I don't know what others will say, but there is a certain awkwardness to the way that words can be put together. Just because they
can be put together in that way doesn't mean that they
should be.

I know that probably sounds elitist, but I mean to say that there are certain combinations of words that just don't flow well.
I don't know that it's incorrect to say, "to John's and my house". It looks grammatically correct to me. I think it might be better to say something like "Would you like to come over to the house John and I share?"
I also wonder why you're avoiding simply saying "our house". Is there a reason to communicate that you and John share a house but do not have anything else in common? If so, "the house (that) John and I share" or "the house that John and I are sharing" gives that information to the listener. Otherwise, I'd just say "our house."