"I invite you on an adventure with me" really means "I invite you to go on an adventure with me." But the phrase is so common in English that in modern English you can omit "go on", and just say "I invite you on a (trip, adventure, outing, picnic, journey, expedition, voyage, etc.)".
So the sentence already has to in it, but we can omit "to go".
"I invite you to an adventure" is the same syntax as "I invite you to a party." That is acceptable.
BUT we can't add "with me" to that sentence. Even using "party", if we say "with me" we add "go": "I invite you to go to a party with me."