"Set off" has many meanings. www.thefreedictionary.com has a listing that shows seven different meanings. We use context to determine which sense of "set off" is meant. There are key cues to determining that you mean "to start a journey or trip". Two of them are "on" or "for".
We set off for London.
We set off on a trip.
We set off for Europe.
There are other possible hints that you mean "to start a journey":
We set off over the snow.
We set off with suitcases in hand.
We set off under power.
If you take this verbal "hint" away, the meaning becomes ambiguous. "We set off a journey" might mean that you instigated a series of events that caused someone else to take a journey. It would be an odd way to say it, but it's possible. This would be meaning 1a ("to give rise to; cause to occur") on the Free Dictionary website.
Since it's missing the cue we're expecting, it now has to be re-interpreted for what particular meaning of "set off" you're using.