To go out to dinner usually means that one is invited somewhere, so it's not something one can spontaneously undertake.
To go out for dinner usually means that one decides to go out, rather than stay in and prepare food at home,
For these reasons I'd be more likely to say 'Let's go out for dinner'.
Invitations are generally for an activity or to an event or place. Meals usually use "for" in modern English. unless it is a formal occasion, in which case it would probably be given a name or a description. This does not mean that "for" is used with every verb, and you can certainly go out to dinner, as has been explained above. I have a feeling that "to" used to be more widely used than it is now.The "over" examples mean the person was invited to Tom's house, right? While the "out" ones imply a restaurant, right? If so, which forms do you prefer? "To dinner" or "for" dinner?
Tom invited me over to dinner and don't know what to say.
Tom invited me over for dinner and don't know what to say.
Tom invited me out to dinner and don't know what to say.
Tom invited me out for dinner and don't know what to say.