Actually, it should be spouse, not spouses. 'Neither of us' is singular, so only has one spouse and one family.
"Neither of us is leaving our spouse or family"
"We are both staying with our spouses and families"
Obviously this quote comes from a polygamous culture
The sentence is a mixture of singular/plural:
Neither of us - usually singular
is - singular
our - plural
spouses - plural
family - singular
If each of us didn't have plural spouses ...
My guess is that our is used to avoid a sex-specific pronoun. That has persuaded a plural verb. So why isn't it:
Neither of us is leaving his spouse or family;
Neither of us is leaving her spouse or family.
Because us consists of one man, one woman.
Perhaps they are having an affair, but have decided to keep it quiet and remain with their spouses and families.