I wonder if a time-element would not make for a better distinction. Or even better: 'with' denoting not only a temporary, but also a provisional
association.
'I belong with them while their mother is sick'.
Although I belong to the Military, right now I belong with them.
It seems to me that more permanent or institutionalized associations are always 'to'.
So would be, in most cases 'I belong to those who tell the truth', although a situation is possible where it would be with'. Imagine all those who do
tell the truth are being rounded up to be punished. Your friend asks you
what you will do. You decide to stick to your principles. 'I belong with the
ones....