People who knew that my children were from my first wife rather than my late second wife would sometimes say something like "Did you take the children?" to her, because "your children" would imply that she was their mother. She thought of them as hers in every way except biologically, but saying "the children" avoids a potential difficulty. There was never any confusion about which children were meant.
I don't know who 'the wife' is. It was your example. It's rude because it's a very dismissive way for a man to refer to his partner, as if she were part of the furniture. I suppose the same would be true of 'the husband', but I've never heard a woman say that.