I think that with this defining relative clause you should use "that", or the pronoun may even be omitted. I think you are right to be uncomfortable with your placing of "in the washing machine" so very far away from "washed".
There is still a problem though, with your second version. Your mother didn't buy you the washing machine, did she?
You have put a lot of information into that one sentence, but you can remove a few surplus words:
"If hadn't washed that jumper (that) my mother bought me for my birthday in the washing machine, it wouldn't have shrunk."
Or you could say "If hadn't machine-washed that jumper my mother bought for my birthday, it wouldn't have shrunk."