You don't need a comma for phrases that come before the subject, but it often sounds better with one. Here, however, we definitely should use one, because the first phrase could be misread as taking an object - in two ways, in fact:
(1) Instead of answering, her mom glanced up into the rear view mirror.
(2) Instead of answering her, mom glanced up into the rear view mirror.
(3) Instead of answering her mom, glanced up into the rear view mirror.
I've used a comma in all three, to show what the possibilities are. Without a comma, it could mean any one of these three. In (1), her mom didn't answer, but it doesn't say who she didn't answer. In (2), mom didn't answer her. (We can use just 'mom' to refer to our mother. 'Mom' is AmE, but the same is true for 'mum', 'mother' etc.) And in (3) someone didn't answer her mom. This last one is ungrammatical, because now there is no subject for the main clause (who glanced?), but it's still easy to read the sentence this way and get stuck if there's no comma.