I've heard both, or at least, both sound fine. Using the preposition to my ear makes it sound more colloquial, or more "nag-like", somewhat more intense.
I see it's been touched on but, "nag at" to me sounds pretty informal "She was nagging at me" conjures up images of a trailer trash couple (I don't mean to offed anyone, that's just the way I perceive it).