I suggest that you use "by" to talk about something that frustrated you: I was frustrated by her refusal to answer my question.
It is pretty common to use "with" if you want to say that a person frustrated you: She was frustrated with her son because he wouldn't clean his room. It is also common to use "with" in talk about things, but it is especially common in talk about people.
"At" and "over" are also possible, but I don't think they are necessary. It probably doesn't make sense to worry about these two because "by" and "with" should be enough to express most of your ideas.
"For" would be useful if you wanted to talk about the length of time during which you felt frustrated: I was frustrated for hours before I finally figured out how to fix the engine.