At or by my side mean physically there beside you. "On my side" means she supports you morally. You can't use them interchangeably, as "standing on my side" means she is literally standing on your body!
If I tried to show a difference between at or by your side it would be possible, I think. By your side is a colloquialism used by a lot of speakers to mean "on your side." We could not use "at" in the same sense. In this usage, there would be a difference, but only bedcause "by your side" is different because of usage.
What do you feel when use this colloquialism and say something like "He is by my side"?
1. He is beside me and helps me; the spatial sense of "by". Or,
2. He acts by my authority, and therefore helps me if I want him to; the authoritative sense of "by" (as in "by law", "by the rules").