At least in the U.S., a phrase like "of a Sunday" is generally used only in texts that are very old, very literary, or both. It's an idiom, and it's perfectly grammatical. I've NEVER heard it used in speech. I don't know how common it is in British usage.
The regular way to say it would be "He would often come to see me on Sunday" or "He would often come to see me on Sundays."