I recently finished my studies ...
I have
recently finished my studies ...

(note the word order)
Both sound fine to me. As a teacher, present perfect is probably more correct, but as a native speaker I would probably be more likely to use the past simple. I would only use "I have recently finished my studies" if it was very, very recently.