Čarobnjak is way more common. Nowadays, Veštac is rarely* used to mean čarobnjak. I would say it is very rarely used with this meaning.
I believe that čarobnjak can be anywhere on the spectrum from good to bad, while veštac is normally seen as a bad person.
There is a phrase stari veštac which means "an old stager", "an old hand", someone with a lot of experience and expertise, a seasoned person who is hard to trick, but can easily trick you.
*For me, at least, as I don't see the two words as synonymous.