Right to privacy is correct. If you know the exact difference between right to and right of, you know more than I do.
Perhaps right to is more concerned with individual or moral rights. We all have a right to life.
My father left me that money in his will, so I've got a right to it.
Perhaps right of is more of a legal term. You have the right of way when the traffic lights are green.
It's a private road. The public have no right of access. He was found guilty but he has the right of appeal.
Right to property or right to freedom of speech is legally enforceable. So they are legal rights. If I remember correctly, I have seen Right OF property and Right OF freedom of speech used by some careful users of the language.
A right of something is a general concept, established in law or by treaty, not an individual right. A right of property, a right of inheritance, a right of freedom of speech, a right of way, a right of appeal, etc are rights established in law. A right of free passage is a right established by treaty or international agreement.
An individual citizen benefits from those statutory rights, so has a right to property, a right to his father's estate, a right to free speech, a right to benefit from a right of way (which becomes "it's my right of way"), a right to appeal, etc. A ship or an aeroplane is entitled to make use of a right of free pasage, so it has a right to free passage.
You will, of course, see examples which don't follow this distinction. A right to something can also be a general (perhaps nebulous) concept - eg a right to life. That fits in with rhitagawr's point about individual and moral rights. I say nebulous because we also have a right to death, and that trumps any right we might have to life.