I have not seen 'a followed site' many times. I think 'a followed man' would be idiomatic, but as for a site, I wonder if 'a site that you follow' would be more idiomatic.
I think that it would be natural to follow some thing that moves with its own will, but a site would have no will that drives itself. So, I, once again, must say that 'the site you watch' or 'the site you keep reading' would be more idiomatic.
I don't have a problem with a "followed site" and neither does Microsoft, apparently. I don't see "a followed man" being idiomatic; in fact, a Google search shows only 32 results, three of them entries in English language forums.
Since we can follow a sports team, I don't see any reason not to follow a site, especially one that routinely changes its content.
There is no reason to think that one particular construction can work for other similar-looking constructions. I would be surprised if that's not also true for Japanese, but even if it's not, you will only confuse yourself and slow your learning if you attempt to make up rules that don't work or try to bend a language to fit your wishes or beliefs.
"A followed site" is something you're not going to see very often – in this case, it's specifically for a computer situation and it can be thought of as shorthand, so I wouldn't be too concerned about whether or not you should use it in normal conversation or writing (you wouldn't).
I'm not at all annoyed or irritated – but I did make my study of Cantonese easier by deciding early on that I would just learn what native Cantonese speakers said and not worry about why they said it or why there were occasional exceptions.