Can you in all honesty say that there is no word that someone might call you that would NOT be offensive to you? If that's true, then I believe you are in a very small minority.
As I just wrote (after your post) obviously some words carry a specific and negative meaning. If you're Jewish and I call you the 'k-word' then obviously that's a problem, because the word has an obviously strongly negative meaning. "Guys" does not. There's close to zero ambiguity when using the 'k-word' whereas "Hey guys..."
I mean, take the aforementioned "'How you guys doing?'" and replace "guys" with the racist slur I mentioned. Not only is that not in the same ballpark, it's not even the same game.