Your context is a little confusing. There's "I", "an interviewer", "a woman", "the interview" (is that the interviewer or the interviewee), "him" and "he". How many people are there?
It doesn’t strike me as particularly rude. And “don’t include me” is a natural phrase, but too vague for this situation, unless you explain what you don’t want to be included in.
But, as I said, I’m not convinced that his comment was meant aggressively/rudely. Could you have misconstrued it? Normally, if you say that you want to hear someone’s opinion, it’s a compliment.
I think "Everyone but me!" or "I didn't ask to hear him/her." could work if said in a jokey manner, otherwise they themselves would be construed as being rude.
You could use the famous line attributed to (amongst others) Samuel Goldwyn: You may include me out.