We've had a very similar conversation about, I think, men scratching their noses.
As I recall, either singular or plural is accepted, depending on context. The difficulty with the singular hat is that it would have to be very large.
One solution this time around is to shift to singular.
When conversing with women (or with a woman), a gentleman will remove his hat.