If there were 5 men at the meeting - there were 10 women.
"I have twice as man complaints about Howard as I have about Fred." That means that there are two times more complaints about Howard as complaints about Fred. "Twice as many" means, "two times as many." If the boss got 20 complaints about poor Howie - he had 40 about inept Fred.
Oops! It should be noted, by his saying, "I'd say," it means he is giving an approximate perception of the number of people at the meeting. In this case, if there were 5 men, there'd be anywhere from 8-12 women. Or thereabout. "I'd say" implies an estimate.