I don't think that with necessarily implies that the glasses are in his hand. With could mean that they are in his hand, but in a conversational context I think it would be clearly understood that the man was wearing them.
Person A: "Which man do you mean?"
Person B: "The man with the dark glasses."
I think the average person would understand that the man was wearing them. I would even go as far as to say that if he were holding them, it would be mentioned specifically: "The man with the dark glasses in his hands."
To be completely clear though azzurri you may say,
"The man with the dark glasses on" or "The man wearing the dark glasses"