Clearly 'a sweep' is regarded as an opprobrious epithet in this context, and I'm not clear of the nature of the insult suggested.
We have one serious, but inconclusive, thread on the subject -
'sweep' as noun [Your sweep's leaving.]
The Urban Dictionary makes its usual riotously obscene suggestions, but includes 'the female in a relationship' -
I'm going to the party with my sweep (or something similar).
Perhaps then the suggestion is of effeminacy, though I incline to the idea that it refers to a 'sweep', a chimney-sweep, so the reproach is about lack of social distinction, added perhaps to juvenility, and impecuniousness.
Maybe it's linked to the word 'scrubber', on which the Urban Dictionary is more helpful:
"Scrubber" is probably of British origin, used as a detrogatory term to describe scullery maids or other low-ranking women in domestic service. These were often young women viewed by higher-ranking people as licentious and dirty (particularly their hands). Due to the lack of occupations for women in the 19th and early 20th century women usually were forced to work in service or as prostitutes, and many had dealings with both. The actual term "scrubber" may refer either to her occupation of scouring floors, or to the actual implement used to clean the floor - the scrubbing brush.
In my experience 'scrubbers' are irretrievably female, yet the person addressed in the OP is probably male, if I am any judge, so maybe that is a false trail. Chimney-sweeps were male and young (6 - 14 or so). They had to be small to be able to get up the confined spaces. It was clearly a terrible job because they operated naked and soot is carcenogenic; cancer of the scrotum was a common disease among the poor children.
I'd welcome further suggestions.