I think you could say either. In this example it really is the number or quantity that does (or doesn't do) something - it's not just a synonym for '(not) many' - but also of course the bacteria do (or don't do) omething.
If you reduce the subject of the verb to its minimum, you would get: "Such a small number of bacteria (does/do) not cause food poisoning." Such, small, and of bacteria are adjectival to "number." Number is singular, therefore "Such a small number of bacteria does not cause food poisoning."