People come and go in numbers, not in amounts, usually. A large number of people are leaving... It is the people that are leaving, not the number. However, 'number' can be seen as the grammatical subject so I would not be shocked to hear someone saying 'A large number of people is leaving'.
'Are'. You're interested in the people, not the number or amount. It's only the people who are leaving. There are some words where either is possible, such as 'majority': a majority of senators are in favour of the bill. (The senators, the people, are in favour.) A majority of senators is needed to pass the bill. (As a matter of law, the number of votes needs to be a majority: a majority is needed.)