The difference is not in the chat but in the type of conversation you want to use this word in.
The speakers, the setting, the tone of formality/informality will all influence the choice.
A brief chat with your employer
a little chat with the cashier at the supermarket
just two examples