"Jolly impulsiveness" is perfect. Since so many people associate "gay" with sexuality these days, its use as a synonym for "jolly" or "happy" has declined. Whoever talks about the rarity of "gay abandon" in the finance department is definitely trying to be funny.
Yes, approximately. It's become a stock phrase, although it's becoming increasingly avoided because of the connection of the word "gay" with homosexuality. Perhaps a better meaning would be "uninhibited enthusiasm".
In Oz it means doing something potentially dangerous, risky, or detrimental for self or others without much concern for the consequences, and is only a 'tired cliché' when used often and indiscriminately by same person.
Gay (or carefree to avoid the use of "gay") abandon is, as has been mentioned, a stock phrase or cliché.
What it means with reference to the finance department is less clear. It could be "total disregard for rules" or "decisions made without any concern for the consequences". Or it might refer to the feeling by members of the department that they are not subject to any constraints.
It probably means the absence of complete seriousness in regard to work, which is (hopefully) very rare in a finance department.