The metaphor comes from old farmhouses and settlers' cabins with wooden floors. Inevitably, holes would appear as knots dropped out of the surrounding wood. The holes were called "rat holes" because rodents used them for coming and going .....
Thus, "pounding money down a rat hole" became a phrase meaning that it was money wasted.
This may be a regional phrase, too. In my area, you would be far more likely to hear someone 'is throwing money away' or 'the money's going down the drain', or even that someone 'is pissing money away'.
The only phrase I regularly hear related to rathole is 'that place is a rathole', meaning it's a mess, or it's nasty.