Following on from Diabolique's point: because "It saddens me" is usually associated with with mature speakers, it is usually associated with more serious and far reaching topics, so one is more likely to say, "It saddens me to see the state of the environment today" than "It saddens me to miss the party"
However, this is more of a common practice than a hard-and-fast rule, so they can be used interchangeably.