Actually, the difference between "I swam" and "I went swimming" exists in French too. "J'ai nagé" means "I swam," so presumably there's some other information about you being near a body of water elsewhere in the sentence. But "J'suis allé nager" means "I went swimming," so I would assume that you went to a beach, a pool, etc. for the express purpose of swimming.
Nb. French actually follows the same "to go X'ing" pattern for all verbs. There are actions (hiking, dancing, running, biking) that we normally think of as happening not in the normal space where we're located, actions that we have to "go do." French has the same distinction for all these verbs.
So, just as you would say "I was at the beach all day Sunday. It was awesome - I swam, I read my book, I got a great tan" in English, you would use "j'ai nagé" in that context in French.