After thinking about it a bit, I noticed that activities that end in an -ing are usually preceded by "to go" when you intend to go do that activity...
- Go swimming
- Go fishing
- Go bowling
- Go riding a horse
And logically it makes sense because it's just like saying "I'm going (to the pool) to swim" or "I'm going (to the bowling alley) to bowl" or "I'm going (to the lake) to fish". This is exactly why I believe we use "to go". "Bowling" primarily represents the act of playing this particular game, not the game itself (necessarily), as evidenced in my explanation. (Assuming that's true, then I don't believe there is a word solely for the game itself.)