There are lots of threads about gerund infinitive that discuss this topic in relation to specific activities. Usually they are about settings where either form can be used but there is a difference in meaning.
In this particular example, only the second form is correct in normal contexts.
The funny (perhaps) thing is that if you select a different activity, where you don't need a verb-noun combination, then both forms are fine, if a little brief.
I went surfing.
I went to surf.
I went singing.
I went to sing.
I went playing golf.
I went singing madrigals.