Yes! You get some very strange looks if you went around saying that! To be 'turned on' by something only really has sexual connotations. The only other way to 'turn on' would be for switching something on (i.e. 'turn on the TV') but that has a very different meaning.
The term "turn on" does not exclusively to suggest sexual connotations, but it does suggest some kind of excitement as a response to the activity, so perhaps (without wishing to upset any keen fishermen and fisherwomen out there) fishing is not a good example.
Going to the theatre turns me on.
Being at a grand prix race turns me on.
I'd say that the one could possibly say something like "I love fishing; the moment I understand I've caught a big fish, the struggle to reel it in, the combination of sheer physical force and dexterity needed not to lose the fish turn me on".
Mind you, if you ask me, our scaly cousins should remain where nature put them since I prefer my meat to walk or fly prior to its taking the road to dinnerhood.