Here's the definition of "give up on" in the Collins COBUILD Phrasal Verbs Dictionary:
If you give up on someone or something, you stop trying to do something that involves them, because you think you will never succeed or understand them [...] NOTE: abandon means almost the same as give up on.
Whether or not it's "wrong" to say "I gave up on the race" would depend on what you are trying to say with the sentence. I can think of situations in which it might work. For example, let's imagine there was a once-a-year car race through the streets of your town. You went to watch it, but it was pouring with rain and you got fed up because you couldn't work out who was winning and who was losing. You might say "I gave up on the race and went off to the pub for a drink".