Juliamay2, please remember that we need to know the context for this Forum. (Where will the sentence be used, and with what meaning? Who is the speaker and who is being addressed? What comes before and after?) See Rule 3. See also the English Only guidelines. Remember also to consult the dictionary and search for earlier threads on the topic.
Venue is correct here, because you think of the building in which the concert occurs (eg it's a beautiful venue for a wedding ceremony). Site is vaguer and suggests the area or location of something rather than the actual building (eg the site of the Battle of Hastings).
"Lots of people gathered at the venue where the concert took place." is awkward.
A venue is, by definition, where people come [to see something [take place]].
I would accept, (i) "Lots of people gathered at the concert venue." or (ii) "Lots of people gathered at the venue." but, in this case, the purpose of there being a venue must have been presented earlier.
"Lots of people gathered at the site where the concert took place." is fine. "the site where the concert took place." = venue.
I suspect PaulQ may be right about the proper use of "venue," but neither indigospooky's sentence about the "beautiful venue for a wedding ceremony" nor the original sentence about "the venue where the concert took place" sounds off to me. The loose use of "venue" as "place where something happens" may be becoming more common, though I have no particular opinion on whether this should be decried or not. It's not a word I use much.
For what it's worth, I would refer to a concert's "site" only if it were outdoors. Otherwise, I would probably talk about the hall (or theatre, or bar, or club) where the concert took place.