"At first sight" without an object, just by itself, is an idiom. As such, the phrasing is more or less fixed. When it comes to idioms, the grammar rule and sense are more flexible because everyone is just used to hearing that specific phrase.
"At the first sight of X" in my example is not an idiom, not a "saying." The article would then be used because people wouldn't be thinking: "that's not how the phrase/idiom is supposed to be."