Grammatically "key of the door" is correct. It is not about location of the key but about the key that belongs to the door, this makes it possessive. Another example would be "the father of the bride", in other words the "bride's father". In this case using the 's is acceptable, but when two inanimate objects are being discussed the 's can't be used. Think about "the car's door". This is incorrect as both car and door are inanimate. Therefore you have to say "the door of the car". Likewise you cannot say "the door's key" (two inanimate objects), so it has to be "the key of the door".Shouldn't it be the "key to the door"?