Perhaps because the Middle Ages were a longer period of time made of other divisions of time during which many rulers came and went. It is also used to refer to the history of all of Europe, northern Africa, and the Middle East. The specific dates of the "middle age" of each country or region differed.
The Victorian Age is a shorter period of time during which there was only one monarch, Victoria, and it only refers to the history of England and the countries connected to England culturally.
Queen Victoria's reign was very long and it was at the culmination of her reign that England had her Empire (Imperialism). It was not until after she passed that the Empire began to break up. England had China, India, Africa and the Commonwealths during her reign. QV also had a huge impact on social morals during her reign because she was very in love with her husband.
Victorian Age simply represents her reign and its lasting power after her death. The Middle Ages spans centuries, without a specific reference to an historical personage: on the other hand, as a subset of the Gothic period (evolving from the Dark Ages) there was the Age of Charles Martel (Charlemagne) which represents the time span of his reign.
"English is one of the few languages to interpret "Middle Ages" as a plural; the equivalents in most other languages, as Medieval Latin medium aevum, French moyen âge, German mittelalter, and Italian il Medioevo are all singular." Maybe this was to avoid confusion with middle age or mid-life which was first used in the fourteenth century.(Mavens' Word of the Day)
In the sense of a long time individual speakers are divided between using the singular and plural. For example, It's an age since I've seen her and It's ages since I've seen her.
In respect of the Middle Ages I can only think that speakers think of there being several ages within that period (different dynasties, changes in society etc.). What happened in the Bronze Age? Well, one thing only - people used bronze and started to compose some epic stories!
"The plural form of the term, Middle Ages, is used in English, Dutch, Russian and Icelandic while all other European languages uses the singular form. This difference originates in different Neo-Latin terms used for the Middle Ages before media aetas became the standard term. Some were singular (media aetas, media antiquitas, medium saeculum and media tempestas), others plural (media saecula and media tempora)." (Reference.com)