It is usually understood that "three days" here is representing a length of time or a period. "(A period of) three days isn't long enough for a good holiday." If you change it to are you would be talking about the length of each individual day. I suppose it's simply an idiomatic quirk of English.
(Slow to type today... I saw SoundShift's post after finally posting mine. Well, you have confirmation, at least.)