A task can be daunting or intimidating, and they mean roughly the same in that case, but a person can only be intimidating, not daunting.
Edit: just realised i didn't actually answer your question. I think "The prison was intimidating" sounds fine but "The prison was daunting." sounds wrong. I'd like to hear others' opinions on this, but I feel like "daunting" can't be used for objects either, unless a task is implied. E.g. you could say "The new house is daunting" if you were talking about the huge task of decorating and repairing the house, but if it was just a house that looked scary, you couldn't use the word daunting.