It can’t. I don’t think you can ever describe a building as windswept. It might be possible for an accomplished writer to make it work in a very specific scenario, but in general the word is simply not used that way, for obvious reasons.
Windswept describes climatic/environmental conditions over many years. It's sort of as semi-permanent condition. And it is generally used as an adjective before a noun - windswept hills, windswept plains, etc.
The Dust Bowl forced thousands of farmers to abandon the windswept plains of the central United States.
What I think you could have, on top of a hill, for instance, is the windswept ruins of a castle. It's been sitting there abandoned for many years, exposed to wind and rain and nature. You couldn't describe the individual rooms that way but you could describe the whole thing that way. At this point, it's part of the landscape.