I am not sure what you mean. In what sense is the person a guest? Is this someone who is temporarily living at your house? If so then you would say: "I was at a guest`s place yesterday" meaning I visited the house of a guest of mine.
I think we need more explanation of what you are talking about.