The structure of the reply suggests that permission was asked of someone with some authority, eg pupil to teacher. In this case, I think yes, you may would be the expected answer.
Asked between equals, I think you're more likely to encounter alternative ways of granting permission: no problem or go ahead or sure or of course. And if you may was paired with of course, I don't think it sounds pompous. I can imagine saying 'Of course you may'.