Like Newg, I prefer "time" here to "while" or "moment."
I would say something like We only had one hour for shopping/to go shopping.
However, if the specific amount of time is not specified, I personally would naturally say: We didn't have much time for shopping/to go shopping or, as Newg suggested above:
We had little time to go shopping/for shopping.
I respectfully disagree.
At least in AE, if you say We didn't have much time for shopping/sightseeing/visiting friends/dining in fine restaurants/etc.,
it doesn't necessarily imply that these activities were not on your list of priorities.
It could well be that you would have liked to have spent three hours shopping but only had one hour.
I could especially see saying "We didn't have much time for shopping"
in the context of an organized tour where the structured schedule
allows for only a brief period of "free time" to shop.