What is your relationship to the driver? Is the driver hired? Is he or she a friend of yours? Are you driving around your own town or somewhere else? Is the purpose of the drive to see the sights in a random manner or is it simply for the pleasure of riding in the car? Are you and the driver the only occupants of the car or do you have a group of people with you?
Driving aimlessly around in a car could be cruising or taking a joy ride or just taking a drive/going for a drive.
I'm assuming that you and the driver agree on what you are doing. The way you've described someone else doing it to you makes me want to suggest "poorly planned kidnapping".
We used to say "go for a ride", when the point was to travel in the car, maybe looking at the scenery, rather than to arrive anywhere in particular. It means the same thing as AngelEyes' suggestion, but it's less formal.