The idea of karaoke venue which is not also a bar is almost inconceivable in Britain, and I would not use the term "karaoke bar" to describe such a place here. However, a "bar" does not necessarily have to serve alcoholic drinks; we have "tea bars" and we used to have "temperance bars", although neither of these would be described simply as "bars".
What would you call the place if it did not have karaoke? My guess is "karaoke cafe" would be most apt. After all we have (or used to have) "internet cafes" which were primarily about internet access, and the "cafe" part was incidental.
The quotes and most information in this post are from Wikipedia's article "Karaoke".
"In Asia, a karaoke box is the most popular type of karaoke venue." These boxes are rooms you rent to do karaoke. Some places have many boxes and also serve food. Some hotels have boxes.
I've seen these on Korean TV shows. In Korean they are "song rooms". Each is a small soundproof room with sofas, a large video screen, and the karaoke equipment. Typically a group of 2 to 5 friends rents the room.
In Europe and North America, a "karaoke bar, restaurant, club or lounge" has a small stage (sometimes) and karaoke equipment. In those places, everyone in the bar or restaurant hears you sing: there is no separate room to rent.
"Private karaoke rooms, similar to Asia's karaoke boxes" are common in Toronto, LA, Chicago, NYC and San Francisco.
In my opinion, karaoke is much less popular (in recent years) in the U.S. than it is in China, South Korea and Japan. I can't comment on UK countries.