Bantu languages are painfully difficult, from my experience. I live in South Africa and have studied Xhosa (an nguni / southern Bantu language) for more than 4 years. I'm nowhere near fluent, after a year of studying a Romance language (French) and a Germanic language (German) they're nearly at par with my xhosa!
I take your point, Loopin, and it's true that other East African Bantu languages like Kikuyu, Kikamba or Luganda can be incredibly difficult to learn.
But Swahili started out its life some 8-900 years ago as a creole between the coastal Bantu languages and Arabic. While I've never studied the historical development of the language, my assumption has been that this had an overall effect of simplifying the phonology, losing tonal structure (Kikuyu, for one, has tones) and adopting a simple penultimate accent on almost all words in the language, and also losing all diphthongs. The phonetics of Swahili are about as simple as Japanese.
The grammar has simplified too, losing all but about five noun classes. The verb forms are largely regular for all verbs.
Another factor involved is that the vast majority of Swahili speakers speak it as a second language. Except for the Swahili people along the coast most East Africans speak their tribal language as their native language. This has tended to simplify the commoner expressions, and there is a well-known gulf between what is considered "proper" Swahili, suppsedly based on what Swahili people in Zanzibar (and to a lesser extent Mombasa) speak and what you will generally hear on the street in Nairobi.
I could give some examples but I'm getting off topic. Not everybody knows that Swahili has an extensive medieval literature, mostly in poetry and written in the Arabic alphabet. It's an interesting language to learn.