Hi,
I am a Mandarin speaker from Taiwan, I hope I could provide you some observation.
But I only focus on 'A-not A' and 'option-related' interrogative function of '嗎' and '呢', and ignore other functions they might possess.
The following sentences are both eligible and only have subtle difference in meanings to me and to my friends I inquired.
A. '現在有沒有新的訪客呢?' -> soften than B
B. '現在有新的訪客嗎?'
1. In A, '有 沒有'-> Exist or doesn't exist -> provide two options -> you can answer '有' or '沒有'
In B, '有' -> Exist -> provide only one option, and the other possible option need to be inferred by your self -> you can answer '有' or '沒有'
2. '有沒有', as in A, can only go with '呢' but not '嗎'
'有', as in B, can only go with '嗎' but not '呢'
The difference lies in that '呢' goes with questions which explicitly states out 'all possible options', while '嗎' doesn't.
'嗎' is the real 'A-not A' question marker, having only two options in meaning, although it only states out one option, and the other is omitted.
Therefore,
C. (O) '你喜歡小美, 小玲, 或是 小橋 呢?' -> you ask the speaker to 'pick one' from the options you provide (but of course you can said 'none of the above')
'Do you like girl A, girl B, or girl C?'
D. (OX)'你喜歡小美, 小玲, 或是 小橋 嗎?' -> Most of my friends find it weird, but they accept this one under one condition that the listener view the
sentence as a condensed sentences consisting three independent Qs (你喜歡小美嗎?你喜歡小玲嗎?你喜歡小橋嗎?)
In English this kind of condensed sentence is like 'I gave Mary a book, John a coke.' to omit the repetition.
However, when we change '或是' to '還是'.
E. (O)'你喜歡小美, 小玲, 還是小橋 呢?' -> eligible and same with C.
F. (X)'你喜歡小美, 小玲, 還是小橋 嗎?' -> not eligible, because '還是' imply that 'you need to pick up one' , and thus rules out the the possibility of the
interpretation of condensed sentence as that in D.
I hope they make sense to you. Although I am a native Chinese speaker, I am interested in the hidden rules that we're not aware of.