I am not that good at Arabic yet, but let me answer as I see it: if I am not mistaken, no you can’t use the indefinite relative pronoun ما in that sense. ما is used with verbs (in perfect and rarely in imperfect tense) only to make them negative. But you can use ما in the sense of “certain” or “some” in appositive construction (بدل) after some noun (this first noun will be in accusative case), as in:
نَوْعاً مَا - somewhat, to some/certain extent;
يَوْماً مَا - some day.
Note: of course, if there is some preposition before نَوْعاً and يَوْماً, these 2 nouns will change their cases to genitive (also called oblique).
It can’t be used in a similar way to لا طالبَ في الفصل, meaning that it can’t replace لا النافية للجنس. But it can be used in the same way as لا طالبٌ في الفصل - it can replace لا النافية للمفرد.
The sentence would be ما طالبٌ في الفصل.
In this case you can also use إنْ as in: إنْ طالبٌ في الفصل.
All three mean and act in the same way that ليس does.