I don't think we use "not to speak of " in the same way as "not to mention". We do say "to say nothing of":

*John can do arithmetic, to say nothing of calculus*. I'm not sure that this is a good use of the expression, because there isn't anything extraordinary about being able to do arithmetic.

*John is only six years old, and he can do arithmetic - to say nothing of/not to mention algebra and calculus. That child is a genius.*

"To say nothing of/not to mention" means "and in addition", so there is an emphatic, cumulative effect.

Thank you for bringing 'to say nothing of' into the discussion.

Can you say:

3.. John can do arithmetic, not to mention calculus.

1. John can do calculus, not to mention arithmetic.

What's the difference between 3 and 1?

What about 4 and 5?

4. John can do arithmetic, to say nothing of calculus.

5. John can do calculus, to say nothing of arithmetic.

What's the difference between 3 and 1?