It really depends on context, but you could certainly say 'the number (being) called' and 'the number of the caller' respectively.
'Called/calling number' could work, but I'm not sure the meaning of either would be readily apparent unless the terms were situated in a particular context which makes it clear.