To dig, in this sense, is a figurative use of its meaning of "to prod someone/something" and is commonest in "to dig someone in the ribs." This means to poke someone forcefully (often with a finger or the elbow but also anything pointed but not sharp) so as to annoy or provoke them (or sometimes to attract their attention.)
This meaning is somewhat different from the usual meaning of "dig" in, e.g. "He dug the garden with a spade."