Actually, Sophie, we should insist on more context here. But the sentence, "get him good and spooked," suggests that this is your MI5 man again and that he is interrogationg a suspect. If that is the case, then...
... nibble around the edges means ask questions that are not directly related to the crime being investigated, but which might lead to information or reactions that help the investigation. It's a metaphor that, like most metaphors, only works in a known context.