Actually, Sophie, we should insist on more context here. But the sentence, "get him good and spooked," suggests that this is your MI5 man again and that he is interrogationg a suspect. If that is the case, then...
... nibble around the edges means ask questions that are not directly related to the crime being investigated, but which might lead to information or reactions that help the investigation. It's a metaphor that, like most metaphors, only works in a known context.
Here's more context. Hope it will help.
A former IRA operative still bears a major grudge against the head of the section D of the MI5. He tells one of the MI5 agents that all of section D must assemble at the location of a bomb he intends to detonate in London. If they refuse, he will make sure that hundreds of civilians are killed and he knows they would sacrifice themselves rather than risk that.
Later on, the former IRA operative says to the man who contracted him to do the deed : Harry (the head of section D) is pissing in his pants. I'm sure he's gathering his team to his bosom right now. As soon as I've run them all to ground, I'll do the deed. (...) I need to nibble around Harry's edges a little more. Get him good and spooked.
The expression is [still] being used to suggest that instead of a direct attack or confrontation with Harry, the IRA-man is intending to "prod" him with smaller gestures, messages and such.
Consider this situation: You wish to force your next-door neighbour to move away.
First you play loud music. Then you park your car directly in front of his house, even in front of his driveway. Then you sneak out an night and dig up all his rose bushes. You might even put a dog turd through his letter-box.
None of this actually forces him to move away, but you are nibbling around the edges to make him feel unwelcome.