He is dead serious.I tried looking for an online source (please don't post a link in this thread -- it's against the rules without prior permission) to listen to the episode but couldn't find one. Perhaps someone else watched the episode.
The only other interpretation I can think of is that he's being ironic. He might be saying that it's pitiful to be a non-felon and he's supposed to hang out with non-felons and share their pain for living such a boring life. It's a stretch, but it's possible. Was he smiling when he said it?