It doesn't need a comma, but I was imagining a scenario in which a comma would be acceptable. If the "it" refers to something in a previous sentence then a comma would work (with "if" replaced by "whether"). For example:
A: "I want to change the logo."
B: "I don't think it'll make any difference. Brand familiarity is not that important to us."
A: "To me it makes a difference, whether brand familiarity is significant or not"
However, I feel I may be straying from the point. In the original sentence the comma does not work at all.