It's grammatically correct with or without the comma but, as Paul says (post #2), there's a subtle difference with the pause indicated by the comma. I would say that it might suggest that the speaker is not sure that the other person would want the speaker's company for the visit to the hospital.