In place of "love" I would probably put "feelings," but otherwise those two words can replace "affect." They do imply a slightly different meaning, however: if it "affects" your work, it is merely a factor, one of many, but if it governs (or rules) your work, it is the deciding factor in your work, more important than anything else.
"Affect" is much broader that "govern". Affect means to produce an effect or a change on something. The exact way that happens is open. The lover could be distracted, angry, happy, sad, always on the phone, always writing IMs, away from work, leaving early, coming in late, etc. "Govern" has a sense of controlling, setting direction, ruling, or regulating. There is a little overlap, but I would not use "govern" or "rule" in the context given in this sentence.
Right! Your context does not indicate that some rule or regulation can intrude on your love life. You ought to be able to find other substitutes for affect. Govern or rule would give the wrong impression.