"ne sme se" is a general statement of fact, without direct imperative meaning: approximately, it means "one does not" or "one may not". Of course, it does have prohibiting connotations if used in direct speech: when you tell someone "One does not smoke here", well, it implies that he should distinguish the cigarette immediately (i.e. in a more polite manner than "don't")
"nemoj" is a simple imperative: "don't"
"ne čini to" ("don't do that") is close to "nemoj".