I think "to be better off" has to do with conditions being better, as opposed to "to be better" which has to do with...being better?
"I am better off than you.": This means that I am in a better situation than you. A lot of times, this has to do with money like "homeless people suffer more than those who are better off."
"I am better than you.": This means that there is something about me that makes me better than you (perhaps in terms of intelligence, talent, personality, etc).
Better and worse are for specific traits, whereas better off and worse off are for overall, general states. I'm becoming a better runner (specific). Since moving to Toronto, I'm better off (everything is a little better on average).