broke away from the rest of the group is the correct form in this sentence. They separated and moved away from the other people in the group.
broke off is used when something is abruptly separated (literally or metaphorically).
In the high winds several branches broke off the tree.
After a big argument, John broke off his friendship with Jenny.
You can also use “break off [from]” to mean breaking away from a group etc.:
The city formally broke off from King County when the citizens voted on February 28, 1990, to … take control of its … future development.
American psychology broke off from its European roots between 1890 and 1910…
The Messenians break off from the Achæan association, and take up arms.
Meanwhile, ordinary people in America, anonymous people from out of the masses—known today, as Jean-François Revel observes, as “the middle classes”—were breaking off from conventional society, from family, neighborhood, ……