Modifiers are not the same as the thing they modify. You may as well say that "car" cannot be modified by "green" because "car" is not a colour..
The subject is "your duty". You could have a number of duties in different situations, so a modifier is used to say which situation applies in this instance. Your duty as a good citizen is to obey the law. Your duty as a member of this forum is to write in correct English to the best of your ability. Your duty as an assembly worker is to check products for defects.