To defer is to trust someone else (an authority) more than yourself on an issue. It is a sign of integrity to sometimes trust someone else more than you in this way and to bet on their judgement over yours. A TIPAESA structure explicitly defers to some authority in its last resort, e.g. by attribution. Knowing where we defer to others is the basis of command hierarchy.
To refer, by contrast, is to invoke knowledge (which might come with its own source and thus authority structure). To infer is to invoke abstract reasoning methods. Neither of these involve deferring to some body.
Typically repute must be assigned by, or audited by, some authority. A credit rating is the most obvious example. Other examples such as the approval of candidates for a political party by its leader, or priestly hierarchy, e.g. Vatican.