Yeah, ok, I see your problem now.
Authority in the sense that I'm using it -- and in the sense that anyone who's informed about these matters would -- does not mean "agency". The problem that constitutions are intended to solve is what members of a national community have the legitimate right to rule over other members in it -- that is, who in a society has a monopoly on the legitimate use of coercive force in order to enforce justice, property rights and so on, in that society? In a monarchy, the monarch is vested with that exclusive right. In a constitutional republic, free citizens consent to a document that outlines the various positions of a government which operates as the sovereign power. That document specifies the limitations of privileges associated with each office, the way people come to assume those offices, and so on, and it's called the constitution. The American Constitution is authoritative within the American political system because Americans, as I said, consent to it, and give it that power, which is the basis of its legitimacy as a document: they recognize it as the supreme basis which invests the various offices of the federal government with their authority.
So authority here has to do with who has the legitimate right to exercise power and rule over others -- not simple "agency". The electoral college is legitimate (i.e., it is not a sham) from a constitutional perspective because the constitution grants it the authority to select who the president is. The fact that the way the institution now operates is not the same as what it once was, or that it's not how the founding fathers intended, again, from a constitutional perspective, is not relevant.
Whatever Alexander Hamilton, or whoever else, wrote in the Federalist Papers isn't part of the US Constitution. It's not binding, and it's only potentially relevant if some issue related to the electoral process comes up in a federal court and an originalist judge decides to let it inform his or her opinion. But the mere fact that things have changed doesn't undermine the constitutional authority of the electoral college to select the president, even if there are very compelling reasons to question whether the current electoral system is a good or fair way to select the president.