Somebody has to run the thing before recommending it

Every directory in this space counts. None of them runs anything.

Checked on 14 August 2026, the largest catalogues of agent tooling describe their own method in the same terms. One states that its crawlers sweep the registries on a schedule, refresh stars and install counts, and that an editor reviews borderline listings — and nowhere claims that anything is executed. Another lists 22,070 servers and publishes no methodology page at all. A third compares eight marketplaces and finds that not one of them claims to verify a listing works; the strongest guarantee on offer is a static security scan before publication. The reference benchmark for model performance publishes a scope statement with no reproducibility policy, no raw data commitment and no conflict-of-interest disclosure anywhere on it.

Stars, installs and search demand all measure how many people found a thing. None of them measures whether it still works. That gap is the entire reason these nine indexes exist, and it is why the first figure on this page is a failure rate rather than a total: a count of what is out there is a number anybody can produce, and a count of what is broken is a number somebody had to go and check.

Toolproof is the masthead over the nine, not a tenth index. Each one keeps its own name, its own site and its own engine. What is published here is the one thing they did not have separately: a single method they are all held to, stated in advance, with the conflicts named. The methodology page writes it out before it says anything else.