An open-source tool that turns scattered public documents into structured, sourced, comparable scorecards.
The Ordinizer is a software tool that generates record-based scorecards for public viewing on the web. A record-based scorecard is a structured evaluation assembled from what an entity itself has put on the record — its enacted laws, its filings, its own published materials — with every score citing the passage it rests on. A scorecard is meant to be more useful than the raw data it draws on: it selects the questions that matter and arranges the answers so a reader can act on them, rather than leaving them to assemble the picture from scattered documents.
We've produced two implementations: one for reviewing municipal ordinances (scorecard forthcoming), another for reviewing software products (scorecard forthcoming). The general case for why local decision-making needs open analytical infrastructure is made in our companion white paper Civic Analytics (link forthcoming).
Who is this for? Both sides of the evaluation, as with any scorecard. On one side are the people choosing or judging: residents and board members comparing their municipality's laws to its neighbors', organizations selecting software. On the other side are the entities being scored: a municipality can see where its code is silent on questions its peers have addressed, and a vendor can see how its published capabilities read against the field. A scorecard serves the second audience as much as the first — practitioners need the landscape view too, and before now they had no better access to it than anyone else.
Most scorecards in circulation — analyst quadrants, college rankings, advocacy report cards — are opinion-based. A team of reviewers forms judgments through briefings, interviews, surveys, and expertise, then compresses those judgments into scores. The method has two structural limits. The judgment is subjective: the evidence behind a score is sometimes shared, sometimes cherry-picked, and the reader is ultimately asked to trust the reviewer's authority. And it is expensive: paying qualified humans to evaluate means coverage goes only where the economics support it.
Opinion-based scorecards also enjoy a marketing engine that keeps them prominent. The ranked entities promote the rankings themselves — every "Leader" badge on a vendor's homepage, every "Top 20" banner on a university's site, advertises the ranking's authority for free. The publisher and the ranked share an interest in the scorecard's legitimacy.
The Ordinizer's scorecards rest on a different foundation: what the entities themselves have published. Every score derives from the entity's own documents — the ordinance as written, the vendor's own materials — and every cell cites the passage it rests on. Anyone can check the work. The scorecard doesn't ask "what do experts think of this town's wetland protections?" It asks "what do this town's laws actually say?" — the same questions, the same rubric, applied uniformly to every entity.
The record-based scorecard draws on a few established ideas.
The namesake of the genre, , is internally focused: an organization defines its own goals and then measures its performance against them, on the premise that institutions steer by what is made visible in structured form. Corporate scorecarding extends this inward comparison across divisions and departments, scored on common measures — the comparison itself serving as a motivator. The record-based scorecard applies the same discipline outward, across entities rather than within one.
A second foundation comes from transparency policy: the practice of requiring organizations to report information outward to the public. In Full Disclosure, Archon Fung, Mary Graham, and David Weil examined when such disclosure actually changes behavior. Nearly all their flagship cases are , where entities report on themselves. Their central finding is that raw disclosure fails unless someone makes it comparable, standardized, and embedded in the decisions people already make. Documents sitting in scattered PDFs on municipal websites are disclosure without that layer. The Ordinizer supplies the missing layer — it is the intermediary the transparency chain has lacked.
A record-based scorecard rests on what the entity itself has put on the record — a mandated disclosure (an SEC filing, an incident report), an enacted law, or its own published materials. The category has two branches. Data-backed scorecards work from structured records: quantitative filings that can be tabulated directly, as fiscal stress monitors and toxic release rankings do. AI-driven scorecards work from unstructured documents — statutes, policies, published materials — that until recently only human readers could evaluate. The two are complementary, and the second no longer has to wait for governments to publish clean structured data: the documents already exist. is the nearest precedent for the whole category.
Objective ratings have empirical support of their own. found that structured evaluation, the same questions and criteria applied uniformly, consistently matches or outperforms holistic expert judgment, which varies widely from one evaluator to the next. A rubric applied by machine is that discipline at scale: every entity scored the same way, every score auditable. Herbert Simon's account of real decision-making supplies the complementary point: people decide with whatever information is cheaply at hand, not with everything that exists. The cross-entity view has lived in a few expert heads, costly to reach, so most decisions have been made without it. A standing scorecard changes what is cheaply at hand.
One caution the literature also supplies: when a scorecard becomes a target, entities optimize for the score rather than the substance — . The Ordinizer's design is a partial defense — because every score cites its source and the rubric is public, gaming the score means changing the actual documents, and disagreement with a score has something concrete to argue with.
The Ordinizer does not remove human judgment from evaluation — it relocates it. People define the realm, the domains, the questions, and the rubrics: the judgment-laden work. The machine does the reading and scoring at scale: the labor-intensive work. This division makes expert reviewers more efficient rather than obsolete. An expert's time goes into designing good questions and auditing contested scores, not into reading the four-hundredth PDF. The judgment that shapes the scorecard lives in a public, criticizable rubric instead of hidden in a reviewer's head.
The claim to objectivity should be stated precisely: the evidence trail is objective — documents, citations, a reproducible method. The choice of what to ask and how to grade it involves judgment, published for anyone to challenge.
The tool's limits track the limits of the record. The Ordinizer can assess what a policy states, not how it is followed — that would require different data: enforcement records, outcomes, observation. It can assess what a product's materials say it does, not how well users feel it delivers on the claim. The score measures the record, and the record is what an entity has committed to paper.
This is where subjective reviewers add value the record cannot. Interviews, user surveys, site visits, practitioner experience — the methods of opinion-based review — reach what documents don't capture. The two approaches are complements rather than rivals: the record-based scorecard establishes the documented baseline uniformly and cheaply, and human reviewers can then direct their attention to the entities and questions where the record and reality most need checking against each other.
Step back and the territory of document analysis sorts into three regions.
Where evaluation is lucrative, machine reading already happens — behind paywalls. Financial firms run AI extraction across SEC filings, earnings calls, and risk disclosures at sector scale, comparing language across peers and flagging changes. That analysis is sold to clients or held as proprietary trading edge, and the same data-driven analysis of public information is now branching into prediction markets, where structured reading of disclosures and public records feeds priced forecasts. The capability exists; the results stay private.
Where entities benefit from being ranked, opinion-based scorecards flourish on marketing energy — Gartner, US News, and their kin, sustained by the promotional loop between publisher and ranked.
Between these lies a large middle region: public data that could be assembled into comparisons but hasn't been. Municipal statutes. Small-market vendor capabilities. Local governance practices. The documents are public, the questions matter to real decisions, but no market will pay analysts to assemble the answers and no marketing engine promotes the results. The bottleneck was never the data or the demand — it was that assembly required paid human readers, and these domains couldn't pay. Machine reading collapses that cost. The long tail of public disclosure can finally be scored.
That middle region is the Ordinizer's territory, and it is arguably the largest of the three.
The Ordinizer is an open-source package. You download it and follow the instructions to configure it for your own realm.
You define a realm — an area of inquiry, such as conservation law or participatory governance software. Within it, you specify domains (the major categories of concern), a set of questions for each domain, and a scoring rubric for each question. Then you select the entities to analyze — municipalities, vendors, agencies — and point the system at their published materials.
The AI reads each entity's corpus and answers every question against the rubric, citing the passages that support each score. The result is a scorecard: entities across one axis, questions grouped by domain down the other, every cell scored, sourced, and reproducible. Where an entity's documents are silent on a question, that is recorded — "this town has no such provision" is often exactly the decision-relevant fact.
The current realms are a starting point. The architecture generalizes to any realm where entities publish documents and uniform questions are worth asking across them: procurement, housing policy, environmental compliance, institutional governance. Expanding coverage — more counties, more realms — needs collaborators: people to define domains and rubrics in areas they know, and people to identify the corpora worth reading.
There is a longer game here too. Governments and citizens have persistently failed to track, or regularly demand, open data in reusable form — the failure the Civic Analytics paper documents. AI-driven scorecards offer a partial workaround, since they read the documents that already exist. But they also generate the demand: once a realm is scored, the questions the documents cannot answer become visible, and those gaps are a concrete, specific argument for what structured disclosure should exist. The scorecard is both a substitute for missing open data and a case for producing it.
The Ordinizer is developed under NYSeeds.org. The whole apparatus is open source: results publish to a public Git repository, and the method itself is free to inspect, reuse, and extend. Contact: jongarfunkel@gmail.com.