An open-source tool that turns scattered public documents into structured, sourced, comparable scorecards.
The Ordinizer (short for "Ordinance Organizer") is a software tool that generates record-based scorecards for public viewing on the web. A record-based scorecard is a structured evaluation assembled from what an entity itself has put on the record — its enacted laws, its filings, its own published materials — with every score citing the passage it uses. A scorecard is meant to be more useful than the raw data it draws on: it selects the questions that matter and arranges the answers so a reader can act on them, rather than leaving them to assemble the picture from scattered documents.
Our initial implementation was a . We've since generalized the code to look at any matrix of data, such as a . The general case for why civic decision-making needs open analytical infrastructure is made in our companion white paper Civic Analytics.
We've built this for two sets of users. We'd like ordinary residents to be able to see the local laws and compare them. We'd like board members and other advocates to use this knowledge to bring the laws up to the "best practices" of neighboring communities.
Most popular scorecards — analyst quadrants, college rankings, advocacy report cards — are opinion-based. A team of reviewers forms judgments through briefings, interviews, surveys, and expertise, then compresses those judgments into scores. The method has two structural limits. The judgment is subjective: the evidence behind a score is sometimes shared, sometimes cherry-picked, and the reader is ultimately asked to trust the reviewer's authority. And it is expensive: paying qualified humans to evaluate means coverage goes only where the economics support it.
Opinion-based scorecards also enjoy a marketing engine that keeps them prominent. The ranked entities promote the rankings themselves — every "Leader" badge on a vendor's homepage, every "Top 20" banner on a university's site, advertises the ranking's authority for free. The publisher and the ranked share an interest in the scorecard's legitimacy.
The Ordinizer's scorecards rest on a different foundation: what the entities themselves have published. Every score derives from the entity's own documents — the ordinance as written, the vendor's own materials — and every cell cites the source links. We think it's valuable for a start to get answers to questions like "what do a given town's laws actually say?" and "what capabilities does this vendor actually support?"
The record-based scorecard draws on a few established ideas.
The namesake of the genre, , is internally focused: an organization defines its own goals and then measures its performance against them, on the premise that institutions steer by what is made visible in structured form. Corporate scorecarding extends this inward comparison across divisions and departments, scored on common measures — the comparison itself serving as a motivator. The record-based scorecard applies the same discipline outward, across entities rather than within one.
A second foundation comes from transparency policy: the practice of requiring organizations to report information outward to the public. In Full Disclosure, Archon Fung, Mary Graham, and David Weil examined when such disclosure actually changes behavior. Nearly all their flagship cases are , where entities report on themselves. Their central finding is that raw disclosure fails unless someone makes it comparable, standardized, and embedded in the decisions people already make. Documents sitting in scattered PDFs on municipal websites are disclosure without that layer.
A record-based scorecard rests on what the entity itself has put on the record — a mandated disclosure (an SEC filing, an incident report), an enacted law, or its own published materials. Data-backed scorecards work from structured records: quantitative filings that can be tabulated directly, as fiscal stress monitors and toxic release rankings do. AI-driven scorecards work from unstructured documents — statutes, policies, published materials — that until recently only human readers could evaluate. is the closest public precedent for this.
Objective ratings have empirical support of their own. Paul Meehl's research on found that structured evaluation, the same questions and criteria applied uniformly, consistently matches or outperforms holistic expert judgment, which varies widely from one evaluator to the next. A rubric applied by machine can do it at scale. Most decision-makers can only , and don't have the big picture unless it's drawn out. A standing scorecard gives us more information at hand.
The huge challenge of the reliability of scorecards is people trying to game them: optimizing for the score rather than the substance — . The Ordinizer's design is a partial answer to this: because every score cites its source and the rubric is public, gaming the score means changing the actual documents.
With the Ordinizer, our aim is to make the review process more efficient. People define the realm, the domains, the questions, and the rubrics: the judgment-laden work. The machine does the reading and scoring at scale: the labor-intensive work. An expert's time goes into designing good questions and auditing contested scores.
The limitations are that we can only assess what a policy states, not how it is followed — that would require different data: enforcement records, outcomes, observation. It can assess what a product's materials say it does, not how well users feel it delivers on the claim. The score measures the record, and the record is what an entity has committed to paper.
This is where subjective reviewers add value the record cannot. Interviews, user surveys, site visits, practitioner experience — the methods of opinion-based review — reach what documents don't capture. This could give us the best of both worlds: automate what we can, and human reviewers can then direct their attention to the entities and questions where the record and reality most need checking against each other.
The private market already uses machine reading. Financial firms run AI extraction across SEC filings, earnings calls, and risk disclosures at sector scale, comparing language across peers and flagging changes. That analysis is sold to clients or held as proprietary trading edge, and the same data-driven analysis of public information is now branching into prediction markets, where structured reading of disclosures and public records feeds priced forecasts. The capability exists; the results stay private.
The public market sees opinion-based scorecards flourish on marketing energy — Gartner, US News, and their kin, sustained by the promotional loop between publisher and ranked.
We're after the middle market — public data that could be assembled into comparisons but hasn't been: municipal statutes, small-market vendor capabilities, local governance practices. The documents are public, the questions matter to real decisions, but no market will pay analysts to assemble the answers and no marketing engine promotes the results. Machine reading collapses that cost. The long tail of public disclosure can finally be scored.
The Ordinizer is an open-source project on GitHub. You download it and follow the instructions to configure it for your own realm.
You define a realm — an area of inquiry, such as conservation law or participatory governance software. Within it, you specify domains (the major categories of concern), a set of questions for each domain, and a scoring rubric for each question. Then you select the entities to analyze — municipalities, vendors, agencies — and point the system at their published materials.
As for the data, you need to seed it with desired websites. While we had initially used a manually-entered set of URLs for the ordinances, we are starting to work with the new LOCUS dataset to discover specific municipal ordinances programmatically. The documents are pulled in via web-crawling and then indexed into a vector database; the last step is to do the RAG analysis. All of these steps are presently done via command-line utilities; we are looking towards potentially making it all web-based.
The architecture generalizes to any realm where entities publish documents and uniform questions are worth asking across them: procurement, housing policy, environmental compliance, institutional governance. For new coverage — more counties, more realms — we'll need more collaborators such as local subject experts.
Our larger goal is to drive more civic analytics — providing the basic data for government and citizens to make more informed decisions. AI-driven scorecards offer a partial workaround, reading through the documents that already exist. But the gaps they uncover matter too: missing inputs point to what should be tracked, and sometimes to policies that need updating in their own right.