How we measure
Everything below is what our software actually does, written so that a records clerk, an IT director or a council member can check it. Where we estimate, we say so and we show the sample.
1. What we look at
Only the public website. We never log in, never request access, never install anything. We fetch the same pages a resident would open, follow the links on them, and note every document they point to. We identify ourselves in every request, so a system administrator can see us in the logs and block us. We keep to about two requests per second and we respect the site's robots file.
2. What we count
A document is a PDF file linked from the site. We stop crawling at a fixed number of pages, so on very large sites the count is a floor, not a ceiling, and we say so on the city page. Word, Excel and PowerPoint files are on the roadmap and are not counted in the pilot edition.
3. What we test
We download a random sample of the documents found, usually two hundred, and check each one for:
- A tag tree. The internal structure that tells a screen reader what is a heading, a paragraph, a list, a table. Without it the reader gets a flat stream of words, or nothing.
- Real text. A scanned page is a picture. A screen reader cannot read a picture. We check whether the first pages contain text at all.
- A declared language and a title. Small things that decide whether a reader pronounces the document correctly and announces it by name.
- Whether it is still online. Dead links are removed from the estimate.
These are the checks that can be run by software with confidence. They are necessary conditions for WCAG 2.1 Level AA and PDF/UA (ISO 14289-1), not the whole of either standard. A document that passes all of them can still fail a human review. A document that fails the first one cannot pass.
3a. The second opinion
Every sampled document is also run through veraPDF, the open-source reference validator maintained by the PDF Association, against the PDF/UA-1 profile. We publish its failure rate next to ours. In the pilot the two agreed on every document we called untagged; veraPDF additionally fails most of the documents that do have a tag tree, for reasons a tag tree alone cannot show. That gap is real, and it is why our repair work ends with a person and a screen reader, not with a checker.
4. How we grade
The grade uses one number: the share of live documents in the sample that have no tag tree. The scale is fixed and applies to every city the same way.
| A | Under 10% of documents lack screen-reader structure |
|---|---|
| B | 10% to under 30% |
| C | 30% to under 55% |
| D | 55% to under 80% |
| F | 80% or more |
4a. What repair actually does, and what it cannot do
An order does not start with an opinion. It starts with the same measurement, run over the files themselves, and then a repair pass that we test the same way. Some of the work is mechanical and a program does it well: recognising the text on a scanned page, writing the structure a screen reader follows, naming the language, giving the document a title, marking the tab order on a page of form fields.
We have run that pass over 111 documents from two cities and measured the result with veraPDF. The number of failed rules fell from 785 to 198, 75% fewer. 39 documents gained a structure tree they did not have. 36 of 111 came out passing PDF/UA-1 completely.
The other 75 did not, and the reasons are the reason a person is on this job. A machine cannot know what a photograph is for, so it cannot write the sentence a blind reader hears instead of it. It cannot tell which row of a table is the header when the layout only implies it, or what an unlabelled box on a permit application is asking for. Those are decisions about meaning, and we make them by reading the document.
Two checks run on every repaired file and both can stop it. The text must still be there afterwards, character for character. And the content must still be reachable through the structure we wrote: a file can be made to pass a validator by declaring the whole page decorative, which produces a technically conforming document that a screen reader reads as silence. No file in the test was returned that failed either check.
5. What we do not claim
- We do not claim that a document we repaired is “compliant” in a legal sense. We state which checks it passes, and a person reviews every file we return.
- We do not claim that our count is your legal obligation. The Title II rule exempts documents that were already published and are not used to obtain a service. Only a review of your own site can settle which documents that covers.
- We are not a government body and we have no authority over anyone. Nobody is required to buy anything from us.
6. Corrections
If a figure on a city page is wrong, write to reports@documentaccessibilityinstitute.org. We re-run the measurement, publish the new result with the date, and keep the old one visible underneath it.