Why there is no honest OCR leaderboard

What would a fair OCR benchmark need?

Three things: a set of pages, a correct answer for each, and every tool run the same way on them. Each is harder than it sounds.

The pages must look like real work and be new to every tool. The correct answers, called an answer key, must be the exact text of each page. And all tools must be scored with the same rules. If any of the three is weak, the final number looks precise but means little.

Why are answer keys so hard to make?

Because a person has to type every word, and a second person has to check it. An answer key is the exact correct text of a page, and OCR is scored by how far its output differs from the key.

For a bank statement, the key is every transaction row, with the right date, amount and sign. For an Urdu book page, it is every letter and mark. One mistake in the key unfairly marks a correct OCR result wrong, or lets a wrong one pass. Doing this well is slow and costs real money, so many published scores skip the second check or use text copied from a file that may itself be wrong.

Why not share the documents people really use?

Because the pages that matter most are private. Bank statements, ID cards, medical forms and contracts belong to someone, and sharing them is not an option.

Public sets fill the gap, and they have limits. A research team reviewing earlier work (OmniDocBench) found that existing benchmarks cover few document types. The large public sets of bank statements are synthetic, which means generated by a program, and their labels are not checked by a person. We have a set of 400 Indian statements that we keep out of our own scoring for this reason: we cannot trust its labels enough to say a score on it is real.

Why can’t one company test every competitor?

Paid tools must be bought, set up and run the way their makers intend, and they change often. Testing every one fairly means buying every plan and re-running the work each time a tool updates.

A shortcut is to test a few and leave out the rest. That leaves out the tool that might be best for you. Another shortcut is to let each company score itself, which is how most rankings arise. We do not think that is dishonest in every case, but the person publishing the chart also chose the pages, the rules and the tools it compares with.

What is wrong with one headline score?

It hides where a tool fails, and OCR fails for different reasons on different pages. A tool can read clean English print almost perfectly and still fail on a photo, a table or a script it was not built for; see why some scripts are harder for OCR. One average cannot show that.

A score can also mislead on what it counts. A page that is mostly right can still lose the one number that mattered. A good score on one kind of page says nothing about yours.

That is why a ranking built on someone else’s pages rarely settles which tool you should use. Other tools are good too, and a fair reader would also try the free built-in OCR on their phone or computer before paying for anything.

What should I do instead?

Test your own pages. Pick five that look like your real work, include your hardest page, type a paragraph and every name, date and amount, and count the differences for each tool.

1. Choose five pages: three typical, one bad photo and one with your hardest feature.
2. Type out one paragraph and every name, date and amount from each page.
3. Run each tool on the same files without editing them.
4. Count wrong or missing names, dates and amounts first, then other wrong words.
5. Pick the tool with the fewest errors that matter. Retest when your documents change.

The full steps are in how to test any OCR tool on your own documents. If you want a feature comparison instead, see how FastOCR stands against other tools on our comparison pages, and keep in mind that we wrote them.