OCR accuracy is a product decision, not a model score

Receipt Scanner photographs a paper receipt and turns it into a structured expense record: merchant, line items, tax, total. The first working version read clean receipts well and crumpled thermal paper badly — accuracy sat below 60% on the receipts people actually have in their pockets.

The number that eventually mattered was not the one I started optimizing.

The input is the problem, not the engine

My first instinct was that the OCR engine was the weak link. It was not. Tesseract does reasonably well on text that is flat, aligned, and high contrast. Thermal receipts are none of those things by the time they reach a wallet:

• the paper curls, so the baseline bends across the width;
• the photo is taken hand-held at an angle, so the text is skewed;
• thermal print fades unevenly, so contrast varies within one receipt;
• the background is usually a table, so the crop is ambiguous.

Every one of those is an image problem that exists before a single character is recognized. Feeding a better engine a worse image is the wrong lever.

The multi-pass preprocessing pipeline addresses them in order:

captured frame
  -> contrast normalization   (even out uneven thermal fade)
  -> adaptive thresholding    (local, not global — handles gradients)
  -> deskew via Hough         (find dominant text lines, rotate to flat)
  -> OCR

Adaptive thresholding is the one that moved the number most. A global threshold picks one cutoff for the whole image, which fails the moment one half of a receipt is faded and the other is not. Thresholding against a local neighbourhood treats each region on its own terms.

That pipeline plus post-processing brought real-world accuracy to roughly 92%.

Arithmetic is a free validator

The interesting correction step is not visual at all. A receipt carries internal redundancy that most documents do not: the line items are supposed to sum to the total.

That means the document can check itself.

If extracted items sum to the printed total, confidence in both readings goes up. If they disagree, the mismatch localizes the error — and often the specific failure is recoverable, because OCR confusions on digits are systematic rather than random. 8 reads as B, 5 as S, 1 as l, and a misplaced decimal shows up as an order-of-magnitude gap rather than a small one.

Where a document contains its own check digit, use it. Structural redundancy is cheaper and more reliable than a better model.

This generalizes past receipts. Invoices have subtotals. Bank statements have running balances. Timesheets have day totals. Anywhere a document was designed for a human to verify by hand, that verification path is available to you too.

Running it on the device changes the tradeoff

OCR runs client-side, in Tesseract.js WebAssembly workers, rather than on a server.

That decision buys three things and costs one:

The last row is real. A mid-range phone from four years ago takes noticeably longer than a laptop, and there is no server-side fix — I cannot deploy a faster machine on their behalf. WebAssembly workers keep the main thread responsive during the work, so the app stays interactive, but the wall-clock time is theirs.

I took that trade because the capture moment is the one where connectivity is least likely. You photograph a receipt when you are handed it: in a shop, in a car park, on the way out. A design that requires an upload right then fails exactly when it is needed.

Offline-first means the record exists before the sync

The flow persists locally before it attempts anything remote:

1. capture and preprocess locally
2. run OCR in a WebAssembly worker
3. normalize merchant, items, tax, total
4. save a pending record on the device
5. sync to Supabase when connectivity returns

Step 4 before step 5 is the whole idea. The user's expense is recorded the moment it is captured. Synchronization is a background concern about durability and multi-device access, not a precondition for the app being useful.

On the server side, Supabase row-level security scopes records per user, so the isolation boundary is enforced by the database rather than by every query remembering to filter. That is the same instinct I wrote about in designing tenant boundaries: a rule that depends on developers remembering it is not a boundary.

The remaining 8% is a UX question

Here is where the framing shifted for me. A 92% field-level accuracy sounds close to solved. It is not, and the arithmetic shows why: a receipt with twelve line items and 92% per-field accuracy has roughly a 63% chance of containing at least one error somewhere.

So most receipts are wrong in some small way. No amount of additional preprocessing changes that, because the failures that remain are ones where the information is genuinely not in the image — thermal print that has faded past recovery cannot be recovered by any pipeline.

That reframes the goal. The question is no longer "how do I get to 99%" but:

1. Does the app know which field it is unsure about?
2. Is correcting that field faster than typing the receipt from scratch?

Confidence-aware extraction beats confident extraction. A value the app flags as uncertain, positioned next to the cropped region of the image it came from, is a one-tap correction. The same value presented as a fact is a silent error in someone's expense history — worse than no extraction at all, because it will not be checked.

The best remaining improvement is not in the pipeline. It is at capture time: guiding the user to a flatter, better-lit, better-framed photograph prevents errors that no amount of post-processing can undo.

The takeaway

I spent most of my effort on the part that was measurable and most of my learning on the part that was not. Preprocessing took accuracy from 60% to 92% and that work was worth doing. But the product only became trustworthy when it stopped presenting every extraction with equal confidence.

Accuracy is what your pipeline achieves. Trust is what your interface does about the gap.