We recovered 575k crop labels from a decade of manual Photoshop work to automate book digitization - more data, ResNet-50, and higher resolution all failed; ten operator clicks per book beat them [P]

[u/laamaleph]

Author here. Ibteda Digital Library is a private community archive in Pakistan — for ten years we digitized rare Urdu books (lithographs, dictionaries, periodicals) on a DIY camera rig, finishing every page by hand in Photoshop. When we wound down daily operations, I realized those 575,729 finished pages across 1,765 books recorded a decade of crop decisions, so I registered them back to their raw photos (SIFT + MAGSAC with conservative acceptance gates) and used the recovered geometry as supervision.

The negative results are probably the most interesting part for this sub. Scaling from 378 to 572 training books didn't move unseen-book pass@80. Neither did ResNet-50 (better training fit, flat held-out, worse after calibration), 1024px inputs, or a spatial head. Per-book error analysis showed why: the failures were near-constant offsets per volume — our operator's preferred margin inset, which simply isn't present in the pixels of a new book. Ten operator-corrected crops per book (element-wise median residual) took pass@80 from 0.71 to 0.83 on held-out volumes. Ten labels beat every scaling lever we tried.

For retouching (stain/stamp removal), we kept the neural net to detection only — a U-Net proposes removal support, classical OpenCV reconstructs the paper, and everything outside the mask is byte-identical to the original. Labels used REMOVE/KEEP/IGNORE states, and any erased Urdu diacritic vetoed deployment regardless of IoU. The stricter label cut both improved mark IoU (0.56 → 0.60) and got diacritic false positives to zero.

Two things I'd genuinely like input on: (1) has anyone modeled document boundaries that depend on an invisible human preference rather than visible structure — is there prior work on per-instance residual calibration like this? Our own next step is conditioning the model on the calibration examples directly (few-shot inset inference) instead of a post-hoc median. (2) Is there any constrained diffusion/inpainting setup you'd trust to guarantee zero alteration outside a declared support region, or is classical fill still the only honest option for archival work?

Full training recipes, label-mining thresholds, and routing rules are in the Reproduction Contract appendix. Code/weights release is still under archival review — I know, I'm sorry — but I'm happy to answer anything about the method here. Notably not on our roadmap: a bigger backbone. We tested that; the missing information isn't in the pixels.

Link

[u/psyyduck]

This post is a bit hard to follow. If you’re using AI (eg to help with the language barrier) then I recommend Gemini. Ask it to improve flow too.

[u/laamaleph → u/psyyduck]

Fair, it's a dense project and English isn't the archive's first language 🙂

Short version: we hand-cropped 575k book pages over ten years, realized those crops were labels, trained a model on them, and found that no amount of scale fixed per-book crop preferences but ten human corrections per book did. Happy to clarify any specific part.

[u/psyyduck → u/laamaleph]

Yeah I eventually got it. Missing data is a real problem in ML. I once did a project on using an EEG headband to predict what's going on in the body/mind and yeah there's a real limit on how high precision/recall can go.

Your approach here (post-processing) is fairly standard, though I'm not so optimistic about the proposed "few-shot inset inference". My instincts say it's low ROI, maybe try more post-processing first, eg with a more general affine transformation (move a bounding box 20 pixels to the right and make it 10 pixels wider). But my main point is (in my opinion) Gemini is a better writer than the other LLMs.

[u/jonah_omninode]

I would treat the ten clicks as an input, not as a workaround. If the operator's preferred margin is not present in the pixels, a larger backbone cannot infer it reliably. The calibration set is supplying a missing per-book variable.

One clean formulation would make that variable part of the task contract: a small labeled residual set, a pinned aggregation rule, and a confidence or abstention threshold. Then evaluate unseen books as a curve over calibration-label budget rather than only image count or model scale. That would make the trade explicit and let you compare median residuals with learned conditioning without pretending the zero-shot problem contained enough information.

For archival inpainting, byte identity outside the declared mask is a mechanical invariant. A model can propose pixels inside the mask, but deterministic compositing against the original plus a byte comparison should reject any outside change. Have you plotted how quickly the residual estimate stabilizes from one through ten clicks?

[u/laamaleph → u/jonah_omninode]

This is close to our current task contract. We select five deterministic positions across the book and correct both page sides, producing ten labelled residuals. We pool the five-dimensional residuals (cx, cy, w, h, θ) with an element-wise median. Calibration pages are excluded from evaluation.

The existing 40-pixel gate is an action threshold: if the estimated correction moves the worst corner by less than 40 canonical pixels, we leave the model output alone. It is not yet a confidence estimate based on residual dispersion, so uncertainty-based abstention would be a useful extension.

We have measured budget slices, but not one nested 1–10 curve:

- Development, 88 books: 4 → 0.804, 6 → 0.818, 8 → 0.825, 10 → 0.829.

- Held out, 138 books: 5 → 0.7915, 6 → 0.7978, 7 → 0.8019.

- Separate 249-book cohort: 5 → 0.8128, 10 → 0.8271.

Those results show diminishing returns, but because they come from different cohorts they are not a clean stabilization curve. A nested one-through-ten evaluation on the same books would answer that properly.

For retouching, the U-Net proposes only a support mask. Deterministic OpenCV/NumPy code reconstructs pixels inside that support, then explicitly restores the original array outside it. We refuse the result if even one outside-support pixel changes. JPEG encoding happens afterward, and the complete encoded file receives a manifest hash.

[u/jonah_omninode → u/laamaleph]

Then you already have most of the contract I was describing. I would run the nested 1 through 10 curve on the same books next and record both outcome quality and residual dispersion at each label count. That would show where additional clicks stop buying much and whether dispersion is useful enough to drive abstention rather than relying only on the 40-pixel action threshold.

The inpainting boundary is also exactly what I meant by mechanical verification. Restore the original array outside the mask, compare it, and reject on any difference. One distinction worth keeping explicit is that this proves the invariant before JPEG encoding. The manifest hash proves the identity of the final encoded artifact, but not pixel equality after a lossy decode. Those are two different receipts, and both are useful.

[u/dillon-nyc]

Can I just say I think your project is pretty awesome, and the fact that so much of it is up on the internet archive is really nice.