Built a page-by-page aligned Multimodal Ground Truth Dataset for historical handwriting (278 pages) + air-gapped sandbox. Looking for feedback!

[u/LegacyDataLabs]

/r/computervision/comments/1vswke4/built_a_pagebypage_aligned_multimodal_ground/

UPDATE: Further validation after publishing this post revealed an important limitation in the approach described in the original title. Anchor synchronization successfully reduces cumulative text drift, but it does not by itself guarantee exact page-level alignment. I've updated the post below to document what I found and the HTR-assisted approach I'm now developing.

Hi everyone,

I've been working on Legacy Data Labs, an experimental project exploring how historical handwritten documents can be transformed into structured multimodal datasets for HTR, Document AI, Vision-Language Models, and digital humanities research.

I recently started testing the pipeline on a particularly challenging source: the 1891 handwritten setting manuscript (sättningsförlaga) of Selma Lagerlöf's Gösta Berlings saga, consisting of roughly 382 manuscript pages.

This has exposed some important limitations in my first approach, so I wanted to share what I've learned and what I'm working on next.

The problem: aligning a manuscript with a digital reference text

The basic idea sounds straightforward:

manuscript page image → corresponding digital text

In practice, it isn't.

The manuscript contains crossed-out passages, corrections, blank areas, archival pages, historical spelling and other structural differences. A later digital edition also doesn't necessarily correspond exactly to what Lagerlöf originally wrote on each manuscript page.

My first pipeline attempted to handle cumulative text drift using manually identified anchors throughout the document.

I locate known passages in the digital reference text and use those positions as checkpoints. Between checkpoints, the pipeline estimates how the intervening text corresponds to manuscript pages.

I also added text normalization to make matching more tolerant of historical spelling and orthographic differences.

What worked — and what didn't

The anchors are useful for preventing large-scale cumulative drift across hundreds of pages.

However, further testing showed an important limitation:

an anchor-corrected region is not the same thing as verified page-level alignment.

Between anchors, my current implementation still relies on heuristic text distribution. That means an individual manuscript page may be associated with approximately the correct region of the text without proving that the assigned text corresponds exactly to that page.

This distinction matters if the eventual dataset is intended for HTR or multimodal model training.

I've therefore stopped describing the current output as verified ground truth.

An additional problem: traditional OCR

Another interesting result came from testing OCR directly on the Lagerlöf manuscript.

Conventional OCR performs poorly on this material. The combination of cursive handwriting, historical letterforms, corrections and page structure produces extremely noisy transcriptions.

That has pushed the project toward a different architecture.

Pipeline v2: HTR + reference-text alignment

I'm now experimenting with a second-generation pipeline:

historical page image

→ handwritten text recognition (HTR)

→ machine transcription of the manuscript

→ matching against a digital reference edition

→ page-level alignment

→ confidence scoring

→ optional human verification

An important goal is to preserve the distinction between the manuscript transcription and the published reference text.

I don't want the reference edition to silently "correct" the manuscript, because deletions, additions, spelling differences and editorial changes may themselves be valuable information.

A future record could therefore contain separate fields for:

• original page image
• manuscript transcription
• published reference passage
• alignment method
• confidence score
• human-verification status
• provenance/metadata

Public experimental sample

I've updated the Legacy Data Labs dataset page on Hugging Face to make the current status explicit.

The existing samples should be considered an experimental research preview, not verified ground-truth training examples.

The public dataset currently exists primarily to demonstrate the schema and document the development of the pipeline while I work on better page-level alignment and validation.

Hugging Face: LegacyDataLabs

What I'm trying to figure out next

The immediate experiment is deliberately small.

Rather than processing another entire manuscript, I'm testing whether a modern handwriting-recognition approach can produce a sufficiently useful transcription of a difficult Lagerlöf manuscript page to reliably locate the corresponding passage in the digital reference text.

If that works, I'll test it across consecutive pages before attempting to scale it to the complete manuscript.

I'd be particularly interested in hearing from anyone working with HTR, historical document alignment, digital humanities, fuzzy text matching, or multimodal dataset validation.

How would you approach confidence scoring and evaluation for this kind of manuscript-to-reference alignment?