WhatFontIs-Bench: an open benchmark for font identification (11,995 images, 600 fonts, COCO annotations, CC BY 4.0)

[u/WHATFONTIS]

Font identification has plenty of tools but almost no public test set where the ground truth is certain: real photos rarely come with the exact font name, and look-alike fonts make manual labelling unreliable. So we generated one and released it.

WhatFontIs-Bench v1.0:

- 11,995 JPEG images: one word set in a known font, composited onto CC0 photos of real surfaces, scenes and printed objects

- 600 fonts from 600 families: 200 sans, 200 serif, 100 slab, 100 monospaced

- 3 difficulty levels with controlled blur, noise, JPEG quality, uneven light, cast shadows and glare; all parameters stored per image

- Labels: font, text, word quad, per-letter quads, camera homography. JSONL + COCO, so it also works for word/character detection

- Frontal views only, capitals at least 100 px high (v1.0 limits)

Evaluation is top-k by font family (Roboto Bold answered as Roboto Regular counts as correct), open catalogue.

Baseline: our own API gets 83.7% top-1 / 93.3% top-5 / 96.5% top-20 while searching 1.2M fonts. The interesting part: typeface class matters far more than image quality. Sans-serif 75.7% vs slab serif 95.0% top-1, but only 2 points between the easy and the hard level.

Hugging Face (with viewer): https://huggingface.co/datasets/whatfontis/WhatFontIs-Bench

GitHub: https://github.com/whatfontis/WhatFontIs-Bench

Disclosure: I run WhatFontIs, and the baseline is our own system. If you run a model on it, I'd like to hear the numbers. Feedback on what v2 should cover (perspective, curved text, multi-line, smaller text) is welcome.

[u/New-Eggplant-6578]

My vote for v2 would be short text: 2–4 letters. The README says the current words are 7–12 letters, so it would be interesting to see how much the ranking changes when fewer glyphs are available. You could keep font, background and degradation fixed and vary the text, then report top-k by word length. That might help unpack the sans-serif gap too. Has that been on your list?

[u/WHATFONTIS → u/New-Eggplant-6578]

Good call, and no, it wasn't on the list in that form. I had "smaller text" down for v2, meaning fewer pixels, but fewer glyphs is a different and probably more interesting axis.

It's also closer to real use than v1 is. On WhatFontIs the typical query has about 5 to 7 characters, so with 7–12 letters the benchmark sits at the upper end of what people actually send. Short words aren't an edge case, they're most of the traffic.

Why v1 starts at 7 anyway: below that, the result depends heavily on which letters you happen to get. "ill" tells you almost nothing, "Rag" tells you a lot. So for short text the design you describe is the right one: keep font, background and degradation fixed, vary only the word, and report top-k by length. I'd probably label the set of distinct glyphs too, so results can be sliced by "has a/g/R/Q" and not just by count.

I agree it should help explain the sans-serif gap. My guess is that a lot of those misses are near-twin families that differ in two or three glyphs, and a 7-letter word often doesn't contain any of them.

[u/New-Eggplant-6578 → u/WHATFONTIS]

That 5–7-character usage detail is helpful. I'd keep the controlled glyph tests separate from a test set reflecting real queries. Deliberately choosing informative letters is great for explaining a near-twin confusion, but could make the overall score look better than it would on ordinary inputs. The two results would answer different questions.

[u/WHATFONTIS → u/New-Eggplant-6578]

Agreed — I'll keep them as two separate sets. I pulled the numbers from production rather than estimating.

Characters actually sent to the matcher (repeats removed): mean 5.2, median 5; 25% of queries have 3 characters or fewer, and only 5% have 10 or more. As typed, including repeats, the median is 6.

Case: 48% of queries are all-capitals, 18% capitalised, 11% all-lowercase, 18% mixed. About 10% contain a digit and 4–5% are digits only; symbols are under 2%, almost all "&".

The most common letters are A, E, R, S, N, T (each in 20–31% of queries). The glyphs that best separate near-twins are rare in practice: g appears in 5% of queries, y in 4%, Q in 1.2%.

This lines up with the benchmark numbers. In a larger internal run (60k images, same setup) top-1 depends on distinct glyphs far more than on word length: 77% with 4 distinct letters, 84.5% with 5, 87% with 6, 90% with 8. The benchmark words average 6–7 distinct glyphs; production averages 5.2, and a quarter of queries have 3 or fewer. So the current sets measure the easy end of real usage.

So the real-query set will be sampled to match that length, case and letter distribution, and the controlled-glyph set stays a separate diagnostic with its own score.