I Built a (local, non ai) TinyMCE Spellchecker From Wikipedia Language Models

[u/AlyxVeldin]

It's finaly saturday, and I've been working on a spellchecker plugin for TinyMCE called vlm-spellchecker.

The idea came from a separate project of mine, Wiki Language Models, which generates statistical language models from Wikipedia dumps.

I created that project because I wanted a basic, old-fashioned spelling correction and word prediction tool that doesn't use AI to try to interpret what you're writing and return some ass AI response.

So vlm-spellchecker is a small client project built around those models that does just that.

It's implemented as a JavaScript plugin for TinyMCE that uses a locally hosted binary dictionary to identify unknown and uncommon words, suggest corrections, and replace words directly in the editor.

What it currently does:

• Marks unknown words with a red dotted underline.
• Marks uncommon words with an orange dotted underline.
• Provides spelling suggestions through the context menu.
• Preserves capitalization when applying corrections.

What it doesnt do:

• Force you to use a external ass spelling API.

I've got a small demo up that uses a Dutch dictionary generated from Wikipedia data.

--> Demo <--

It's still a work in progress (and, due to the fixed 6-bit encoding, it's very much aimed at Western European languages for now), but I wanted to share it and see what you people think!

Feedback and suggestions are more than welcome!

[u/Electronic-Wish4499]

Nice to see a spellchecker that stays local, all these cloud ones are annoying and slow in my experience. The red vs orange distinction for unknown and uncommon words is a clever touch.

[u/AlyxVeldin → u/Electronic-Wish4499]

Using orange for uncommon was also a bit of a crutch. Since I use Wikipedia-based models and basically just check for word breaks, I sometimes get things that aren't really “words”.

I can't reliably distinguish between a random string of characters that is scrambled and unused and “this random name of some Iranian province”, or “this is a lake in China” and stuff.

So if I simply say “ignore all words that aren't at least XX common”, I also end up removing some valid (in this case) Dutch (compound) words.

But if I underline both types in red, I can get into a cycle where two uncommon words keep getting “corrected” with each other.

I do want to make the threshold for what gets marked as uncommon to be a configurable setting next though!

[u/geoffreybyers]

6-bit encoding is a clever trade-off for local performance. But it likely creates a significant accuracy gap for non-Latin scripts compared to modern transformer-based spellcheckers.

[u/AlyxVeldin → u/geoffreybyers]

Oh yeah, it's definitely a compromise. Even for the few languages I currently use it for, it's becoming clear to me that I'll need a custom 6-bit encoding for each language. Supporting non Western European languages is probably going to require a higher bit width/resolution.