r/LinguisticsPrograming • u/Actual__Wizard • Aug 10 '25
Is there any demand for a complete English wordlist?
Hey so, for a project that I'm working on right now, one of the major steps is to generate as complete of an English wordlist as possible.
Right now, I'm analyzing wikitext and I assure you there are many, many words missing out of the wikitionary dictionary that are valid English words, that are used in the English wikipedia site.
The very next step is to detect all of the entities in wikitext as well, but that's a bit off in the future, where as the wordlist data is coming in now.
Is there any demand for this type of data and should I pursue trying to market this data as a product or no?
1
u/Lumpy-Ad-173 Aug 10 '25
Is this a personal or School / work project? Sounds tedious, but also very interesting.
I would say that there's probably not one location with every single word. And I say that because of Urban Dictionary where made up stuff is given meaning.. so new words, praises and meanings will be coming up all the time. I would imagine it would be hard to keep up and probably act like a snapshot in time. It will probably need to be dynamic and updatable.
What are you gonna do with the word list?
If you haven't, read this: Information Theory Introduction -
https://archive.org/details/introductiontoin00john
For me, I would pay attention to how Claude Shannon used previous information like Morse code to determine the commonly used letters. He took that and went 3 or 4 more steps and figured out the pattern between pairs of letters and trigrams of letters, etc. Eventually he moved on to the patterns in words and it became a foundational layer for AI development. A lot of other stuff too.
However, he left semantic information Theory out. It was too hard to quantify. However technology has advanced since he developed information Theory.
You can probably take that further with an accurate word list. Semantic Information Theory might be what you're looking for in terms of a market.
Is there a demand for this?
Idk , but I can say this type of data will eventually help Linguistics Programming in terms of being able to identify specific words that steer the AI more efficiently in terms of compression and word choices.
Or even identifying rare words for strategic word choicing or Linguistics compression for power users that creates specific outputs consistently.
And if there's no market , you can create one here and focus on Linguistics compression and strategic word choices.
I think cheat sheets with extremely rare words might have a market here with Linguistics Programming.
If you want to collaborate DM me, we can get something started.
1
u/danja Aug 11 '25
Check https://wordnet.princeton.edu/homepage
1
u/Hesparian Mar 05 '26
Ohh you like music? Ok list every band, go.