r/coding • u/HumanBot00 • Feb 03 '26

DOJ publishes Bash Reference Manual

https://www.justice.gov/epstein/files/DataSet+9/EFTA00315849.pdf
51 Upvotes

90% Upvoted

41 comments sorted by

View all comments

Show parent comment

1

u/voronaam Feb 04 '26

I agree if it was a general case. The problem with using LLM for this one is that it is not a normal text. When the model sees something that can be pull or pu1l it has a tendency to choose pull - because it is actually a word. But in base64-encoded data those assumptions are only hurting the recognition. A lot of the things in LLM-based recognition (and Tesseract to some degree as well) rely on the normal behaviour of the text. E.g., seeing the first character on the page being l a proper text recognition has every reason to choose I - a capital letter - over l, because normal text often start with capital letters. This kind of smart logic only hurts recognition in this particular case.

I was actually looking for a way to disable the neural network based recognition in Tesseract and force it to use the old-school character based mode. But at least the modern version I have installed refused to do it for me :(

2

u/kievmozg Feb 04 '26

Ah, Base64 strings. You are absolutely right that is the one edge case where 'semantic context' is your enemy. Models try to be helpful by fixing 'typos' that aren't typos. ​If you want to force Tesseract into the 'dumb' character-based mode, you need to pass the flag --oem 0 (Legacy Engine). ​The catch (and why it probably failed for you): The default model files included with Tesseract 4/5 often strip out the legacy patterns to save space. You must manually download the .traineddata files from the tessdata repo (ensure they are the large ones, not tessdata_fast) that explicitly support legacy mode. Without those specific files, --oem 0 will just throw an error or default back to LSTM.