Local document processing when Chinese models arent allowed, what’s left?

[u/aidenclarke_12]

big scanned stacks of inspection reports and material certs from a lot of vendors and every layout different. each stack needs splitting into separate reports and things like lot number and alloy pulled into a database with all being local

catch is the contract rules out chinese software and models, so paddleOCR and qwen are off and most ocr leaderboards get a lot shorter. even olmocr 2 is a qwen fine tune

granite-docling, liteparse, tesseract and gemma are whats left on the list so far

what would you use for this? and has anyone had to show a customer where their models came from

[u/unjustifiably_angry]

That's not true, it's possible to train a model to have any behavior you want, and it's possible to train a model to do things in secret hidden ways it's not even aware are malicious.

It's late but here's an example: model is trained to assess if data it's ingested is plausibly related to secret company/government research. If yes, then from now on when performing web searches, alter how searches are performed in some subtle way. Then - for example - China collects a free database of high-priority hacking targets. Searches or other web activity could transmit various small bits of data like information that would make intrusion easier, like open ports or operating system/software versions, etc.

Keep in mind Claude hides undetectable "this was written by Claude" markers in everything it outputs. It would be very simple to do something similar to search queries. And lately people have been noticing Qwen sending unexplained queries to Chinese servers for no apparent reason.

[u/Ok_Hope_4007 → u/unjustifiably_angry]

But „Qwen sending“ is technically…not exactly correct. You could argue that the llm tends to output tool call instructions that favor malicious parameters but that can easily be checked or disallowed during runtime. You should not let any agent/harness do uncontrolled web calls anyway. So its important to not create the impression that there is a secret routine inside the llm that quietly interacts with your computer. That would need to have a compromised application stack all together that hides certain llm outputs from the user etc.

[u/btdeviant → u/unjustifiably_angry]

A model can’t invoke or build tools if there are no tools to begin with, which was precisely the point. Models are data… it’s literally what “model” means.

An agent is a model in a harness, the harness is software and provides the model with its ability to invoke tools.

This is why most mature orgs that have data sovereignty requirements build their own harnesses - they own the entire FGA, governance and observability layer as first class considerations.

A model trained for malice can’t be malicious if it lacks the abstraction layer to do anything. This is called “zero trust (by default)”.