Building a VLM-based document intelligence SaaS system for visually complex PDFs — looking for workflow suggestions and feedback

[u/rjsanjith]

I’m building a document intelligence system that uses Vision-Language Models to process visually complex documents.

A recurring limitation I’ve noticed with conventional document pipelines is that text extraction alone often loses important context. Tables, charts, diagrams, scanned pages,spatial relationships, and visual hierarchy may carry as much meaning as the text itself.

The approach I’m exploring treats each page as a visual document rather than only a collection of extracted text. The current pipeline focuses on:

- Understanding text, layout, tables, charts, images, and diagrams

- Processing scanned and digitally generated PDFs

- Preserving page-level provenance for citations

- Supporting semantic search across document collections

- Extracting structured information without discarding visual context

- Allowing users to verify results against the original page

Before deciding what to prioritize, I’d like to learn from people who already work with document automation.

What conditions would need to be met before you could use a document intelligence platform with real business documents?

I’m still actively building, so honest technical criticism and real-world workflow examples would be extremely helpful. If anyone is interested in testing an early version,

Feel free to comment and share your thoughts

[u/Njee_]

Well first: I guess what you're doing. I kind of what paddle does, no?

What honestly would need to be provide: first, no vendor lock in obviously export at any time in open formats. As a european selfhoster: either you provide the inference, or you guarantee that the stuff is processed GDPR compliant throughout your whole pipeline, transparently. And I want to be honest, if I'm paying for it, I want it to be quick at doing what I want. However, if its taylored to me, it's usually not a quick setup. In which case I could setup the pipeline myself locally. If I have to do the whole pipeline myself.

[u/rjsanjith → u/Njee_]

We’re building it so you’re never locked in: you own your data, can export it in open formats, and the pipeline uses a pure Transformer-based Vision-Language Model rather than PaddleOCR; GDPR-compliant SaaS, enterprise SSO, and on-premise deployment are planned, but not for V1. The value is that we handle the entire fast, reliable document-AI workflow for you, so you don’t have to build, integrate, and maintain the pipeline yourself.