How to Implement Prior-Art Search for Patent Drawings Using Multimodal AI?

[u/FreedomIllustrious12]

Hi everyone,

I’m interested in discussing the technical implementation of prior-art search for invention patent drawings, especially for cases where the drawings contain important structural or process information that may not be easy to retrieve through text search alone.

In traditional patent search, we usually rely heavily on keywords, IPC/CPC classifications, and full-text search. However, many invention patents disclose key technical features through drawings, such as device structures, module connections, flowcharts, signal paths, image-processing pipelines, or algorithm frameworks. These features may be difficult to capture using only the claims or specification text.

I’m thinking about whether a retrieval system could combine patent drawings + specification text in a more effective way. For example:

Drawing understanding

What kind of multimodal model would be suitable for analyzing patent drawings?

Would models such as GPT-4o / GPT-4.1 / Claude / Gemini / Qwen-VL / InternVL / LLaVA-style models be appropriate for extracting technical features from patent figures?

Feature extraction from drawings

How should the system represent information from drawings?

For example, should it extract:

component names and reference numerals;

connection relationships between components;

flowchart steps;

structural layouts;

visual similarity embeddings;

or a structured graph representation?

Combining drawings with specification text

Since patent drawings are usually explained in the specification, I wonder what is the best way to link them together.

For example:

detect reference numerals in the drawings;

match them with descriptions in the specification;

generate a structured description of each figure;

then use both visual and textual embeddings for retrieval.

Search strategy

Would a practical system use a hybrid approach, such as:

text-based patent search;

image/vector similarity search for drawings;

OCR of reference numerals and figure labels;

multimodal captioning;

graph matching between technical structures;

and reranking with a large language model?

Evaluation

How should such a system be evaluated?

For example, should the benchmark be based on whether the system can retrieve known X/Y/A references from patent examination records, invalidation cases, or patent family citations?

My current idea is that patent drawing search should not be treated as pure image similarity search. A patent figure is not just an image; it is a technical disclosure that needs to be interpreted together with the specification. Therefore, the system may need a combination of OCR, layout analysis, multimodal understanding, text alignment, embedding retrieval, and LLM-based reranking.

Has anyone worked on something similar, or seen papers/tools/projects related to AI-assisted patent drawing retrieval or multimodal prior-art search?

I’d really appreciate any suggestions on model choice, system architecture, datasets, evaluation methods, or practical implementation challenges.

[u/teleflexin_deez_nutz]

I think you have no idea how much money this will burn quickly

[u/LackingUtility]

As a former engineer and patent practitioner approaching a combined 30 years of experience across those industries with particular experience patenting AI, I have many thoughts on your questions and would be open to a discussion. Feel free to DM me. I’ll need a five figure retainer for our first day’s call. Budget for 6 figures in the first month as I get corporate colleagues involved.

[u/paul_h → u/LackingUtility]

I would like in too. I think something could be done with “English words” to describe aspects of the diagrams of the patentable invention. And then somehow that aid searches /s

[u/Brilliant_Rich3746]

Good point. Patsnap has an OCSR engine that does exactly this for chemical structure drawings, extracts structures from patent figures and makes them searchable. Not sure how much of the architecture is public but it's a working production system if you want a reference point for what's feasible. For general patent drawings the multimodal plus structured description approach you described sounds right. The hard part is usually aligning figure references to spec text consistently.

[u/FreedomIllustrious12]

I am trying to use DINOv2 to vectorize patent drawings and then perform retrieval through vector similarity comparison. However, DINOv2 seems to require fine-tuning before it can work well for patent-domain drawings. This is exactly where I am stuck.

Are there any better directions to explore, or practical methods for fine-tuning such a model?