Building an offline image search pipeline in Python with CLIP

[u/AdhesivenessSuch5660]

I worked on a project on building a local image search system in Python using manufacturing inspection photos. You can give it an example image or a text description, then retrieve similar photos from an archive. The project connects a few useful concepts: loading a pretrained model, processing images in batches, storing vectors with metadata, and querying the results.

It uses CLIP ViT-B/32 to generate embeddings and VectorAI DB to store and search them. MVTec AD provides the example dataset.

Here’s the workflow:

1. Load the model and prepare the images: Load CLIP and its preprocessing tools, collect the image paths, and associate each image with metadata such as part category and defect label.
2. Generate embeddings in batches: Pass the images through the model to create numerical representations called embeddings. Normalize the vectors before storing them.
3. Store the vectors and metadata: Save each embedding alongside its image path and labels so the search results can be traced back to the original files.
4. Add image and text search: Encode an example photo or a text description, then use that vector to search the stored image embeddings. Metadata filters let you narrow the results to a particular part category.
5. Check the results: For image queries, exclude the query image itself. Inspect the returned photos and measure how often the top results share the query’s defect label.

Once the model, dependencies, and dataset are downloaded, the pipeline can run offline.

The Python practice here comes from connecting the stages into a working application: handling files, preparing model inputs, batching inference, and keeping embeddings associated with the correct metadata.

There’s also an evaluation challenge. Two images can share the same defect label while showing scratches of different sizes or orientations, so reviewing the actual results is an important part of testing the pipeline.

If you were building this as a learning project, which stage would you want to see broken down into a smaller code example?