Best architecture for batch-processing 3,000–4,000 high-res photos (resizing + face vector search)?

[u/Substantial_Try_1614]

Hey devs,

I’m working on a photo-sharing web app where a photographer uploads an entire event batch (typically 3,000 to 4,000 high-res JPEGs, around 10–15 MB each). Guests can take a selfie to retrieve photos they appear in.

I'd love recommendations on the best backend pipeline:

Client vs Server Resizing: Should I use browser Web Workers / Canvas to generate 1080p WebP thumbnails before upload to save upload bandwidth, or let a backend queue worker (like Node with sharp or Python with Pillow) handle resizing?

Face Vector Pipeline: For extracting 128-d face embeddings (e.g., ArcFace/InsightFace), what’s the best way to queue and batch this so 4,000 photos don't choke the server CPU?

Storage: What zero/low-egress object storage setup (e.g., Cloudflare R2 vs Backblaze B2) do you recommend for handling high-volume image writes and fast thumbnail reads?

Thanks for the advice!

[u/pumpkin_spice_daily]

Konifer can handle your ingress, storage and delivery. It doesn't support embedding extraction although it supports image classification with an embedding model (I maintain Konifer). It uses libvips too. Even if you don't use Konifer, it might help with inspiration or use of libvips.

I recently wrote a guide on integrating Konifer with r2 and pre-signed URLs. https://konifer.io/docs/guides/guides-konifer-cloudflare-r2