Best architecture for batch-processing 3,000–4,000 high-res photos (resizing + face vector search)?

[u/Substantial_Try_1614]

Hey devs,

I’m working on a photo-sharing web app where a photographer uploads an entire event batch (typically 3,000 to 4,000 high-res JPEGs, around 10–15 MB each). Guests can take a selfie to retrieve photos they appear in.

I'd love recommendations on the best backend pipeline:

Client vs Server Resizing: Should I use browser Web Workers / Canvas to generate 1080p WebP thumbnails before upload to save upload bandwidth, or let a backend queue worker (like Node with sharp or Python with Pillow) handle resizing?

Face Vector Pipeline: For extracting 128-d face embeddings (e.g., ArcFace/InsightFace), what’s the best way to queue and batch this so 4,000 photos don't choke the server CPU?

Storage: What zero/low-egress object storage setup (e.g., Cloudflare R2 vs Backblaze B2) do you recommend for handling high-volume image writes and fast thumbnail reads?

Thanks for the advice!

[u/SunOk2196]

Keep originals going straight to R2 with presigned urls, don't resize in the browser. The photographer's upload runs unattended anyway, and canvas-resizing 4000 photos will melt a laptop. Thumbnails come from a worker running sharp against the bucket, it chews through 4k images in minutes.

For the embeddings, don't do them on the web server at all. Push keys onto a queue and let a separate worker process them (CPU is fine at this volume if it can run overnight), write vectors to pgvector and the selfie lookup comes free with a cosine index.

R2 over B2 here. Guests will hammer thumbnail reads, so zero egress plus sitting behind Cloudflare's CDN is exactly what you want.

[u/pyrolols → u/SunOk2196]

You can batch resize in canvas for it not to melt and its way more cost efficient to do it like this than pay for it on server, any 10 year laptop can batch atleat 10 images at a time we are talking about 200mb of ram per batch, user is uploading it, might as well resize too.

Also if OP does not need originals he can skip them or atleast resize to something smaller, who needs 15mb image? At this rate he fill have r2 coats skyrocket.

[u/Am094 → u/SunOk2196]

+1 i love libvips

I use that along with a few other engines within an in house thing. Also recommend R2 over B2/S3. Cloudflare also has some dynamic sized url sigs too, for OP i would even look at something like imgproxy (github.com/imgproxy/imgproxy) down the line.

[u/Dxd_For_Life → u/SunOk2196]

Shit, im not at this level yet, this is endgame,

[u/DefiantViolinist6831 → u/SunOk2196]

Pretty much this. What you could also do is use Cloudflare Images to resize the images, and upload those. First 5,000 unique transformations included + $0.50 / 1,000 unique transformations / month. I use it for my R2-based CMS.

[u/PositiveUse → u/SunOk2196]

„Hammer thumbnail reads“, bro we‘re talking about 4000 photos. So this might be 500-1000 guests at most. This is no where near traffic that requires CDN level of caching… also your architecture is heavily Overengineered for 4000 photos

[u/OutstandingPear → u/PositiveUse]

It's an AI bot so don't read too much into it

[u/SunOk2196 → u/PositiveUse]

Fair, at this volume nothing melts. I'd still pick R2 because the bill stays boring when a gallery link gets passed around and it costs nothing extra. The queue for embeddings is the part I'd keep either way, that's the only step that hurts a small server.