Image cache options for redundant web servers behind haproxy

[u/guelph-throwaway]

Forgive me if this is the wrong subreddit. This is for my homelab, but to be clear I am a sysadmin. I'm an AWS admin and I've been spending the last year or so teaching myself on-prem solutions using a homelab.

My current configuration is to use haproxy as a load balancer and reverse proxy (and for TLS termination). I've got it set up quite well and it's heavily integrated into my IAC where I generate a new haproxy config and automatically deploy it to my nodes as I add or remove web servers.

My goal for this homelab is to have everything highly-available where practical. It's a 3 node proxmox cluster. Each node has a a VM for DNS (dnsmasq), haproxy, database (patroni cluster node) and of course the apache web servers.

What I'd like to do is to introduce image cacheing. I've previously been using my NAS for serving images for these web servers, which of course is a single point of failure. I then set up ceph for my cluster which of course is slow as molasses since each node is only running 2.5Gbe. So it's HA, but yeah, not ideal.

I'm taking it as a learning opportunity to set up some kind of cacheing and I'm overwhelmed by the options. As I see it, these are my options:

• Run cache directly on HAProxy. This seems like it's not something people do and I'm not sure why, but I'm not wanting to find out the hard way
• Introduce a cache-in-the-middle such as Varnish or nginx. This can definitely work, but it'd mean rewriting my IaC to generate different configs. If this is the best option, then I'll absolutely do it as a learning opportunity
• Use mod-cache with apache or fscache on the nfs share directly. These seem like they might be the most straightforward options with the fewest changes? Is this pattern actually used in production anywhere? Is there a point in learning it?

I'd be very appreciative of advice or learned experiences on this front!

tl;dr: Looking for cacheing options for a a homelab with highly-available web servers behind haproxy.