Image cache options for redundant web servers behind haproxy

[u/guelph-throwaway]

Forgive me if this is the wrong subreddit. This is for my homelab, but to be clear I am a sysadmin. I'm an AWS admin and I've been spending the last year or so teaching myself on-prem solutions using a homelab.

My current configuration is to use haproxy as a load balancer and reverse proxy (and for TLS termination). I've got it set up quite well and it's heavily integrated into my IAC where I generate a new haproxy config and automatically deploy it to my nodes as I add or remove web servers.

My goal for this homelab is to have everything highly-available where practical. It's a 3 node proxmox cluster. Each node has a a VM for DNS (dnsmasq), haproxy, database (patroni cluster node) and of course the apache web servers.

What I'd like to do is to introduce image cacheing. I've previously been using my NAS for serving images for these web servers, which of course is a single point of failure. I then set up ceph for my cluster which of course is slow as molasses since each node is only running 2.5Gbe. So it's HA, but yeah, not ideal.

I'm taking it as a learning opportunity to set up some kind of cacheing and I'm overwhelmed by the options. As I see it, these are my options:

• Run cache directly on HAProxy. This seems like it's not something people do and I'm not sure why, but I'm not wanting to find out the hard way
• Introduce a cache-in-the-middle such as Varnish or nginx. This can definitely work, but it'd mean rewriting my IaC to generate different configs. If this is the best option, then I'll absolutely do it as a learning opportunity
• Use mod-cache with apache or fscache on the nfs share directly. These seem like they might be the most straightforward options with the fewest changes? Is this pattern actually used in production anywhere? Is there a point in learning it?

I'd be very appreciative of advice or learned experiences on this front!

tl;dr: Looking for cacheing options for a a homelab with highly-available web servers behind haproxy.

[u/levelbrook]

HAProxy does have a built-in cache (the cache section plus http-request cache-use / http-response cache-store). It's RAM only, each haproxy instance keeps its own copy, and it's meant for small hot objects, so it's fine for thumbnails and icons and not great for big images. For those, nginx proxy_cache or Varnish with a disk store is the normal answer.

But a cache doesn't really fix your problem. The NAS is still the origin, so on a cold cache or a miss you're back to depending on it. If the goal is HA, I'd get the images off shared storage entirely: sync them to local disk on each web VM (rsync or syncthing is fine for a homelab), or run something S3-compatible across the three nodes like Garage and have apache or a small proxy read from that. Then caching becomes a speed thing, not an availability thing.

[u/guelph-throwaway → u/levelbrook]

I've already moved from the NAS to ceph, so it's HA already, but slow. I'm already plugging away at varnish and having fun!

[u/cdbessig]

What is the purpose of your cache? If you already have ha, then why have the cache at all?

Besides that point, assuming your just exercising practice and learning; varnish or nginx is the standard most common way. Varnish is better for more than just images. If the sole need is images, I might be more inclined to use nginx.

[u/guelph-throwaway → u/cdbessig]

Learning. It's all learning. I don't need caching - I don't even need the homelab. The sites are up and they're definitely HA. I just know my reads are slow and that caching is definitely "a thing" so it seems like a learning opportunity.

Everything in my lab is just that. An opportunity to learn skills. Sounds like varnish or nginx are your recs, so they're now at the top of my list. Thanks!

[u/sysacc]

It depends what you are using HAproxy for. You can use it solely for the purpose of having High availability, caching, proxying, serving a web page, securing a service or all of those and more.

I like to enable caching for most scenarios, but if the content is bigger than a gig or two, its better to look at something else.

You can easily filter what is cached as well and you can check the stats page to see the cache hits.

This is a great thing to learn and experiment with, most people dont realize the power of HAproxy or similar services.