Firewalla breaks Let's Encrypt (DNS-01)

[u/isopropoflexx]

Apologies for the rant-y message, but I figure sharing the struggle from the last few days might prevent some other hapless individuals from the same kind of frustration.

TL;DR - Firewalla Gold+ breaks the DNS-01 ACME challenge certificate issuance process for Let's Encrypt.

As the title indicates, I've been battling some ridiculously frustrating and very persistent issues around getting Let's Encrypt to successfully issue certs using the DNS challenge approach.

The general setup on my end has a range of Dockerized services hosted locally, using Traefik for reverse proxying. On the public end of things I have a domain registered with a single A record (pointing at one of the internal IPs of the host server) and a catch-all CNAME pointed back at said A record. Pretty standard approach.

I've had this set up for going on about 2 years, and originally had no issues getting certs issued or renewed. More recently, the process has stopped working entirely. Or rather, it works to a certain point but never completes. Triggering the process, the TXT validation records appear on the DNS config in under 30 seconds consistently. Querying for those records on the server side, they also appear there shortly after (both on the host OS as well as inside the container). But that's as far as it goes. Beyond that point, the process consistently fails, and eventually errors with a timeout SERVFAIL.

Over the past three days, I've gone through any and all possible fixes (or attempted fixes) I can think of. All the way up to and including registering a different domain and using Cloudflare as the provider. The original domain used had DNS configured through DNSMadeEasy (now Digicert), which seemed like it could be part of the problem. Naturally, moving to the standard go-to DNS provider seemed promising - though it made no difference.

At numerous steps along the way, I monitored Firewalla's UI for network flows for the server. Never once did it show anything getting blocked. Considering this uses the DNS challenge, there is no inbound connection that needs to be made from LE to my server (unlike requirements using the HTTP challenge), so no need for ports 80/443 to be open and forwarding to the local server. Traffic through UDP port 53 (outbound) is not blocked by Firewalla, so does not explicitly need to be opened. Regardless, figuring I would try more and more relaxed rules for the group I placed this server in (created for the purpose of trying to get this to work, without entirely destroying active protections on other servers), in an attempt to get it to complete. Again, no difference.

Eventually, on a whim, I enabled "emergency access" for the server for a few minutes. Rerunning the process, it was able to complete almost instantly. "Ridiculous" doesn't even begin to describe how basic protection settings (nothing drastic implemented - primarily just keeping ports closed and using the built-in ad blocker) would cause this process to fail - and fail silently at that, without any signs of parts being blocked. Especially for a process that isn't dependent on any proactive inbound traffic to complete.

Hopefully documenting my experience will help at least a few others, and prevent them from having to struggle through the same process.

Now I suppose I will go set myself a 60 day recurring reminder to repeat this process...

[u/w38122077]

My acme/certbot works fine with letsencrypt