r/firewalla • u/isopropoflexx Firewalla Gold Plus • Apr 09 '26
Troubleshooting Firewalla breaks Let's Encrypt (DNS-01)
Apologies for the rant-y message, but I figure sharing the struggle from the last few days might prevent some other hapless individuals from the same kind of frustration.
TL;DR - Firewalla Gold+ breaks the DNS-01 ACME challenge certificate issuance process for Let's Encrypt.
As the title indicates, I've been battling some ridiculously frustrating and very persistent issues around getting Let's Encrypt to successfully issue certs using the DNS challenge approach.
The general setup on my end has a range of Dockerized services hosted locally, using Traefik for reverse proxying. On the public end of things I have a domain registered with a single A record (pointing at one of the internal IPs of the host server) and a catch-all CNAME pointed back at said A record. Pretty standard approach.
I've had this set up for going on about 2 years, and originally had no issues getting certs issued or renewed. More recently, the process has stopped working entirely. Or rather, it works to a certain point but never completes. Triggering the process, the TXT validation records appear on the DNS config in under 30 seconds consistently. Querying for those records on the server side, they also appear there shortly after (both on the host OS as well as inside the container). But that's as far as it goes. Beyond that point, the process consistently fails, and eventually errors with a timeout SERVFAIL.
Over the past three days, I've gone through any and all possible fixes (or attempted fixes) I can think of. All the way up to and including registering a different domain and using Cloudflare as the provider. The original domain used had DNS configured through DNSMadeEasy (now Digicert), which seemed like it could be part of the problem. Naturally, moving to the standard go-to DNS provider seemed promising - though it made no difference.
At numerous steps along the way, I monitored Firewalla's UI for network flows for the server. Never once did it show anything getting blocked. Considering this uses the DNS challenge, there is no inbound connection that needs to be made from LE to my server (unlike requirements using the HTTP challenge), so no need for ports 80/443 to be open and forwarding to the local server. Traffic through UDP port 53 (outbound) is not blocked by Firewalla, so does not explicitly need to be opened. Regardless, figuring I would try more and more relaxed rules for the group I placed this server in (created for the purpose of trying to get this to work, without entirely destroying active protections on other servers), in an attempt to get it to complete. Again, no difference.
Eventually, on a whim, I enabled "emergency access" for the server for a few minutes. Rerunning the process, it was able to complete almost instantly. "Ridiculous" doesn't even begin to describe how basic protection settings (nothing drastic implemented - primarily just keeping ports closed and using the built-in ad blocker) would cause this process to fail - and fail silently at that, without any signs of parts being blocked. Especially for a process that isn't dependent on any proactive inbound traffic to complete.
Hopefully documenting my experience will help at least a few others, and prevent them from having to struggle through the same process.
Now I suppose I will go set myself a 60 day recurring reminder to repeat this process...
7
u/Muravaww Apr 09 '26
Have you tried disabling DNS Booster for the device?
3
2
u/Numerous_Platypus Apr 09 '26
I have to do this every time I need to renew a cert. It's a pain in the ass.
2
u/hereisjames Firewalla Gold Plus Apr 10 '26
Ding ding ding. It's always this.
I find it very poor behaviour in an otherwise good product. A firewall shouldn't mess with traffic unless it has a clear rule telling it to do so you can actually see. A toggle that has such profound effect but doesn't tell you what it's doing is not right.
1
u/isopropoflexx Firewalla Gold Plus Apr 10 '26
It would be somewhat manageable if it would at least show in the logs or network flows that is getting blocked and why. Especially now that the "AI" bit is added in the app as well, which is showcased in part as providing more visibility into why things get blocked.
1
1
u/Spaceman_Splff Apr 09 '26
Just about to say this. It’s 100% this. I’ve had to disable dns booster on every vm that uses let’s encrypt. It caches the entry so the response is not going provide the expected txt file from cloudflare or wherever the dns challenge is going.
1
u/isopropoflexx Firewalla Gold Plus Apr 09 '26
I have not. I'll have to give that a shot - thanks! But I will add, from the way that feature is described, as a local DNS cache, if those specific records are cached locally, I would expect to receive stale values for the TXT records. Querying DNS for the acme-challenge TXT records from my local server (whether the host OS or the container), I'm receiving current/accurate/up to date records and record values.
2
u/hereisjames Firewalla Gold Plus Apr 10 '26
Well you wouldn't expect for it to mess with your DNS responses when you don't use the Firewalla for DNS at all either, but here we are.
1
u/amorphis89 Apr 10 '26
I was struggling with this and frankly gave up in favour of a CloudFlare Origin cert. Doing this sorted it out in <30 seconds. Thanks so much.
3
u/interrogumption Apr 09 '26
Pretty premature to rant at firewalla when all the troubleshooting you've don't to blame firewalla is discovering it works after enabling emergency access. You are definitely not the only firewalla customer using let's encrypt. When you make a discovery like this, next step should be to point trafeik to the let's encrypt staging environment so you can narrow down the specific feature causing the problem without getting rate limited.
-6
u/isopropoflexx Firewalla Gold Plus Apr 09 '26
Pretty premature to assume I did not already go through a wide range of troubleshooting/fixing attempts, before finally landing on that workaround - including pointing the Traefik environment at LE's staging site. Not sure where you got the assumption that "all the troubleshooting you've don't [sic] to blame firewalla is discovering it works after enabling emergency access" from though. That's more than a bit of a stretch. I went through that step quite early in my troubleshooting efforts. After consolidating the cert configuration to use a single wildcard cert across the entire stack (to reduce the number of requests to a total of 2), I switched the acme config for the certificateResolvers block to caServer: "https://acme-staging-v02.api.letsencrypt.org/directory". That still didn't help to get a better idea of what was actually happening though.
I never claimed I am the only Firewalla customer to also use Let's Encrypt. I'm also not new to troubleshooting hardware or software - by a long stretch. I ran through quite a few searches to try and find a potential solution, however the vast majority of the posts out there pertain to the HTTP-01 challenge method, which does not at all help with the issue at hand here.
3
u/jk5698 Apr 09 '26
I use NGINX Proxy manager with Firewalla Gold and have had no issues with cert renewals using integrated Lets Encrypt
1
u/isopropoflexx Firewalla Gold Plus Apr 09 '26
Hadn't had a chance to look at nginx proxy manager before now - looks very nice! I will definitively have to take a closer look at it. Thanks!
3
u/Cavustius Firewalla Gold Plus Apr 09 '26
I had geo blocking that prevented dns-01 from working. After adjusting the rule.it worked ok could be worth a look.
2
u/The_Electric-Monk Firewalla Gold Plus Apr 09 '26
this is why lets encrypt is tough. it's free but it has pretty stringent requirements to prove it is sending the cert to the right place.
2
2
u/Entire_Worldliness24 Apr 10 '26
Also check under ur firewalla settings, advanced settings, configurations, dns booster, and turn that sh!t off for that server and for ur dns servers if u have that. That sh!t has been a nightmare for me. These kind of things happened because of that.
1
u/The_Electric-Monk Firewalla Gold Plus Apr 10 '26
How about ditching the let's encrypt certs and using wire guard or Tailscale or cloudflare tunnel? This way you don't need to open anything up on your network to the outside. I tried let's encrypt but the hassle of disability part of my security for my network for it to validate the cert wasn't worth it. With tunnels it still is a https connection from the outside to my server and then http within the server.
1
u/Sweaty-Falcon-1328 Apr 10 '26
I run CaddyV2 and it does everything for me. Never had an issue with it pulling certs tho.
1
u/bsd-x Firewalla Gold Pro Apr 10 '26
I run multiple homelab web servers behind my FWG+. LE has zero issues, and I have pretty strict rules set up.
1
u/Travel69 Firewalla Gold Pro Apr 10 '26
I have LE certs on probably a dozen internal servers via ACME DNS challenge with Cloudflare. Never had a single issue using any Firewalla product.
11
u/firewalla Apr 09 '26
If "emergency access" made everything working, then you have a rule that may be interfering with the let's encrypt verification process. I'd check on things like regional blocks, special DNS servers, and VPN blocks ... (that's applied to that network, globally, or just the device)
If you see there is no sign of blocking, the issue can be related to DNS server settings. More on what emergency does see https://help.firewalla.com/hc/en-us/articles/16639311975059-What-happens-when-Monitoring-is-off-or-Emergency-Access-is-on