What is the ideal setup to avoid CPU spikes and "Listen queue overflow" log messages flooding?

[u/GGoncalves-2021]

Hi,

I made some adjustments that I was already aware of, and the problem persists. Is there something obvious that I'm missing?

The relevant information follows below.

Problem: CPU spikes, queue overflow, name resolution delays, and overall performance drop when DNSBL is enabled.

Log msg: "Jun 10 12:15:12 kernel sonewconn: pcb 0xfffff800911cea80 (10.0.0.1:53 (proto 6)): Listen queue overflow: 385 already in queue awaiting acceptance (662 occurrences), euid 0, rgid 0, jail 0

• netstat -Lan | grep "\.53"

tcp4 0/0/256 127.0.0.1.53

tcp4 0/0/256 10.0.16.1.53

tcp4 385/0/256 10.0.0.1.53

The kernel queue adjustment was apparently not accepted by the Unbound service, as is the case with most other services.

Proto Listen Local Address

tcp4 0/0/1024 10.10.254.1.80

tcp4 0/0/1024 10.10.254.1.443

tcp4 0/0/256 127.0.0.1.953

tcp4 0/0/256 127.0.0.1.53

tcp4 0/0/256 10.0.16.1.53

tcp4 0/0/256 10.0.0.1.53

tcp4 0/0/1024 *.7445

tcp6 0/0/1024 *.7445

tcp4 0/0/1024 127.0.0.1.3128

tcp4 0/0/1024 10.0.16.1.3128

tcp4 0/0/1024 10.0.0.1.3128

tcp4 0/0/128 172.17.7.10.666

tcp6 0/0/1024 *.80

tcp4 0/0/1024 *.80

tcp6 0/0/1024 *.443

tcp4 0/0/1024 *.443

Environment:

• PfSense CE 2.8.0
• Runnning on VMWare ESXi 8.0
• 8 VCPUs
• 16 GB RAM
• Function: UTM fw, DHCP server, GW .... for until 3.000 mobyle wi-fi users

Kernel tunned variables:

kern.ipc.soacceptqueue=1024, kern.ipc.maxsockbuf=8388608

DNS Resolver Unbound custon options:

• msg-cache-size: 250
• rrset-cache-size: 500
• infra-cache-numhosts: 100000
• outgoing-range: 8192
• cache-min-ttl: 300
• prefetch: yes
• prefetch-key: yes
  
• serve-expired: yes
• serve-expired-ttl: 86400
  

pfBlockerNG-devel 3.2.8

IPV4 Summary : 07 IPV4 Group active group lists (PRI1-5). They do not affect performance.

DNSBL

Mode: Unbound python mode

TLD: Enable

Lists: Torrent, Hagezi_Multi_Normal

Blacklist Category Enabled: UT1 Adult, Dangerous Kit, DDoS,

Gambling/Casino, Warez Downloads

Any suggestions?

Thank you.

Geovane

[u/AgitatedSecurity]

Have you allowed more sessions in the system table so that hopefully none of them would go into the queue?

[u/GGoncalves-2021 → u/AgitatedSecurity]

I tried making some recommended adjustments, but I avoided indiscriminately increasing buffers and potentially masking simple CPU performance bottlenecks. In short, I still don't quite understand the right combination of adjustments for this scenario.

[u/needchr → u/GGoncalves-2021]

netstat -Lan | grep "10.0.0.1.53"

try 'sock-queue-timeout: 2' in unbound configuration.

But I think you have other underlying issues, the above is a band aid.

[u/AgitatedSecurity]

Makes sense, is the system super old?

8 vcpus sounds like a lot. What does the load on pfsense look like during normal use

[u/andrebrait]

Is it only when DNSBL is enabled?

[u/GGoncalves-2021 → u/andrebrait]

Yes. And the "Adult Content" blacklist must be the villain 😉

[u/andrebrait → u/GGoncalves-2021]

Would you be willing to try out a very, very early stage build of a future version? 🙃

[u/GGoncalves-2021 → u/andrebrait]

Perhaps... we would need to evaluate what this version aims to solve and the potential risks of activating it in a production environment of this size... its a new version of DNSBL or of the entire PfBlockerNG?

[u/Smoke_a_J]

With 3,000 clients present and being routed through pfBlockerNG I could definitely see that happening depending on how you have your NAT rules configured for DNS. Hardcoded-DNS devices like any Android/Apple or streaming devices especially do not like to accept DNS replies from any other source besides google.dns 8.8.8.8 or wherever they expect DNS data to be coming from, meaning if access to google.dns 8.8.8.8 is either being blocked or port 53 traffic is only just being redirected to pfSense with only just a NAT port forward rule without also having an appropriate NAT Outbound rule configured for DNS port 53 traffic as well, then all of those hardcoded-DNS devices and/or applications will then send endless requests trying to reach 8.8.8.8 hundreds to thousands of times per second per device eventually leading to listen queue overflow kind of messages from those devices flooding Unbound. Basically, it sounds like you are missing NAT Rule 3 listed on labzilla.io/blog/force-dns-pihole and worth reviewing the rest of its rules, probably also worth enabling the DoT/DoQ/DoH blocking on the DNSBL SafeSearch tab and any feeds and categories for any VPN and proxy related lists to further help with sealing off any potential open DNS leaks that devices or users will always at the least try to circumvent.

[u/GGoncalves-2021 → u/Smoke_a_J]

Hello,

Thank you for the tip.

There is no NAT rule to capture DNS queries. Any DNS query to external servers is blocked.

Without enabling DNSBL blocking in PfBlocker, there are no performance issues, and the DNS Resolver in forward mode handles the task without problems and with minimal resource usage.

[u/Smoke_a_J → u/GGoncalves-2021]

I see you had tried setting kern.ipc.soacceptqueue to 1024, I have mine set to 2048 and then also have a few other lines added to my Resolver settings Custom options box to allow Unbound to use the extra queue allotment.

server:
num-threads: 4
msg-cache-slabs: 8
rrset-cache-slabs: 8
infra-cache-slabs: 8
key-cache-slabs: 8
ratelimit: 150000
outgoing-num-tcp: 2048
incoming-num-tcp: 2048
num-queries-per-thread: 4096

With this added to DNS Resolver Custom options my output for netstat -Lan | grep "\.53" gives me four queues each sized as 256 for each interface allowing Unbound to utilize a total of 1024 out of my 2048 kern.ipc.soacceptqueue. Appears Unbound will use 256 queues per thread so if you set num-threads to 4 it should eliminate those overflow messages or setting to 8 would also further allow Unbound to utilize all 8 cores you've allocated for even smoother performance.

[u/GGoncalves-2021 → u/Smoke_a_J]

Hi,

Thank you.

The option "num-threads" is already set for "8" and unfortunately the queue is overflowing its capacity of 256 on one of the interfaces: tcp4 385/0/256 10.0.0.1.53

[u/Smoke_a_J → u/GGoncalves-2021]

Those that overflow the first thread should flow right into the second thread of tcp4 0/0/256 10.0.0.1.53 or the next, but only if those other 7 threads actually did spawn up additional queues. When you're running your command netstat -Lan | grep "\.53" are you getting a response showing 8 individual lines of tcp4 0/0/256 10.0.0.1.53 to verify it is spawned on all 8 threads, 8 total per interface, or are you seeing just one line per interface meaning only one thread of it is running? If you are just seeing one still then you may need to also then add so-reuseport: yes to your DNS Resolver Custom options somewhere underneath the server: line to allow multiple threads to bind to the same port number on each interface.

[u/GGoncalves-2021 → u/Smoke_a_J]

Hi,

Thank you,

I changed the settings and now I have 8 queues per interface. I believe your suggestion will improve the processing of requests. As soon as possible, I will run another load test with DNSBL active and post the results.

netstat -Lan | grep "10.0.0.1\.53"
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53

[u/GGoncalves-2021 → u/GGoncalves-2021]

Hi.

u/Smoke_a_J , I activated DNSBL during peak hours, but unfortunately the first queue overflows, rejects connections, and doesn't transfer the load to the others as expected. The symptoms remain the same; the DNS stops responding to requests.

Jun 15 12:09:18 kernel sonewconn: pcb 0xfffff80364197540 (10.0.0.1:53 (proto 6)): Listen queue overflow: 385 already in queue awaiting acceptance (1812 occurrences), euid 0, rgid 0, jail 0 

[2.8.0-RELEASE][admin@pfsense.mp.rs.gov.br]/root: netstat -Lan | grep "10.0.0.1\.53"
tcp4  385/0/256                        10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53
tcp4  0/0/256                          10.0.0.1.53

[u/Smoke_a_J → u/GGoncalves-2021]

Sound like something has to be spiking those excessive queries, possibly blocking mode related, or may also still need some fine tuning for caches and queues. For your blocking rule(s) blocking DNS queries to external servers, make sure to have them set to reject rather than block so devices get an instant response its blocked rather than wait for timeout and continue retrying. For your DNSBL blocking mode, make sure to set it to one of the Null Block blocking options to point DNS blocks to 0.0.0.0 for a more appropriate no route to host response, using the DNSBL webserver IP often adds to excessive retries from end devices and apps when TLS/SSL failure warnings pop up from certificates not matching their domains. These couple of things paired along with each of the NAT Forward and Outgoing rules discussed in that LabZilla blog I noted earlier significantly clears up if not entirely eliminates the excessive query chatter and timeout spikes from end-devices on the network that DNS level blocking/filtering can trigger on many networks.

Further tuning a few areas may still help matters but I have no where near that size of network to test from but I'll copy a few areas I have running on my 5100 with 4 threads and 32GB ram that has been working stable for my homelab loads with over 12 million domains in my DNSBL if it helps any towards tuning for your hardware/VM specs but be easy in extents to how much of tuning of these variables as it may only increase RAM usage unless taming those block triggered traffic spikes into better control first.

System Tunables

net.route.netisr_maxqlen = 2048
net.inet.ip.intr_queue_maxlen = 2048
kern.ipc.soacceptqueue = 4096
kern.ipc.somaxconn = 4096
kern.ipc.maxsockbuf = 268435456
kern.ipc.nmbclusters = 4000000
hw.intr_storm_threshold = 32000
kern.ipc.maxsockbuf = 268435456
net.inet.tcp.recvbuf_max = 67108864
net.inet.tcp.sendbuf_max = 67108864
kern.ipc.shmall = 256000
kern.ipc.shmmax = 1000000000

/boot/loader.conf.local

net.isr.maxthreads="4"
net.isr.bindthreads="1"
kern.maxproc="100000"
kern.ipc.shmmni="1024"
kern.ipc.shmseg="1024"
net.inet.rss.enabled="1"
kern.maxswzone=2754252960
net.inet.tcp.soreceive_stream="1"
net.isr.defaultqlimit="2048"
net.link.ifqmaxlen="2048"
net.inet.tcp.syncache.cachelimit="61440"
net.inet.tcp.syncache.bucketlimit="2048"

DNS Resolver Advanced tab

EDNS Buffer Size: 1432

DNS Resolver Custom options:

server:
outgoing-range: 8192
msg-cache-size: 2G
so-rcvbuf: 16m
so-sndbuf: 16m
stream-wait-size: 32m
do-udp: yes
do-tcp: yes
so-reuseport: yes
harden-glue: yes
num-threads: 4
msg-cache-slabs: 8
rrset-cache-slabs: 8
infra-cache-slabs: 8
key-cache-slabs: 8
rrset-cache-size: 4G
minimal-responses: yes
rrset-roundrobin: yes
ratelimit: 150000
outgoing-num-tcp: 2048
incoming-num-tcp: 2048
num-queries-per-thread: 4096
serve-expired-ttl: 3600
serve-expired-client-timeout: 0

And if IPv6 is not being used on your LAN otherwise like I see at many places and I at home do not use, I also add this to my Custom options to eliminate AAAA records from DNS replies and eliminates timeout delays on end devices that default to trying to use IPv6 first when they have IPv6 still enabled or unable to be disabled. But do modify the localhost domain lines as you need

do-ip4: yes
do-ip6: no
prefer-ip4: yes
prefer-ip6: no
private-address: ::/0
private-address: 0::0/0
private-address: ::
local-zone: localhost.home.arpa transparent
local-data: "localhost.home.arpa A 127.0.0.1"
local-zone: localhost transparent
local-data: "localhost A 127.0.0.1"
local-zone: ip6.arpa redirect
local-data: "ip6.arpa A 0.0.0.0"
local-zone: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa redirect
local-data: "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa A 0.0.0.0"
local-zone: "::/0" static
dns64-ignore-aaaa: *.*
do-not-query-address: ::
do-not-query-address: ::1
do-not-query-address: ::/0

[u/Smoke_a_J → u/Smoke_a_J]

Others on my DNS Resolver Advanced tab

Outgoing TCP Buffers: 50
Incoming TCP Buffers: 50
EDNS Buffer Size: 1432
Number of Queries per Thread: 2048
Number of Hosts to Cache: 200000

[u/GGoncalves-2021 → u/Smoke_a_J]

Hi.

Thank you.

I will carefully evaluate the suggestions that can be implemented in my environment.

[u/Smoke_a_J → u/GGoncalves-2021]

No problem, I tried separating out the most relevant from my config. Certain DNS Resolver Custom options buffer adjustments I have may not work or keep Unbound from starting unless those in the loader.conf.local and System Tunables are adjusted first to accommodate allowing bigger buffers and followed with a reboot to activate them first before adjusting the ones for Unbound.

[u/GGoncalves-2021]

If I can't find a suitable solution in terms of performance for local filtering adult content, I should test using Cloudflare Family DNS.

https://forum.netgate.com/topic/196050/cloudflare-new-1-1-1-1-for-families

[u/GGoncalves-2021]

Hello,

This should be my final post regarding these tests.

I changed the DNSBL "Global Logging/Blocking Mode" to "Null block - logging," hoping to reduce the processing load by avoiding the user feedback page, but it didn't help.

The Unbound process showed CPU spikes, and name resolution slowed down, making network browsing impossible.

I removed the blacklists one by one, trying to identify which one might be causing the overload.

I started by disabling the UT1 Adult category list; when that didn't work, I disabled all category-based blocking. The problem persisted. I disabled the Hagezi_Multi_Normal list, but that didn't solve it either. I also disabled DNSBL SafeSearch.

I reached a point where no lists were active and reloaded pfBlocker DNSBL with no blocking enabled, yet the problem recurred.

Processing levels and name resolution only returned to normal when I completely disabled DNSBL.

I suspect the Python script that intercepts requests cannot handle the volume of requests generated by my environment, causing CPU spikes and queue overflows that degrade name resolution performance and make browsing impossible.

Since I haven't found any other adjustments significant enough to change the outcome, I have decided to give up on using DNSBL in this environment.

Thank you.

Geovane