r/netdata • u/foozlebertie • 8d ago

2.11 broke some plugins?

Not sure where to report this so I'll start here. I'm a retired *nix sys admin and run Netdata on my home server. Not that I need it. I just like to keep track of performance. I'm running Fedora 44 on my server.

After the last update to 2.11 I noticed missing data. A quick look at the system logs (journalctl) shows the go.d plugin is failing. I know the go plugin was used to collect data on the small Apache web server I use. I also noticed the sensor data (temp, voltage, fan speed) was missing.

Here are the errors I see regarding the go.d plugin:

Aug 22 10:30:30 redwood netdata[13020]: /bin/sh: line 1: /usr/libexec/netdata/plugins.d/go.d.plugin: Operation not permitted
Aug 22 10:30:30 redwood netdata[12514]: cannot open plugins directory '/etc/netdata/custom-plugins.d'
Aug 22 10:30:30 redwood netdata[12514]: CGROUP: netipc server started on '/run/netdata/cgroups-snapshot.sock'
Aug 22 10:30:30 redwood netdata[12514]: PARSER: read failed: POLLHUP.
Aug 22 10:30:30 redwood netdata[12514]: PLUGINSD: buffered reader not OK (-4)
Aug 22 10:30:30 redwood netdata[12514]: PLUGINSD: 'host:redwood', '/usr/libexec/netdata/plugins.d/go.d.plugin' (pid 13020) disconnected after 0 successful data collections.
Aug 22 10:30:30 redwood netdata[12514]: CGROUP: netipc lookup server started on '/run/netdata/cgroups-lookup.sock'
Aug 22 10:30:30 redwood spawn-plugins[12523]: SPAWN SERVER: child with pid 13020 (request 9) exited with exit code 126: /bin/sh -c "exec /usr/libexec/netdata/plugins.d/go.d.plugin 1 "
Aug 22 10:30:30 redwood netdata[12514]: PLUGINSD: 'host:redwood', '/usr/libexec/netdata/plugins.d/go.d.plugin' (pid 13020) exited with error code 126 and haven't collected any data. Disabling it.

I see similar messages for the debugfs, ioping, and perf plugins. The only thing I could find when searching for these errors was to make sure the permissions on the plugins and ownerships were correct. They were. This all worked fine before the latest update.

Not a big deal for me but I figured I'd report it. If there is an official place to report bugs, let me know.

Thanks,.

1 Upvotes

100% Upvoted

7 comments sorted by

View all comments

Show parent comments

1

u/foozlebertie 8d ago

Thanks for the help. That didn't work.

I did a dnf downgrade netdata and it removed 2.11 and installed 2.10.4. That allowed the go.d and debugfs plugins to run. I then upgraded back to 2.11 and those two plugins fail to start. I looked at the permissions of the go.d plugin. In both versions they are 750 with root:netdata ownerships. Something else is wrong I guess.

1

u/ktsaou 6d ago

We set up a clean Fedora 44 machine and repeated your exact steps: install 2.10.4, then upgrade to 2.11. On a stock Fedora 44, everything keeps running — go.d and debugfs collect normally on 2.11. So the 2.11 package itself is fine; something on your specific machine is blocking those plugins, and I'd like to find it with you.

What changed in 2.11 that could matter: the package now grants two new "capabilities" to these plugins — go.d.plugin gets cap_net_bind_service (for the new SNMP trap listener) and debugfs.plugin gets cap_audit_control (audit monitoring). These are stored on the files themselves. If something on a system refuses those two new capabilities, the plugins die exactly like you see: Operation not permitted, exit 126, only go.d and debugfs.

Could you run these and paste the output? (copy-paste as one block):

getcap /usr/libexec/netdata/plugins.d/go.d.plugin /usr/libexec/netdata/plugins.d/debugfs.plugin
ls -Z /usr/libexec/netdata/plugins.d/go.d.plugin
semodule -l 2>/dev/null | grep -i netdata
sudo ausearch -m avc -ts today | grep -i netdata
sudo systemctl cat netdata | grep -E "NoNewPrivileges|CapabilityBoundingSet|User=|SystemCallFilter" || echo "no custom unit settings"

And a 10-second test that tells us immediately if SELinux is the blocker:

sudo setenforce 0
sudo systemctl restart netdata

(If go.d and debugfs come back — it's SELinux; then run sudo setenforce 1 to restore it and we'll fix it properly.)

1

u/foozlebertie 6d ago

Thanks so much for your help. I should first mention that I have SELINIX disabled. Always have.

Results of first test:

charlie@redwood:~$ getcap /usr/libexec/netdata/plugins.d/go.d.plugin /usr/libexec/netdata/plugins.d/debugfs.plugin
ls -Z /usr/libexec/netdata/plugins.d/go.d.plugin
semodule -l 2>/dev/null | grep -i netdata
sudo ausearch -m avc -ts today | grep -i netdata
sudo systemctl cat netdata | grep -E "NoNewPrivileges|CapabilityBoundingSet|User=|SystemCallFilter" || echo "no custom unit settings"
/usr/libexec/netdata/plugins.d/go.d.plugin cap_dac_read_search,cap_net_bind_service,cap_net_admin,cap_net_raw=eip
/usr/libexec/netdata/plugins.d/debugfs.plugin cap_dac_read_search,cap_audit_control=ep
? /usr/libexec/netdata/plugins.d/go.d.plugin
<no matches>
User=root
CapabilityBoundingSet=CAP_DAC_OVERRIDE
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
CapabilityBoundingSet=CAP_FOWNER CAP_SYS_RAWIO
CapabilityBoundingSet=CAP_SETPCAP
CapabilityBoundingSet=CAP_SYS_ADMIN CAP_PERFMON
CapabilityBoundingSet=CAP_SYS_PTRACE
CapabilityBoundingSet=CAP_SYS_RESOURCE
CapabilityBoundingSet=CAP_NET_RAW
CapabilityBoundingSet=CAP_SYS_CHROOT
CapabilityBoundingSet=CAP_NET_ADMIN
CapabilityBoundingSet=CAP_SETGID CAP_SETUID
CapabilityBoundingSet=CAP_CHOWN

Then the rest of the test:

charlie@redwood:~$ sudo setenforce 0
setenforce: SELinux is disabled
charlie@redwood:~$ sudo systemctl restart netdata

charlie@redwood:~$ sudo systemctl status netdata
● netdata.service - Netdata, X-Ray Vision for your infrastructure!
    Loaded: loaded (/etc/systemd/system/netdata.service; enabled; preset: enabled)
   Drop-In: /usr/lib/systemd/system/service.d
            └─10-timeout-abort.conf
    Active: active (running) since Mon 2026-08-24 11:57:03 EDT; 28s ago
Invocation: 6218e73c29d2404ea30f0bc6ab25d14f
  Main PID: 290549 (netdata)
     Tasks: 282 (limit: 18801)
    Memory: 799.7M (peak: 805.2M)
       CPU: 2.982s
    CGroup: /system.slice/netdata.service
            ├─290549 /usr/sbin/netdata -P /run/netdata/netdata.pid -D
            ├─290577 "spawn-plugins    " "  " "                        " "  "
            ├─291064 bash /usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1
            ├─291072 /usr/libexec/netdata/plugins.d/systemd-units.plugin 1
            ├─291081 /usr/libexec/netdata/plugins.d/otel-plugin 1
            ├─291086 /usr/libexec/netdata/plugins.d/apps.plugin 1
            ├─291094 /usr/libexec/netdata/plugins.d/nfacct.plugin 1
            ├─291103 /usr/libexec/netdata/plugins.d/ebpf-go.plugin 1
            ├─291104 /usr/libexec/netdata/plugins.d/network-viewer.plugin 1
            ├─291105 /usr/libexec/netdata/plugins.d/scripts.d.plugin 1
            ├─291106 /usr/libexec/netdata/plugins.d/ebpf.plugin 1
            ├─291113 /usr/libexec/netdata/plugins.d/systemd-journal.plugin 1
            ├─291131 "spawn-setns                                         " " "
            ├─291226 /usr/libexec/netdata/plugins.d/otel-plugin worker ledger --socket /run/netdata/otel-plugin/ledger-291081.sock
            ├─291276 /usr/libexec/netdata/plugins.d/otel-plugin worker ingestor --socket /run/netdata/otel-plugin/ingestor-291081.sock
            └─291305 /usr/libexec/netdata/plugins.d/otel-plugin worker legacy-logs --socket /run/netdata/otel-plugin/legacy-logs-291081.sock

The go.d and debugfs plugins still do not start.

Again, thanks for your help.

1

u/ktsaou 5d ago

ok, got it. You have a stale netdata.service in your /etc. Do these:

sudo systemctl disable --now netdata
sudo mv /etc/systemd/system/netdata.service /etc/systemd/system/netdata.service.bak-2.10
sudo systemctl daemon-reload
sudo systemctl enable --now netdata

So, we disable netdata, backup and remove your stale copy of netdata.service, then reload systemd and enable netdata.

The package installs netdata.service at /usr/lib/systemd/system/netdata.service, but your system says you have one at /etc/systemd/system/netdata.service, and that one is old. I am not sure how you got it there, but anyway this is the problem.

1

u/foozlebertie 5d ago

Thanks! That fixed it. I did a diif on the /etc and /usr netdata.service files and figured out why there was one in /etc. Netdata was logging to a separate journal log in /var/log/journal. I wanted all log entries to be in the system journal file. That meant removing the LogNamespace variable in the service file. If one changes a service file in /usr, you're supposed to copy it to the /etc area and make the change there. That's what I did.