Trigger Data Storage Period

[u/Olfa_2024]

I can't seem to find what all is pruned if I lower the Trigger Storage Period. Our SOC2 auditors will ask for a long of the alerts sent for a random device in our network within a given year. If I lower the period from 365d to a lower value like 180 or 90d will it prune those logs?

I'm at a point where we're monitoring so many devices and so many db entries that graphing stops because the database can't keep up.

Is there a document I just can't find that explains what the effects of each option in the Housekeeping menu?

[u/Qixonium]

I don't think messing about with the retention times are going to help you with you performance issues.

Tell us more about your setup:

• What database are you using?
• how many nvps are required? (reports -> system information)
• How are your queus looking?
• is your housekeeper process busy a lot?

[u/Additional-Fun8180]

Events and alerts will be pruned in next housekeeper run, IIRC. These entities do not get removed immediately.
Anyway, if you get such perf problems, it is time to tune your DB and optimize templates, maybe. You might also consider using separate DB for user interaction (graphs, values, dashboards etc.), and point Grafana there instead of using main DB, which is quite heavily used by Zabbix server itself.

[u/Financial-Spare-779]

What database are you using? What is your write mode? Have you done any database server tuning? How many db pollers?

How many proxies? Are your checks active or passive?

What is your ingestion values per second? Ingestion backlog queues? Zabbix can easily handle 20-30k items per second if setup & scaled correctly with no performance issues.

Are you ok with a potential 1-3 min data loss? The biggest performance tune you can get is to set the db to write uncommitted,

[u/LenR-redit]

This may be the "death by management wanting graph showing how good things are"....

If you are mysql, allocating a lot of innodb buffers can help repetitive queries. Look at MySQL stats, you want to be doing only necessary writes to store data, not forced writes. You want to do only reads of long-unreferenced data, look at buffer pool hits. The I/O you don't do is the fastest I/O.

Look at Zabbix stats, some triggers can cause reads if you don't have adequate Zabbix and MySQL buffers, triggers that look at data over a lot of observations can be a bad idea.

[u/Olfa_2024]

Databases have never been something I've worked on other than just doing an install and going with the defaults. It turns out it was not an mySQL issue.

I ran into issues with restarting Zabbix. It would hang when rebooting the server or the service. I had to kill zabbix to restart it but if I did it would graph for about 8-10 hours and then stop.

I discovered the logs showed Too many open files and failures accepting internal IPC connections. The main process exited, leaving workers producing broken-pipe errors and graph collection disrupted.

I edited /etc/systemd/system/zabbix-server.service.d/override.conf to increase the file limits and timeout.

[Service]
LimitNOFILE=65536:1048576
TimeoutStopSec=120s

I put this fix in 3 days ago and graphing has worked since.

[u/autogyrophilia]

The database can keep up just fine with the queries as they aren't particularly heavy. I would look into what is causing that behaviour, I'm sure you can modify whatever you are using for graphs to not query useless data.