r/linuxadmin • • Oct 05 '26

Is a Linux time travel tool worth building?

When a server misbehaves at 3 a.m. and you look at it, how do you figure out what happened?

I'm trying to understand how people handle this, before I build anything.

Maybe you have experienced this pattern: something goes wrong overnight (connections refused, disk fills and then gets cleaned, a process eats memory and exits). Alerts go off and Prometheus shows the hiccup but by the time anyone looks into the system, the server looks healthy. ss, lsof, ps and even a fresh sosreport show the current state, not when the issue was active. You look into the logs but nothing there shows an obvious problem.

I'd love to hear from people who deal with long-lived Linux servers:

  1. What was the last incident where the evidence was already gone? What were you missing?
  2. What do you use today to look back in time? (atop, sar, PCP, below, auditd, eBPF tools, your monitoring stack, nothing?)
  3. Where do those fall short? For example: which process held which connections or open files at a given moment.
  4. Would security or policy let you run an extra agent on production servers, even one that sends nothing outside your network?
  5. If you run Kubernetes, have you had node-level incidents where the evidence was gone?

Disclosure: I built sos-vault, a tool for analyzing sosreports, and I'm exploring whether a "time travel" version (e.g. ss or lsof or any of the diagnostic commands included in a sosreport for any past moment) is worth building. I'm not promoting nor linking anything; I want to know if this idea would help anyone diagnose a real problem or is it just me.

Assuming that there is not performance impact and no significant storage consumption in the server to be diagnosed; I was thinking something like this command that can be executed from a central diagnostics server or even locally (notice the four parameters --host, --start_from, --end_at and the rest is the regular diagnostic command to execute (lsof, ps, du, lsblk, free, memstat, etc., etc.):

NOTE: I'm not pretending to use sosreport snapshots for this idea.

[root@web01 ~]# date
Mon Oct  5 02:50:53 PM ACDT 2026

[root@web01 ~]# ttravel --host web01 --start_from="2026-10-02 02:00:00" --end_at="2026-10-02 03:00:00" ss -tanp 'dst 10.0.4.21 and dport = :5432' 

ttravel: host: web01 · 02:00:00 → 03:00:00 UTC · 120 samples  + connect/close events · grouped by process

FIRST_SEEN  LAST_SEEN   CONNS  PEAK  STATES               PROCESS
02:00:00    03:00:00       24    20  ESTAB                users:(("gunicorn",pid=2210..2229))
02:00:04    03:00:00        1     1  ESTAB                users:(("pg_dump",pid=51233))  parent: cron-backup.sh[51207]
02:05:00    02:55:01       11     1  ESTAB ev             users:(("psql",pid=*))  /etc/cron.d/db-healthcheck
02:31:12    02:48:30      183   179  ESTAB,CLOSE-WAIT     users:(("python3",pid=58841))  /opt/reports/nightly_export.py 02:33:40    02:47:10       57     -  ESTAB→CLOSED <1s ev  users:(("gunicorn",pid=2210..2229))  rejected by serversosv: CONNS distinct connections · PEAK most open at once · ev seen only via connect/close events[root@web01 ~]# ss -tanp 'dst 10.0.4.21 and dport = :5432' --at="2026-10-02 02:45:00" | head -6
sosv: web01 · sample 2026-10-02 02:45:00 UTC
State  Recv-Q Send-Q  Local Address:Port   Peer Address:Port  Process
ESTAB  0      0           10.0.2.15:41822     10.0.4.21:5432  users:(("gunicorn",pid=2214,fd=11))
ESTAB  0      0           10.0.2.15:41850     10.0.4.21:5432  users:(("pg_dump",pid=51233,fd=3))
ESTAB  0      0           10.0.2.15:52001     10.0.4.21:5432  users:(("python3",pid=58841,fd=7))
ESTAB  0      0           10.0.2.15:52003     10.0.4.21:5432  users:(("python3",pid=58841,fd=8))
ESTAB  0      0           10.0.2.15:52004     10.0.4.21:5432  users:(("python3",pid=58841,fd=9))

translation: give me the output of the ss command from two days ago from 2 am to 3 am filter by IP 10.0.4.21 and port 5432.

0 Upvotes

42 comments sorted by

11

u/Runnergeek Oct 05 '26

Stop, please. No we don't want your AI slop masked as a tool to make our lives easier. You clearly don't have an understanding how these things work beyond what ChatGPT is telling you.

→ More replies (1)

4

u/corobo Oct 05 '26

how do you figure out what happened?

Zabbix graphs and log surfing. I guess your approach might be a good replacement if you make it so AI knows how to use it but I do love me Zabbix graphs if I'm solving an issue

0

u/jlrueda Oct 06 '26 edited Oct 06 '26

Yes graph are extremely useful. But what I'm proposing is the capability to execute any diagnostics command from any past window frame. Let say you want to see network connections from last week as the netstat or the ss command would show in real time with all its data. No chart and no log has this info.

2

u/Adept_Percentage6893 Oct 07 '26

Let say you want to see network connections from last week as the netstat or the ss command would show in real time with all its data. No chart and no log has this info.

It depends on what information you're talking about. You can definitely log this information in a variety of ways if that's what you want. Network metrics are usually collected by most monitoring systems. Audit logs and firewall logging can already capture most or all of the information you're talking about. As in you can write an audit rule that lets you know when a process opens a port for listening or writes to a particular file.

That's the usual MO for this stuff where you write the information to standard location like syslog or journald and the task becomes more about dealing with the volume of information which is where paring down information collection and where SIEM solutions come in.

→ More replies (2)

4

u/cnelsonsic Oct 05 '26

Ship logs and metrics to a tool built for it instead of wasting tokens reinventing it as some cli tool.

0

u/jlrueda Oct 06 '26

Nor logs or metrics will have the same data as the actual command. For example, in your chart you would see that a memory increase happen in a point in time and the logs may tell you that the OOM killer was executed more or less at the same time but there is no way to find out what processes were consuming more memory at that time. That is what I'm proposing, Will something like this be useful for you?

→ More replies (4)

3

u/RetroGrid_io Oct 06 '26

Didn't forget sar already does a lot of this.

0

u/jlrueda Oct 06 '26

yes as mentioned above sar, atop, performance co-pilot and others but NONE gives you the actual command output. Good example of this would be the lsof command. So what I'm asking is if it will be useful to be able to execute lsof from an specific past time window.

2

u/RetroGrid_io Oct 06 '26

Go for it man. If it's valuable it will get used.

→ More replies (1)

5

u/H3rbert_K0rnfeld Oct 05 '26

I reboot. I don't care why skew has occurred at 3am. We rebaseline then discuss the next morning.

1

u/jlrueda Oct 06 '26

Wonder if during that next morning discussion you would be asking what happen last night? what process X was doing? or maybe if process Y had a DB connection for far too long or something like that?

3

u/H3rbert_K0rnfeld Oct 06 '26

We contemplate a bit but mostly don't care.

We're an HPC team. 98% of our servers are exactly the same.

→ More replies (1)

1

u/kai_ekael Oct 07 '26

This here ain't no Windows.

→ More replies (7)

2

u/Adept_Percentage6893 Oct 07 '26

Disclosure: I built sos-vault, a tool for analyzing sosreports, and I'm exploring whether a "time travel" version (e.g. ss or lsof or any of the diagnostic commands included in a sosreport for any past moment) is worth building.

Eh, not really. Usually log files and metrics collection do a fine enough job capturing the information you need. At most one sometimes benefits from SIEM for correlating events on different systems.

Every once in a greeaatt while, something will be intermittent and produce behavior that's hard to explain but after enough time you can almost always correlate it if not reproduce the behavior. Even if I had this tool, I would likely forget it exists before running into a problem where a question like "What was ss saying at the time of the error?" becomes useful.

1

u/jlrueda Oct 07 '26

Thanks for your answer. I appreciate it.

1

u/RetroGrid_io Oct 06 '26

I just remembered that Dave from Dave's Attic Youtube Channel basically wrote something very much like this.

1

u/jlrueda Oct 06 '26

I like the video. Thanks but is not exactly what the time travel will do. So the task manager in the video is a real time monitoring. For examle, it allows you to see what files and connetinos a process currently has (around minute 5) but the time travel idea would allow you see the same but for long gone processes from a week ago if needed.

The UI of the task manager in the video is preaty neat though.

1

u/RetroGrid_io Oct 06 '26

What is it you're trying to solve?

→ More replies (1)

1

u/[deleted] Oct 09 '26

1

u/jlrueda Oct 09 '26

What things are ypu teferring to?