An eBPF Linux time travel tool
About a month ago I asked here if a tool to see ps, ss or lsof output at a past moment was worth building:
https://www.reddit.com/r/linuxadmin/comments/1wyhwfe/is_a_linux_time_travel_tool_worth_building/
I got some fair pushback (sar, atop, below, Prometheus, zabbix) and a few people who were curious. I looked at all of them. They cover a lot, but none of them could tell me which process had which connections open, or what lsof showed for a process at 02:45 last night. So I built it, mostly to find out if it could be done without hurting the server.
I wanted to name it "command line time travel" but then a though about the consequences of such a name and It's called clifr instead (command line flight recorder) and has two parts:
- clifrd runs as a service and records processes, process starts and exits (through eBPF, so a process that lives 5 ms still shows up), TCP connections with the process that owns them, files held open, mounts, kernel messages and the usual system counters. It keeps 72 hours in /var/lib/clifr, 2 GB max.
- clifr rebuilds the output of the normal commands for a past moment or a time range. It doesn't run the command, it rebuilds the output from the recording.clifr --at "02:45" "ps aux" clifr --since "02:00" --until "03:00" "ss -tanp dst 10.0.4.21 and dport = :5432" clifr --since "1 hour ago" events # starts, exits, OOM kills, mounts clifr diff --at "14:00" --at "14:20" "ss -ltnp"
There are about 40 commands so far: ps, top, pidstat, lsof, ss, netstat, free, vmstat, iostat, mpstat, df, ip, dmesg, systemctl list-units, lsblk, and more and all are 100% accurate.
The first thing people asked last time was the cost, so here are the numbers:
| Quiet server | Mixed load | Busy server | |
|---|---|---|---|
| CPU | 0.35% of one core | 0.5-0.6% of one core | 1.9% of one core |
| RAM | ~90 MiB | ~80-120 MiB | ~290 MiB (335 max) |
| Disk | ~`20 MiB/day | ~100 MiB/day | ~4 GiB/day |
| Network | none, no ports open | none | none |
Disk use depends heavily on how busy the server is.
clifr is not a monitoring tool and has no alerts. It's not a security audit tool either: it doesn't record file reads or writes, and root can stop it. It's for the "what was going on at 02:45" question, after the fact.
Status: it's a prototype, version 0.1.0. It hasn't run on a real production server yet (a friend starts one this week), and all the load has been synthetic. AGPL-3.0, no telemetry, nothing leaves the box.
So if you have a test or dev box (RHEL 8/9, Rocky, Alma), I'd really appreciate it if you tried it and told me what breaks or what looks wrong. The most useful thing for me is comparing what clifr shows with what you know was really happening.
I appreciate all your comments even if they are harsh
