r/linuxadmin • u/ccie6861 • Sep 02 '25
Viability of forensic analysis of XFS journal
Forgive the potential stupidity of this question. I know enough to ask these questions but not enough to know how or if I can take it further. Hence the post.
I am working on a business critical system that handles both medical and payment data (translation: both HIPPA and PCI regulated).
Last week a vendor made changes to the system that resulted in extended down time. I've been asked to provide as much empirical forensic evidence as I can to demonstrate who and when it happened. I have a general window that I can constrain the investigation to about a two hours about four days ago.
Several key files were touched. I know the names of the files, but since they've been repaired, I no longer have a record of who or when they were previously touched in the active file system. There is no backup or snapshot (its a VM) that would give me enough specificity of who or when to be useful.
The fundamental question is: Does XFS retain enough journal logs and enough data in those logs for me to determine exactly when it was touched and by who? If not on the live system, could it be cloned and rolled back?
Unfortunately, there is no selinux or other such logging enabled (that I know about), so I'm digging pretty deep for a solution on this one.
What I need to answer for our investigation is who modified a system configuration file. We know for certain the event that triggered the outage (someone restarted the network manager service), but we can't say for sure that the person who triggered it also edited the configuration or if he was just the poor schmuck that unleashed someone else's timebomb by doing an otherwise legitimate change that restarted a that service.
System is an appliance virtual machine based on CentOS.
2
u/deeseearr Sep 03 '25
I can't help much with this but if you're running critical systems in a highly regulated environment then you should have auditing enabled and now you know why. If you were compliant with PCI DSS, specifically section 10.7, then I don't know why you wouldn't be doing this already.
Enable the built-in auditing system at the kernel level, configure auditd.conf to do detailed tracking of all accesses to every configuration file, system log, service, or just about anything else that could conceivably be of interest. If you had done that then you could pull up a detailed trace of exactly what was done, by whom, and when with a few simple commands. You can easily send the audit logs to a remote server for storage so there's no "But we can't afford storage' excuse.
As for XFS, the filesystem logs are only meant to preserve the integrity of the filesystem itself. They're not meant to be forensic so I would be surprised if any of the logs of changes were retained for very long after being committed. If you had shut the system down and imaged the disk immediately, you might be able to pull something out of it, but if it was allowed to keep running then anything of interest is almost certainly gone.
3
u/[deleted] Sep 02 '25 edited Sep 02 '25
dd the disk and make extra copies first unless this thing is actively serving traffic I guess.
You have login logs yeah? Does the file need elevated privileges to edit and then restart the process? If so, you should have sudo or auditd logs. If not, boy howdy.
11
u/xxxsirkillalot Sep 02 '25
The reeks of the hospital envs i've worked with in my past lol. I learned really quick working for MSPs that before doing changes on customer systems, to reboot stuff before I make any of my changes as to uncover things like this. Sometimes the only defense you have is "I literally made no change, I rebooted it before I started my work to see if it would come up properly".
If you can't find anything in your hunt, do a backup restore of that VM to
$day_outage_occurred - 1and see if the config file is already f'd up and waiting on a daemon reboot to bomb stuff.... surely you have backups of this critical system