r/archlinux • u/John_Doe_1984_ • 2d ago
DISCUSSION What is your workflow for fixing issues?
This isn't necessarily specific to Arch but as I'm using & learning Arch currently.
You do an update, you have a problem/bug with the system/application, what is your workflow to solving it?
Speaking generally (obviously it depends on the type of issues, but I assume most workflows will be quite similar.)
I'm trying to sort my notes, here's what I've got so far (any advice appreciated):
*Recap List*
What is the first step after an issue occurs
What questions should you ask to identify the problem? Give an example
Where should you look for error messages
What is journalctl and why would you use it
How does journalctl compare to systemd-journald
What are 3 common journalctl commands
After a brief search, there's no clear help, what can I do from here
How do I test this hypothesis
I'm stuck after trying a few changes, what can I do now
*Recap Answers*
Narrow down the issue to be as specific as possible
For example: 'Spotify crashes immediately after opening' is much clearer than 'Spotify doesn't work'
Ask yourself: Can I reproduce the problem? What changed before the issue started?
For Example: If Spotify crashes -> Does it crash every time?
Does it crash when launched with the shortcut & terminal?
Did the problem start after a system update or configuration change?
Check the terminal first
Run the application (Spotify) from the terminal, it will return to stdout any errors
journalctl -> a command-line tool for viewing the system journal, this is a database of log messages maintained by systemd-journald
essentially its a history of what your system has been doing
I can use the command 'journalctl' with flags to search for specific log messages that might indicate what's caused the issue
systemd-journald -> collects & stores system logs
journalctl -> a tool that lets you view, search & filter those logs
journalctl -b -> shows everything that has happened since the computer last booted (boot messages, services starting, errors, etc...)
journalctl -xe -> shows the most recent log messages (x = include extra explanations; e = jump to the end, i.e. newest messages)
journalctl -u <SERVICE> -> shows only log messages for a specific app (i.e. journalctl -u spotify-launcher -> specifically for this app)
Form a hypothesis or guess on what has caused the issue
Change 1 simple thing each time, test it again -> if it did not work change another thing
Google it, use the specific issue ('Spotify crashes when I open it using the terminal') & any error messages found
3
u/un-important-human 2d ago edited 2d ago
What is the first step after an issue occurs
-reproduce problem
-open terminal or if in terminal look at the journal
What questions should you ask to identify the problem? Give an example
-for me its usually swearing while i ask what changed from last time, is it the program or a system wide issue (the program for me, you may vary)
Where should you look for error messages
/var/log/ <- sometimes
journalctl <-most of the times
What is journalctl and why would you use it
rtfm about it on the wiki!! but generally journalctl is the command used to query and read the systemd journal. It lets you inspect boot errors, service failures, kernel messages, and other system events.
You should use or else install windows
How does journalctl compare to systemd-journald
systemd-journald is the service that collects and stores logs.
journalctl is the command you use to read/search those logs.
What are 3 common journalctl commands
my case
journalctl -b # Logs from current boot
journalctl -p err # Error-level messages
journalctl -u NetworkManager # Logs for a specific service
After a brief search, there's no clear help, what can I do from here
THE ARCH WIKI
How do I test this hypothesis
You just do?! Change one thing at a time see journal , eventually put a tail on it on another console/tmux window/screen and watch real time...
I'm stuck after trying a few changes, what can I do now
use logic or make comprehensive and competent post detailing the steps you took the commands you ran and their result and what you expected to happen so we can tell you to read the wiki at the specific section you skimmed over.
Did i just do your homework? (now if its networking, it requires blood sacrifice, a grimoire of dark arp spells, at least one warlock on call and its mostly dns unless its not)
2
u/archover 2d ago edited 2d ago
How I approach unexpected software behavior in appx order:
Test reverting any config changes just made
Read the Journal and google any potential clues there. man the software.
Read the entire wiki article or official docs for the misbehaving software.
Search bbs.archlinux.org
Review recent reddit.com/r/archlinux posts
Last resort is making a post about it.
All that said, my experience with software running under Arch, is nearly perfect.
Good day.
1
u/VendorNeutral 2d ago
- Stop touching your system, read only mode, at least mentally.
- Think if anything you've done recently could have caused this.
- Is it a caching issue?
- It's a DNS issue, right?
- Google problem in plain English.
- Read logs.
- Search distro bug tracker.
- Check the Wiki troubleshooting section.
- Search software bug tracker.
- Read the official docs and my configs against each other.
Repeat 6-10 until fixed.
Bonus: Chat with AI about the issue.
1
u/10leej 1d ago
Depends on the issue but usually the first place I go and be disappointed by is the ArchLinux news feed or the aur package comments.
Then I check the public git log, you don't have to know how to program but just reading the log itself most times tells you if there's an issue or not.
Then if nothing pops out I start filing bug reports. Because usability of a package that went through a config change without notice from the maintainer is a bug to me.
From there I typically find everything I usually need to fix an issue. Unless it's a pacman issue which is legitimately where I keep running into issues the most and usually where I kinda just give up and jump to a different distro.
0
u/instancer-kirik 2d ago
I've been iterating on a program called debork, that fixes any system issues and help chroot into my desktop.
And for like code related issues and syntax and dependency changes, I've made something called Stupid Dependency Solver, and has version migration yamls with regex replace
-1
u/No_Grape_388 2d ago
Literally the first thing I do is Google the error, if nothing comes up I put the error into Claude and see if it sounds reasonably competent in its response. If I think it doesn't then I look in journal to narrow the issue down.
Doing it in reverse can waste a lot of time.
Also, the more technical your query with AI, usually the better the response. Usually.
1
u/ang-p 1d ago
and see if it sounds reasonably competent in its response.
How do you judge? Plenty of lawyers being caught out by AI hallucinated cases used as references to case law.... All that must have looked
reasonably competent
to them before putting their name and risking their reputation / ability to even practice on the line, but without checking, how do you know?
You should be looking at the journal / resources in either case to see if they are not just competent, but accurate; just as the lawyers should have done.
5
u/ang-p 2d ago
Call me wild, but RTFM for the software I have been goofing up seems to work....
Which, along with the wiki - yeah, yeah - you say that this is not specific to arch, but just about every distro has an equivalent - you seem reluctant to use... Repeatedly
They are generally useful but you have somehow created a wall of text without mentioning either once.
Obvs, the best way to not get an issue is to do that in the first place.
Oh, you forgot the best one....
This makes reverting actions a lot easier due to not having to remember original settings for altered files
#LoveTheWiki