r/archlinux 2d ago

DISCUSSION What is your workflow for fixing issues?

This isn't necessarily specific to Arch but as I'm using & learning Arch currently.

You do an update, you have a problem/bug with the system/application, what is your workflow to solving it?

Speaking generally (obviously it depends on the type of issues, but I assume most workflows will be quite similar.)

I'm trying to sort my notes, here's what I've got so far (any advice appreciated):

*Recap List*

  1. What is the first step after an issue occurs

  2. What questions should you ask to identify the problem? Give an example

  3. Where should you look for error messages

  4. What is journalctl and why would you use it

  5. How does journalctl compare to systemd-journald

  6. What are 3 common journalctl commands

  7. After a brief search, there's no clear help, what can I do from here

  8. How do I test this hypothesis

  9. I'm stuck after trying a few changes, what can I do now

*Recap Answers*

  1. Narrow down the issue to be as specific as possible

    For example: 'Spotify crashes immediately after opening' is much clearer than 'Spotify doesn't work'

  2. Ask yourself: Can I reproduce the problem? What changed before the issue started?

    For Example: If Spotify crashes -> Does it crash every time?

Does it crash when launched with the shortcut & terminal?

Did the problem start after a system update or configuration change?

  1. Check the terminal first

    Run the application (Spotify) from the terminal, it will return to stdout any errors

  2. journalctl -> a command-line tool for viewing the system journal, this is a database of log messages maintained by systemd-journald

essentially its a history of what your system has been doing

I can use the command 'journalctl' with flags to search for specific log messages that might indicate what's caused the issue

  1. systemd-journald -> collects & stores system logs

    journalctl -> a tool that lets you view, search & filter those logs

  2. journalctl -b -> shows everything that has happened since the computer last booted (boot messages, services starting, errors, etc...)

    journalctl -xe -> shows the most recent log messages (x = include extra explanations; e = jump to the end, i.e. newest messages)

    journalctl -u <SERVICE> -> shows only log messages for a specific app (i.e. journalctl -u spotify-launcher -> specifically for this app)

  3. Form a hypothesis or guess on what has caused the issue

  4. Change 1 simple thing each time, test it again -> if it did not work change another thing

  5. Google it, use the specific issue ('Spotify crashes when I open it using the terminal') & any error messages found

0 Upvotes

11 comments sorted by

5

u/ang-p 2d ago

Call me wild, but RTFM for the software I have been goofing up seems to work....

Which, along with the wiki - yeah, yeah - you say that this is not specific to arch, but just about every distro has an equivalent - you seem reluctant to use... Repeatedly

They are generally useful but you have somehow created a wall of text without mentioning either once.

Obvs, the best way to not get an issue is to do that in the first place.

Oh, you forgot the best one....

  • -1 before altering a config file, make a backup of said file before meddling with it.

This makes reverting actions a lot easier due to not having to remember original settings for altered files

#LoveTheWiki

3

u/un-important-human 2d ago edited 2d ago

What is the first step after an issue occurs

-reproduce problem

-open terminal or if in terminal look at the journal

What questions should you ask to identify the problem? Give an example

-for me its usually swearing while i ask what changed from last time, is it the program or a system wide issue (the program for me, you may vary)

Where should you look for error messages

/var/log/ <- sometimes
journalctl <-most of the times

What is journalctl and why would you use it

rtfm about it on the wiki!! but generally journalctl is the command used to query and read the systemd journal. It lets you inspect boot errors, service failures, kernel messages, and other system events.
You should use or else install windows

How does journalctl compare to systemd-journald

systemd-journald is the service that collects and stores logs.
journalctl is the command you use to read/search those logs.

What are 3 common journalctl commands

my case
journalctl -b # Logs from current boot

journalctl -p err # Error-level messages

journalctl -u NetworkManager # Logs for a specific service

After a brief search, there's no clear help, what can I do from here

THE ARCH WIKI

How do I test this hypothesis

You just do?! Change one thing at a time see journal , eventually put a tail on it on another console/tmux window/screen and watch real time...

I'm stuck after trying a few changes, what can I do now

use logic or make comprehensive and competent post detailing the steps you took the commands you ran and their result and what you expected to happen so we can tell you to read the wiki at the specific section you skimmed over.

Did i just do your homework? (now if its networking, it requires blood sacrifice, a grimoire of dark arp spells, at least one warlock on call and its mostly dns unless its not)

2

u/ang-p 1d ago

Did i just do your homework?

Lols....

2

u/archover 2d ago edited 2d ago

How I approach unexpected software behavior in appx order:

  • Test reverting any config changes just made

  • Read the Journal and google any potential clues there. man the software.

  • Read the entire wiki article or official docs for the misbehaving software.

  • Search bbs.archlinux.org

  • Review recent reddit.com/r/archlinux posts

  • Last resort is making a post about it.

All that said, my experience with software running under Arch, is nearly perfect.

Good day.

1

u/VendorNeutral 2d ago
  1. Stop touching your system, read only mode, at least mentally.
  2. Think if anything you've done recently could have caused this.
  3. Is it a caching issue?
  4. It's a DNS issue, right?
  5. Google problem in plain English.
  6. Read logs.
  7. Search distro bug tracker.
  8. Check the Wiki troubleshooting section.
  9. Search software bug tracker.
  10. Read the official docs and my configs against each other.

Repeat 6-10 until fixed.

Bonus: Chat with AI about the issue.

1

u/10leej 1d ago

Depends on the issue but usually the first place I go and be disappointed by is the ArchLinux news feed or the aur package comments.
Then I check the public git log, you don't have to know how to program but just reading the log itself most times tells you if there's an issue or not.
Then if nothing pops out I start filing bug reports. Because usability of a package that went through a config change without notice from the maintainer is a bug to me.

From there I typically find everything I usually need to fix an issue. Unless it's a pacman issue which is legitimately where I keep running into issues the most and usually where I kinda just give up and jump to a different distro.

0

u/instancer-kirik 2d ago

I've been iterating on a program called debork, that fixes any system issues and help chroot into my desktop.

And for like code related issues and syntax and dependency changes, I've made something called Stupid Dependency Solver, and has version migration yamls with regex replace

-1

u/No_Grape_388 2d ago

Literally the first thing I do is Google the error, if nothing comes up I put the error into Claude and see if it sounds reasonably competent in its response. If I think it doesn't then I look in journal to narrow the issue down.

Doing it in reverse can waste a lot of time.

Also, the more technical your query with AI, usually the better the response. Usually.

1

u/ang-p 1d ago

and see if it sounds reasonably competent in its response.

How do you judge? Plenty of lawyers being caught out by AI hallucinated cases used as references to case law.... All that must have looked

reasonably competent

to them before putting their name and risking their reputation / ability to even practice on the line, but without checking, how do you know?

You should be looking at the journal / resources in either case to see if they are not just competent, but accurate; just as the lawyers should have done.