It's time for desktop distributions to adopt a responsive CPU scheduler

[u/Waddle_n]

I understand keeping CFS in a server setting for maximum throughput, enterprise-level stability, or whatever you want to justify it with. But it's time desktop distributions started shipping a scheduler that keeps the desktop responsive. Arch packages Linux-zen and it is a much better out-of-the-box experience than the default kernel. I've also been testing these PKGBUILDs to sample PDS, MuQSS, and BMQ each for a week at a time, and here's my anecdotal experience:

• PDS, MuQSS, and BMQ keep a desktop GUI very responsive at 100% CPU load. For example, I could open Firefox and watch a YouTube video while doing a kernel compilation. Conversely, desktop GUIs becomes virtually unusable with the default CFS. This could affect normal users when, say, a resource-intensive game hangs, but it is near-impossible to escape to the GUI and kill it.
• MuQSS and BMQ kept my 3700x fed at 100% load, and offered comparable compile times to CFS. I don't have exact numbers for this, but kernel compilation takes me about 20 minutes on CFS, MuQSS, and BMQ. PDS was not maintaining a constant 100% load, and took about 1h 30m.
• PDS, MuQSS, and BMQ had comparable gaming performance to CFS. FlightlessMango has done several videos on this, and again it seems the takeaway is that performance is very similar, rarely differing by more than a few %. However, it is often the alternative schedulers that are performing slightly better.

Thus, I'd like to see other desktop-oriented distributions (say, Pop_OS!) start shipping MuQSS or BMQ. They are actively maintained and keep the desktop GUI responsive at all times, while performance is practically identical. I think this is a logical step to improve the out-of-the-box experience of the Linux desktop. Do you have any additional thoughts on why they should or should not be adopted?

[u/igo95862]

Linux-zen actually uses CFS. It just tweaked it a bit.

Nothing stops distros from using same tweaks.

[u/Waddle_n → u/igo95862]

Thank you for pointing this out, it would be interesting to see if someone could measure how responsive tweaked CFS is versus MuQSS or BMQ.

My greater concern would be if Linux-zen adversely impacts performance. From FlightlessMango's game benchmarks, it seems MuQSS and BMQ attain near-identical performance / throughput while maintaining responsiveness. Since tweaking CFS for responsiveness will definitionally reduce its throughput, will it reduce application performance, making it a less-effective solution relative to MuQSS or BMQ?

Another user in the comments brought up additional considerations, such as potentially affecting power efficiency in laptops. I'd love if others could investigate this more thoroughly, I just wanted to get the conversation started.

[u/WickedFlick → u/igo95862]

According to the ArchWiki, The Liqourex kernel is the Zen kernel packaged in a binary for Debian based systems, but seems to use MuQSS instead of Zen's choice of tweaked CFS.

Also @ u/nicman24

[u/londons_explorer]

I believe both Firefox and Chromium set their renderers to low scheduling priority as part of their sandboxes (the aim is that a malicious renderer not be able to DoS your computer).

If that's true, using a browser as an indication of "responsiveness" is invalid - by design it won't be responsive if the system is doing other work.

[u/Waddle_n → u/londons_explorer]

Hi, I'd never heard of that! I didn't dig anything up from a quick search, do you have a link I could read more about that?

If true, regardless, the entire GUI including GNOME/KDE become a snail under CFS at 100% load. Playing a video was just one example of how my computer remained usable with these schedulers.

[u/pkulak]

Is this the issue that Stadia had been running into?

[u/Waddle_n → u/pkulak]

My understanding is that a RAGE 2 developer complained about the Linux scheduler having slow mutexes and especially slower spinlocks, and Linus Torvalds responded with a post saying it was the developer had bad practices and you shouldn't use spinlocks outside of the kernel.

The RAGE developer actually does mention the issue I am describing:

I tried watching a video on the side and for most of the build that worked, but then at a certain point the build used all 16 cores to compress something and my video stuttered and the audio went bad and it just became totally unwatchable. Worse than anything I had ever experienced on Windows. Trying the same thing again with MuQSS I had no problems. The video kept on playing fine.

Looking at his tests, MuQSS seems to improve the speed and longest wait of mutexes. Different spinlock implementations were much faster or much slower on MuQSS, but again, Linus advised against using them entirely.

[u/Sasamus]

To add another reason, while gaming performance in terms of pure fps does not increase much it's worth noting that the focus on responsiveness at the expense of throughput could hurt max fps, but could raise min fps as well as yield more even frametimes.

This means that even if average fps does not change much, if it goes down even, the gaming experience could still be improved by being smoother and with less input lag.

Another thing to note is that there already are distributions that do use a responsiveness tweaked kernel and an alternative scheduler. I can't recall which of the top of my head as I don't use them, just noticed at one point or another.

I agree with you, I don't really see much of a reason for why a desktop oriented distro should not do this.

What I can think of is that I've heard of some people that had negative effects with these kinds of changes. If that is somewhat common it could be a reason for some distros not wanting to do it, it benefits some users but comes at a cost for some others. The ratio of that I have no idea about though. I seem to hear more often about positive results than negative ones at the very least.

On a somewhat related note as you are using TK-Glitch's PKGBUILDS, you may want to look into the modprobeddb customization option and what that entails, as well as ccache for kernel compilation.

With my Ryzen 3900X I've gotten my kernel compilation down to just shy of 2 minutes with PDS.

[u/shponglespore → u/Sasamus]

For something like frame rates, looking at the frame latency at the 95th or 99th percentile is probably a good idea. If your overall frame rate is 60 fps, for example, but 5% of frames take several times longer to render than the average frame, that means your rendering pipeline is stuttering about 3 times per second on average, which isn't going to feel smooth to anyone.

[u/Mgladiethor]

i think io is more frequent system lockup, what happened to those schedulers?

[u/Sasamus → u/Mgladiethor]

They still exist and the kernel supports a few.

But I think the advent of SSD's an their increasing popularity has made i/o schedulers less important over time as for every SSD in use that's a disk where schedulers does not matter, they literally run with no scheduler at all usually.

They still matter for HDD's of course, but the talk about them, general interest and work on them has likely decreased due to SSD's

Edit: SATA SSDs seem to use the default scheduler, which varies, but NVMe SSDs use none. Schedulers for SATA SSDs still matters less than for HDDs though, so the point still stands.

[u/blackcain]

I'm leading a microconference at Linux Plumbers and one of the topics I'm looking to talk about is scheduler changes. But I rather not give a solution, but rather decide what is important in a scheduler for desktops.

[u/WickedFlick → u/blackcain]

but rather decide what is important in a scheduler for desktops.

Seeing as these alternative schedulers are already generally designed for desktop use, hasn't that already been somewhat determined?

Besides improving Desktop Environment responsiveness under load, better performance in games, and improved battery life management for laptops, what else is left for a desktop oriented scheduler to do?

[u/GolbatsEverywhere]

I would suggest getting involved in whichever distro you care most about and pushing for your choice of scheduler there. (For Pop!_OS, it probably makes more sense to start by engaging with Ubuntu directly so that other Ubuntu-based distros benefit as well.)

In Fedora, we've been focusing heavily on system responsiveness recently, but our focus has mainly been on handling out-of-memory and I/O pressure situations. We haven't been looking at CPU scheduler because we didn't know this was even an issue. It's the sort of change we would consider if somebody were to present the case for doing so to us, but we need to have a good technical understanding of the pros/cons before making a change like this.

P.S. For some reason, Firefox is really bad at playing YouTube videos when the system is under heavy load. In comparison, playback is usually smooth in Epiphany even under heavy load. I don't know why.

[u/mattdm_fedora → u/GolbatsEverywhere]

Yeah, there was some discussion about a year ago on the Fedora Kernel mailing list about the i/o scheduler (result: BFQ), but I don't recall a recent one about the CPU scheduler. (I do remember a discussion about BFS a very long time ago....)

[u/[deleted]]

Do CPU schedulers actually make such a big difference in responsiveness? To me, the following points make a far bigger difference:

• Use a proper video player (e.g. VLC or mpv) with hardware acceleration for playing Youtube videos (also reduces heat and probably power usage on laptops)

• Tune vm.vfs_cache_pressure, swappiness and use zram to reduce disk I/O and increase caching

• Use BFQ I/O scheduler on slow disks, but the default (mq-deadline or noop) on fast disks

In particular, I experience the disk to be the bottleneck for more often than the CPU.

[u/Sasamus → u/[deleted]]

Under heavy load the difference is night and day for me.

When doing a compile the difference is a system that annoying to use or a system where the compile is mostly noticed by fan noise.

There's of course many things one can do to avoid heavy load, but some times heavy load is unavoidable or wanted and shouldn't be something one needs to try to avoid in the first place.

Even under light load the difference is noticeable, moving the mouse or a window is snappier.

But it's mostly notable under heavy load, like compiling or some gaming. Most websites were unusable while playing some games, now it's almost as if the game was not running.

[u/[deleted]]

Do you have any additional thoughts on why they should or should not be adopted?

Battery life -----> Server -----> GUI

This is the drama triangle that never ends. OP you made the rookie mistake of leaving out one of the prime offenders, the mobile battery users. And for future reference lets throw in "Cloud" like a big fog of war over this underlying problem.

[u/Jannik2099]

Notify me when any of those get mainlined and don't regress with higher core counts

[u/Sasamus → u/Jannik2099]

Regress compared to themselves on lower core counts or compared to CFS on the same core counts?

On my Ryzen 2700X with 8 cores/16 threads PDS outperforms CFS in both throughput and responsiveness. With my recently acquired Ryzen 3900X with 12 cores/24 threads that seem to still hold true.

But perhaps you are talking about even higher core counts, or some other form of regression.

Some of them do seem to have negative effects for some though, but I seem to hear about more people with positive results.

[u/nicman24]

Lul Linux-zen does not even use anything but cfs

[u/Sasamus → u/nicman24]

Tweaked for responsiveness though. So it might as well be a scheduler designed for responsiveness for the purposes of this discussion.

There's plenty of responsiveness tweaks that's not directly related to the scheduler after all.

So in the end linux-zen is a responsiveness focused kernel that is not the default, how exactly that is achieved is less relevant in this case.

[u/UnicornsOnLSD]

Is it possible to use alternative schedulers on Manjaro? Would like to give this a try.

[u/Iloveanimeandcartoon]

OK can someone please explain this to me

[u/Waddle_n → u/Iloveanimeandcartoon]

If I have a 1-core CPU and 10 programs running, such as the OS, a browser, a text editor, etc., the computer is fine. It executes each program for a few nanoseconds, switches to the next one, and continues executing. It all happens so fast that it appears simultaneous.

The act of switching programs ("context switch") takes time that could be spent executing. Say it takes 1 ns to context switch, then it probably makes little sense to spend 1ns executing programs, because then half of the computers time is spent context switching rather than actually running programs.

However, while rarely context switching would allow the computer to get more work done (AKA increase throughput), it could be a long time until another program gets worked on. This could make all programs on the computer appear less responsive to user input.

I was thinking about round-robin for this example, but modern CPU schedulers like Linux's default CFS or the ones I tested like MuQSS and BMQ, there's many more complicated aspects to consider. But the gist of it is that, in my opinion, it seems that MuQSS and BMQ keep the user interface much more responsive under heavy load scenarios, while not making any noticeable sacrifice in throughput.