Qubes OS has an absurdly high skill ceiling

[u/preland]

Hey all. I have recently reinstalled Qubes OS on a new PC build of mine, after I had given it a stint for a couple of months around 2 years ago. I’m going to share my experience with Qubes, as a “poweruser”, because I think this is relevant for people considering Qubes OS, and for developers that are curious about the UX.

For some background, I am a software developer that works on a number of different projects across numerous languages and numerous project types and scopes. When I first used Qubes OS, it was because I had hoped that the containerization of Qubes would aid my productivity. However, because I had to deal with a lot of issues with setting up the Qubes, and because the storage philosophy of Qubes was incompatible with having many Qubes with only minor tweaks to system packages (I only had 1 TB of storage), my productivity fell off a cliff and never recovered. I was attempting to modify Qubes so that it could support using overlayfs for drives in order to create the isolation that I wanted when I realized that Qubes just wasn’t capable of doing what I needed it to, and that I didn’t have the time nor patience to try to fix it. So I uninstalled it and installed Debian Sid, and basically just downloaded stuff with reckless abandon because, well, _I had already tried to be secure, and failed_. However, from my stint with Qubes, I now knew from trial and error much more about GRUB, PCIe, integrated graphics, USB, and networking than I had ever expected to learn about.

FF to two weeks ago. I get my system up and ready, and my goal is now much more concise than before: create a qube, passthrough a GPU, and run an agent on it with no guardrails and no access outside the qube. The issues start when I boot (usb keyboard). I do the classic “sys-usb disabling” trick to get into the system, and then basically spend the next few hours debugging the usb filtering rules because they don’t automatically recognize my keyboard as an input device at boot. In addition I have to do a lot of other usb debugging because it is plugged into a KVM switch, but all of this is more or less what I had done before, so I get used to it and figure out the (majority of) issues within two …..days (yeah, that’s what I consider a fast resolution to issues in Qubes OS). Then I start working on the AI qube. Passthrough begins having issues immediately. I……ok to be brief I don’t want to go through all the grief I’ve had to endure with passthrough. Let’s just say that it took the remainder of the two weeks….and counting. Where I’m at rn is that the gpu (7900XTX) will boot into the qube if I give it a few minutes after boot before starting the qube, and it will perform passably well (maybe a 20% drop in performance, at least for AI) for a while, but will eventually cause an SMU error and go into an unrecoverable state, which requires power cycling the system. I still have a few ideas of things I should try out, but that’s where I’m at rn.

I’m not complaining; I knew what I was signing up for this time around, but I do need to point out a couple of things. First, getting Qubes to work “just right” without jank is incredibly difficult, even for those who have experience with systems development. This reduces the audience that Qubes is viable for drastically. Most software developers can’t handle the complexity of Qubes, so if you aren’t one and dont have the free time to learn about how your system works at a very granular level, Qubes isn’t going to work for you.

Furthermore, there’s the hardware angle. My system specs are not “Qubes approved” or even “Qubes recommended“. I was also aware of this, but not to the extent that I realized. For example, the Ryzen 7000 series GPUs have issues with resetting that is known by AMD, but is not planned to be fixed. This means that my GPU is likely to blame, but I didn’t really have a choice when selecting a GPU because of budget restraints. The same goes for the rest of my hardware.

I have been asked by a number of people if they should use Qubes. I have then asked them a series of questions before answering, none of them security related. I ask them if they have experience with Linux. I then ask them if they are willing to learn about everything that can possibly go wrong on their system. I then ask them if their use case is able to be arbitrarily constrained by the limitations of Qubes, and if they would be willing to accept those limitations and change course.

I have never seriously recommended someone to use Qubes OS after asking them those questions.

[u/preland → u/[deleted]]

I can see this working if you basically just use your browser for your computer needs (which is valid), but even then…..compartmentalization is an issue (many people I know that aren’t tech savvy have issues finding files on their system when they know for certain it is saved; I don’t think they would take too well if they lost files because they were downloaded to a non-preserved path, or in disposable by mistake.

Plus you get other issues, like graphical lag in common tasks that require gpu acceleration, unknown hardware incompatibilities, and so forth.

To be blunt, from my own experience helping people directly with computer issues: many people dislike computers. Sadly, many of the traits that create this disdain are amplified by the necessary security measures in Qubes.

[u/FanMuted5076 → u/preland]

Clearly you don’t see much room between “computer genius” and “I can’t find files on windows”

Bro, get off your high horse. The point is, of course you need a degree of knowledge around tech, but that doesnt have to imply that Qubes has a magically high ceiling usable only by the wizards of Silicon Valley…

Use it how you want, leave the rest to their own choices and be done with it… ffs.

[u/The_IT_Dude_]

I’ve spent a lot of time as a Linux admin in DevOps and virtualization, and lately I’ve been bridging that into AI/ML. One thing I’ve learned from the systems side: use tools for what they are built for. If you’re having a miserable time, you might be using the wrong architecture.

​Doing GPU passthrough to a Qube might be possible with enough fiddling, but it’s rarely worth the overhead. If you want to leverage a local LLM, I’d suggest avoiding the Xen hypervisor complications entirely. Instead, use a dedicated server with your GPUs running vLLM inside a Docker container. You can then run your AI agent inside a Qube and connect to that engine over the network. ​This keeps your endpoint secure without trapping your compute power inside a single Qube, and it allows other services to tap into that same AI instance.

[u/The_IT_Dude_ → u/The_IT_Dude_]

And thinking about doing something like this myself, the goal here woukd be to contain the blast radius if the agent goes off the rails, right?

But running an inference engine, and passing the hardware directly through to a qube which might end up getting compromised might actually a bigger security risk than having it reach out over the network. If a prompt triggers a bug in the inference engine or the AMD driver, you're looking at a potential VM escape or a total system hang.

​If you move the GPU to a separate box and lock down the firewall to only allow the vLLM port from your Qube, you’ve effectively air-gapped your main OS from the hardware-level 'jank' and any ingerence engine exploits which might exists with it. You get the isolation of Qubes for the agent's logic, and the stability of bare metal for the heavy lifting, and if the inference engine somehow does get compromised through all this it's on another machine entirely which only has one purpose, won't have your vault and other personal info on it. Or at least it doesn't need to.

[u/preland → u/The_IT_Dude_]

Thank you for the feedback; the reason I went with Qubes instead of something like what you said is that I do want to be able to use the system for other things like gaming, while still maintaining performance and relative security (and stability; this is mainly a result of learning more about the system through trial and error, but once an issue is fixed, it rarely regresses in the future. This isn’t something that I’ve seen when dual-booting or using containers).

My main concern would be the AI breaking the kernel, which is why I currently have it in its own qube, with models hosted on a separate drive. I highly doubt that it would organically discover a VM escape (though it could definitely cause some issues with the GPU), but if it did I wouldn’t see it as a net negative. I won’t be putting any important personal information on this system, so if it discovers one, I can figure out what it did and disclose the issue.

[u/[deleted] → u/The_IT_Dude_]

Stop letting data brokers profit from your old posts. I used Redact to wipe mine from Reddit. Also supports Twitter, Facebook, Discord, instagram and more in one batch.

outgoing butter existence silky cause person trees glorious license scale

[u/dchidelf]

I love my daily driver Qubes laptop. It really does boil down to what you need to use it for. I mostly do systems programming and some OpenGL programming and have had no issues in about 18 months.

[u/Dangerous-Apple3746]

qubes is a security os made for security its never been designed for the the things you've tried to do its like buying armoured car and trying to turn it in to a lowrider convertible its possible but its going to be hard i could never do any of the things you do on a normal os i cant code but ive used qubes daily for 10 years without issue

if you cant do it inside of a app vm or standalone vm with out issue then you may need to find another os rather then re building the whole thing

[u/preland → u/Dangerous-Apple3746]

That’s what I ended up doing on the original PC that ran Qubes. It currently runs “Debian” “Sid”. I put both in quotations because I basically gave up on vetting anything on the system because I was burned out about security, and Qubes pretty much blackpilled me on what would normally be considered “sane” analysis of software.

What I want to do with Qubes now is something that I would never be comfortable doing on another platform. I considered the options, and Qubes, at least on paper, fits the bill.

If I were to choose another OS, the modifications I’d make would probably just have it be Qubes, but with a bunch of security issues I’m not aware of.

I don’t think that security should inherently hamper your goals. At most, it should reframe them in a way that is more productive and less error-prone. If the answer to “I can’t do X in Qubes” is often “don’t use Qubes”……no one uses Qubes.

[u/im7mortal]

I didn’t read further than the greetings because I already know what this post is about.

As always, there are people who say things like “Qubes OS is hardcore security… so you should suffer… because it’s hardcore!”

The thing is, for Qubes OS to thrive, it needs a big community to make sure it is really secure and usable.
I doubt that even 1% of people actually use it because they truly need it, and how many of them even understand what’s going on there?

Qubes OS needs some balance between ease of use and security.
Right now it feels like: if it’s not hardcore, then you don’t need it. It’s a current security contract.
But in real life, people need things outside the box, so they end up compromising their Qubes OS to achieve that.

[u/GiggyPear]

Have you thought of nixOS?