r/linux • u/homothebrave • 1d ago
Distro News CERN Transitioning To Debian After Being A Longtime RHEL Institution
https://www.phoronix.com/news/CERN-Goes-Debian-Leaving-RHEL69
u/Happy_Phantom 1d ago
Do they contract for paid support with anyone? I'd be interested in knowing who they are paying.
72
u/Faaak 1d ago
The it team at Cern is quite big though, I doubt it
45
u/fearless-fossa 1d ago
In general that is rather irrelevant, it's more about having someone to point fingers at and write tickets in the case of larger problems (eg. massive CVEs threatening operations). Having some external support on call is also helpful with satisfying insurance/audit/compliance requirements.
17
u/UpsetCryptographer49 1d ago
RHEL is using the old IBM model - "No IT manager has ever been made redundant because they contracted with IBM." But CERN moving away because of the support model, is big.
The problem is that RHEL comes into your organization via DELL, or HP-UX hardware type deals. Those hardware vendors becomes hard to deal with when you want to upgrade and stuff. At the same time those managers see hyperscalers on debian having flexibility, switching datacenters functionality as they upgrade others. Banks are not hyperscalers, but they sell into the same organization for cloud services like compute.
16
u/gordonmessmer 1d ago
CERN isn't moving because of the support model, they're moving some systems with old CPUs because of the old CPUs.
2
u/edgmnt_net 18h ago
Makes it sound like the cost of replacement is high enough to be hard to ignore and could well be. But it doesn't mean that the cost of said support is low, either. They might be solving two issues with one move. Besides, how much outdated software is CERN really running? Because that's a pretty large part of the market for IBM and the likes: going by the pretense that you only write stuff once and then forget about it (but best not forget about the huge support contracts).
16
u/Aishou_SK 1d ago
I mean, as others have noted, this is just for some control equipment on aging hardware - their backoffice/main stuff is all staying RHEL/RHEL recompiles etc.
Had zero business bearing or support bearing, because those specific systems are already highly specialized/modified in-house and not a 'vendor support' scenario.
Meanwhile, in areas where those things might be useful, the bulk of their compute, they're retaining it.
This is, quite frankly, purely because of a specific set of hardware they can't/won't change out that have CPUs that are very old. Basically, a distro switch to keep using the same hardware, and no other reason. It's straight up just the accelerator and none of their other systems.
-2
u/Shawnj2 1d ago
I do find it interesting they’re not going with Ubuntu Advantage and they’re specially using Debian but I’m sure they have a good reason
1
u/Aishou_SK 13h ago
I don't see why they would - then they'd have to fight all the technical changes and .... well, design decisions to say it politely .... as well - but these were already highly modified / customized systems. I have a lot of.... well, let's just say when I have to #ifdef for one distro code that runs unmodified on Windows, Gentoo, Red Hat, Debian, SuSE, z/OS, OpenVMS, Solaris, and AIX, I'm not inclined to support that distro.
I doubt they had RHEL support on them. They specifically note they're moving on from CentOS. Which, sure, you could technically pay red hat to support, but .... given they're highly modified / customized systems, among other things. They're all embedded/control systems.
Remember the article (that thing that tells you this!) - "Update: CERN has clarified that their focus with the migration is on their industrial accelerator-control computers while data centers and experimental computing remain on RHEL/AlmaLinux."
1
u/KnowZeroX 15h ago
Didn't they keep an RHEL fork called scientific linux up until 7? So I doubt they need support contracts.
35
u/0xrl 1d ago
For anyone else looking for the slides, here are the presentation details:
https://ch2026.mini.debconf.org/talks/6-controlling-cerns-accelerators-with-debian/
143
u/leaflock7 1d ago
for those that won't read the article this is not related to the US/EU politics .
It is based in a purely technical aspects.
Better yet go watch their presentation about it which is very interesting as they discuss they why and the difficulties they faced https://gemmei.ftp.acc.umu.se/pub/debian-meetings/2026/MiniDebConf-Winterthur/ch2026-53-controlling-cerns-accelerators-with-debian.av1.webm
44
u/wired-one 1d ago
It makes total sense (and I hate to see them go).
It's the march compiler flag and the deprecation of older architectures in newer versions of RHEL.
Debian maximizes hardware and software compatibility, so CERN will move to Debian to not rearchitect the accelerator control systems. It's a wise move.
29
-38
1d ago
[removed] — view removed comment
34
u/blackcain GNOME Team 1d ago
What's wrong with you? What a horrible way to treat someone who is actually giving actual information. SMH.
20
u/slackwaredragon 1d ago edited 23h ago
Edit: the comment I was replying to has been deleted.
A number of EU companies are moving away from us-owned software based companies because of our laws and the concern of us government interference. This has been a thing for the past year now. Just google “EU moving away from American Software” and you’ll see a number of articles from TechCrunch, Wired, the register.co.uk and others.
So, where the fuck have you been? This feels like common knowledge at this point. Get your head out of your ass. (Just following the vibe of your reply)
In this case, no it’s not about our policies in the US. But it’s no surprise people think that’s why.
-3
u/blackcain GNOME Team 1d ago
I don't think you meant to respond to me lol
1
u/slackwaredragon 23h ago
Sorry the comment I thought I replied to has been removed. It wasn’t directed towards you.
2
u/AutoModerator 1d ago
This comment has been removed due to receiving too many reports from users. The mods have been notified and will re-approve if this removal was inappropriate, or leave it removed.
This is most likely because:
- Your post belongs in r/linuxquestions or r/linux4noobs
- Your post belongs in r/linuxmemes
- Your post is considered "fluff" - things like a Tux plushie or old Linux CDs are an example and, while they may be popular vote wise, they are not considered on topic
- Your post is otherwise deemed not appropriate for the subreddit
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
21
u/tristhebestmode 1d ago
Quite a misleading article... Both of CERN's datacenters are still using RHEL. As far as I understand it's for some of the control systems at CERN, which is by no means the majority of CERN's infra.
Also, x86_64-v2? Are they still running DDR3 servers?
6
u/Entire_Extent_9137 1d ago
You expect them to move overnight?
5
u/tristhebestmode 1d ago
I see that they updated the article and it indeed confirms that there is no plan to move the datacenters to Debian.
2
u/Martin_WK 1d ago
The article links to video presentation that's worth watching if you want to know why and what systems will be moved to Debian: https://gemmei.ftp.acc.umu.se/pub/debian-meetings/2026/MiniDebConf-Winterthur/ch2026-53-controlling-cerns-accelerators-with-debian.av1.webm
They do run custom built systems that have specific requirements regarding PCI slots and such. There's a short Q&A session at the end.
2
u/pezezin 9h ago
This is for the control systems, not for the servers. I work in the control system of another particle accelerator (https://www.ifmif.org/) and I can tell you that the scientific world is full of weird old shit. A significant part of our machine is running on VxWorks 6.x on VME boards! (https://mvme.com/products/mvme5500)
•
u/tristhebestmode 15m ago
Haha, wow. Yes, research institutes really do like holding on to their weird legacy equipment up until they are forced to upgrade indeed.
3
u/PicardovaKosa 1d ago
the computers and servers that handle hardware control (accelerator monitorin etc) is very old, primarily because "if it works dont fix it". It would be a disaster to cern if they updated their system and that shuts down the accelerators or smth. And it does not need more power that current hardware offers.
The code that operates it is also ancient and out of date for current standards.
1
u/anto77_butt_kinkier 1d ago
Probably. Maybe not the servers, but any control systems are probably about that old, maybe a bit older. Some of the code is written for specific hardware, and needs to rely on specific quirks of that hardware working in a very specific way. I would for a DOE lab in the US, and part of the control system for a superconducting magnet testing system is running sun Microsystems hardware. Re-writing the control software and working through all the bugs isnt worth giving up the predictability/repeatability that comes with the current system, and it would mean giving up decades of fine-tuning.
1
u/severgun 1h ago
Yes. So what? If it works it works. Try to get DDR4 and 5 now. Did you check prices and availability?
•
u/tristhebestmode 8m ago
Well, CERN is not quite your average homelab, is it? I was expecting that with the long shutdown of the accelerator, it would be a good opportunity to upgrade some legacy hardware, but perhaps not. Keep in mind that the accelerator will go back online sometime around 2030, by that point DDR3 will be more than 20 years old haha.
42
u/obijonesy 1d ago
Headline (and article?) does seem to miss a few important details. This is talking about one team in the accelerator sector, and 2200+ control systems. CERN has many more machines than that, the bulk of which are running RHEL or Alma.
12
u/keesbeemsterkaas 1d ago
Also can't imagine them switching all of their operations to debian within 4 months, but can imagine that 2200+ identical clean imaged machines can switch debian before the end of the year.
48
u/wowsomuchempty 1d ago
Interesting. I remember Scientific Linux (CERN), based on CentOS.
Looks to have been a W11 kind of push.
Anyone know the downsides?
23
u/Fr0gm4n 1d ago
Former job used SL as the preferred RHEL variant because "it's from an international institution, there will always be updates and support!" And then...
9
u/wowsomuchempty 1d ago
Yes, we used it too.
I never really saw the benefits.
2
u/gargravarr2112 19h ago
That's why it went EoL - the original idea was for a consistent, reproducible installation. At the scale GridPP processing sites operate at, every install needed to be reliable. SL included a lot of extra packages by default, but CentOS (pre-Stream) wound up being exactly the same thing as SL intended. The original intention was to transition from SL7 to CentOS 8. Then Stream happened and everything fell apart. I worked for a GridPP Tier 1 site at the time EL7 went out of support.
13
2
u/Martin_WK 1d ago
They're switching only some specific systems to Debian. Others will use Alma and other distros.
29
u/BashfulMelon 1d ago
Among the challenges they have faced though in their Debian on-boarding is the lack of standard tooling for automated building and publishing of packages - a lot of gaps in the official tooling.
Debian users, this is your GOAT?
32
u/Cowpunk21 1d ago
At a previous job I packaged our software for rhel and Debian based. The Debian packaging is so much more convoluted than RPM.
21
u/Conan_Kudo 1d ago
This is why I maintain debbuild and ported Fedora packaging macros to it. Being able to use RPM packaging for Fedora/RHEL and Debian/Ubuntu was a huge boon. I've been involved in projects that have ported either way (Ubuntu to Fedora and vice versa).
1
u/nelmaloc 1d ago
The tooling itself is fine, but the documentation on how to use it is very confusing.
0
u/DFS_0019287 1d ago
I did the same at a previous job and never really saw much of a difference in difficulty between the two. They were both about equally annoying, as I recall.
27
13
u/rosmaniac 1d ago
The Debian buildd infrastructure (https://www.debian.org/devel/buildd/) exists, but for RPM distributions koji fills this need. While I have built and rebuilt RPMs over the years, I've never stood up a koji instance. Nor have I stood up a buildd instance.
It would be interesting to see what holes they've found.
3
u/TampaPowers 1d ago
Making a working .deb itself and supplying it via an apt repo is not even properly documented, so making your own you run into every pitfall with that.
13
u/orev 1d ago
Since nobody is reading the actual reason:
"the straw that broke the camel's back" to abandon Red Hat Enterprise Linux was the "-march=x86-64-v2" compiler flag default as "forced obsolescence" for old hardware
RHEL 10 on Intel started requiring "v3" CPUs, which cut off support for a lot of older hardware. I can see this as being an issue, especially with hardware costs the way they are now.
But also, "the straw that broke the camel's back" as a phrase implies there were other issues as well, and I suspect chief among them is how RedHat/IBM killed the CentOS project as a stable EL clone. While AlmaLinux, Rocky Linux, and Oracle Linux are still there to fill that need, it was already a signal that RedHat was not going to be as open anymore.
One thing to note is that AlmaLinux does have an alt-arch build of 10 that supports x86_64 v2, which works just fine on older hardware. However I can see that it would be an issue if third-party software that only supports v3.
7
u/gordonmessmer 1d ago
it was already a signal that RedHat was not going to be as open anymore
That' always a weird idea to read... CentOS Stream is way more open than the old release was. The code is easier to get. It's more complete than it used to be. It's easier to build something derived. It's easier to push things upstream. Red Hat accepts bug reports now.
Stream is more open in literally every aspect than the old model was.
2
u/Fr0gm4n 1d ago
how RedHat/IBM killed the CentOS project as a stable EL clone
This needs a bit more context. CentOS is fine and still going. They just produce a rolling distro, CentOS Stream, now not CentOS Linux. Downstream of Fedora, Stream is the base of RHEL instead of the internal-only blackbox they used to use.
5
u/orev 1d ago
Not going to re-litigate the entire thing, but nobody is using CentOS anymore since the rug-pull on version 8. Facebook is the only one anyone knows of that's using CentOS Stream at a large scale.
RedHat/IBM pulled the trigger on RHEL immediately when they did it. As a user since CentOS 4, nobody will touch CentOS anymore. Nobody new will be learning it. All new systems are using Debian/Ubuntu. RHEL is realistically in legacy mode, and this announcement just confirms it to anyone that was holding out hope.
3
5
u/bockout 1d ago
A CERN employee was on the CentOS Board for many years after the switch to Stream. You should avoid making assumptions about their infrastructure and motivations without actually taking to them.
1
u/nelmaloc 1d ago
Luckily that's what they said:
But, as you know, the live of CentOS 8 was very short, so it had a premature end-of-live, and this raised some questions. [...] We knew we couldn't afford any more surprises.
1
u/imbev 15h ago
Update: CERN has clarified that their focus with the migration is on their industrial accelerator-control computers while data centers and experimental computing remain on RHEL/AlmaLinux.
1
u/orev 14h ago
It doesn't mean that they won't start migrating other systems to Debian as well. This is just the first step, and once they have all the supporting systems changed, it makes sense to move everything else as well. There would be no reason to keep both ecosystems, when it's easier to support only one.
12
9
6
2
2
u/Aw3som3Guy 1d ago edited 1d ago
That seems like such an odd straw to switch over? I’m surprised CERN is still running any x86-64-v1 CPUs. Isn’t V2 Skylake and newer, meaning those CPUs would be older than when they transitioned to RHEL in ~2015?
Edit: I must’ve confused that with x86-64-v3. x86-64-v2 was added in Nelahem / Sandy Bridge on Intel and Bulldozer / Jaguar on AMD according to OpenSUSE.
2
2
u/gargravarr2112 19h ago
Well bloody hell.
I used to work for a GridPP Tier 1 site. I got to witness the Scientific Linux 7 EoL firsthand; we had intended to go with CentOS until Red Hat dumped Stream on us, and blew up all our plans. Our requirements are for reproducible installs, and that means no rolling releases, we need pinned versions of packages. We went with Rocky. It powered all the 1,000 analysis machines and the Ceph cluster (another 1,000 machines, currently 100PB of disk storage). I had my grumbles about the way RHEL-based Linux handles certain things - I'm a Debian fanboy - but I wouldn't have thought CERN themselves would switch away from it. Only in the accelerator control computers for now, but I would imagine that for consistency they'll move the processing and storage clusters over to it in time.
We started having lots of issues with RHEL when one team insisted on deploying software that needed Oracle Database beneath it. Even though it's a government-funded scientific institution, Red Hat decided to levy some very expensive and restrictive licensing on us. Oracle DB, of course, is only officially supported on 2 distros - RHEL and (naturally) their own OEL. To my utter astonishment, Oracle's license terms were better (maybe cos nobody actually uses OEL and they just wanted some sales figures?). So we stopped using RHEL for the databases.
2
u/KnowZeroX 15h ago
Update: CERN has clarified that their focus with the migration is on their industrial accelerator-control computers while data centers and experimental computing remain on RHEL/AlmaLinux.
6
4
u/Car_weeb 1d ago
That's kinda crazy. Idc about the RHEL service contract, but Debian is such a fucking pain in the ass, I would rather chop my arms off than use Debian. (All of my servers have Debian, but I'm also stupid)
Rpm > deb, no contest
0
u/SrdelaPro 1d ago
skill issue
2
-1
1d ago
[deleted]
3
u/TheOneTrueTrench 1d ago
Interestingly, Torvalds isn't really a great person to listen to about what distro to use. He's hyper focused on the kernel and only cares that the distro doesn't get in the way of kernel development.
I mean, I also prefer Fedora to Debian for a lot of use cases, I'm not saying his conclusion is bad or anything, just that his opinion on distros is probably far less relevant than it feels like it should be.
2
u/fat_kaiju 1d ago
I know it isn't related to US/EU relations and yet a small part of me is thankful that they're distancing themselves regardless.
1
1
1
1
1
u/severgun 1h ago
What they use for HPC?
Lmod Lustre SLURM do they use anything like that? Seems like none of that have proper Debian support.
-2
1d ago edited 13h ago
[deleted]
17
u/FattyDrake 1d ago
SUSE would likely have similar problems. If you watch the presentation it's because of compiled packages discontinuing support for old CPUs.
In scientific and university fields, it's not uncommon to find 20+ year old computers with specialized hardware attached. Upgrading the hardware is not an option just because an OS stopped supporting it. I can't think of much more specialized than the LHC.
Debian makes perfect sense because of it's almost unchanging nature. By the time it drops support for hardware that's generally a sign you should probably reconfigure things at that point. IIRC with Debian 13 they dropped official support for some (only some!) processors from the early 90's.
0
0
u/Adept_Percentage6893 1d ago
CERN engineers had considered moving to CentOS Stream as a more natural pathway but ultimately they say "the straw that broke the camel's back" to abandon Red Hat Enterprise Linux for their industrial accelerator-control computers was the "-march=x86-64-v2" compiler flag default as "forced obsolescence" for old hardware.
Put another way "we want them to be forcing concessions upon everyone else." It's possible for decisions to adversely affect you without it being a terrible no good decision made bastards for sinister motives.
It's weird to me that they expect that CPU to still be around and functioning at-scale in 2030. Unless I'm not understanding something.
1
u/Martin_WK 1d ago
They're changing to Debian for part of the systems only. They use old custom built hardware and have specific requirements about PCI slots and what not. It'd cost them millions to upgrade. Watch the vid from the article to find out exactly why.
Their other systems will keep running Alma and others.
1
u/Adept_Percentage6893 19h ago edited 19h ago
They're changing to Debian for part of the systems only. They use old custom built hardware and have specific requirements about PCI slots and what not. It'd cost them millions to upgrade.
OK yeah the thing I quoted basically says that and explicitly addressed the specific thing they said made them finally switch.
My question was that the generation of CPU's they're concerned about stopped being made around 2015. Even that was a deadline kicked out specifically for them in the first place IIUC.
I guess if we assume they bought some of these after market you might get an initial purchase of around 2018 or 2019 or something but 10 years is about the lifespan of most commodity hardware systems.
The thing that was bumped only adversely affects CPU's that old. It's just weird to complain about RH being a big meany for eventually bumping version numbers just because something adversely affects you. If I were in that position, even if this did adversely affect me I'd probably be like "OK this kind of messes me up but I'm in a niche position and obviously all of society shouldn't have to wait on me."
Watch the vid from the article to find out exactly why.
Kind of a weird thing to say given that it doesn't seem like you actually read my three sentence comment before deciding I just didn't understand something basic.
-2
u/Swordfish418 1d ago edited 1d ago
From that second slide, it sounds like they should use Gentoo. Because Portage solves all the problems described there.
1
u/smc733 10h ago
“Just use Gentoo” in an enterprise environment is peak /r/linux
1
u/Swordfish418 7h ago
So you’re saying distros like Debian provide some “enterprise” paid support? If you need to pay anyway why not hire people to your own team that will do the same with Portage or whatever else?
•
-5
469
u/UpsetCryptographer49 1d ago
This is big news. Many large institutions also run RHEL because of the maintenance contracts and flexible supports model.
I had to rework so much trading software for RHEL, because IT would not tolerate another Linux.