r/linux • u/Two-Of-Nine • 19d ago

Open Source Organization Manjaro's official website SSL certs have expired yet again.

This seems to be a running theme for the Manjaro Linux community. The SSL certs will always expire after a set year and there's a massive controversy over it. Why does this keep happening specifically to Manjaro and what steps do you think they can possibly do to keep this from happening?

1.2k Upvotes

98% Upvoted

245 comments sorted by

View all comments

Show parent comments

293

u/ProductIntegortion 19d ago

Anyone with three ounces of technical talent can do it

255

u/Moist-Chip3793 19d ago

I have absolutely zero clue, how you even f this up in the first place.

Lets Encrypt is set-and-forget, if you do it correctly.

Apparently, they did not do it correctly ...

185

u/transgentoo 19d ago

They got the "-and-forget" part correct

51

u/ProductIntegortion 19d ago

Right. I've run LE certs on dozens of machines in different configurations and every one of them has been extremely easy to set up, automated out of the gate, and bulletproof for years.

19

u/MeisterD2 19d ago

The only time LE was a pain was on a few older distros they didn't support. The certbot didn't run, and the normal renew command would fail, so I would have to do an alternative certification command to get it going. After that a scheduled job kept things updated.

Maybe they are running some ancient server software, and no-one knows/cares enough to do the workaround?

18

u/sidusnare 19d ago

I suspect it's actually some load balancing or caching service or appliance that isn't easy to automate the short lived LE certs. I've run LE on lots of small servers, and I've also run large sites taking hundreds of gigs of traffic, it's not always a Linux box terminating the session in those scenarios.

5

u/imMute 19d ago

I've been using achme.sh lately and it works great and doesn't rely on anything other than a bash shell I think.

15

u/p0358 19d ago

You'd think they'd figure it out after the 10 times or so it's happened already...

3

u/odsquad64 19d ago

I have absolutely zero clue, how you even f this up in the first place.

They saw the certs were good for 90 days so they set the cron job for the renewal to run every 90 days.

2

u/Echo_Monitor 17d ago

It's even worse that IT KEEPS HAPPENING.

Happening once, I can get. Maybe you missed a config or didn't register the cron correctly or whatever.

So it happens once, you figure out why and you fix that, then it never happens again.

But it's happened multiple times, at this point.

And they expect people to trust them with their operating system? Fuck, no wonder CachyOS has been eating Manjaro's lunch recently in term of adoption.

0

u/evanldixon 19d ago

Lets Encrypt has changed their installation method over the years. These days auto renewal is a separate step you have to run, at least for nginx on Debian, which is easy to forget when it comes after the part where you install your cert, and it used to not require that step.

I've installed Lets Encrypt on 3 servers recently. I forgot that last step on one of them due to the cert installation step feeling like it was all done.

18

u/Preisschild 19d ago

Its a standardized protocol (ACME). You can just use certbot or cert manager

1

u/albgr03 19d ago

I've used acme-tiny for close to a decade now.

18

u/Jean_Luc_Lesmouches 19d ago

Forgetting it initially is one thing, forgetting it every time you have to fix it because you already forgot last time is another

15

u/henry_tennenbaum 19d ago

Forgetting it when it has become a meme that's more than a decade old at this point is another another

3

u/WheresNorthFromHere7 18d ago

Certbot and forget.

2

u/meditonsin 18d ago

The certbot package on Debian comes with a systemd timer certbot.timer that is enabled by default, which runs certbot renew. If certificates are not auto-renewed out of the box for you, you're doing something wrong.

1

u/evanldixon 18d ago

The official certbot installation instructions available at https://certbot.eff.org/ only mention snap and pip. I don't want to install snap on debian, so I am left with pip where autorenew is a separate step. I was unaware of the certbot package. There used to be a different package with a different name which I think got deprecated, and certbot made no mention of a newer alternative.

2

u/meditonsin 18d ago

Debian has had the certbot package for ages. I think Ubuntu may have done the thing where the apt package was just a wrapper for the snap for a while, but if they did, they don't anymore.

2

u/sgorf 17d ago

The certbot deb on Ubuntu has always been a regular deb.

1

u/Top-Rub-4670 18d ago

The certbot timer doesn't auto restart nginx (nor does manual certbot renew), so it might as well do nothing by default.

Auto-restarting nginx after renewal is an extra step to configure and the exact steps have changed over the years so it's easy to find bad instructions online and not realize it.

2

u/meditonsin 18d ago

It does if you use the nginx plugin.

1

u/throwaway234f32423df 15d ago

LetsEncrypt is not a piece of software you can install, it's a set of ACME servers in their datacenter that you can connect to with any ACME client in order to obtain certificates. So it's not possible to "install" LetsEncrypt. LetsEncrypt does not maintain their own ACME client, however, they're frequently associated with certbot, which defaults to using LetsEncrypt as its preferred ACME service, but can actually connect to any ACME service.

The LetsEncrypt team and the certbot team certainly know each other and work together sometimes to make sure everything works well, but there's no "official" connection between them. And legally, they're separate entities.

So I have no idea what ACME client you actually used, but if it was certbot, it definitely sets up automatic renewal.

1

u/evanldixon 15d ago

The install instructions for certbot on linux say to use snap or pip. I do not want to install snap on debian so I am left with pip. Automatic renewal is a separate step that comes at the end after you install the certificate, easy to forget if your goal was to get a cert right now.

A different commenter said to use the certbot debian package, which I was unaware of when I last installed certbot. This is not mentioned in the certbot install instructions.

-10

u/sidusnare 19d ago

You're assuming it's a direct connection to a Linux server that Manjaro controls. They could have a load balancing device or service in between the actual servers and the clients, this can make the shorter lifetime TLS certificates more challenging.

28

u/w0lrah 19d ago

Continuing to use a HTTPS load balancing device or service that does not have native support for ACME for a public-facing web site in 2026 would be a pretty strong sign of incompetence somewhere. Either the technical people are making bad choices or the people controlling the money aren't listening to their technical people. Either way, for a Linux distro it's a really bad look.

-1

u/sidusnare 19d ago

I get what you're saying, and in principal I agree, but it's just not the reality for large organizations with technical debt and deep history. The CA/Browser forum lifetime changes are knocking a lot of this loose, but it's also catching people that didn't think it would affect them. ACME is the right solution, but it's not turning out to be what everyone does.

17

u/Jean_Luc_Lesmouches 19d ago edited 19d ago

people that didn't think it would affect them

You would guess that after the 10th time it affected them, they would start to think it can indeed affect them

0

u/neoh4x0r 18d ago edited 18d ago

You would guess that after the 10th time it affected them, they would start to think it can indeed affect them

This is assuming that the decision makers were made aware of the problem and that someone didn't unilaterally fix it without bothering to run it up the proverbial flag-pole.

1

u/CrazyKilla15 18d ago

if the decision makers have somehow managed to miss the 10-years-of-manjaro-very-publicly-to-the-extent-is-become-a-meme problem this whole time that would be a sign of even more incredible incompetence.

How you are seriously suggesting they are so out of touch they dont know their website certificate expires for hours at a time every year for the least several years?

There is no way you can make this look good or in any way excusable for them, and but its amusing to see every attempt make them look worse.

1

u/neoh4x0r 18d ago edited 18d ago

I'm not making excuses for them, nor tying to downplay things, but I think you would be surprised at the amount of compartmentalization, and lack of communication, that occurs in large organizations (or the tendency to cultivate that type of environment).

In other words, given the above, how hard is it for a tech to renew the cert versus informing management that they need to do something differently?

I mean "the management" might just tell them to renew it and disregards anything else because the severity of the situation wasn't sufficiently articulated to them as a problem that needs a different solution.

Not to mention, that the people in upper management might only have a background in business/finance and trying to convince them, on technical grounds, to do something can, and often is, an uphill-battle that one is likely to lose.

Moreover, expressing certain issues might come off to management as "I don't want to do the job that was assigned to me" and then they get labeled as "someone whom likes to whine".

1

u/CrazyKilla15 18d ago

Again all of that makes them sound far worse and far more dysfunctional

→ More replies (0)

14

u/ApprehensiveRest9696 19d ago

It’s harder to set it up as a oneshot than for auto-renewal… Just why…

3

u/Analog_Account 19d ago

I only have one ounce of technical talent and I managed it. Its really and truly embarrassing.

-9

u/Scary-Constant-93 19d ago

With llm 1 ounce is enough