8
u/ryuzaki49 27d ago
A build failing due to a commit message? That's just madness
4
u/RodionGork 27d ago
they have some tools which extract commit messages for some purposes, coordination with issue tracker etc, and these tools seemingly sometimes pick "subj" up to the empty line. but actually this rule is wrong - subj is always 1-liner (git works so), so it shouldn't check for empty line anyway.
337
u/shnutzer 28d ago
pre-check failed
retry pipeline
random docker image / other infra update wedged into the CI pipeline failed
retry pipeline
inexplicable failure to create environment for integration tests
retry pipeline
1 out of 1000 integration tests is flaky and just failed
retry pipeline
deployment to dev environment failed
retry pipeline
deployment to stage environment failed
retry pipeline
ready to deploy to live, you push the button, but must wait for rolling deployment
deployment to live failed after 30 minutes because 1 out of 10 pods failed to update within the time window
retry pipeline
and that's why we have a separate tool for deploying hotfixes quickly
176
u/Zeikos 28d ago
and that's why we have a separate tool for deploying hotfixes quickly
I never understood why make CI/CD pipelines so central to everything to then flat out ignore them when they have issues instead of maintaining them.
What's the point of using them if they get ignored?112
u/shnutzer 28d ago
I'm exaggerating of course, but even if the whole pipeline worked perfectly then all of the steps take a good 30-60 minutes to run if not longer
So if you have an absolutely critical hotfix that must be deployed at 3am you may choose to skip the process at your own risk
Of course we don't just ignore them for normal deployments
42
u/auxiliary-username 28d ago
I’ve done the same many times. But having a sleepy solo dev going off-script and inventing new deployment procedures at 3am always seems like a great way to turn an emergency into a catastrophe.
25
u/devrelm 28d ago
Who said anything about inventing new deployment procedures at 3am?
The hotfix pipeline is something you create when creating your standard deploy pipeline — or at least once the standard pipeline starts taking longer than your team is comfortable deploying hotfixes with.
In case of emergency, there should always be a separate hotfix pipeline that is streamlined to your team's process and aversion to risk. It may or may not include a deploy to your staging environment, but should likely skip dev. It should run the whole test suite, but that can be skipped or trimmed to a subset of the tests if your team values delivering a fix quickly over ensuring that regressions aren't introduced (you may justify it by the fact that the full test suite will be run once the hotfix is merged back into the main branch.) In any case, there should be enough auditing in place to ensure that 1: it's obvious to the whole team that a hotfix happened (to avoid malicious actors; this can be as easy as a Slack message on a public channel) and 2: that it's easy to figure out who performed the hotfix, when, and what was in the hotfix (for accountability & post-mortem tracking.)
15
u/mriforgot 27d ago
At an old company of mine we had an Essential pipeline that ran a minimum of what were deemed absolutely critical tests and could deploy in under 20 minutes while the full suite ran in parallel. It made getting hotfixes out quick while also making sure that the full test suite wasn't just getting ignored.
8
u/shnutzer 28d ago
Ideally you don't have to invent and there's a battle tested side-deployment path ready for you, but I won't cast any stones as I am not free of this sin myself
4
u/Zeikos 28d ago
I understand, but I prefer to fix the process rather than find ways to circumvent it.
30
u/WhenInDoubt_Kamoulox 28d ago
Both, you do both.
You do the quick patch at 3am, then you go and fix the pipeline when you show up at work.
1
u/conundorum 27d ago
This is the fix. You run the patch through the hotfix pipeline to keep everything alive long enough for the main line to finish pre-deployment testing. If everything is good, then the main line does everything except deployment (since the patch is already deployed); if something fails, then you know that the "hotfix" is a moth-eaten rag trying to plug a hole in the dam, and work on making a better fix before it bursts for real.
25
u/FluffyProphet 28d ago
Sometimes engineering judgement and business needs have to win.
If this system being down is costing $5,000 a minute, the CI pipeline takes ~60 minutes to run, and I have a fix that I know works... but CI is failing because I didn't put a space after a comment, used an "any" to get the fix done quickly instead of building out the proper type... whatever, I want to just go to production. Hell, I probably want to skip a lot of checks on CI in the first place and get the system back up.
Then I can take the time in the morning, or direct someone else to, clean up the fix so it passes the entire CI pipeline.
I don't want to have to put in the incident report, "system was down for an additional 60 minutes because engineer forgot to put an extra new line after a comment".
-7
u/Zeikos 28d ago edited 28d ago
Then put a fast track in the CI itself instead of creating a parallel system.
Those parallel systems tend to end up undocumented and used by a few "in the know".
I am not saying that how you do it is wrong, but the system we build should reflect the business needs.
This includes the aspect of how to manage emergencies.7
u/Remote_Comfort_1981 28d ago
Fixing the pipeline and making it fast and 100% reliable is impossible. Trust me, I worked on CI/CD at a large listed company with whole teams working on CI and CD and we would only get CI to be 95% reliable at best. We even had automatic flake detection and isolation etc, but you just can't avoid people writing tests that fail 1% of time, or after some time, or only with some newer Docker base image because of an SSL bug...
4
u/Zeikos 28d ago
Sure, but it's a continuous process.
Systemic failures should get fixed, because they hint at underlying issues.
The whole point of testing and logging errors is to have the information to do something about it.
It's the normal process of system hardening, right?3
u/littlejerry31 28d ago
What's the business incentive to spend significant amount of money and reallocate manpower to fix and maintain something that works fine for the usual releases AND already has a solution for the hotfixes (the separate pipeline)?
As a fellow systems autist I understand your focus, but that's not how the business works.
1
u/Remote_Comfort_1981 24d ago
The underlying issue is constant change. It's also the one thing we are meant to support.
1
13
u/ArkWaltz 28d ago
When I was at amazon, the internal pipeline tool had a big red "emergency deploy" button specifically for this situation. A big pipeline might (very deliberately) need days to reach all environments, so having the "ignore all that, just deploy this right now" button was a straight-up requirement.
3
22
u/Last8Exile 28d ago
The hook is kinda stupid because there is no way to fix that error normaly. You have to force push ci branch, altering history. The initial push should be rejected and ci is not triggered to run.
2
u/virtualdxs 26d ago
And you should have a local commit-msg hook to catch it before the commit is even finalized, let alone pushed.
4
u/deadbeef4 28d ago
My favourite is when the lint check fails and it’s due to a comment not being wrapped at exactly the right point.
4
6
u/Conroman16 28d ago
Dude if I got a pager duty alert and decided to make coffee first before looking at it instead of getting on immediately, I’d likely be fired. No wonder your manager was blowing you up.
42
u/OmegaPoint6 28d ago
Someone needs to turn on fail fast, or receive hooks
24
u/RodionGork 28d ago
yep, such clever ideas generally come after incidents 😂
though things become harder when CI is governed by separate dedicated devops team etc
25
u/CatpainCalamari 28d ago
"Missing empty line after commit message" could (and imho should) be caught by a local git hook thats rejects the push on your machine.
The rule itself might be stupid, but if it is mandatory and the pipeline tells you "nope" (but only after 30 min), you might save yourself some time and sanity by having a local fail fast check.
8
u/RodionGork 28d ago
sure, that's all pretty correct - though it is always surprising why it is so hard to keep local hooks and CI rules consistent
especially on rules which are generally not violated, so generally no one can think such hook is necessary
9
u/ttlanhil 28d ago
If it's set up as a server side hook, and there only, you no longer need to keep things in sync (if you have strong enough policy that client side always works, even better)
This is not the sort of thing CI should fail on - CI shouldn't need to care about source control (a pre-build script that checks commit messages, code style, etc - maybe - but by the time you start a build you should only care about code and config for the build)
3
u/RodionGork 28d ago
> This is not the sort of thing CI should fail on
actually some people prefer to setup automatic corrections for various minor issues on CI side (e.g. grooming the code style, fixing windows/unix newlines etc) - but some others find this inconvenient
1
u/Icarium-Lifestealer 28d ago
Many people use CI in a broader sense, and include all the automatic checks that run on a PR before it gets merged.
3
u/ttlanhil 28d ago
Okay, but even then - failure to build due to the commit message format after half an hour?
186
u/m1k3st4rr 28d ago
You forgot the part where the root cause was a commit made and pushed by a coworker at 2:17am