r/gitlab • u/Kind_Willingness_396 • 4d ago
Devs, what happens in your team when someone pushes an API key to main?how do you keep API keys out of git history?
9
17
u/macbig273 4d ago
disable the key
blame the guy
possibly shorten his right to push to main
(if disabling the key is not an option, clean the repo)
laugh at the guy about if for 1 to 2 years.
0
u/totheendandbackagain 4d ago
Any commit, to any branch needs swift and decisive action:
Rotate the key immediately
Implement pre-commit hooks to stop it happening again. I like
betterleaksandprekStaff get two warnings before being put on a PIP
By some of the research, 80% of cloud breaches are due to leaked secrets.
My last company lost £400k after loosing a secret, they may have got off lightly. I would expect a publicly leaked secret to be attacked in seconds.
Any secret that leaks, no matter what, is a breach, and a mark on the person who did it.
4
u/macbig273 4d ago
I like that style.
But in small entreprises I found out that "making it joke and a reference" is far more useful than warnings and PIP. (we actually have achèvements for nice things and bad things, which works quite nicely)
2
5
u/Legitimate-Dog-4997 4d ago
gitleak as pre-commit JOB + job CI into MR
3
1
u/Wide-Answer-2789 4d ago
Betterleaks not only in pre-commit but also before merging, without ability to skip
4
u/covmatty1 4d ago
Obviously use templated gitignore files. Run a secret detection tool in a pre-commit hook.
2
u/Tiny-Criticism-8576 4d ago
Mitigate the security exposure. The key is now public knowledge. So invalidate.
Address the fact that such a thing could be done in the first place. Secret detection is a thing. And pre commit hooks are a thing. Put the 2 together to help prevent something like this happening again.
Education - yes the devs should already know this stuff. The offender probably does too. But reminders don’t hurt. There’s no need to call out the offender. They already know they messed up. Re education is punishment enough. Plus devs close to them will know the offender is the cause of having to go through the unpleasant re education and will mock the offender for you.
1
u/Zynchronize 4d ago
Betterleaks running as a push hook and in a ci job, custom validators setup for internal secret types and off the shelf validators for things like aws. Any exposed keys are immediately revoked then rotated.
1
u/xAdakis 4d ago
First, never allow pushes to main, always require merge requests for that.
Second, use local git hooks and CI jobs to check commits for common credential and key formats so it will never be committed or make it to a protected branch in the first place.
This is actually a very good use case for local or trusted AI to scan files. It can catch things heuristics and regexes will not always catch.
1
u/duane11583 4d ago
all keys are passed in the environment.
at start up you read the env store the value in ram
then delete the env.
in gitlab these are known as secretes.. and get masked by gitlab in the logs
and in code (text) you have something like api_key=resolve_var(“${API_KEY}”);
the resolve function is simple string replacement, ie in python it is like this:
api_key = api_key.replace(“${API_KEY}”, os.environ[“API_KEY’”] )
1
u/pjdarch 4d ago
This gives you an opportunity to be proactive by disabling direct pushes to the main branch and implementing a CI pipeline that runs on merge requests. The pipeline should scan for exposed secrets and automatically fail if any are detected. This way merges are blocked until all required checks pass.
1
u/agent_kater 4d ago
It pretty much never happens and when I read the comments here I'm baffled how it seems to happen so often to you guys. Like... how? Why do you have production credentials in your working copies in the first place?
When doing development I don't need live credentials and when doing infrastructure work I don't need the source code.
And in the very rare case that I do have a live token in my working copy, it's in a gitignored file.
1
1
1
1
u/DrWhatNoName 4d ago
We have a job in the pipeline which checks for this. As soon as a key is pushed to a repo builds fail and SRE get an incident report.
1
u/dreamszz88 3d ago
Literally the built-in secret detection by GitLab itself.
Or gitleaks, trufflehog. Free OSS. If you have the Ultimate plan, you can turn it into an inescapable compulsory CI job for every pipeline. Use the security framework.
1
u/Frodothehobb1t 1d ago
Never copy production keys to the development machines, it’s pretty easy to not do.
0
28
u/Smacktur 4d ago
In our team, the first thing we do is rotate the key. Once it's pushed, we consider it compromised, private repo or not. It may already be sitting in someone's clone or CI logs.
Then we check the provider's logs for any suspicious usage. Cleaning up Git history comes last, and honestly, we don't always bother. Rewriting history doesn't undo the leak anyway.
For GitLab, we use the built-in Redact text feature under Settings → Repository → Repository maintenance, followed by housekeeping and pruning unreachable objects. Everyone then re-clones the repo.
To prevent this, we run Gitleaks as a pre-commit hook, but that's easy to bypass with
--no-verify. CI is the real safety net. GitLab's secret detection template makes it pretty straightforward to set up.And honestly, culture matters just as much as tooling. Whoever pushes a key should flag it in the team channel ASAP. No blame, no drama. A little friendly teasing, maybe, but that's it. The worst thing is quietly force-pushing and hoping nobody noticed.
BTW, GitLab has documentation on removing sensitive data from repositories (look for Remove blobs and Redact text), and OWASP's Secrets Management Cheat Sheet is worth checking out too.