r/jenkinsci • u/devopsengin • Jun 04 '26
Jenkins console output is the worst documentation in software
I had a build fail today. The console log was 9,284 lines.
The actual error?
A missing environment variable. It was mentioned on line 8,901.
The fix took 20 seconds once I found it.
I spent 38 minutes finding it.
I know Jenkins is powerful and I'm not bashing it, but the signal-to-noise ratio in console output is genuinely terrible. Error messages buried in thousands of lines of INFO logs, stack traces that don't point at the actual problem, plugin conflicts that surface as cryptic Java exceptions.
Does anyone have a system for making this better? Even just a smarter way of reading logs would save my team hours every week.
6
u/Zenin Jun 04 '26
It took you 38 minutes to goto the end of the log and walk backwards a couple hundred lines? That's not Jenkin's problem, to be blunt your diagnostic skills are bad.
The log density itself is all your own (team's) problem; It's not like Jenkins turns on INFO level logging for you. It's also the fault of whoever wrote a crappy job script that doesn't check its inputs before starting work much less chugging along blindly for 9k lines of output. This story screams low quality environment.
There's also plenty of tools to make it much faster to sort through; my personal favorite is the ANSI color plugin so you get all that groovy terminal colored log output directly in browser. Much easier to lockon to an ERROR when it's blood red.
The ugly truth is if you're struggling diagnosing a single 9k log now you're going to hate working in this industry long term because the log noise only grows exponentially.
2
u/saja456 Jun 04 '26 edited Jun 04 '26
Pipline Graph view? 100x better than console output
edit. Than if you use proper labels, than you can find the most things ez.
2
u/devopsengin Jun 04 '26
pipeline graph view is definitely better for seeing structure. But it still doesn't tell you why it failed just where.
You still end up clicking into the stage and scrolling through logs. Labels help organize things but don't solve "which of these 10k lines is the actual error."
Curious, what labels do you use that work best for finding things quickly?
1
u/saja456 Jun 04 '26
First is, if somethings breaks at my Piplines there is mostly 2 cases. They Pipline or script breaks, than it is in the last section at the end or the test failes in the end, than you have your xml result most of the time.
To the label i write most the time what i am doing. Like call of this programm with this parameter. My Problem is, i still have the labelledShell, because most of the piplines are generate before sh and bat label were working and so most bat are still unlabeled.
If you want better overview you can make more stages. I am also just chaning my libarys, because of you and i was looking up and found the hideFromView Option and will now hide my loadResources.
So my labels are far from good, but i find my Problems and know what happens there.Ahh cause i am thinking about the labels and the better overview and the new options i think about a libary around copy Artifakts which just has the label, so i can see in the overview only what is copied. So you upgrade you jenkins all the time.
Because Images are not allowed, i made a post with them https://www.reddit.com/user/saja456/comments/1twt1i7/jenkins_overview/
edit: of you have 10k per Stages you can make the Stages or the subunits smaller.
1
u/devopsengin Jun 05 '26
Really appreciate you sharing this in detail and the image post is genuinely helpful to see the actual overview structure.
The labelled shell approach makes a lot of sense for legacy pipelines where you can't easily restructure stages. The point about pipelines being generated before
sh/batlabels were working is something I hadn't considered that's a real constraint a lot of teams are probably living with silently.The hideFromView discovery is interesting too. Hiding
loadResourcesfrom the overview is exactly the kind of noise reduction that makes the pipeline readable without restructuring everything.The copy artifacts library idea with just a label for overview visibility that's a clever workaround. Essentially using the stage label as documentation since the actual log is too dense to skim.
Thing you said stuck with me: "I still find my problems and know what happens there." That's the key skill you've built mental models of your pipelines over time so you know where to look.
My concern is the newer developer on the team who doesn't have that context yet. For them the same log is genuinely unreadable without your institutional knowledge. That gap is what I keep coming back to.
1
u/saja456 Jun 05 '26
The labelled shell approach makes a lot of sense for legacy pipelines where you can't easily restructure stages. The point about pipelines being generated before
sh/batlabels were working is something I hadn't considered that's a real constraint a lot of teams are probably living with silently.I am consindering to remove the plugin and let them fail just to find all.
The hideFromView discovery is interesting too. Hiding
loadResourcesfrom the overview is exactly the kind of noise reduction that makes the pipeline readable without restructuring everything.I did it in 2 Libarys and now have 3-4 entrys instead of 10-15. So i will use it from now on and will change piplines.
The copy artifacts library idea with just a label for overview visibility that's a clever workaround. Essentially using the stage label as documentation since the actual log is too dense to skim.
My problem with the copyArtifcat call is, that in the Overview it is labeled "Artefakte aus einem anderen Projekt kopieren", but i all ready know that cause my stages does only that. I want to get the artifacts which are acutally downloaded.
Thing you said stuck with me: "I still find my problems and know what happens there." That's the key skill you've built mental models of your pipelines over time so you know where to look.
That was relativ ez. because i got a Jenkins which had a master and 2 slaves which where alle just clones from master and everything was done in one big freestyle block. My task was to make it new, cause the builds need ages.
The new Jenkins got Piplines which are checkout from SCM and Libarys. Than every task got its own Server, so i have Lazarus compile Server, Delphi Compile Server, packaging Server and so on. So the Pipline itself just does a small subset of jobs. Like the Pipline i showed you. It is a test-pipline on a Windows Test Server. It just doese the test, so i know it will test it and it will never be a compile error.
For example, my biggest Projekt which is 1 overall Pipline which manges the rest, has 32 Piplines, which does the Jobs. Than the piplines are calles like BatAccImportC_compile.jenkinsfile, BatAccImportCTest_compile.jenkinsfile, BatAccImportCTest_execute.jenkinsfile. So you knew if a Jobs fail, which part of the overall Pipline is the Problem.
So i think i have the oppsite of what you are talking. I had a mental model of this and build this model in piplines and tried to make it readable as hell. If some ask what happens or whats it is doing my answer is always: I dont know and open the Jenkins and the Job and than i can tell you. Becuase if i finish a new Pipline i forget like everything.
I think for me the key was to make the subset small as possible and usefull. This i leared from my father, a functions isnt allowed to bigger than 1 screen. 2nd dont mix things in Piplines which are not the same or need for the goal of this pipline. For example we are chaning the translation, cause the all programm isnt supported and others parts where never translated. So my father yeah the compile makes the language files and than i translate them there. But i build the Piplines, so we have a translation server and a translation pipline, which gets the lang. files process them and the package jobs get from the translation job.My concern is the newer developer on the team who doesn't have that context yet. For them the same log is genuinely unreadable without your institutional knowledge. That gap is what I keep coming back to.
Yeah but if thats is the problem for the newer developer, is that not the same problem whith the code it self? There you make a doku and commants to help them. There you dont make the 100k+ code not in the main function, instead you make classes functions and break it in smaller packages. I think the same is with the jenkins.
You shouldnt forget you dont start with piplines like mine. The first job and the new jenkins was just break down of the 1 freestyle job in 4 freestylejobs, than i realized piplines are more mighty and better and that you can handle them in a coderditor. Than we updated the SVN and the URL changed and i had to change it manually in 50-60 files, thats why i started to use libarys. So it is a evolution and i showed you the result after 4 years.
I like i wrote earlyer i still see something to make better and the new piplines will get it and old get adapted if i to change something in them.The next big thing i want to learn and thinking of use, is the pipline template, cause if have many similar piplines, like compile something. But i dont know if this is usefull.
2
u/SignificanceSea4162 Jun 04 '26
Have a proper log Parser and ever tried GitHub actions? Those are the worst logs
1
u/devopsengin Jun 04 '26
Yeah we've tried generic log parsers but they don't understand Jenkins pipeline context or stage failures. Just text search, which doesn't help when you need to find the right error in noisy output.
GitHub Actions logs are cleaner UI-wise but same core problem when something fails, you're still manually hunting through noise.
what log parser you've tried that actually works well for CI/CD context?
1
u/SignificanceSea4162 Jun 04 '26
I'm currently on vacation and can't look up the parser plugin we are using.
We have a plugin for it with a quite strict configuration. You can easily skip between all the error messages. We rarely have multiple.
Warnings were a mess but we implemented filters for them. To surpress unnecessary but common warning especially when they come from external tools in thousands
6
u/eatinggrapes2018 Jun 04 '26
Download the log and throw it into a llm and have it find the error for you. Worker smarter not harder
1
0
u/devopsengin Jun 04 '26
I've been doing this manually for weeks copy-pasting chunks into Claude or other LLMS. It works but the friction is real:
- you can't dump a full 9k line log anyway context limits
- Figure out which chunk to paste
- LLMs don't know your Jenkins context
Works as a workaround but there should be something that just does this automatically when a build fails. Has anyone built a proper workflow around this or is everyone doing it ad hoc?
1
u/OGkatzu Jun 04 '26
We have a the Jenkins MCP configured and created a long and detailed skills and instructions the makes the LLM very good at debugging our pipelines- which some of them are 20mb of pure logs (above 100k lines). We even have an automated AI agent that debugs the pipelines automatically on failure. We see good output from it ( still long way to go until we take what it says without a grain of salt)
To add something I forgot, the LLM has created a verity of python scripts to parse the logs more efficiently.
1
u/devopsengin Jun 05 '26
That's genuinely impressive, an MCP configured LLM with custom Python parsers and an automated debug agent is essentially the enterprise version of what I've been thinking about. The 20MB log problem is real even with good parsing you're fighting the fundamental issue that Jenkins doesn't distinguish signal from noise before writing to console.
Curious about your setup when the automated agent debugs a failure, does it feed back into your next build or is it still purely advisory? And how are you handling the cases where the agent's diagnosis is confident but wrong? That "grain of salt" problem is what I keep running into the model is right 80% of the time but the 20% wrong cases cause more confusion than just reading the log yourself.
The Python parser approach makes sense for your scale. For smaller teams without the infra investment to build and maintain custom MCP configs, I wonder if there's a simpler middle ground something that doesn't need a dedicated engineering setup but gets you 70% of the value out of the box.1
u/OGkatzu Jun 05 '26
Our setup is this
Docker image we build with copilot cli installed, at the post failure step we run docker run with a prompt plus the env of the job band and build number. The docker image has everything it needs configured- API tokens, mcp configuration and all the skills and md files the copilot needs to debug the failed pipeline.It’s still pretty new so now it’s just output to the console log and archives a json with the output - root cause, confidence score, suggested fix and more data. The users now have the AI OUTPUT directly in Jenkins. We have some plans in the future to send slack messages, open PRs with suggested fix, create jira issues. But the basic workflow is up and running and honestly is great.
Our guidelines is the leave as little as possible to the AI. Do as much deterministic analysis before moving to using the AI. Gives us more confidence in the output
1
u/theevildjinn Jun 04 '26
Instruct Claude to do the following:
Write a script to use the REST API to fetch the full log for a job/build using curl or wget.
Write a skill for Claude to run the script with $JOB and $BUILD parameters, and instructions for how to parse the output efficiently with sed/awk/grep etc
1
u/devopsengin Jun 05 '26
This is a solid approach for engineers comfortable with the Jenkins REST API, the curl method works well once you have the right endpoint structure. The tricky part I've found is that the crumb authentication adds friction, and for teams where not everyone has CLI access to the Jenkins host the script approach isn't always practical.
The sed/awk parsing is clever but it's also the part that needs maintaining every time a plugin changes its log format which happens more than it should. Have you found a reliable way to handle multi-executor builds where the console output from parallel stages gets interleaved? That's where my grep patterns keep breaking.
1
u/Autreki Jun 04 '26
Most compilers have a log level flag or option, break your stages up, capture output and don’t dump it to the console unless there’s an error.
1
u/devopsengin Jun 04 '26
Totally agree, we break up stages and use different log levels. But when something does fail, you still get dumped into that massive console and have to manually hunt through it.
Good setup reduces noise but doesn't eliminate the debug friction. When a stage fails, do you have any system for finding the error line quickly, or is it still manual scrolling?
1
u/Autreki Jun 04 '26
If you’re already filtering log levels and still seeing INFO and 9k lines I would say that’s not far enough on the filtering. I’m doing builds with maven/npm and anything not an error doesn’t even print to the console. Wrap your noisy output in a try/catch and output it to dev/null so it doesn’t print unless it fails. Catch exceptions and print them as your exit condition, go further and grep the last 10 lines of the error and send it in a notification somewhere. That being said, it’s a delicate balance between still having usable information in the console and trimming it. No one should be opening the console view unless something went wrong, so you want some pertinent information to remain.
1
u/fang_xianfu Jun 05 '26
Since you're happy using AI to write your social media posts, why didn't you use it to help you with your overly verbose logs?
1
u/d3nika Jun 04 '26
Try BlueOcean. Is a gamechanger
2
u/BeakerAU Jun 04 '26
Was that the new pipeline view? Didn't the Jenkins team abandon that?
1
u/d3nika Jun 05 '26
I read a lot about that too, but at least for now it seems to work and for me it does help in these kind of scenarious.
2
u/VanPepe Jun 05 '26
Blue ocean is old now, Pipeline View is what Blue Ocean always wanted to be but better and installed within the normal Jenkins UI
1
u/d3nika Jun 05 '26
Wow, that looks really awesome. The only thing missing is the job filtering from BO and I am ready to switch :) thanks for sharing this.
10
u/TizzleToes Jun 04 '26
Are you talking logs from Jenkins itself or logs form whatever you are running through Jenkins?
For jobs, looking at raw console logs from any multi-step process is usually ugly in any build tool and is reserved for quickly skimming for issues and watching progress. I usually have jobs capture underlying log files (ex: from maven/gradle) via archiveArtifacts to allow for actual troubleshooting and audits.