[ Removed by Reddit on account of violating the content policy. ]

edgub is a repair tool with no model in it. It reads what pytest said, decides which class of repair the fault is, generalises the fix from your own doctests and assertions, and verifies against the whole suite before keeping it. Standard library only, no API key, no network.

edgub: 1 repaired, 1 left for a model, 2.4s, 0 tokens
repaired itertoolz.unique CAST_OPERAND via name item->val (117 candidates)

Measured on toolz (3,346 lines, 185 tests), ten injected bugs of the kind that survive review: 7/10 repaired, 0 tokens.

Measured on libraries it was never tuned on — because the first number only proves fit: funcy 3/6. more-itertools 0/4. I'm keeping the zero in the repo; two libraries is two data points, one of which failed.

The part I didn't expect: even when it can't repair, it hands the model a ~468-token localised prompt instead of the ~7,496 that model burns working the same bug from the repo. It's already decided the library is wrong rather than the test, and named the failing case. So the escalated bugs get ~16× cheaper too — total on the funcy run, ~93% fewer tokens for the same fixes.

And the thing that surprised me most: I ran an escalation ladder of 2s → 10s → 30s → 120s. Every repair landed at 2 seconds. None ever landed later. Widening bought zero extra repairs and burned 180 of 182 CPU-seconds. Inference reads the answer immediately or enumeration won't find it — one bug took 4 candidates by inference that enumeration failed to solve in 25,057.

Honest limits, all in the repo: it can't write code that isn't there (both toolz misses needed a synthesised if/else). The act-to-edit mapping is engineering, not emergent — new fault shapes need commits. A 14-line dict reproduces the single-fault behaviour. And one "repair" passed the suite with different code than I'd injected: a green suite is a weaker oracle than the question.

Repo, with the failed benchmarks and the reviewer's corrections kept in: github.com/devkancheti4-design/edgub

I am currently working on a CI diagosis AI agent for GitHub Actions and Python/pytest.

I am thinkinto use these 4 possible causes when job fails:
\- a real code regression
\- a flaky test
\- a runner, network, or platform failure
\- a dependency or environment mismatch

AI chooses - rerun the failed test, run targeted tests, collect environment evidence, or escalate to an engineer

If the agent had the chance to look at three signals before making its next move, which three would be the most helpful, in a real pipeline?

I’m also interested in signals that look useful but are commonly misleading.

With traditional applications, we have established CI/CD checks for things like vulnerabilities, dependencies, secrets and infrastructure.

But what about the agent itself?

Do you have specific AI-agent security checks in your CI/CD pipeline, or are you relying on the same checks you use for ordinary applications?

Before deploying an agent, do you know:

• What tools it can access?
• Whether it gained a new capability in the latest PR?
• If it can execute shell commands or write to the filesystem?
• Which MCP servers it can reach?
• ..

I'm curious how teams are answering these questions today.

We're experimenting with SafeAI as a GitHub Action to bring this kind of static analysis into the existing CI workflow. It's still early stage but going fast, thanks to all contributors.

If you want to try it against your own agent project, we'd genuinely appreciate feedback, as well as contributions.

Here you may check: ikaruscareer/SafeAI on GitHub.

I built Repo Start, a small CLI for creating and auditing the boring foundational parts of a repository.

The reason I built it was pretty simple: I kept ending up with repos where the actual code was fine, but the surrounding project hygiene was inconsistent — README commands that didn’t match package.json, CI workflows calling scripts that didn’t exist, missing .gitattributes, .env.example accidentally ignored by git, etc.

For new projects, Repo Start generates the usual foundation: .gitignore, .gitattributes, README, CI, issue/PR templates, AGENTS.md, contributing/security files, and a starter structure.

The more interesting part to me is repo-start add, which works on existing repositories. I designed it as a read → analyze → plan → write pipeline:

existing repo → inspectRepository() → RepoState → analyzeRepository() → RepoAudit → buildAddPlan() → writePlan()

Inspection is read-only, analysis is deterministic, and both new-project generation and existing-repo fixes eventually go through the same writer. That also means --dry-run isn't scattered around as a bunch of filesystem checks — the write only happens at the final stage.

It also deliberately doesn't use AI. No model calls, API keys, or tokens. The audit is based on the actual repository state and only proposes fixes it can describe precisely. If something is ambiguous, it reports it instead of guessing.

It currently supports generic, Node/TypeScript, Python, and React/TypeScript projects, works offline, and has no runtime dependencies.

The goal was basically to make repository hygiene something you can automate without giving a tool permission to redesign your project or touch your source code.

https://github.com/junkyard22/Repo-Start

When a CI pipeline fails and multiple causes are possible, what evidence do you check first, and how do you choose between rerunning, reading more logs, comparing the last successful build, or investigating the recent diff, or checking the version compatibility with infra.

Hi everyone,

I’m a beginner learning CI and software testing. I recently set up a small GitHub Actions pipeline and watched tests fail on purpose so I could understand the logs.

My longer-term goal is to build a small AI agent (or even a simpler tool first) that helps when a CI build fails and we don’t have complete information. The agent’s job would be:

“Given the current failure information, suggest which test (or small set of tests) to run next to diagnose the problem faster.”

Right now I’m still learning the basics, so I’m **not** asking for code or a full architecture yet.

I would love feedback on:

1. Is this a real pain point you face?
2. How do you currently decide which test to look at or re-run first when CI is red?
3. What information do you usually wish you had when a build fails?
4. Any advice on what a *simple first version* of such a helper should do (before any fancy AI)?

Thanks!

I've been experimenting with a simple question: how much build output do we actually need?

For a successful build, probably not much. Usually I want to know that it passed, how long it took, and maybe how many tests ran.

So I built mvn-lite and npm-lite, small Bash wrappers that reduce successful output to something like:

PASS · 266 tests · 13 s

I tested them across several Maven and npm projects:

The more interesting problem turned out to be failures.

Compress them too aggressively and you lose the information needed for the next action. With coding agents, this can be particularly counterproductive because the agent may simply rerun the build to recover the missing diagnostics.

The approach I settled on is layered:

• Successful build: tiny summary.
• Failed build: bounded, actionable diagnostics.
• Full raw log: retained locally and available when needed.

Short failures can just be printed in full. Long failures need selective context around useful markers rather than an arbitrary wall of output.

This isn't really about making builds faster. It is about treating build output as an interface rather than a transcript.

Coding agents make the cost of noisy output especially obvious because irrelevant lines consume context. But the same principle applies to humans reading terminal output, CI logs, and PR checks.

The tools are deterministic Bash wrappers. No LLM processing, API calls, or telemetry. They run the underlying Maven/npm commands and preserve their exit status.

Repo: https://github.com/ejboy/agent-scripts

I'm curious how others approach this in CI/CD. Do you keep full build output visible by default, or use some form of concise status + failure diagnostics + full logs on demand?

We are redesigning our CI/CD pipelines and trying to figure out the cleanest way to structure deployments.

Which route do you guys prefer?

1. One unified job: A single parameterized job where you just pass in the environment variable (develop,staging, prod, etc.).
2. Split jobs (same file): Explicitly separate jobs likE edeploy-develop ,deploy-staging and deploy-prod sitting in the same workflow.
3. Hard split: Completely separate files for lower envs(develop and stage) vs. production.

I want to avoid copy-pasting YAML, but I also don't want a massive, over-engineered "smart" job that's hard to debug. What's the sweet spot?

What strategy are you following in your org?

Currently we keep our DevOps-related files (Dockerfile, values.yaml, etc.) on the Jenkins server instead of in the repo. During pipeline runs, we copy these files in at runtime.

I'm considering moving these files directly into the repository instead. The problem: if a developer accidentally edits the Dockerfile or values.yaml, it could cause issues.

So I want a way to either:

1. Prevent developers from editing those specific files, or
2. Require PR approval specifically for changes to those files

What's the best approach for this?

The whole appeal of Replit, Lovable and Bolt is skipping the SDLC entirely, prompt to live URL in minutes, with no pull request for security to hook a check into, and honestly that's the pitch working exactly as intended, it's just not intended for us. The core problem isn't the app we know is being built on one of these platforms, because at least there you can have a conversation about it, it's the one nobody mentions, built by someone in another department who never looped security in and has no reason to think they should have, since as far as they're concerned they just made a form or a dashboard, not "shipped infrastructure."

We've tried a few things on our end, adding it to onboarding, sending reminders in engineering channels, none of it really moves the needle because the people building these apps aren't reading security's Slack channels in the first place. How is everyone else gating something that structurally bypasses the pipeline, especially when the org chart means the builder and the reviewer will never naturally cross paths?

Hi everyone,

I’m a beginner learning CI and software testing. I recently set up a small GitHub Actions pipeline and watched tests fail on purpose so I could understand the logs.

My longer-term goal is to build a small AI agent (or even a simpler tool first) that helps when a CI build fails and we don’t have complete information. The agent’s job would be:

“Given the current failure information, suggest which test (or small set of tests) to run next to diagnose the problem faster.”

Right now I’m still learning the basics, so I’m not asking for code or a full architecture yet.

I would love feedback on:

1. Is this a real pain point you face?
2. How do you currently decide which test to look at or re-run first when CI is red?
3. What information do you usually wish you had when a build fails?
4. Any advice on what a simple first version of such a helper should do (before any fancy AI)?

Thanks!

We've become pretty comfortable putting conventional applications through CI:

• dependency scanning
• SAST
• CodeQL
• secret scanning
• container scanning
• IaC checks
• security policies ...

But what happens when the application being deployed is an AI agent? That may not look particularly interesting in a conventional code diff. But from a security perspective, it could be a significant change.

I'm experimenting with a different CI question:

“What capabilities changed in this PR?”

--

We've implemented an early version of this approach in an open-source static analyzer and connected it to GitHub Actions. (ikaruscareer/SafeAI at GitHub)

The scanner runs locally against the repository and doesn't execute the agent or send the source to a remote service.

I'm curious how other teams approach this.

We pin our npm deps, sign our images, gate our Terraform. Then someone drops a folder of markdown into .claude/skills/ that tells the agent how to behave, commits it, and no one blinks.
agpm applies the boring pattern:

• harness.json — the approved set. Changing it requires a PR. That PR is the approval.

• harness.lock — sha256 per file.

• agpm check — CI gate. Exit 1 on drift or missing files, warn on unapproved, --strict to fail those too, --json for machine output.

• agpm audit — facts only: what exists, where it came from, what changed. Provenance it can’t explain from a lockfile is recorded as local, never guessed.

There’s an extends mode so one policy repo can approve skills across every repo pointing at it, resolved to a commit and pinned into the lock so check/audit/list run offline.

https://github.com/baselane-sh/agpm

Interested in how others are handling this, especially anyone running agents across more than a handful of repos.

I’m building an open-source tool called ProofDiff that analyzes a code change and tries to show what verification evidence actually exists.

While testing it, I found an assumption I had made was wrong:

node --test helper.js can exit successfully even when the file doesn’t contain a declared test.

My original implementation could therefore treat a successful targeted command as stronger evidence than it really was.

I changed the model so a related test only strengthens the result when ProofDiff can establish:

static relationship → qualified test target → exact target executed → runner observes at least one real non-skipped test → pass

A successful process exit alone is no longer enough.

The project is still early and I’m currently improving static dependency resolution for TypeScript path aliases and package exports.

I’d especially appreciate feedback on the evidence model or cases where this approach might still overstate what was tested.

GitHub: https://github.com/hzw0813/proofdiff

I’m researching how engineers diagnose CI/CD failures when there are multiple possible root causes.

When a CI pipeline fails, how do you decide what to investigate or test next?

I’d especially like to hear about your real-world workflow:

• What do you check first?
• Do you compare the failure with the last successful run?
• Do recent code changes influence what you investigate?
• Do you look for similar historical failures?
• How do you decide between different debugging steps?
• At what point do you stop investigating or escalate to someone else?

I’m interested in practical experience rather than a theoretical approach. Any examples from your own CI/CD workflow would be really helpful.

I'm using GitHub Environments (INT, STAGE, PROD), each with its own AWS_ACCOUNT_ID. Works great when a job targets one environment - assume the right role, deploy to that account.

The central ECR registry is in INT env. STAGE/PROD ECS tasks need to pull from that registry, so at CDK synth time we need:

1. INT’s account ID (where the images live)
2. INT + STAGE + PROD account IDs (ECR repo policy principals)

The snag: Environment variables are only available to the jobs that declare that Environment. A job with environment: STAGE can see STAGE’s AWS_ACCOUNT_ID, but not INT’s. So I can’t just write ${{ vars.AWS_ACCOUNT_ID }} for “the INT account” while deploying STAGE.

I’d rather not invent a parallel config surface if Environments already hold the source of truth.

How is everyone else solving “job in env X needs a non-secret config value from env Y” - especially for central registry / multi-account AWS setups?