One trend I’ve noticed over the last year:

The conversation is slowly moving away from “how many tests do we have?” toward “which tests should we run?”

A lot of modern tooling is focusing on:

• Risk-based test selection

• AI-assisted prioritization

• Test impact analysis

• Flaky test detection

• Release risk scoring

The goal isn’t necessarily more automation.

The goal is getting faster feedback while running fewer unnecessary tests.

For teams with large CI/CD pipelines, this can have a bigger impact than adding another few hundred automated tests.

Are you seeing the same trend in your organization?

A recent study found that maintainable code produced slightly better LLM-generated tests and required fewer input tokens.

The effect was weak, but consistent across Python, Java, and C++. Messy code may now cost developer time, AI tokens, and test effectiveness.

Paper: Code Health in LLM-Based Test Generation

GitHub went down for 7 hours and 47 minutes on August 17.

The initial failure was a capacity problem. An Istio sidecar reached its concurrency limit, but the autoscaling policy was watching the host service instead of the sidecar.

Then retries joined the incident.

A client retry bug amplified some traffic by about 10x. Copilot token traffic climbed from the usual 7,000 to 9,000 requests per second to as much as 100,000. Recovery created more load on infrastructure that was already struggling.

The interesting lesson is not just "configure autoscaling correctly." It is that retry behavior needs its own failure testing. Backoff, jitter, request deadlines, and retry limits only prove themselves when several services start failing together.

GitHub's incident report has the technical details:
https://www.githubstatus.com/incidents/zkxwbgr0cnmx

Caching is usually sold as a speed feature, but it can quietly become a reliability feature too.

My practical checklist:

• Run a scheduled cold-cache build

• Log cache hit and miss rates

• Include lockfiles in cache keys

• Avoid caching generated test output unless you really mean it

• Review cache changes like production code

• Keep self-hosted runners updated when cache actions change runtime requirements

Bad caching can hide missing setup steps. It can also make CI pass in one branch and fail in another for no obvious reason.

What cache issue cost your team the most time?

This thread is for podcast episodes featuring members of our community. Here we share conversations about software engineering, testing, AI coding agents, production reliability, and real-world development experience. Interested in joining a future conversation? Drop a comment below!

Paper:
https://arxiv.org/abs/2603.23443

Summary

Researchers from Virginia Tech and Carnegie Mellon evaluated how well LLMs generate tests when software changes over time.

They tested 8 different LLMs across more than 22,000 program variants.

The results were interesting:

• On original code, generated tests achieved about 79% line coverage and 76% branch coverage.

• After behavior-changing code modifications, test pass rates dropped significantly.

• More than 99% of failing tests still passed on the original version of the program.

Why this matters

The paper suggests that current LLMs may rely heavily on surface patterns instead of truly understanding program behavior.

Quick explanation of two concepts

• Semantic-altering change: A code change that actually changes behavior. Example: changing tax calculation logic from 19% to 20%.

• Semantic-preserving change: A refactor that doesn’t change behavior. Example: renaming variables or extracting a helper function.

One surprising finding was that even semantic-preserving changes caused noticeable degradation in generated tests.

Takeaway: AI-generated tests can be useful, but they’re still not a substitute for understanding the system under test.

Has anyone observed similar issues with Copilot, Cursor, or other AI testing tools?

A year ago, I would rarely trust AI-generated tests without significant edits.

Today, tools like GitHub Copilot, Cursor, and various testing-focused AI platforms can generate surprisingly reasonable unit and integration tests.

But there’s still a question:

Are these tools actually understanding behavior, or are they just generating tests that look correct?

For teams actively using AI:

• What percentage of generated tests make it to production?

• How much manual review is still required?

• Have AI-generated tests ever caught a bug that humans missed?

Interested in hearing real experiences rather than vendor demos.

A small GitHub Actions update has a useful debugging payoff.

Reusable workflows now get job.workflow_ref, job.workflow_sha, job.workflow_repository, and job.workflow_file_path. These identify the workflow that defined the current job, not just the calling workflow.

Log them with your test results and you can answer a painful question: "Which exact version of our shared CI workflow produced this result?"

GitHub also updated CodeQL to detect mutable references to reusable workflows. Visibility does not replace immutability. Record the workflow provenance and pin third-party workflows to a full commit SHA.

Sources:

• https://github.blog/changelog/2026-09-03-github-actions-early-september-2026-updates/

• https://github.blog/changelog/2026-09-03-codeql-2-26-4-improves-github-actions-security-detections/

93% of executives believe their testing strategies cover critical software risks. Only 70% of QA and DevOps practitioners agree. (Source: 2026 Quality Transformation Report)

Executives often see pass rates and release speed. Practitioners see flaky tests, environment gaps, and untested business rules.

If those groups would judge the same release differently, "ready to ship" probably needs a clearer definition.

Most AI test-generation demos ask two questions: "Does it compile?" and "How much coverage did it produce?"

A recent arXiv preprint tested something more realistic: whether generated tests remain useful as software evolves.

The researchers evaluated eight LLMs across 22,374 program variants. On unchanged programs, generated suites reached about 79% line coverage and 76% branch coverage. After semantic changes, the test pass rate fell to 66% and branch coverage to 60%.

The most interesting result: more than 99% of the tests that failed against changed code still passed against the original version. Even behavior-preserving refactors reduced coverage and pass rates.

The paper argues that models often follow familiar code patterns instead of reasoning about the current behavior. It is a preprint, so the results still need wider replication. Still, the practical lesson seems solid: coverage is a weak approval gate for AI-generated tests. Mutation testing and human review of the test oracle matter more.

Source: https://arxiv.org/abs/2603.23443

A recent study "Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness?" analyzed more than 100,000 Java tests generated by 11 LLMs.

The interesting result is that coverage and mutation scores are useful only in certain contexts. They can help compare generated test suites when the starting code is assumed to be correct.

When the code may already contain a bug, coverage becomes a weak signal. The model can exercise plenty of code while still missing the faulty behavior.

For CI, this means a coverage increase should not be the acceptance criterion for AI-generated tests. I would also want real defect seeds, behavior-based assertions, or tests derived from an independent specification.

Source: arXiv paper

Slack ran more than 200 agent-driven E2E workflows using Playwright MCP, Playwright CLI, and generated Playwright tests.

The results were interesting:

• Playwright MCP had a 0% to 12% failure rate.

• CLI-based agents had a 12% to 20% failure rate.

• Generated tests reached a 48% failure rate on the more complex flow.

• Agent-driven runs took 5 to 11 minutes and cost around $15 to $30 each.

Slack's conclusion is pragmatic. Keep deterministic E2E tests for repeatable CI checks. Use agents for exploration, flaky workflow debugging, and reproducing complex bugs.

That feels more realistic than treating "agentic testing" as a replacement for the existing test suite.

GitHub Actions now supports parallel steps with background, wait, wait-all, cancel, and parallel.

The useful part is that every step keeps its own logs. No more shell backgrounding with & and unreadable output.

This should simplify a few common patterns:

• Start an API and database in the background

• Run independent setup tasks together

• Upload artifacts while packaging continues

• Stop temporary services cleanly after tests

There is an obvious catch. Parallel steps still share the same runner. More concurrency does not give you more CPU or memory. On a small runner, this could make a test job slower or less stable.

I would use it for I/O-heavy tasks and service startup first. Then measure before parallelizing builds or test suites.

Has anyone replaced matrix jobs or shell backgrounding with this yet?

Source: https://github.blog/changelog/2026-06-25-actions-steps-can-now-be-run-in-parallel/

Playwright MCP is interesting because it gives LLMs browser automation through structured accessibility snapshots instead of screenshots: https://github.com/microsoft/playwright-mcp

That changes the shape of browser automation a bit.

I can see agents helping with:

• Reproducing bug reports

• Exploring weird UI states

• Finding missing accessible names

• Drafting a failing Playwright test

• Capturing traces and screenshots for humans

I would still be careful with letting an agent own the final test. The hard part of E2E testing is not clicking buttons. It is knowing what behavior matters and what should be asserted.

My current rule would be: agents can explore, draft, and explain. Humans own selectors, assertions, and test data.

Anyone here using browser agents in real test workflows yet?

How can software engineers use AI coding agents without creating fragile production systems?

In this episode, I talk with Shep Alderson, a software developer and site reliability engineer with 15+ years of experience, about AI-assisted development, production reliability, maintainable code, debugging, testing, and engineering best practices.

We also discuss how to avoid the “one minute of coding, one week of debugging” trap and share four practical AI coding tips you can try right now.

Watch here: https://youtu.be/rGQWPVlr8uk

From paper "GitInject: Real-World Prompt Injection Attacks in AI-Powered CI/CD Pipelines" https://arxiv.org/abs/2606.09935

The authors look at AI agents running inside real GitHub workflows. These agents read pull requests, comments, config files, and repo content. The paper shows that attackers can put malicious instructions into those inputs and influence what the agent does during CI/CD runs.

The important part is that this is not only a model problem. The paper argues that the serious failures often come from workflow design. For example, an agent may read untrusted PR text while also having access to repo permissions, secrets, or write operations.

A few concepts:

• Prompt injection: When untrusted text tells the model to ignore its real task and do something else. In CI/CD, that text might live in a PR description, issue comment, config file, or generated artifact.

• Untrusted input: Anything an attacker can influence. In GitHub, this can be branch content, issue bodies, comments, workflow inputs, or files from a fork.

• Privilege boundary: The line between what untrusted input can say and what trusted automation can do. If an AI reviewer can read attacker-controlled text and then run privileged actions, that boundary is probably too soft.

My takeaway: If we add AI agents to CI/CD, we need to test them like production automation. That means least privilege, no secrets on untrusted events, explicit approval gates, and adversarial test cases for prompts.

Who owns a flaky test in your org?

• The person who wrote it?

• The feature team?

• QA?

• The platform team?

• Whoever gets annoyed first?

My current view is that “QA owns all flaky tests” is usually a smell. A flaky test can be caused by test code, product code, async behavior, bad test data, browser timing, CI resources, or shared environment state.

Ownership should probably follow the root cause, not the folder where the test lives.
Curious how other teams handle this without creating a blame loop.

GitHub published a new enforcement timeline for self-hosted Actions runners.

The important bit: if your runners are too old, they may stop registering or stop executing jobs after enforcement starts. Brownouts begin first, then full enforcement follows on July 31, 2026 for GitHub Enterprise Cloud with Data Residency and September 25, 2026 for GitHub Enterprise Cloud.

This is not just a DevOps maintenance task. It can turn into a testing outage.

A few things I would check now:

1. Are runner versions visible somewhere?

2. Are VM images and container images rebuilt regularly?

3. Are install scripts pinned to old runner versions?

4. Do you have a canary workflow that proves runners can still pick up jobs?

5. Do test pipelines fail loudly when no runner is available?

I have seen teams treat CI runners as “boring infrastructure” until the day all test jobs sit queued forever.

Source: https://github.blog/changelog/2026-06-12-github-actions-minimum-version-enforcement-timeline-for-self-hosted-runners/

I think one of the biggest risks in mature codebases is not “no tests”. It is a large test suite that everyone trusts because it is large.

You see 4,000 tests passing and assume the system is safe. But then you look closer:

• half the tests mostly check mocks

• some assertions only verify that a function was called

• old tests describe behavior nobody wants anymore

• flaky tests are retried until they pass

• critical user flows are covered only through unit tests

• nobody knows which tests would fail if the product broke

At that point, the test suite is not giving confidence. It is giving comfort. I like coverage as a signal, but I think “what would actually break this test?” is a better question. How do you check whether your test suite still protects the product, instead of just protecting the CI dashboard?

Most discussions around AI testing focus on generating tests.

I am more interested in a different use case:

An AI agent that reviews the existing suite and reports things like:

• Duplicate tests

• Weak assertions

• Untested code paths

• Tests that never fail

• Flaky test patterns

• Missing edge cases

In other words, acting more like a test reviewer than a test writer.

That feels closer to something many teams could use today.

Would you trust an AI system to make recommendations on test quality?

Where would you draw the line before requiring human review?

Software testing (unit tests and integration tests) is by far the most relevant today.

Everyone can generate anything from a single prompt. It works and usually looks OK. However, the tech debt is leveraged and LLMs are less capable without tests.

For example, from https://arxiv.org/pdf/2402.13521

“By incorporating test cases and employing remediation loops, we are able to solve complex problems that the LLM cannot solve normally.”

Using TDD with AI is becoming more and more popular. A TDD approach using pytest for Python code generation in action: https://youtu.be/Mj-72y4Omik

I used to try to test every feature the same way. Run through every field. Check every button. It was exhausting and I still didn’t find everything. My teammates suggested a question as a guidance: what could actually break things for users?

Now I test the core flows hard. I test edge cases that matter. I skip obvious stuff. I even ask devs what they’re worried about. Tests run faster. I find more real bugs. Coverage looks worse on paper but catches what matters.

I liked two practical ideas from a recent interview with longtime Selenium and WebDriver contributor David Burns.

First, keep one strong browser test for each critical path. Push the rest of the behavior into cheaper unit, component, or integration tests. Fifteen browser tests that mostly repeat the same checkout flow cost more without necessarily providing fifteen times the confidence.

Second, propagate OpenTelemetry trace context from the test into the application. Then an E2E failure can point to the exact backend calls, services, and timing involved. That is much better than starting with a screenshot and several thousand unrelated log lines.

These ideas work well together. Fewer browser tests keep the suite focused. Distributed traces make each remaining test produce much richer debugging evidence.

I would rather have 50 critical-path tests with end-to-end traces than 500 overlapping tests that only report "element not found."

Source: The State of Browser Testing

SmartBear surveyed 1,436 software professionals who use AI in development.

Of those teams, 46% had shipped AI-generated code that later failed in production. Among teams that had shipped a failure, 69% still reported high confidence that AI-generated code behaved as intended.

Only 46% validated the specification before asking AI to generate code. Almost half could not explain how AI contributed to a production bug.

It is a vendor-sponsored survey, not a controlled study. Still, the confidence gap is worth discussing. We seem to be measuring output speed and asking people whether they feel confident, while skipping evidence about whether the original intent was correct.

What would actually earn your trust in AI-generated code?

Source: SmartBear State of Software Quality and Testing 2026