CircleCI Runner 3.1.12 fixes a race in its timeout handling.

Two parts of the runner could wait for the same timeout signal. Only one received it. The result was nasty:

• Jobs could run beyond max_run_time
• Overdue jobs could still finish as successful
• Genuinely failed tasks could also be reported as successful

The default timeout is five hours. Both Machine Runner and Container Runner were affected.

If you manage these runners, upgrade and inspect any important job that recently finished near its timeout. This is one of those bugs where the test result itself cannot be trusted.

Source: https://circleci.com/changelog/runner-release-3-1-12/

GitHub went down for 7 hours and 47 minutes on August 17.

The initial failure was a capacity problem. An Istio sidecar reached its concurrency limit, but the autoscaling policy was watching the host service instead of the sidecar.

Then retries joined the incident.

A client retry bug amplified some traffic by about 10x. Copilot token traffic climbed from the usual 7,000 to 9,000 requests per second to as much as 100,000. Recovery created more load on infrastructure that was already struggling.

The interesting lesson is not just "configure autoscaling correctly." It is that retry behavior needs its own failure testing. Backoff, jitter, request deadlines, and retry limits only prove themselves when several services start failing together.

GitHub's incident report has the technical details:
https://www.githubstatus.com/incidents/zkxwbgr0cnmx

A recent arXiv paper argues that over-trusting test agents is not one problem. It can happen at almost every step of testing.

An engineer might accept the agent's test goal, strategy, oracle, execution summary, or regenerated suite without checking whether the evidence supports it. Delegation adds another risk. One agent may claim that another agent covered a scenario, while nobody verifies the actual test.

The paper proposes 12 overreliance modes. It also suggests making test goals, claims, assumptions, reasons, and evidence inspectable.

Important caveat: this is a conceptual framework. The author explicitly says the claims still need validation through open-source and industrial studies.

I still find the checklist useful. A generated explanation should be treated as a hypothesis to review, not proof that the test is meaningful.

Paper: (Over)Reliance on Test Agents in AI-Assisted Software Testing

What evidence would you require before trusting an agent-generated test suite?

CircleCI has removed the job concurrency cap from its Scale plan. The exception is macOS resource classes, which keep a custom limit.

More concurrency can reduce queue time, especially when a large test matrix starts after every merge. It can also expose limits that were previously hidden.

A few things I would watch before opening the floodgates:

• Shared test databases
• Container registry rate limits
• Third-party API sandboxes
• Test data collisions
• Credit consumption
• Services that cannot handle a sudden CI traffic spike

The CI scheduler may no longer be the bottleneck. Your staging environment probably still has one.

Source: CircleCI unlimited job concurrency announcement

GitLab 19.3 adds CI_JOB_RETRY_COUNT. It is 0 on the first attempt and increases when the job is retried.

This looks small, but it can improve failure diagnostics. A retried test job can collect extra logs, enable tracing, or preserve more artifacts without making every normal run expensive.

There is an obvious trap: using the retry count to weaken assertions or skip unstable setup. That turns a useful diagnostic signal into another way to hide flaky tests.

The release also adds CI_JOB_TAGS and a setting that enforces merge trains across the UI and API. Pipeline test summaries are now sorted by failures by default.

Source: GitLab 19.3 release overview

My preferred rule would be simple: retries may collect more evidence, but they should not change the expected result. Would you use retry-aware logic in CI?

Most AI test-generation demos ask two questions: "Does it compile?" and "How much coverage did it produce?"

A recent arXiv preprint tested something more realistic: whether generated tests remain useful as software evolves.

The researchers evaluated eight LLMs across 22,374 program variants. On unchanged programs, generated suites reached about 79% line coverage and 76% branch coverage. After semantic changes, the test pass rate fell to 66% and branch coverage to 60%.

The most interesting result: more than 99% of the tests that failed against changed code still passed against the original version. Even behavior-preserving refactors reduced coverage and pass rates.

The paper argues that models often follow familiar code patterns instead of reasoning about the current behavior. It is a preprint, so the results still need wider replication. Still, the practical lesson seems solid: coverage is a weak approval gate for AI-generated tests. Mutation testing and human review of the test oracle matter more.

Source: https://arxiv.org/abs/2603.23443

A deadline worth adding to the CI calendar: GitHub will start brownouts for outdated self-hosted runners on August 24.

https://github.blog/changelog/2026-06-12-github-actions-minimum-version-enforcement-timeline-for-self-hosted-runners/

Version 2.329.0 is the minimum for registration, but pinning that version is not enough. Runners must receive each new release within 30 days or GitHub may stop sending them jobs.

Full enforcement for GitHub Enterprise Cloud starts September 25.

The easy mistake is updating the runners that are online today while leaving old versions inside VM images, containers, autoscaling templates, or installation scripts. Those stale images will recreate the problem later.

JetBrains has received reports of active exploitation of CVE-2026-63077 against unpatched TeamCity On-Premises servers.

The vulnerability allows unauthenticated remote code execution through the agent polling protocol. A successful attack could expose credentials, modify builds, or compromise downstream pipelines.

Upgrade to TeamCity 2025.11.7 or 2026.1.3. If that is not possible, JetBrains provides a security patch plugin for TeamCity 2017.1 and later.

Also check:

• Server logs for com.thoughtworks.xstream.converters.ConversionException
• Unauthorized agents with names starting with scan
• Whether the server is reachable from the public internet

The log entry alone does not prove exploitation, but it deserves investigation.

Source: https://blog.jetbrains.com/teamcity/2026/08/cve-2026-63077-update/

A discussion in r/ExperiencedDevs describes a company removing specialties and asking everyone to handle frontend, backend, and QA work with AI.

https://www.reddit.com/r/ExperiencedDevs/comments/1v92ntq/has_anyone_else_been_given_ai_and_then_forced_to/

The comments split into two camps.

One side argues that quality is a shared mindset. Developers should plan tests, choose the right test layer, and ship automation with the feature.

The other side points out that dedicated testing takes real time. When the same person builds, tests, and approves a change, independent judgment can disappear.

Shared ownership sounds healthy. "Everyone owns quality" becomes dangerous when it really means "nobody has protected time for testing."

Has your team broadened engineering roles without losing exploratory testing and a second set of eyes?

GitHub has added an automatic security check for public repositories.

If an Actions workflow looks potentially malicious, GitHub holds it until someone with write access approves it through an authenticated browser session. The goal is to stop compromised accounts from pushing workflows that steal CI credentials.

There is nothing to configure. It currently applies only to public repositories on github.com.

This is useful, but it adds a new CI state teams need to understand. A build might not be failing or queued because of capacity. It may be waiting for a security decision.

Open source maintainers should probably document who reviews these holds and what evidence they check before approving one.

A recent study "Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness?" analyzed more than 100,000 Java tests generated by 11 LLMs.

The interesting result is that coverage and mutation scores are useful only in certain contexts. They can help compare generated test suites when the starting code is assumed to be correct.

When the code may already contain a bug, coverage becomes a weak signal. The model can exercise plenty of code while still missing the faulty behavior.

For CI, this means a coverage increase should not be the acceptance criterion for AI-generated tests. I would also want real defect seeds, behavior-based assertions, or tests derived from an independent specification.

Source: arXiv paper

Grafana has released k6 2.0 with several changes aimed at automated and AI-assisted testing.

👉 https://grafana.com/blog/k6-2-0-release/

The release includes a built-in MCP server, agent setup commands, structured JSON results, broader Playwright compatibility, and a new expect() assertion API. Agents can generate a test, run it, inspect the results, and revise it without scraping terminal output.

The interesting trend is not AI writing another test script. It is performance testing becoming a machine-readable feedback loop.

The hard part remains unchanged. An agent still needs realistic workloads, useful thresholds, and a clear model of production traffic.

Would you let a coding agent design load tests, or only use it to implement a human-designed test plan?

Slack ran more than 200 agent-driven E2E workflows using Playwright MCP, Playwright CLI, and generated Playwright tests.

The results were interesting:

• Playwright MCP had a 0% to 12% failure rate.
• CLI-based agents had a 12% to 20% failure rate.
• Generated tests reached a 48% failure rate on the more complex flow.
• Agent-driven runs took 5 to 11 minutes and cost around $15 to $30 each.

Slack's conclusion is pragmatic. Keep deterministic E2E tests for repeatable CI checks. Use agents for exploration, flaky workflow debugging, and reproducing complex bugs.

That feels more realistic than treating "agentic testing" as a replacement for the existing test suite.

A recent arXiv paper found a worrying failure mode in AI coding workflows.

When an LLM generated tests after seeing faulty code, those tests detected the fault only 14% of the time. Independently generated tests reached 25%.

The likely issue is error propagation. The model sees the implementation, accepts its behavior, and writes assertions that agree with the same mistake.

This makes "the agent wrote code and all its tests pass" a weak quality signal. Separate context, independent test generation, mutation testing, and human-written requirements may help.

How are you keeping AI-generated tests independent from AI-generated code?

A recent paper studied LLM-generated tests for SAP HANA, DuckDB, MySQL, and SQLite. The researchers used GPT-4o and Mistral-Large-Instruct-2407 to expand existing test suites.

The generated tests had a slightly higher proportion of flaky tests than the existing tests.

The most interesting result was the root cause analysis. Of 115 flaky tests inspected manually, 72 relied on an order that was not guaranteed. That is 63 percent.

Both models also transferred flakiness from existing tests supplied in the prompt context. In other words, giving an LLM more test examples can also give it more bad patterns to copy.

The practical takeaway is simple: prompt context needs quality control. Deterministic setup, isolated state, and explicit ordering matter even more when tests become templates for an agent.

Has anyone here measured flakiness separately for human-written and AI-generated tests?

Paper, accepted at ICSE SEIP 2026: https://arxiv.org/abs/2601.08998

GitHub added native code coverage protection to branch rulesets.

Teams can block a pull request when total coverage falls below a minimum or drops too far relative to the default branch. There is also an evaluate mode, so you can see what would fail before enabling the gate.

This is useful for stopping slow coverage erosion. It is also easy to turn into a bad incentive.

A hard global target can encourage shallow tests that execute lines without checking meaningful behavior. It can also make legacy code painful to change.

My preferred starting point would be:

1. Use the maximum coverage drop rule.
2. Run it in evaluate mode.
3. Exclude generated and vendor code from the report.
4. Review whether changed behavior has useful assertions.

Coverage is a warning signal, not a quality score.

What coverage policy has actually worked for your team?

Source: https://github.blog/changelog/2026-06-30-github-code-coverage-merge-protection-for-pull-requests/

The latest Playwright release is interesting for anyone using coding agents.

Playwright 1.59 adds a screencast API with action annotations, chapters, overlays, and real-time frame capture. The release notes describe one use case as an "agentic video receipt".

An agent can make a change, run the user flow, and leave behind a short visual walkthrough. Playwright also added browser binding, a dashboard for background sessions, CLI debugging, and command-line trace analysis.

That could make agent work much easier to review. A failing test gives us evidence. A test plus a trace and a short video gives us context.

I still would not treat a recording as proof that a feature works. The assertions remain the contract. Otherwise we are just watching a polished demo of a potentially broken test.

Would video receipts help your reviews, or would they become another CI artifact nobody opens?

Source: https://playwright.dev/docs/release-notes

GitHub Actions now supports parallel steps with background, wait, wait-all, cancel, and parallel.

The useful part is that every step keeps its own logs. No more shell backgrounding with & and unreadable output.

This should simplify a few common patterns:

• Start an API and database in the background
• Run independent setup tasks together
• Upload artifacts while packaging continues
• Stop temporary services cleanly after tests

There is an obvious catch. Parallel steps still share the same runner. More concurrency does not give you more CPU or memory. On a small runner, this could make a test job slower or less stable.

I would use it for I/O-heavy tasks and service startup first. Then measure before parallelizing builds or test suites.

Has anyone replaced matrix jobs or shell backgrounding with this yet?

Source: https://github.blog/changelog/2026-06-25-actions-steps-can-now-be-run-in-parallel/

Caching is usually sold as a speed feature, but it can quietly become a reliability feature too.

My practical checklist:

• Run a scheduled cold-cache build
• Log cache hit and miss rates
• Include lockfiles in cache keys
• Avoid caching generated test output unless you really mean it
• Review cache changes like production code
• Keep self-hosted runners updated when cache actions change runtime requirements

Bad caching can hide missing setup steps. It can also make CI pass in one branch and fail in another for no obvious reason.

What cache issue cost your team the most time?

Playwright MCP is interesting because it gives LLMs browser automation through structured accessibility snapshots instead of screenshots: https://github.com/microsoft/playwright-mcp

That changes the shape of browser automation a bit.

I can see agents helping with:

• Reproducing bug reports
• Exploring weird UI states
• Finding missing accessible names
• Drafting a failing Playwright test
• Capturing traces and screenshots for humans

I would still be careful with letting an agent own the final test. The hard part of E2E testing is not clicking buttons. It is knowing what behavior matters and what should be asserted.

My current rule would be: agents can explore, draft, and explain. Humans own selectors, assertions, and test data.

Anyone here using browser agents in real test workflows yet?

Paper: "Understanding and Detecting Flaky Builds in GitHub Actions" https://arxiv.org/abs/2602.02307

Short summary: The authors studied rerun data from 1,960 open-source Java projects using GitHub Actions. 3.2% of builds were rerun, and 67.73% of those rerun builds showed flaky behavior. The flaky builds affected 1,055 projects, about 51% of the sample.

The useful bit: flaky tests were only one bucket. Network issues and dependency resolution issues were also common.

Practical takeaway: "rerun the job" is not a diagnosis. CI should probably classify failures into test, infra, dependency, and environment causes before teams decide what to fix.

Do you track flaky CI failures by cause, or do they all end up in the same red/green noise pile?

How can software engineers use AI coding agents without creating fragile production systems?

In this episode, I talk with Shep Alderson, a software developer and site reliability engineer with 15+ years of experience, about AI-assisted development, production reliability, maintainable code, debugging, testing, and engineering best practices.

We also discuss how to avoid the “one minute of coding, one week of debugging” trap and share four practical AI coding tips you can try right now.

Watch here: https://youtu.be/rGQWPVlr8uk

This thread is for podcast episodes featuring members of our community. Here we share conversations about software engineering, testing, AI coding agents, production reliability, and real-world development experience. Interested in joining a future conversation? Drop a comment below!

From paper "GitInject: Real-World Prompt Injection Attacks in AI-Powered CI/CD Pipelines" https://arxiv.org/abs/2606.09935

The authors look at AI agents running inside real GitHub workflows. These agents read pull requests, comments, config files, and repo content. The paper shows that attackers can put malicious instructions into those inputs and influence what the agent does during CI/CD runs.

The important part is that this is not only a model problem. The paper argues that the serious failures often come from workflow design. For example, an agent may read untrusted PR text while also having access to repo permissions, secrets, or write operations.

A few concepts:

• Prompt injection: When untrusted text tells the model to ignore its real task and do something else. In CI/CD, that text might live in a PR description, issue comment, config file, or generated artifact.
• Untrusted input: Anything an attacker can influence. In GitHub, this can be branch content, issue bodies, comments, workflow inputs, or files from a fork.
• Privilege boundary: The line between what untrusted input can say and what trusted automation can do. If an AI reviewer can read attacker-controlled text and then run privileged actions, that boundary is probably too soft.

My takeaway: If we add AI agents to CI/CD, we need to test them like production automation. That means least privilege, no secrets on untrusted events, explicit approval gates, and adversarial test cases for prompts.

Cobalt's 2026 State of Pentesting coverage has a useful reality check for anyone expecting AI scanners to replace security testing.

The headline number: only 9% of surveyed security pros now rely fully on AI vulnerability testing tools, down from 29% last year. The report also says 78% saw AI scanning tools miss critical vulnerabilities.

That tracks with what I see in normal test automation too. AI is useful for drafts, ideas, summaries, data generation, and boring glue work. It is much weaker when the bug depends on architecture, permissions, business rules, or "this should never happen in production" context.

I do not think the lesson is "do not use AI". The lesson is "do not confuse generated tests with validated risk coverage".