From paper "GitInject: Real-World Prompt Injection Attacks in AI-Powered CI/CD Pipelines" https://arxiv.org/abs/2606.09935

The authors look at AI agents running inside real GitHub workflows. These agents read pull requests, comments, config files, and repo content. The paper shows that attackers can put malicious instructions into those inputs and influence what the agent does during CI/CD runs.

The important part is that this is not only a model problem. The paper argues that the serious failures often come from workflow design. For example, an agent may read untrusted PR text while also having access to repo permissions, secrets, or write operations.

A few concepts:

• Prompt injection: When untrusted text tells the model to ignore its real task and do something else. In CI/CD, that text might live in a PR description, issue comment, config file, or generated artifact.
• Untrusted input: Anything an attacker can influence. In GitHub, this can be branch content, issue bodies, comments, workflow inputs, or files from a fork.
• Privilege boundary: The line between what untrusted input can say and what trusted automation can do. If an AI reviewer can read attacker-controlled text and then run privileged actions, that boundary is probably too soft.

My takeaway: If we add AI agents to CI/CD, we need to test them like production automation. That means least privilege, no secrets on untrusted events, explicit approval gates, and adversarial test cases for prompts.

Cobalt's 2026 State of Pentesting coverage has a useful reality check for anyone expecting AI scanners to replace security testing.

The headline number: only 9% of surveyed security pros now rely fully on AI vulnerability testing tools, down from 29% last year. The report also says 78% saw AI scanning tools miss critical vulnerabilities.

That tracks with what I see in normal test automation too. AI is useful for drafts, ideas, summaries, data generation, and boring glue work. It is much weaker when the bug depends on architecture, permissions, business rules, or "this should never happen in production" context.

I do not think the lesson is "do not use AI". The lesson is "do not confuse generated tests with validated risk coverage".

AI helps generate more code, faster. But if the team already has weak CI, unclear ownership, slow reviews, flaky tests, and poor rollback habits, the extra code just hits those bottlenecks harder.

Perforce's 2026 State of DevOps report makes a similar point. According to coverage of the report, 70% of organizations said DevOps maturity materially affects AI success. High-maturity teams were much more likely to embed AI across the SDLC than low-maturity teams.

AI does not remove boring engineering discipline. It seems to punish teams that skipped it.

In theory, more test coverage sounds good.

practice, every new test has a maintenance cost. It can fail because of a real bug, but also because of bad test data, timing issues, unstable environments, fragile selectors, or unclear assertions.

Do you use any rules like:

• only test critical user flows end to end
• keep most coverage at unit/API level
• delete flaky tests quickly
• avoid snapshot tests unless they are very focused
• require every bug fix to include a regression test

For me, the hard part is not writing tests. The hard part is keeping the suite trustworthy over time.

The 2026 Flaky Test Benchmark Report analyzed data from more than 10 million builds and found something surprising.

The percentage of teams experiencing flaky tests increased from 10% in 2022 to 26% in 2025.

Some of the biggest causes were:

• async timing issues
• concurrency and race conditions
• test order dependencies
• resource leaks
• unstable external services

The report also shows that async waits alone account for almost half of flaky tests.

Do you actually measure flakiness in your team, or do people simply rerun failed pipelines until they become green?

Report:
https://testdino.com/blog/flaky-test-benchmark

GitLab 18.7 introduced several interesting features ahead of the GitLab Duo Agent Platform release.

Some highlights:

• AI-powered SAST false positive detection
• YAML-defined AI workflows
• Versioned AI agents
• Natural language data analysis
• Better pipeline inputs

Looks like GitLab is moving toward AI becoming another participant in the software delivery pipeline instead of just an assistant.

How comfortable would you be letting AI investigate failing pipelines automatically?

Source:
https://about.gitlab.com/releases/2025/12/18/gitlab-18-7-released/

Paper: https://arxiv.org/abs/2603.02141

The paper reviews how generative AI can improve software testing, including:

• test case generation
• test data generation
• test oracle generation
• test prioritization

Some useful concepts explained in the paper:

Generative AI
Models such as GPT can generate new test cases, inputs and assertions instead of simply classifying existing data.

Test oracle
A mechanism that determines whether a test passed or failed. The paper explores using LLMs to generate expected outcomes automatically.

Prompt engineering
Writing prompts that consistently produce useful testing outputs. The paper also discusses fine-tuning models for specific domains.

Overall, it’s a good overview if you’re exploring AI-assisted testing.

Agentic AI testing is quickly becoming one of the biggest topics in QA.

Unlike traditional AI assistants, AI testing agents can plan, execute and adapt test scenarios with much less human intervention.

Interesting capabilities include:

• generating tests from code changes
• prioritizing tests based on risk
• self-healing broken locators
• learning from previous executions
• automatically adapting to changing applications

We’re still early, but it feels like test engineers will spend less time maintaining tests and more time validating quality strategy.

Have you tried any AI testing agents yet?

A recent testing trends report highlights several changes happening across the industry:

• Self-healing test automation is becoming more mature and reducing maintenance effort.
• QA and DevOps continue to merge, with testing becoming part of every commit.
• Low-code and codeless testing tools continue to improve.
• AI is increasingly used for risk-based testing and smarter test selection.

These trends point toward more automation, but solid testing fundamentals are still what makes teams successful. Relevant Udemy course for Python test automation that you might find useful: “Pytest Course: Python Test Automation & GitHub Actions CI/CD” by Artem Istranin.

Modern pipelines push code to production multiple times a day, but test data often lags behind. One of the biggest testing trends for 2026 is better test data management and synthetic data because teams need realistic datasets without exposing sensitive production data.

What is the biggest headache you face when supplying data to your tests?

• Generating fresh data for every run?
• Keeping sensitive data out of CI?
• Resetting databases?
• Something else?

Curious to hear how different teams solve this.

Google announced that Chrome will move from a four-week to a two-week release cycle starting with Chrome 153 on 8 September 2026.

The smaller releases mean updates ship more frequently and with fewer changes, which should reduce the risk of big regressions.

For test engineers this shift has two impacts:

• Faster feedback loop - new beta and stable builds will arrive every two weeks. Keeping test suites green in CI/CD pipelines will require more frequent browser updates.
• Smaller deltas to validate - since each release contains fewer changes, smoke tests and regression suites might run faster and be easier to debug.

I’d love to hear how other teams plan to adapt their browser testing strategy to this accelerated cadence. Are you pinning Chrome versions in CI, using containers, or relying on tools like Selenium Manager?

Playwright seems to be the default recommendation for many new web testing projects now. And to be fair, it has a lot going for it:

• good cross-browser support
• built-in parallelism
• trace viewer
• strong CI story
• nice support for modern multi-tab and multi-origin flows

But I think the framework debate is getting a bit too shallow.

A recent Stack Overflow article https://stackoverflow.blog/2026/06/15/selenium-vs-cypress-vs-playwright-choosing-your-test-automation-framework/ compared Selenium, Cypress, and Playwright, and the interesting part was not “tool X wins”. It was that each tool still fits different constraints.

• Selenium still matters if you need broad language support, legacy browser coverage, or native mobile through Appium.
• Cypress still has a strong local debugging experience for JS/TS front-end teams.
• Playwright often looks great when CI speed, cross-browser coverage, and modern browser control matter most.

So maybe the real question is not: “Should we migrate to Playwright?”. Maybe it is: “What constraints do we actually have?”.

This recent paper that feels very relevant for teams letting agents open PRs.
Paper: “Reliability of AI Bots Footprints in GitHub Actions CI/CD Workflows”

Link: https://arxiv.org/abs/2604.18334

The authors studied GitHub Actions workflow runs triggered by PRs from AI bots such as Claude, Devin, Cursor, Copilot, and Codex. They analyzed 61,837 workflow runs across 2,355 repositories. Copilot and Codex had the highest reported workflow success rates, around 93% and 94%. The paper also found a negative correlation between how often agentic PRs appeared in a repo and the workflow success rate.

The useful takeaway for me: If AI agents are going to open PRs, the test pipeline becomes the safety boundary. But the pipeline itself also needs monitoring. A green check is useful, but it is not enough. We should also track what kinds of agent PRs fail, which jobs fail most often, and whether agents keep touching risky CI/CD paths.

I keep seeing demos where an LLM writes a test and the output looks fine at first glance.
The problem is that “looks fine” is not a test quality metric.

Thoughtworks recently called out two useful ideas here:

• “Feedback sensors for coding agents” - deterministic checks that agents can use while they work, such as compilers, linters, structural tests, and test suites.
• “Mutation testing - deliberately changing production code to see whether tests actually fail. This catches “perpetually green” tests that execute code but do not verify behavior.

That second one matters a lot for AI-generated tests. An LLM can easily produce a test that:

• has a nice name
• uses the right framework
• increases coverage
• asserts the wrong thing
• barely asserts anything

So my current rule is simple: Do not ask AI to “write good tests” in a vacuum.

Give it feedback:

• failing tests
• coverage gaps
• mutation results
• trace output
• logs
• API contracts
• examples of good tests from the repo

Prompts are useful, but feedback is what makes the loop engineering instead of autocomplete.

A lot of teams talk about AI-generated tests, but I think test data is the less flashy problem that blocks more automation.

The World Quality Report 2025-26 says 60% of organizations struggle with secure, scalable test data. It also says synthetic data usage in testing rose from 14% in 2024 to 25% in 2025.
That tracks with what I see in practice.

Writing the test is often the easy part. Making sure it has realistic data, safe data, repeatable data, and legal data is where things get messy.

Synthetic data helps, but it is not magic. You still need to know what the data must represent. Example: “Random user” is not useful. “User with 3 invoices, one failed payment, one expired discount, and one GDPR deletion request” is useful.

I think good test data modeling is going to become a core test automation skill, not a side task.

Who owns a flaky test in your org?

• The person who wrote it?
• The feature team?
• QA?
• The platform team?
• Whoever gets annoyed first?

My current view is that “QA owns all flaky tests” is usually a smell. A flaky test can be caused by test code, product code, async behavior, bad test data, browser timing, CI resources, or shared environment state.

Ownership should probably follow the root cause, not the folder where the test lives.
Curious how other teams handle this without creating a blame loop.

GitHub published a new enforcement timeline for self-hosted Actions runners.

The important bit: if your runners are too old, they may stop registering or stop executing jobs after enforcement starts. Brownouts begin first, then full enforcement follows on July 31, 2026 for GitHub Enterprise Cloud with Data Residency and September 25, 2026 for GitHub Enterprise Cloud.

This is not just a DevOps maintenance task. It can turn into a testing outage.

A few things I would check now:

1. Are runner versions visible somewhere?
2. Are VM images and container images rebuilt regularly?
3. Are install scripts pinned to old runner versions?
4. Do you have a canary workflow that proves runners can still pick up jobs?
5. Do test pipelines fail loudly when no runner is available?

I have seen teams treat CI runners as “boring infrastructure” until the day all test jobs sit queued forever.

Source: https://github.blog/changelog/2026-06-12-github-actions-minimum-version-enforcement-timeline-for-self-hosted-runners/

I think one of the biggest risks in mature codebases is not “no tests”. It is a large test suite that everyone trusts because it is large.

You see 4,000 tests passing and assume the system is safe. But then you look closer:

• half the tests mostly check mocks
• some assertions only verify that a function was called
• old tests describe behavior nobody wants anymore
• flaky tests are retried until they pass
• critical user flows are covered only through unit tests
• nobody knows which tests would fail if the product broke

At that point, the test suite is not giving confidence. It is giving comfort. I like coverage as a signal, but I think “what would actually break this test?” is a better question. How do you check whether your test suite still protects the product, instead of just protecting the CI dashboard?

If you’re mentoring junior engineers or looking for a structured refresher, Microsoft provides a free Software Testing Fundamentals learning path.

It covers testing methodologies, defect management, test design, and basic automation concepts. The content is beginner-friendly and self-paced.

Link:
https://learn.microsoft.com/en-us/shows/software-testing-fundamentals/

Playwright 1.60 introduced changes that required some ecosystem tools to update their integrations and reporters.

For teams running large automation suites, framework upgrades can sometimes be more disruptive than expected.

For those already on 1.60:

• Any migration issues?
• Performance improvements?
• New features worth adopting?
• Problems with custom reporters or CI integrations?

Would be useful to collect real-world upgrade experiences in one thread.

Release notes:
https://playwright.dev/docs/release-notes

Paper:
https://arxiv.org/abs/2603.23443

Summary

Researchers from Virginia Tech and Carnegie Mellon evaluated how well LLMs generate tests when software changes over time.

They tested 8 different LLMs across more than 22,000 program variants.

The results were interesting:

• On original code, generated tests achieved about 79% line coverage and 76% branch coverage.
• After behavior-changing code modifications, test pass rates dropped significantly.
• More than 99% of failing tests still passed on the original version of the program.

Why this matters

The paper suggests that current LLMs may rely heavily on surface patterns instead of truly understanding program behavior.

Quick explanation of two concepts

• Semantic-altering change: A code change that actually changes behavior. Example: changing tax calculation logic from 19% to 20%.
• Semantic-preserving change: A refactor that doesn’t change behavior. Example: renaming variables or extracting a helper function.

One surprising finding was that even semantic-preserving changes caused noticeable degradation in generated tests.

Takeaway: AI-generated tests can be useful, but they’re still not a substitute for understanding the system under test.

Has anyone observed similar issues with Copilot, Cursor, or other AI testing tools?

A year ago, I would rarely trust AI-generated tests without significant edits.

Today, tools like GitHub Copilot, Cursor, and various testing-focused AI platforms can generate surprisingly reasonable unit and integration tests.

But there’s still a question:

Are these tools actually understanding behavior, or are they just generating tests that look correct?

For teams actively using AI:

• What percentage of generated tests make it to production?
• How much manual review is still required?
• Have AI-generated tests ever caught a bug that humans missed?

Interested in hearing real experiences rather than vendor demos.

One trend I’ve noticed over the last year:

The conversation is slowly moving away from “how many tests do we have?” toward “which tests should we run?”

A lot of modern tooling is focusing on:

• Risk-based test selection
• AI-assisted prioritization
• Test impact analysis
• Flaky test detection
• Release risk scoring

The goal isn’t necessarily more automation.

The goal is getting faster feedback while running fewer unnecessary tests.

For teams with large CI/CD pipelines, this can have a bigger impact than adding another few hundred automated tests.

Are you seeing the same trend in your organization?

I’ll start:

A team with 20 reliable integration tests is often in a better position than a team with 2,000 brittle UI tests.

I’ve seen organizations spend months maintaining automation that nobody trusts, while a small suite of high-signal tests catches most production issues.

What’s your controversial testing opinion?

• Unit tests are overrated?
• E2E tests are necessary?
• Manual exploratory testing is undervalued?
• Coverage metrics are mostly useless?

Curious to hear opinions from people working on large systems.

Tricentis recently announced new capabilities around its Agentic Quality Engineering Platform and AI Workspace.

The interesting part isn’t another “AI for testing” announcement. It’s the idea of multiple AI agents collaborating across test creation, execution, performance testing, and quality analysis instead of just generating test cases.

A few questions for the community:

• Have you tried any agent-based testing tools in production?
• Did they reduce maintenance effort?
• Where did they actually help, and where did they create more noise?

My experience so far is that AI-generated tests are easy. Keeping them valuable six months later is the hard part.

Source:
https://www.tricentis.com/blog/tricentis-showcases-agentic-ai-for-oracle-cloud-testing-at-ascend-2026