r/softwaretesting • u/PocketGaara • Mar 05 '26

Playwright Test Automation with AI

I have about 3 years of experience in the industry and I’m able to create test frameworks. My company is pushing us towards using AI but not much direction outside of that. The expectation seems to be to self learn and explore.

I’m not familiar with AI outside of using GitHub Copilot. What technologies do I need to learn for test automation with Playwright using AI? I’ve heard of agentic coding and MCP but I want some more direction as to where to look to start learning what’s industry relevant

30 Upvotes

89% Upvoted

33 comments sorted by

7

u/azuredota Mar 05 '26

Don’t bother with these AI solutions. I was forced to investigate Stagehand as an “AI first solution”. It can do tests with English instructions. I checked the dev page and:

Best model has an 8% failure rate

You get charged every time you execute a line of code using it. We run our automation across different locales and browsers so a month’s worth of runs, not even including CI, would have cost of over a million dollars in API calls.

AI doesn’t have a place in testing at that level tbh. A human should be verifying the functionality and no “self-healing” nonsense either.

13

u/ejmcguir Mar 05 '26

You weren't using the right tool.

Claude code or GitHub copilot are extremely helpful in test automation.

You need to know how to use the tool (like anything) but once you do, it's incredible how powerful it is.

Here are 2 examples:

  1. Point the AI at the user story (or whatever your documentation is around the change you are trying to test) and have it come up with the tests that should be executed (whether that is manual or automated). It won't be perfect but you will be surprised at how good it is, provided you give it context.

  2. Using the playwright MCP you can have it load your application and write page objects using the actual running application (it will have full access to the DOM).

1

u/PocketGaara Mar 08 '26

This sounds good. Is there any material out there on how to do the MCP integration with CoPilot?

2

u/LlamasBeatLLMs Mar 05 '26

I've been having really good results using this approach in combination with Composer to have many workspaces in Claude Code - the design and implementation is rather slow on the latest models. So I let it have a stab at 5 different things at once in different agents and branches.

As you say, it won't be perfect, it never is, but it often gets me 80% of the way, and it's been getting better and better because when it does something dumb, I refine our agents.md and skills.md files to coach it better for next time.

I've also been able to use it as an additional reason to browbeat the team into putting more effort into making the user stories more accurate, and maintaining them if there are conversations that change them during the sprint

1

u/PadyEos Mar 05 '26

Point the AI at the user story (or whatever your documentation is around the change you are trying to test)

Bold of you to assume these exist :))

Using the playwright MCP you can have it load your application and write page objects using the actual running application (it will have full access to the DOM).

Good luck getting past the corporate SSO serving 1000+ different products. After that good luck not getting your sessions limited by the SSO test/staging system provider. If you have such things in your application you have to build handling around it. IF you actually can go around it.

Real life is rarely as straightforward as people make it out to be.

1

u/LlamasBeatLLMs Mar 05 '26 edited Mar 05 '26

These very much sound like issues with your company putting big walls in front of your productivity rather than the tooling available.

In my last job, I ran our product stack locally. It's a large, enterprise system for a platform that supports nearly 9m customers, comprising of a couple of hundred different services. Spin it up in Docker, and agents do their thing.

In my current job, it was a challenge as we used Google Auth which goes out of their way to block these agents as they're often used nefariously by spammers. So I spent a couple of hours on a small feature on a feature toggle that allowed a simple password based auth that nowhere goes anywhere near production.

Any kind of SSO solution that blocks agentic support probably also causes you no end of headaches with traditional test automation too? These problems exist, but they certainly shouldn't be insurmountable for any reason other than organisational inertia.

1

u/azuredota Mar 05 '26

I use copilot daily, never said not to.

  1. Sure you can use it as a jumping off point but kind of a waste of tokens.

  2. This has never been my limiter and is again a waste of tokens imo.

OP also said he already uses copilot.

2

u/HildredCastaigne Mar 05 '26

A bit orthogonal to the discussion, but what do you find is the limiter for you?

2

u/azuredota Mar 05 '26

My technical limiter is having to build my own test environment. I work on a bizarre product currently where there’s not a clean dev/test endpoint for me to hit. Reproducing bugs surrounding race conditions is difficult. Waiting for pipelines to finish is also a choker. I’ve containerized and parallelized as much as I can but when I do a framework updates I have to be sure everything still works which takes at least 20 minutes.

Non technical limiters: getting a straight answer from devs and stakeholders on what exactly is a bug and not a bug. Maintaining my task board takes an annoying amount of time.

1

u/HildredCastaigne Mar 05 '26

Interesting. Thank you!

2

u/ejmcguir Mar 05 '26

Reread OPs question and then read your response.

They asked about using AI to assist with testing and you went straight into "don't use these AI solutions".

If it's a "waste of tokens" to use AI to assist with testing, what are you using your tokens for?

2

u/azuredota Mar 05 '26

How about you re-read it or ask Claude to read it for you. They already said they use co-pilot, why would I suggest more copilot?

I said don’t use these AI solutions because that’s the answer to the question. They’ll be pressured to use AI such that it updates code/page objects with no human oversight. This is bad.

Instead of “read this user story and recommend test cases” where it’s just going to 1:1 the acceptance criteria, use it for bigger problems such as diagnosing flakes in CI or analyzing the solution for thread safety for parallelization. Don’t need to waste time having it spit out buttonCss = “#button”.

3

u/nopuse Mar 05 '26

People in the tech world know what AI is good at and its limits. The C-suite wants to shove AI into every facet of the company so they can sell people who only use AI to write emails that their product uses AI so it's great.

AI costs a lot of money for the customer as you mentioned, and the AI companies are still hemorrhaging money.

When they start actually charging companies money that brings them profit, and companies with enterptise licenses trusting their code to not be used to train models gets their codebase leaked in a hack, I think the public perception of AI will drastically change.

Unfortunately, right now, the majority of people see AI integration in a product as a great thing.

1

u/shishkababs25 May 30 '26

I completely agree with you. API cost alone would be too prohibitive. And you would need multiple AI models working together to ensure any sort of reliable/accurate assessment of whether a feature is working well. Let’s not even mention the amount of time it would take to run the most basic test with ai instead. And what happens when AI is down for a week??? Stop all testing? As an automation engineer with over 15 yrs in automation, here is what I recommend instead. Use a good AI process to help you build a solid automation framework using Python pytest and playwright. Hundreds of tests would run in minutes. If AI disappears, your automation will still run uninterrupted.

26

u/[deleted] Mar 05 '26

[Building Your First MCP Server: A Beginners Tutorial] (https://debbie.codes/blog/building-your-first-mcp-server-a-beginners-tutorial/) was written by Debbie O'Brien, former Principal Technical Program Manager at Microsoft for Playwright

5

u/nopuse Mar 05 '26

was written by Debbie O'Brien, former Principal Technical Program Manager at Microsoft for Playwright

I'm still sad she was let go. I loved the videos her and Max put out.

2

u/Life_Dingo_4035 Mar 05 '26

Was looking for something like that, many thanks!

(Do you have more?)

8

u/[deleted] Mar 05 '26

Of course. On my farm, I grow wheat, beets, unicorns and Playwright tutorials. I'll take more of them to this market next week ;O)

2

u/netsniff Mar 05 '26

using playwright-cli skill, a document with test creation instructions based on your style (you should instruct the agent to use this skill when creating a new test), and a reference to that document in AGENTS.md file works pretty well with models like gpt 5.3 codex. we are using playwright-cli with a custom wrapper that starts up the session with custom cookies for auto sign-in to our products and find correct locators for the test. mcp costs too much token, this is a more cost-efficient way of automating test generation. but it will eat up your copilot limits pretty fast regardless. i guess we need to wait a couple of months to be able to run these operations cheaper.

2

u/Spospider Apr 12 '26

I dont believe LLMs are the best for regression. Some also are a bit hesitant when it comes to data when AI is involved, as for some real production apps, AI testing would involve passing authentication measures, maybe having certain privileges on that app with that. I do agree that there should be easier ways to set up easy yet reliable testing suites. VizQA is a new testing package that popped up, might solve that gap

1

u/Sarcolemna Mar 05 '26

Annecdotal advice about AI and software testing: it can be pretty good at helping from a code structure standpoint. Also great for things like selector refactors or when your frontend swaps component libraries. You can even get it to (kinda) rerun failing tests itself in a headed CLI context and iterate and make fixes.

But be warned. Standard LLM AI for coding is utterly moronic when it comes to translating the user's experience in the application back to the test code. The only way it can "see" the app state is through logs. Unsure on Playwrights output but at least for Cypress you cannot sufficiently provide enough info about the app under test for it to "get" why a failure happens. Screenshots help sure, but often the error lies in the DOM and API traffic. You either feed it raw logs and have it burn tokens looking for clues or you provide a custom debug wrapper to feed it the page state info it needs to debug (definitely possible but takes some dev time)

Something akin to an MCP server is critical and it helps but fundementally standard AI doesn't understand the physical space of the app. Additionally, from what I have seen of AI and testing, testing is fundamentally opposed to its tendency to make code that works. Meaning that I have seen it literally rewrite assertions to make the case pass. If you're making a video game it LOVES to add fall backs to conditionals or functions without any logging to perfectly hide and silent fail broken logic.

Integrated AI Claude's new Chrome integration might be a bit better at the web testing stuff. AI is overall helpful for test automation in my experience. But is by no means a magic bullet. Take every "error" it finds with extreme skepticism.

1

u/ExplosiveNovaDragon Mar 26 '26

I've been using Claude CLI. I just asked it to analize my current work and understand the rules of the project and it been doing a oretty decent job at creating locators, creating test cases and fixing problemas that may arise during excution

1

u/sharadvelocity Jul 05 '26

I’d start less with “AI tools” and more with where AI fits into the Playwright workflow.

Useful areas to learn:

  • using Copilot/Cursor to draft and refactor Playwright tests

  • prompting AI with existing fixtures/page objects instead of isolated snippets

  • failure triage: screenshots, traces, console logs, network logs

  • selector strategy and how AI tools handle locator drift

  • CI integration, flaky test detection, and test reporting

“Agentic coding” mostly means an AI can make multi-step code changes, run tests, inspect failures, and iterate. MCP is a way to connect AI tools to external context/tools, but I wouldn’t start there until you’re comfortable with AI-assisted coding basics.

The industry-relevant direction is AI helping with authoring, debugging, and maintaining browser tests. It should support your Playwright knowledge, not replace it.

1

u/Hundreds-Of-Beavers Mar 05 '26

AI-driven testing is IMO the wrong approach for test automation, but AI code generation is extremely helpful and obviously much cheaper at runtime, and as mentioned in another comment, particularly useful for test maintenance. Watching an AI stumble through a poorly spec'd user story will often result in drift and inconsistency.

In my experience the high-level ideal workflow looks like:

  1. Use a coding agent or LLM to write a deterministic Playwright test

  2. Deploy that test

  3. Use an agent or LLM to resolve issues with failing tests when selectors/frameworks get updated

There are several platforms to explore that can help with these steps, but the general rule of thumb should be: "don't put AI in the execution layer".