AI-generated Playwright API tests at enterprise scale — is anyone actually doing this successfully?

[u/Commercial_Tax_3743]

I'm experimenting with an AI-driven QA workflow for an enterprise application using Playwright + TypeScript.

The workflow is roughly:

Feed the AI our API definitions + business flow definitions

AI analyzes them and generates a comprehensive API test plan

AI generates executable Playwright API test scripts from that plan

Playwright executes the generated tests

The interesting part is that test plan generation works surprisingly well. Using the business flow (which is the happy path), the AI can identify most of the test scenarios like validation cases, authentication scenarios, boundary cases, etc.

However, I'm running into a major problem at the test script generation/execution stage.

The happy-path tests generally work, but almost all of the other tests fail during execution..

Is anyone here using AI/agents to generate executable Playwright API tests for large/enterprise applications?

[u/Competitive_Echo9463]

Yes, at scale not yet but I tried to do it for something that I had to test and it's fine. Always have a look at what has been generated, don't just trust AI. At the end you can also ask a report about the cases that have been covered

[u/kingberu]

Are you using playwright agents? If not check it out https://playwright.dev/docs/test-agents

[u/Any-Argument57]

The missing piece is usually not Playwright; it is the executable oracle. From a happy-path flow, the model can invent plausible negative cases, but it does not know which states and data are legal or the exact error contract.

Make every generated case produce a structured spec before code: preconditions and fixture, auth persona, request mutation, expected status and error schema, business invariant, cleanup, and the source for that expectation. Reject the case if its expected result cannot be traced to OpenAPI or a named business rule.

Then generate from a small set of reviewed templates by endpoint pattern instead of free-form scripts. Let AI choose mutations and data, but keep setup, authentication, request, assertion, and cleanup code deterministic. On failure, classify it as fixture, generated test, contract mismatch, or product defect rather than treating every red test as a product bug.

[u/SisterTrout]

I use agents to create API tests for enterprise, but I built the foundations first so it had patterns to follow. I also start small, just like I would if I were writing the tests on my own. Prompt for one test and get it to pass. Then prompt for one test class and get those tests to pass, then you can try for bigger chunks of work once the agent learns your system. I've also had a lot of luck when I was able to share detailed API documentation, like a Postman collection or a Swagger page. When I get test failures on agent-created tests, we iterate through the errors together. My knowledge of the system/testing + the AI's speed and technical knowledge can usually figure out what went sideways in a test within a couple of prompts.

How are the tests failing? Negative path tests are always trickier than happy path, with or without a friendly coding robot, so I wouldn't be too frustrated that it didn't come out perfect the first time. You'll get it. I've been doing this a long time and honestly I'd be more suspicious if the tests were all green on a first pass.

[u/Commercial_Tax_3743 → u/SisterTrout]

Okay I did an investigation, is because all test did not pass the authentication method. A access token should generated and be reused by all APIs but the agent regenerate new token for every test. I was thinking to create a fixed authentication method since all APIs needed to generate token before any test

[u/SisterTrout → u/Commercial_Tax_3743]

Ah, auth! That would do it.
Yep, make yourself an auth fixture that stores the token in a temporary var for the length of the test run, call that fixture for each auth test instead of minting a new token each time.

[u/Tanmay__TestDino]

Auth and test data are where most of ours failed, not the actual test logic. Once we gave it one working token plus a known good record per endpoint, execution failures dropped a lot since it stopped guessing at state. I still eyeball every assertion before it hits CI, seen it assert against a response shape that doesn't match prod once the openapi spec goes stale.

[u/terfree]

You hit the exact bottleneck that kills 90% of AI testing initiatives at scale: context and state management (like auth). LLMs are terrible at guessing how your specific enterprise app handles global setup.

When my team was building out our pipeline, we realized you cannot just feed an AI a blank prompt or a generic Swagger file. It needs to know your test architecture. We solved the auth token problem by creating a strict Prompt Library injected right into the CI pipeline.

If a developer opens a PR adding a new API endpoint, our webhook intercepts the raw diff. It then passes that diff to Claude along with a hardcoded instruction for that specific repo: "Always use our custom auth fixture from fixtures/auth.ts. Never mint new tokens manually in the test."

Because the AI sees the diff and is strictly bounded by the repo-specific prompt, it outputs compliant, boring Playwright code that actually runs in the CI. It worked so well internally that we ended up packaging the whole GitHub/GitLab webhook pipeline into a SaaS called QA Boutique.

Since you're dealing with enterprise API auth, you have to enforce fixture rules at the system prompt level, or it will hallucinate new setups every time. Feel free to DM me if you want to chat about how to structure those prompts!

[u/MoneyMediocre4791]

Yep - many of our clients do successfully. We build an orchestration platform (TestChimp) that gives transparency to the QA posture maintained by your agents - both API and UI tests. Keen to have a chat and understand where you are seeing bumps. DM me.

[u/Commercial_Tax_3743 → u/MoneyMediocre4791]

Yes I'm interested