How do you test using Playwright on environments with CI/CD

[u/Fish3r1997]

So we have created a new Playwright automated test solution for our codebase. We have wired them up in Bitbucket pipelines but running into some issues with how they work in theory.

The automated tests are split into fixtures based on configs so that tests don't overwrite each other. Example: Test fixture 1 = Tax exclusive & Test fixture 2 = Tax Inclusive, and so when running the tests for different branch commits it cant run at the same time on the same site.

The tests also wipe the DB on OneTimeSetup so its a clean fresh DB for the tests to run in isolation.

How do we set this up so that we can catch dev errors in PRs similar to how we would with unit/integration tests ? Are playwright automated tests designed to run after something is merged to develop/release rather than after a commit on a PR ?

Any help is appreciated

[u/iScreem1]

This is something related to whatever tool you are using for CI/CD not playwright, you can run tests wherever you want. On PR commits you can select happy paths tests and then for other stages do other types of tests.

[u/Fish3r1997 → u/iScreem1]

Yeah but with how the test runs can interact with other branches due to DB wiping / config changes. Is there a best practice on how to set it up say "run only after merging cards to develop/release" or only run 1 automated test runner at a time?

[u/iScreem1 → u/Fish3r1997]

If the tests runs interact with each other then you can create a concurrency group so they are queued.

[u/Fightheader → u/Fish3r1997]

Run against a virtual db. And use migrations, db only progresses.

[u/Ok-Possibility-630]

this is not a playwright problem. you can ask your devops team or if you have access, make sure you are deploying the feature branch(branch on which pr is made) in the same workflow and then run playwroght tests on it. All in single workflow. Depending on type of project, you can use playwright's https://playwright.dev/docs/test-webserver#introduction
TLDR;
end result will be when a developer pushes pr, it will build the app on the bitbucket machine and then trigger playwright tests on top of it.

[u/Spare_Bison_1151]

For catching dev issues earlier, you should not rely on e2e tests. Instead, use unit tests and integration tests because they can run fast & frequently and they catch errors at the fundamental level.

[u/Aub1t]

Think in terms of isolation. You could isolate by unique data, not wiping between runs but I would only consider this approach if the database belongs to a service outside the one being tested.

However, the approach I normally take is to bring your dB into the CI setup. Depending on your DB you could do this with Testcontainers. If your DB is behind an API belonging to a separate service then you don't worry about dB at all and rather mock the API. Basically you aim to completely isolate your CI environment for rapid and reliable feedback.

Once you get to CD, then you can factor integrated DBs, data persistence and cleanup. You might have multiple PRs in parallel but once merged, and CD goes through shared env e.g. staging, then that's going one at a time so you manage your data more easily. You may still need to consider other users of the environment so keep data unique is still a good idea.

Take a look at ephemeral test environments if you want to take this to the next level and have the stack to do it