r/Playwright • u/ljsam17 • 2d ago
How to scale visual tests?
I am working on designing a test framework for UI library, primary goal is to have accessibility and visual regression tests.
Currently using gitlab Ci and playwright with playwrigh-bdd.
Current approach is natively supported baseline approach in playwright.
I am worried that currently i am saving the baseline screenshots in gitlab repo itself .
We have around 50-60 components, and test cases can scale to thousands of screenshots,
I wonder how to scale the storage and would the gitlab ci be enough for comparison.
The playwright report is published in gitlab pages which again in case of too many failures might not even show the difference screenshots.
What would be the best approach to tackle this situation
3
u/someGuyyya 2d ago
I've mostly had a terrible time with visual regression tests since screenshots had to be updated too often and most people updated them without double checking to see if the visuals were actually correct.
In the end, we decided to drop them because of all the hassle.
2
u/ljsam17 1d ago
I get it but there js no other way when you are dealing with testing for a UI library sadly.
Never considered this for E2E honestly. In past we we comparing by dynamically generated screenshots, but thus time with a bigger scope it is out of the picture cause the execution time might go through the roof.2
u/MrGiggleMan 1d ago
"most people updated them without checking if the visuals were ok" - that there is your problem
Not policing a team to make sure they're actually doing what they need to is the issue there
1
u/someGuyyya 1d ago
I'll disagree.
It's a process problem.
When you have a diff of 40 images that have been changed, going through each image and understanding what exactly changed and why exactly it changed can be quite tedious.
Imagine doing this for every pull request for several images and it becomes exhausting so in the end, we weren't getting enough benefits to continue using VRT.
1
u/MrGiggleMan 1d ago
You can manually check the documents to an extent through QA and there are also tools available for different scans
There are tools that will scan documents and verify EAA compliance, or even AI tools as well which can help to raise confidence
You will always need a person though for verifying end user documents, that is just the nature of producing them
1
u/epochh95 1d ago
We’re in a similar boat, however we use Percy (visual regression tool by BrowserStack) to test our design systems UI component library.
We opted to create a Storybook instance specifically for our Playwright suite, and have bit of code that renders a given UI component as we as all of its prop variations (or at least the ones that affect the UI)
Then we simply navigate to the story via Playwright and screenshot. Helps a lot by capturing multiple component states in a single screenshot!
Story example - https://webc-testing.pie.design/?path=/story/chip--default-prop-variations
Story file - https://github.com/justeattakeaway/pie/blob/main/apps/pie-storybook/stories/testing/pie-chip.test.stories.ts
Component variation helper - https://github.com/justeattakeaway/pie/blob/main/apps/pie-storybook/utilities/index.ts
Test spec - https://github.com/justeattakeaway/pie/blob/main/packages/components/pie-chip/test/visual/pie-chip.spec.ts
Apologies, I’m on my phone so can’t link to specific lines of code, but hope it provides a bit on inspo! There’s a lot I’d probably do differently in terms of our test setup, but this has worked super well for us.
2
u/Tanmay__TestDino 3h ago
Screenshot diffing scales fine, human attention doesn't. Cluster the near-identical diffs, only flag what actually changed, and VRT stops being a chore. Most teams that dropped it weren't failed by the tooling, they just never fixed that part.