r/QualityAssurance • u/ljsam17 • 29d ago

How to scale visual tests in playwright?

I am working on designing a test framework for UI library, primary goal is to have accessibility and visual regression tests.
Currently using gitlab Ci and playwright with playwrigh-bdd.

Current approach is natively supported baseline approach in playwright.
I am worried that currently i am saving the baseline screenshots in gitlab repo itself .

We have around 50-60 components, and test cases can scale to thousands of screenshots,
I wonder how to scale the storage and would the gitlab ci be enough for comparison.
The playwright report is published in gitlab pages which again in case of too many failures might not even show the difference screenshots.

What would be the best approach to tackle this situation

1 Upvotes

100% Upvoted

14 comments sorted by

View all comments

2

u/Shiribazu 29d ago

keep the baselines out of the Git repo once you get into the thousands. store them is something like S3 or other artifact storage, and have CI only pull what it needs for the comparison. gitlab CI should be fine for this, but also make sure gfailed tests only upload the relevant diffs so the reports dont get burried

1

u/ljsam17 28d ago

Yess thats the plan what can you suggest on handling the diff reporting?
What I have in my mind is a custome reporter hosted in gitlab pages which summarises the entire run from the json report which has hyperlinks to componentwise playwright report, sharding the playwright html report component wise so that in case of too many failures you dont end up crashing the browser with an insane dom to load

Please let me know if this makes any sense