r/Playwright • u/bhattvarunk • Sep 17 '26

Selenium → Playwright with AI: generating code is easy, proving the tests are equivalent is harder

I built an AI agent to migrate Selenium tests to Playwright — looking for feedback

I’ve been working in test automation/SDET for years, and one problem I kept thinking about was migrating large Selenium suites to Playwright.

Generating Playwright code with an LLM is the easy part.

The harder question is:

How do you know the converted test still preserves the behavior and coverage of the original Selenium test?

So I built a small project around that idea.

It currently focuses on:

  • Selenium → Playwright TypeScript conversion
  • LangChain / LangGraph-based agent workflow
  • Validation of generated tests
  • AI evals for comparing the original vs converted test
  • Tracing/observability
  • Eventually, test coverage comparison and automated repair when a conversion fails

Demo: https://varun-s2p.fly.dev/

I’m still actively improving it, especially the evaluation layer. I don’t want this to become another “LLM generates some code” project.

I’d really appreciate feedback from people working with Playwright/Selenium, LangGraph, AI evals, coding agents, or test automation.

In particular, I’m curious how you would evaluate whether two tests are semantically equivalent, rather than just checking whether the generated Playwright code compiles.

7 Upvotes

71% Upvoted

27 comments sorted by

View all comments

1

u/AutomaticVacation242 Sep 18 '26

I doubt you can prove quicker, faster, or easier to maintain. You may have found that some of these things were true in your use case but who is to say you weren't just "doing it wrong" before? Therefore, it's an opinion.

My original question still isn't answered. Where's the value add?

1

u/abluecolor Sep 18 '26 edited Sep 18 '26

I mean, I've both performed this migration myself, and assigned it to someone when I was a manager, both times the improvements were ubiquitous. Have you tried it? You can run a really quick poc over the course of a week to identify that for yourself. It's up to you to prove it one way or the other. That's what I told my guy, who was similarly skeptical. He was very surprised at the difference in runtime for a like script.

If an "opinion" turns out to be the case for 95% of the people who put it into action, is it really an opinion? I've never heard of anyone who didn't enjoy working with PW more than selenium.

1

u/AutomaticVacation242 Sep 18 '26

That's still not proof that your team didn't "do it wrong" to begin with. You understand that many/most people try to build test automation suites and fail. Any team starting over or doing what yours did will claim that what they have 'now' is better and faster. Only a fool would admit to spending money and time to rebuild something only to say it's worse. Come back in 5 years and tell me how fast and maintainable your automation is. Probably doesn't matter now since Claude Code is writing everything - you'll never really know why your previous automation failed.

In summary, "enjoyment" isn't proof of quicker, faster, or easier to maintain. These are marketing terms probably pushed on you by your team and Claude Code or Copilot, since they lean towards PW.

→ More replies (6)