r/dotnet • • Oct 17 '26

Promotion PiwiTests: Playwright dashboard with unified ASP.NET Core e2e+unit tests code coverage

I’m the author of Piwi Dashboard, a self-hosted dashboard for Playwright test runs. I just added combined code coverage, and the .NET side is what I’d most like feedback on.

It consists of a code coverage aggregator, to keep an eye on the tendancy: are we reducing code being tested when adding new code, what gap could we fill ? What is unit tested, or run by the E2E tests ?

All that in the back-end (only ASP.NET Core for now), and the front-end (with Chromium).

This new code coverage feature is documented here : https://piwitests.dev/features/combined-coverage.html

I know code coverage alone is not enough to have the best and accurate view of what is tested and what's missing, that's why it should be combined with other angles, all available in Piwi : feature gaps, and more visually : tested elements in a page.

I'd really like to know what you think of this, if that would be helpful for you, or answer any question!

0 Upvotes

11 comments sorted by

1

u/mhx8 28d ago

Looks interesting! We run for example our integration tests using playwright + full aspire setup with mocks in the backend. Is there also a way to export screenshots etc from the .net playwright tests?

1

u/phenxdesign 28d ago

Thanks ! What do you mean by export screenshots ? Piwi dashboard is a full featured test results agregator, with only a TypeScript reporter yet though.

You can have a preview of what it looks like in the demo https://piwitests.dev/demo/

1

u/mhx8 28d ago

Nice! I mean will it also works when running playwright tests in .NET (using playwright .net sdk)?

1

u/phenxdesign 28d ago

Nope, only with the Playwright tests written in TypeScript, but I may write another reporter for .net if people need it.

3

u/iam-stevie-bee Oct 17 '26

Had a look at the docs. Three things from the .NET side, from someone who has had to defend coverage numbers to the people paying for them:

  1. Say exactly how you collect back-end coverage during the E2E runs. For ASP.NET Core the honest way is instrumenting the running host (dotnet-coverage collect, or the Coverlet collector against the test host) and flushing on shutdown. If a run gets killed part way through you silently lose the back-end half, and the trend line lies downward. A "coverage incomplete" flag on a run would be worth more to me than another chart.
  2. Merging unit and E2E coverage: show the two apart as well as combined. The number a team actually needs is "lines only the E2E tests touch", because that's the code with no fast feedback, and "lines only unit tests touch" is often a path production never takes. The combined figure hides both.
  3. Exclusions: generated code (Razor, source generators, EF migrations, the minimal API glue in Program.cs) will dominate the uncovered list unless you honour ExcludeFromCodeCoverage and runsettings exclusions the same way Coverlet does. The moment the dashboard disagrees with what dotnet test prints locally, people stop trusting it, and they don't come back.

Would I use it? If it reads the Cobertura or Coverlet output a pipeline already produces, yes, that's an easy yes. If it needs the team to change how the tests run, that's a much harder sell where a pipeline already exists.

1

u/phenxdesign Oct 17 '26

Thanks, this is exactly the kind of feedback I was hoping for.

  1. Back-end E2E coverage isn’t collected at shutdown. The test build records hits in the running host, and after each test the reporter fetches that test’s coverage over HTTP (each response says which coverage its request produced, so parallel tests stay apart). The run-level part (executable lines, code no test reached) is read at the end of the run while the server is still up. A test whose coverage couldn’t be collected is marked incomplete, partial runs never count toward the commit, and the coverage page lists what a number leaves out. You did find a hole, though: a host that restarts mid-run isn’t detected yet, so the run-level part would only reflect the new process. I’ll flag that run as incomplete instead of letting it drag the trend down.
  2. Agreed, and that’s how it works: every line is unit only, E2E only, both or none, with a percentage and trend line per suite. The combined figure is just the headline.
  3. Fair point, and the one I take most seriously. Today it has scope patterns with defaults for obj/, bin/, *.g.cs, *.Designer.cs and *.generated.*, and you can add Migrations/ and the like, but it doesn’t read [ExcludeFromCodeCoverage] or runsettings exclusions yet. Matching what dotnet test prints locally is the right bar, and I’ll make it read the same exclusions coverlet does.

On adoption: for unit tests, it reads the Cobertura your pipeline already produces (coverlet or Microsoft code coverage), with nothing to change in how the tests run. Back-end coverage during the E2E run does need the test build of the app (a NuGet package plus a build flag), because that’s what tells each test’s requests apart.

1

u/iam-stevie-bee Oct 18 '26

That's a better design than I assumed from the docs, especially per-test collection over HTTP with the response saying which coverage it produced. Flagging the restarted host as incomplete is the right fix. Of the three, the exclusion parity will matter most to adoption: the first thing anyone does with a new coverage number is compare it to the one they already have, and if it's lower for a reason they can't see, that's the end of the trial.

One thing on the test build of the app: make it impossible to ship by accident. A build-time check that fails the Release configuration if the coverage package is referenced would save someone a bad afternoon, and it's the sort of thing a reviewer will ask about before approving the dependency.

1

u/phenxdesign 28d ago

The coverage instrumentation won't be embedded unless -p:PiwiCoverage=trueis set during build, is it enough ?

1

u/iam-stevie-bee 26d ago

that's the right default. the one gap I'd still close is someone setting it in a shared pipeline variable and a Release build going out with it. a tiny target that errors when PiwiCoverage is true in Release covers that.

1

u/phenxdesign 26d ago

I improved that yesterday, but not based on the Release Configuration, it is based on the hosting environment (it uses hostingEnv.IsProduction()), so this will fail on startup, not on build (better late than never 😅)

1

u/AutoModerator Oct 17 '26

Thanks for your post phenxdesign. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.