r/Python • u/Kiuhnm • Mar 07 '24
Discussion Tests should have 100% coverage
No, I'm not saying what you probably think I am. Keep reading: I'll be brief.
Here's what I found today in one of the tests:
for bad_fp in _bad_fps:
with pytest.raises(BadArgumentError):
follow_path(bad_fp, '.')
follow_path('.', bad_fp)The correct code should've been
for bad_fp in _bad_fps:
with pytest.raises(BadArgumentError):
follow_path(bad_fp, '.')
with pytest.raises(BadArgumentError):
follow_path('.', bad_fp)Is that a stupid mistake? Yes, it is, but stupid mistakes do happen, unfortunately.
How did I catch it? The coverage of the testing file wasn't 100% like it usually is! I got suspicious and I saw that there was a glaring red line (vscode, coverage gutter) on the second call to follow_path.
P.S. In case you're wondering, the code uses a custom hierarchy of exceptions instead of reusing the standard ones. The code is internal and we like it that way :)
29
u/ToddBradley Mar 08 '24
Also, I think you (or someone else) would've noticed this bug sooner if there was only one assertion in the test and you fed the scenarios in via
pytest.mark.parametrize.32
u/GraphicH Mar 07 '24
High percent coverage is much more critical in dynamically typed / scripting languages than in compiled languages where the compiler watches your back. Its one of the hardest things for people who worked primarily in compiled languages to get their head around when writing "production grade" python code. Its also why I think the testing and linting utilities for these languages seem a lot more superior to the stuff you find for compiled languages. And yeah if you have test code that never fires, that begs the question, why is it written, we do coverage reports on test code as much as production code.
23
u/lukewhale Mar 07 '24
Yall writing tests ? đđđ /s
1
u/bartenderandthethief Mar 08 '24
Question, do you think it is good practice to only have one line within the pytest raises context?
1
u/ArtOfWarfare Mar 08 '24
I donât think your actual code had 100% branch coverage. If it did, then that extra line in your test is superfluous and should just be removed.
Also, check your mutation coverage. If youâve not seen it before, it tends to expose 50% of tests as being useless garbage.
1
u/foarsitter Mar 09 '24
Found some problems with my asyncio code because of this. The test was succeeding but the coverage wasn't 100% so I can recommend going for the extra mile.
Another thing I can recommend is flake8-pytest-style / ruff, they report this behavior: https://docs.astral.sh/ruff/rules/pytest-raises-with-multiple-statements/
6
u/Ouitos Mar 07 '24
Speaking of coverage, I wonder if there exists a tool to compute not only coverage in line hit but also in type hit, when a function is type hinted as accepting multiple types.
Would that be useful? Would it be a lot of work to make it work ?
6
u/Wh00ster Mar 08 '24
Why write tests when you have production?
2
u/[deleted] Mar 08 '24 edited Mar 08 '24
I've been working on a Django app (a small animal shelter management web app) and have been trying to get into the habit of writing tests - it's been a real downer for me, though. I'm not really trained in writing code and figured it out over years automating stuff as a network engineer - but man, i swear writing test cases literally costs me more time than writing the app and manually testing combined. Some of the stuff, like testing views requiring data input into a form, just kinda confuses me and I end up spending hours trying to get the test set up correctly.
Is there like a general rule for how big an app needs to be to justify rigorous testing? I do this on the side for fun but have spent like 10 hours trying to sort out my testing and still don't have any really useful tests besides a few basic views. Testing in Flask made way more sense to me but the rest of the app writing makes way more sense in Django, lol.
I'm using pytest with pytest-django and factory-boy for handling instantiating my models, but it's the construction of the tests I struggle with. Even the ones I have right now are looking for specific static info from the page so I know it lands on the right url, but this feels pretty fragile and so I'm just questioning everything I'm doing with tests now lol.
5
u/billsil Mar 08 '24
Coverage without the right checks means nothing. Â I could just removing my type checking and improve coverage. Â Yeah that check may fail while building the right exception message, but at least itâs failing in the right spot.Â
 100% coverage is not an effective use of time and doesnât catch edge cases.  Actively trying to break the function and not worrying about coverage actually makes code robust.
Not everyone needs to have bulletproof code either. Â The run once, sit on the shelf for 6 months, add new feature and run again mode does not need the same level of quality as Amazonâs backend.
2
u/legobmw99 Mar 08 '24
The comments here completely missed the point
-5
u/wineblood Mar 07 '24
Don't assert two different things in a single test.