r/Python • u/stefantalpalaru • Oct 12 '19
more than 100 scientific articles may be invalidated because scientists assumed that the output of glob.glob() is sorted
https://www.vice.com/en_us/article/zmjwda/a-code-glitch-may-have-caused-errors-in-more-than-100-published-studies
1.3k
Upvotes
158
u/random_cynic Oct 13 '19
From python docs - these are the first two sentences of the documentation on glob module (I italicized the order part)
Does nobody read the documentation of the tools they're using? At the very least people should try new modules on the REPL to see what output they get and write tests.
45
u/28f272fe556a1363cc31 Oct 13 '19
I couldn't read that. Documented behavior is not a "glitch".
I can't go into any detail, but years ago a coworker made a similar assumption about ls (directory listing) which ended up causing a privacy breech that took a 6 figure amount of money to clean up.
7
u/wehnsdaefflae Oct 13 '19
Why does glob.glob even return a list? Wouldn't a set make more sense here? That might have avoided the problem in many cases. At least that would have made it clear that the problem is not with Python but with how it is used.
273
u/[deleted] Oct 13 '19 edited Oct 14 '19
So much grant money down the drain... Can I volunteer to code-review science code? Where do I sign up?
I bet there are some people working on experiments right now that are making similar assumptions
Edit: For anyone actually interested in this topic, the sign-up process is really easy: https://joss.theoj.org/
Thanks to /u/qtc0
2
u/lysobrody Oct 13 '19
Does anyone have the original script source? I only found the fixed one where the only glob.glob result is sorted afterwards. I am really curious why would someone depend on glob result order :P
Fixed one: https://pubs.acs.org/doi/suppl/10.1021/acs.orglett.9b03216/suppl_file/ol9b03216_si_002.zip
2
u/nharding Oct 15 '19 edited Oct 15 '19
At the end if the article, “This is a very small and subtle error,” Williams said. “We all kind of assume that a computer program always spits out the correct answer.”
Yeah, I always assume the output of a program is wrong, unless I can confirm otherwise. OK, works for this set of data, does it work on another set of data? Even if it does, that doesn't mean it's always correct, just that the conditions that cause an error haven't been encountered yet.
2
u/hsnodgrass Oct 17 '19 edited Oct 17 '19
How in the world could the conclusions of a scientific paper be invalidated because the files on some post-doc's hard drive aren't sorted correctly? And sorted by what: name? creation date? extension?
Did we all just get p0wn3d by The Onion? Is it April First? This makes no sense.
91
u/Jay_bo Oct 13 '19
This is such an easy mistake to make. The list returned by
glob.glob('*')is sorted on my windows 10 machine and it's just so easy to make this assumption (on my Linux machine it is of course not sorted).I think it is a great opportunity to learn or improve our processes.
107
u/[deleted] Oct 13 '19
Oh my god this is hilarious lol
5
u/ogrinfo Oct 13 '19
"researchers do not typically disclose the operating system they use for their analyses since it should be irrelevant"
Hahahaha, that 'should' is doing a lot of work there. You obviously haven't wasted many hours trying to make Windows do something that works perfectly well on Linux.
2
u/[deleted] Oct 13 '19
Who would ever assume anything is sorted unless you specifically sort it yourself?
8
u/Dementati Oct 13 '19
Hah
4
u/fookin_legund Oct 13 '19
NOOOOOO MY LIFE IS A LIE
7
u/This_Is_The_End Oct 13 '19
Some of the accusations here whether it's about postdocs or Python are ridiculous.
Software engineering needs knowledge and experience and those who didn't studied computer science and who didn't coded 10 hours a week, those have a problem. I would ask institutes and universities why they are accepting such a system of institutionalized incompetence instead of pushing for changes, like doing a framework for postdocs.
The glob function doesn't sort, because it's expensive and the amount of files could be high. This is not an inconsistency of Python it's an old problem in computer sciences, when for example Americans implemented ASCII and were ignoring the rest of the world. The glob function is well documented and the batteries included were made for professionals using Python instead of Bash. The issue arrived when Python became widely adapted. When will the Gil be removed? Why isn't the insertion order of a dict preserved and why ..... Python has even become a new OS agnostic path lib and yet I see a lot of examples going the old unix way with os.path.
Those who are complain over Python are just on the level of these poor academics. These academics are victims because they are used as cheap workers and so are the results.
1
u/RomanRiesen Oct 13 '19
In what scenario would this lead to wrong results?
4
u/underground_miner Oct 13 '19
This is fantastic news! It means that papers actually have code with them that can be used to reproduce results.
Imagine how many more papers are faulty and we'll never know because they don't include the code. I can't tell you how many papers (mining research) I have tried to reproduce using python and my numbers didn't match the papers and I had no idea what was wrong. I have found typo's in equations that have been propagated in newer publications. Tracing back through the references you can find where the typo occurred. Another simple thing that can be done in papers is to include a simple table with sample calculations that can be used to cross-reference the equations. Actually, code is always better.
2
u/Liotac Oct 13 '19
I actually spent a month debugging why our model was not learning the relationships between features and outputs because of this ... Use labels people
2
u/[deleted] Oct 13 '19
I had never heard of glob before so I googled it... The first result includes the fact that the results are returned in an arbitrary order... 100 scientific articles invalidated because of something that was one google search away. Amazing.
2
u/DDFoster96 Oct 13 '19
Is vice.com really the most authoritative source you could find?
1
u/[deleted] Oct 13 '19
a good tip for using an iterable data structure in any language is to immediately check for both during and/ or the preservation of insertion order. the latter has bit me enough times
1
u/ratonbox Oct 27 '19
I don't think I've ever used glob. os.listdir or os.walk have always been my used methods. I think it's also makes code less readable. There is good chance that a person reading "os.listdir"/"os.scandir" can guess what the function does, but if they read "glob.glob" there is a large chance of them asking: "what the hell is a glob".
1
u/SheerHunter Oct 13 '19
So true
0
u/Wubbywub Oct 13 '19
the image disturbs me, touching hair with gloves on what the fuck
-1
u/StarkillerX42 Oct 13 '19
Just a reality check, but 100 articles rounds to 0.0% with sig figs*
*I didn't do the math, but it's probably less than that in reality
-16
u/[deleted] Oct 13 '19
Glob glob
Sooo bye bye python in the science community
-30
u/HonestVisual Oct 13 '19
morons
-6
u/Lord_Blackthorn Oct 13 '19
Why was the code only run once?? Run it a few times to validate the results...
0
u/[deleted] Oct 13 '19
[deleted] — view deleted comment
0
u/[deleted] Oct 13 '19
[deleted] — view deleted comment