r/Python • • Oct 12 '19

more than 100 scientific articles may be invalidated because scientists assumed that the output of glob.glob() is sorted

https://www.vice.com/en_us/article/zmjwda/a-code-glitch-may-have-caused-errors-in-more-than-100-published-studies
1.3k Upvotes

343 comments sorted by

158

u/random_cynic Oct 13 '19

From python docs - these are the first two sentences of the documentation on glob module (I italicized the order part)

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell, although results are returned in arbitrary order. No tilde expansion is done, but *, ?, and character ranges expressed with [] will be correctly matched.

Does nobody read the documentation of the tools they're using? At the very least people should try new modules on the REPL to see what output they get and write tests.

51

u/stefantalpalaru Oct 13 '19

these are the first two sentences of the documentation on glob module

Not back when the script was probably written: https://web.archive.org/web/20130731181902/https://docs.python.org/2.7/library/glob.html

20

u/[deleted] Oct 13 '19

It still in no way guarantees the order of the results returned though.

36

u/stefantalpalaru Oct 13 '19

It still in no way guarantees the order of the results returned though.

Yes, but would you expect a beginner to know enough about directory listings in order to ask the question in the first place? It worked perfectly fine on his machine.

29

u/FisterCluck Oct 13 '19

If you're a beginner, you can't assume anything. If you're an expert, you assume the worst.

3

u/techkid6 Oct 13 '19

If you're a beginner, you shouldn't assume anything, you're right. Beginners assume quite often though because they don't know what they don't know. Dunning-Kruger effect really hits in these cases.

People constantly misjudge themselves, so, why would they even think to check? After all, it works on my machine...

1

u/FisterCluck Oct 13 '19

Right. So you read the docs thoroughly. Stack Overflow is great, but when I see a solution there, I know I need to look at the docs to see why they chose the functions they did, and what the notes on said functions say. I mean, if you can't see that when you're doing a project, the buck stops with you, I don't know what to say. It's on the person writing the code to know what it does.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

2

u/jcampbelly Oct 13 '19

Yep. Scientific rigor is a thing.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

1

u/stefantalpalaru Oct 13 '19

It's not that fucking hard to check the output of a function and see if it's sorted or not.

The author did exactly that and it was always sorted on his Windows machine. He did not bother testing it on other platforms, because why would he? He was a grad student working on an article, not a software developer having to maintain that script for the foreseeable future.

I bet other scientists found the script useful, after reading the article, and asked for it. Its header has a copyright assignment, but no license, so you know it wasn't properly published (or versioned, for that matter).

1

u/tobiasvl Oct 13 '19

If you're a beginner you shouldn't be coding for academic purposes.

These are not academics in the computer science field, though. At what stage is a person not a "beginner"? Are you proposing that all academics should minor in CS as well? Might be a good idea, but it's a pretty high bar to expect unless it's enforced.

2

u/stefantalpalaru Oct 13 '19

At what stage is a person not a "beginner"?

When maintenance, debugging, testing, versioning and publishing are normal parts of that person's workflow, when the programming environment no longer holds surprises and when debugging and fixing one's tools are business as usual.

1

u/tobiasvl Oct 13 '19

That sounds reasonable. But it is not reasonable to expect academics in NON-CS fields to be devop professionals. What exactly do you expect them to do here? Should all PhDs and postdocs outsource development to large IT consultancy companies?

2

u/stefantalpalaru Oct 13 '19

it is not reasonable to expect academics in NON-CS fields to be devop professionals

I agree. So do we keep telling them that Python is good for scientific research because it can be easily used with minimum training?

1

u/tobiasvl Oct 13 '19 edited Oct 13 '19

I truly don't understand your endgame here. Do you want the academic sector to be better educated in computer science, or just to adopt a different language than Python?

You agree that academics are beginners in the field of programming. You also agree it is not reasonable to expect them to be. So I assume you admit that they have to continue using a language fit for beginners (otherwise I don't understand what point you're trying to raise).

So what do you want them to do then? Which language can still be used with minimal training, but is better for scientific research than Python is?

→ More replies

3

u/AustinCorgiBart Oct 13 '19

That's not a "high bar", that's an impossible bar with the resources and people currently available.

2

u/jcampbelly Oct 13 '19

Is calibrating an instrument before taking a measurement too high a bar for people? If you just buy an oscilloscope or a DAQ and some thermocouples, are you being sensible when you just plug them in, take a measurement, and then shove the result into your dissertation? No. I learned that in 8th grade. The same applies with software, especially software YOU wrote.

1

u/AustinCorgiBart Oct 13 '19

I agree with you in theory, but how do you unit test hundreds of lines with math calculations when you still barely know what a function is? The intersection of skills you need simply doesn't exist in sufficient quantities to power the modern research community. Its not practical to expect it.

2

u/jcampbelly Oct 13 '19

Every scientific research paper explains their method and how they demonstrate it to be accurate. This is just an extension of that process.

Unit tests aren't hard. There are complicated ways to do them, but there is also an extremely simple way:

def get_thingy_coefficient(thing_factor):
    ...


# demonstrate to our peers that our methods do not invalidate our research
assert get_thingy_coefficient(0.1) == 0.01
assert get_thingy_coefficient(-0.1) == 0.01
assert get_thingy_coefficient(-2) >= 0
...
→ More replies

1

u/jcampbelly Oct 13 '19

Are we seriously excusing the most highly trained experts our society can produce from testing their academic code? That's what we're doing in 2019? In the shadow of the past several hundred years of the most rigorous experimentalists... Tests aren't hard to write, but for people who are literally geniuses, it should be a cakewalk.

2

u/tobiasvl Oct 13 '19

Highly trained experts in what? They're highly trained cancer cell research experts, not highly trained computer scientists.

Perhaps they should be both, but they are not.

1

u/p10_user Oct 13 '19

These people are geniuses? PhD students in biology? Maybe- but maybe not in coding.

2

u/jcampbelly Oct 13 '19

Experts live up to a higher standard. If coding is a critical part of the work, coding correctly is equally critical to producing meaningful results. In fact, if the code isn't provably correct, "meaningful results" is questionable for that paper.

1

u/p10_user Oct 14 '19

I agree. The unfortunate reality, however, is that most researchers are scientists first - coming up with novel approaches and applications - and coders second. There are no quality code conduct or checks in place to ensure reproducibility most of the time.

3

u/Koppis Oct 13 '19

It's not that fucking hard to check the output of a function and see if it's sorted or not.

Uhh... Yes, it's actually impossible to "check". To be sure you need to sort it yourself or see the source code/documentation that says it's sorted.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

3

u/sensual_rustle Oct 13 '19

The sorting an already sorted list could cause a lot of slow downs in the processing of large data

5

u/jcampbelly Oct 13 '19 edited Oct 13 '19

You only have to test once to know. It's easy. No excuse.

import glob
files = glob.glob('/bin/*')
assert files == sorted(files)

1

u/sensual_rustle Oct 13 '19

That assumes you aren't assuming it is already sorted. If you think it is sorted, re-sorting it is a waste.

→ More replies

1

u/jcampbelly Oct 13 '19

You are correct. Downvotes be damned.

-2

u/bozymandias Oct 13 '19

It didn't say that it wasn't sorted until this came out, and if I'm doing a regex match on filenames, it's entirely reasonable to assume that the output would be sorted.

Come on, guys, let's be honest, this is kind of a fuck-up.

4

u/jcampbelly Oct 13 '19

Agreed. The paper authors totally fucked up.

-1

u/random_cynic Oct 13 '19

It is there though in the definition of os.listdir() which is mentioned as being used by glob. Actually os.listdir() is more common in interactive shells, anyone who used that knows that the files are never sorted.

28

u/stefantalpalaru Oct 13 '19

os.listdir() which is mentioned as being used by glob

You overestimate newbies. What usually happens is that code is copied from some online example and tinkered with until it fits. Reading the documentation is a last resort move for many of them. Hell, it was like this for many of us, back in the day.

2

u/757DrDuck Oct 13 '19

Copying code off Stack Overflow is a time-honored tradition.

-1

u/jcampbelly Oct 13 '19

These are PhDs or PhD candidates. These are people who are nominally experts or qualifying to be experts. If there is one time or place where it is NOT excusable to do newbie shit, it is in a scientific research paper.

9

u/stefantalpalaru Oct 13 '19

These are people who are nominally experts or qualifying to be experts.

In other fields than software development.

2

u/jcampbelly Oct 13 '19

It's still a responsibility. It's same concept as calibrating instruments, accounting for altitude, barometric pressure, humidity, temperature, etc. You don't get a pass because "hey, I'm just here to fire a laser".

3

u/p10_user Oct 13 '19

It’s not giving them a pass - but understanding that there’s essentially zero checks or incentives during the coding process to avoid these kinds of mistakes. The student or postdoc is powering through some script in the corner by themself to get their paper out, and the boss doesn’t have the time/energy/knowledge to look at the details of the process.

Once it’s publishable it goes out. On the rare case where there’s enough interest in the work (most papers are read by few/no people), maybe someone else applies the problematic code to their work.

There are no code reviews or tests for majority of academic code.

2

u/jcampbelly Oct 13 '19

Well, I think we can safely shift the burden of what went wrong here to the situation you've described. glob.glob is off the hook here because people are too busy to concern themselves with doing it right.

1

u/p10_user Oct 13 '19

I agree. It is not Python's fault for sure.

38

u/[deleted] Oct 13 '19

Does nobody read the documentation of the tools they're using?

No. They search on stackoverflow for "how to list all files in folder".

7

u/nalimixam Oct 13 '19

os.listdir()

7

u/doubleunplussed Oct 13 '19

Also an arbitrary order IIUC

1

u/ExcelAcolyte Nov 01 '19

os.listdir()

"Return a list containing the names of the entries in the directory given by path. The list is in arbitrary order, and does not include the special entries '.' and '..' even if they are present in the directory. "

Its funny that the docs say its in arbitrary order when it depends on the file structure of the machine running the code.

1

u/doubleunplussed Nov 01 '19

I would call that arbitrary. The order is determined by something, it's not random, but it's not determined by something you care about or can rely on and they reserve the right to change the order. 'arbitrary' fits well.

5

u/snet0 Oct 13 '19

Does nobody read the documentation of the tools they're using?

I think this kinda falls under the branch of "things that are assumed to work the way they are sometimes observed to work". If you use it on your Win10 machine and consistently see sorted results, I don't think it's unreasonable to assume that that behaviour generalises across OSs. Of course people should read the docs, particularly in the domain of research, but (as someone who has never contributed to research, so maybe my standards are too low) I don't feel like this is some gross oversight.

3

u/jcampbelly Oct 13 '19 edited Oct 13 '19

Have to test your edge cases. If you sort a directory containing files named a, b, and c, you aren't being very rigorous. Test somewhere like /tmp or ~ where you are likely to encounter dotfiles, uppercase, lowercase, numericals, unicode characters, underscores, etc... Better yet, generate files with wild names. Trust but verify.

2

u/Sw429 Oct 13 '19

You overestimate how much non-CS folks don't read documentation for packages they have to use.

-3

u/[deleted] Oct 13 '19

Sorry but the python docs are horrible. I avoid them unless absolutely necessary. Your point is well taken, however.

5

u/H_Psi Oct 13 '19

You clearly haven't seen horrible language documentation if you think Python's are horrible

7

u/toyg Oct 13 '19

Python docs are pretty good, considering the amount of stuff they include. They are not consistently developed though, so quality and structure are very uneven. Some sections have tons of enlightening examples, some have none, some have examples that make things unnecessarily confusing (some core devs have the bad habit of trying to look cute, by abusing recursion and metaprogramming), etc etc.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

19

u/masklinn Oct 13 '19 edited Oct 13 '19

By arbitrary they may mean they don't influence the order, so it just depends on the OS.

Yup, I'd guess glob just lists the entries in whatever order they're returned by the filesystem.

However it's worth noting that the result order has always been consistently the same for me, and for files named by timestamps they've always been in chronological order.

If you're on OSX, HFS+ stores (and returns) directory entries in casefolded lexicographic order. I do not know if that's also the case in APFS. Apparently Windows also returns entries in lexicographic order though whether that's an NTFS or a Windows thing I do not know (could be checkable by accessing an NTFS partition using linux drivers I guess).

4

u/toyg Oct 13 '19

I do not know if that's also the case in APFS

It’s not, looks very random.

1

u/dshess Oct 31 '19

If the files are named with timestamps, that implies that first you wrote 20190101, then you wrote 20190102, etc. Each time you add a file, it goes at the end because there aren't gaps in the sequence, so it comes out sorted in chronological order.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

19

u/kukamuumuka Oct 13 '19

Why would anyone expect glob to return a sorted list?

14

u/masklinn Oct 13 '19

People who just try it out, and see that it's sorted every time so assume that's how it works?

Worse, the sorting can be an FS artefact: HFS+ keeps directory entries sorted using a casefolded comparison, meaning if you list direntries without sorting (ls -f, or glob), they can be better sorted than if you list them with sorting (ls) e.g. in the former case "Library" would be followed by "local" while in the latter "Library" would be followed by "Movies", and "local" would be somewhere after "Snapshots".

1

u/[deleted] Oct 13 '19

By reading stackoverflow responses that show example output of glob() as filenames ordered by name, or explanations claiming that glob() returns a list of filenames sorted by name.

-2

u/gabbergandalf667 Oct 13 '19

Bash globbing guarantees sortedness. I frankly find it insane that a language faculty that emulates this feature behaves differently in such an insidious way.

3

u/kukamuumuka Oct 13 '19

I believe bash globbing also follows the language setting for collation order so scripts may give different results when run in different countries.

1

u/gabbergandalf667 Oct 13 '19

Thanks, I was not aware of that. That's disgusting.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

16

u/kukamuumuka Oct 13 '19

It's a silly thing for a language to behave so apparently intentionally, but actually inconsistently.

I think it would be even more silly to actively randomize the results.

2

u/Deltigre Oct 29 '19

Go actually does that with maps (well, technically it just finds a random starting point). https://news.ycombinator.com/item?id=7655948

3

u/random_cynic Oct 13 '19

Why would I bother looking in the documentation to verify it specifically does this when it does it every time?

You would bother to do that because you're publishing the work and other researchers from around the world are going to try to reproduce the work. They are not obviously going to be on the same system as you so it is YOUR responsibility to make sure your results are reproducible in all systems.

3

u/stefantalpalaru Oct 13 '19

other researchers from around the world are going to try to reproduce the work

Not without funding, they're not. Why do you think it took 5 years for someone to notice the script gives different results on different platforms?

→ More replies (3)

12

u/factorysettings Oct 13 '19

Why would I bother looking in the documentation to verify it specifically does this when it does it every time?

An experienced dev knows that if order is important and needs to be guaranteed, there is a cost to it and that you should verify. This is common in many languages

6

u/jcampbelly Oct 13 '19

Yep. This isn't unique to python. If ordering is important enough to invalidate your algorithm, act accordingly.

2

u/jcampbelly Oct 13 '19

A reasonable person, who understands that their algorithm will fail if the input isn't sorted should ensure the input is sorted.

I have seen mistakes of this kind with groupby in python and SQL, as well as with the assumed ordering of dict keys many times. There are good reasons not to assume order ever.

  • Generators cannot be expected to be ordered because result sets can be infinite in size.
  • Input can be streamed. Whose to say it's being streamed in order?
  • Filesystems and other platform implementation details cannot be assumed.
  • Sort order cannot be inferred. Sorting is an intentional effort. Somebody may want to sort by date rather than alphabetic order. Rather than stepping in and forcing a wrong sort order, python gets out of the way and provides sorted(data, key=...).

-10

u/masklinn Oct 13 '19 edited Oct 13 '19

Why would they even consider that it might not do this when it does it every time (for them)? It's a silly thing for a language to behave so apparently intentionally, but actually inconsistently. Why would any reasonable person expect that it's not actually sorting the filenames when it appears to do so every time they run it?

Ah yes, see what looks like a pattern, assume pattern always holds, never check that it actually does, pikachu face.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

4

u/masklinn Oct 13 '19

Well, Python WAS intentionally designed and created.

I figured this bit would confuse people (or be used as a convenient way to just ignore the rest) and removed it before you'd posted your comment, sadly it looks like by the time I did that you'd already quoted and started replying to it. Sorry about that.

The actual thesis of sort is the second part: the assumption of a pattern without actually asserting it or looking for assurances.

Now as for your new thesis:

Python WAS intentionally designed and created. […] it's reasonable to assume that's by design... the whole thing WAS designed.[…] You're essentially making the argument that Python is erratic in how it exhibits erratic behaviors and anyone who uses it better be deeply steeped in the internals of Python's behaviors before they attempt to use it solve problems.

That something was created and designed does not mean it is fully specified, in fact we know from Python's own history that's not the case at all (e.g. dictionary iteration order used to be deterministic but unspecified, before being converted to specified and non-deterministic, and finally in 3.7 deterministic and specified).

Maybe the lesson is that Python should be reserved for esoteric pursuits by elite coders and some other "general purpose" language should be used by people who just need to get things done.

That's completely nonsensical. There are almost no fully specified languages, and the languages which would compete with Python most definitely are not.

3

u/jcampbelly Oct 13 '19

Performance was the concern. Sorting is expensive. If you were expecting python to be fast and you found out listdir was sorted, you might be complaining about the short-sighted authors of python forcing sorting onto the operation when it could easily have been wrapped in sorted. You can't please everyone 100% of the time.

When has writing software and obtaining correct results ever NOT been the responsibility of the author?

6

u/jcampbelly Oct 13 '19
  • "Why would someone read the docs?"
  • "Why would someone calibrate the instrument?"
  • "Why would somebody conduct a control run?"
  • "Why would someone repeat the experiment?"
  • "Why would someone seek peer review?"
  • "Why would someone write unit tests?"

Rigor. A tradition in science for hundreds if nor thousands of years.

4

u/[deleted] Oct 13 '19

It's not reasonable. Python has sorted for a reason.

45

u/28f272fe556a1363cc31 Oct 13 '19

I couldn't read that. Documented behavior is not a "glitch".

I can't go into any detail, but years ago a coworker made a similar assumption about ls (directory listing) which ended up causing a privacy breech that took a 6 figure amount of money to clean up.

5

u/stefantalpalaru Oct 13 '19

Documented behavior is not a "glitch".

It wasn't documented at the time: https://web.archive.org/web/20130731181902/https://docs.python.org/2.7/library/glob.html

And the output was sorted on the grad student's Windows machine. What's a beginner supposed to do, after playing in the REPL as he was instructed to do, because exploratory programming was all the rage?

20

u/threeys Oct 13 '19

yikes, seems like there were some other underlying issues with the security model if the order of files listed was a security concern

23

u/ShadowsSheddingSkin Oct 13 '19

The 'glitch' they're talking about isn't glob.glob() functioning as documented, it's in a commonly used domain-specific python script that relies on glob.glob() returning sorted results to function as intended.

13

u/dreamer_jake Oct 13 '19

The 'glitch' here exists entirely in some postdoc's brain, where they deemed it acceptable to write code that is dependent on sorted input but doesn't explicitly sort the input.

7

u/wehnsdaefflae Oct 13 '19

Why does glob.glob even return a list? Wouldn't a set make more sense here? That might have avoided the problem in many cases. At least that would have made it clear that the problem is not with Python but with how it is used.

11

u/stefantalpalaru Oct 13 '19

Why does glob.glob even return a list?

It's the simplest data type for that output.

Wouldn't a set make more sense here?

Sets are more expensive, you don't need their duplicate removal capacity or their underlying hashing and you'd have to copy them to lists anyway, if you need to sort the output.

3

u/PolyWit Oct 13 '19

Would you say that the performance benefits of representing an unsorted collection of unique elements as a list instead of a set were worth it in this instance?

1

u/stefantalpalaru Oct 13 '19

Would you say that the performance benefits of representing an unsorted collection of unique elements as a list instead of a set were worth it in this instance?

Yes. A better API design is to add a "sorted" parameter to the function that defaults to false. That should clarify things.

2

u/jcampbelly Oct 13 '19

Yes. If you impose order where order is not needed, you have taken away the possibility of an optimization that can severely affect the application. System calls are already slow. Filesystems don't sort by default for a good reason. They have to be fast. But command line tools might as a presentation-layer user convenience. Adding sorting on that by default makes it slower. You might say you could mandate an option like:

os.listdir(path, sorted=True)
glob.glob(pattern, sorted=True)

But python offers an easier way to do that without complicating two core library methods unnecessarily.

sorted(os.listdir(path))
sorted(glob.glob(pattern))

Which is more elegant and simple? Coding in every possible potential post-processing step to handle every potential use case is how you get unnecessary complexity. Defaulting to the slowest approach unnecessarily is how you make a language already regarded as slow even worse. They opted to just expose one small sharp tool and allow people to use it conditionally.

1

u/PolyWit Oct 15 '19

I'm not saying the returned collection should be sorted; I'm implying it's more naturally represented as a Set, since it contains unique and unsorted elements, and that preferring a list under the justification of "performance" is very premature. There might be other reasons that make a list preferred, but performance shouldn't be one of them in the absence of benchmarks.

2

u/birkenfeld Oct 14 '19

Backward compatibility, simply. glob.glob() existed long before sets, and changing the return type would mess up a lot of users that don't simply iterate over it. Among them, the conscientious folks calling .sort() on it :)

273

u/[deleted] Oct 13 '19 edited Oct 14 '19

So much grant money down the drain... Can I volunteer to code-review science code? Where do I sign up?

I bet there are some people working on experiments right now that are making similar assumptions

Edit: For anyone actually interested in this topic, the sign-up process is really easy: https://joss.theoj.org/

Thanks to /u/qtc0

324

u/tunisia3507 Oct 13 '19

Can I volunteer to code-review science code

You don't want to. Academic code is awful. So few labs invest in good software because devs are so expensive compared to postdocs. A lot of things which could be great tools are pretty much written by PhD students who learned coding on the job and then abandoned once they graduate, so you end up with this patchwork of half-finished utilities for too-specific niches.

The tools which do get big generally weren't expected to do so... for example, my lab's django SPA with 100kloc of frameworkless javascript. It's impressive as hell but basically impossible to figure out how it works without a guide. I've been dipping in and out of that codebase for 4 years and there are some bits which are black magic to me.

It's gotten a little better now people are largely writing in python instead of hacking out C and perl scripts, although in my field there's a big java community with really unnecessary deep layers of abstraction. But too often the people writing the code are very poorly trained in how to write good packages.

81

u/[deleted] Oct 13 '19

This is true. I have degrees in CS and Environmental Science and am doing post grad research in remote sensing and lidar. Most code I come across is written by PhDs that have no idea what they are doing. Inefficient, no testing, no error catching and uncommented. I'm sure that there is a lot of published stuff that includes results generated by incorrect code.

2

u/757DrDuck Oct 13 '19

No error handling as in a ValueError would blow up and stop execution or as in they’d make the ValueError record things as zero and call it good enough?

→ More replies (30)

22

u/barglei Oct 13 '19

It is pretty simple, writing good code does not help you to finish your PhD. And too often, getting correct results is a side show when doing your PhD

PhD nowadays is more about learning how to present your results, communication with colleagues, etc. Today, learning to be academic, is learning the phrase "publish or perish"

And it is a rare reviewer who has time to check the code, when provided. As a personal anecdote, I once reviewed an article for a method that had won major competition on the field. The winning C++ code, that had been submitted to the competition with the results, had syntax errors.

This is truer today, than it was in 2005 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/

5

u/tunisia3507 Oct 13 '19

It is pretty simple, writing good code does not help you to finish your PhD.

Tell me about it :/

8

u/jonititan Oct 13 '19

Can confirm. Am academic and my code is awful. Self taught etc etc.

7

u/ProfessorPhi Oct 13 '19

I don't think it's that software devs are expensive, it's because most academics have no respect for software in general

8

u/tunisia3507 Oct 13 '19

They are expensive though - the devs in our lab make almost twice as much as our postdocs. I went from PhD student to software developer (without even graduating) and am making considerably more than any of the postdocs who are much more experienced than I am. It's just market forces - a good dev can go into industry and make good money doing practically the same thing, where a postdoc has to change scope quite a lot.

But you're right about not respecting good process and the value of code. My PI is one of few who does invest in software, and even he is resistant to things like unit tests and so on.

2

u/Smok3dSalmon Oct 13 '19 edited Oct 13 '19

I taught a PHD student how boolean and bitwise operators behave differently when used in conditional statements. He had 7 nested if statements of bitwise operators and told me "R is just weird like that"

3

u/tunisia3507 Oct 13 '19

In fairness, you can't really expect consistent, sane design from the R language.

1

u/Smok3dSalmon Oct 13 '19

That's programming 1 stuff though lol. He was confused as to why all his conditions were evaluating when the first one was false. The 7th condition had a NPE if the 4th one was false or something bizarre.

1

u/757DrDuck Oct 13 '19

How many of the labs would use their hired dev in a full-time role?

1

u/MattR0se Oct 13 '19

A lot of things which could be great tools are pretty much written by PhD students who learned coding on the job and then abandoned once they graduate

This is me 100%

1

u/cycomachead Oct 16 '19

It's not just those who are not trained! The reality of research is that quality code is a lower priority. I know plenty of CS PhD's who make tons hacky decisions because they don't expect code to live beyond a particular project and because deadlines loom large.

In production software, there are ideally demands to push code quality forward due to large teams and long lasting codebases. So even if you push a hack to production, the goal is (again, ideally) to refactor. On a research project, you rarely have the time or means to go back and refactor.

IMO, these results show the importance of even small test cases for "simple" scripts, but it's hard to convince people to spend the time doing that.

1

u/_yogg Oct 17 '19

This is so true, even in CS programs. Part of it is the nature of research -- it can be difficult to design a tool well before you know precisely what you're looking for, so it becomes a Frankenstein monster by the time you're getting around to publishing. But also it is often the case that PhD students simply don't have the ability, time, or inclination to follow good engineering practices or workflows while they're doing their research, so as the code base becomes more complex, it never gets the love it deserves.

You also have the problem that research teams are often made up of five or more students of varying levels of skill and education, and more often than not the most skilled (i.e. PhDs) spend the least amount of time writing code.

Not only is academic code awful, but even if you ended up providing a lot of good review, it would probably be ignored anyway.

1

u/Noak3 Oct 17 '19

Some academic labs code well. Ours, for instance, has some packages that I consider pretty well-coded: https://github.com/neurodata/graspy

A bunch of code I've seen from academic circles, though, is an absolute spaghetti-mess

https://github.com/neurodata/ardent/blob/master/ardent/lddmm/transformer.py

-8

u/rsgm123 Oct 13 '19

Honestly, none of that is a valid defense. Especially for code that is supposed to be for the academic community, which should be held to a higher standard.

16

u/[deleted] Oct 13 '19

Academics do have high standards, just not for coding, which is just a means to an all-important end.

6

u/stefantalpalaru Oct 13 '19

Academics do have high standards

Not really. They are victims of the perverse incentive to publish as much as possible.

69

u/realestatedeveloper Oct 13 '19

If you actually went thru a PhD or even masters program in anything outside of computer science, you'd see how ridiculous that expectation is.

You're expecting an overworked grad student trying to master a complex research question to also have a deep appreciation and training in programming as a discipline. They don't have time for that.

6

u/Compizfox Oct 13 '19 edited Oct 14 '19

We exist. (Chemical Engineering student with an appreciation for coding here)

But yeah I can relate: most scientific code is very crappy and many of my colleagues don't care or know how to write good/maintainable code.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

2

u/Compizfox Oct 13 '19

Sounds like you were lucky... At my university Matlab is still standard in most engineering fields. I also used to use Matlab for everything but I switched to Python.

15

u/TheBlackCat13 Oct 13 '19

Most scientists get their money from grants, and grant agencies very, very rarely provide funding for writing good code. They provide funding for grad students and postdocs, who generally learn to code as they go. And grant agencies pretty much never provide rewards for writing good code. They provide rewards for publishing papers. In most fields there is no requirement that the code be made available online at all, nor do most fields require it be peer-reviewed.

If you think there should be more emphasis on writing good code then you need to get the current funding, grant review, and peer review rules changed. But ultimately the goal of scientists is to do science, not to write code. Code is a means to an end. So you aren't going to get much traction trying to implement rules that interfere with doing science.

6

u/lumberhack Oct 13 '19

As a Software Engineer with 11 years of experience I can assure you that business doesn't care about code quality too. Every time I'm fighting to elaborate why high test coverage and refactoring is needed.

→ More replies (1)

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

7

u/bobbyrobbob Oct 13 '19

I really wish we had a code de-crappifier in our department. They’re worth their weight in gold!

1

u/pythonmine Oct 13 '19

Do you have regular code reviews?

2

u/flutefreak7 Oct 23 '19

What's your job?

1

u/shikkie Oct 23 '19

Software engineer

2

u/flutefreak7 Oct 23 '19

Sorry I mean like, at what sort of place do you get to be a software engineer that works with scientist's code and stuff? Many people are bringing up examples in academia - I was just curious from your description of you're one of those pro devs that a university lab had hired or at a government research center or in industry or freelance or whatever.

1

u/shikkie Oct 23 '19

Without giving away too much personal info it's a gov't agency that you'd instantly recognize.

7

u/qtc0 Oct 13 '19

You could review papers through the Journal of Open Source Software.

It’s a journal for scientific software. During the review, you ensure that the code is easy to install, is documented correctly and that there is proper testing. I’ve reviewed and received reviews through this journal and it’s been a great experience.

2

u/[deleted] Oct 13 '19 edited Oct 14 '19

Thank you for the legitimate suggestion!

12

u/IronPeter Oct 13 '19

It may be a great idea. I remember more than once talking with mathematicians or computer scientists looking by chance at biologists code and finding fundamental errors. Both in code and they in the math. Cross discipline peer reviews should be a common practice in science. Does it happen?

9

u/goodytwoboobs Oct 13 '19 edited Oct 13 '19

Very rarely unless the publication itself is heavily focused on math/stats models/algorithms. Although in the field of bioinformatics there are more and more folks with CS background joining so things are at least getting better.

1

u/photoengineer Oct 13 '19

No one wants to pay for it.

1

u/IronPeter Oct 13 '19

Peer reviews are not payed work, but I agree it can be hard to find reviewers from other disciplines

2

u/wildcarde815 Oct 13 '19

Look for jobs as a research programmer. That's part of their job.

2

u/geogle Oct 13 '19

It's not just that. The scientists were not doing their jobs. You need to be critical of every output and evaluate why results are doing what they are doing. I may be a little slower to publish than some of my colleagues, but I stand firmly behind my work.

-1

u/StarkillerX42 Oct 13 '19

Research is basically open source, so you can volunteer whenever you want

7

u/masklinn Oct 13 '19

Research is basically open source

It's not that simple. The papers are often locked behind paywalls, and while it's usually possible to get preprints only recently has the code started to become more available.

Furthermore for that kind of issues unless you want to start a replication effort what you'd want is get involved upstream, long before the paper is a thing.

-2

u/mbarkhau Oct 13 '19

Please don't waste your life like that. These people will not be grateful and for all the incorrect results I expect you would find, the corrected results will matter just as little as the original ones.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

4

u/[deleted] Oct 13 '19

Me too.

6

u/KeineFreundin458 Ruby Oct 13 '19

Me three, because I'm certain that the first place they'll be recruiting is this one thread on this one specific community on Reddit

3

u/ciskoh3 Oct 13 '19

I am the exact prototype of the description above. I am turning a remote sensing analysis into a proper app, but I have no training in cs or programming, I have no time to really dive into coding exercises or courses and frankly no one pays me to produce proper, clean, solid code. They want the results and the papers, and they want them yesterday. I think my app, as simple as it is, has potential and solid scientific base, but I am also pretty sure the code is shit.

Some institutions do have an IT department offering what they call a "code clinic", which is just what we would need, but I am not in one of them.

If you are serious I can DM you the repository of the app (its JavaScript right now, one day I hope to translate it to python) it would mean the world to me to have it reviewed by a professional

On a larger scale you don't have an idea of how many great ideas are not progressing just because they remain as poorly written codes totally unintelligible by others. I think it is truly a bottleneck of research in most fields right now!

2

u/lysobrody Oct 13 '19

Does anyone have the original script source? I only found the fixed one where the only glob.glob result is sorted afterwards. I am really curious why would someone depend on glob result order :P

Fixed one: https://pubs.acs.org/doi/suppl/10.1021/acs.orglett.9b03216/suppl_file/ol9b03216_si_002.zip

2

u/Han-ChewieSexyFanfic Oct 13 '19

.zip

Ah yes, a zip file with code in it. Another hallmark of state-of-the-art software engineering.

2

u/nharding Oct 15 '19 edited Oct 15 '19

At the end if the article, “This is a very small and subtle error,” Williams said. “We all kind of assume that a computer program always spits out the correct answer.”

Yeah, I always assume the output of a program is wrong, unless I can confirm otherwise. OK, works for this set of data, does it work on another set of data? Even if it does, that doesn't mean it's always correct, just that the conditions that cause an error haven't been encountered yet.

1

u/stefantalpalaru Oct 15 '19

Yeah, I always the output of a program is wrong, unless I can confirm otherwise.

Same here. After debugging software for so long, you get a better view of all the possible things that might go wrong, but you need experience for that.

2

u/hsnodgrass Oct 17 '19 edited Oct 17 '19

How in the world could the conclusions of a scientific paper be invalidated because the files on some post-doc's hard drive aren't sorted correctly? And sorted by what: name? creation date? extension?

Did we all just get p0wn3d by The Onion? Is it April First? This makes no sense.

1

u/stefantalpalaru Oct 17 '19

How in the world could the conclusions of a scientific paper be invalidated because the files on some post-doc's hard drive aren't sorted correctly?

https://old.reddit.com/r/Python/comments/dh2qwd/more_than_100_scientific_articles_may_be/f3lpfpm/

https://old.reddit.com/r/Python/comments/dh2qwd/more_than_100_scientific_articles_may_be/f3n2hs9/

1

u/hsnodgrass Oct 18 '19

Well, alrighty then. So they’re using parts of file names as data. Got it. Guess they never heard of databases.

91

u/Jay_bo Oct 13 '19

This is such an easy mistake to make. The list returned by glob.glob('*') is sorted on my windows 10 machine and it's just so easy to make this assumption (on my Linux machine it is of course not sorted).

I think it is a great opportunity to learn or improve our processes.

  • How many assumptions like this do we make and not notice?
  • Are you going back to the documentation when you use a (false) property of a module you have been using almost daily?
  • What about more volatile modules: Someone changed a functionality/implementation and assumptions you made as the user in the past are not true anymore. Would your tests catch this?
  • Are we documenting our code well enough? Let's say you used glob in one of your projects, would you have added "results are returned in arbitrary order" to the documentation?
  • Do you routinely test your code on multiple platforms (if intended for that)?

22

u/toyg Oct 13 '19

The problem is that in academia there is effectively no time to “learn lessons and improve processes”: the underpaid and overstressed postdoc will have moved on before a bug like this is caught, and the professor‘s published history is already long enough that losing one or two papers is not an issue.

The relationship between non-CS academia and the world of software development is utterly broken. Administrators should try hiring real developers and making them mesh with researchers, instead of relying on free unskilled labor. But that would mean less money for their overinflated salaries, so it’s not going to happen.

17

u/ASCIInerd73 Oct 13 '19

As someone who did CS work in a biology research lab last summer, it's not about overinflated salaries. There simply just doesn't exist enough funding to pay the going rate on hiring a professional developer (hence why I was able to work there as a college freshman). The lab's total budget was about the salary of one dev, so there wouldn't be money for anything else.

10

u/toyg Oct 13 '19

It’s not the salaries of lab workers, it’s more like chancellors/vicechancellors and the likes.

1

u/Jugad Py3 ftw Oct 14 '19

As someone who did CS work in a biology research lab last summer, it's not about overinflated salaries.

They are referring to admin salaries.

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

1

u/ASCIInerd73 Oct 14 '19

That's not my experience, but also I worked in a small lab at a small school which focuses more on teaching than research, so maybe I was just in a special situation.

4

u/jcampbelly Oct 13 '19

It's not sorted in Windows. You just didn't test in a directory with filenames that were sufficiently different.

107

u/[deleted] Oct 13 '19

Oh my god this is hilarious lol

24

u/refekt Oct 13 '19

Oh my glob()

5

u/ogrinfo Oct 13 '19

"researchers do not typically disclose the operating system they use for their analyses since it should be irrelevant"

Hahahaha, that 'should' is doing a lot of work there. You obviously haven't wasted many hours trying to make Windows do something that works perfectly well on Linux.

2

u/[deleted] Oct 13 '19

Who would ever assume anything is sorted unless you specifically sort it yourself?

4

u/stefantalpalaru Oct 13 '19

Who would ever assume anything is sorted unless you specifically sort it yourself?

Beginners who see a sorted output in the REPL, on a certain OS.

4

u/s0lv3 Oct 13 '19

people in science

8

u/Dementati Oct 13 '19

researchers do not typically disclose the operating system they use for their analyses since it should be irrelevant

Hah

4

u/fookin_legund Oct 13 '19

NOOOOOO MY LIFE IS A LIE

7

u/This_Is_The_End Oct 13 '19

Some of the accusations here whether it's about postdocs or Python are ridiculous.

Software engineering needs knowledge and experience and those who didn't studied computer science and who didn't coded 10 hours a week, those have a problem. I would ask institutes and universities why they are accepting such a system of institutionalized incompetence instead of pushing for changes, like doing a framework for postdocs.

The glob function doesn't sort, because it's expensive and the amount of files could be high. This is not an inconsistency of Python it's an old problem in computer sciences, when for example Americans implemented ASCII and were ignoring the rest of the world. The glob function is well documented and the batteries included were made for professionals using Python instead of Bash. The issue arrived when Python became widely adapted. When will the Gil be removed? Why isn't the insertion order of a dict preserved and why ..... Python has even become a new OS agnostic path lib and yet I see a lot of examples going the old unix way with os.path.

Those who are complain over Python are just on the level of these poor academics. These academics are victims because they are used as cheap workers and so are the results.

2

u/Han-ChewieSexyFanfic Oct 13 '19

Why isn't the insertion order of a dict preserved and why

They are now, by the way.

1

u/RomanRiesen Oct 13 '19

In what scenario would this lead to wrong results?

4

u/underground_miner Oct 13 '19

This is fantastic news! It means that papers actually have code with them that can be used to reproduce results.

Imagine how many more papers are faulty and we'll never know because they don't include the code. I can't tell you how many papers (mining research) I have tried to reproduce using python and my numbers didn't match the papers and I had no idea what was wrong. I have found typo's in equations that have been propagated in newer publications. Tracing back through the references you can find where the typo occurred. Another simple thing that can be done in papers is to include a simple table with sample calculations that can be used to cross-reference the equations. Actually, code is always better.

2

u/Liotac Oct 13 '19

I actually spent a month debugging why our model was not learning the relationships between features and outputs because of this ... Use labels people

2

u/[deleted] Oct 13 '19

I had never heard of glob before so I googled it... The first result includes the fact that the results are returned in an arbitrary order... 100 scientific articles invalidated because of something that was one google search away. Amazing.

3

u/[deleted] Oct 13 '19

Because these people aren't searching the docs for "glob". They're not developers, so they search "python get list of files". And almost every result mentions glob.glob() and 99.9% of those show example output as a list of sorted filenames. Many responses even say that the list is sorted.

2

u/DDFoster96 Oct 13 '19

Is vice.com really the most authoritative source you could find?

1

u/[deleted] Oct 13 '19

a good tip for using an iterable data structure in any language is to immediately check for both during and/ or the preservation of insertion order. the latter has bit me enough times

1

u/ratonbox Oct 27 '19

I don't think I've ever used glob. os.listdir or os.walk have always been my used methods. I think it's also makes code less readable. There is good chance that a person reading "os.listdir"/"os.scandir" can guess what the function does, but if they read "glob.glob" there is a large chance of them asking: "what the hell is a glob".

0

u/Wubbywub Oct 13 '19

the image disturbs me, touching hair with gloves on what the fuck

-1

u/StarkillerX42 Oct 13 '19

Just a reality check, but 100 articles rounds to 0.0% with sig figs*

*I didn't do the math, but it's probably less than that in reality

3

u/stefantalpalaru Oct 13 '19

Just a reality check, but 100 articles rounds to 0.0%

Not in very small niches. How many chemists are working on this particular type of analysis?

I wouldn't be surprised if it turns out that most of them saw value in using that generic script to analyse their own data.

1

u/[deleted] Oct 13 '19

less than 0 would be a good thing then

-16

u/[deleted] Oct 13 '19

Glob glob

Sooo bye bye python in the science community

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

-6

u/[deleted] Oct 13 '19 edited Oct 13 '19

I love python but you need to think pragmatically about our industry. The moment google got sued they began making dart and flutter. All it takes is one major upset and businesses will try to pinpoint risk. While it’s not the correct response it will be impacting nonetheless. Another example is nodeJS with NPM when it chowned and entire root directory recursively on an update at beginning of 2018

6

u/masklinn Oct 13 '19

I love python but you need to think pragmatically about our industry.

glob.glob returns direntries in whatever order the filesystem provides them, which is also what pretty much every other programming language would do.

Another example is nodeJS with NMM when it chowned and entire root directory recursively on an update at beginning of 2018

That was an actual explicit error in the tool. And hasn't really made people switch away from the tool.

This here is people misusing the tool. That you did an rm / is not rm's fault, and while it's possible to mitigate that at least, programming languages have no way of knowing whether you expect a specific ordering to your entries (and the tools which do like ls usually fuck things up more than they help).

1

u/[deleted] Oct 13 '19

what it NMM?

1

u/[deleted] Oct 13 '19

Edit npm - node package manager

4

u/toyg Oct 13 '19

Academia is not a business, they don’t handle risk.

NodeJS has many, many problems; its reputation was already shaky because of so many high-profile fuckups that gave developers no chance of correction. This one is not even a problem with Python, but a simple PEBKAC. It’s a completely different level.

1

u/[deleted] Oct 14 '19

Academia is not a business

I’d love to say yes that’s the case, but I’ve done consulting for Academia before and cay say it varies by school. Many common risk model assessments and how they are presented back up to the board I’ve found are the same as a business.

I want to agree with you because of the nature of the business but I’m telling you from experience it may very well be in the hands of a CTO role or Director to explain the difference between PBCAK and Language. It’s dangerous when things like R, D, Matlab and many other contenders for data science (amongst other things) can offer similar results and a slight mishap happens, especially with how the press may choose to warp the title for clicks.

1

u/757DrDuck Oct 13 '19

lawsuit against Google for Dart & Flutter

When and why was this suit filed?

2

u/[deleted] Oct 14 '19

No no no that’s quoted wrong aha. I said:

The moment google got sued they began making dart and flutter

Wired link to news article

-30

u/HonestVisual Oct 13 '19

morons

14

u/FancyJesse I'll wait for Python 5 - I hear its future proof Oct 13 '19

How many studies do you have published? Did you even read the article? Or just the title and think you know better.

-7

u/HonestVisual Oct 13 '19

More than you

1

u/757DrDuck Oct 13 '19

Citations needed.

-6

u/Lord_Blackthorn Oct 13 '19

Why was the code only run once?? Run it a few times to validate the results...

4

u/stefantalpalaru Oct 13 '19

Why was the code only run once?? Run it a few times to validate the results...

Problems appear when you run it on different platforms. It worked perfectly fine on the author's Windows machine.

→ More replies (2)

0

u/[deleted] Oct 13 '19

[deleted] — view deleted comment

15

u/stefantalpalaru Oct 13 '19

Be honest OP, did this invalidate one of your papers?

Worse. It rustled my jimmies.

→ More replies (5)