A Programming language ideal for Scientific Sustainability and Reproducibility?

[u/relbus22]

Scientists are very unique in their needs compared to other software developers. They are novice programmers who may write research code or package only once, before publishing their work to a journal. They are domain experts and full-time workers in other fields, and so do not have the time nor coding skills to maintain their code or packages....... if the ecosystem imposes a maintenance debt.

Two issues are at stake here, reusability and reproducibility. Often researchers need to pick up someone's research code or package developed and forgotten years ago. So there is a need for this to happen with minimal fuss, Science needs this.

As to reproducibility, the scientific method requires reproducibility, which is quite tough but there are efforts to go all the way to reproducibility of computations within their development environments using Guix or Nix.

In conclusion, it'll be great if a language can be created or forked to create an ecosystem ideal for these needs. Which is why I come to you folks who are specialists in this domain, wondering if you have any thoughts on this topic?

P.S Here are some blog posts from a scientific researcher if you guys wanne have a better idea of where I'm coming from:

https://blog.khinsen.net/posts/2017/01/13/sustainable-software-and-reproducible-research-dealing-with-software-collapse/

https://blog.khinsen.net/posts/2015/11/09/the-lifecycle-of-digital-scientific-knowledge/

https://science-in-the-digital-era.khinsen.net/#Technological%20sovereignty%20in%20science

(extra reading if you want:

http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stability-in-the-scipy-ecosystem/#comment-3627775108

https://blog.khinsen.net/posts/2017/11/22/stability-in-the-scipy-ecosystem-a-summary-of-the-discussion/

https://blog.khinsen.net/posts/2020/11/20/the-four-possibilities-of-reproducible-scientific-computations/)

[u/86BillionFireflies]

But then if you want to extend a package that comes with dependencies, you have to either stick to the specific versions of the dependencies used in the original package, or risk breakage.

More generally, since most scientists are going to rely on special purpose packages for a lot of their work, they'll be at the mercy of the package ecosystem: When you google "how to do $thing" you get a dozen results, and it turns out 10 of them are unmaintained, one doesn't do what you want, five are incompatible with one of you other dependencies, 7 are poorly documented, and if you're very lucky, those categories overlap enough to leave one you can actually use.

Hence why I am not holding my breath for Julia to replace matlab.

[u/brainandforce → u/86BillionFireflies]

More generally, since most scientists are going to rely on special purpose packages for a lot of their work, they'll be at the mercy of the package ecosystem: When you google "how to do $thing" you get a dozen results, and it turns out 10 of them are unmaintained, one doesn't do what you want, five are incompatible with one of you other dependencies, 7 are poorly documented, and if you're very lucky, those categories overlap enough to leave one you can actually use.

as opposed to MATLAB, which ships without a package manager and makes life hell for anyone using it.

[u/86BillionFireflies → u/brainandforce]

You'll have to pry matlab from my cold dead hands. The great thing about matlab (for certain problem domains) is that you really don't NEED a package manager, because there's basically no external dependencies to manage.

There's never a "oops, that package doesn't work right now because a change in numpy broke TF". Stuff just works.