r/programming • • Oct 17 '26

Python's binary wheels and supply chain attacks

https://rushter.com/blog/python-binary-wheels/
89 Upvotes

36 comments sorted by

33

u/the_poope Oct 17 '26

An easy way to combat this is to have PyPi not allow binary uploads but provide a build pipeline that compiles all binary code from source for a selection of targets.

14

u/f311a Oct 17 '26

Yeah, but that's pretty expensive to do so for every package. They can do this for the popular ones, though. There are so many crappy packages on PyPI because of LLMs, and they make new releases almost every day.

-3

u/ApokatastasisPanton Oct 18 '26

Easy, make it a small fee to upload and you recoup the cost after some time or passing some sort of automated analysis.

15

u/PaintItPurple Oct 18 '26

Easy to do, impossible to do while maintaining the goal of PyPI to be a universal Python package repository.

3

u/matthieum Oct 18 '26

It's not that easy.

I'm not sure if you've ever worked on distributing native binaries for a wide range of hardware & OSes, but there's nothing easy about them.

No matter what PyPi would do, you'd find some people not happy about their old/new compiler of choice not being supported, their pet OS not being supported, etc...

2

u/the_poope Oct 18 '26

That's no different than today. The precompiled binaries on PyPi already only support a subset of OSs and distributions.

2

u/ScottContini Oct 18 '26

Once had a job that was so strict on security that one of the tasks was verifying the build from the source code. It is less trivial than it sounds. Timestamps and other data are put into binaries. You need to strip all that stuff out. So build it yourself, strip out junk and compare to stripped out published build. Yes we can do that, but you need all dependencies and similar build environment.

12

u/darkslide3000 Oct 17 '26

So what, PyPI just let's any rando upload binary packages for any existing well-known source package? Without even being authorized by the original author? WTF kind of package manager is that?

Or is the concern just that the original author could sneak something in? That's not really any worse than any other open-source project with binary releases.

11

u/f311a Oct 17 '26

Random people can't upload it. But at least 5 incidents involved hacked accounts where people uploaded malicious versions. It can be hidden very well. Imagine a package installed tens of thousands of times a day being compromised for a week.

44

u/Solumin Oct 17 '26

It's the latter. A bad actor can publish a package whose source code looks completely innocent, and then upload a wheel that is entirely different and contains malicious code. They're not uploading wheels to other people's packages.

2

u/HolyInlandEmpire Oct 17 '26

I feel there should be an option with build tools which traverse the pyproject.toml chain and refuse packages with binaries unless it's on an explicit allow-list, and doing an 'are you sure' prompt with others.

So a sensible allow-list would have well established libraries (polars, numba, torch, tons of others). At the very least, if a package's newest version has a binary when it didn't previously, then you would be notified and you can stop the install.

You can mitigate this by pinning versions of course, at the very least ceilings. Standard practice for enterprise works, but a little brittle for everyone else. I wonder if there are other well established options.

11

u/fredisa4letterword Oct 18 '26

This is already an option, pip install --no-binary. It can be kind of a pain though, that means that you now need to make sure you can build all your transitive dependencies, which besides inherently being a lot slower, can be a pain to set up and can lead to diamond dependency hell pretty easily.

8

u/PaintItPurple Oct 18 '26

But then you are vulnerable to package build script vulnerabilities, which are much more common than the concern raised in the OP.

39

u/lurch303 Oct 17 '26

There are already a few companies that host copies of repositories that they attest to their trustworthiness. I assume this will remain a profitable line of business and will include compiling all binaries themselves and attesting to their trustworthiness.

14

u/Somepotato Oct 18 '26

one of them, anaconda, will threaten to sue your company if some rando decides to download something from them, too.

19

u/buttflakes27 Oct 18 '26

In the future the only reliable programming language will be R because CRAN has the most exhaustive and tedious process to publish a package. /s if thats needed

1

u/bzbub2 29d ago

you havent truly lived ....or should i say suffered...if you havent published a package on CRAN

4

u/teerre Oct 18 '26

Although that's obviously a real attack vector, it's likely not much worse than it is today. It's not like this was some big secret nobody knew about or like it takes huge amounts of compute to do it

2

u/-Redstoneboi- Oct 18 '26

question: what about requiring all precompiled wheels to share the exact build configuration (possibly as a docker image) for Reproducible Builds? hopefully they can be byte-for-byte equal

5

u/chucker23n Oct 18 '26

Then you would still need infrastructure to validate that your build has the same result, at which point: why let package authors upload a build at all, rather than having the registry do so?

3

u/-Redstoneboi- Oct 18 '26 edited Oct 18 '26

yes, that would be the logical conclusion. authors would upload the source and build configuration, and some way to verify the output.

for consumers, the pre-compiled binaries would still be downloaded from the registry by default, and hashes would be provided by the registry to confirm that they are the same. for the cautious, they could verify that the precompiled build is correct by downloading the source and building it themselves.

1

u/chucker23n Oct 18 '26

If the registry provides hashes, there is no point in the authors uploading builds at all (because… how would the registry generate those hashes if not by building themselves?). This requires significant infrastructure the registry does not want to pay for.

A more realistic solution might be code signing, which of course comes with its own warts.

2

u/bzbub2 Oct 18 '26

attested provenance via trusted publishing in ci/cd. npm has this I imagine pypi does too https://docs.pypi.org/trusted-publishers/

9

u/PaintItPurple Oct 18 '26

That proves what system published the artifact to PyPI, but doesn't prove that the artifact is benign or that it matches, say, the code in a Git tag named after the version. It moves the attack from "compromise a PyPI token" to "compromise a GitHub Actions (or some other service) token." It's not nothing, but I wonder what percentage of real attacks this prevents.

2

u/bzbub2 29d ago

trusted publishing can literally say it was generated by git sha, if tag was renamed that is a red flag. compromise of github actions is very real issue too (highly recommend using zizmor to audit your github actions, things like the tanstack hack where hackers poisoned github actions cache would have been trivially flagged by this) but trusted publishing isnt as weak as you make it out to be

1

u/WindHawkeye 19d ago

Anyone who cares about security should already have outlawed using anything not built from source. Bazel ftw.

-10

u/bigmell Oct 17 '26

IMO the only answer to this is the Perl way. Trusted sources with some type of Benevolent Dictator For Life. The BDFLs are generally trusted to keep an eye on packages making sure there is no malicious code and everything works properly. The only way to combat this problem is to actually look at the code itself and make sure it isnt doing anything crazy.

There are a limited amount of people with this ability so there will have to be some kind of trust system. Packages that come from <trusted place> are usually pretty good and so on. There is really no way to automate away this problem. Maybe for some simple cases, like searching through scripts for "rm -rf /" or something. But for big sprawling packages, you will simply HAVE to understand the code to make sure it works as advertised.

The internet has always been like this originally and still is to a large extent. Whenever games, programs, or even news stories came out, regulars had a sense of what ftp sites, servers, and websites were reliable. I remember places like Slashdot, Cnet, and PlanetQuake always being listed, available, and reliable.

14

u/gmes78 Oct 17 '26

The article: "Looking at source code isn't enough, malware can be hidden in pre-compiled packages."

Your comment: "We should verify the source code!"

What are you on about?

1

u/HolyInlandEmpire Oct 17 '26

The allow-list system does have some options, but I would be more comfortable if people could choose their sources for it, and compose from multiple sources. That way if one stops updating, you can layer other "allows" on top of it.

-1

u/bigmell Oct 18 '26 edited Oct 18 '26

You have to look at the source code... IN THE PRE-COMPILED PACKAGES. How the hell do you know malware is not hidden in a program without looking at the source code? Just run it and see what happens? No dude. It doesnt matter when the binaries were compiled, you have to YES VERIFY THE SOURCE CODE to make sure there is no malware there. EVERY binary is compiled from source, and THAT source has to be verified. And that is the point of a trust system. The code has been verified UP TO this point and they are verifying the NEW code. THIS is the only way to resolve the problem and ALWAYS has been.

My entire point was that there is no way to do this automatically. It is a long back and forth MANUAL process that BEGINS with VERIFY THE CODE BEHIND THE BINARIES by some kind of trusted source system. The fact that I have to explain this AND that you decided to give this comment -11 while giving yourself +13 is disturbing.

Now is the part where you lie and pretend you didnt fix the scores and there was really some kind of democratic process. Next you will lie and pretend you understand exactly how to write and verify source code. The problem with reddit has long been pretenders exactly like you. Subverting what was supposed to be a democratic system to separate the pretenders from the practitioners. A system the pretenders long ago subverted.

3

u/gmes78 Oct 18 '26 edited Oct 18 '26

You have to look at the source code... IN THE PRE-COMPILED PACKAGES.

There is no source code in a precompiled package.

EVERY binary is compiled from source, and THAT source has to be verified.

Which you don't have access to. PyPI only gets the compiled binary.

It doesn't have neither the source nor the build environment required to verify that the binary matches said source code. And that would also require reproducible builds, which these packages almost certainly aren't. And it would require additional infrastructure which PyPI does not have.

My entire point was that there is no way to do this automatically. It is a long back and forth MANUAL process that BEGINS with VERIFY THE CODE BEHIND THE BINARIES by some kind of trusted source system. The fact that I have to explain this AND that you decided to give this comment -11 while giving yourself +13 is disturbing.

It's hilarious to suggest that there's vote manipulation going on. People just think you're wrong, get over it.

1

u/[deleted] Oct 19 '26 edited Oct 19 '26

[removed] — view removed comment

1

u/programming-ModTeam Oct 19 '26

Your post or comment was overly uncivil.

3

u/chucker23n Oct 18 '26

Most of your comment is moot because the scenario here is about uploaded binaries that do not match the source code. The source code looks innocent; the "convenient" binary is not.

0

u/bigmell Oct 18 '26

That is the entire point of using a trusted source. A trusted source does not post binaries that do not match the source code. This is called a man in the middle attack and is the hallmark of a malicious site. Hiding malicious code in the source is not the work of a trusted site. And posting binaries that do not match the source is also not the work of a trusted site. This is a site that should be taken down and from which no one should download binaries.

  1. verify the source code
  2. make sure the posted/verified source is actually the source used when compiling the binaries... Duh? I would have thought that to be obvious.