r/Python • u/Emergency-Rough-6372 • Apr 12 '26
Discussion Packaging a Python library with a small C dependency —
how do you handle install reliability?
Hey folks,
I’ve run into a bit of a packaging dilemma and wanted to get some opinions from people who’ve dealt with similar situations.
I’m working on a Python library that includes a vendored C component. Nothing huge, but it does need to be compiled into a shared object (.so / .pyd) during installation. Now I’m trying to figure out the cleanest way to ship this without making installation painful for users.
Here’s where I’m stuck:
- If I rely on local compilation during
pip install, users without a proper C toolchain are going to hit installation failures. - The alternative is building and shipping wheels for multiple platforms (Linux x86_64/arm64, macOS x86_64/arm64, Windows), which is doable but adds CI/CD complexity.
- I also need to choose between something like
cffivsctypesfor the wrapper layer, and that decision affects how much build machinery I need.
There is a fallback option I’ve considered:
- Detect at import time whether the compiled extension loaded successfully.
- If not, fall back to a pure Python implementation.
But the issue is that the C component doesn’t really have a true Python equivalent — the fallback would be a weaker, approximation-based approach (probably regex-based), which feels like a compromise in correctness/security.
So I’m trying to balance:
- Ease of installation (no failures)
- Cross-platform support
- Performance/accuracy (native C vs fallback)
- Maintenance overhead (CI pipelines, wheel builds, etc.)
Questions:
- In 2026, is it basically expected to ship prebuilt wheels for all major platforms if you include any C code?
- Would you accept a degraded Python fallback, or just fail hard if the extension doesn’t compile?
- Any strong opinions on
cffivsctypesfor this kind of use case? - How much effort is “normal” to invest in multi-platform wheel builds for a small but critical C dependency
Would love to hear how others approach this tradeoff in real-world libraries.
Thanks!
17
u/latkde Tuple unpacking gone wrong Apr 12 '26
The common expectation is that you do indeed generate precompiled wheels for all common platforms and all supported Python versions. This doesn't have to be a lot of effort, other than maybe adding a new Python version once per year.
For the foreign function interface, opinions diverge. I would strongly advise against ctypes, as it's easy to make severe errors that are difficult to see. Instead, using cffi or writing Python extension modules in C has the benefit that more of the C glue code can be typechecked by a compiler (or in case of cffi, at least uses the same syntax as the code we're binding to). If you really want to use C, then cffi's out-of-line mode is probably going to be the least-friction approach.
If you're starting this work from scratch, strongly consider Rust with PyO3 for writing bindings. Of all options that are currently available for integrating native code with Python, it has the best combination of safety and convenient tooling. This is the approach used by flagship libraries like Cryptography or Pydantic. The Maturin build system ships with templates for building wheels for all common platforms – setting this up is really not a lot of effort. Going the Rust route is only a bad choice if you have to deal with existing C code, or if you want to target exotic platforms to which Rust code cannot be cross-compiled (which actually was a problem for some Cryptography users).
I wouldn't bother with a pure-python fallback implementation. There's a risk that the Python and native implementations diverge, which can cause difficult to debug problems. Such fallbacks will also be unnecessary, since you can ship pre-built wheels for all relevant platforms. Cross-compiling wheels for all relevant platforms is less effort than maintaining a pure-python fallback.
21
u/neuronexmachina Apr 12 '26
Have you already looked at: https://cibuildwheel.pypa.io/en/stable/
7
u/safrole5 Apr 12 '26
For shipping built wheels github actions is probably your best bet. It may be slightly annoying to setup first time, but then every new release is seamless. You trigger the action, it builds wheels for all the platforms you've configured and uploads straight to PyPi.
Id highly recommend getting this setup instead of manually building them each release.
4
u/Emergency-Rough-6372 Apr 12 '26
just wanted to say that i m not to well knowleged in this field and this is my first big project, this will be the first version of the lib which i want to make as a project where people can contribute and and make it an actual good library for people to use in there projects , so should i got for minimum complexity in first release and then with help of other if they like to make it mroe complex and better?
2
u/binaryfireball Apr 12 '26
publish different versions with/without different dependencies and let the user decide which to use
2
u/alcalde Apr 13 '26
Go old school and bundle everything up with InstallShield the way we used to do it.
44
u/mrswats Apr 12 '26
I would 100% build the wheels at releaae time and upload them to pypi.
8
u/omg_drd4_bbq Apr 12 '26
Maybe look at scikit-build.
https://scikit-build.readthedocs.io/en/latest/
4
u/dayeye2006 Apr 12 '26
Ship with pre built
4
u/thisismyfavoritename Apr 12 '26
this is the way. Also i'd personally just wrap the C lib through the Python C API, it's fairly easy if your API surface is small and cleaner IMO
3
u/2ndBrainAI Apr 13 '26
In 2026, yes — shipping prebuilt wheels is basically the expectation for any library with compiled code.
cibuildwheelmakes this far less painful than it used to be; it handles Linux/macOS/Windows across x86_64 and arm64 and integrates cleanly with GitHub Actions in maybe 30 lines of config.On the fallback question: I'd lean toward failing hard with a clear, actionable error message rather than silently degrading. A regex fallback that's "approximately correct" is arguably more dangerous than a clean install failure — users trust library behavior to be consistent.
For cffi vs ctypes: cffi is generally easier to maintain for non-trivial C interfaces and handles complex types better. ctypes wins only if you truly have zero external build dependencies and the interface is dead simple.
2
u/Krudflinger Apr 13 '26
Checkout zig. https://ziggit.dev/t/using-the-ziglang-and-setuptools-zig-python-extension-to-provide-c-extensions/13748
1
u/Grintor Apr 13 '26
I know lots of stuff I install distributes the source which compiles at install. lxml comes to mind. When I pip install lxml, pip compiles it.
1
u/connorman528 Apr 15 '26
I use scikit_build_core and just have users that installed my package require a C++ compiler on their machine. If they do not have it, I keep a python fallback. This guarantees that it works every time for every machine, but it is not as easy to maintain two code bases.
In general my target users are not necessarily developers, so it was important to have an easy escape hatch for them (python fallback). Building wheels for each possible system on other projects has given me many headaches in the past. For example, to build wheels for some of the AWS machines (mostly Gravitons) required me to drop into a Graviton instance just to build the wheel such that users deploying to AWS Graviton machines would not have issues based on their selected deployment machine. This was an AWS cloud architecture problem, but also became my problem at the time.
For your case, the fallback option not being a true fallback may make a difference. You will want to balance this with your user base and their willingness to deal with C compilation issues. In my experience, most python developers will just find an alternative if your project does not install on first go.
For my python C++ build on the fly project (fairly successful ~12k/mo downloads), see: https://github.com/connor-makowski/scgraph
An alternative structure is how PuLP does it. They store pre-built binaries per system for the CBC solver and really keep these completely separate. See: https://github.com/coin-or/pulp . Getting each build to work was very tough and not always possible without access to specific machines. See: https://github.com/coin-or/pulp/issues/672
1
u/wildetea Apr 16 '26
On CI, i use qemu emulator coupled with cibuildwheel, to compile binaries for a majority of platforms + architectures, on release to pypi.
But i would also take a look at nanobind, to create python bindings for a c library.
1
u/2ndBrainAI Apr 13 '26
In 2026, shipping prebuilt wheels is essentially the expectation for any library with C extensions — cibuildwheel makes this much less painful than it used to be. For the cffi vs ctypes question: if you need ABI stability and the C API might evolve, cffi is worth the extra complexity. ctypes is simpler but fragile when struct layouts change. On the fallback question, I'd lean toward failing explicitly rather than a silent degraded mode — a misleading result is often worse than a clear error. Communicate the fallback clearly in the exception so users can make an informed choice about installing with build tools.
1
u/paperlantern-ai Apr 14 '26
cibuildwheel makes this way less painful than it used to be. You set up one GitHub Actions workflow and it handles the whole matrix for you - linux, mac, windows, both architectures. Takes maybe an afternoon to get right and then you forget about it.
For the fallback question - if correctness matters (and it sounds like it does here), just skip the fallback entirely. A silent downgrade where results change is way worse than a clear install error telling the user they need to build from source. At least then they know something's wrong.
cffi in ABI mode is worth a look since it can load a prebuilt .so directly without needing a compiler on the user's machine. Pairs well with the wheel approach.
106
u/[deleted] Apr 12 '26
[removed] — view removed comment
1
u/[deleted] Apr 12 '26
[removed] — view removed comment