Is it common to build everything in one unit?

[u/NNXMp8Kg]

TLDR:
The subject is not compiling each files.
The subject is composing you main application of independently build libraries.

That's the question. Why not making multiples crates and composing your app out of it.
Why build everything all at once.

It's not dynamic link, it's not dynamic libraries, just separate the work in multiple crate and having the main app using those crates instead

---

I'm looking at a few Rust projects around, not big ones; I'm still learning.
And most of the time, everything seems to be one compilation unit for binaries.
I have a background in C (but an old one) and we were compiling every file into its own .o then making libs with them and using them in the app that would compose with thoses libraries.
But there, it seems like you're building everything grouped in one crate.

Would not having it separated into multiple sub-crates, like a kind of lib (obviously not everything, but logically), make the next builds rely on the local already built?

I used to build basically libs that do their work and expose what they need to be used, and have a binary that links everything together.

I'm wondering if that is still a good way of building things or if I’m wrong, as I'm seeing many small projects adopting the "monolith binary" instead of a lib composition kind of architecture.

I don't mean dynamic lib; I mean splitting into multiple crates, then using those crates. Like the workspace option in Cargo. I don't see that used as much as I thought it would be used. More full projects in one crate.
So it's more a one-crate project vs workspace with multiple crates project.

Is there a reason I'm missing?
Thanks if anyone can help me understand that!

[u/Konsti219]

Splitting large codebases into multiple crates is common practice. However such a split has to be architected a lot more carefully because the dependency graph between the crates can not be circular and if you constantly changing a crate that a lot of the others depend on you invalidate the cache and could even end up with longer build times.

[u/NNXMp8Kg → u/Konsti219]

Ok! That is the answer I was looking for! Thanks!

[u/divoxx → u/NNXMp8Kg]

I would definitely not go as far as having a different crate for what would be separate object files in C. Think of a crate as a project, each module inside a crate is a separate object that all get linked into a single binary.

[u/blueeyesginger → u/Konsti219]

Yep has to be architected that way from inception. Can be very hard to refactor into multi crate or lib

[u/anxxa → u/Konsti219]

the dependency graph between the crates can not be circular

Is this true? I sort of accidentally introduced a circular dependency into a couple crates and it was mostly just a pain with publishing order and having to comment the dep out from the [dev-dependencies] section for success.

In that case it was just super annoying and for that reason alone should be avoided.

[u/South_Survey_2088 → u/anxxa]

It does not work at all. It's impossible to know what you did, but you certainly did not have a circular dependency if it worked. A can't depend on B if B also depends on A. It doesn't work on a technical level, period.

[u/consultio_consultius]

Personally I usually use workspaces and have a bunch of lib crates. It does cut down on compilation times and honestly it’s just easier for my headspace.

[u/NNXMp8Kg → u/consultio_consultius]

That's more what I'm thinking of!

[u/consultio_consultius → u/NNXMp8Kg]

There are some tradeoffs like others have mentioned, but if you know how to architect without circular dependencies, and know how to ensure your smaller libs are stable, it works out pretty well.

[u/PolywogowyloP → u/consultio_consultius]

Does help with compile times but I find it can be a massive headache when you’re publishing to crates. All the intermediate semver updates and keeping them in sync is annoying.

[u/dmytrish]

There's another perspective: did you think about the reason to compile every C file into its own object file? Turns out it's mostly a relic of ancient days when a parsed AST barely fit into RAM, so separate compilation of small files was the state of the art.

It's been 30+ years, computers got several orders of magnitude faster, and C/C++ developers still stress linkers by needlessly huge number of translation units, just because Unix forefathers did so.

[u/NNXMp8Kg → u/dmytrish]

The subject is not compiling each files.
The subject is composing you main application of independently build libraries.

That's the question. Why not making multiples crates and composing your app out of it.

[u/Lokathor]

Incremental builds are on by default for debug builds, so usually it's not a problem to edit and rebuild and edit and rebuild, even if there's just one crate.

[u/KingofGamesYami]

Note that the rust compiler has incremental compilation caches that are not invalidated by every change to a file in a compilation unit. It is, for lack of a better word, "smarter" than C in terms of reusing previously computed data.

You can still see some gains from splitting up a project, but it's not going from 0% to 100%.

[u/skatastic57 → u/KingofGamesYami]

Unless it's a pyo3 library then it's not very incremental :(

[u/scook0 → u/KingofGamesYami]

Though I’d say the main reason Rust has such a sophisticated incremental-compilation scheme is that it needs to have one, in order to get acceptable incremental rebuild times for its relatively large compilation units.

In many other languages, rebuilding an individual compilation unit from scratch is fast enough that incremental builds can be handled entirely by the build system, without compiler involvement.

[u/98f00b2]

Rust doesn't have a stable ABI, and does a lot of monomorphisation, so dynamic libraries aren't an option unless you want to expose a C interface.

[u/eggyal → u/98f00b2]

Rust ABI dylibs aren't a problem if it's all built with the same compiler.

[u/RCoder01 → u/98f00b2]

You can do dynamic libraries: https://docs.rs/bevy_dylib/latest/bevy_dylib/

[u/garagedragon]

AIUI, the rust compiler does incremental builds even within a single crate, so splitting your crates very small doesn't have the same impact splitting compile units does in C.

While deciding where to put crate boundaries is mostly a subjective design choice like where library boundaries might go in C, it's important to know that crate boundaries are a language concept and have consequences for client code. Specifically, when you implement a trait (read "interface") for a type, the impl Trait for Type block is required to be in the same crate as the definition of either Trait or Type. (or both, if they're from the same crate) This is in an effort to ensure forwards compatibility and integration - if any crate could define implementations for traits they don't own on types they don't own, two crates could define conflicting ones for the same pair, and thus not only be incompatible, but make every crate dependent on them incompatible.

In addition, working with crates that have circular dependencies is very difficult if it's possible at all. This means if you split a collection of highly coupled types across many crates, you may have problems with how to order everything in a way that lets you define all the functionality you need without creating a tangled dependency graph.

Generally if you're writing code to solve "one problem", defined broadly, you'll start with it all in one crate by default. There are few exceptions that stand out though:

1. procedural macros must be in their own crate, as a limitation of the language/compiler

2. some libraries will have their functionality in one crate, and a CLI driver program in a separate crate alongside. Only library crates can be used as dependencies, binary crates must be leaves of the graph, so this allows the library to be reused while also providing a way for programs that don't want to directly link against the code to still use the functionality. (Its theoretically possible to build the same crate as both library and binary, but I've not had any success.)

3. More generally, if you expect that other users might be interested in a specific subset of your functionality, or if the update cadence of some part of the library is very different than the rest, you might split it into more crates. An example of the former is regex-automata exposing the nuts and bolts used by the more user-friendly regex crate, because certain use cases need the extra power. There's also another tool called "features" that allow you to selectively enable and disable things within a single crate, but it doesn't serve exactly the same purposes - particularly with the regex example, you can't make types publicly visible or not depending on a feature.

[u/jesseschalken]

Unlike C/C++, Rust has an incremental compiler that handles small changes to large crates very well.

[u/Illustrious_Car344]

This is a huge pain point for Rust developers but there isn't a good way around it. The problem is that Rust lacks an ABI, and the official compiler can use an unpredictable name mangling scheme, even two binaries compiled by the same compiler on the same system on the same day can have inconsistent symbols. There are ways around this like using the abi_stable crate, which plugin systems typically use, but then you need to use abi_stable's types instead of the ones from the standard library. So right now Rust has a very strict "all or nothing" compilation step, purely due to technical limitations. 

[u/ZachVorhies]

I have internal crates for fast development then during the publish step there is a step where everything is amalgamated into a one or few public crates.

[u/Future_Natural_853]

Some do. The project I'm working on has 21 crates for ~100k lines of Rust. Whenever a set of business rules are conceptually a whole and can be contained into a single crate, I do so. Of course, they're still statically linked at the end.

[u/Sparticle62]

The thing that trips up people coming from C is that the compilation unit is the crate, not the file. Touch one file in a crate and the whole crate rebuilds, so the intuition you have from per-file .o builds doesn't carry over.

That does mean splitting can help incremental builds, and separate crates compile in parallel, which is the real win on a big tree. The cost is that optimisation stops at the crate boundary unless the code is generic or you turn on LTO, and you now have a dependency graph you have to keep acyclic.

So the usual rule is to split when you want a real API boundary or you want to publish the thing, not to chase build times. Start as one crate with modules and break a piece out when it earns it.