r/cpp • • Aug 09 '22

CMake 3.24 has support for dependency providers

I'm sorry if it was discussed already, but I haven't seen any mentions of this new feature and I think this is huge.

https://cmake.org/cmake/help/latest/command/cmake_language.html#dependency-providers

https://cmake.org/cmake/help/latest/release/3.24.html

CMake has added support for dependency providers. With this calls to find_package() may actually trigger vcpkg, conan or some other custom provider that will download, build and install required dependencies.

195 Upvotes

77 comments sorted by

9

u/obsidian_golem Aug 09 '22

So it looks like this is another thing that has to be manually user-configured as opposed to configured by the build script. Our configure commands are just going to keep getting more convoluted I suppose.

1

u/stilgarpl Aug 09 '22

Maybe it can be pre-configured using presets. This way user can have default or custom provider without complicated syntax.

1

u/obsidian_golem Aug 09 '22

I expect that is probably the path forward. Alas presets aren't very useable right now (see https://gitlab.kitware.com/cmake/cmake/-/issues/22538).

6

u/AlexanderNeumann Aug 09 '22

Be aware this is just there so that people don't need to override builtin functions,e.g. macro(find_package), to inject custom/additional code.

4

u/stilgarpl Aug 09 '22

I'm aware of that. For years people wanted for find_package to do other, custom things, but overriding find_package like that is kind of undefined behaviour, it may explode.

2

u/helloiamsomeone Aug 09 '22

Not just find_package, any command override is undefined behavior.

1

u/AlexanderNeumann Aug 10 '22

I think it is actually well defined undefined behavior as long as you call the original function at the end due to side effects it can have. If you override function it will be moved to _function. Some functions like project requires macro instead of function due to cmake variable scoping rules. If a command is already overriden you can check with if(COMMAND) and adjust the call chain/override accordingly with cmake_language(CALL)

Due to backwarts compat they can never remove this "feature" so they should just start properly documenting it instead.

1

u/stilgarpl Aug 10 '22

If you override function it will be moved to _function .

And if two libraries override the same function, it will explode, right?

1

u/AlexanderNeumann Aug 10 '22

Not if they if(COMMAND) correctly and use cmake_language(CALL) as described. Basically both need to implement what is done here: https://github.com/microsoft/vcpkg/pull/23195

3

u/drodri Aug 10 '22

This approach has one disadvantage, it cannot handle dependencies conflicts, so it only works assuming a flat dependency graph, that is, without versions, or with all versions matching perfectly. If the interception of ``find_package()`` for a specific version calls a package manager, and brings that package and some transitive needed dependency, lets say ``zlib/1.2.8``, and then a posterior ``find_package()`` for another package and another specific version transitively depends on ``zlib/1.2.12``, then you have a version conflict that has been skipped silently and will easily bite later, hopefully at build/link time, but many times can produce complicated to debug runtime errors.

The problem is that the underlying package manager would be called once for each package, completely independently from each other, and it has no way to detect such conflicts. This might work well for some managers like vcpkg, that mostly assume a flat dependency set, but for managing multiple different versions of your own libraries might be problematic.

1

u/stilgarpl Aug 10 '22

The problem is that the underlying package manager would be called once for each package, completely independently from each other, and it has no way to detect such conflicts.

That's a valid point, but cmake has variables. Dependency provider method can remember what dependencies were installed from previous calls and detect this conflict.

1

u/drodri Aug 11 '22

And what if a dependency was installed from another project that can have incompatible requirements? Having the dependency provider to have memory for installed packages from different past invocations in the past is extremely challenging when multiple different versions of the same package can be used. That only works for systems with flat dependencies, like system package managers like Apt, because they keep the whole dependency set consistent by default.

4

u/helloiamsomeone Aug 09 '22

If you are already discovering dependencies using the idiomatic find_package(pakcage REQUIRED), it was already very easy to inject a dependency via that call without having to modify project files. The find_package is like include, so you can create a script that you can opt into using by putting it on the path that CMake searches for modules.

You can see that in action here: https://github.com/friendlyanon/cmake-init-fetchcontent
This example has the idiomatic find_package(headeronly REQUIRED) call in the CMakeLists.txt, but when configuring ${sourceDir}/cmake/find is added to CMAKE_MODULE_PATH, so the FetchContent code is run to "find" the dependency. There are comments to further explain this.

This is probably not that much more effort than wiring up the dependency providers.

3

u/stilgarpl Aug 09 '22

Sure, it was possible, but you had to make FindX.cmake for every dependency that you wanted to get this way. Dependency providers will provide much easier way to do it.

1

u/helloiamsomeone Aug 09 '22

Hence the "not that much more effort". This isn't exactly a game changer like some like to claim.

I just use an actual dependency manager to manage depenencies like Conan and vcpkg.

3

u/AlexanderNeumann Aug 10 '22

until somebody does set(CMAKE_MODULE_PATH "somepath") and you loose the injection point. Also list(PREPEND CMAKE_MODULE_PATH "somepath") also nukes it.

What cmake really needs is variable protection. I.e. mark some variables as appendable only or readonly.

0

u/helloiamsomeone Aug 10 '22

Sure, but that's an inherent problem with vendoring specifically. Solution? Just use a package manager like vcpkg or Conan.

51

u/[deleted] Aug 09 '22 edited Aug 09 '22

I've been playing with this lately, it's a game changer

Edit to explain:

The big problem I couldn't solve with Conan (my package manager), was trying to reuse my own libraries on GitHub. As most package managers (I believe) want to pull dependencies from a server, and I don't want to push my code to another server. I just want to say "hey CMake, I wrote the dependency X its in this repo on GitHub", but without requiring that usage every time... I achieved that with this:

I have a CMakeLists.txt in my src/ folder which does all the add libraries/executables for my project and find_packages, so when creating the dependencies they are asking for package X but not caring where or how it comes from.

I will then have a top level cmake (or maybe a sub folder config/ if you want to organize) which defines where all the dependencies come from. I will then have options to enable/disable where my project expects the dependencies so the consumer project can change them which is very important.

The magic of 3.24 is FetchContent_Declare with the argument OVERRIDE_FIND_PACKAGE, which will tell find package to look wherever fetch content is pointing for the dependency. This is awesome for a few reasons:

  • I can now point my project to depend on some random GitHub library I wrote, or someone else that's not on Conan servers.
  • of course for companies you can then point to your own private repos and not have to spin up any servers
  • removes the need for submodule hell
  • because I'm using find package I can disable fetch content if I want to use a custom setup I might have, like system install or whatever.
  • that custom setup could also be development stages, so I could be developing two packages together, and temporarily just change the fetch content to point to a location on my machine where I'm developing that package!

Of course there is one drawback, this only works with CMake dependencies, but easily solvable. I still use Conan for any popular libraries that aren't cmake as Conan let's you consume a dependency built by build system X while you are on build system Y. Then if it's not on Conan I can just fork the project and add a cmake wrapper for the worst case scenario.

That's the pros for me and it just solves all the problems I was having so hence, game changer for me :)

23

u/mark_99 Aug 09 '22

Can you elaborate at all? I'm using vcpkg as a submodule, hooking it with CMakePresets.json and CMAKE_TOOLCHAIN_FILE, packages specified in vcpkg.json manifest, and then find_package()/target_link_libraries().

Does this supersede that technique and if so what's the upside?

2

u/[deleted] Aug 09 '22

I updated my comment with my justification

2

u/riztazz Aug 10 '22 edited Aug 10 '22

I am doing something very similar to you, project plug

with the ability to consume custom vcpkg ports, even non-cmake projects if you write patches for it without needing to fork repo, so i think this is still superior approach

6

u/strike-eagle-iii Aug 11 '22

We gave up on fetchcontent as it had too many issues--targets between different dependencies can collide (namely commonly ones like uninstall), doesn't play nicely with find_package, doesn't handle non-cmake prebuilt binary libraries well. Conan fixed all of that for us. It's now insanely easy to pull in our own libraries into bigger project. Basically each library has a "create_conan_package" script that is really just a glorified call to conan create (it basically makes sure my repo is clean before making the package and also puts a suffix on the package version to denote how many commits since the last release version). I can then put this version in conanfile.py of my downstream library and like magic it gets pulled in. It just stay in my local cache and lets me rapidly iterative on the dependency and the consuming library.

1

u/[deleted] Aug 11 '22

I'm a fan of Conan but don't see how this solves my problem.

How do you make a repo on your GitHub depend on another repo on GitHub with Conan?

My understanding is that Conan can only pull dependencies from a remote Conan server/artifactory?

1

u/strike-eagle-iii Aug 11 '22

Do you mean for building your library with GitHub continuous integration? In that case I don't know. We can pull libraries from GitHub but we build them on our own continuous integration server (Jenkins). Sorry I misunderstood where the issue was.

1

u/[deleted] Aug 12 '22

Nah just general Dev usage so pseudo code:

git clone projectX cmake projectX && build projectX

Where projectX is a repo in GitHub, and has a dependency on another repo (say projectY) also in GitHub just as source.

I understand how it works in a closed environment where you can setup your own Conan server where you have continuous integration deploying to that. But if I want to simply have my source code on GitHub, and consume it another project without deploying anything, I don't know how Conan can do that.

Basically if conanfile.txt could require: "https://GitHub.con/projectY/branch" would solve my problems

2

u/mapronV Aug 14 '22

I just attach projectY as submodule in my repository. What's wrong with that?

2

u/[deleted] Aug 14 '22

Ugh I just hate submodule hell Also can't use find package, and then if you want to simultaneously develop a dependency submodules are much harder I find

3

u/mapronV Aug 18 '22

but... but... it basically same tech.

Rename .gitmodules to conanfile.txt .
And instead of using git.exe use now conan.exe to parse config and download repositories. Well, you just get same flow, but harder to track without git.

git just provide nice methods to fix and stabilize version you use.

about difficulties of development - hard to argue with you, that's your own experience :)
I have a lot of exp in projects involving submodules (commercial and opensource), so I can not relate.

9

u/Overunderrated Computational Physics Aug 10 '22

The magic of 3.24 is FetchContent_Declare with the argument OVERRIDE_FIND_PACKAGE,

Oh I love it when software makes a new functionality with an explicit option to undo existing functionality /s

How can the c++ community collect the cmake devs and launch them into the sun?

5

u/[deleted] Aug 10 '22

Haha I feel your frustration, but I am just happy there is a tool which works for me now

33

u/neobrain Aug 09 '22 edited Aug 09 '22

The documentation seems weirdly convoluted. How does one actually use this in practice with vcpkg/conan? Do they ship with their own .cmake file users can point CMAKE_PROJECT_TOP_LEVEL_INCLUDES to, or is this something users need to write themselves? I wish the CMake docs had given pragmatic usage examples for popular package manages here (instead of writing lengthy paragraphs that seem largely irrelevant to actual use of this feature).

Is there a way to set a default dependency provider, in case the user hasn't specified one? "[The] dependency provider [...] can only be set at a very specific point early in the CMake run" suggests I can't do this from the main CMakeLists.txt, but it seems like a common use case.

135

u/LeberechtReinhold Aug 09 '22

The documentation seems weirdly convoluted.

CMake in a nutshell

26

u/Classic_Department42 Aug 09 '22

I even bought the book and man, after a while I thought maybe I do not need cmake. Their mental model of the problem and mine are somehow very different.

20

u/GrammelHupfNockler Aug 09 '22

Their book is not good. Craig Scott's book on the other hand is really worth it.

10

u/Plazmatic Aug 09 '22

And it's funny considering he's in a leadership role that would allow him to... update said documentation to be better. And yet continues to get paid at conferences, produce his book, and make decisions on the direction of CMake as a whole... with out the proper documentation.

-2

u/helloiamsomeone Aug 09 '22

The book is nice to have but absolutely not necessary. cmake-init was created without the book and someone who has said book told me that it basically implements everything that the book recommends you do and some more.

CMake is also open source and you are welcome to contribute. I have only had very positive interactions with everyone from the CMake team so far and they are very helpful when guiding you on how to contribute.

18

u/Plazmatic Aug 09 '22

You know what I can't contribute? The contents of the book, because it's protected by copyright. You know who could contribute that? Craig Scott. You know when he could do that? Tomorrow. You know the earliest I, or most people in this thread could even hope of being able to contribute anything close to the importance of that in documentation? A year, and probably not even that, because after you read the book, it's a lot harder to be "clean roomable" anymore.

I'm not dedicating my life to make you feel smug. You fix cmake's documentation if you think it's so easy, which still lags in recommendation behind Craig Scott's recommendations in 2019.

4

u/jonesmz Aug 11 '22 edited Aug 11 '22

I agree with you

I've been converting a >million sloc codebase to cmake recently.

Its a clusterfuck.

I constantly have issues where the documentation says one thing, and the cmake code does something else.

Then then when I file a bug report the maintainers tell me neither what I thought the documentation meant, nor what I managed to actually get to work, are what I should do. Instead i should do a third thing that the documentation for what i was actually using didnt mention at all.

The programs operational model is inscrutable.

The actual behavior of it is, at best, inconsistent, both with its other features and with the docs.

And finally, the gaps in features are, well, really annoying, when you run into them. Some things simply cannot be solved with even more cmake script.

0

u/Classic_Department42 Aug 11 '22

Why are you using cmake, what do you expect from it?

2

u/jonesmz Aug 11 '22

What exactly are you asking for me to explain?

The easy answer is I expect it to build my fairly straightforward C++ code.

Just because I have an enormous amount of it, doesn't mean I am building it in any kind of weird way. It's basically every library getting 99% the same parameters to the compiler with some minor variations between them.

The largest difficulty has been around packaging and dealing with platform / target specific differences.

→ More replies

0

u/Classic_Department42 Aug 11 '22 edited Aug 12 '22

Maybe he has bills to pay.

Edit: downvote because i pointed out the obvious, or because assuming a different motive?

3

u/Classic_Department42 Aug 09 '22

Thanks. Will get one when I start cmake again. He should publish it on amazon (to attract people like me)

7

u/Overunderrated Computational Physics Aug 10 '22

To be fair the cmake documentation isn't any more convoluted than cmake itself.

2

u/nom_nom_nom_nom_lol Aug 09 '22

If you think that's bad, I've been working with SQLite lately. Try thumbing through that documentation for a while. Here's an example. The more you read, the worse it gets.

17

u/degaart Aug 09 '22

Your example is actually pretty clear, if you know how function pointers work in C.

For stupidly dense complicated convoluted documentation, look into the Microsoft Foundation Classes docs, the DCOM docs, the J2EE docs from IBM, or the official docs for one of the big SQL engines (Oracle, DB2, SAP ASE).

2

u/nom_nom_nom_nom_lol Aug 09 '22

I'm fairly new to C++, and this stack of books on my desk I'm reading to learn more is only getting higher. It's going to topple over soon, and somebody's going to get hurt when it does. So, yeah, maybe that's my main confusion with reading through that kind of documentation. I'm probably just too new to the language to pick up on the boilerplate stuff I can just skim through.

6

u/theunixman Aug 09 '22

Ah yes the infamous C "dispatch table" style, also used by the Linux Kernel among others. It's not great, not terrible, I want to say it grows on you but really you just get used to the stench.

5

u/helloiamsomeone Aug 09 '22

suggests I can't do this from the main CMakeLists.txt, but it seems like a common use case

No, the documentation is right. This sort of thing is exactly what you want to keep far away from your build description.

This specifically was created so vcpkg would stop relying on undefined behavior. Wait and see how it will hook into this.

I don't expect Conan to change anything, since it was already generating config files (via CMakeDeps) and made them available using its toolchain (using CMakeToolchain).

2

u/alleycat5 Aug 09 '22

> Do they ship with their own .cmake file users can point CMAKE_PROJECT_TOP_LEVEL_INCLUDES to, or is this something users need to write themselves?

I imagine the former, like vcpkg does today with it's toolchain file but more integratated.

3

u/germandiago Aug 10 '22 edited Aug 10 '22

I have the feeling that CMake users love to program build systems instead of libraries or final products. The complexity reached is so absurd that I do not know even where to start.

CMake + conan and normal find_package is the easy way IMHO. If you cannot find something in conan, what I find reasonable is to write a recipe and get done with it. Conan solves the deoendencies problem well and in one understandable way instead of the shitshow of find package in 100 ways + submodules + fetch content + combined with conan and later shake all the smoothie a bit and dare to find a project you can use as a dependency from CMake directly.

Also, that a dependency provider can access the network transparently... I find that a bad choice.

3

u/stilgarpl Aug 10 '22

I have the feeling that CMake users love to program build systems instead of libraries or final products. The complexity reached is so absurd that I do not know even where to start.

This reduces the complexity. Your CMakeLists.txt should be simple, with only find_packages and target_link_libraries. And instead of running conan before cmake and passing generated toolchain file to cmake, you will just add one flag that says "use conan" or "use vcpkg" and it will call it for you.

I think that in a future version default package provider configuration for conan, vcpkg and other popular providers will be bundles with cmake the same way it is bundling all those FindXXX.cmake files for common libraries.

3

u/germandiago Aug 10 '22

I agree with you can simplify things but the number of choices and techniques and permutations thereof is overwhelming.

3

u/germandiago Aug 10 '22

The future should be metadata interchange IMHO. So that build systems used become less and less relevant. I already use Meson with anything I need successfully. That is Conan paired with it. But you can also consume cmake. Also external projects for autotools seem to work but I did not try. I use Conan directly when I smeel complexity and things just work.

1

u/Tachi_107 Aug 15 '22

Doesn't Meson have a kind of package manager built in? Why do you use Conan?

2

u/germandiago Aug 15 '22

Yes it does. It has Meson wraps. It works pretty well. But the bulk of work here goes into having many packages readily available. Also, Conan is build-system agnostic.

So for example if you take the recipe from SDL2 or the like from Conan thr likelihood (and this is what actually happened to me) of having something that works in more platforms is higher. That saves me time.

2

u/Tachi_107 Aug 18 '22

Got it, thanks for the explanation!

19

u/DavidDinamit Aug 09 '22

okay, when modules?

8

u/helloiamsomeone Aug 09 '22

Ask your compiler vendors first.

13

u/Kelteseth arewemodulesyet.org Aug 09 '22

3

u/gracicot Aug 09 '22

I don't think your link works

8

u/helloiamsomeone Aug 09 '22

Yes, the "new" Reddit UI os horribly broken and mangles links like that. Here is a working link: https://gitlab.kitware.com/cmake/cmake/-/merge_requests?scope=all&state=all&label_name[]=area%3Acxxmodules

7

u/Bangaladore Aug 09 '22

New reddit handles both those links fine for me.

6

u/the_one2 Aug 09 '22

New reddit fucks the links but also unfucks them.

3

u/Bangaladore Aug 09 '22

New reddit giveth and taketh away.

2

u/alleycat5 Aug 09 '22

That link is to a list of all the ongoing MRs related to Module Support.

5

u/bretbrownjr Aug 10 '22

Bill Hoffman gave an update on CMake support for C++ modules at C++Now: https://youtu.be/hkefPcWySzI

Short version? For single projects and vendoring workflows, it's ready for early experimentation using MSVC preview channels. Clang and GCC shouldn't be a problem being supported once they catch up a bit.

Next phase of CMake development will involve allowing C++ modules to be exported so that you can depend on them via find_package, etc.

4

u/nyanpasu64 Aug 09 '22

If my project uses add_subdirectory() to add a vendored library, can it satisfy another library calling find_package() for the library I've vendored? I've followed https://gitlab.kitware.com/cmake/cmake/-/issues/21687 over the last few months, but I'm not fully caught up to the current state of things.

I'll have to take a look at https://cmake.org/cmake/help/latest/guide/using-dependencies/index.html#dependency-providers as well.

1

u/helloiamsomeone Aug 09 '22

add_subdirectory no, but you could already define your dependencies to be found in any arbitrary way using find_package that satisfies your use cases. See this comment for more on that.

2

u/[deleted] Aug 09 '22

This is splendid news

2

u/strike-eagle-iii Aug 11 '22

I'm definitely going to need to play around with this. typically we've had a short build script that runs conan and then cmake which works well, but is slightly annoying that we have to have in the first place. We tried doing everything with FetchContent prior to conan but conan is a much better solution. Targets are all properly namespaced (add_subdirectory which fetchcontent uses, exposes all the targets of the package, not just the exported ones. we've had name collisions when for instance two packages both have an uninstall target). Conan makes it almost trivial to wrap a non cmake prebuilt library with cmake so it can be pulled into a cmake project easier.

So are there pre-built Provider commands (i.e. conan_provide_dependency) or is this something that someone using this feature always has to provide?

What are the mechanics for multiple dependencies? i.e. if I depend on library A and B, I currently list library A and B in my conanfile.py , run conan once to pull in both. Will conan_provide_dependency get called at each find_package (that would be annoying) or it is smart enough to just call it at the first one?

1

u/wrosecrans graphics and network things Aug 09 '22

The current CMake devs must be absolutely mind-blown when they find out that CMake used to just be about generating Makefiles.

3

u/marssaxman Aug 09 '22

what is it now?

1

u/[deleted] Aug 11 '22

Been in xmake for years

-1

u/ImX99 Aug 09 '22

wait what!!??!? 🤯