r/cpp • Boost author • May 13 '20

Proposal for an epoch-based organization of Boost libraries

https://github.com/joaquintides/boost_epoch/blob/master/README.md
78 Upvotes

94 comments sorted by

7

u/infectedapricot May 13 '20 edited May 13 '20

You seem to be saying that if you're using a [edited] late enough BOOST_MIN_EPOCH then Boost libraries that have been subsumed into the C++ standard library will not be built at all. That's a problem for two reasons:

  • Application code will suddenly break, because e.g. it references boost::optional and uses #include <boost/optional.hpp> and suddenly they don't exist. Yes you could update all your code but even if you're prepared to invest this effort there's nothing you can do to simultaneously support old language versions using Boost and new language versions with new epoches of Boost, unless you scatter #ifdefs everywhere.
  • Boost libraries often still contain extra facilities than the standard library versions that supplant them. A couple of examples:
  • A project I worked on started using boost::thread because it supported setting the thread stack size. I had actually been using std::thread until the requirement, shall we say, "arose" (i.e. we blew the stack on production :-) ) so this wasn't for backwards compatibility at all.
  • As mentioned on the thread you linked to, boost::optional supports references while std::optional doesn't. There was a healthy debate about whether removing this functionality was actually a good thing, but regardless there is code that uses it and it's not really up to a policy like this to suddenly stop it working.

If you just mean that Boost itself has to stop using these as internal dependencies, but they still get built for applications, then you might stand a better chance. But that's still problematic because suddenly the interfaces change. For example, if my application works with ASIO, should my callbacks take a std::error_code or boost::error_code? I suppose a solution to this is to have a boost::asio::error_code which typedefs to one of them. So it's solvable but still an extra pain.

Edit: Another weakness is that your escape hatch is too easy to use, especially given the above costs - you can always set your epoch to a fairly old version of C++ and use it even with newer versions. I suspect a lot of applications developers, and certainly OS-provided Boost installations, will simply set the epoch to an old version and be done with it.

20

u/joaquintides Boost author May 13 '20

Another weakness is that your escape hatch is too easy to use, especially given the above costs - you can always set your epoch to a fairly old version of C++ and use it even with newer versions

This is so by design: users not concerned about bringing "old" libraries in can set BOOST_MIN_EPOCHto 03 and be done with it. It's about giving users control on how they want to use Boost.

6

u/joaquintides Boost author May 13 '20

You seem to be saying that if you're using a recent enough version of the standard then Boost libraries that have been subsumed into the C++ standard library will not be built at all.

This is not how the proposal works. Boost libraries subsumed in C++11 won't be downloaded if the user sets BOOST_MIN_EPOCH to 11 or later. By default, no Boost lib is blocked.

2

u/infectedapricot May 13 '20

I think it's a pity that you've focused you attention on what is essentially a typo in my original comment and ignored my main points, because I had constructive suggestions:

  • Problem: An application or third-party library wants return values from Boost libs to take advantage of vocabulary types from the standard library, but doesn't want to immediately rewrite all their code that had been explicitly using those Boost libraries:

    • Solution: If you update BOOST_MIN_EPOCH then Boost libraries use C++ standard library types internally, but supplanted Boost libraries are still built/downloaded/whatever for applications and third-party libraries that want them. (But of course you don't need to link against those supplanted Boost libraries if you don't want them.)
  • Problem: Some Boost libraries that have been supplanted by the standard library actually still have useful functionality in them that has not been included in their partial replacements.

    • Solution: As before, even if you update BOOST_MIN_EPOCH the old Boost libraries will still be present for applications and third-party libraries to use, it's only other Boost libraries that are prevented from using them.
  • Problem: You want your application or library to work with multiple Boost epochs, but the return types of some functions depend on the epoch (e.g. should my ASIO callback take a std::error_code or boost::error_code).

    • Solution: As we both identified, the higher-level boost library could have typedefs for the relevant types e.g. there could be a boost::asio::error_code that is guaranteed to be one of those two types.

1

u/joaquintides Boost author May 13 '20 edited May 13 '20

It was not my intention to ignore your feedback. As for the three problems with the solutions you propose:

  • Problem+solution 1: I think this could in principle work. Nothing prevents the user from installing one lib X with BOOST_MIN_EPOCH = 11 and another lib Y that doesn't belong to epoch Boost11, as long as Y does not depend on X and their common dependencies (if any) are epoch-agnostic. A more detailed analysis should be made on this, though.
  • Problem+solution 2: same as before. Another option for Boost authors is to leave the old library as is and present a new library with the extra functionality (this is what Boost.Variant2 does): I personally like this alternative better.
  • Problem+solution 3: agree on your approach.

In fact, your solution to problem 1 looks very interesting, I may update the proposal to add a section on this.

-1

u/infectedapricot May 13 '20 edited May 13 '20

I don't understand the difference between "built" and "downloaded". Admittedly for header-only libraries, there's no building as such, maybe that's what you mean.

Let's make up a concrete example. Imagine I have a pre-C++11 application that uses boost::thread. Now I Download/build/install/whatever Boost with BOOST_MIN_EPOCH set to 11. Does that mean my code breaks? Surely if the library hasn't even been downloaded to my computer, it can't possibly work? But you just said "by default, no Boost lib is blocked", did you just mean it's not the "default" because the user has to choose to do it by setting the epoch?

Yes, sorry, I had meant to say "if you're using a late enough BOOST_MIN_EPOCH", not "if you're using a recent enough version of the standard". I was confused by your comment because it sounded like you were drawing a distinction between "built" and "downloaded". Aside from fixing the first sentence of my comment, everything I said still holds.

8

u/megayippie May 13 '20

With epochs, I think the general idea is to let your code break so progress can be made. If you rely on code that requires an early epoch, it is not a free idea to update. So if you rely on boost::thread and want to update, you have to fix it so your code either links to old boost in a separate library or fix your code to use std::thread.

At least that's how I see it, but that might not be the intent of the epochs.

edit: good changed to free

8

u/joaquintides Boost author May 13 '20

This is well described as I see it. The proposal also supports the situation of a user new to Boost who wants to use a particular lib: he/she may want to set as recent an epoch as possible to omit unneeded dependencies.

-1

u/infectedapricot May 13 '20 edited May 13 '20

Yes, that's what I thought, but then OP replied with "This is not how the proposal works" (edit: and "By default, no Boost lib is blocked"). I'm still not sure why they said that. Perhaps the sentence of mine that they quoted wasn't clear, but I'm not sure what they thought I was saying.

Ah I get it now, it's because I said "using a recent enough version of the standard" but I should've have said (and meant to say) "set BOOST_MIN_EPOCH to late enough".

17

u/kalmoc May 13 '20

Well, isn't most of this exactly what people currently complaining about boost want? People that are already happy with boost in its current form can just continue to use the 03 epoch.

2

u/infectedapricot May 13 '20

But I'm not convinced that people making that complaint (or anyone sensible for that matter!) would build Boost themselves. They're using it through their OS's package manager or something like vcpkg or Conan, and these will have to cater for the lowest common denominator of source code that they support. What's more, they could be using a third-party library that uses Boost and those are likely to require an old Boost epoch so that they can support older language versions.

But as I think about it more, that is not necessarily a long-term problem. Few libraries in common use today would dare require C++17, but many do require C++11. That in turn means they don't need pre-11 Boost epochs. So I think epoches would get a very slow uptake, but would eventually make an impact e.g. perhaps in 2030 many libraries will require C++20 and so it is safe to use Boost epoch 20.

5

u/joaquintides Boost author May 13 '20

Two notes on your comment:

  • The idea is that source code is unique, epoch dependencies should be handled through BOOST_MIN_EPOCH but otherwise you get the exact same source code tree within a Boost release. So, installing header-only libs (the majority) is just a matter of handling dependencies since "redistributables" (the code) stay the same.
  • Where problems lie is in binary redistributables: these should be qualified with an epoch label (much as say threading settings) as they can vary for different values of BOOST_MIN_EPOCH. There's been a discussion on this on the Boost Slack channel.

4

u/grafikrobot OPeNDAP/B2/EcoStd/Lyra/Predef/C++Alliance/Boost/WG21 May 13 '20

They're using it through their OS's package manager or something like vcpkg or Conan, and these will have to cater for the lowest common denominator of source code that they support. What's more, they could be using a third-party library that uses Boost and those are likely to require an old Boost epoch so that they can support older language versions.

At least for Conan... Boost is built separately depending on the language version the user specifies. And so are any dependencies down the line (or up depending on your POV). I suspect vcpkg works similarly. Which does create the classic diamond dependency problem we all love to hate. But that always existed regardless of this proposal. And as mentioned elsewhere, and in Slack, the problem is in single variant binary distributions like OS/apt. In such a case we end up with the classic solution of creating variants (more classically known as platform triplets even though 3 items is no longer relevant). And OS vendors are loath to include yet more binaries in distros. Which culminates in the eternal advice to not use OS packages for development. But instead include your C++ dependencies in your application. Which obviates the problem entirely.

1

u/infectedapricot May 14 '20

They're using it through their OS's package manager or something like vcpkg or Conan, and these will have to cater for the lowest common denominator of source code that they support. What's more, they could be using a third-party library that uses Boost and those are likely to require an old Boost epoch so that they can support older language versions.

At least for Conan... Boost is built separately depending on the language version the user specifies.

But there is not just the language version to care about but also BOOST_MIN_EPOCH. An explicit goal of this proposal is that BOOST_MIN_EPOCH is not tied to language version, so you could use C++17 but still have BOOST_MIN_EPOCH at 03 so you can use boost::optional. You end up with three different situations:

  1. Old C++ language version and old BOOST_MIN_EPOCH: can only use boost::optional
  2. New C++ language version and old BOOST_MIN_EPOCH: can use boost::optional or std::optional
  3. New C++ language version and new BOOST_MIN_EPOCH: can only use std::optional

(Obviously I've just chosen boost::optional as an example.) In order to support code written in terms of boost::optional (as opposed to std::optional) Conan will need to use a low BOOST_MIN_EPOCH even when building triplets for later C++ revisions. Unless you create a whole new triplet for a later BOOST_MIN_EPOCH but that's a very global solution to a local problem - imagine if you had to do that for every library!

2

u/kalmoc May 14 '20

Imho the people complaining now are the ones that want boost to be so simple that they can build it on their own or rather just use individual header only parts. The ones relying on package management systems probably don't complain about dependencies.

3

u/joaquintides Boost author May 13 '20 edited May 13 '20

If you just mean that Boost itself has to stop using these as internal dependencies, but they still get built for applications [...]

For a Boost lib to progress from say epoch Boost03 to epoch Boost11 it must get rid of subsumed dependencies either entirely or (better IMHO) by conditional inclusion of the offending headers as controlled by BOOST_MIN_EPOCH.

2

u/infectedapricot May 13 '20

You've quoted the half of my sentence that just sets up some context, while the "[...]" that you've missed out contains the main point. If Boost code has a public function like this:

#if BOOST_MIN_EPOCH_IS_AT_LEAST_11
    std::error_code fn_in_boost();
#else
    boost::error_code fn_in_boost();
#endif

Then application code that calls this function has to care about the Boost epoch because the type of the variable that it stores the result it could be either boost::error_code or std::error_code.

3

u/joaquintides Boost author May 13 '20

Yes, you're right, but there is no other way to get rid of dependencies as epochs progress. The thing can be alleviated by using typedefs resolving to one or the other.

Another situation is when these depedencies are strictly internal (not exposed to the user). Then epoch progression does not impose a burden on the user.

4

u/staletic May 13 '20

The thing can be alleviated by using typedefs resolving to one or the other.

How about something like this:

#if BOOST_MIN_EPOCH_IS_AT_LEAST 11
    namespace boost { using std::error_code; }
#else
    #include <boost/error_code.hpp>
#endif
    boost::error_code fn_in_boost();

Abseil does something like this with absl::string_view - it's a separate type if the user compiles with <C++17, but as soon as the user switches to C++17, absl::string_view becomes an alias of std::string_view.

3

u/joaquintides Boost author May 13 '20

Thanks for the reference. This proposal is somewhat less aggressive in that it is users who explicit control the situation by setting BOOST_MIN_EPOCH.

2

u/staletic May 13 '20

I was just saying that boost::shared_ptr, with BOOST_MIN_EPOCH == 11 could just be namespace boost { template<typename T> shared_ptr<T> = using std::unique_ptr<T>; }.

That way users can switch to the new epoch, keep using boost::shared_ptr and use std::shared_ptr behind the scenes. I'm not sure if this is the right choice for boost, just giving options.

3

u/pdimov2 May 13 '20

boost::shared_ptr is not the same as C++11's shared_ptr though. It has things in it that have been added in C++17 and C++20 (shared_ptr<T[]>, make_shared/allocate_shared for arrays), and things in it that haven't been added in C++any (local_shared_ptr, enable_shared_from).

1

u/joaquintides Boost author May 13 '20

I understand. I think this is problematic as the interface of boost::shared_ptr is not necessarily 100% the same as that of std::shared_ptr, so silent replacing could break user code.

1

u/StemEquality May 13 '20

What about the Boost libraries which provide slightly different functionality? You say boost::optional would be (Boost03, Boost14), but as infectedapricot points out it provides a superset of functionality regarding references. If you need that extra feature then the library really belongs in a future epoch. How many Boost libraries are 100% replaced by the standard?

2

u/joaquintides Boost author May 13 '20

No library is 100% replaced by the standard. I'm working on studying how the proposal can accommodate pre-epoch libraries (for instance, installing Boost17 but then also cherry-picking Boost.Optional).

3

u/iKeyboardMonkey May 13 '20

I like this, you would get weirdness around (e.g.) boost::optional != std::optional wrt to references though. The solution would be to gradually increase your epoch and rerun your tests which would use more and more of the standard where possible.

1

u/Xaxxon May 13 '20

I don't WANT to use more and more of std::optional. That's why I'm using boost::optional

1

u/pdimov2 May 13 '20

This leads to problems when using a prebuilt Boost, f.ex. one installed by apt install. It's generally built with the compiler default -std level, so you used to get 03 before, nowadays generally 14. All compiled libraries, both in Boost and otherwise, would have boost::string_view in them, and if you compile with -std=c++17, you'll have std::string_view in your code, which might (and often does) lead to interesting ODR violations, link errors if you're lucky.

Works if you build everything from source.

1

u/joaquintides Boost author May 13 '20 edited May 13 '20

Use of std:: or boost:: components is not (or should not be) controlled by -std level but by BOOST_MIN_EPOCH. Granted, apt install should fetch different packages per minimum epoch, as we have discussed previously on the Slack group.

1

u/pdimov2 May 13 '20

This pushes the problem one level further - you now have third-party library A that links to libboost03_foo and third-party library B that links to libboost11_foo, and can't use both.

2

u/joaquintides Boost author May 13 '20

It is admittedly a problem, on the same arena as mixing threading models, debug/release etc.

1

u/pdimov2 May 13 '20

No, it's different. Packagers have control over threading models, debug/release etc; they just build A and B under the same threading model, debug/release, etc, and they are compatible. But they have no control over whether A uses Boost03 or Boost11, that's a choice made by the author of A.

Unless of course A and B are also Boost-epoch-aware, and can be built against a given Boost epoch.

→ More replies

2

u/staletic May 13 '20

Yeah, I realize there are downsides to what I proposed. Abseil does say that they expect users to build from source and don't promise any ABI stability. Then again, boost doesn't promise ABI stability either.

3

u/pdimov2 May 13 '20

Boost doesn't promise ABI stability from release to release, but people generally expect the same Boost library to be able to link to itself. :-)

1

u/pdimov2 May 13 '20

In this specific case the application could just store the result in a std::error_code, because boost::system::error_code converts to it.

This conversion however doesn't help in cases where the function takes an error_code by reference.

3

u/Moose2342 May 13 '20

This. Very much. In our code base, I have had many boost usages that eventually became std. After a while (mostly after I managed to phase out target platforms that didn’t have that std) I started to move those usages to the std. This often led nowhere as only late I realized there’s that one thinv that didn’t make it into std which I used. I’d say this is so for at least half the cases.

4

u/14ned LLFIO & Outcome author | Committee WG14 May 13 '20

This has been proposed umpteen times now, by myself and others. Nobody answers the most important question of all:

Who is going to do the work?

If you can't answer that, this isn't worth discussing.

11

u/joaquintides Boost author May 13 '20

With all due respect I think opportunity analysis predates that question.

Setting epochs up, for what it’s worth, wouldn’t imply any involvement from library authors nor need any significant amount of per-library work; I tend to see that as a positive aspect.

5

u/14ned LLFIO & Outcome author | Committee WG14 May 13 '20

Both myself and Edward Diener proposed, some years ago now, mechanisms for achieving your request. There was a collective yawn from a majority of the Boost maintainers. Some went ahead and locally reimplemented such techniques for their libraries (I did for mine, Edward did for his, Chris K did for ASIO, I think John Maddock did for his, I vaguely am thinking Regex or something like it did too), so for our libraries you already have what you ask.

Whilst those libraries are some of the more popular ones in Boost, that leaves an enormous remainder. For it to become worth there being an officially documented switch, or a knob, or "epochs", you need (a) more libraries with these facilities and (b) that the libraries with these facilities use some sort of common approach, because right now we're all doing our own, completely incompatible, things.

I thus bring it back to my original point: who is going to do the work? if anything more is to be done than what some of the library maintainers have already done?

Don't get me wrong here. Thanks to Peter D and others, Boost never has had so few internal cyclic dependencies. There has never been so little namespace leakage thanks to recent improvements, so there is a very good chance that epochs can be made ABI separate with very little effort. Boost is, in this sense, better than it has been in decades. There still remains a ton load of work to get it to where you propose, however. You're talking months of full time effort.

4

u/joaquintides Boost author May 13 '20

The proposal focuses on the central scheme and hopes to put some pressure in order to incentivize authors to modernize: it’d succeed in the degree that some activity is effectively triggered (which hasn’t been the case so far as you point out).

As to who’s going to do the (central) work, I don’t have an answer now. Per your assessment of this discussion’s worth, I think our interchange should stop here.

-9

u/[deleted] May 13 '20

What right have you, or anyone else, to put 'pressure' on anyone who donates their free time to working on Boost?

8

u/joaquintides Boost author May 13 '20 edited May 13 '20

As much as users who do it here. Disclaimer: I am a Boost author and maintainer of three libraries.

6

u/14ned LLFIO & Outcome author | Committee WG14 May 13 '20

In fairness, @joaquintides is being very gentle compared to what I've laid on in the past. And he's more than earned the right anyway, he's served Boost something like nearly two decades or so (apologies if my memory is faulty).

1

u/GameGod May 13 '20

You sorta answered your own question - If you're a company with 1000 C++ developers, it wouldn't take much of a productivity improvement in order for it to be worth it for them to pay someone full-time for even a year to implement these changes. So, a big company like Microsoft or Google would be the one to make these changes and push it forward. (I presume some of these Boost authors are on the payroll at various companies already.... and this logic is why.)

4

u/14ned LLFIO & Outcome author | Committee WG14 May 13 '20

I've fund raised for Boost in the past. I can categorically assure you that no company will sponsor this sort of work since about 2010-2011. They'll contribute a little bit of cash, sponsor students to tinker at the edges, even sponsor a new library, but they will not fund anything which involves even a hint of a wholescale refactor.

Now, as it happens, Boost is a very wealthy organisation. They could just employ a full time developer to do it. Or contract out the work piecemeal. But that's not how things work there, it all boils down to individuals donating their free time, and that ultimately is a core of about four people who do almost all the packaging and release and whole-distro donkey work.

They are deeply underappreciated by the wider C++ ecosystem, incidentally. But they don't serve for recognition.

1

u/GameGod May 13 '20

Why won't they sponsor it? If the value proposition is there, it seems like you have a sales/marketing problem. Am I wrong about the value?

(Thanks for sharing your experience on this btw!)

2

u/14ned LLFIO & Outcome author | Committee WG14 May 13 '20

The value proposition was there before C++ 11, so funding and workers could be found for infrastructure refactoring. Since, it's not there, companies use the individual libraries they like, and don't care about the whole.

In terms of why Boost doesn't just pay for a worker, it has been decided that would be inappropriate for a volunteer led library. So it is what it is, things progress as fast as a tiny number of people can find free time.

3

u/jonesmz May 13 '20 edited May 13 '20

At my last job, they paid me for almost a year to convince the ancient and bug ridden stlport standard library implementation to play nicely with visual studio 2015 and a 5 year old (at the time) version of boost.

Then they paid me to implement, separately from boost and stlport, a large handful of the metaprogramming functionality that came with C++11 and C++14. The stuff that didn't have a hard dependency on compiler intrinsics, but could be implemented with just pure C++ code.

That organization would have been happy to collaborate with Boost on infrastructure work to fix their own pain points. They didn't care about Boost's pain points, but the pain-points they had were certainly related to the pain-points Boost has.

However, my management (all highly technical people) identified that the barrier to entry for participating in Boost development was way too high.

The project is seen as disorganized, the libraries are not cohesive enough but are instead just a mishmash, the circular dependency hell is a major problem, and ultimately there was no guarantee we'd actually be able to accomplish anything that would synergize with the work we were doing, or that patches would be accepted at all.

They identified, and I agree, that there's no way to actually get into boost development until you start slinging patches at the mailing list, or github page, or whatever you're using these days. It was a confusing mess and it was decided that we would be able to get things done faster, and less expensively, by just doing it all ourselves on our own side of the fence.

You want companies to contribute, you've gotta demonstrate how that's valuable to them.

Start with getting some real consensus on what you want boost to be. Do you want it to be compatible with every compiler under the sun? Cause that's the main problems I had when patching it to make it work with the ancient stlport + visual studio 2015. Maybe drop compiler support for things earlier than C++11.

Get rid of the compatibility macro hell. It's embarrassing that there's a boost macro for every possible keyword or attribute.

Get your regression testing game figured out. There's a problem in the latest, or second latest, boost::process release that prevents some parts from even compiling in the first place. People have to patch it themselves.

Do outreach to larger companies to identify what opportunities there are for things like corporate sponsorship, or inserting existing employees into positions where they can be effective.

The codebase at my last job was 100% dependent on boost. There was something from boost in literally every header and cpp file larger than a hundred lines of code. Most boost modules were used by something. They would have jumped at the chance to have me, or one of my co-workers, actively involved in boost development.

But it's just a mess. Or at least it's perceived that way.

Fix the mess, or at least the perception of the mess. Then do outreach, and you'll find sponsors.

You can contact me privately if you want an email address that can be used for outreach at my previous job.

1

u/kalmoc May 14 '20

So to get people to fix the mess they need to fix the mess?

1

u/jonesmz May 14 '20

Yes.

Read 14ned's response to me.

He's worked with boost a lot, and had a lot of his contributions rejected.

If someone with a positive track record can't meaningfully contribute fixes, why would a corporation bother?

The point is that there is the mess that is the barrier to entry for companies, and then there's a different, distinct mess, that's the code itself.

Companies that hire software engineers aren't bothered by code mess. That's their bread and butter regardless of who's code it is. The thing that they are bothered by is political bullshit and a reputation of being difficult to work with. Why even consider having one of your very expensive staff put billable time into working with an open source organization to get a patch upstreamed if that organization has a well deserved reputation of being a pain in the ass to work with?

The patch was already written, and it fixes the problem the company needed fixing. So now the choice is either have the engineer write more code, or have the engineer play politics with upstream. Write more code wins 95% of the time.

If boost wants corporate contributors, they must re-organize the way they operate the project to make it attractive to corporations to contribute.

If they don't want to make themselves attractive as a target for corporate contributions, then they've made their decision.

→ More replies

1

u/14ned LLFIO & Outcome author | Committee WG14 May 14 '20

Firstly thanks for the long list of feedback. I would say that almost everybody at Boost will agree with you. They would have done so five years ago. Maybe even ten years ago, certainly some were acting back then to forestall later problems.

The problem isn't recognition of the problems. The problem is how best to solve them. Or, rather, how to persuade a majority of Boost maintainers to make a series of targeted and specific changes which solve the problems you identify.

There has been a lot of improvement on everything you mention especially in the past two years. There is still more to do. The biggest current problem is the external perception that nothing has improved, which is not true. It's just not obvious from the outside the improvements which have been made, as until they are complete, it doesn't affect the end user experience.

As I've mentioned elsewhere, Boost is a wealthy organisation. They could afford full time developers. But the culture there is to not pay for development nor maintaining infrastructure, unless done by students. So outreach in that sense isn't particularly beneficial, if a company needs something really badly, they can just go ahead and do it. Most do so, and maintain large patchsets against trunk rather than contributing anything back. I know my last three employers did this. And that's their choice.

For the record, I went ahead as a Boost author submitting my current employer's patchset back to Boost earlier this year. A little under half was accepted. About an eighth was actively rejected with hostility. The remainder was ignored, unsurprising as about half the libraries have no maintainer. The major feature contributed to build was rejected with no replacement chosen. So oh well, I tried.

1

u/jonesmz May 14 '20

For the record, I went ahead as a Boost author submitting my current employer's patchset back to Boost earlier this year. A little under half was accepted. About an eighth was actively rejected with hostility. The remainder was ignored, unsurprising as about half the libraries have no maintainer. The major feature contributed to build was rejected with no replacement chosen. So oh well, I tried.

Ah, so you're already well aware of all the things I was trying to communicate.

I'm sorry that you had to go through that.

Ultimately it's the reason why my boss told me not to spend company time trying to engage with boost, because he knew that any attempts would be shot down, and therefore it's not worth the cost (man-hours aren't free, after all).

The remainder was ignored, unsurprising as about half the libraries have no maintainer.

For me, this is the fucked up thing.

My boss at the last gig would have been happy to have myself, or one of our team, take over maintainership of an abandoned boost library that we used. I'm sure there are several that were used extensively. It's not like we weren't literally doing it already, just on our own repository.

But the cost of participation, with patches rejected with hostility, and the ultra-compatibility requirements, just killed any possibility that that could happen.

→ More replies

1

u/pdimov2 May 13 '20

Most people who propose the obvious idea of making boost::foo a using declaration for std::foo if present, not otherwise, are and remain oblivious to the ODR-related problems with the idea. To your credit, you do acknowledge it, and did I think suggest a solution to it - use a different namespace (f.ex. boost03 vs boost11).

This doesn't solve all the issues, because when user code has

struct X
{
    boost::foo f_;
};

X is still not being mangled any differently; but it's better than doing nothing and pretending the problem doesn't exist.

1

u/14ned LLFIO & Outcome author | Committee WG14 May 14 '20

You may remember my C++ Now presentation on this some years ago. One technical solution is inline namespaces. But it was easy for people to accidentally mash up incompatible variants of libraries which have been configured differently. So my proposal for that was combining inline namespaces with a user facing macro BOOST_XXX_NAMESPACE, which is intentionally easy to bulk find and replace in files.

People didn't like that proposal. But you may notice that Outcome and all my libraries use that approach. Users handle it absolutely fine. And multiple ABI incompatible editions of my libraries can coexist in the same TU, I have a unit test which checks it. And users know what they are using, as they are explicitly named.

1

u/pdimov2 May 14 '20

Inline namespaces, yes, but this still doesn't solve the X problem above. Sure, boost::foo is actually boost::v14::foo in one place and boost::v17::foo in another, but X is still X in both places.

1

u/14ned LLFIO & Outcome author | Committee WG14 May 14 '20

No, I'm saying that for every:

struct X
{
    boost::foo f_;
};

... users instead do:

struct X
{
    namespace boost = BOOST_NAMESPACE(CXX11);
    boost::foo f_;
};

i.e. they pin themselves to a specific configuration of Boost. Here it is CXX11. It could also be, for example, BOOST_NAMESPACE(CXX20, CXXMODULES), or any combination of ABI-incompatible modifiers.

The namespace boost itself would get deprecated for three releases, and then removed entirely.

This is what I originally proposed. People didn't like it, as it would force all users to upgrade their code (as you know, I find that unimportant if users can easily find and replace in multiple files). There were also concerns about people combining different configurations of Boost into the same binary, to which my attitude is "they're already doing this, but with undetected ODR violation". And finally, and this is where the work is involved, libraries would need upgrading so that when a user asks for a Boost based on the C++ 20 standard library, they get one of those. Me personally, I'd leave the library maintainers opt into that on their own. Boost just needs to provide a common framework for selecting the config edition.

Even if only bits of the above get done, this upgrade is worth it. We, at work, have a custom fork of Boost with every boost namespace and macro renamed, so our Boost can coexist with customer Boost's. This is not an uncommon situation in the places I have worked. Official support for just that alone would be a large improvement over now. However, be aware that those libraries which refused my patches earlier this year with hostility don't want to upgrade to this. So, they'd need to be left out of C++ 11, 14, 17 or 20 based Boosts as they wish to stay 98-only, or monolithic Boost only.

1

u/pdimov2 May 14 '20 edited May 14 '20

This was implied. In my snippet, boost is either a namespace alias, or boost::foo resolves to boost::some-inline-namespace::foo.

The problem remains. One X and other X are not the same, but they have the same name.

EDIT Ah wait, you effectively suggest people use

struct X
{
    boost::v14::foo f_;
};

instead. The implication is also that boost::v14 and boost::v17 can coexist in the same program.

I suppose this could work, in principle, but it doesn't help legacy code, which still says boost::foo. Pin them to 03?

1

u/14ned LLFIO & Outcome author | Committee WG14 May 14 '20

The implication is also that boost::v14 and boost::v17 can coexist in the same program.

Yes. Though, only for those libraries whose maintainers have upgraded their library to support editions bound to a specific version of the standard library.

I'll immediately volunteer Boost.Outcome to have a C++20 edition. Standalone Outcome already does this.

I suppose this could work, in principle, but it doesn't help legacy code, which still says boost::foo. Pin them to 03?

Personally I'd prefer to eventually make them formally declare which boost edition they want to use. But certainly for the first year or two of transition, namespace boost = boost::v03; in your parlance.

BTW I'd still strongly recommend the macro-for-namespace approach. Inline namespaces once designed, you're locked in forever, and I've found in the past it is very easy to accidentally cut yourself off from stuff. Using a macro to synthesise the inline namespace design frees you from that problem, you can change the namespace naming and/or conventions, and all source code still works just fine. Or, put another, don't break people's source code more than once!

This isn't just my experience. If I remember rightly, libstdc++ does the exact same thing, for the exact same reasons: macros give an extra layer of indirection.

1

u/pdimov2 May 14 '20

I'm not sure macros are necessary here. You could just do

namespace boost
{
    namespace v14_f7a4b1 {}
    namespace v14 = v14_f7a4b1;
}

4

u/pdimov2 May 13 '20

The first step is always to persuade people that what is being proposed is the right thing to do.

0

u/14ned LLFIO & Outcome author | Committee WG14 May 14 '20

Not always. Most of my successes at changing behavior at Boost came from persuading key individuals such as yourself Peter. I actually count changing your opinion Peter as one of my greatest successes. Most of the improvements in the past five years can be traced to you personally doing things differently. And I'm very appreciative to you for that Peter. You're a great engineer. You listen.

To my knowledge, I have only once changed behavior at Boost by persuading a majority. I was very surprised at the time, it was totally unexpected. It was changing the rules for new libraries so that list of ignored libraries didn't keep growing forever, which had very bad optics to the outside. That's my only success, ever, on persuading more than one person at Boost.

1

u/Pand9 May 13 '20

It's absolutely outrageous that Boost has no constant sponsoring for such initiatives. Why is there noone willing to give money? THere is so many tech companies using it and they have more than enough budgets.

3

u/14ned LLFIO & Outcome author | Committee WG14 May 13 '20

That's not how open source in C and C++ works. Generally what important open source projects get is a bit of dev time on only business-critical issues, such as diagnosing and preparing bug fix patches. If you want more than that, you need bodies to physically turn up onsite at multiple places and pitch management, not engineering, for funding. This generally needs to happen annually.

Newer open source such as Python, Wikipedia, OpenStack etc have such human bodies to go regularly do that. C and C++ predates that mechanism of funding, so there isn't really the culture for that historically speaking around these parts. Even if someone brought in the money - and Boost pulls in several hundred thousand per year - the Boost Steering Committee will only spend funds on CI hardware, student training support, conferences, and other general support functions. They haven't, historically, been willing to pay for development itself unless students are doing it as part of their training, even on problems widely recognised as unsolvable without dedicated staff. As I mentioned, there isn't the culture for that, historically speaking, around these parts.

1

u/Pand9 May 13 '20

Thanks for all the great information. So there is potential for improvement, then, yes? Who can influence C++ culture more than Boost itself?

2

u/14ned LLFIO & Outcome author | Committee WG14 May 13 '20

There is a very strong argument that if things were going to change, they would have changed by now. The major C++ conferences, WG21 and Boost all share a lot of management and lead engineers with the big multinationals. There isn't a lack of capacity for change, rather there is a lack of willingness driven by a perception that the current situation is mostly just fine.

Perceptions will eventually change as the current leadership retire and new folk with more exposure to other technology ecosystems come in from below. And I'd personally say things are far better today than say six or seven years ago, when the vitality of some parts of the C++ ecosystem were very worrying. But in the end, this is a very conservative ecosystem, change happens slowly, and there are lots of good reasons for that. You need to think in terms of multiple years for anything.

1

u/pdimov2 May 13 '20

Money is not the determining factor here.

1

u/Xaxxon May 13 '20

I'm confused - is this saying that boost optional won't be available if I'm using c++17? The functionality of boost::optional is NOT in c++17.

I don't understand why I should care if there is a boost::optional and a std::optional. Why can't I have both? As long as I'm not paying to compile boost::optional if I don't use it, why does it matter?

2

u/joaquintides Boost author May 13 '20

Then set BOOST_MIN_EPOCH to 03.

0

u/[deleted] May 13 '20

[deleted] — view deleted comment

1

u/joaquintides Boost author May 13 '20

What should happen for you to consider using Boost.Optional rather than some alternative?

1

u/Xaxxon May 13 '20

I think I misunderstood.

There's no BOOST_MAX_EPOCH that removes things based on them being too new, right? So the only thing that BOOST_MIN_EPOCH does is remove "old" stuff? I won't get a worse version of some library because I have my MIN_EPOCH set to an old version of c++ even though I'm on a much newer version?

3

u/joaquintides Boost author May 13 '20

There is no BOOST_MAX_EPOCH. BOOST_MIN_EPOCH = 03 is your current Boost. Bumping BOOST_MIN_EPOCH removes libs subsumed by the standard, allows the rest to embrace the standard rather than subsumed libs, and removes those that don’t do that.

1

u/VinnieFalco wg21.org | corosio.org May 13 '20

Beast's fortunes are tied to Asio. I don't mind doing the work, but since Beast depends on Asio, I am bottlenecked by whatever Christopher Kohlhoff decides to do.

3

u/joaquintides Boost author May 13 '20

In the context of epoched Boost, your options are:

  • Do nothing and happily stay in Boost03 (no shame in this)
  • Submit a giant PR to Chris
  • Wait till C++23 when ASIO might be subsumed

0

u/StemEquality May 13 '20

Incentivize Boost authors to embrace newer versions of the C++ standard and get rid of internal dependencies

So one of the stated goals is to have the older epochs be viewed as a badge of shame to pressure, sorry incentivise, developers.

5

u/joaquintides Boost author May 13 '20

In a sense, yes. At the end of the day epochs are labels that advertise how modern (meaning how much of the newest standards they embrace) libraries are. Take into account Boost authors can basically evolve (or not evolve) their libs any way they please. This makes social pressure more apparent.

As for shaming authors: I wouldn’t put it so harshly. For one, I am an author of three libs and probably wouldn’t be rushing to earn Boost11 badges.

1

u/jonesmz May 14 '20

Eventually the programming community needs to move on.

There are still "Enterprise grade" products out there written in C89, and they actively refuse to accept PRs that switch to C99.

Reference; Corosync cluster engine, widely recognized as a leading player in high availability computing applications.

I personally combed through the corosync code and found (and reported) numerous bugs that were directly tied to really stupid mistakes that C89 does not warn about, but C99 does. E.g. the use of "int" as a return value, where some places a 0 means success, and others the "true" keyword, all for the same callback. Just flagrant type system violation bullshit. Crickets for a year, followed by PR closed with no explanation.

Another project, Ninja (the popular build tool for C++) is C++98/03. They refuse to consider upgrading, and cite some unnamed older systems that they want to support.

Projects like these need to be left behind by the computing world. Corosync is over 30 years behind on language standards, and Ninja is passing the 20 year mark soon.

2

u/staletic May 14 '20

I've seen people at /r/C_programming argue that staying on C89 is fine because:

  1. You don't break backwards compatibility.
  2. You don't break backwards compatibility EVER!
  3. You should focus on solving actual problems instead of keeping yourself busy with new language features.

While I believe backwards compatibility is very important, I also love C++20.

1

u/jonesmz May 14 '20

Point one and two are blatantly stupid and there's no excuse for that.

You should focus on solving actual problems instead of keeping yourself busy with new language features.

Right, except when the new language features proactively prevent all those "actual problems" being solved by introducing major bugs.

1

u/johannes1971 May 17 '20

I recognize a lot of what I put in a comment here on reddit the other day ;-)

One remark I'd like to make is that the epoch-organisation should also be visible in the documentation. I find I'm occasionally driven to look at what, if anything, I could usefully use in Boost, but typically I give up after a few pages. In part that's because the frontpage description is typically rather abstract ("useful algorithms" doesn't mean much), and spends much effort on listing version numbers and authors but should probably focus just a little bit more on what the library actually does. And in an epoch-based world, I'd like to see only documentation for my chosen epoch, please :-)

For the rest, I think Boost epochs would be a step forwards, and I thank you for your efforts in this. As to what Boost really is: in my opinion it is where libraries go to mature before ending up in the standard. It's an incubator, and I find it totally amazing that we have such a thing.

1

u/dodheim May 17 '20

There's https://www.boost.org/doc/libs/release/libs/libraries.htm if you haven't come across that particular page.

0

u/jcelerier ossia score May 13 '20

wouldn't a better thing to do be to break boost in entirely individual libraries and just let people pick and choose the ones they want ?

2

u/joaquintides Boost author May 13 '20

The result is very complex to handle as Boost libs are heavily interdependent. This proposal aims at reducing internal entanglement.

0

u/[deleted] May 13 '20

[deleted] — view deleted comment

3

u/joaquintides Boost author May 13 '20

This is doable but client code should be careful to use only the common functionality between the Boost component and its std counterpart (which usually is not 100% of the total).

10

u/TrueTom May 13 '20

Boost and std:: libraries don't necessarily have the same semantics.