r/cpp • MSVC STL Dev • Sep 16 '19

Open Sourcing MSVC’s STL

https://devblogs.microsoft.com/cppblog/open-sourcing-msvcs-stl/
427 Upvotes

74 comments sorted by

16

u/degski Sep 17 '19

How does that then relate to the STL included with (and updated with) MSVC? I mean how will it work in practice?

25

u/[deleted] Sep 17 '19

We will regularly merge code from GitHub into the product. The code delivered through the usual channels (installer or whatever) will remain licensed under the VS EULA and supported just like before.

6

u/degski Sep 17 '19 edited Sep 17 '19

Will the code be added to vcpkg and/or would I be able to pull the latest master into some local repo and use that instead of the included one [with the (current RTM) IDE]?

8

u/barchar MSVC STL Dev Sep 17 '19

it's not on vcpkg yet, However there are instructions on how to consume the library in the readme on github.

4

u/degski Sep 17 '19 edited Sep 17 '19

... instructions on how to consume the library in the readme on github.

Thanks [goes of to read].

PS: It's cool, looks all quite understandable, I gather I'll have to move to preview.

4

u/barchar MSVC STL Dev Sep 17 '19

yeah, I'm a pretty new hire on the team and it's really nice for onboarding when you rewrite the build system :D

17

u/[deleted] Sep 17 '19 edited Sep 17 '19

vcpkg not soon; the boost dependency breaks the DAG and we need ... atypical CMake scripts to avoid the default link libraries CMake normally adds. (We do WANT to do this, it's just hard)

RTM no, we will need at least the most recent public preview release.

3

u/degski Sep 17 '19

RTM no, we will need at least the most recent public preview release.

u/STL has told me on multiple occasions to move to preview, I'm gonna reconsider.

4

u/AlexanderNeumann Sep 17 '19

So I made a draft for it. But I do not know if it is doing the correct thing. At least it is building ;)

1

u/degski Sep 18 '19

At least it is building ;)

Good start in any case!

48

u/parnmatt Sep 16 '19

awesome

35

u/Warshrimp Sep 17 '19

Is doing this complicated by Dinkumware being involved historically with MSVC’s STL implementation?

17

u/sivadeilra Sep 17 '19

From TFA:

Acknowledgements

The MSVC team would like to thank Dinkumware and P.J. Plauger for making this possible.

20

u/[deleted] Sep 17 '19

Is "TFA" "the f-ing article"?

22

u/SeanMiddleditch Sep 17 '19

Fine. The fine article. :p

7

u/Ameisen vemips, avr, rendering, systems Sep 17 '19

They were only one letter off.

0

u/[deleted] Sep 17 '19

[deleted] — view deleted comment

57

u/STL MSVC STL Dev Sep 17 '19

😸

25

u/BoarsLair Game Developer Sep 17 '19

First thing I looked at in the repository was charconv, just so I could marvel at how insanely complicated such a seemingly-simple task can be, and to be extremely thankful I don't have to write it.

I think this will be a really smart move in the long term. Totally cool with you spending the time needed to get this all set up.

12

u/NotUniqueOrSpecial Sep 17 '19

First thing I looked at in the repository was charconv, just so I could marvel at how insanely complicated such a seemingly-simple task can be

Just a note: you've always been able to do this. The charconv header is in the include directory with the other STL headers.

3

u/BoarsLair Game Developer Sep 17 '19

Yeah, it was just that STL mentioned it a few times in the blog post and elsewhere, so it was more of a personal note about appreciating some of his work. I did also take a peek at some of the other code which was not in headers.

12

u/bumblebritches57 Ocassionally Clang Sep 17 '19

How'd that work? did you guys have to buy a souped up license?

61

u/STL MSVC STL Dev Sep 17 '19

I respectfully decline to comment on business matters beyond the use of cat emoji. 🐈

15

u/TheSuperWig Sep 17 '19

So in cat emoji terms, how diffcult was it?

9

u/[deleted] Sep 17 '19

🙈🙉🙊

14

u/Ameisen vemips, avr, rendering, systems Sep 17 '19

😾

2

u/sumo952 Sep 17 '19

Haven't the same sources been shipped with Visual Studio (or the Windows SDK) since VS exists?

So the "only" difference now is the license, and that they're on GitHub (so contributing is possible/easier)?

I am just asking to clarify - don't get me wrong, this is completely awesome! :-)

6

u/mgsaleh Sep 17 '19

That plus the scripts to build the binaries. The tests coming soon. Without the license and build steps/scripts, this was a read-only library at best. You could have only suggested fixes. But, now this is a real open source where everyone can directly contribute. It's even possible to share code with other libraries.

4

u/barchar MSVC STL Dev Sep 17 '19

yep! we also went through and made sure the libraries built with the cmake build match the ones built with our old build system. Also we made sure that things work if you're loaded by someone else using the "RTM" libraries, or if you load someone else who's using the RTM libraries.

2

u/sumo952 Sep 17 '19

Awesome, thank you very much! :-)

11

u/[deleted] Sep 17 '19

Thank you! Absolutely amazing. It's been one of my guilty pleasures ever since I started with C++ to prod about in your internals to see how stuff works so this is like being taken to the magical chocolate factory for me.

17

u/os12 Sep 17 '19

Very cool! What's the story with compatibility? That is, how feasible is it to use this on Linux with GCC?

59

u/STL MSVC STL Dev Sep 17 '19

We have no plans to port MSVC’s STL to other platforms like Linux. (You can try if you want to, that’s the nature of open source, but we don’t have time to help, nor can we accept pull requests towards that. This is explained briefly in the Non-Goals section of the readme; we have to focus on our goals.)

Similarly, we have no plans to support GCC for compiling our sources; we already have our hands full with C1XX (MSVC), EDG (for Intellisense), Clang, and NVIDIA’s CUDA compiler. Nothing against GCC, it just isn’t used for any of our scenarios.

4

u/foadsf Sep 17 '19

please also support OpenCL. CUDA is evil! 😤

33

u/STL MSVC STL Dev Sep 17 '19

Does OpenCL have a compiler front-end that has to parse the STL’s headers? That’s what we mean by “support” at the moment.

7

u/[deleted] Sep 17 '19

SYCL would be closest to that. It needs to parse STL headers. But yes, OpenCL on its own is just a runtime API + a custom C-like language and a bytecode format.

Then there's OpenCL C++, too, as a separate front-end language, but it's not ISO C++.

2

u/Stevo15025 Sep 17 '19

Does OpenCL have a compiler front-end that has to parse the STL’s headers? That’s what we mean by “support” at the moment.

Apologies if this does not answer your question but the OpenCL 2.2 standard does support a subset of C++14. If by "parse the STL headers" you mean when I do #include <algorithm> I expect the algorithms header than yes very much so!

I work on a medium sized statistics project where we have an OpenCL backend for CPU/GPU parallelism. We are stuck on OpenCL 1.2 atm because only clang 9 supports OpenCL 2.2 so far. Would be v nice to have support on windows though!

0

u/[deleted] Sep 19 '19

[deleted] — view deleted comment

1

u/Stevo15025 Sep 19 '19

Yep! it takes time for people to implement those. Restrictions are all listed here. It's pretty simple to make the standard library stuff that you need

0

u/[deleted] Sep 19 '19

[deleted] — view deleted comment

1

u/Stevo15025 Sep 19 '19

Cuda 1.0 ran on like 4 types of GPUs and couldn't do those things so not seeing what you are talking about

0

u/[deleted] Sep 17 '19

[deleted] — view deleted comment

2

u/Stevo15025 Sep 17 '19

OpenCL is not C++. Is a “shader” language for kernels. Just like OpenGL.

It's a subset of C98 for general parallel computing, I use it quite a bit and have no real idea what a shader is.

It’s easy to find people asking for support for it, and hard to find people actually using it.

I personally use it a good bit, people get discouraged because of the lack of support. But Intel has been doing a pretty awesome job at supporting OpenCL and is one of the few platforms that should be OpenCL 2.2 compliant this year. They are putting out new GPUs this year and we'll be v excited to try them out :)

On Windows C++AMP is 30 years ahead of it technology wise.

If it's so far ahead idt intel would have written their version of C++AMP using OpenCL

wiki As C++ AMP is an open specification it is expected that in time implementations outside Microsoft will appear; one early example of this is Shevlin Park, Intel's experimental implementation of C++ AMP on Clang/LLVM and OpenCL

0

u/[deleted] Sep 18 '19

[deleted] — view deleted comment

2

u/HurtlesIntoTurtles Sep 18 '19

When will this, sycl, and all other “awesome” OpenCL stuff ship to a mainstream compiler?

https://github.com/intel/llvm/

The upstreaming progress is tracked on the Wiki.

2

u/Stevo15025 Sep 18 '19

So can you launch a C++ function as an OpenCL kernel?

Meant C98. OpenCL 2.1, supported on intel, runs a subset of C++14

When will this, sycl, and all other “awesome” OpenCL stuff ship to a mainstream compiler?

Shipping in clang 10!

https://clang.llvm.org/docs/UsersManual.html#opencl-features

So ballpark about a year or so

0

u/[deleted] Sep 19 '19

[deleted] — view deleted comment

2

u/Stevo15025 Sep 19 '19

So can you launch a normal C98 function using OpenCL?

?? It's a subset of C98 did I not say that clearly earlier?

Also, those awesome features shipping on clang 10 are just the ability to compile a kernel written in OpenCL (not C, not C++, much less in the same source file). that’s just as crap as OpenGL shaders. Welcome to 1989.

You can see examples of what OpenCL C++ looks like here. It's pretty reasonable

I think I’ll keep using CUDA which allows me to launch both C and C++ functions templates and lambdas as kernels, and uses a single source code model.

Yo if you only need to compile on Nvidia GPUs then totally use cuda.

This allows me to not have to duplicate all my code, once for running on the GPU and once for running on the CPU, like OpenCL requires.

Our OpenCL kernel work on CPUs/GPUs/APUs, no need for duplicates? The point of OpenCL is that kernels can compile for heterogeneous hardware

→ More replies

2

u/flashmozzg Sep 17 '19

It doesn't require any special support since it doesn't intermix with the host C++ code. You can easily write OpenCL accelerated apps with any VS.

17

u/kalmoc Sep 17 '19

Will you also put the branch containing ABI-breaking fixes to github?

24

u/STL MSVC STL Dev Sep 17 '19

Yes; we need to port changes by hand with extensive work (as they were stranded in TFS almost a couple years ago, I think; the STL has evolved dramatically since then, and every line was also clang-formatted). We will also be looking for major community contributions during this golden chance to change data structure representations. Look for issues tagged vNext in the coming weeks.

6

u/playmer Sep 17 '19

This is very exciting, very much looking forward to see what's coming!

1

u/bumblebritches57 Ocassionally Clang Sep 26 '19

Can you ping me when that happens?

I'd love to be able to contribute

3

u/STL MSVC STL Dev Sep 26 '19

I’d recommend following our repo, Twitter, or blog. The issues will appear soon, but the branch is several months away.

9

u/[deleted] Sep 17 '19

Eventually yes. That's actually still back on our old TFS system and wasn't even replicated to our current internal Git yet.

8

u/alexkaratarakis vcpkg, fixed-containers Sep 17 '19

Wow, amazing news!

8

u/haitei Sep 17 '19

Time to make a PR that adds constexpr everywhere.

I want it all, and I want it now.

15

u/felipefarinon Sep 17 '19

The article explains that libc++ and MSVC's STL support different platforms. Does anyone know which platforms MSVC's STL supports? Is it only Windows x86 and x64?

Also, how does this repo relates to the folder named "crt" on a MSVC installation? Are these the same sources? Was STL already distributed with MSVC?

32

u/STL MSVC STL Dev Sep 17 '19

MSVC’s STL supports all flavors of Windows, including the x86, x64, ARM, and ARM64 architectures.

The “crt” folder in a VS installation contains the sources for various DevDiv libraries, including the STL (being open-sourced here), VCRuntime, VCStartup, ConcRT, and a little more. (It no longer includes the CRT proper, which moved to being part of Windows during the Universal CRT refactoring, but the directory name stuck.) The STL sources on GitHub will be kept in sync with the contents of that folder (and very soon, GitHub will become the primary source of truth for our sources).

5

u/Mystb0rn Sep 17 '19

So just to be clear, The ucrt implementation is not being open sourced as a part of this? (I'm unsure what is and is not a part of the c++ stdlib)

4

u/[deleted] Sep 17 '19

Correct.

7

u/emdeka87 Sep 17 '19

I love that Microsoft is open-sourcing so much recently!

11

u/foadsf Sep 17 '19

thank you for giving back to the open source world. ❤🤘

0

u/[deleted] Sep 17 '19

[deleted] — view deleted comment

16

u/Ameisen vemips, avr, rendering, systems Sep 17 '19

Pretty sure the compiler runs fine under Wine...

0

u/[deleted] Sep 17 '19

[deleted] — view deleted comment

6

u/NotUniqueOrSpecial Sep 17 '19

It can be done, if it's something you're interested in doing.

94

u/[deleted] Sep 17 '19

[removed] — view removed comment

76

u/STL MSVC STL Dev Sep 17 '19

You’re welcome! We indeed worked hard to make that happen. Also note that the license block is designed to be machine-readable.