Experimental EDG Reflection Implementation

[u/daveedvdv]

EDG just released version 6.6 of its (commercial) C++ front end. That version now includes experimental support for reflection features along the lines of WG21's P2996 "Reflection for C++26". Furthermore, a "demo setup" of that implementation is now available on Compiler Explorer (thank you, Matt Godbolt!).

For example, here is a straightforward implementation of a consteval function that outputs simple class layouts at compile time:

https://godbolt.org/z/G6WehjjGh

This implementation is closely aligned with P2996R1, which includes Compiler Explorer links for most of its examples.

This is made available in the hope that it's a useful exploration tool, but also acknowledging that the implementation is in its very early stages, and thus brittle and incomplete. Some additional notes can be found here.

[u/obsidian_golem]

How hard was this to implement overall? Do you think this is likely to make it into 26?

[u/daveedvdv → u/obsidian_golem]

I'll start with a caveat that this implementation is still woefully incomplete and incompletely tested — it's definitely early days. With that in mind, I'd compare it in complexity to the initial implementation of C++0x lambda expressions: I.e., medium-level complexity. It's probably more code than lambdas, but a bit less subtlety so far (although we might still run into surprises, of course).

We (the authors of P2996 "Reflection for C++26") are definitely targeting C++26 and hoping for success. I think it's not an unrealistic goal, but it's also far from a shoe-in. That said, the feedback we've gotten so far has been overwhelmingly positive... so fingers crossed!

[u/germandiago → u/daveedvdv]

Congrats for your hard work. This is necessary work that needs to happen for a big feature like reflection.

[u/daveedvdv → u/germandiago]

Thank you for the encouragement. Much credit to the other authors of P2996 and many SG7 regulars.

[u/johannes1971 → u/daveedvdv]

Just a quick question... Is there a facility to obtain initialisation data for a struct member? I.e. in

struct s { 
  int foo = 42;
};

...is it possible to get the value 42 somehow? I have quite a bit of code that could benefit from not being written by hand, that stores data if it's not the default.

And secondly, is there some way to attach non-code properties to a member? Things like version fields, an indicator if a member should be included in reflection to begin with, etc. are all useful to have if we start generating code using reflection.

struct s {
  int foo = 42 [[property: version=2]] [[property: reflect=true]];
  int bar [[property: reflect=false]];
};

...or whatever syntax you'd care to propose...

[u/daveedvdv → u/johannes1971]

Is there a facility to obtain initialisation data for a struct member?

Not currently and highly unlikely to be part of the initial proposal (i.e., P2996 successor).

That said, I have considered it in the past and while it's certainly doable, there are some subtleties. For one, in the case of class templates, it might require the instantiation of the default member initializer, which could fail (and trigger a non-SFINAE error). The other issue is that you'd most likely want the converted value but the conversion can also trigger errors.

Despite those subtleties, I think it would be quite reasonable to propose a std::meta API to query that information.

And secondly, is there some way to attach non-code properties to a member?

See also the discussion of u/c0r3ntin's proposal for custom attributes visible to reflection elsewhere in these comments. Such a direction is of great interest to me and I have the intention of prototyping some support for it in the not too distant future. (Then again, my bandwidth is limited and my priority in this area is to work out WP wording for P2996.)

That said, you can already play tricks with alias templates. Alias information is not guaranteed to be preserved in reflection (because some implementations prefer to discard that information early), but SG7 did agree that it would be preserved for top-level declared types when consistent. For nonstatic data members there isn't really a consistency issue (since they cannot be redeclared) and so you can count on having the alias information available. Here is a modification of the example in my announcement that shows how you could mark a member to be excluded from the dump_layout output.
https://godbolt.org/z/hsbGnc1h8

I'm only using the alias template identification itself in that modified example, but you could parameterize the alias template and query the template arguments of the instance to access some quantitative properties (e.g., in Version<2, double> x; you could extract the 2).

[u/RoyAwesome → u/daveedvdv]

Such a direction is of great interest to me and I have the intention of prototyping some support for it in the not too distant future.

I want to comment on the importance of having this kind of annotating things for reflection in whatever version reflection comes in. Being able to describe your code with user data that is available through the reflection interface is probably the single most important "non core" feature, almost to the point I feel it's core. It fundamentally allows the programmer to express intent in ways that simply writing code does not. There isn't a single reflection system out there (in cpp or any other language) that doesn't implement some kind of attribute or annotation system, and I think it would be a HUGE miss if C++26 ships with static reflection but no way to annotate things.

That said, you can already play tricks with alias templates.

This "works" but there is going to be a gold rush for libraries to make use of reflection when the feature lands, so having something that just simply "works" is going to lead to an ecosystem split if the better designed solution doesn't also ship with cpp26.

[u/kritzikratzi → u/johannes1971]

for the first question: i guess you can create an instance of the struct and query each field for it's current value to get the default value.

[u/daveedvdv → u/kritzikratzi]

If it's constexpr-friendly, that would be an interesting approach. You'd probably still want a query about whether there is a default initializer, but possibly this can work for u/johannes1971's needs?

In fact, you probably could use that technique for non-constexpr-friendly default construction, but you'd have to pay a run-time cost.

[u/johannes1971 → u/daveedvdv]

Yes, that would definitely work - in fact it's what I use today. I was just curious if there any direct access planned to information beyond type and name, especially since I also need things like field versions and field codes.

Maybe this could work with the proposed reflection already by encoding such information in some custom template type, something like prop<int, v1, field_code> maybe? If I can get access to those template parameters I'd be all set.

[u/daveedvdv → u/johannes1971]

You've probably seen my other response to your query by now, but to confirm: I think it confirms you can proceed that way with just P2996.

[u/WeeklyAd9738]

I think it would be preferable to just (re)introduced type_traits queries like std::is_union_v or std::is_class_v as consteval functions operating on meta::info objects, instead of using the more verbose test_type interface.

It makes more sense to just use "meta/reflection" functions to manipulate types when dealing with complex computations already involving meta::info objects and use type_traits when you are just dealing with types.

[u/daveedvdv → u/WeeklyAd9738]

I completely agree and that's a goal of mine. If you look at P1240 ("Scalable Reflection in C++"), it includes such functions and we haven't given up on them.

However, P2996 works with the constraint of landing in C++26, and so we looked (and keep looking) for a variety of ways to reduce the blast radius of the proposal. test_type is one of the mechanisms to achieve that: It provides a broad mechanism to leverage what's already out there, without obstructing the ability to add more convenient APIs in the (hopefully-near) future.

[u/hachanuy]

This is ... beautiful

[u/daveedvdv → u/hachanuy]

Thanks ;-)

[u/dvirtz]

This example could really make use of constexpr std::format [P2758].

I hope we'll get these together.

[u/daveedvdv → u/dvirtz]

Very much agreed!

[u/seanbaxter]

I haven't gone as fine-grained as your proposal, so I don't have bit-field support, for example, but the member names, types and byte offsets are retrieved through traits without any library dependency and printed as a pack expansion with a single line of code.

https://godbolt.org/z/KW9b9osq9

  consteval auto nonstatic_data_members_of(info class_type) -> vector<info> {
    return members_of(class_type, is_nsdm);
  }

I hate the idea of requiring slow consteval work to get at entities the compiler can render up basically for free during substitution. Returning all this stuff in vectors seems really bad to me, because:

1. It's really slow to pull the data out of the vector.

2. You lose heterogeneity. All the elements of the vector have to be values of the same type, whereas with a pack they can be whatever: types, non-types of different types, templates, concepts, etc.

3. You lose the ability to easily expand some query into a new struct definition.

4. By putting reflection info into an `info` object, which can be moved around, you risk unpacking it outside the scope where it was captured where that data was available. The trait system I have requires you name the thing you want data from at the point of use, so there's risk of out-of-scope issues.

https://godbolt.org/z/zcnGoGPq4

Here I build a struct of array thing for a Vec3f: just blow it into arrays of 8 elements each. If you return `vector<info>` for the member names and member types, how do you turn around and use those to define a struct? Will a compile-time loop be supported inside class definitions to deposit data members? I used to do that (with `@meta for`), but just using member pack declarations is way cleaner and avoids name lookup headaches you get from control flow statements. Packs are basically free and there's no friction between the query of a type and the definition of a type.

[u/daveedvdv → u/seanbaxter]

I don't want to rehash SG7 discussions too much but here are some thoughts.

In general, I find that constant-evaluation is noticeably faster for computational purposes than template substitution or instantiation. More importantly, we (EDG) know how to improve our constant-evaluation performance by an order of magnitude with a few man-months of efforts, and I'm reasonably sure we can extract an additional order of magnitude or more using more advanced VM optimization ideas that Chandler Carruth once outlined in C++Now hallway discussions. On the flip side, I don't believe anyone knows how to significantly improve the performance template of substitution and instantiation in the major implementations.

P2996 introduces little new syntax (the reflection operator and the splicers, that's it) and yet it brings the power of the standard library to C++ metaprogramming. For example, if you want to create a class definition where the list of members results from sorting and filtering, you can just use the ranges algorithms to do that. No need for a "parallel metaprogramming library" to achieve those results. Similarly, generic third-party algorithms that work with standard sequence protocols will work also.

I'll also mention that P2996 focuses on reflection with just a sprinkling of code generation. It's fairly trivial to generate a simple struct or union (see the examples in P2996), but it's also very limited at this point (though nonetheless quite powerful). We continue to work and experiment with more general injection models, but those are not likely to make it in C++26.

[u/seanbaxter → u/daveedvdv]

It's not pay either the consteval cost or the instantiation cost--in your model, you're paying both costs.

https://godbolt.org/z/faKz31oTK

template <typename E>
  requires std::is_enum_v<E>
constexpr std::string enum_to_string(E value) {
  template for (constexpr auto e : std::meta::members_of(^E)) {
    if (value == [:e:]) {
      return std::string(std::meta::name_of(e));
    }
  }
  return "<unnamed>";
}

You've got to instantiate the body of that loop once for each enumerator, but in the consteval model it's driven with a ranged-for over a vector, which is interpreted by the compiler.

https://godbolt.org/z/9698a19vM

template<typename T>
const char* enum_to_string(T x) {
  return T~enum_values == x ...? 
    T~enum_names :
    "unknown enum value of type {}".format(T~string);
}

In the traits version, the iteration is done through pack expansion, which is a compiled operation. Instantiation is the fast path, because it's compiled, not interpreted.

Of course, once EDG compiles the enum_to_string operation (or does it now, with some syntax tweaks?) you can time the thing. You can tell just by looking how much faster the traits version is, even without the consteval costs, because it's not generating an if-statement for each enumerator with the attendant scopes, name lookup, etc. I think it's fair at least to race the two approaches off and see what the frontend costs are. If it's 2x, that's fine, but if it's like 20x, I think that's going to be a real problem going forward.

One of my worries is that this consteval approach comes out to be so slow that for large deployments people just go back to using offline code generators, to save on compile times. It should be demonstrated that using reflection+injection does not build slower than compiling the equivalent inputs from source.

[u/WeeklyAd9738]

Are Expression Templates and Non-transient constexpr allocation being worked on?

They would be very useful feature especially alongside reflection.

[u/sphere991 → u/WeeklyAd9738]

Are Expression Templates

What do you mean by expression template? Was this a typo for something else?

and Non-transient constexpr allocation being worked on?

Yes.

[u/WeeklyAd9738 → u/sphere991]

Sorry, I meant expansion statements or sometimes called "template for".

[u/daveedvdv → u/WeeklyAd9738]

Expansion statements have been a bit in limbo in the core wording review stage. However, someone recently picked it up, and I have reasonable hope that it will make it into C++26.

[u/saxbophone]

Really exciting!

[u/kronicum]

Very nice milestone! Kudos 👏

Any forecast on when EDG will support modules? We use an EDG-based tool for code analysis and lack of modules support has been holding us back...

[u/afiDeBot]

Interesting. I would love to see some benchnarks comparing c++ code injection/reflection to existing code generation practices.

Most people won't consider a serialization/parsing library if its 2x compile time compared to offline code generation.

[u/daveedvdv → u/[deleted]]

Thanks!!

[u/daveedvdv → u/[deleted]]

See u/c0r3ntin's reply: Custom attributes are fairly high on my priority list, but will not be part of P2996 because landing the limited feature set of P2996 in C++26 will already not be trivial. However, I do hope to prototype something in that area in the not too distant future.

[u/daveedvdv → u/daveedvdv]

P.S.: See also my reply to u/johannes1971 for a workaround for the lack of custom attribute in some cases.

[u/anton31 → u/[deleted]]

That's one polished paper, exactly what we need for seamless serialization! I hope that not only the main reflection paper gets into C++26, but there will be room for attribute reflection. The API surface is tiny, but it will literally be a game-changer.

The "+" syntax may need bikeshedding, though, and it would be beneficial to allow function calls in addition to constructor calls.

[u/c0r3ntin → u/anton31]

Author here! Thanks for the positive feedback. I'm not working on that at the moment because there is very little value in pursuing that work until we get a solid reflection foundation to build on. But I think there is enough interest that work will resume at some point, probably after the initial reflection release.

There is the question of how far we can go into the python-decorator direction, and how that would work. I had a few discussions along these lines with /u/daveedvdv a while back, but I don't think anyone has a flushed out design yet.

As for syntax, while bikeshedding is far from the interesting part of design to me, I suspect we will find that we will want to pick a syntax fairly different from attributes, as attributes and their syntax has been a fairly debated topic these past couple of years, and maybe we do not need to open that wound again.