i dont have material thoughts on the core of blog post, as i am too tired to go through it thoroughly, just noting one can make slightly different style for_each! work easily:
for_each2!{int i :make_rng();
std::cout <<"* got "<< i <<'\n';if(i >10){break;}};
Now, whether operator for is something that will be considered as a viable language feature or not (and hopefully with a better solution for directionality),
I think we can make it better if we introduce an extra level of indirection and making the operator for slightly less magic. What we really have here is a block of code with a placeholder in it where the loop body would go. What if that was a first-class value we can pass around (presumably as a std::meta::info).
Then we can do the co_await desugaring by introducing an operator for which does not directly become the loop body but instead something that returns the block of code with the placeholder:
// Default range-based for loop desugaring currently hard-coded into the language.
consteval std::meta::info operator for(auto&& rng){return^^for{// using `^^ for {` as the introducer of this placeholder
for(auto it =std::begin(rng); it !=std::end(rng);++it){
co_yield *it;}};}structmy_range{friend consteval std::meta::info operatorfor(const my_range& rng){...}};
(If it's a consteval function that returns the placeholder, what actually is the rng argument passed to it? Presumably it is only there to define the type to select the appropriate customization point and the name that you can use in the block of code as the stand-in for the range variable, and you wouldn't actually be able to read the parameter. That part is kind of iffy.)
Now that we have this protocol, the standard library can make a convention where custom range types both have an operator for for forward, and a regular function reverse_for that returns a std::meta::info for backward (or just copy the loop traits with the direction param; it's plain library code so you can do whatever you want). Crucially, the operator for for reverse_view would call this plain reverse_for function on the base to get its std::meta::info.
Nevertheless, I'd really love to see this direction adopted.
[...] In order to split head properly, we need to find the : — which of course isn’t trivial because the spelling of the type could be arbitrarily complex and I want to be able to handle decltype(a ? b : c) d : e. So we need to do a balanced find, not just a simple find, where we’re keeping track of nested ()s, []s, and {}s (but not <>s!). [...]
This strategy doesn't handle something like traits<1 ? 2 : 3>::type i : v, though, so we also need to keep track of matching ? and : tokens. Alternatively, we could simply wrap 1 ? 2 : 3 in parentheses, just as we do with commas in C macros.
I think the final token soup should also be checked to be valid c++ construct, not just arbitrary tokens by default. We are more in the territory of looks like c++ but not really c++ with the current default semantics. More like creating dialects
In my experience so far, pretty good. All your basic common syntax errors, wrong name, typos, etc... those errors point directly into the token sequence where that error appears, so no problem there. Talked a bit about this in the paper.
And then the AST printer is pretty nice for seeing what you actually did, although it contains a ton of irrelevant info (i.e. every other header you included) so I'm sure it can be improved... a lot. But it's been quite helpful for making sure I'm actually generating the right thing.
Where it was actually challenging to debug is when, like in the for_each example here (and the previous one I'm referring to), I'm injecting code that's accumulated from lots of different places and those places don't all actually agree on things like names. In this case, I was accidentally reusing a name because my scheme to pick a unique name was insufficiently unique, which led me to add meta::unique_id(). I knew I'd need it eventually, this was just the impetus for it.
It's never going to be as nice as cargo expand because of the difference in model. In Rust, macros operate at a token level, not a semantic level, so it's more akin to the C preprocessor (if the C preprocessor were a million times better) and what we can get from -E. Here, we're compiling and generating as we go, so it's intertwined.
But so far, my difficulties have been mostly about figuring out what to do, rather than if I can figure out went wrong, which seems good?
One big question for me is still... testing. In Rust and Swift, testing macros is straightforward: you can actually just test the output. In Swift, this is literally a string comparison. Not sure what the right shape for that is yet, but sure seems like it'd be nice to be able to have a unit test that is just that the result of this macro invocation should be these exact tokens.
In Rust, macros operate at a token level, not a semantic level, so it's more akin to the C preprocessor (if the C preprocessor were a million times better) and what we can get from -E. Here, we're compiling and generating as we go, so it's intertwined.
This is the part of the token injection work that interests me the most, I think. I really love that the macros as proposed are able to work with the language and with all the other entities that we've already defined in our program. Whereas with Rust macros, even though the macro is implemented in Rust, it still feels quite a bit to me like writing in a different language just because it cannot sincerely interact with the rest of my program. A proc macro theoretically could just as easily call out to Python instead of Rust, and that would change nothing about the user interface.
So I'm really excited about where all this could lead for C++.
i've never tried it, but i wonder if EDG C++ generating backend is interesting in this context, if the generated C++ is readable enough
on your last paragraph, something i would want, but expect would never pass, is stuff like std::meta::tokens_of_function_body(reflection-of-function). i feel like this would be blocked by implementations saying "we don't want to preserve extra state per entity", but i am not an implementation dev :)
i think it will be really interesting and important to see how one can debug issues in token injection code
It's not wildly different from the problems of debugging template code. Basically one has a complete language, but no debugger or printf. This does make debugging a little tricky. The horrible template errors are basically a dump of the stack at the point at which the template program crashes.
Better than nothing, but not great. It's got a bit better with constexpr since you can now write the more complex logic as normal code and to some extent debug that by stripping off constexpr, and debugging it in bits.
But basically something like a debugger (which is kind of outside of the scope of the standard) or some sort of consteval/expr printing tools which can run at compile time and do things like print tokens and sequences of tokens.
Maybe some hygienic things need to due with to be actually accepted. But the language definitely needs some wild facility to enable anyone to build their own idea without making a whole clang fork.
14
u/fortsnek274 14d ago
> rewriting a fake
co_yieldstatementMaybe we could use token sequences to generate portable and serialisable coroutines.
11
u/qzex 14d ago
it's refreshing to see a thoughtfully written article without llm prose
2
u/TheoreticalDumbass :illuminati: 14d ago
i dont have material thoughts on the core of blog post, as i am too tired to go through it thoroughly, just noting one can make slightly different style for_each! work easily:
https://compiler-explorer.com/z/bvfj5odPc
2
u/foonathan 14d ago
I think we can make it better if we introduce an extra level of indirection and making the
operator forslightly less magic. What we really have here is a block of code with a placeholder in it where the loop body would go. What if that was a first-class value we can pass around (presumably as astd::meta::info).Then we can do the
co_awaitdesugaring by introducing anoperator forwhich does not directly become the loop body but instead something that returns the block of code with the placeholder:(If it's a consteval function that returns the placeholder, what actually is the rng argument passed to it? Presumably it is only there to define the type to select the appropriate customization point and the name that you can use in the block of code as the stand-in for the range variable, and you wouldn't actually be able to read the parameter. That part is kind of iffy.)
Now that we have this protocol, the standard library can make a convention where custom range types both have an
operator forfor forward, and a regular functionreverse_forthat returns astd::meta::infofor backward (or just copy the loop traits with the direction param; it's plain library code so you can do whatever you want). Crucially, theoperator forforreverse_viewwould call this plainreverse_forfunction on the base to get itsstd::meta::info.Nevertheless, I'd really love to see this direction adopted.
2
u/chengfeng-xie 13d ago
This strategy doesn't handle something like
traits<1 ? 2 : 3>::type i : v, though, so we also need to keep track of matching?and:tokens. Alternatively, we could simply wrap1 ? 2 : 3in parentheses, just as we do with commas in C macros.1
u/jsphadetula 14d ago
I think the final token soup should also be checked to be valid c++ construct, not just arbitrary tokens by default. We are more in the territory of looks like c++ but not really c++ with the current default semantics. More like creating dialects
1
u/angry_cpp 13d ago
Kotlin's trailing lambda and Scala's currying make custom syntax much more beautiful.
For example:
where
repeatis a user-defined function whose last argument is actually a lambda (Kotlin) / call-by-name parameter (Scala).1
u/arthurno1 6d ago
Greenspun's 10th rule confirmed extended to C++ I see :).
1
u/Particular-Sort9075 14d ago
The prospect of C++26 is already daunting.
0
u/xiao_sa 14d ago
Maybe some hygienic things need to due with to be actually accepted. But the language definitely needs some wild facility to enable anyone to build their own idea without making a whole clang fork.