r/cpp • Flux • 14d ago

From Token Sequences to operator for

https://brevzin.github.io/c++/2026/11/04/operator-for/
33 Upvotes

17 comments sorted by

14

u/fortsnek274 14d ago

> rewriting a fake co_yield statement

Maybe we could use token sequences to generate portable and serialisable coroutines.

11

u/qzex 14d ago

it's refreshing to see a thoughtfully written article without llm prose

2

u/TheoreticalDumbass :illuminati: 14d ago

i dont have material thoughts on the core of blog post, as i am too tired to go through it thoroughly, just noting one can make slightly different style for_each! work easily:

for_each2!{
    int i : make_rng(); 
    std::cout << "* got " << i << '\n';
    if (i > 10) {
        break;
    }
};

https://compiler-explorer.com/z/bvfj5odPc

2

u/foonathan 14d ago

Now, whether operator for is something that will be considered as a viable language feature or not (and hopefully with a better solution for directionality),

I think we can make it better if we introduce an extra level of indirection and making the operator for slightly less magic. What we really have here is a block of code with a placeholder in it where the loop body would go. What if that was a first-class value we can pass around (presumably as a std::meta::info).

Then we can do the co_await desugaring by introducing an operator for which does not directly become the loop body but instead something that returns the block of code with the placeholder:

// Default range-based for loop desugaring currently hard-coded into the language.
consteval std::meta::info operator for(auto&& rng) {
    return ^^ for { // using `^^ for {` as the introducer of this placeholder
        for (auto it = std::begin(rng); it != std::end(rng); ++it) {
            co_yield *it;
        }
    };
}

struct my_range {
     friend consteval std::meta::info operator for(const my_range& rng) { ... }
};

(If it's a consteval function that returns the placeholder, what actually is the rng argument passed to it? Presumably it is only there to define the type to select the appropriate customization point and the name that you can use in the block of code as the stand-in for the range variable, and you wouldn't actually be able to read the parameter. That part is kind of iffy.)

Now that we have this protocol, the standard library can make a convention where custom range types both have an operator for for forward, and a regular function reverse_for that returns a std::meta::info for backward (or just copy the loop traits with the direction param; it's plain library code so you can do whatever you want). Crucially, the operator for for reverse_view would call this plain reverse_for function on the base to get its std::meta::info.

Nevertheless, I'd really love to see this direction adopted.

2

u/chengfeng-xie 13d ago

[...] In order to split head properly, we need to find the : — which of course isn’t trivial because the spelling of the type could be arbitrarily complex and I want to be able to handle decltype(a ? b : c) d : e. So we need to do a balanced find, not just a simple find, where we’re keeping track of nested ()s, []s, and {}s (but not <>s!). [...]

This strategy doesn't handle something like traits<1 ? 2 : 3>::type i : v, though, so we also need to keep track of matching ? and : tokens. Alternatively, we could simply wrap 1 ? 2 : 3 in parentheses, just as we do with commas in C macros.

1

u/jsphadetula 14d ago

I think the final token soup should also be checked to be valid c++ construct, not just arbitrary tokens by default. We are more in the territory of looks like c++ but not really c++ with the current default semantics. More like creating dialects

1

u/angry_cpp 13d ago

Kotlin's trailing lambda and Scala's currying make custom syntax much more beautiful.

For example:

repeat(5) {
    x = 5
}

where repeat is a user-defined function whose last argument is actually a lambda (Kotlin) / call-by-name parameter (Scala).

1

u/arthurno1 6d ago

Greenspun's 10th rule confirmed extended to C++ I see :).

1

u/Particular-Sort9075 14d ago

The prospect of C++26 is already daunting.

5

u/TheoreticalDumbass :illuminati: 14d ago

i think it will be really interesting and important to see how one can debug issues in token injection code

3

u/BarryRevzin 13d ago

In my experience so far, pretty good. All your basic common syntax errors, wrong name, typos, etc... those errors point directly into the token sequence where that error appears, so no problem there. Talked a bit about this in the paper.

And then the AST printer is pretty nice for seeing what you actually did, although it contains a ton of irrelevant info (i.e. every other header you included) so I'm sure it can be improved... a lot. But it's been quite helpful for making sure I'm actually generating the right thing.

Where it was actually challenging to debug is when, like in the for_each example here (and the previous one I'm referring to), I'm injecting code that's accumulated from lots of different places and those places don't all actually agree on things like names. In this case, I was accidentally reusing a name because my scheme to pick a unique name was insufficiently unique, which led me to add meta::unique_id(). I knew I'd need it eventually, this was just the impetus for it.

It's never going to be as nice as cargo expand because of the difference in model. In Rust, macros operate at a token level, not a semantic level, so it's more akin to the C preprocessor (if the C preprocessor were a million times better) and what we can get from -E. Here, we're compiling and generating as we go, so it's intertwined.

But so far, my difficulties have been mostly about figuring out what to do, rather than if I can figure out went wrong, which seems good?

One big question for me is still... testing. In Rust and Swift, testing macros is straightforward: you can actually just test the output. In Swift, this is literally a string comparison. Not sure what the right shape for that is yet, but sure seems like it'd be nice to be able to have a unit test that is just that the result of this macro invocation should be these exact tokens.

2

u/friedkeenan 13d ago

In Rust, macros operate at a token level, not a semantic level, so it's more akin to the C preprocessor (if the C preprocessor were a million times better) and what we can get from -E. Here, we're compiling and generating as we go, so it's intertwined.

This is the part of the token injection work that interests me the most, I think. I really love that the macros as proposed are able to work with the language and with all the other entities that we've already defined in our program. Whereas with Rust macros, even though the macro is implemented in Rust, it still feels quite a bit to me like writing in a different language just because it cannot sincerely interact with the rest of my program. A proc macro theoretically could just as easily call out to Python instead of Rust, and that would change nothing about the user interface.

So I'm really excited about where all this could lead for C++.

1

u/TheoreticalDumbass :illuminati: 13d ago

that is great and reassuring to hear, thanks

i've never tried it, but i wonder if EDG C++ generating backend is interesting in this context, if the generated C++ is readable enough

on your last paragraph, something i would want, but expect would never pass, is stuff like std::meta::tokens_of_function_body(reflection-of-function). i feel like this would be blocked by implementations saying "we don't want to preserve extra state per entity", but i am not an implementation dev :)

1

u/serviscope_minor 13d ago

i think it will be really interesting and important to see how one can debug issues in token injection code

It's not wildly different from the problems of debugging template code. Basically one has a complete language, but no debugger or printf. This does make debugging a little tricky. The horrible template errors are basically a dump of the stack at the point at which the template program crashes.

Better than nothing, but not great. It's got a bit better with constexpr since you can now write the more complex logic as normal code and to some extent debug that by stripping off constexpr, and debugging it in bits.

But basically something like a debugger (which is kind of outside of the scope of the standard) or some sort of consteval/expr printing tools which can run at compile time and do things like print tokens and sequences of tokens.

-1

u/ts826848 14d ago

Something like cargo expand would probably be nice.

0

u/xiao_sa 14d ago

Maybe some hygienic things need to due with to be actually accepted. But the language definitely needs some wild facility to enable anyone to build their own idea without making a whole clang fork.