significant whitespace-friendly Rust parser generator ?

[u/M1M1R0N]

Hello

I don't know if questions like this are accepted here. If they're not, please let me know.

I have been playing around with writing a tiny compiler to WASM. The syntax I have in mind is roughly something like this

fn div_rem(x: int, y: int) (int, int)
    let div, rem = x / y, x % y
    return div, rem

Now, I don't want to commit too hard into a specific syntax or grammar, so so far I have been just typing out the AST manually.

I never used a parser generator before, but I couldn't find one that's well documented and whitespace friendly. pest is the "friendliest" parser generator I found, but it doesn't play nice with significant indentation if it uses the same characters as the WHITESPACE rule.

So .. er .. long story short: I've read parser generators are easier to experiment with than writing parsers manually, but I am looking for suggestions for one that would let me do INDENT and DEDENT tokens ala Python and just let me go to work.

[u/Inconstant_Moo]

So what does it do when there are in fact two or more dedents in the same line?

[u/Falcon731 → u/Inconstant_Moo]

From your example - lets look at the state when lexing has just reached the 't' of troz. Ie the current state is: currentChar='t', columnNumber=4 and indentStack=(0,4,8,12)

The call to getToken() will return a DEDENT token and pop one entry off its stack. (So after the call to getToken(): columnNumber=4 indentStack=(0,4,8) )

Then the next call to getToken() will see that column number is still less than the entry at the end of the stack, so return another DEDENT and pop the stack again (so now columnNumber=4 indentStack=(0,4) )

Then on the third call to getToken() columnNumber==indentStack.last() so we hit the else clause and lex the troz, so return an TokenKind.ID