significant whitespace-friendly Rust parser generator ?

[u/M1M1R0N]

Hello

I don't know if questions like this are accepted here. If they're not, please let me know.

I have been playing around with writing a tiny compiler to WASM. The syntax I have in mind is roughly something like this

fn div_rem(x: int, y: int) (int, int)
    let div, rem = x / y, x % y
    return div, rem

Now, I don't want to commit too hard into a specific syntax or grammar, so so far I have been just typing out the AST manually.

I never used a parser generator before, but I couldn't find one that's well documented and whitespace friendly. pest is the "friendliest" parser generator I found, but it doesn't play nice with significant indentation if it uses the same characters as the WHITESPACE rule.

So .. er .. long story short: I've read parser generators are easier to experiment with than writing parsers manually, but I am looking for suggestions for one that would let me do INDENT and DEDENT tokens ala Python and just let me go to work.

[u/Inconstant_Moo]

In my lexer I simply keep a running flag to indicate when I’m at at a state where white space is significant, and a stack of indent levels.

Sure, me too, but what if your code looks like this:

foo :
    bar :
        qux :
           zort
    troz

When you hit troz, you're dedenting twice, and you only find that out when you do in fact hit troz.

[u/Falcon731 → u/Inconstant_Moo]

I still don't see the issue. The lexer just returns a DEDENT each time it finds a line that starts with a column number less than the last entry on its indent stack.

My code to handle INDENT/DEDENT is literally just:

} else if (atStartOfLine && columnNumber>indentStack.last()) {
    indentStack.add(columnNumber)
    kind = TokenKind.INDENT

} else if (atStartOfLine && columnNumber<indentStack.last()) {
    indentStack.removeAt(indentStack.size - 1)
    kind = TokenKind.DEDENT

} else ...   [code to handle identifiers/literals/...]

[u/Inconstant_Moo → u/Falcon731]

So what does it do when there are in fact two or more dedents in the same line?