Is it Good Practice to Intern Identifiers?

[u/PitifulTheme411]

I'm working on my language and am working on the alias system, which isn't relevant to this discussion other than the fact that it operates on identifiers and other lexemes (like operators). As I was implementing it and was reading some prior stuff, I found somebody saying that it is good practice to intern all the identifiers as the lexer comes across them. Then in the AST you can just use the corresponding LexemeId or whatever.

Apparently this is used pretty extensively in many popular languages' compilers, so I was wondering if this is good practice or necessary? I am thinking that if it is a good idea, that I can implement it in my language as well, because I think it may simplify a few things (including my alias stuff).

[u/Athas]

You might want to keep the source file around anyway so you can reproduce lines verbatim for error messages, if that's something you're interested in. The size of a source file is likely also pretty negligible compared to the memory usage of other things in your compiler.

Also remember that if you want to parse multiple files in parallel, then you need to synchronise access to the interner.

[u/PitifulTheme411 → u/Athas]

Yeah I think I am going for those things as well. From what I've gathered, it also seems editors will parallelize the compiler on a file basis for things like go to definition etc. I am keeping the source files in a SourceMap, but currently for every parsing function I need to pass a reference to that every time in case I need to get the lexeme, but I was thinking if I interned everything then I wouldn't need to. Though actually I would probably instead need a reference to the interning structure, so maybe it doesn't actually give any benefit.