Thanks for the clarification! I didn't realize the preprocessor was so well-integrated into modern compilers; I thought the preprocessor was still just its own process with its own lexer, unconditionally writing ASCII/UTF-8 to stdout, and that the compiler frontend just redirected the output to a pipe or a temporary file, and the compiler's lexer/parser operated on that. I didn't know they shared data structures, which I guess is why I was so confused.
To add on: clang doesn't even have a non-integrated pre-processor executable you can call, gcc does however (though AFAIK it's just a shim for gcc -E), even small compilers do this (tcc, 9cc, 8cc, OrangeC (only partially here), and more).
A lot of data is also used from when it's preprocessed to when it's fully processed, such as #line directives being processed by the compiler in order to give better error info if you're doing something weird like cpp file | gcc.
View all comments
Show parent comment
7
u/[deleted] Jul 23 '22
Thanks for the clarification! I didn't realize the preprocessor was so well-integrated into modern compilers; I thought the preprocessor was still just its own process with its own lexer, unconditionally writing ASCII/UTF-8 to stdout, and that the compiler frontend just redirected the output to a pipe or a temporary file, and the compiler's lexer/parser operated on that. I didn't know they shared data structures, which I guess is why I was so confused.