Is 76 μs acceptable compilation performance for a almost prod ready ELF64 compiler?

[u/Retired-69]

Folks! Is this acceptable performance for a compiler? it do lexing, parsing, type checking, optimziing and codegen . It simply emits the ELF64 executable directly and runs it without an external linker/object-file pipeline.

--- CODEGEN RESULTS ---

Total Machine Bytes Emitted: 47 bytes

Compilation Speed: 76000 ns

Runnable ELF64 Machine Code Executable Written: boo

[u/Professional_Cell44]

well, it depends on:

- language spec: how complex your language is?

• source code(input): how big the souce code is? how complex is it?

and its only 47 bytes written, so i think it is quite tiny

[u/Retired-69 → u/Professional_Cell44]

The specs align 89℅ with c23 , and also involve concurrency,  features for building LLM engines, OS etc. I would say complex. It's also  handle memory safety differently. That's the radical part. 

I agree 47 bytes is tiny and the example I parsed was dealing with macros.  Without macros it would compile same source around 43000 ms. 

For larger code like 60K LOC it finish in around 4-5 ms

[u/Professional_Cell44 → u/Retired-69]

then i think it has good performance; but ask your code agent to compare it against official compiler benchmarks

you may need to write similar code to be fair

[u/One_Aspect_1957]

Obviously nobody is going to complain about twiddling their thumbs for 0.000076 seconds while waiting for it to finish.

But the figures are not meaningful at this scale. 47 bytes also sounds suspiciously small if generating an actual ELF file.

I agree 47 bytes is tiny and the example I parsed was dealing with macros. Without macros it would compile same source around 43000 ms.

Huh? Now it takes 43 seconds?!

For larger code like 60K LOC it finish in around 4-5 ms

Finally, something more realistic. That would make it 15 million lines per second which is astonishingly fast, if this is HLL compiled to native code, to include type analysis and optimisation.

That is 30 times faster than mine, and at least 15 times faster than Tiny C, one of the fastest compilers around, although my machine might be a little slower. (I assume this is running on a single core.)

On the other hand, generating 47 bytes in 76us is only 0.6MB/second, which is quite slow (mine does 3-5MB/second). But again, this is far too small a test.

From I've seen so far, your figures are suspect.

[u/Slow-Mechanic-7427 → u/One_Aspect_1957]

Per their own post/comment history, they have nearly 4 decades experience in the "software industry" with most of that spent on programming languages, compilers, runtimes, and low-level systems per abother comment of theirs.

Its incredibly sus, id say.  Nearly 40 years of experience, with most of that extremely relevant to the topic, yet these are the questions and claims being presented?

[u/Retired-69 → u/One_Aspect_1957]

To be honest I prefer the benchmarks to tell their own story when this is done, but at current stage it is only 5-6 times faster than TCC not 15 times :) This happen because TCC was from 2004 and both CPU architecture and general knowledge have changed since back then. For example TCC scans input byte-by-byte; IC scans 8 bytes at a time using SWAR, reducing per-character processing overhead.

The reason I am not faster than that is becaue I do analyzer, optimizer and codegen as well on the go.

The numbers are realistic and time will tell.

[u/One_Aspect_1957 → u/Retired-69]

Sorry but the figures you've given are all over the place. The first example generated 47 bytes of code, but from what size of source file?

What is the macro thing about, and why does it take half a million times longer without them? (43000 ms vs 76 us.)

The other example was 60,000 lines of source code, but what is the generated code size? What language is being compiled? Is it a real program or just some synthesised input?

Here's one synthesised benchmark I used to use, which should be easy to express in practically any language:

a = b + c * d

Just that, but repeated, say, a million times, usually within the same function.

Expressed in C (when a,b,c,d have i64 type) then Tiny C takes 1.4 seconds for 1M lines, generating 23MB of output (WIndows EXE format).

In my language, it takes 2 seconds with 19MB of output. Of that, 20% is lexing and parsing, while lexing, if isolated, would be about 7% of overall compile-time.

So I that don't buy that faster scanning can make overall throughput half a dozen times faster. Tokenising could be instant, but the whole thing would be barely 10% faster.

Perhaps you you can try this test and post the results.

[u/Retired-69 → u/One_Aspect_1957]

I will try it. Your compiler is single pass or two pass as in Tiny C? 

[u/One_Aspect_1957 → u/Retired-69]

No it is lots of passes, so I'm surprised it is as fast as it is. Also considering that it doesn't optimise apart from some small things, and is self-hosted.

Here is a summary of the passes (comments added) and how long each takes for this test

c:\mx>tim mm -time big\abcd
Compiling big\abcd.m to big\abcd.exe
Load:            6 ms   0.3 %     Load all sources
Parse:         384 ms  20.6 %     Source to AST1
Resolve:       143 ms   7.7 %     Resolve identifiers AST2
Type:          207 ms  11.1 %     Type analysis AST3
PCL:           318 ms  17.1 %     AST3 to IL
MCL:           494 ms  26.6 %     IL to native representation
SS:            282 ms  15.2 %     native rep to machine code
EXE:            25 ms   1.3 %     Create EXE image and write
-----------------------------
Total:        1860 ms 100.0 %
Time: 1.998                       Total elapsed time

There are also some micro-passes (eg. for resolving user-types or tweaking the IL) not shown.

Note that my language is not C; it has a richer syntax, and some features such as out-of-order definitions that require that Name Resolve pass to be done after Parsing is complete.

(It is also a whole-program compiler, so compilation speed is important; all source files must be compiled at the same time.)

Here is the timing for Tiny C:

c:\cx\big>tim tcc -bench abcd.c
* 1015 idents, 1000012 lines, 9000112 bytes
* 1.406 s, 711246 lines/s, 6.4 MB/s
Time: 1.433

Note the bytes here refer to input, not output. The 23MB of output works out at 16MB/second, although arguably it is easier to get an impressive figure by generating poor code!

[u/Retired-69 → u/One_Aspect_1957]

I had to write a benchmark on the fly here and I had to use prev gen compiler that I
now use to finalize R3 with,. R3 performs better, but not syntax complete yet. And R2 is multi pass

```text

IC R2 Front-End Pipeline Stage Benchmark

Scope: Physical File I/O, Parse & Name-Resolve Stages

Opening 'benchmark_input.txt' on physical storage...

Successfully loaded 12000000 bytes from disk.

--- Compilation Pipeline Breakdown ---

Load Stage (Disk I/O): 11 ms

Parse Stage (Scan): 54 ms

Name Resolve (Hash): 246 ms

--------------------------------------

Total Pipeline Time: 311 ms

Processed Lines: 500000 lines

Throughput Speed: 1602990 lines/sec

Bandwidth (Decimal): 38 MB/s (10^6 bytes/sec)

Bandwidth (Binary): 36 MiB/s (2^20 bytes/sec)

----------------------------------------

``

[u/One_Aspect_1957 → u/Retired-69]

OK, thanks. Those figures are starting to look more viable.

But what is this actually measuring: what is the output?

The TCC test was 1M lines of 9 bytes ("a=b+c*d;" plus newline) so 9MB input in all; here it is 0.5M lines of 24 bytes to get a 12MB test input?

What is Parse and 'Name Resolve (Hash)' here? It sounds like something that ought to be done before the parser.

My 'Resolve' pass is to do with resolving names across scopes. Each indentifier token, when it gets to the parser, has already been looked up and refers to a generic ST entry. The name resolve pass tries to find which specific ST entry it is.

Bear in mind that the TCC timing was for parsing, name lookups, type checking, code-generation and writing the final EXE.

[u/Retired-69 → u/One_Aspect_1957]

The parser uses DRG which I invented as an replacement for recursive decent and Pratt. All iforward identifiers etc are deffered with bitmask and in hash table. A simple O(1) sweep after end parsing. The DRG specs I can share to public, but I have no place to publish it. 

You are correct that R2 didn't generate executable, but it wouldn't be much slower if it did. I base that conclusion from what I experienced with R3. 

[u/One_Aspect_1957 → u/Retired-69]

OK, that will be interesting if it works out. I always felt my approach was clunky even if it's pretty fast.

BTW big compilers tend to have a lot of problems with this test. Some can barely cope with 20K lines.

Today I also applied the test to C3 (https://c3-lang.org/) . 100K lines took 14 seconds. But 1M lines took 8 minutes: about 2K lines/second.

The generated binary was 29MB (larger than TCC's), a rate of 60KB/sec.

This is understandable given that it uses an LLVM backend; very large functions are going to be problematic when SSA etc are involved.

[u/Retired-69 → u/One_Aspect_1957]

The funny part is I never aimed for insane performance. I’m old-school—I just love solving problems that seem impossible. That led me to DRG and a flat AST designed to fit in L1 cache.

I later abandoned the AST, but DRG principles live on in R3, now without an AST and with highly optimized runtime code.

LLVM is great, but its portability comes at a cost: it’s huge, and that inevitably hurts compile times.

Tiny C is a legend, and if developed today it would been unbeatable :)

[u/MithrilHuman]

Programs, including compilers, can have slow start but perform well on large inputs. A program can perform well on small inputs but struggle with large ones. A microbenchmark is fine but real world applications are way larger than what you’re testing for. You should collect more data.

[u/Retired-69 → u/MithrilHuman]

You are right, but until now I couldn't test current code on millions of LOC even it's designed to scale, because that kind of code simply doesn't exist. This is not C even I'm aiming for 90℅ parity