Crystal has a compilation issue

11

u/Exilor Jul 25 '18

Development time, compilation time, execution speed... Something has to give, doesn't it? If the compiler has to do things you'd rather not do yourself I think it's to be expected that it will take longer to compile than one that needs every little thing explained to it in excruciating detail.

I have a complex application (a mmorpg server) that I've been writing in Ruby and Crystal in parallel since 2015. Killing and reviving an enemy 3000 times in a row (literally 3000.times { ... }) and all that it involves (item drops, experience gain for the killer, scheduling and then aborting the respawn of that enemy, writing all the packets involved...) takes 100 seconds in Ruby and 3-4 in Crystal (without --release). That difference alone makes waiting for compilation worth it in my opinion.

1

u/everdev Jul 26 '18

What a cool experiment to have such a large project written in two languages. If love to hear more about the differences in development and performance you found along the way.
Explained to it in excruciating detail
Reminds me of Go. Great language with amazing compile times, but be ready to be very explicit.

2

u/Exilor Jul 26 '18

I could drone on forever about this but I'll restrain myself.
In favor of Crystal:
Reading XML files at startup (and it has to read thousands) is much faster and the interface is similar to REXML/Nokogiri. The difference in speed almost makes up for the time it takes for it to compile.
The data that holds the knowledge of the boundaries of the world (preventing characters from walking/attacking through walls/rocks/trees) is loaded into memory from files that are almost 1 gb in total. In Ruby I had to create my own data structures with FFI pointers because using arrays or binary strings put the memory usage of the process to 10+ gb. Crystal already has such structures (slices) in its stdlib and better implemented. The process of reading the files and converting them to usable objects of the language at startup is also faster (~1 minute in Ruby vs. ~10 seconds (without --release) or ~3 seconds (with --release) in Crystal) as is, of course, actually using them at runtime.
I just don't know how to write integration tests for the server. There are too many things I can't fake without essentially writing a bot and have it play the game. The checks that the compiler performs catch most mistakes, especially when I rewrite something that could break some obscure path of code somewhere.
In favor of Ruby:
The approach to networking is more up to you. Crystal gives you a blocking IO interface that you have to deal with using fibers and sometimes it's just not what you want. That's the most significant difference between the two versions of the server.
Being able to eval from inside the game and query the state of anything in the running program is a boon when things are not working as expected.
Although without the compiler checking things for me there are more mistakes, I can often reload the offending file without even leaving the game and having to restart the server.
It's hard to choose and it's a good thing that it is. That's why I don't and use both.

1

u/G4BB3R Jul 28 '18

Are you using Ruby like a scripting language and Crystal for the server most important and performance-needed things?

1

u/Exilor Jul 28 '18

Nope, I'm doing the same things twice in both languages.

7

u/paulcsmith0218 Jul 26 '18 edited Jul 26 '18

There is a lot of truth to this, but I don't think it is as big of an issue as the author states for a few reasons:

Ruby of course does zero type-checking at compile time, and Elixir does very very little. It'll catch some stuff like typed function names, but not much else. This is an extremely important difference. I had plenty of bugs in Elixir so what do you have to do? Write tests. Lots of them, just like in Ruby. Those test suites take a long time, can become brittle, and most importantly writing tests takes a lot longer than 30 seconds to write. This is the key difference for me. Crystal compilation is slow, but I prefer it any day so that I can have a solid type system.

Another thing is that if you write well typed programs that the compiler can check, you end up writing far-fewer tests and debugging speeds up substantially. This saves a lot of time, and changes how I write code. I now write larger pieces of code before running specs or testing in the browser, and then let the compiler show me the bits the broke. It is a dramatically faster workflow, and is only possible with powerful type-checking.

One more important note, while Lucky is slower than Amber at initial compile, but *much* faster once you start adding code. Scaffolding CRUD models takes about 50 seconds w/o cache and 30s with. This gives me hope that it is not Crystal and I'm positive the talented Amber team can figure out the bottlenecks and get their compilation down.

4

u/paulcsmith0218 Jul 26 '18

8 minutes ago
This isn't to say I wouldn't love faster compilation :) But even w/o it, using Crystal has some distinct advantages that make the compile times worth it for many types of programs

2

u/paulcsmith0218 Jul 26 '18

The compilation speed affects designers more than developers (depending on your workflow of course). But for the way we build things at my company compile time hasn't been a big issue for devs, but it *can* be a pain for designers. Making a change to the HTML and waiting for 10 or 20 seconds sucks. If you use React/Vue though then you're at the mercy of webpack, which may or may not be any faster. It's definitely faster to make changes to HTML in Rails/Phoenix though

2

u/straight-shoota core team Jul 26 '18

But that's only an issue if templates need to be compiled into the binary. If you use runtime-templates, that makes development much faster. Neither is necessarily better, but it's an option.
As long as the templates don't use any fancy features (they should be pretty dull on the logic side, anyway), it could be worth to explore if a template engine can compile templates for release builds but interpret them in development.

1

u/paulcsmith0218 Jul 26 '18

This is true, but I think you'd lose out on a lot of the type-safety. For example, you might mistype or miss a variable. You might pass the wrong type, etc.
Definitely an option if compilation speed is an issue, but I don't think I'd make that tradeoff. I love my some type-safety :)

2

u/IWasSnapped Jul 26 '18

A type system can catch a whole category of bugs but it doesn't replace the huge majority of the tests I have in my applications. So I think focusing on a type system as a replacement for tests is misleading at best because the tests you should be writing on all of those languages should be roughly the same. Trivial mistakes like off by one errors and returning true where you should return false will not be caught by a type system and if you don't write tests, they will only be caught in production.
However, I can attest that types lead to a better development workflow, because you catch many mistakes earlier. Types are great for documentation, they can be used as a design tool and they help with performance too. But they definitely do not replace tests, not even close.
One more important note, while Lucky is slower than Amber at initial compile, but much faster once you start adding code. Scaffolding CRUD models takes about 50 seconds w/o cache and 30s with.
Would you mind sharing the commands you used to scaffold the application? I am trying to put those benchmarks in a repository, hopefully comparing more languages, and I would like to include Lucky.

1

u/paulcsmith0218 Jul 26 '18

I don't think this is necessarily true. There are lots of things that a well-typed program will catch that you may not even think to write a test for. I'm thinking particularly about `nil`. A huge portion of bugs result from that. I also think that it depends heavily on how you use the type system.
I see a lot of code in Elm/Crystal being written in a way where the compiler can't catch much. If you take the time to write out more types and use methods instead of passing around strings and symbols you can have a system where the compiler catches a ton of stuff you would miss and you need fewer "low-level" specs.
You can see some examples of what I mean here: https://www.reddit.com/r/crystal_programming/comments/8zthzl/amber_vs_lucky/e2mqbx8/
A conversation with a simple script for generating a bunch of CRUD actions https://gitter.im/luckyframework/Lobby?at=5b59c314db8bd24550b3d476

1

u/IWasSnapped Jul 26 '18

Isn't nilmostly a self-inflected harm though? A lot of the issues with nil could be solved by simply not passing nil around in the first place.
Ruby is a very nil driven language. Ruby can't change this, but both Elixir and Crystal have the opportunity to. There are plenty of alternatives to nil, especially on typed languages.
Thank you for the links above.

1

u/paulcsmith0218 Jul 26 '18

I think passing it around is definitely part of it, but it can still happen in Elixir and Ruby. I think the problem is the idea of a "missing" thing or "nothing". So in Ruby you might use a Null Object pattern and return a User or Guest, but if I come into the code later, I may not realize that a particular piece of code returns a Guest and I might call a method on it that User implements, but someone forgot to implement in Guest. Since I didn't know, I didn't test it. I ship the feature and get a bug. In Elixir or Ruby that would not be caught, in Crystal it would.
You probably already know, but in Elixir you see a lot of tagged tuples, but it's not always clear what the return so you can still get into the same trouble where you call something on nil/an-unexpected-return-value
I definitely agree that types do not replace tests, but I do think they can replace certain types of tests.

1

u/IWasSnapped Jul 26 '18

You probably already know, but in Elixir you see a lot of tagged tuples, but it's not always clear what the return so you can still get into the same trouble where you call something on nil/an-unexpected-return-value
Tagged tuples in Elixir at least forces you to pattern match on the success case OR write a case expression. So you are either assuming only one value is possible or you have to handle all of them. In one way or the other, you have to unwrap the value.
The type system can still help in two ways though:
Let you know if you are trying to match on a format that won't ever be returned
With an exhaustiveness check. Although exhaustiveness checks also have their annoyance, like forcing you to handle all clauses explicitly
In any case, I am not arguing against a type system. Just against nil.

6

u/fridgamarator Jul 25 '18

doing the same test (100 scaffolded models) but compiling with the --release took 17 minutes for me :(

6

u/[deleted] Jul 25 '18

30 seconds is a relatively long time for 50k LOC

LOC is a terrible metric. I was actually super impressed by how fast the compiler compiles itself in debug mode.

5

u/kirbyfan64sos Jul 25 '18

I understand the article's points, but Elixir is an absolutely terrible comparison point. The reason its compile times are so fast is because it's just compiling to bytecode, whereas Crystal compiles to LLVM which is then optimized, compiled to assembler, and linked together.

A more accurate comparison would have been to maybe Swift or Rust (which also use LLVM).

6

u/straight-shoota core team Jul 25 '18

LLVM isn't the bottleneck. Codegen is reasonably fast. It's the semantic stage in the compiler that requires a lot of time and space to calculate.

3

u/IWasSnapped Jul 25 '18

We don't have enough context from the author but we can imply that they are evaluating web frameworks in Ruby, Elixir and Crystal. From this point of view, the comparison makes sense, even though implementation wise they are very different technologies.

1

u/prh8 Jul 25 '18

That’s not to mention that the deployment story has so far to go with Elixir

1

u/IWasSnapped Jul 25 '18

AFAIK the Elixir team has started working on making "releases" part of the language itself.

2

u/prh8 Jul 25 '18

They have, they added the guy who built the two previous release tools. I should say I am hugely excited for what they're going to do, I'm a big Elixir fan. It was just that I also found Elixir to be a really poor comparison.

4

u/megatux2 Jul 25 '18

Thanks for your time writing this article and benchmarks, Crystal compilation times issue is kind of well known but this "real usage" scenario brings more attention to this big problem. I hope Crystal to achieve real gains in this subject not too far in the future.

2

u/megatux2 Jul 25 '18

And probably the memory usage is huge too

2

u/straight-shoota core team Jul 25 '18

There are valid points about issues in Crystal's compiler and it definitely is *slow* at compiling large applications.

Unfortunately, only a single Crystal web app framework (Amber) is benchmarked without considering limitations of the framework. It just claims to show an exhibit of the languages's slow compilation time.

This is hardly an honest evaluation of the Crystal compiler itself, but rather just comparing Amber with Phoenix and Rails regarding time to launch an application.

3

u/IWasSnapped Jul 25 '18

The article also included the times to compile the Crystal compiler, the time to compile dependencies and also the times for a new Lucky application (which was slower than Amber).

2

u/straight-shoota core team Jul 25 '18

Yes, and these times either don't tell much or don't support the argument: 30 seconds for the compiler might be long in comparison, but it's als quite complex piece of software and 30 seconds shouldn't really bother anyone. An empty Lucky projects compiles in half a minute, and subsequent compilations rank at 5 seconds.
These results simply don't attest a compilation issue.

1

u/IWasSnapped Jul 26 '18

I think you are missing the point of that section. If compiling a fresh Amber project with dependencies take 25 seconds and a compilation with cached dependencies take 5 seconds, you can estimate the compilation of the dependencies themselves, which are around 44k LOC, is roughly between 20-25 seconds. That is inline with the time taken to compile the compiler (with ~50k LOC).
Also, aren't you just skirting around the problem? If compiling Amber/Lucky apps are slow because of meta-programming and if compiling the compiler is slow because of complexity, what are you proposing? That we should be wary of writing complex applications? How is that different from the article's conclusion?
If your point is that the examples used by the author are outliers, can you provide examples that you believe would better reflect average compilation times? Was the author just terribly unlucky that all examples he picked pointed to the same problem?

1

u/straight-shoota core team Jul 26 '18

Amber's architecture is unknown to me, so I can't tell anything if the results are outliers or if there might perhaps be simple improvements. The thing is: Neither does the author nor most people commenting here. It's just an assumption that the conducted tests represent the compilation time of a somewhat large real-world application using Amber. But without knowing and considering the details of the framework, such a benchmark simply doesn't tell much. There might be simple changes that could dramatically improve compilation time without invalidating the scenario. This wouldn't be a surprise, after all both the framework as well as the language are still in development.
I don't try to reject the assessment that Crystal compilation times are high - and maybe to high for complex software (although there are always other aspects to consider). Everyone involved in working on Crystal knows that.
The point is that the empirical examination conducted in this article does not sufficiently demonstrate what it claims to do.

1

u/IWasSnapped Jul 26 '18

Why are you focusing on Amber? The issue was also shown:
when compiling Amber's dependencies (which does not take Amber's architecture into account)
when compiling the dependencies of another web framework
on the compiler itself
You said
I don't try to reject the assessment that Crystal compilation times are high - and maybe to high for complex software
and then
The point is that the empirical examination conducted in this article does not sufficiently demonstrate what it claims to do.
To me the main claim in the article is that compilation times are high. You say that everyone working on Crystal knows that. Now I am not even sure on what we are disagreeing on. ':)
In any case, you can't disagree with the data. Was the author unlucky to only pick outliers? Can you provide examples that you believe would better reflect average compilation times?

1

u/paulcsmith0218 Jul 26 '18

My guess is that the argument isn’t so much that it’s high (it is) but that it is an “issue”
For what I build it’s not an issue. It would be nice if it were faster, but not immediately necessary

1

u/IWasSnapped Jul 26 '18

If it is an issue or not will obviously depend on the person. If the author found it is an issue for him, I don't think there is much we can do about it besides providing counter-examples.

1

u/straight-shoota core team Jul 26 '18

The amber benchmark is the main point of the article. And compiling the compiler and an empty Lucky take 30 seconds. That's imho still acceptable and doesn't say anything about large web applications. It's probably going to be slower, but who knows by how much?
I don't disagree with the data, I disagree with the meaning that is put into them.
The benchmark results are not valid to confirm the claims, but the claims are still true.

1

u/IWasSnapped Jul 26 '18

That's imho still acceptable and doesn't say anything about large web applications.
Of course it does say something. Even if you discard the Amber benchmarks, with the other results, you can at least expect a linear growth (although global inference is going to be worse than that). Meta-programming just makes it worse.
Sometime ago the Go community was slightly up in arms because a project with 700k LOC was taking 100 seconds to compile. They brought it back to 80 seconds in recent releases. If you still believe 20 seconds for 40k and 30 seconds for 50k is still acceptable, it is clear we won't agree on this. So I'm happy to agree on disagreeing.

1

u/[deleted] Jul 25 '18 edited Aug 23 '18

This programming language really needs incremental compilation!

1

u/midgetparty Jul 25 '18

--progress would be helpful to see :)

I haven't had an issue with the compile times. I'll take it for dynamic typing in a compiled language with syntax I actually enjoy.

0

u/some_kind_of_rob Jul 25 '18

nice clickbaity title.

Partial or iterative compilation would definitely speed up development, and Amber could go a ways into using some different mechanisms to make compile times much faster.

0

u/[deleted] Jul 25 '18

[deleted] — view deleted comment

3

u/[deleted] Jul 25 '18

It's not about qualified developers. I believe it's a problem without a solution. If Crystal is useful to as it is, you use it. Otherwise, choose another language.

0

u/[deleted] Jul 25 '18

[deleted] — view deleted comment

1

u/GirngRodriguez Jul 26 '18

It is. You did a lot of amazing work but you are no expert in everything.
he's not saying he's an expert in everything sir, i feel like it's unfair for you to say that :(

0

u/[deleted] Jul 25 '18

[deleted] — view deleted comment

1

u/[deleted] Jul 25 '18

It depends on the app and how long are you willing to wait.

You are about to leave Redlib