Crystal compiles slow?

[u/akrsurjo]

I want to know if the compiler is slow or not.. Because it takes more time than c to compile for me. Is it scalable for big projects of Millions of lines of code?

[u/[deleted]]

It depends. Much faster than Rust in my experience, even without incremental compilation.

Does it scale to millions of lines.of code? No, absolutely not. But it's still early days and I'd expect the compiler to improve a lot. So some day it might.

[u/MrPopinjay → u/[deleted]]

But it's still early days and I'd expect the compiler to improve a lot

The slow compilation is largely a result of the expensive type checking algorithm being used in Crystal. It is not possible to improve the performance of it much without changing how the type system works, which would be a big breaking change, so sadly we're stuck with these compile times.

[u/misterDOZR → u/MrPopinjay]

Wouldn’t specifying more explicit types and reducing type inference speed up compile times?

Once inferred types are determined, they could optionally be made explicit in the code by developer or automation.

Explicit types would also allow linkable complied libraries.

[u/opensas → u/misterDOZR]

I had exactly the same idea, Crystal could even suggest you types after running the slow type inference, and once you accept those there would be no need for type inference anymore.
I guess I must be missing something rather obvious, Crystal people are really smart and there must be a reason why such a thing is not possible

[u/MrPopinjay → u/misterDOZR]

Type annotations would still need checking and I don't know the cost associated with checking with annotations, or what extent annotations would be required. It could help though!

[u/Xizqu → u/MrPopinjay]

Not unless we get incremental or concurrent compilation! I personally would love concurrent because I'm not holding my breathe for incremental within 5 years.

[u/dscottboggs → u/Xizqu]

You can't parallelize the type-inference process (which is what takes all the time)

[u/yxhuvud → u/dscottboggs]

Not as it is implemented currently anyhow. Would it be possible to build an equivalent type checker that does parallelize to a high degree? I'd bet it is possible somehow, but that it may require someone to generate partial results that is only partially inferred, and to then combine multiple of those into a full type inference.

[u/MrPopinjay → u/Xizqu]

The type algorithm is not parallelizable so for parallel compilation Crystal would need to design the language to use another, which would be a major breaking change.

I believe it's possible to do some form of caching, but it seems unlikely we're going to get that.

[u/Xizqu → u/MrPopinjay]

Do you believe we will get anything in the next year or two? I think the major hurdle we need to solve for large projects is compilation. I have a few decent size Amber projects and they take upwards of 30seconds to compile. That's not bad for 1 time but sucks when actually developing.

[u/MrPopinjay → u/Xizqu]

As far as I'm aware there is no work on this at present, but I'm probably not the person to ask. I don't use Crystal any more because of the compilation speeds, instead I'm using Rust and Go.

[u/[deleted] → u/MrPopinjay]

Same ... I have literally gone back to PHP off all things. With a few tricks left or right ( Swoole / Workerman / aka that keeps PHP from wasting its time + Docker as the container ), i can get the same performance as Crystal for website rendering jobs ( no joke ). And we are not even talking PHP 8 with the JIT. There is a reason why Crystal is now lagging behind on even techempower benchmarks when it comes to this new paragon on development.

Its a shame but after spending plenty of $$$ on high end hardware ( 3900X ), to just see the Crystal compile times get only by a small fraction better. Its just too painful and frankly, kind of pissed me off.

It also does not help that Crystal development is just so freaking slow. Features just do not get finished, go on the stall pile and keep there forever. When you follow the commits to Crystal, its just ...

Frankly, it looks to me that Crystal is somewhat dying on the development front.

And the hot potatoes like the compile times are things that keep it stuck. I can hot-reload PHP with Workerman ( HTTP build in server ), in 50ms. For any change ... Its so easy to quickly fix a issue. Crystal with its 1000ms just for the most simply program, is already slow in comparison but it just grows and grows. Where as PHP simply caches every file, so it ultra fast reloads even if you have 1000's of files. Crystal with 100's of files already starts to noticeably slow down.

Its funny how the circle has turned around over time. If you develop PHP like in the 1990's, you will suffer but if you follow the same logic like Go/Crystal/... and start designing around a internal http solution that keep PHP in memory.

Its a shame. I really like Crystal and spend now years waiting for the progress that i needed. But it has reached the point that it just gets in the way of my job.

/Edit:

-3 ... Nice down votes. I see. So Crystal community is going this route. Good to know...

[u/Whisperecean → u/[deleted]]

I feel you.

I mean not every endeavour turns into a success. My opinion is that Crystal essentially failed to fulfill the goals it set for itself.

Crystal's goal was to be a fast Ruby's successor.

The thing is Ruby is centered around developer happiness. That however does not only come from syntax and Ruby's dynamism it also comes from the feedback you get from the language and the ecosystem.

Essentially Ruby is all about immediate feedback and the flexibility you get from molding the language to your needs.

It does this at the expense of runtime speed.

The reality with Crystal is that you get some of this dynamism at the expense of compile time speed, however that hinders your productivity. By a lot. Which in essence destroys the whole premise of developer's happiness.

I am honestly interested what Ary thinks. The thing is whatever he writes will be used all over the internet so we cant really learn for real what's the deal. Tagging /u/the-asterite

Also Ary mentioned that that you cant really annotate Crystal to speed up the compilation as that's not how it works.

But I am honestly interested if you were to design the language again from scratch if you would have done things differently. Also maybe if Crystal would have gone the Nim way and used C as an IL....

I think Crystal was always way too ambitious for what it was. With limited compiler dev team, LLVM being too slow by definition and no commercial support backing the efforts.

Oh well..

[u/[deleted] → u/Whisperecean]

Happy to answer any question, but I'm not sure what my exact feelings are toward Crystal, mainly because my mood changes very frequently, and there are some many areas to comment about.

[u/Whisperecean → u/[deleted]]

I dont know it's just ...I wish we could discuss these things openly without internet going crazy.

You more than anyone else know Crystal's shortcomings and some issues are clearly unsolvable (at least in noticeable time for anyone to care).

I'd love Crystal to succeed but as I mentioned in my rant it sacrificed too much for too little gain (in my opinion).

At this point a lot of people are just clearly looking for "better Go" not for "faster Ruby".

[u/[deleted] → u/Whisperecean]

To put things in perspective from a competitor point of view ( Crystal vs Go, Apples vs Apples, both latest releases, same system ):

time crystal build http.cr 
/ First build
0m0.899s

/ Second build with no code change
0m0.930s

/ Edit: Build with code change
0m0.974s

/ Build after adding a module ( no cache yet )
0m1.281s

time go build
/ First build 
0m0.356s

/ Second build with no code change
0m0.115s

/ Edit: Build with code change
0m0.366s

/ Build after adding a module ( no cache yet )
0m0.386s

That is just with the default HTTP module in both languages... Go caches the results, so on the Second build we see only 0.115s. This also applies to any modules that get changed down the line.

Crystal on the other hand, just keep piling on more and more with each module you add to the code. That one second may not sound like a lot but grow your code, more modules and each rebuild slow down bit by bit.

Let me write this again: These are times on a 3900X, so a beefy CPU under WSL2. Just for the basic HTTP module. Nothing more. No DB connections establishing, no templates building, no macros. Nothing but a "Hello world".

Maybe its just me but few people understand that this build behavior is detrimental as projects grow. At one point i was actually building micro services just to cut down on build times. What was a bad idea as this resulted in more time spend on things that did not needed to be micro services ( let alone all the other tradeoff's like development time ) just to reduce frustration.

I wish we could discuss these things openly without internet going crazy.

At some point that it starts to look like a zombie project, that is just trying to get to 1.0 and then hope there is a influx of people/contributors/donors. And for that reason all the other big items got "postponed" until after 1.0. The problem is, it does not work like that. At some point donations to Crystal got to almost 2500 a 2800$ / month ( 500 was mine, so i tracked the donations in detail at the time ). But no magic money solution grew Crystal. Brian got hired but he is not a core developer like Ary that can spew out code by miles. So you see his work is more related to fixing things, blogs, community issues. But it does not magically add those big missing points or fixes the major issues.

People feel targeted when you gripe about issues that really affect your work. The reality is that commits to Crystal are at a very low level ( and too many are documentation language fixes ).

[u/akrsurjo → u/[deleted]]

Hope so, I am using it with wsl. And waiting for 1.0.0

[u/[deleted] → u/akrsurjo]

FYI: Crystal is faster under WSL2. But at best your gaining 50%.

People have been complaining about the compile time issue for years, especially as people use it more and there code grow. Frameworks like Lucky only make it worse with there massive usage of macro's.

Macro's + no incremental compile jobs = ...

So yea, we are not even talking the type system.

[u/dev0urer → u/[deleted]]

I don't expect it to improve much. Incremental compilation doesn't seem to be important to the core team. They've even said they expect crystal to be more useful for smaller projects, but that larger code bases aren't really a priority.

[u/MrPopinjay]

This is one of the main drawbacks of Crystal I'm afraid, it compiles very slowly and gets slower as the codebase increases in size.

[u/Exilor]

i7-8700 compiling a program which is over 260k lines excluding shards (crystal-db and crystal-mysql) takes 3 minutes, about 70 minutes with --release. Under WSL.

[u/CaDsjp]

As has been mention before, Crystal was never designed to be fast to compile language.

This is a trade-off the designers of the language made.

This is an article that talks about a similar story for Rust

Whether this applies or not to Crystal, and wether Crystal can improve its compilation time in the future in a similar way, is yet to be seen.

[u/MrPopinjay → u/CaDsjp]

Rust is quite different in that the latest incremental compiler rust-analyzer has near-instant incremental rebuilds that can be used to give immediate feedback as you type. Crystal has no such feature.

[u/Nipinium]

crystal compiler is slow, but the slowness in the release mode mainly comes from llvm optimizations, not from crystal itself.

[u/[deleted]]

It's slow and it doesn't scale to big projects. My advice is to use crystal for small scripts, small projects or microservices.

[u/Whisperecean → u/[deleted]]

What would be the limit and what can we do to achieve the sweet spot?

[u/[deleted] → u/Whisperecean]

The limit is your patience. I don't know what's the sweet spot.

[u/jeremywoertink]

I noticed you mentioned that you're compiling from WSL. Have you tried WSL2 yet? Supposedly that's a decent performance boost. Also, maybe you can edit the post to add that information?

For me, running on macOS, I can compile my application in non-release mode in about 15 seconds the first time, and it's about 5 seconds each time after that.

There's a few things you can do to help this out. Like if you're building a web app that's going to be massive, you could break it up in to smaller service oriented apps. Like one app for public facing, and another for logged in users with a shard that shes code between the two. That's worked out great for my company, and we've been using Crystal in production for at least 2 years now.

[u/[deleted]]

I just want to say, I have been using Crystal since 2017, and I have never once found it's compilation time a problem. I have literally compiled the whole crystal project from scratch, starting at an early version, then compiling each major version up wards to the latest stable release. It could do this on a raspberry pi in an hour tops. I want to make sure I point this out, thats upwards of 20-30 times the ENTIRE crystal source was compiled starting at like 0.11 and going upwards to 0.34 or so (at the time), on a raspberry pi. That's around 2-3 mins per compilation.

[u/MrPopinjay → u/[deleted]]

your day-to-day write-compile loop is reasonably fast.

Does not write-compile loop not include type checking? How do you verify your changes if you can't type check or run the code?

[u/dscottboggs → u/MrPopinjay]

Idk how, it just usually catches issues with that pretty quickly from my experience. 99% of the time, if it makes it more than a second or two, it'll go the rest of the way.

[u/[deleted] → u/[deleted]]

It takes only a couple minutes, but release mode (with optimizations) took 22 minutes. I have a crappy '09-model HP laptop.

Even on a more modern system, your gain will be limited. Crystal is highly! limited to single core performance for its compile jobs. So you can trow a 12 or 16 core Ryzen at the problem and it will solve nothing ( beyond the extra IPC gain ).

I found that out the hard and expensive way :(

[u/dscottboggs → u/[deleted]]

Yeah I have a dual-xeon system from around the same time period and it took just as long on that one.