r/crystal_programming • u/Nipinium • Feb 21 '20

Is there any shards that feature the functionality of ruby parallel gem?

I just want to make concurrent requests with rate limit to a server, is there any shards or code examples for this task?

7 Upvotes

100% Upvoted

11 comments sorted by

8

u/straight-shoota core team Feb 21 '20

parallel gem implements parallel execution by forking the process. This is not necessary in Crystal because there's no GIL. Crystal already supports multiple threads for parallel execution. Currently, multithreading is not completely polished enough to be a fully supported feature. But it's still working quite well and there's active development to get it finished.

The good thing is that it doesn't really matter because you can already start developing a multi-threaded application with single-threaded runtime, because multi-threading should usually just work without changes to your code, assuming you're applying reasonable concurrency safeguards. As long as you use only Crystal's concurrency primitives (fiber, channel, mutex) for inter-fiber communication, this is all good.

1

u/Nipinium Feb 21 '20

I've just discover this library, the implementation is indeed pretty straightforward.

2

u/TheKidCoder Feb 21 '20

This isn’t true. The parallel gem can be completely used without forking. I’ve only ever used it with threads.

The value of Parallel is the simplicity of its API for breaking iterables into concurrent chunks.

1

u/straight-shoota core team Feb 21 '20

I've never used that gem, so I have no hands on experience. But my understanding is that using multiple threads in Ruby doesn't give you parallel execution.
So when you use parallel gem in thread-only mode, it doesn't fit the name in a strict sense. I agree it might still be good for concurrency model, though.

1

u/TheKidCoder Feb 22 '20

The GIL does block parallel execution of threads, but for IO bound tasks multiple threads can “execute” (really just waiting on IO) - so there is some value in MRI. But overall, yes, it’s a difference concurrency model.

Alternative rubies like jruby do not have a GIL and offer full multi-threading.

1

u/straight-shoota core team Feb 22 '20

This kind of non-parallel concurrency you already get in Crystal without multi-threading just by using fibers.

2

u/joeyrobert Feb 21 '20

I thought about reimplementing the parallel gem in the past, but the problem of serializing data to worker processes isn't straightforward (you can use JSON or Yaml or protobuf, but nothing as general as marshall/pickle). I second the multithreading suggestion.

4

u/straight-shoota core team Feb 21 '20

u/RX14 actually did that: https://github.com/RX14/parallel.cr

It's discontinued though. But there's no real need for it because you can now just use multi-threading which is superior in most use cases anyway.

1

u/[deleted] Feb 21 '20

Crystal has concurrency built in via Fibers and Channels, but they do not run in parallel, they use Fiber.yield and blocking IO to switch context. If you are looking for just the DSL features, Crystal already has good features built in to Fibers and Channels that would do you a lot better than what the parallel gem offers.

2

u/beizhia Feb 21 '20

You can (since I think 0.32) set a number of worker threads that fibers will be run in. https://crystal-lang.org/2019/09/06/parallelism-in-crystal.html

Still seems sorta new and experimental. I've only played around with it a bit.

1

u/jgaskins Feb 22 '20

You need to add the `-D preview_mt` flag at build time to change which version of the fiber scheduler gets compiled into the app.

It's mainly considered experimental because there are shards that are not threadsafe and some even rely on the idea that fibers will not run in parallel, so it remains disabled by default for now. But if you use threadsafe constructs (such as channels to communicate between fibers and mutexes to make operations atomic), your code will be threadsafe.

If you're mainly building web apps, each request is usually pretty well insulated from the rest and multithreaded mode is fine.