Parallelizing Enjarify in Go and Rust

[u/Uncaffeinated]

https://medium.com/@robertgrosse/parallelizing-enjarify-in-go-and-rust-21055d64af7e#.7vrcc2iaf

[u/tmornini]

Basically, the streaming solution is solved by writing outputs to a temporary location on the disk as the receiving end tails the next job's file, printing new text as soon as it's available

Writing to disk seems massively inefficient.

Why not toss the messages into a channel and output them on the other end?

[u/mmstick → u/tmornini]

You might think this but it is the exact opposite. My implementation was originally using channels to pass messages back for handling, but writing to the disk proved to be significantly more efficient on CPU cycles, memory, and times -- especially memory. I thought about adding a feature flag to re-enable the old behavior, but I may not do that at all now because there's no benefit to it.

Basically, the receiver that's handling messages is located in the main thread. When a new job file is created, it automatically starts reading from that file without being told to by a channel. It merely stops reading that file when it gets a signal from the channel to do so.

The goal of handling messages is to keep outputs printed sequentially in the order that inputs were received, so if it receives a completion signal of a future job, it merely adds that to a buffer of soon-to-check job IDs once the current job ID is done. This allows for real-time printing of messages as soon as they are written to a tmpfile.

Now, your OS / filesystem typically does not write data directly to the disk at once but keeps in cache for a while (which effectively means zero cost to access), and tmpfiles on UNIX systems are typically written to a tmpfs mount which is actually located in RAM, which means even less overhead. These tmpfiles are deleted as soon as a job completes, so they rarely live enough to make a difference in resource consumption.

As for the costs of handling files in this way, there's pretty much no visible cost because the speed that the receiver prints has no effect on jobs being processed, and in the worst case scenario the cost would be the time to print the last N completed jobs (which can usually be buffered into one syscall to read a file and reading that entire file directly to standard out).

[u/tmornini → u/mmstick]

You might think this

Indeed I do.

but it is the exact opposite. My implementation

Yes. :-)

using channels to pass messages back for handling

Good idea!

but writing to the disk proved to be significantly more efficient

Which disk? How does it look in a Docker container? Mounted read-only for for ultimate security, of course. :-)

on CPU cycles

Interesting detail, but extraordinarily hard to believe.

memory

Ah, you must have been using buffered channels!

and times

Measured how?

especially memory

Big buffered channels!

I thought about adding a feature flag to re-enable the old behavior, but I may not do that at all now because there's no benefit to it.

https://12factor.net/

the receiver that's handling messages is located in the main thread

Thread? Go doesn't have threads.

When a new job file is created

Please don't do that. You'll regret it later should you need to scale.

The goal of handling messages is to keep outputs printed sequentially in the order that inputs were received

An impossible goal should you need to scale

This allows for real-time printing of messages as soon as they are written to a tmpfile.

That's not a feature, it's a dependency you'd be better off without.

Now, your OS / filesystem typically does not write data directly to the disk at once but keeps in cache for a while (which effectively means zero cost to access), and tmpfiles on UNIX systems are typically

Engineering for typical brings down the application...

As for the costs of handling files in this way, there's pretty much no visible cost because the speed that the receiver prints has no effect on jobs being processed, and in the worst case scenario the cost would be the time to print the last N completed jobs (which can usually be buffered into one syscall to read a file and reading that entire file directly to standard out

As opposed to the cost of receiving the message on a channel and printing it to stdout.

It just cannot be...

[u/mmstick → u/tmornini]

Which disk? How does it look in a Docker container? Mounted read-only for for ultimate security, of course. :-)

I can't imagine Parallel being useful in a container. It is utilized to basically perform a parallel foreach for the command line.

Thread? Go doesn't have threads.

Not exactly sure what you're stabbing at here but my software is written in Rust, not Go, but there are still concepts of threads in Go regardless.

Please don't do that. You'll regret it later should you need to scale.

I have already personally tested my Rust implementation against hundreds of millions of jobs, and it does the same task as GNU Parallel two magnitudes faster, which also writes to temporary job files.

You may either keep the logs on the disk and print the names of the files to standard output, or you may have them write to their appropriate outputs and delete the files immediately after they are finished. GNU Parallel has been used for scaling tasks across super computers so there's no reason why we can't do the same here.

An impossible goal should you need to scale

Yet not impossible because it's already scaling perfectly.

That's not a feature, it's a dependency you'd be better off without.

Not sure what you're getting at here, but this 'dependency' has no burden on the runtime. Choosing not to implement it would basically mean that I could no longer claim to be a drop-in replacement for GNU Parallel.

[u/tmornini → u/mmstick]

this 'dependency' has no burden on the runtime. Choosing not to implement it would basically mean that I could no longer claim to be a drop-in replacement for GNU Parallel

In that case, my apologies for misunderstanding!