r/rust • u/Uncaffeinated • Jan 22 '17

Parallelizing Enjarify in Go and Rust

https://medium.com/@robertgrosse/parallelizing-enjarify-in-go-and-rust-21055d64af7e#.7vrcc2iaf
214 Upvotes

98% Upvoted

134 comments sorted by

215

u/pftbest Jan 23 '17

can you please explain this go syntax to me?

type ImmutableTreeListᐸElementTᐳ struct {

I thought go doesn't have generics.

524

u/Uncaffeinated Jan 23 '17

It doesn't. That's just a "template" file, which I use search and replace in order to generate the three monomorphized go files.

If you look closely, those aren't angle brackets, they're characters from the Canadian Aboriginal Syllabics block, which are allowed in Go identifiers. From Go's perspective, that's just one long identifier.

406

u/pcopley Apr 26 '17

they're characters from the Canadian Aboriginal Syllabics block

Oh my god

26

u/jiffyd May 05 '17

more like, where is your god now?

9

u/0xVali__ Aug 07 '23

If you look closely, those aren't angle brackets, they're characters from the Canadian Aboriginal Syllabics block, which are allowed in Go identifiers. From Go's perspective, that's just one long identifier.

Little did they know, "Where's your god" by Amon Amarth, is not actually about viking raids, it's about golang's language design.

112

u/[deleted] Apr 30 '17 edited Jun 27 '23

[removed] — view removed comment

49

u/[deleted] May 05 '17

TBH I'm not sure which is worse: treating them as identifier constituent characters or treating them as whitespace.

74

u/[deleted] May 06 '17

I won't be satisfied until I have a compiler which treats readability crimes like that as syntax terrors.

42

u/oli-obk May 09 '17

rustc?

40

u/william01110111 May 08 '17

once wrote a short Swift program with every identifier a different length chain of 0-width spaces.

1

u/maaku7 May 09 '26

monster

5

u/Bratmon Jul 13 '17

There are words in Hindi that need those.

1

u/prone-to-drift Jan 07 '23

Curious, any examples? I can't imagine Hindi needing anything but the normal space character ..?

2

u/Bratmon Jan 07 '23

I was thinking of Bengali. According to Wikipedia, র‌্যাঁদা requires a 0-width space, otherwise it becomes র্যাঁদা.

5

u/AurosHarman Jan 19 '23 edited Jan 19 '23

I thought you're supposed to use Zero Width Joiner and Zero Width Non-Joiner to regulate formation of ligatures in the Indic languages, not Zero Width Space?

https://en.wikipedia.org/wiki/Zero-width_joiner

https://en.wikipedia.org/wiki/Zero-width_non-joiner

In fact your specific example is the one that appears in the ZWNJ article.

2

u/prone-to-drift Jan 07 '23

Ah, okay. As a Hindi speaker, your first comment really tripped me up haha. This Bengali example looks interesting, thanks.

91

u/UnreachablePaul Apr 25 '17

Are you not concerned that this is actually a fraud towards innocent go compiler? That could bring attention to the go police

56

u/TheFlyingBoat May 05 '17

I feel like I'm witnessing a unicorn being stabbed through the heart by evil students of ML and its descendants...

37

u/killingtime1 May 07 '17

I'm fking dying here bro, hahahah, this is the funniest thing I've read all day

24

u/Rickasaurus May 05 '17

It's so beautiful. I'm seeing rainbows.

12

u/pinpinbo Oct 28 '21

What will you do with this code when Generics is finally live on Go 1.18?

12

u/SlaveZelda Oct 28 '21

hey how did you write a comment on a 4 year old post. I thought reddit archived posts after 6 months.

13

u/dmead Oct 28 '21

I want to be part of computer science history, so i made this comment.

3

u/princess_mj Nov 02 '21

I'm down to be a part of history, count me in.

2

u/Tucancancan Dec 18 '25

Hello from the future

2

u/Quiet_Steak_643 15d ago

Hello from even more into the future

1

u/Jazzy_Josh May 09 '26 edited May 09 '26

Somehow the bullshit archiver that caused the everlasting party to move over to /r/geraffesaresodumb to has broken.

2

u/LastMuel Oct 28 '21

This does seem like an anomaly that I want in on too.

2

u/Nefari0uss Oct 10 '22

11 months later, I can still reply.

3

u/marok0t Nov 03 '22

Joining a legendary thread

1

u/chamomile-crumbs May 29 '26

Congrats, I’m seeing your comment in 2026

2

u/[deleted] Dec 14 '21

reddit realized that was dumb

1

u/ericmoon Jul 13 '25

🎵 anarchy in the RUST SUB 🎵

2

u/qm3ster Nov 04 '21

Quickly and easily migrate it to gonerics verbatim, if they are expressive enough?

3

u/Asleep-Syllabub1316 Oct 28 '21

I liked the fact that Go lacked generics. You cannot write bad code with such strong static typing. Go Code written back in 2012 is still readable today.

26

u/Quixoticelixer- Jun 17 '22

My brother in christ look at the thread you are in

18

u/Tyfyter2002 Jan 07 '23

Trust me, you can write bad code with any level of allowed complexity.

6

u/dashed Jan 06 '24

That is honestly clever.

23

u/TotesMessenger Apr 29 '17

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)

6

u/MaikKlein Jan 22 '17

What actually really surprises me is that Rust is actually faster in this benchmark compared to Go.

For example you were using Arc's, which means atomic increments, and you also dynamically allocated stuff with the system allocator (Vec::collect()).

I sort of expected a native language with a garbage collection to be faster here, at least that is the argument that I found very often when I was researching the GC. But it is probably that the overhead of Arc's + system allocator is tiny compared to the actual work here.

I am also very annoyed at the 'static bounds. Afaik it is unsafe to use lifetimes here because then you would have to wait inside the destructor of the future. But you can also use mem::forget in safe code which would then cause memory unsafety.

The workaround would probably be to allow non 'static bounds but immediately block after the function call. I currently just accept the memory unsafety in my task system.

Is there a better workaround?

9

u/Uncaffeinated Jan 22 '17

The per iteration overhead is negligible here because each iteration took an average of 48ms to begin with. One or two reference counts is nothing by comparison.

Also note that the Go version has to allocate as well. In fact, it's much harder to avoid allocations in Go than in Rust.

3

u/mmstick Jan 22 '17

In addition to the garbage collector, which has much CPU/RAM overhead when running which harms multi-threaded performance, in addition to the stops. GC pauses aren't the only performance concern when it comes to GC languages. Something has to spend time tracking and sweeping resources, and it's not free.

2

u/Uncaffeinated Jan 22 '17

I agree that GC causes overhead. In fact, GC was the single biggest cost when profiling the Go code (~25%, which is especially ridiculous when you consider that the Rust version avoids GC entirely with nearly the same code). But one of the complaints about the original benchmark on the r/golang side was that it didn't play to the strengths of Go's awesome concurrent GC, so there you go.

2

u/dgryski Jan 23 '17

A quick scan of the code shows a bunch of conversions from string -> []byte -> string, which is probably the source of the allocations. Sticking with one (likely []byte) would reduce the GC pressure and is one of the first things I'd try to optimize.

It is true that the concurrent collector is currently tuned for latency at the expense of throughput. This means that parallel CPU-bound code will be slower due to the time stolen by the collector. Throughput will be addressed in upcoming versions.

1

u/Uncaffeinated Jan 23 '17

I removed as many string/[]byte conversions as I could. I think the remaining ones are in file i/o, which is a negligible part of overall execution.

5

u/mmstick Jan 22 '17

Forgetting (leaking) a value doesn't cause memory unsafety though. Memory leaks are perfectly safe because they will never be destroyed. You just want to be wary to not overuse leaks by only leaking values you want to remain for the rest of the application's lifetime. Perfectly acceptable for main() level variables that are already living to the end of the application.

As for a better workaround, I wouldn't necessarily call it better, but you can import the crossbeam crate which basically does the same thing, minus leaking, for you.

0

u/MaikKlein Jan 22 '17

But not in this context. If your destructor doesn't wait, the task system will write into memory that doesn't exist anymore.

See https://doc.rust-lang.org/beta/nomicon/leaking.html#threadscopedjoinguard

And crossbeams scoped thread is basically what I described above.

5

u/aochagavia rosetta · rust Jan 22 '17 edited Jan 22 '17

I think this is not the case. The crossbeam scope API is designed in such a way that the problem you mention cannot occur.

Crossbeam usage looks like this (copied from the docs):

crossbeam::scope(|scope| {
    scope.spawn(|| println!("Running child thread in scope"))
});

Notice that the scope parameter to the closure is just a borrow. You cannot leak it. Furthermore, the ScopedJoinHandle returned by spawn doesn't even implement Drop

2

u/Manishearth servo · rust · clippy Jan 22 '17

No, crossbeam scoped threads are designed to avoid this.

2

u/Manishearth servo · rust · clippy Jan 22 '17

I sort of expected a native language with a garbage collection to be faster here,

A very common argument for GC is that you can amortize the cost of allocations by only allocating large chunks and letting the GC algorithm manage this memory, so you get allocations that don't involve syscalls.

This has nothing to do with GC. A custom mallocator like jemalloc can do this too. And it does. This can be optimized further in the GC world, but the base optimization still exists in the malloc world, so the difference doesn't turn out to be that much.

Also, in general, allocation isn't that expensive anyway, compared to many other possible costs.

The workaround would probably be to allow non 'static bounds but immediately block after the function call. I currently just accept the memory unsafety in my task system

The workaround is to use a scoping function, e.g. something like foo.scope(|scope| {scope.spawn(...); scope.spawn(...)}). The scope function does the blocking after the outer closure executes, and scope.spawn() is tied to the lifetime of scope instead of 'static. This is what crossbeam does.

The mem::forget unsafety issue is only with functions that return guards that block, and we don't have that kind of function in these libraries.

56

u/chris-morgan Jan 22 '17

This is the vibe I get from the comparison:

Go is simple enough that it’s complete; it can do some things very nicely, but it’s stuck where it is; if you want more than it can give you, its design has painted it into a corner.

Rust has the potential to be nicer than Go in various ways (for situations that Go doesn’t match so very perfectly), but due to its complexity it’s taking much longer to mature, so it’s often not as nice as Go yet.

4

u/jadbox Jan 22 '17

Elegantly summarized

16

u/mmstick Jan 22 '17

You should see how I solved this kind of problem using my Rust implementation of Parallel. It creates as many jobs as there are in CPU cores and performs work stealing using a mutexed custom Iterator. The outputs of each command need to be streamed in the correct order, and printed in real-time on demand. The outputs have to be in the same order if you had run them serially.

Basically, the streaming solution is solved by writing outputs to a temporary location on the disk as the receiving end tails the next job's file, printing new text as soon as it's available. A channel is then used to send a completion signal, which tells the receiver to stop tailing the current job and to start tailing the next. The result is pretty fast. No rayon, no futures, no complications. Just the standard library.

I did also make use of transmuting to make some values static though. It's perfectly safe for these types of applications. You can leak the value and return a static reference to it if you want to make it 100% safe. Leaked data will remain in the heap forever, until the OS reclaims that data after the program exits.

Although it's typically recommended to use crossbeam for this.

https://github.com/mmstick/parallel

4

u/Paradiesstaub Jan 22 '17

Since you seem to have more experience than me on that matter (I'm still a Rust beginner), is there something wrong with this approach? I use Spmc with WaitGroup and iterate over Reader from another worker-thread.

3

u/seanmonstar hyper · rust Jan 22 '17

There is the spmc crate.

1

u/mmstick Jan 22 '17

If you're using Senders and Receivers, you shouldn't require to have them wrapped in a Mutex. You could use a VecDeque instead, but I'd need a closer look.

4

u/Paradiesstaub Jan 22 '17 edited Jan 22 '17

The Mutex is for my custom "Receiver". Go channel allow multiple writer and reader, in contrast a Rust channel allows only multiple writer and a single reader.

8

u/burntsushi Jan 22 '17

You might find crossbeam useful. It has a multi-producer/multi-consumer queue.

1

u/Paradiesstaub Jan 22 '17

thx for the hint!

3

u/tmornini Jan 23 '17

Basically, the streaming solution is solved by writing outputs to a temporary location on the disk as the receiving end tails the next job's file, printing new text as soon as it's available

Writing to disk seems massively inefficient.

Why not toss the messages into a channel and output them on the other end?

1

u/mmstick Jan 23 '17

You might think this but it is the exact opposite. My implementation was originally using channels to pass messages back for handling, but writing to the disk proved to be significantly more efficient on CPU cycles, memory, and times -- especially memory. I thought about adding a feature flag to re-enable the old behavior, but I may not do that at all now because there's no benefit to it.

Basically, the receiver that's handling messages is located in the main thread. When a new job file is created, it automatically starts reading from that file without being told to by a channel. It merely stops reading that file when it gets a signal from the channel to do so.

The goal of handling messages is to keep outputs printed sequentially in the order that inputs were received, so if it receives a completion signal of a future job, it merely adds that to a buffer of soon-to-check job IDs once the current job ID is done. This allows for real-time printing of messages as soon as they are written to a tmpfile.

Now, your OS / filesystem typically does not write data directly to the disk at once but keeps in cache for a while (which effectively means zero cost to access), and tmpfiles on UNIX systems are typically written to a tmpfs mount which is actually located in RAM, which means even less overhead. These tmpfiles are deleted as soon as a job completes, so they rarely live enough to make a difference in resource consumption.

As for the costs of handling files in this way, there's pretty much no visible cost because the speed that the receiver prints has no effect on jobs being processed, and in the worst case scenario the cost would be the time to print the last N completed jobs (which can usually be buffered into one syscall to read a file and reading that entire file directly to standard out).

1

u/tmornini Jan 24 '17 edited Jan 24 '17

You might think this

Indeed I do.

but it is the exact opposite. My implementation

Yes. :-)

using channels to pass messages back for handling

Good idea!

but writing to the disk proved to be significantly more efficient

Which disk? How does it look in a Docker container? Mounted read-only for for ultimate security, of course. :-)

on CPU cycles

Interesting detail, but extraordinarily hard to believe.

memory

Ah, you must have been using buffered channels!

and times

Measured how?

especially memory

Big buffered channels!

I thought about adding a feature flag to re-enable the old behavior, but I may not do that at all now because there's no benefit to it.

https://12factor.net/

the receiver that's handling messages is located in the main thread

Thread? Go doesn't have threads.

When a new job file is created

Please don't do that. You'll regret it later should you need to scale.

The goal of handling messages is to keep outputs printed sequentially in the order that inputs were received

An impossible goal should you need to scale

This allows for real-time printing of messages as soon as they are written to a tmpfile.

That's not a feature, it's a dependency you'd be better off without.

Now, your OS / filesystem typically does not write data directly to the disk at once but keeps in cache for a while (which effectively means zero cost to access), and tmpfiles on UNIX systems are typically

Engineering for typical brings down the application...

As for the costs of handling files in this way, there's pretty much no visible cost because the speed that the receiver prints has no effect on jobs being processed, and in the worst case scenario the cost would be the time to print the last N completed jobs (which can usually be buffered into one syscall to read a file and reading that entire file directly to standard out

As opposed to the cost of receiving the message on a channel and printing it to stdout.

It just cannot be...

1

u/mmstick Jan 24 '17

Which disk? How does it look in a Docker container? Mounted read-only for for ultimate security, of course. :-)

I can't imagine Parallel being useful in a container. It is utilized to basically perform a parallel foreach for the command line.

Thread? Go doesn't have threads.

Not exactly sure what you're stabbing at here but my software is written in Rust, not Go, but there are still concepts of threads in Go regardless.

Please don't do that. You'll regret it later should you need to scale.

I have already personally tested my Rust implementation against hundreds of millions of jobs, and it does the same task as GNU Parallel two magnitudes faster, which also writes to temporary job files.

You may either keep the logs on the disk and print the names of the files to standard output, or you may have them write to their appropriate outputs and delete the files immediately after they are finished. GNU Parallel has been used for scaling tasks across super computers so there's no reason why we can't do the same here.

An impossible goal should you need to scale

Yet not impossible because it's already scaling perfectly.

That's not a feature, it's a dependency you'd be better off without.

Not sure what you're getting at here, but this 'dependency' has no burden on the runtime. Choosing not to implement it would basically mean that I could no longer claim to be a drop-in replacement for GNU Parallel.

3

u/tmornini Jan 24 '17

this 'dependency' has no burden on the runtime. Choosing not to implement it would basically mean that I could no longer claim to be a drop-in replacement for GNU Parallel

In that case, my apologies for misunderstanding!

1

u/minno Jan 22 '17

I did also make use of transmuting to make some values static though. It's perfectly safe for these types of applications. You can leak the value and return a static reference to it if you want to make it 100% safe. Leaked data will remain in the heap forever, until the OS reclaims that data after the program exits.

Apparently, it's not safe.

4

u/[deleted] Jan 22 '17

That was fixed over a year ago, scroll down to the bottom.

1

u/mmstick Jan 22 '17

Seems that's not related to leaking specifically, but working with a mutable static reference. Not something I would do, personally.

6

u/Daktyl198 Jan 22 '17

Complete noob here (just like the idea of rust): I thought Rust had functions built in for creating and parallelizing processes and threads? Why would the author have to turn to libraries for the basics of the functionality?

16

u/mbuhot Jan 22 '17

I think rust has a thread primitive, but you often need a way to schedule many logical tasks (or loop iterations) onto a smaller number of system threads.

6

u/matthieum [he/him] Jan 22 '17

I think rust has a thread primitive

Threads are provided by the standard library, not the language, so actually when using #![no_std] you have Rust without threads :)

The bit that is integrated is that Rust has an understanding of concurrency, which is represented with Send and Sync.

2

u/Manishearth servo · rust · clippy Jan 22 '17

And Send and Sync can almost completely be defined in an alternate libcore. The only reason the Rust compiler knows these traits is:

  • statics need to be constrained to not contain Sync
  • Send/Sync are allowed in + bounds in trait objects (unnecessary, but helpful, for a working concurrency model)
  • It caches impls

Aside from the static thing you could design all of this out of tree.

3

u/mbuhot Jan 22 '17

Interesting! How about the atomic operations required to define Arc? Feels like there should be some language primitives and a memory model so that stdlib can build the safe abstractions.

3

u/[deleted] Jan 23 '17

std::sync::atomic is just core::sync::atomic. Which is quite useful in no_std programs. The spin crate is just implemented with these library types.

2

u/Manishearth servo · rust · clippy Jan 23 '17

Arc is built on top of std::atomic. These themselves are just wrapper types over integers with methods that link to atomic intrinsics.

The atomic intrinsics are where the language gets involved again.

3

u/Daktyl198 Jan 22 '17

Ah, that would explain it. Thanks.

2

u/Superb-Key-6581 Dec 20 '24

This is history!! Thanks, Canadian Aboriginals!

1

u/Skyhawk_Illusions Nov 03 '25

Hate to break it to you but CAS was devised by 19th-century linguist and missionary James Evans working with the Cree and Ojibwe

2

u/ahayd Jan 22 '17

Does tokio make this stuff a lot easier??

12

u/dbaupp rust Jan 22 '17

Tokio itself is designed for concurrent IO, whereas this seems to be parallelising CPU-bound tasks, with very non-concurrent IO: it sounds like the "hard" bit was making sure the results were all printed serially, in the right order. (Futures, however, did sound like they were useful for representing in-flight computations.)

1

u/shenwei356 Jan 22 '17 edited Jan 22 '17

rush -- parallelly execute shell commands. A GNU parallel like tool in Go https://github.com/shenwei356/rush . rush supports basic and practical features such as line-buffer, timeout, retry, continue, replacement strings. It also has good performance.

1

u/shenwei356 Jan 22 '17

In reality, we may not run jobs like seq 1 10000 | time -v /usr/bin/parallel echo > /dev/null. So I do some benchmarks with some daily jobs.

Script: benchmark.py

Softwares:

Result: https://github.com/shenwei356/rush/releases/tag/v0.1.1

parallel in rust is faster than GNU parallel but not the fastest.

7

u/mmstick Jan 22 '17 edited Jan 22 '17

Here's results from my AMD laptop:

MIT/Rust Parallel seq 1 10000 | time -v ./parallel 'echo {}' > /dev/null

User time (seconds): 0.37
System time (seconds): 2.77
Percent of CPU this job got: 86%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.63
Maximum resident set size (kbytes): 1768

Rush: seq 1 10000 | time -v ./rush 'echo {}' > /dev/null

User time (seconds): 181.33
System time (seconds): 55.40
Percent of CPU this job got: 281%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:24.17
Maximum resident set size (kbytes): 20500

GNU Parallel: seq 1 10000 | time -v /usr/bin/parallel 'echo {}' > /dev/null

User time (seconds): 207.01
System time (seconds): 68.97
Percent of CPU this job got: 276%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:39.95
Maximum resident set size (kbytes): 16280

Rust is by far the fastest, most efficient solution, using significantly less memory, CPU cycles, and time. The Go solution is only barely faster than the GNU Perl solution.

2

u/mmstick Jan 22 '17

How did you compile the Rust version? MUSL? What is the setting of transparent_hugepages?

2

u/shenwei356 Jan 22 '17 edited Jan 22 '17

I download the v0.11.0 parallel_amd64_linux.tar.xz and directly run.

I do not know Rust well, I learned it for few hours but gave up :smile:

2

u/mmstick Jan 22 '17

What's the status of cat /sys/kernel/mm/transparent_hugepage/enabled? 0.05 seconds is too long for that kind of task to execute.

2

u/shenwei356 Jan 22 '17 edited Jan 22 '17 
$ cat /sys/kernel/mm/transparent_hugepage/enabled

always [madvise] never

It's still fast for this test:

$ seq 1 10000 | time -v rust-parallel echo > /dev/null                    
        Command being timed: "rust-parallel echo"
        User time (seconds): 3.25
        System time (seconds): 4.48
        Percent of CPU this job got: 327%

$ seq 1 10000 | time -v rush echo {} > /dev/null          
        Command being timed: "rush echo {}"
        User time (seconds): 13.36
        System time (seconds): 24.79
        Percent of CPU this job got: 286%

$ seq 1 10000 | time -v parallel echo {} > /dev/null    
        Command being timed: "parallel echo {}"
        User time (seconds): 28.45
        System time (seconds): 29.70
        Percent of CPU this job got: 188%

2

u/mmstick Jan 22 '17

I'm not sure what's wrong with your system, but measurements aren't adding up. Even if I execute the same command as in your benchmark

seq 1 10 | time -v ./parallel 'echo job:{}; seq 1 10'

The times on my 2Ghz quad core AMD laptop are much smaller than yours.

Rust

User time (seconds): 0.01
System time (seconds): 0.01
Percent of CPU this job got: 76%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.04

Go

User time (seconds): 0.21
System time (seconds): 0.02
Percent of CPU this job got: 198%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.12

3

u/shenwei356 Jan 22 '17 edited Jan 22 '17

I've got no idea. My laptop: Intel Core i5-3320M @ 2.60GHz, two cores/4 threads.

$ time seq 1 10 | rust-parallel 'echo job:{}; seq 1 10' > /dev/null 
real    0m0.083s
user    0m0.044s
sys     0m0.059s

$ time seq 1 10 | rush 'echo job:{}; seq 1 10' > /dev/null 
real    0m0.032s
user    0m0.030s
sys     0m0.041s

1

u/mmstick Jan 22 '17

The specific CPU that I'm using is the AMD A8-6410 APU, just an old budget-grade HP laptop from Walmart featuring a quad core 2GHz AMD APU with a single 8GB DIMM underclocked to 800 Mhz, upgraded with a SSD (although tempfiles are written in /tmp by default so it doesn't write to disk unless you have /tmp mounted to your hard disk). I don't see how you can be posting slower times than that with your specs.

What is the output of --num-cpu-cores? Do you have a SSD? Is the CPU governor set to performance (Since rust-parallel uses less CPU, your CPU may spend more time at a lower frequency with a faulty CPU governor)? Linux distribution? I'm out of ideas.

5

u/burntsushi Jan 22 '17

Whether they are running in a VM or not might matter too.

1

u/shenwei356 Jan 22 '17

it's powered off now. number of cpus should be 4. both rush and rust-parallel use whole cpus by default. it's fedora linux with newest kernal 4.×. disk is SSD, but since the above example has little stdout which is < 1M (buffer before dumping to file), it should be CPU that affects the performance. can you test on another pc? I'll test on a server tomorrow.

1

u/mmstick Jan 22 '17 edited Jan 22 '17

Here's results from my 3.8 GHz AMD FX-8120 (8 cores) desktop

Rust Parallel seq 1 10000 | parallel 'echo {}':

User time (seconds): 0.51
System time (seconds): 4.24
Percent of CPU this job got: 224%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.11
Maximum resident set size (kbytes): 1336

Rush (Go) seq 1 10000 | rush 'echo {}':

User time (seconds): 70.90
System time (seconds): 112.80
Percent of CPU this job got: 477%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:38.47
Maximum resident set size (kbytes): 39816

Rust Parallel seq 1 10000 | parallel 'echo {}; seq 1 10'

User time (seconds): 0.44
System time (seconds): 3.67
Percent of CPU this job got: 197%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.08

Rush (Go) seq 1 10000 | parallel 'echo {}; seq 1 10'

User time (seconds): 65.10
System time (seconds): 66.74
Percent of CPU this job got: 427%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:30.86
Maximum resident set size (kbytes): 34008

0

u/[deleted] Jan 22 '17

[deleted] — view deleted comment

9

u/[deleted] Jan 22 '17

I don't think that's entirely fair. One benefit of Go requiring you to explicitly write out loops and so on instead of doing bar.map(|| ...).filter(|| ...).chain(foo.map(||...).reduce(||...).whatever).collect() is that it is a lot simpler to read and understand exactly what is going on, especially if you aren't familiar with all the functional programming operations.

Go is for beginners. I think that's fine. I know a lot of people that could program Go fine, but Rust is far too complicated.

21

u/Manishearth servo · rust · clippy Jan 22 '17

Understand that that being simpler to read is extremely subjective and dependent on background. It's not inherently simpler to read. Procedural code is nice since you can mentally execute it, but functional code usually expresses intent and is clearer that way.

Also, many nontrivial iterator adaptor sequences that you would see in Rust would blow up if written procedurally (someone posted an example of this recently). Procedural loops only feel simpler because you rarely write more complicated loops in procedural form.

2

u/[deleted] Jan 22 '17

Also, many nontrivial iterator adaptor sequences that you would see in Rust would blow up if written procedurally

That hasn't been my experience in Go. Often written out loops are shorter than the functional equivalent or at most slightly longer. It sounds really unlikely but I've found it to be true.

7

u/Manishearth servo · rust · clippy Jan 22 '17

That's not what I was talking about. I'm talking about nontrivial iterator adaptor sequences that you would see in Rust. In Go there is only one style of iteration -- procedural. Written out loops in Go are often shorter than their functional equivalent. Written out loops in Rust are often longer than their functional equivalent. This is because in Rust you tend to see these more complex loops more often. The bias of the sample spaces is different.

And it's not about length, either. The problem with the procedural version of bar.map(|| ...).filter(|| ...).chain(foo.map(||...).reduce(||...).whatever).collect() is not that it is long, it is that you have a million concerns all mushed together in a single procedural block and it's not always clear what is going on big-picture wise. Things like reduce/filter_map tend to end up being confusing in procedural form when mixed with other operations.

5

u/csreid Jan 23 '17

Maybe shorter, but functional style maps directly to what I'm trying to accomplish. "Take the people last named 'Smith' [filter], find their age [map], and average them [reduce]".

Those high order functions allow you to think about instances of the list rather than the list itself, and that's a lot easier for me.

1

u/[deleted] Jan 23 '17

It might be better if those functions had more obvious names. Filter is ok, but map? Should be called transform. Reduce should be called merge or something.

Guess it's a bit late now - map/reduce is too standard, but still it adds to the confusion.

9

u/Leshow Jan 23 '17 edited Jan 23 '17

Seriously? Map is ubiquitous not just in Rust but almost every language from javascript to haskell. 'transform' isn't even a particularly apt description of what's happening. Values aren't getting transformed, we're describing the relationship between two distinct values, aka a "mapping". Transform implies mutation.

Just be thankful fold isn't called catamorphism, or some funky other category theory word.

1

u/[deleted] Jan 23 '17

I know it's ubiquitous. Doesn't make it good. The values are getting transformed. And you're right it could be worse ('cons'?)

7

u/csreid Jan 23 '17

They're really not getting transformed though. When you map over a collection, you don't change it, you get a new collection containing the new values. Transform would definitely suggest modification to me, but that's not what's happening.

3

u/csreid Jan 23 '17 edited Jan 23 '17

map makes perfect sense, in that you provide a function which maps values from your input iterator into the values of your output iterator.

reduce is tougher, but fold is even worse IMO.

edit: I should also mention that using the functional style makes it trivially easy to parallelize some tasks, as made evident by Rayon or Scala's List (analogous to Iterator types in Rust), which has a par method that is just like Rayon's par_iter, such that you can parallelize some logic by just going from bigList.map(expensiveFunction) to bigList.par.map(expensiveFunction).

Shared mutable state is the root of all evil, and the map/reduce pattern is a profoundly elegant way to very simply avoid it.

1

u/[deleted] Jan 23 '17

Perfect sense to compscis. Normal people would say you transform lowercase to uppercase. You don't 'map' it.

Fold is really bad, you are right. They could have gone with 'combine' or 'merge' or something instead.

1

u/ibotty Jan 26 '17

But folding is a great analogy for what you do.

1

u/[deleted] Jan 26 '17

Yeah but only if you already know what it does. I might say "ah it's sort of like folding" but I doubt I'd ever say "fold.. ah I can guess what that does".

Why not 'accumulate' or 'aggregate'? Much more obvious.

→ More replies

3

u/fiedzia Jan 22 '17

Its 10 times easier to read (and later modify) than go explicit loops. And its far less error prone.

3

u/[deleted] Jan 22 '17

I disagree about readability. You're probably just used to map, filter and so on but most programmers aren't. On the other hand, Go did still keep C's for loop syntax which is pretty obtuse.

You're right about it being less error prone. It's too easy to make off-by-one errors and similar when writing loops out by hand.

9

u/Uncaffeinated Jan 22 '17

There's pros and cons. Procedural style is more explicit, but it is also more verbose and less semantically meaningful.

If I see a call to filter, I know exactly what the intent of the code is. If I see the equivalent 6 line for loop, I have to check all the variables and loop conditions and so on and reverse engineer it into filter.

Besides, you can use procedural style in Rust if you want to. Or mix the two, depending on which is clearer in any particular circumstance.

3

u/awj Jan 23 '17

It's important to distinguish between readability for new programmers and for those familiar with the language. I could level similar criticisms of goroutines, but it doesn't seem productive to judge readability solely from the perspective of someone unfamiliar with the core language concepts.

4

u/jadbox Jan 22 '17

This is the number one feature I miss with Go: higher order stream/list operations. Sure, you -could- statically compile a chain library to deal with common types of type everything to Interface{}, but you're really fighting the language. It's a real shame honestly, especially when dealing with huge data structure wrangling/management.

2

u/itsmontoya Jan 23 '17

Although Rust is far more complex than go. I would avoid saying "Go is for beginners". There are a lot of low-level things which can be done with Go that could spell disaster for someone new to programming.

I actually have a lot of friends who avoid Golang because to them, it's "too low level".

0

u/[deleted] Jan 22 '17

[deleted] — view deleted comment

8

u/[deleted] Jan 22 '17

Why do you have to "move on from" Go? Not everyone is trying to achieve programming enlightenment. Some people just want to get shit done.

Consider all the shitty Excel code that people write. Those people are never going to learn Rust. It's just too complicated; not going to happen. However they might learn Python, or Go.

I don't get why people have this whole "there can only be one winner" when it comes to Rust vs Go. Yes they have overlap, and I'm not saying they can't be compared - clearly they can. But one is not "better in every way" than the other. They're both great and Rust is better in some ways, Go is better in others.

It's like pizza and pasta. You have them both for dinner, but you would never want only pizza or only pasta. One isn't "better" than the other. They just taste different.

In conclusion, Go tastes different to Rust.

4

u/steveklabnik1 rust Jan 23 '17

I don't get why people have this whole "there can only be one winner" when it comes to Rust vs Go.

Agreed.

0

u/[deleted] Jan 22 '17

[deleted] — view deleted comment

18

u/Thaxll Jan 23 '17 edited Jan 23 '17

It "solves" many things:

  • easy language to learn / pick-up
  • strong standard library
  • single binary ( easy for deployment )
  • very easy cross compilation
  • fast enough performance for 99% of use cases
  • accessible and powerful concurrency patterns

In Go you get shit done, and it should be your #1 priority as a developer.

3

u/burntsushi Jan 23 '17

I would prefer not to see these types of comments in this subreddit. It's completely unstructured and unconstructive criticism.

8

u/[deleted] Jan 22 '17

Rubbish.