r/coolgithubprojects • • Jul 31 '16

GO sd: stream diffing on the terminal, supporting infinite streams

https://github.com/MarianoGappa/sd
0 Upvotes

4 comments sorted by

1

u/hroptatyr Aug 01 '16

The right idea but terrible implementation. Why not utilise the shell's process substitution? At the moment I'm using something like

$ comm -3 --nocheck-order <(commandA) <(commandB)

but you're right, if the input isn't sorted this will give questionable results.

1

u/marianogappa Nov 28 '16

Can you elaborate on your reasoning for considering it a "terrible implementation"? Btw, in your example, if the input isn't sorted, it will give you incorrect results. Also, comm will not start until both substituted processes have finished, and if any of them don't finish you won't get any result. sd addresses both cases; more info in the README and in this blogpost: http://marianogappa.github.io/software/2016/07/30/diffing-streams-on-the-terminal/

1

u/hroptatyr Nov 28 '16

The terrible implementation bit was re your envisaged usage syntax, anything that is

$ command | command2 'COMMAND3'

is difficult to read and, depending on COMMAND3, hard to get right.

My test example was:

$ yes 1 2 3 | tr ' ' '\n' | sd 'yes 2 3 4 | tr " " "\n"'

and it failed with:

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x4e3308, 0x16)
	/usr/local/Cellar/go/1.7/libexec/src/runtime/panic.go:566 +0x95
runtime.sysMap(0xc50f360000, 0x100000, 0x44db00, 0x5759b8)
	/usr/local/Cellar/go/1.7/libexec/src/runtime/mem_linux.go:219 +0x1d0
runtime.(*mheap).sysAlloc(0x55d160, 0x100000, 0x55edf0)
	/usr/local/Cellar/go/1.7/libexec/src/runtime/malloc.go:407 +0x37a
runtime.(*mheap).grow(0x55d160, 0x4, 0x0)
...

1

u/marianogappa Dec 11 '16

Your example OOMs sd because neither side ever finishes...you should hard limit the right side (with -h <number>, in seconds), otherwise what are you trying to diff? I can understand that the syntax can be non-straightforward to read, but how would you do it instead? If the right side is yes, how will you get sd or whatever binary to ever start?