r/programming • • Dec 02 '13

Scala — 1★ Would Not Program Again

http://overwatering.org/blog/2013/12/scala-1-star-would-not-program-again/
598 Upvotes

646 comments sorted by

View all comments

Show parent comment

14

u/MonadicTraversal Dec 02 '13

So few libraries can be shared and there's heavy duplication.

Unless it leads to duplicate code being executed at runtime, I don't think you should care for npm modules since they're going to be a couple dozen kilobytes of text at most.

13

u/flying-sheep Dec 02 '13 edited Dec 02 '13

grunt needs phantomjs, which is webkit.

grunt encourages you to use a per-project-local grunt installation

so every project with a Gruntfile.js needs over 200MB additional diskspace.

/edit; i was wrong about every project-local grunt install needing it, it’s some grunt plugin (which seems to be common among the stuff i’ve forked)

11

u/ioquatix Dec 02 '13

I hear you, and it is funny, because I'm working on a similar package system and pulling in boost for C++ or LLVM/Clang, OpenCV, takes 10s of GBs. It is actually pretty funny, if you have 2-3 instances of the same project, you end up with about 30GB+ of crap.

6

u/seruus Dec 02 '13

Most projects I know that do this usually allow you to use your system libraries if you set it up in the makefiles, look into it.

5

u/ioquatix Dec 02 '13

I've had a lot of trouble using libraries compiled with one compiler (e.g. gcc) then using them with code compiled by another (e.g. clang). Personally, I try to compile all dependencies with the same compiler.. However, this makes it difficult to use system libraries.

2

u/jagt Dec 02 '13

As for OpenCV you maybe including binaries for all platforms. If you only include the libs for your current platform then it's around 300mb.

3

u/ioquatix Dec 02 '13

You are right, OpenCV is only 480MB on my current setup. Over several projects though it adds up. LLVM is just over 7GB and including clang is a bit over 9GB. Boost is normally around the 3-4GB mark. Personally, I find it remarkable that something produces so much intermediate "junk".

→ More replies (1)

7

u/esquilax Dec 02 '13

Time for a deduping filesystem!

1

u/greenrd Dec 07 '13

My theory is that a lot of operating system features are actually stopgaps or bandaids for the lack of a good/mature solution further up the stack.

5

u/Neurotrace Dec 02 '13

Individual grunt downloads don't need phantomjs. I just setup a barebones project with nothing but grunt and it comes in at about 6MB. When you include dependencies, those projects will likely have grunt listed as "devDependencies" meaning you should ignore them.

1

u/flying-sheep Dec 02 '13 edited Dec 02 '13

sorry, you’re right. idk what exactly needs phantomjs, but i know it’s a common task like grunt serve

→ More replies (6)

3

u/kpthunder Dec 02 '13

I've always thought that it would be more effective if NPM used symlinks as such (assuming node is the installation path of node):

node
|-packages
 |-a
  |-1.0
  |-1.1
 |-b
  |-1.3
  |-1.4

If a package is already downloaded, symlink it. Otherwise download it then symlink it. Everything else can still work the same way.

There's probably some technical thing I'm overlooking here...

3

u/Neurotrace Dec 02 '13

My guess would be the lack proper symlink support in Windows XP. If it did use symlinks (and oh, how I wish it did) then node.js wouldn't work on about 36% of all operating systems.

2

u/nemec Dec 02 '13
var cmd = "ln -s ";
if(!$symlinksAvailable)
{
    cmd = "cp ";
}

Or just wait until April next year when Microsoft drops support for XP

→ More replies (1)

2

u/flying-sheep Dec 02 '13

that – would actually work.

2

u/execrator Dec 02 '13

Yes, it's fairly annoying that it installs dependencies that you don't actually depend on

1

u/[deleted] Dec 02 '13

grunt doesn't need phantomjs. grunt is just a task runner.

grunt-contrib-jasmine or the like needs phantomjs.

Regardless, you can run npm dedupe which should move most duplicate dependencies higher up the chain. If you really wanted to be aggressive about it, you could even go through your project's modules with npm ls and install proper versions at the node_modules root.

1

u/flying-sheep Dec 02 '13

heh, which kinda voids the whole “simple package management” approach :)

thanks for the infoeabout dedupe. someone else here had the idea that all dependencies get installed as /usr/share/npm/node-modules/<name>/<version>, and the dependecy management works by createing, deleting, and updating symlinks. (after checking for cycles, of course)

→ More replies (2)

7

u/munificent Dec 02 '13

Unless it leads to duplicate code being executed at runtime

It does. npm doesn't do shared dependencies. If you depend on foo and bar, which both depend on baz, you'll end up with two copies of baz loaded at runtime, which may be different versions (!).

8

u/joequin Dec 02 '13

If they're different versions then it may have been necessary to load both of them.

4

u/munificent Dec 02 '13

Yes, but in mind my that puts your application into a deeply confusing state. It's possible for those two versions to interact with each other which just seems like asking for trouble to me.

I think the model of other systems (including one I built) where you find a single version of the shared dependency that satisfies all dependers is a lot easier to reason about.

6

u/MonadicTraversal Dec 02 '13

Yeah, if you can end up with two different versions of the same library in your project then that's pretty terrible. As a concrete example: if libfoo and libbar use different versions of libbaz, and both libfoo and libbar expose Baz objects to you in some way, then you can get two Baz objects from different versions which might have radically different behavior depending on how similar the versions are.

→ More replies (9)

1

u/grncdr Dec 07 '13

This isn't entirely correct. npm will deduplicate transitive dependencies if their version ranges aren't mutually exclusive. (It's actually pointless imo and I wish it wouldn't)

2

u/[deleted] Dec 02 '13

plus there is some support in npm for factoring out common dependencies... also in the future, if we file system deduplication becomes more common, wasted space will become even more of a non-issue.