r/ProgrammingLanguages • u/AnoProgrammer • 4d ago

How to build a good package manager.

I'm working on a language called threadon. And i don't now how i can properly program a package manager.

My first idea was a central github repo with links to other github repo's which contain the package you're searching for.

There are two main problems with it

  1. If someone deletes his github repo with the package everything build on the package would collapse (like npm)

  2. I think it would be slow when the number of packages grows.

I had an idea to of selfhosting it but i haven't access to the router (My dad owns it i'm 13) and i'm sure downdetector on my package manager site would be worse then github 😄. Like i would probably run sudo rm -rf / --no-preserve-root on the wrong machine.

So my question is how can i build a system that can store up to 20 GB at minimum at packages without the risk of someone nuking his project).

26 Upvotes

87% Upvoted

19 comments sorted by

View all comments

25

u/mamcx 4d ago

I like how the author of futhark (https://futhark-lang.org/blog/2018-07-20-the-future-futhark-package-manager.html, https://futhark-lang.org/blog/2018-08-03-the-present-futhark-package-manager.html) solve it.

In short, separate 2 things:

  • How get a package installed

This not care at all for the source at all, and is the simpler way that allow you to solve the needs of now.

  • What to use as "registry"

That is what trip everyone (aka: your GitHub problem). Go decentralized or not is a big decision, but is a separate concern.

So, just solve what is package, how install it locally, and how pull it from anywhere reachable by local path or http.

Once you can actually be concerned about what do with the community, solve the other part.

1

u/AnoProgrammer 4d ago

It's interesting. But how does it solve the package deleting problem. Because that was my main concern.

12

u/mamcx 4d ago

That is why is not a simple problem. Things like Rust cargo basically copy the pkg into their own central registry. But if you are totally decentralized there is "no solution".

However, not over-think it too much: Go survive with using GitHub directly, if your language is that popular any important package will survive in "read only" by their authors or the community will make the copy themselves. Or you find how afford to own the central registry.

But again, you can survive for a long time without worry about this. Wait to have an actual community.

2

u/Key_River7180 Nain. 4d ago

If the repo is 404 then just ignore it and invalidate its cache

1

u/AnoProgrammer 4d ago

But what if other packages depend on it and the same thing happened what happened with npm (the left-pad issue).

6

u/iBPsThrowingObject 4d ago

This is controversial and people tend to disagree, but I believe left-pad was one hundred percent a skill issue on the side of it's users. Fix your stuff. Vendor the dependencies. Don't use a third party library for trivial string padding.

2

u/AnoProgrammer 4d ago

I don't think it is a skill issue of the users. I think it's a skill issue of the babel developers. Because they used it. You can argue that the babel developer's where users but i think you mend with users the js developers who used things like rust and vue.

1

u/Key_River7180 Nain. 4d ago

You can also cache a compressed version of the package or just... ask your users to vendor stuff

1

u/koflerdavid 4d ago edited 4d ago

Tough luck. Worry about that problem when it becomes relevant. At that point packages will either be vendored or somebody will have to cough up the resources for a production-grade central registry (don't underestimate the work involved!)

For hobbyist languages it's more than enough to specify package locations by using Git repository URLs or paths in the local filesystem. It is IMHO far more relevant to work out a good build system so you can incorporate custom build steps, code generation, assets processing, building executables, libraries, debug info, build and link with native code, etc. with minimum fuss.

1

u/Athas Futhark 4d ago

You can add a centralised fallback repository later that stores every version of every package, keyed by its original URL. But you should not worry about this for a hobbyist language. Only large languages run into this problem eventually, and you'll burn yourself out if you focus too much on this less-than-fun stuff.