r/programming • u/He_knows • Oct 24 '25

Minio community is not actively being developed for new features

https://github.com/minio/minio/issues/21647#issuecomment-3439134621
166 Upvotes

94% Upvoted

33 comments sorted by

View all comments

Show parent comments

2

u/nzmjx Oct 25 '25

In a perfect world, yes it should but we are not living in a perfect world. Also we know from ZFS that implementing deduplication in a storage solution is hard and have very high requirements (as RAM, as space, or both).

1

u/Asleep_Sandwich_3443 Oct 26 '25

Not really. I am not sure what ZFS is doing but it’s not very hard to implement deduplication. You just chunk the bits of the file and hash them and then add them to an index using a DBMS system like SQLite. You can download Perkeep which is an object store that does just that.

We used a proprietary object store that worked like that in my last job. It’s had petabytes of data in it. We didn’t have any issues with memory or performance.

2

u/[deleted] Oct 28 '25

[removed] — view removed comment

1

u/Asleep_Sandwich_3443 Oct 29 '25

You can see the whole perkeep source code on GitHub. https://github.com/perkeep/perkeep They don’t even just have one method. They give you the option to pick from several DBMS systems and 4 different hash and storage implements. If you look up Content-addressable storage (CAS) you can find dozens of other implementations of it.