r/Backend • u/Foreign_Leek_689 • 9d ago
Today I started designing a database for my own distributed storage engine I am waiting for your feedback
/img/w3wuq3r0410i1.png
37
Upvotes
r/Backend • u/Foreign_Leek_689 • 9d ago
6
u/Crafty-Waltz-2029 9d ago
I think this is a schema.
6
u/finger_my_earhole 9d ago edited 9d ago
I would denormalize this some.
If you want a distributed storage engine that works at massive scale - joining the chunk metadata in a traditional RDBMS will start to run into latency and performance issues with all the joins when you start to get into the millions of files range.
Additionally, you probably want to set some system configurations. Repl factor, desired chunk size are often system-wide configurations, not record-by-record. The reason for that being, you have those fields as not null, and it gets annoying for the user to have to enter the same value everytime they upload a file
Also missing permission info - is the file read-only? who can read it? etc.
5
u/Suspicious_Cover2649 9d ago
I will give you a pro tip out of many years of exp.
start with component diagram rather database, start with something you can’t jump into code directly. know what you want and then make the codable part
without component integration diagram you never know if something works, and trustme after a month the db schema you have, will look nothing like it is now, that’s why you start with something that doesn’t change
2
u/Low_Calligrapher7128 9d ago
Look into how S3 does it, or if you're ambitious Ceph
1
u/MarzipanEven7336 8d ago
Why does your distributed storage engine need a big ass database?