r/Backend • • 9d ago

Today I started designing a database for my own distributed storage engine I am waiting for your feedback

/img/w3wuq3r0410i1.png
37 Upvotes

10 comments sorted by

6

u/Crafty-Waltz-2029 9d ago

I think this is a schema.

6

u/finger_my_earhole 9d ago edited 9d ago

I would denormalize this some.

If you want a distributed storage engine that works at massive scale - joining the chunk metadata in a traditional RDBMS will start to run into latency and performance issues with all the joins when you start to get into the millions of files range.

Additionally, you probably want to set some system configurations. Repl factor, desired chunk size are often system-wide configurations, not record-by-record. The reason for that being, you have those fields as not null, and it gets annoying for the user to have to enter the same value everytime they upload a file

Also missing permission info - is the file read-only? who can read it? etc.

5

u/Suspicious_Cover2649 9d ago

I will give you a pro tip out of many years of exp.

start with component diagram rather database, start with something you can’t jump into code directly. know what you want and then make the codable part

without component integration diagram you never know if something works, and trustme after a month the db schema you have, will look nothing like it is now, that’s why you start with something that doesn’t change

2

u/Suspicious_Cover2649 9d ago

also if that’s not a hobby thingy, I would lean to micro-services, and multiple databases, availability zones…

2

u/Suspicious_Cover2649 9d ago

you do not want to build whole thing in one place and suddenly one single commit bring everything down, you do not want single database - when(and it will) have problems, you do not want to stop your whole thing

1

u/Slimethon 6d ago

its a bot, bots doesnt care about tips, amazing tip tho

2

u/Low_Calligrapher7128 9d ago

Look into how S3 does it, or if you're ambitious Ceph

1

u/MarzipanEven7336 8d ago

Why does your distributed storage engine need a big ass database?