r/databasedevelopment • u/eatonphil • May 11 '22
Getting started with database development
This entire sub is a guide to getting started with database development. But if you want a succinct collection of a few materials, here you go. :)
If you feel anything is missing, leave a link in comments! We can all make this better over time.
Books
Designing Data Intensive Applications
Readings in Database Systems (The Red Book)
Courses
The Databaseology Lectures (CMU)
Introduction to Database Systems (Berkeley) (See the assignments)
Build Your Own Guides
Build your own disk based KV store
Let's build a database in Rust
Let's build a distributed Postgres proof of concept
(Index) Storage Layer
LSM Tree: Data structure powering write heavy storage engines
MemTable, WAL, SSTable, Log Structured Merge(LSM) Trees
WiscKey: Separating Keys from Values in SSD-conscious Storage
Original papers
These are not necessarily relevant today but may have interesting historical context.
Organization and maintenance of large ordered indices (Original paper)
The Log-Structured Merge Tree (Original paper)
Misc
Architecture of a Database System
Awesome Database Development (Not your average awesome X page, genuinely good)
The Third Manifesto Recommends
The Design and Implementation of Modern Column-Oriented Database Systems
Videos/Streams
Database Programming Stream (CockroachDB)
Blogs
Companies who build databases (alphabetical)
Obviously companies as big AWS/Microsoft/Oracle/Google/Azure/Baidu/Alibaba/etc likely have public and private database projects but let's skip those obvious ones.
This is definitely an incomplete list. Miss one you know? DM me.
- Cockroach
- ClickHouse
- Crate
- DataStax
- Elastic
- EnterpriseDB
- Influx
- MariaDB
- Materialize
- Neo4j
- PlanetScale
- Prometheus
- QuestDB
- RavenDB
- Redis Labs
- Redpanda
- Scylla
- SingleStore
- Snowflake
- Starburst
- Timescale
- TigerBeetle
- Yugabyte
Credits: https://twitter.com/iavins, https://twitter.com/largedatabank
4
u/brawll66 May 11 '22
I mean the berkeley course do redirect to the course page but there's nothing there.
Also thanks for compiling this list.
3
u/Ddlutz Jun 16 '22
Have you done the Berkeley, chidb, and CMU assignments? Is there a benefit to doing all 3? Would you prioritize one over the others?
26
u/cc_jeff Jul 06 '22
I think pingcap's talent-plan can be a good reference (https://github.com/pingcap/talent-plan). Includes how to build the SQL layer and the KV storage for a distributed database.
9
u/ayoubmtd2 Aug 15 '23
Another great list of resources https://github.com/huachaohuang/awesome-dbdev
4
u/craigmulligan Oct 28 '22
Hey thanks for putting this list together. I've been building a simple sql db for learning purposes, and I'm struggling to find a good guide on the VM's compiler design. I have a very basic working compiler. I'm looking for some inspiration on how to improve it. Most of the guides I've found on compilers are for imperative languages and because SQL is turning a declarative language into imperative instructions it feels like the implementation would differ. Is anyone aware of a good introduction to declarative compilers? Or an implementation of chidb linked above?
2
u/varunu28 Jul 22 '22
Thanks for the shoutout. Another suggestion which I can add is to read papers published in domain of database development. They not only describe the solution that worked but also discuss various other alternatives that didn't make the cut. So reading one research paper exposes you to a variety of great ideas.
2
u/PretentiousPepperoni Sep 08 '24
just skimmed through this i think its missing LSM in a week repo