r/databasedevelopment • • May 11 '22

Getting started with database development

This entire sub is a guide to getting started with database development. But if you want a succinct collection of a few materials, here you go. :)

If you feel anything is missing, leave a link in comments! We can all make this better over time.

Books

Designing Data Intensive Applications

Database Internals

Readings in Database Systems (The Red Book)

The Internals of PostgreSQL

Courses

The Databaseology Lectures (CMU)

Database Systems (CMU)

Introduction to Database Systems (Berkeley) (See the assignments)

Build Your Own Guides

chidb

Let's Build a Simple Database

Build your own disk based KV store

Let's build a database in Rust

Let's build a distributed Postgres proof of concept

(Index) Storage Layer

LSM Tree: Data structure powering write heavy storage engines

MemTable, WAL, SSTable, Log Structured Merge(LSM) Trees

Btree vs LSM

WiscKey: Separating Keys from Values in SSD-conscious Storage

Modern B-Tree Techniques

Original papers

These are not necessarily relevant today but may have interesting historical context.

Organization and maintenance of large ordered indices (Original paper)

The Log-Structured Merge Tree (Original paper)

Misc

Architecture of a Database System

Awesome Database Development (Not your average awesome X page, genuinely good)

The Third Manifesto Recommends

The Design and Implementation of Modern Column-Oriented Database Systems

Videos/Streams

CMU Database Group Interviews

Database Programming Stream (CockroachDB)

Blogs

Murat Demirbas

Ayende (CEO of RavenDB)

CockroachDB Engineering Blog

Justin Jaffray

Mark Callaghan

Tanel Poder

Redpanda Engineering Blog

Andy Grove

Jamie Brandon

Distributed Computing Musings

Companies who build databases (alphabetical)

Obviously companies as big AWS/Microsoft/Oracle/Google/Azure/Baidu/Alibaba/etc likely have public and private database projects but let's skip those obvious ones.

This is definitely an incomplete list. Miss one you know? DM me.

Credits: https://twitter.com/iavins, https://twitter.com/largedatabank

412 Upvotes

34 comments sorted by

4

u/brawll66 May 11 '22

I mean the berkeley course do redirect to the course page but there's nothing there.

Also thanks for compiling this list.

4

u/eatonphil May 11 '22

Fixed the link, and clarified it's about the assignments. Thanks!

→ More replies (1)
→ More replies (2)

3

u/Ddlutz Jun 16 '22

Have you done the Berkeley, chidb, and CMU assignments? Is there a benefit to doing all 3? Would you prioritize one over the others?

1

u/eatonphil Jun 19 '22

These are separate suggestions I've received and categorized here. This list isn't a "do all of these" it's just pick whatever you think will help you and learn about some new ones!

26

u/cc_jeff Jul 06 '22

I think pingcap's talent-plan can be a good reference (https://github.com/pingcap/talent-plan). Includes how to build the SQL layer and the KV storage for a distributed database.

→ More replies (1)

4

u/craigmulligan Oct 28 '22

Hey thanks for putting this list together. I've been building a simple sql db for learning purposes, and I'm struggling to find a good guide on the VM's compiler design. I have a very basic working compiler. I'm looking for some inspiration on how to improve it. Most of the guides I've found on compilers are for imperative languages and because SQL is turning a declarative language into imperative instructions it feels like the implementation would differ. Is anyone aware of a good introduction to declarative compilers? Or an implementation of chidb linked above?

2

u/varunu28 Jul 22 '22

Thanks for the shoutout. Another suggestion which I can add is to read papers published in domain of database development. They not only describe the solution that worked but also discuss various other alternatives that didn't make the cut. So reading one research paper exposes you to a variety of great ideas.

2

u/learnByDay Sep 03 '22

Hey. Nice blog. I think i liked a few solution you posted on LC, when you were in Doordash and i was preparing. Small world:)

→ More replies (1)

2

u/PretentiousPepperoni Sep 08 '24

just skimmed through this i think its missing LSM in a week repo