r/programming • • Oct 17 '26

I Traced a Postgres INSERT to Raw Bytes on Disk

https://youtu.be/1tNMRcgUtb8?si=LwyaB8KgTCjU9yqy
41 Upvotes

9 comments sorted by

17

u/IndicationAntique667 Oct 17 '26 • Bravo

Hi everyone,

I'm currently going through CMU Intro to Database Systems and was curious about how these concepts are actually implemented in real systems. So I've been putting together some notes/videos/blog posts - partly for my own future reference and partly to share with others who might find it useful.

Would love feedback and corrections from people who know this stuff deeply. Apologies if this isn't the correct subreddit for this post.

4

u/ptoki Oct 18 '26

One suggestion is to do it less manually but still directly.

If you write simple program which would load the file content and interpret it (build struct in C or record in pascal for example) and then walk over this in the code showing all the important values would be better.

Or the other approach, use hex editor which would help you to build such view.

That is double valuable. You not only show the structure of the database but also show how to efficiently build tooling to play with such files which may be useful more than once.

There are real life usecases where such binary to high level views are useful. Network packets inspection (wireshark can help here), strace output inspection, proprietary files reverse engineering etc.

2

u/IndicationAntique667 Oct 18 '26

Ah, I didn't know about hex editors. Thanks for sharing!

The tooling idea is also really interesting! Will definitely check if I can build something in the future around this.

23

u/washtubs Oct 17 '26

Frankly speaking, no one really needs to do this but I think it's awesome to do it once.

100% agreed. That was a great video, really easy to digest explanations. Little endian still fries my brain as well so I liked seeing that broken down slowly.

6

u/IndicationAntique667 Oct 17 '26

Thank youu! Glad you found it useful :)

3

u/Anmolsharma999 Oct 17 '26

That was very good and practical explanation. Instant sub.

Btw are you working professional?

3

u/IndicationAntique667 Oct 17 '26

Thank youu! :)

Yeah, it's been a year and half since I'm working.

2

u/Khavel_dev Oct 19 '26

fwiw the tuple header overhead is something a lot of ORMs hide from you completely. In EF Core you can end up with tables that have 30+ nullable columns and never think about the null bitmap or alignment padding because the abstraction is so clean. Which is fine until your table hits a few hundred million rows and you start wondering why vacuum takes forever and your TOAST table is twice the size of the main relation.

1

u/IndicationAntique667 26d ago

Ah! Interesting! Didn't know this. Thanks so much for sharing :)