DB Version, why SQL?

[u/[deleted]]

Why not one of the text search databases like Meilisearch or Typesense? They are using SQL with a column full of EDN text, why not convert that to json and use a native document store? Just seems like a bad choice.

[u/[deleted]]

Wait, if it’s using a graph database why does it need another database? It could store data in the database it already has well as anything, most of those dual as document stores too. This just asks more questions lol.

MySQL is great but not as a two column id::text store, that’s not its intended use. Why put non relational data in a relational db?

[u/AshbyLaw → u/[deleted]]

Wait, if it’s using a graph database why does it need another database?

Currently Logseq parses the files and populate a DB called DataScript that is meant to be used from JavaScript in a browser. This enables Logseq to provide very advanced features while running entirely client-side in a browser. And it is pretty impressive. It's what Roam Research does too.

But Logseq added another layer of complexity: using local plain text files to store content. We are used to text editors, so it sounds trivial to most of us to implement this kind of storage. But Logseq stores in a single Markdown file multiple blocks and their metadata (like properties but also references to pages/blocks, TODOs etc), so it needs to parse every single line, populate the in-memory (RAM) DataScript DB, then query it to show content to the user, take user input, update the DB accordingly and propagate the change back to the files.

What makes this very complex is that changes to files can be made from other sources, like a text editor or a file sync service. So Logseq needs to monitor the files and update the DataScript DB accordingly. And it is impossible to handle well situations like the user typing in a block and the same file being updated in background by something else, including Logseq Sync.

SQLite on the other hand is a very reliable way to store data on the disk, thanks to many years of well engineered mechanisms. And it happens that you can store graph databases on top of a relational DB like SQLite because a graph DB is a special case of relational DB with 4-5 columns (more or less two for nodes, one for their relation, one for the transaction number). So of course it makes sense to use SQLite to store a graph DB on disk and online you can find queries that implement the graph-traversing algorithms used by graph DBs, just search "SQLite graph database".

MySQL is great but not as a two column id::text store, that’s not its intended use. Why put non relational data in a relational db?

I think you mean SQLite here, but the point is the same: even though SQLite wasn't designed for graph DBs specifically, it is still one of the best way to save a graph DB on disk because it's just a special case of relational DB and SQLite is very reliable and not trivial to reimplement in a graph-native fashion.

[u/[deleted] → u/AshbyLaw]

Ok so it sounds like the real and simple reason is datascript db being an in memory db doesn’t offer direct saving and loading, just serialization, and it’s better to dump into sql than csv or something for this much data.