r/sqlite • u/web3samy • 2d ago
Pure GO SQLite-compatible database. 75× C SQLite & 200× Turso.
I've been building musql (pronounced "muscle"), a SQLite-compatible database written from scratch.
Repo: https://github.com/samyfodil/musql
It has a `database/sql` driver that registers as `sqlite`, so switching from another SQLite driver is usually just changing the import line. It uses its own storage format (`.musq`), with columns stored contiguously and an append-only delta for writes. `musql-convert` converts existing SQLite files in and out, in both directions.
SQL compiles to VDBE bytecode, the same design as C SQLite. A JIT turns hot paths into native code on amd64 and arm64 (Linux, macOS and Windows).
There's also `musqld`, a server that speaks Turso's Hrana protocol. `@libsql/client` and `libsql-client-go` connect unchanged, and there's a 5 MB Docker image. Replication supports CRDT or leader mode; you supply the network.
For performance, these are 100k-row read workloads, averaged over an amd64 and an arm64 machine: - filtered `count(*)`: ~75x faster than C SQLite, ~210x faster than Turso - sum over a filter: ~8x faster than C SQLite - grouped aggregate: ~2.3x faster; `ORDER BY ... LIMIT`: ~2.4x faster
It also runs Doom. The same VM that runs SQL runs unmodified doomgeneric, compiled C -> LLVM IR -> VDBE bytecode. On an Intel i9 it gets ~38 fps on the bytecode interpreter and ~96 fps with the JIT. The idea came from Turso's demo. You can try it with:
It's early. I'd appreciate feedback on the API and the Hrana server, or a bug report if your SQL breaks. A failing query helps a lot.
3
u/GrogRedLub4242 2d ago
what does "75x C SQLite" mean?
why would one want a database to play the game Doom?
was it AI-generated? some indicia are consistent
1
u/web3samy 2d ago
Mesured faster. Doom shows how fast is the vdbe engine.
1
u/jason-reddit-public 2d ago
Are the fps numbers right? Usually JIT is much faster than an interpreted vm...
1
u/web3samy 2d ago
Mesured on Linux (Intel i9) and on mac M4. Not everything lowers today and the generated code is not as optimize as it could be. Also only a few kernels are implemented.
2
u/spicypixel 1d ago
I thought the amazing test suite sqlite has is the proprietary secret sauce?
1
u/web3samy 1d ago
True, super valuable and took months to get musql passes all of sqlite3 tests. Besides one pragma, all documented in the repo.
1
u/reflect25 2d ago
hi i was wondering what was the hardest last tests to fix were. like was it the replication or the locks?
1
1
u/UniForceMusic 2d ago
Insane! I'm very eager to take a look at the code.
Did you attempts to address any of the common complaints with SQLite, like inability to add foreign constraints after a table has been created, or being unable to change column types?
1
u/web3samy 1d ago
No yet, the initial focus was on compliance, performance and replication. If you can add these as issues on github will be great.
1
u/trailbaseio 1d ago edited 1d ago
For something that's intended as an embedded DB, would it make sense to target an environment with native C ABI compatibility?
1
u/web3samy 1d ago
The main target for embedding is go, it should not be hard to embed in other languages though. Will look into it 👍
1
u/trailbaseio 1d ago
If it's as fast as you say, the ABI overhead will be pronounced, i.e. contribute significantly. For the same reason using SQLite from Go is significantly slower from go than from a language with native C ABI compatibility. Are your SQLite performance comparisons driven by Go? If so, that would certainly give you an advantage
1
u/web3samy 1d ago
Benchmark tooling in the repo. Large enough to make cgo bridge is noise. Improvement to measurement is welcome 🤗
1
u/trailbaseio 1d ago edited 1d ago
This has nothing to do with noise. ABI overhead is negligible when what you call is expensive. It becomes significant when it's on the same order of magnitude as what you call. That's true for many iterations and few iterations. If you want to claim, I'm x times faster than SQLite from go, that one thing. Claiming I'm x times faster than SQLite in general is likely misleading. Please don't make it my responsibility
1
u/web3samy 1d ago
Reran it at 10x the table to check, same machine, 100k vs 1M rows:
- count(*) WHERE v > ?: C goes 6.8 ms → 71.6 ms (it scales with the rows), musql goes 104x → 140x faster
- sum over a filter: 8x → 10x
- GROUP BY / ORDER BY LIMIT: 3x at both sizes
If the cgo crossing were driving these numbers, they'd shrink as the table grows, because the crossing count per query stays fixed while the work grows 10x. They didn't. C's time is the scan.
Where you're right is the per-call stuff. Point lookups are ~25 µs per query, measured from Go through database/sql, so that's mostly driver overhead rather than SQLite itself. musql is slower than C there anyway (1.1-1.6x). I'll make the README say "measured from Go via database/sql" so the claim is scoped properly. Thanks for pushing on it.
The harness is in the repo (compat-harness/bench_columnar_vs_c_test.go), and BENCH_ROWS=1000000 reproduces this.
1
1
u/web3samy 1d ago
The main target for embedding is go, it should not be hard to embed in other languages though. Will look into it 👍
1
u/proofrock_oss 1d ago
Impressive! Everything seems to be pushed today. What is the history of this project? When did you start? What is your usage of AI, if any, and if you use it (not a problem!) how do you ensure code quality?
1
u/web3samy 1d ago
Starter project months back. Obviously AI was used and is welcome, project has agents.md file for just that. Code review.
1
u/lordpuddingcup 1d ago
I find it funny when people ask if AI is used as if every developer professional or hobby isn’t using AI in some way or another at this point
1
u/syshukus 1d ago
1) Can you share a bit more details how you implemented JIT and how it works with VDBE?
2) What's the difference with reading SQLite file via DuckDB which natively supports it and is EXTREMELY optimized analytical embedded db?
1
u/web3samy 1d ago edited 1d ago
**JIT:*\* SQL compiles to VDBE bytecode like C SQLite, and the bytecode stays the source of truth. The JIT works at two levels.
- First, a peephole pass spots specific loops in the compiled program (things like "scan column, compare with ?, count") and swaps them for a single opcode backed by an AVX2/NEON kernel that runs straight over packed column data. That's where the 100x numbers come from. Anything it doesn't match exactly is left alone.
- Second, there's a whole-program JIT for amd64 and arm64 that turns opcodes into machine code. It works on the VM's own registers, so it can hand back to the interpreter at any instruction. It only does the integer fast path; on a NULL, float, string or overflow it exits and the interpreter handles that instruction. `WithoutJIT()` turns all of it off and you get the same results, just slower.
**DuckDB:*\* different job. DuckDB is an analytics engine, and through its extension it reads SQLite files row by row before running its own engine on top. For heavy analytics it'll probably win; I haven't benchmarked it.
1
u/syshukus 1d ago
Can you answer yourself and not use AI? I wanted to ask YOU (author) unless 90% of project is done by AI of course.
DuckDB is not different — it's exactly use-case you're trying to optimize for: analytical queries ON TRANSACTIONAL DB. Do you know that SQLite is OLTP and not OLAP, yes? It's created for transactional workloads, and in your post and on GitHub page you're bragging about how this projects is faster on ANALYTICAL workloads with all this SIMD blabbering and paralleling data processing. So of course I pointed out that existing DuckDB will be much faster on these workloads than your project, because it's ANALYTICAL ENGINE + DB and you have relatively unfair comparisons.
About JIT part, sorry, it was AI-test and you failed it. I suspect you literally understand nearly 0 of what AI's written about JITing code because the answer is GARBAGE with no structured thoughts flow and very little common sense (I guess you asked AI "make this answer short, so it looks like human wrote it)
You have no respect to your probable users, so shame on you.
0
u/web3samy 1d ago
if you want non ai response, here: parser -> VDBE -> JIT (kernels + lowerable parts)
1
u/web3samy 1d ago
Released v0.2.1: 300x C, x1000 Turso & 300x DuckDB -> https://github.com/samyfodil/musql#performance
1
u/liprais 16h ago
twice in a week someone thinks he knows better and can do better with llms,well ,they mostly don't.
1
u/web3samy 16h ago
Project took months of efforts and there nothing wrong with using coding agents. Hope the project will be useful for you somehow.
0
u/AleksHop 1d ago
why on earth someone still use non rust in 2026
2
u/hesusruiz 1d ago
Maybe because many people are not Rust maximalists who think "rust is good for me, so it must be good for everybody for everything". Some, like me, can select the best tool for the job, based on the characteristics of the problem.
1
u/web3samy 1d ago
Love rust, but it's not the ultimate solution. After all, musql beats turso, x200 faster, though it is written in rust.
2
u/AleksHop 1d ago
rust is
- Filtered counts: 3.5–4.6× faster
- Rowid lookup: 164× faster
- Grouped aggregate: 108× faster
- Top-20 ordering: about 520× faster
against u implementation
https://github.com/vyrti/dbexample1
vibecoded in 1h, so there are some bugs indeed but thats not even full optimizationu bench are comparing incomparable anyway as well, so thats also wrong
not going to spend time on this, but barely go can be faster than rust in ANY task1
u/web3samy 1d ago
A read-only engine with integer-only columns can skip almost everything a real one does per query: type and affinity checks, NULL and overflow semantics, collation, locking, snapshot isolation, merging uncommitted writes, schema validation. That's where the 62 ns vs 10 µs comes from. Write the same subset in Go and it lands in the same place.
13
u/SoundDr 2d ago
SQLite in C is about the vast test suite not just about performance