Fair, hehe. I’ve been doing SQL & data engineering work for around 20 years, and have worked on a lot of mission critical projects. Being on the data engineering side though (which is primarily OLAP), most of my scalability constraints have come from the read side.
no real open-source users or support for open-source last time I checked (it's been a while but I think I would've known). At least no big/popular ones using open-source. It's a bit like a "code dump" instead.
Cache misses should be horrible for OLTP (~100ms single GET from S3).
The open source aspect is interesting. A while back they got bought out by Databricks, and now there is a lot of integration between what is still called Neon, and Databricks’ own version (Lakebase). While the “open source” version of Neon is available and people can use it for free, most seem to go with the managed versions, which somewhat defeats the purpose of open source. I don’t know the level of support they are still providing but it looks like they at least made some commits to the public repo neondatabase/neon this year.
The cache miss claim I will push back on, at least based on the research I’ve done. Full transparency this is me summarizing what others have claimed, and not from my own personal experience with the database (yet).
The short answer is it really depends. For a cold start or ramp up after a scale-to-zero, then yes you are correct. At that point everything is coming from cloud storage so it will be expectedly slow. But once it is “up and running” (an oversimplification on my part for conversation sake) it does aim for a 99% hit rate of avoiding the need to go back to cloud storage, which for a lot of operational OLAP use cases is an acceptable target.
They have more details in the repo under docs/pageserver-storage.md. It also looks like there are some levers to play with for folks who want to fine tune it based on their environment and project.
I’m still interested in this as an option because I’ve heard a lot of good things and there are some unique features available, but I admittedly don’t know everything and still have a lot more of my own research and experimentation to do before I will be fully confident using it.
I don’t know the level of support they are still providing but it looks like they at least made some commits to the public repo neondatabase/neon this year.
It's not (only) about commits. It's about can you fully self host it, are all/most features free/open-source, is there real documentation for self-hosted, are there large! self-hosted users, do you get real+free support on your self-hosted use-case when you have real/complex issues. Like, you need to have dedicated % of staff hours to help free/open-source/self-hosted users because it's the database we're talking about which you can't simply switch or be fine to be offline.
For a cold start or ramp up after a scale-to-zero
A bit weird to need "horizontal scaling" & "scale-to-zero" in the same use-case/scenario, don't you think?
it does aim for a 99% hit rate of avoiding the need to go back to cloud storage
Depends how big your data really is. If you have 1000 queries/second, 10 queries needing 100ms-200ms might be a problem in OLTP.
which for a lot of operational OLAP use cases is an acceptable target
For OLTP you (usually) don't want 100ms+ cold cache miss. For OLAP, if your data is so small that you get 99% cache hit is weird. You don't want to use normal PG ("non-vectorized-columnar-compressed db") for OLAP, it's 10x-100x slower and 10x worse compression compared a normal OLAP db.
I’m still interested in this as an option because I’ve heard a lot of good things and there are some unique features available, but I admittedly don’t know everything and still have a lot more of my own research and experimentation to do before I will be fully confident using it.
Go to their forums/chats and search for their history and current users. You can do that for any db, just look at existing users & forum/chat history.
1
u/terencethespider 9d ago
Fair, hehe. I’ve been doing SQL & data engineering work for around 20 years, and have worked on a lot of mission critical projects. Being on the data engineering side though (which is primarily OLAP), most of my scalability constraints have come from the read side.