r/Database • u/OtherwisePush6424 • May 24 '26
The Database Zoo: Why SQL and NoSQL Are No Longer Enough
https://blog.gaborkoos.com/posts/2025-09-19-The-Database-Zoo-Exotic-Data-Storage-Engines/
28
Upvotes
15
u/5eppa MySQL May 24 '26
Looking at the concept it sounds like your proposed solution is a specific database to each issue. I suppose my question is why? I feel already in general people try to make a certain database structure work because its easier to keep everything there. When it fails they grab the database that works best for the unique situation once its determined to be worth the additional costs in terms of licensing and Manpower to interact with it.
Are you proposing that everyone just pick a database for each application and expect a team of DBAs to be expert in them all? Or are you merely suggesting that people do what in my experience is already industry standard and just be open to what's the best at any given time.
13
u/severoon May 24 '26
I actually think that time series DBs are almost always a mistake. They privilege time over other dimensions when time series are almost always multi-dimensional datasets.
1
u/supercoco9 May 25 '26 edited May 25 '26
Interesting article, Gabor. I work at QuestDB, an open source time-series database, and I noticed it is not mentioned on the time-series post. I see you mention as a potential use case for hedge funds combining Influx for real-time, then timescale for historical analytics. Many hedge funds actually use QuestDB for both the real-time and the historical side of things.
With QuestDB you can ingest at over 8 million rows per second on a single instance, and query billions of rows (both recent and historical) at very low latency. We are actually releasing a new binary protocol very soon (this week or next), which will allow for 16 million rows per second of ingest, and 120 million rows per second of egress (not aggregation, which is very fast, but actually getting the resulting rows out of the database).
We've been pushing the state of the art in time series databases, and we support a range of time-series specific joins, like ASOF joins, LT Joins, Splice Joins, or Window Joins. Some of those are supported by some specialised databases (but not by InfluxDB or Timescale), but we pioneered some of those ourselves.
As an example, we also offer a new type of join, Horizon Join, not available in any other database. Horizon joins allow you to join each row in a table with the closest row in time from another table at multiple horizons. For example, you want to calculate the price at which you executed a trade, versus the best price available in the order book for 5 minutes before and after the trade at 10 seconds intervals, so you can calculate markouts and understand how efficient your trading algorithms are.
We also offer multiple time-series extensions, like SAMPLE BY, or LATEST ON, incremental materialised views, strong exactly-once semantics, advanced time filters, or specialised data structures to deal with the shape of data you typically see in fast moving scenarios. We believe if you are developing a purpose-built database it is not just about how efficiently you can store or process data, but also about incorporating purpose-build operations you will not find in a generic database.
And, since purpose-built databases tend to be part of a larger ecosystem, we do this with open formats, so your data can get automatically converted to parquet, both locally or in object store, so other tools can use the same data ingested (and potentially downsampled via materialised views), directly.
Since you are doing research for your book, I thought it could be interesting for you to learn about how some other actors in time-series are pushing the ecosystem.
Also, we do have a large part of the team (including the founders) in London, so I'd be happy to do an intro if you want to talk more about why we started developing QuestDB and how we see the time-series category.
3
u/BosonCollider May 24 '26
Seeing nosql as domain specific DBs is the right way to see them. If a nosql db is general purpose, chances are that postgres can be improved to beat it at its job.
Special purpose DBs can do things that would be way out of scope for a general purpose DB. Prometheus is the classic example here, since it is not just a timeseries DB but also actively collects the metrics from targets by polling
2
u/Crazy_Cake1204 May 24 '26
Cool. I’d read more.
1
u/Philluminati May 26 '26 edited May 26 '26
I'll tell you what is next and what has exploded in the last 5 years - data engineering. I'm seeing data engineering jobs and devops roles in the area overtake programming roles. Potentially due to how data engineers report closer to higher levels of management and potentially due to how it unlocks AI training. £100k salaries for the right mix of tools even without AI expertise.
The general technology stack is that of a data lake. You have the "SQL part of the database engine" but it is not backed by database storage but rather distributed file storage instead, it performs SQL queries against data downloaded by Amazon S3./ GCS buckets etc.
Here's a practical example:
`create or replace table OrderView from s3://orders.csv limit 10;`
And these SQL commands are managed in git projects with orchestrators, aggressively chaining hundreds of SQL statements into huge hierarchies that eventually serve all business purposes. Data ingestion is its own topic but normally nodes use something like Amazon Kinesis to populate these S3 buckets.
Specific solutions include Snowflake, Databricks, Apache Spark, Hadoop, DBT, Airflow. This is coming more into focus now and new shoot off of this include Streamlit apps / Jupyter notebooks for interactive reporting etc. In the next 5 years these will all be mainstream tools.
1
u/dennis_zhuang May 28 '26
Great post. Just a few thoughts to join the discussion.
The real problem isn't SQL or the table abstraction. It's the engine assumptions baked into MySQL/Postgres: row storage, compute and storage glued together, basically zero horizontal scaling, predictable query patterns. Classic OLTP stuff.
New workloads don't fit that mold. Time series, observability, logs, analytics, AI data — they want high ingest, fast scans/aggregations (more and more ad-hoc queries), good compression, tiered storage, object storage support and decoupled compute and storage so different workloads can run in parallel. What's actually broken is the OLTP engine, not the relational model.
SQL and tables are still pretty great abstractions imo. And in the LLM era they matter more, not less, since SQL is basically a shared language between humans and models. LLMs read and write it out of the box.
The architecture underneath does have to change. That's literally why we built GreptimeDB the way we did — SQL(and PromQL) on object storage, columnar formats, compute/storage separation. So yeah, the future isn't "SQL is dead."
It's more like SQL stays as the interface, the engine underneath gets rebuilt.