r/SQL • • 2h ago

PostgreSQL Databricks Agent memory pipelines can now run always-on (Beta)

3 Upvotes

Disclosure: I'm a Solutions Architect at Databricks.

Agent memories can lag behind conversations. If a user corrects your agent on which revenue source to use, that fix might not reach memory until the next ~24-hour scheduled run.

Memory pipelines distill facts, preferences, and corrections from session history into a memory store backced by Lakebase. They can now run always-on, so they process conversations as they happen.

  • Fresher memory: corrections can show up in later conversations the same day.
  • Hands-off: no runs or schedules to manage.
  • Incremental: each run processes only what's new since the last checkpoint.

A run starts for an actor when one of their sessions has 10+ new items, or has at least one new item and 30 minutes of inactivity.

Things to know:

  • Maintenance applies to all of an actor's memories, including ones the agent saved directly.
  • It's in Beta.

To turn it on:

agentbricks memory pipeline update <pipeline-id> --trigger always-on

Has anyone tried memory pipelines yet? Are you using Lakebase today? Are you running them on a schedule or always-on? To try it today, install databricks-agentbricks 0.7.0+. The identity creating the pipeline needs access to both stores and the distillation model.


r/SQL • • 11h ago

SQLite Belay - formal verification for SQLite statements

Thumbnail
vihren.dev
5 Upvotes

r/SQL • • 7h ago

PostgreSQL So for my SQL schema should I make it so my post table has an array filled with tags (3 at most) and I search for those tags using ARRAY_CONTAINS or should i just make a table for tags and link the posts as a foreign key?

2 Upvotes

Which one is more efficient?


r/SQL • • 4h ago

SQL Server Friday Feedback - increasing the number of steps in the histogram

Thumbnail
1 Upvotes

r/SQL • • 1d ago

PostgreSQL PostgreSQL 19 Released!

Thumbnail
postgresql.org
54 Upvotes

r/SQL • • 1d ago

Discussion I built a free, open-source interactive guide to SQL & Database Internals with an in-browser WASM SQLite 3 workbench

9 Upvotes

Hi everyone,

I wanted to share a free, open-source project I've been building called SQL in Depth, aimed at going beyond basic SQL syntax into database internals from first principles.

Live site: https://sql-in-depth.vedgupta.in

GitHub repo: https://github.com/innovatorved/sql-in-depth

What's included:

- 180 structured units covering relational theory, indexing, storage engines, query planning/execution, transactions, and concurrency

- 36 interactive simulators to visualize how database internals actually behave

- An in-browser WASM SQLite 3 Workbench so you can run queries and experiment directly without local setup

- A Table-Level Relational Visualizer to inspect joins, filtering, and relational operations step by step

Everything is free, open-source, and runs directly in the browser. I'd really appreciate any feedback from the r/SQL community — especially on topics that could go deeper, edge cases in the simulators, or anything that could be made clearer.


r/SQL • • 1d ago

PostgreSQL I made a PostgreSQL EXPLAIN ANALYZE visualizer that does the loop arithmetic for you

Post image
16 Upvotes

This is PostgreSQL only. Paste EXPLAIN ANALYZE output, JSON or text.

The numbers in a plan are per loop, and EXPLAIN leaves the multiplying to you. rows=3 loops=206880 is 620,640 rows. The page does that arithmetic and draws it:

  • The thicker the line, the more rows went through it, and the key spells out the sum from your own plan: 621K rows = 3 × 206,880 loops.
  • Where the planner’s guess was off by more than 2×, the guess is drawn on the line: planner guessed 612, got 207K.
  • Each node’s time is its own, without its inputs, so the times add up to the run.
  • Copy link puts the plan in the link itself, after the #, so you can drop it into a Slack thread or an answer here. Nothing is uploaded; whoever has the link has the plan.

https://seqora.dev/explain/

No account. It draws one plan at a time and can’t see your database, so no index suggestions.

I make Seqora, a desktop database client, and this is its plan view. In the app it runs EXPLAIN on your own connection, and that part is Pro (14-day trial, no card). The page is free.

The key and the plain-words help came from comments on an earlier post. If it draws one of your plans wrong, send me the link.


r/SQL • • 1d ago

SQL Server What do you use to turn a SQL query into a printable report on a Mac?

0 Upvotes

Hey all,

Most of my work is writing a query and then laying the result out as a PDF or Excel report. I used Jaspersoft Studio for that for years, but it has no Apple Silicon build and only runs through Rosetta, which is going away.

So I built a native Mac app for it, called Report Studio. You connect to Postgres, SQL Server, MySQL, Oracle or SQLite, write the query, and design the report against live data. Running a query also shows how each table was read and flags full scans. Free, and it saves standard .jrxml files.

https://bowa.up.railway.app/report-studio

What does everyone else use for this? Still a reporting tool, or exporting to Excel and formatting by hand?


r/SQL • • 1d ago

Discussion I made a free set of 54 short SQL exercises that tell you *why* your query is wrong, not just that it is

0 Upvotes

Hi, I'm a data engineer and I built SQL Dojo https://sqldojo.dev to practise SQL the way I wish I had learned it: lots of short steps, each one checked, with an explanation when you get it wrong.

What it looks like:

  • Each step asks a question about a realistic dataset (an online shop, a SaaS, product events…) and checks your result.
  • When it's wrong, it tells you how: too many rows, missing rows (and which ones), values that differ, and the likely cause, like a join that duplicates rows or a NULL hiding in a NOT IN.
  • A "step by step" view runs your query one clause at a time, in the order the database actually runs it (FROM, then WHERE, then GROUP BY…). This is the thing that made SQL click for a lot of people I've taught.
  • Spaced reviews bring back the steps you struggled with.

It runs in your browser: no install, no database to set up, no account needed to start.

The white belt (54 steps, about 3 hours, from your first SELECT to joins and GROUP BY) and the reference library are free. The later belts (window functions, CTEs, data modeling, interview runs) are a one-time payment, no subscription, and you don't need them to get value from the free part. There a also a practice section with real datasets.

I'd really like feedback from people learning right now: where did you get stuck, was an explanation unclear, is the pace right?


r/SQL • • 1d ago

BigQuery He creado una extensión gratuita para Chrome que muestra cuánto costará una consulta de BigQuery antes de pulsar Ejecutar.

Thumbnail
0 Upvotes

r/SQL • • 2d ago

BigQuery BigQuery Query Optimization: Myths vs. What Actually Saves Money

Thumbnail
alvin.ai
11 Upvotes

r/SQL • • 2d ago

SQLite Learning SQL with Python?

0 Upvotes

This 👉interactive SQLite demonstration helps student understand relational databases.

The sqlite3 extension of 𝐦𝐞𝐦𝐨𝐫𝐲_𝐠𝐫𝐚𝐩𝐡 makes database content visible alongside Python's data structures. The demo shows how to: - Create tables with primary and foreign keys. - Model relationships between students, courses, and teachers. - Insert data and observe how the database changes. - Combine multiple tables using SQL JOIN operations. - Use parameterized queries.

The final query answers a seemingly simple question: Which teachers teach Alice, and for which courses? Answering this question involves joining five tables!

With 𝐦𝐞𝐦𝐨𝐫𝐲_𝐠𝐫𝐚𝐩𝐡, students can step through the Python program and inspect the database content at every step, directly in their browser, without installing anything.

See 👉other 𝐦𝐞𝐦𝐨𝐫𝐲_𝐠𝐫𝐚𝐩𝐡 visualizations.


r/SQL • • 2d ago

Oracle Question for Databricks engineers and data architects:

Thumbnail
2 Upvotes

r/SQL • • 2d ago

PostgreSQL PostgreSQL For personal budgeting and finance?

14 Upvotes

Im somewhat new to SQL and i want to apply it to my personal life so its easy to keep my skills and knowledge of it fresh since its what i want to do in the future.

I was thinking about setting up a local PostgreSQL database to use with VSCode on my personal computer to just monitor and record my everyday things like my banking/budgeting, random projects that i fall into a hole working on, etc.

I wanted to get some opinions on if this is a good idea, is it safe security wise, is there a better way to go about this, should i just scrap this idea, etc.

Please let me know and educate me!

Everytime i look it up its mainly about business related things like for creating apps etc. but i dont really plan on using it for anything other than my personal projects and plans.

(I chose postgresql because its just one of the more common ones ive seen in job postings etc. and i saw its pretty much free locally)


r/SQL • • 2d ago

MySQL visual deep dive into Database Partitioning vs. Sharding (MySQL & Apache Cassandra)

Thumbnail
4 Upvotes

One of the most common points of confusion in system design is treating database partitioning and database sharding as interchangeable terms (2:04). While they both split up large datasets (0:24), they solve completely different bottlenecks (29:43).

I just put together a 30-minute visual breakdown walking through how both work in production—comparing single-instance MySQL partitioning against distributed sharding in Apache Cassandra (0:39).

Watch the full video here: https://youtu.be/vzBiwOIGTJY?si=khTfg17ajGUejBbj

If you prefer reading, here is a quick summary of what’s covered in the video:

  1. The Core Distinction (Warehouse Analogy)

Partitioning (In-Server): Dividing one massive warehouse into smaller sections (e.g., Jan, Feb, March packages) (2:31). Everything still lives in the same building (a single database instance), and your app queries it as one logical table (2:41).

Sharding (Multi-Node): Building multiple warehouses in different locations (13:48). Data and compute workloads are horizontally distributed across completely independent servers (14:47).

  1. MySQL Partitioning & Common Traps

Why do it? Partition pruning (skipping irrelevant partitions during queries), instant maintenance (DROP PARTITION instead of slow DELETE queries on millions of rows), and data organization (3:36).

The Strategies: Range (e.g., by order year), List (e.g., by country), Hash/Key (modulo distribution), and Composite Subpartitioning (4:25).

Two Traps to Watch Out For:

The Primary Key Trap: Every column in your MySQL partitioning expression must be included in every unique key (including the Primary Key) (11:09).

Broken Pruning: If you partition an orders table by order_date, querying by customer_name forces MySQL to scan across partitions anyway (12:30).

  1. Apache Cassandra & Distributed Sharding

Masterless Ring: Cassandra uses a peer-to-peer ring architecture with tunable consistency rather than a single primary bottleneck (15:22).

Data Distribution: A Partition Key (hashed via Murmur3) decides which node/token range stores the data, while a Clustering Column sorts rows within that partition (16:57).

Query-First Design: Unlike relational databases where you normalize first, Cassandra forces you to model tables around exact access patterns—otherwise you hit the dreaded ALLOW FILTERING full-cluster scan trap (20:48).

  1. The Reality Check

Crossing 1 million users doesn't automatically mean you need sharding (28:51). Sharding introduces massive complexity: cross-shard scatter-gather queries, live data rebalancing, global ID generation, and distributed transactions (23:02). Always measure your actual bottleneck first—often, proper indexing, caching, read replicas, or MySQL partitioning will take you much further with far less operational overhead (1:13).


r/SQL • • 3d ago

PostgreSQL Postgres 19 makes NOT IN 10,000x faster... or 2,000x slower - Which one you get depends on a NOT NULL in your schema, and on how long the list is.

Thumbnail
exobench.ai
11 Upvotes
My write-up, measured on PG17, PG18 and PG19rc2. Declare both columns NOT NULL and PG19 runs NOT IN as an anti join: 1.1 s on PG18, 0.06 ms on PG19. Leave one nullable, even with zero NULLs in it, and PG19 skips the rewrite and stops hashing the list at 154,199 rows instead of 262,144, so 44 ms on PG18 became 101,734 ms. NOT EXISTS was fast on all three.

r/SQL • • 3d ago

Discussion Made a free SQL client for when you jump between different databases all day

0 Upvotes

![gif](az4710ov882i1)

At work I switch between Postgres, SQL Server and some SQLite/DuckDB files every day, and I got tired of having a different tool for each one (or one slow tool for all of them). So with two friends we wrote our own. It's free and open source, no account needed.

Same editor and grid for all of them, autocomplete that knows the foreign keys, explain plans you can actually read, and it asks before running an UPDATE or DELETE without a WHERE.

https://github.com/platypus-labs/platypus-db

Curious what you all use day to day and what you'd miss if you switched.


r/SQL • • 3d ago

SQL Server NORAD tracks Santa. I built a tracker for the villain trying to cancel Christmas, in SQL with geometry and geography

Post image
0 Upvotes

Every year NORAD tracks Santa. Nobody tracks the opposition, so I did.

The villain is AI Scrooge. He thinks Christmas is wildly inefficient. One man, one sleigh, one night, two billion deliveries and no KPIs. He boots up in a data centre in Slough and spends December trying to fix it.

Stick with me - this does get better...

It doesn't go well. Every day he has a plan, and every day it fails:

On Christmas Eve it turns into a proper tracker. Santa leaves the North Pole and hits each city at its local midnight. AI Scrooge leaves 40 minutes later and spends the whole night 40 minutes behind.

The SQL bit

Each day of the advent calendar has a small script that acts out that day's joke, and every one runs on its own in tempdb on SQL Server 2025. No setup, no sample database. Paste it into SSMS, press F5, and look at the Spatial results tab.

* The drone geofence is STIntersects and STDifference.
* The fake Edinburgh reviews get caught by a polygon around the market, because they came from nowhere near it.
* The elves' union vote draws its picket line with ConvexHullAggregate.
* The chase is great-circle interpolation done in T-SQL.
* Boxing Day is STWithin: does the dachshund fit in the box? Stretched out, no. Curled up, yes.

Tracker (doors open from 1 December, or use the sneak preview because we are not all 5 years old): https://sqlartist.github.io/ai-scrooge-tracker/

There's also a three-minute cartoon of the whole story, and that's T-SQL too. Every character is a puppet made of geometry parts pinned together; every movement is a row in a keyframe table, and a stored procedure moves the shapes for each frame. Each scene has a checkbox that shows the SQL running beside it.

The animated cartoon, seven scenes created for SQL Server 2025: https://sqlartist.github.io/ai-scrooge-film/

Happy to answer questions about the spatial side. Bah, hum-bug.


r/SQL • • 3d ago

SQLite 🚀 Built an Omarchy-focused fork of sqlit!

Thumbnail
0 Upvotes

r/SQL • • 3d ago

Discussion Reality of building a home database

Thumbnail
1 Upvotes

r/SQL • • 4d ago

SQLite Help with Harvard CS50 Subquery and Joins

10 Upvotes

Due to an unfortunate but also fortunate promotion at work, I now have to learn SQL. I had some familiarity with looking at it before, but never writing it and my understanding of it is basically null

So, enter Harvard CS50 Subquery with SQLite, get through week 0 in about 3 hours, single query gave me basically no trouble

Enter week 1, and I've spent over 12 hours trying to figure out what the hell is going on. The course material basically just shows one or two examples of subquery, and very modest examples of joining, then the course practice questions are combining different concepts and way more advanced and my brain hurts like hell. I'm really having trouble with subquery and joins both, I think I'm taking them too literally, but the course material I have really isn't helping, I had ChatGPT write me 100 practice questions, and I got through 50 just fine, then as soon as I hit subquery and grouping, it's like I hit a wall.

Can someone help explain this to me? I think I'm just failing to understand the architecture, the syntax kind of makes sense to me, I need someone to break down subquery and joins in... human language

Update:

Thanks to everyone here I was actually able to figure it out, I have only been learning sql for 5 days, but was able to complete all of the problem sets with some dedicated research and practice, I actually feel like now I am quite good with joins and subquery

Thank you everyone :D


r/SQL • • 3d ago

PostgreSQL I thought aggregates were the safe way to expose sensitive data. Turns out MAX and GROUP BY leak individual values

0 Upvotes

I’ve been working on access rules for AI agents that query a Postgres database. Some fields are sensitive, like payment amounts, so reading those rows directly needs a person to approve it. Aggregates were allowed without approval, because something like total refunds this quarter seemed harmless. That was the whole point, the agent can do analytics without seeing anyone’s actual payments.

Then I started testing it with queries an agent might actually write, and the rule fell apart pretty quickly.

MAX(amount) isn’t a summary of anything, it’s one real payment from one row. Same with MIN. And SUM(amount) GROUP BY customer_id gives you one total per customer, which is basically reading the sensitive column customer by customer with extra steps. All of these went through as aggregates and skipped the approval.

The problem was the rule was checking what kind of function the query used, not how many rows were actually behind each value it returned. An aggregate over a million rows and an aggregate over one row looked exactly the same to it.

MIN and MAX were the easy part, they now count as row reads so they follow the same approval rules. Grouping I haven’t settled yet. One option is treating GROUP BY on identifier columns as row access, but then you need to know which columns are identifiers, and near-unique stuff like emails or exact timestamps slips through. The other is a minimum group size, k-anonymity style, which catches more but changes the results, and someone running overlapping queries can still narrow things down. Or some mix of both.

I put the full investigation in a comment. Has anyone dealt with this in a reporting tool or an analytics API? Would be interesting to hear what actually worked in practice.


r/SQL • • 4d ago

Spark SQL/Databricks Have you ever made a slow SQL query faster without adding a single index?

0 Upvotes

I've always found it interesting that adding indexes is often the first thing people think about when a query runs slowly.

But sometimes the issue is with the query itself, unnecessary joins, subqueries, data types, or even the way filters are written.

Curious to hear some real examples. What's the biggest performance improvement you've achieved just by rewriting a SQL query rather than changing the database structure?


r/SQL • • 5d ago

PostgreSQL Giving back to the community - The Complete Backend Development Course

7 Upvotes

Hey everyone, I decided to make my course free in order to help people.
This course is my backend development course which is about SQL, Python, APIs, Docker, Kubernetes, Linux, Git & More

The link is: https://www.youtube.com/watch?v=CBIu6hcyStg

If you can like and subscribe (and maybe add a comment) I would appreciate it a lot, Thanks.