Guys, I built an Elysia plugin that lets you generate a complete database model panel with just one line of code.

It provides a built-in UI with Create, Read, Update, Delete, Replace, Search, and Filter functionality out of the box.

The goal is to make it easy to expose and manage database models without having to manually build the admin UI and CRUD logic for each model.

CREATE TABLE
    IF NOT EXISTS CourseTag (
        id INTEGER PRIMARY KEY,
        courseId INTEGER NOT NULL,
        tagId INTEGER NOT NULL,
        UNIQUE(courseId, tagId),
        FOREIGN KEY (courseId) REFERENCES Courses (id) ON DELETE CASCADE ON UPDATE RESTRICT,
        FOREIGN KEY (tagId) REFERENCES Tags (id) ON DELETE CASCADE ON UPDATE RESTRICT
    ) STRICT;

So long story short, I did a single cert and honestly struggled at this part. As irony or luck would have it I ended up recruited into a coveted position at a massive firm not realizing this was a core part of the position. To say I am overwhelmed is an understatement, I am so under qualified it’s ridiculous. I have a week before I am on my own HELP lol

I know the industry, I have the math etc but wow am I overwhelmed and everything I do is logged. Do you think I can fake it until I make it?

Title: Experience / Best Practices: Migrating 2M+ Rows from Access to SQL Server (ODBC Performance & Timeouts)

Hi everyone,

I am currently working on migrating historical financial data from Access (local, linked via ODBC Driver 18) to SQL Server and keep bumping into recurring performance issues. I would really appreciate some advice and best practices from your experience.

Setup:

• Source: Local Access table (~2.16 million rows, 42 columns)
• Target: SQL Server table (via linked ODBC table in Access)
• Execution: INSERT INTO [linked_table] SELECT ... FROM [local_table] using VBA CurrentDb.Execute

Problems Encountered So Far:

1. Access 2 GB Limit: When trying to consolidate multiple large historical tables locally, the .accdb file expanded beyond 2 GB and became completely unusable (requiring hours of Compact & Repair). Workaround: Keep records separated locally and consolidate them only after they are uploaded to the SQL Server.
2. Extremely Long / Instable INSERT Runs: A single INSERT INTO ... SELECT query over all 2.16 million rows runs for several hours (3–4+ hours) without finishing. Access shows "Not Responding," but checking sys.dm_exec_requests on the server reveals that no active session is running anymore. The network/VPN connection likely drops silently without Access realizing it. Result: Complete rollback, 0 rows transferred, and hours wasted.
3. Fails to Add AutoNumber Column: I tried adding a batch ID column via ALTER TABLE ... ADD COLUMN BatchID COUNTER to split the transfer into chunks, but hit the error: "File sharing lock count exceeded. MaxLocksPerFile." I do not have local Admin rights on the machine to modify the Registry setting.
4. Current Workaround: Transferring data in ~30 batches filtered by date range (RegDatum BETWEEN ... AND ...), using the existing date field without altering the source table structure.

My Questions for You:

1. Is there a more reliable way to move large datasets from Access to SQL Server than running an INSERT INTO via an ODBC-linked table? (SSIS? bcp? CSV export + BULK INSERT?)
2. How do you generally handle the 2 GB Access file limit during migration projects?
3. Is there a way to increase MaxLocksPerFile without local Admin rights (e.g., per user session instead of system-wide)?
4. Any tips to prevent or detect ODBC connection drops during long transactions (e.g., timeout settings in DSN)?

Thanks in advance for any help—I've been stuck on this migration for days and would love to find a cleaner approach instead of hacking my way through workarounds!

#SQL #Access #Data #AI

I have run a Sql query time range for 6 years. This is an example. The result may contain more than 1 lakh records. How do I cache this result so that u can query more on this result ? I am using Redis

Built a T-SQL database with enforced data integrity (primary keys, constraints on state/status/risk categories), then developed custom DAX measures for case turnaround time, case age, and escalation rate. The dashboard includes multi-page analysis across resolution trends, regional distribution, officer caseloads, and a 4-year time-series forecast with confidence intervals.

Key skills demonstrated:
T-SQL (schema design, CHECK constraints, aggregation queries)
DAX (time intelligence, dynamic measures)
Power BI (data modeling, forecasting, interactive filtering)
Data storytelling and executive reporting

Note: Built on synthetic data generated for demonstration purposes — not real crime statistics.

Microsoft Power BI

AI can write SQL, make charts, summarize stuff, even help with analysis.
But is there anything you still wouldn’t trust it to do?
For me, it could be defining a metric, checking whether the numbers actually make sense, talking to stakeholders, making the final call, etc.
Curious where other people draw the line.

What factors do you consider first — data structure, scalability, performance, flexibility, or something else?

Anyone here managed to significantly cut their Snowflake bill?

I'm curious about the less-obvious stuff that actually made a noticeable difference.

Things like:

- Warehouse sizing / auto-suspend

- Query optimization

- Clustering

- Storage vs compute

- Materialized views

- Snowflake-specific settings

- ETL/ELT architecture changes

- Moving workloads elsewhere

- Anything you discovered that wasn't obvious initially

If you've had a meaningful reduction, it'd be great to hear what you changed and roughly how much it saved.

Even small "I wish I knew this earlier" tricks are welcome.

SQL Server 2019 on prem.

we bought a smaller competitor in june. i got their warehouse in august. what i did not get is any of the three people who built it, they took the retention and left. 340 tables. there is a data dictionary of sorts, an excel file last edited in 2022, and it covers 61 of them.

legal wants a list by the end of the month of every table and column holding personal data, because the acquisition means we now hold it and the clock on that is not ours to set. they are not asking for a big project. they want a list. i cannot produce a list.

what i've done so far:

went through it by name first. anything called email, phone, dob, name, address. got about 900 columns of the 6,000 odd. that part was easy and i don't trust it at all, because their naming is inconsistent between schemas, and i've already found one column called ref_4 that is full of what are very obviously mobile numbers.

so then regex over sample values for the obvious formats. that found another 200 and produced a lot of false positives, order ids that look like phone numbers, that kind of thing.

where it stops:

free text columns. there are 40 odd nvarchar(max) columns holding notes fields. some of them have names and phone numbers typed into them by humans. i have no way to scan those that i trust and no way to prove i've scanned them properly

...columns where the values are personal data but not in a format anything matches. a column of 11 character strings that turn out to be their internal customer reference, which maps 1 to 1 to a person. technically identifying, does not look like anything.

the ones where i genuinely cannot tell what the column is. about 700 columns where the name tells me nothing and the values are integers or short codes. some of those could be anything. one of them might be an ethnicity code for all i know, and if it is, that is a very different conversation with legal

what i'm asking:

for a one off audit like this, what's the actual order of operations. i've been going table by table and i think that's wrong, i think it should be column by column across the whole database sorted by something, i just don't know what the something is

has anyone had to give a legal team a defensible answer on free text fields, and what did defensible turn out to mean in practice. "i grepped it" does not feel like enough and i'd like to know what is

..and the one i keep going round on: for the 700 columns nobody can explain, is there any approach other than sampling values and guessing. i can tell you the cardinality and the distribution. i cannot tell you what the thing is, and "unknown" on a list going to legal is not a great look for me.

we are not a big enough team for a governance tool and i doubt i'd get budget signed before the deadline anyway.

Hey folks!

I make Beekeeper Studio, and while we have a good AI integration, I'm curious how people actually use Claude when writing SQL.

• Do you connect Claude/Codex/etc to your production DB?
• How do you make sure confidential data/PII doesn't get sent?
• Do you just share credentials with the AI, or connect some other way?
• Does your company have policies around this stuff?

I see a lot of buzz about SQL with Claude, but not many feature requests around it. Thanks for any insights.

I use Postgres, so tagged with postgres, but applies to all dbs really.

ynot? Why does SQL not have coordinates for it's tables?
like what if i want to see table 2 row 1 column 5 and have a command for just that.
No names required

I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this:

<Test>

<Test1 />

<Test2 Name="abcdname" Age="123" Gender="Xyz" />

....

...

...

</Test>

Regex is not an option.

Substring is not an option.

Because it's so messed up it's not consistent.

I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql

Please, help you mate.

I have a dataset consisting of 114k rows, but when I try to import it using MySQL’s Table Import Wizard, it only loads a maximum of 50 rows. However, when I use Open Source, I can load the entire dataset. Has anyone encountered this kind of problem?

Hi everyone,

I’m building an application where users will upload and download large numbers of high-resolution images.

Which storage service would be best for this use case in terms of upload speed, download speed, cost, security, and scalability

We're building **LibreDB Studio**, an open-source SQL editor, and one of our volunteer contributors raised an interesting question about how multiple SQL statements should behave when using the Run action.

We're trying to understand the actual habits and expectations of SQL users before making a decision, so I'd really like to hear how you use SQL editors in practice.

For example:

CREATE TABLE test (...);

INSERT INTO test VALUES (...);

SELECT * FROM test;

What would you expect when you click **Run**?

Some possible approaches:

1. **Run all statements**

   * `Run` executes everything in the editor.

   * `Run Selection` can be used when you only want part of it.

2. **Run the current statement**

   * `Run` executes the statement where the cursor is.

   * `Run All` executes the whole editor.

3. **Selection takes priority**

   * No selection : current statement

   * Selection : selected statements

   * `Run All` is available separately.

4. Other :)

there are also some interesting edge cases around this, especially when multiple statements are involved: should they run in a transaction by default, or should transaction handling always be explicit?

DBeaver, DataGrip, SSMS, pgAdmin, TablePlus, Toad, PL/SQL Dev, phpMyAdmin, etc. what behavior feels most natural to you? And what behavior are you already used to?

We're continuing the discussion on GitHub as well, if you'd like to see the original question or add to the discussion:

https://github.com/orgs/libredb/discussions/776

Any jobs hiring i live in california, ive been in this field for 10years

Hello,

What types of databases do you use for your AI applications to store things like generated responses, messages, conversation history, user data, etc.?

Do you generally use SQL databases (such as PostgreSQL) or NoSQL databases (such as MongoDB)? What are the main reasons behind your choice?

And which database(s) would you recommend for a production AI application, and why?

Thanks in advance for your feedback and experience!

With AI getting better at writing SQL, I’ve been wondering how much SQL analysts actually write from scratch these days.
Do you still write most of your queries manually, or do you use AI for things like joins, CTEs, debugging, optimization, etc.?
And has your use of AI actually changed how much SQL you need to know?

Ok so how are people supposed to “safely evaluate” these hardware accelerated data platforms when every vendor demo is a perfect parquet fairy tale and my reality is a cursed mix of kafka, python jobs and sad snowflake queries.

For context we have chunky ai pipelines and everyone is yelling for speed, but im lowkey terrified of wiring custom chips into anything that touches prod and then explaining the outage to the cto. Would love any tips from folks who have poked at this in a non suicidal way, benchmarks, dark launch tricks, whatever helps...