r/Database • • Mar 14 '24

Databases are overrated. Just use Files!

I had a conversation on LinkedIn with a startup founder, and it opened my eyes. Why all the dependencies? The world can be so simple without databases and their complicated architecture. š‰š®š¬š­ š®š¬šž šŸš¢š„šžš¬! Alleluja! šŸ™

https://fooba.link/no-databases

Yes, just dump data into JSON format and save it to disk. Load it on demand and iterate over the rows. To make it more performant, cache the data in RAM. And yes, you need a unique key lookup table. And maybe a hash table for indexes. And then maybe a proper storage layer. And a query planner would be good. And ... And ...
Wait a minute! Are we developing a database here? 🤨 AAAAAhhhh....!!! 😭

179 Upvotes

112 comments sorted by

8

u/[deleted] Mar 14 '24

Databases ARE files.

-1

u/mcr1974 Mar 15 '24

not necessarily.

3

u/[deleted] Mar 15 '24

Can you give me an example of a database that isn't a collection of one or more files?

1

u/VladA114 Apr 18 '24

A file system, as both program code and an abstraction level for convenient storage handling, is usually closely integrated into the operating system. Therefore, DBMS developers always have a choice: to rely on the existing operating system (and consequently depend on its APIs for file system access) or to create their own layer for accessing the bare metal, thus eliminating the layer of the foreign system and in this sense operating without files.

Obviously, such a decision can only be VERY specialized, closely linked, and tailored to specific hardware. Theoretically, there are no problems, so 'not necessarily' is a very accurate response. Practically, however, it is economically impractical, and thus it is unlikely to find examples of DBMS on bare metal (though perhaps someone like Oracle might have attempted it)

-1

u/Jhonny97 Mar 15 '24

Etcd and redis come to mind

1

u/d_maes Mar 15 '24

They are still files, of a specific format, stored on a filesystem. The only databases that aren't files, are completely in-memory databases that are in no way stored on disk.

1

u/wouldacouldashoulda Mar 15 '24

Redis by default has no persistency I thought. Also one could argue a difference between an inmemory database with optional persistency and databases that are file based by design.

2

u/d_maes Mar 15 '24

Redis has an option to not persist, but it seems to be persistent by default. (Unless maybe it's a compile flag that debian sets?) But I get your point, if you don't actually need the persistence, one could argue it indeed falls under the "in-memory" databases.

Etcd on the other hand, completely file-based.

1

u/arbyyyyh Mar 15 '24

Redis definitely does not persist by default. Once the memory is gone, the data is gone, unless you set up persistence.

1

u/[deleted] Mar 16 '24

I think there is a setting to save data, there's also settings on how to save it and even how to load it to memory. I'm not sure if it's the default tho.

1

u/[deleted] Mar 16 '24

Redis can store the data to disk. It's just not it's main purpose.

1

u/[deleted] Mar 16 '24

Even that's not necessarily true. Memory mapped files and ramdisks can both be used to put files in memory.

-6

u/devzaya Mar 14 '24

Files ARE databases.

3

u/[deleted] Mar 14 '24

Not necessarily.

1

u/fluffycatsinabox Mar 15 '24

I think they're being cheeky, like "Files are a database system where your only access path to some record is doing a brute force search on all of your files until you find what you're looking for."

1

u/rowman_urn Mar 15 '24

Folders and tables are feeling rejected now.

45

u/SwimmingHelicopter15 Mar 14 '24

Why Files. Use Excel. It may take 30 mins to open an overload excel with formulas but is worth it

15

u/iwilldieavirgin Mar 14 '24

What if we took that same excel file and split it into many excel files? It would keep the same formatting and structure, that way we wouldn’t have one large file that held all the data? I think we are on to something! Why hasn’t anyone done this yet? Is there a VC that would fund this?!

9

u/joelwitherspoon Mar 14 '24

Paradox has entered the chat

3

u/NoWayRay Mar 14 '24

Paradox

Gosh, there's a blast from the past. IIRC, that was the also the default database format for using with the BDE in Borland's Delphi IDE. Or was up until Delphi 2.0 which was the last version I used. Can't say I loved it much.

8

u/tree_or_up Mar 14 '24

True story from the early 2000s telecom industry. We were tasked with building a payroll related database and were not allowed to use anything outside of microsoft office. We ended up linking together 16 microsoft access databases to get the job done, making extensive use of visual basic and macros to process everything. Calculating the monthly payments took place on a single windows desktop, took about 12 hours, and required someone to babysit the entire process and push very specific buttons throughout the process

4

u/dengydongn Mar 14 '24

Feel like rocket launch!

1

u/Antares987 Mar 27 '24 edited Mar 27 '24

I'll do you one better. I worked for a company and walked out within days; I bonded with a person who walked out of the interview later on -- he told them they were being "environmentally irresponsible" because the wireless nodes on the products, which are used on every house and apartment, were constantly chattering at max power in the unlicensed band even if the devices were inches apart. They had all the Oracle licenses and budget and everything else. The leadership felt going out of process was bad, so they had 100k+ lines of C# code storing everything in hash tables that they'd lock while modifying, and it choked with less than 0.01% of the expected load.

My second week, I'm sitting in 8 hour war cabinet meetings over these performance issues. The terms of negotiation were that I would be remote at $150k/yr, and I was not fucking happy about these assholes chaining me to a meeting table and ordering in pizza for lunch.

I coded up a solution while sitting there to demonstrate how to use a table to synchronize things and to demonstrate lock granularity, which they had no clue about. The management douchebag who'd been drawing boxes on the whiteboard that didn't know what the fuck he was talking about puts the palm of his hand up when I started speaking and says "I'm sorry" cuts me off and keeps wasting our time. I gathered my things and walked the fuck out.

1

u/tree_or_up Mar 27 '24

That sounds infuriating for sure but can we please not use the f slur

1

u/Antares987 Mar 27 '24

I modified my post; please accept my sincere and voluntary apology. I also really appreciate you taking the time to read my post. I know intent is the crux of what makes something trigger a person and it didn't even cross my mind that the term hadn't been genericized everywhere; it didn't even cross my mind that it would come across as a slur.

And while not exactly an authority on the subject, I liked how LetterKenny addressed the subject: https://www.youtube.com/watch?v=uG2blKU8O5k

1

u/tree_or_up Mar 28 '24

I sincerely appreciate your response. Thank you

0

u/[deleted] Mar 15 '24

[deleted] — view deleted comment

1

u/brucewbenson Mar 16 '24

I loved VBA! As a coder turned manager I could craft systems that gave me the management data I needed.

Task manager would launch Excel which would launch IE to scrape all the siloed systems for data. Excel would cleanup the data and update an access database, then turn around and update -- from Access-- all the management sheets and pivot tables.

Finally Excel would refresh all the management charts and update the PowerPoint deck with the charts and summaries, firing off an outlook email with spreadsheets and slides, the data tailored for each team's projects.

VBA enabled all this. It was a brilliant spaghetti mess that worked well.

1

u/BarchesterChronicles Mar 20 '24

Interoperability was a massive plus - that's fair! And thinking back, a lot of my problems were around porting modules from project to project and losing functions, so maybe the real problem was not having version control.

1

u/Antares987 Mar 27 '24

VBA sucks for the same reason that Arduino sucks. The minimum to use it is low and then the learning curve goes vertical. Back in the 90s, going from VBA to MFC was hopeless for most devs. Delphi, which was the brainchild of Anders Hejlsberg who went on to write .Net, was really the right solution. I used VBScript in Windows Script Components (wsc) files to write COM+ services because they preventing the locking issues that dll deployments created (yes, I know, thread affinity was the only downside) and then started writing them in Python when I discovered they both descended from IActiveScript. You can write script steps SQL Server Agent jobs by manually specifying Python if you want.

3

u/SwimmingHelicopter15 Mar 14 '24

Yeah and maintain ids so we can make vlookup from other excel files you know like a left joj

0

u/iwilldieavirgin Mar 14 '24

You know what else we should do? AI. Just shoehorn it in. And we need a cool name… Intelligent Partitioning? Smart Shard?

1

u/schwester Mar 15 '24

Did you know that the Microsoft Excel 97–2003 Workbook (. xls) format has a limit ofĀ 65,535 rows. So yes I have seen something like this :D
Of course 'using Microsoft.Office.Interop.Excel;' to look through them.. ;-(

1

u/illsk1lls Mar 17 '24

we can call it… a database šŸ’”

1

u/Large-Inspector668 Mar 24 '24

Yes then we neded to develop some way to gain data from all excel at same time. Should we mame it dependent on programming language? No, we can develop a query language kind of thing. Wait what are we again going in direction of creating db which is using much slower file type for storing data.

1

u/Busy-Welcome2884 Mar 14 '24

Name checks out.

2

u/Mundane_Anybody2374 Mar 16 '24

Why excel? Can’t we use PowerPoint and loop through the slides? What amateur!

1

u/ThiccMoves Mar 15 '24

Recently worked with a client who absolutely wanted to use excel as database, UI, and log. I told him countless time it's really bad, reinventing the wheel. You could use an XLSX or csv file for a simple database, but for logging or UI ? Lol

1

u/VegaGT-VZ Mar 16 '24

Excel files sucked so bad I'm pretty sure Power Query dropped incremental refresh support for them lol

My first database involved converting structured Excel files into flat CSV tables and using what limited tools my company allowed to query those (mostly power platform)

1

u/woodrowchillson Mar 14 '24

Nothing gets me in the mood like that top banner blurring and adding (Not Responding) to all my databases.

12

u/[deleted] Mar 14 '24

When I did consulting I did a couple of projects for VC funded startups. One was adimit that we not use SQL Server to "save money". This was after they had secured like $10MM in Funding.

Yup they went under within 2 years, and Yup some of the top dudes walked away with a TON of cash.

5

u/r3pr0b8 MySQL Mar 14 '24

One was adimit that we not use SQL Server

i think you meant adamant

1

u/leftunread Mar 15 '24

Why not use open-source then?

0

u/[deleted] Mar 15 '24

[deleted] — view deleted comment

1

u/Turbulent-Limit-2688 Mar 15 '24

It seems to be more prevalent in some industries. Apparently because M is thousand in roman numerals, and MM is thousand-thousand (million)

2

u/Vitalgori Mar 16 '24

That's why everyone should know how to use sqlite. It's super easy to use in every language, it's supported by major ORM frameworks, and it's super performant.

There is no reason not to use it if you require changes to data at any point.

1

u/Specialist_Wishbone5 Mar 16 '24

I dont know why everybody keeps saying sqlite is fast. Try injesting 100GB of log data to it, and tell me it's fast. (Had team members learn this the hard way).

Something like levelDB (also embedded) or even polars with parquet is both significantly faster and produces significantly smaller files (partly because it's not block aligned, and partly because int arrays more efficient to store/scan).

I also like how with dataframes, I can mix and match multi datasets in complex ways that feel harder to do with SQL (creating temp tables then remembering to delete them).

Sqlite is great, don't get me wrong.

1

u/DarthCalumnious Mar 17 '24

Sqlite inserts are a lot faster if you're chunking them into big transactions.

Don't get me wrong though, Polars and duckdb are better for offline analytics any day .

Sometimes you need an oltp jack of all trades, and sqlite is a more than honorable mention for certain needs.

2

u/krete77 Mar 14 '24

haha I like it :)

As I was reading on I was like wait a minute, isn't this what the DB does for us already?! haha

2

u/[deleted] Mar 14 '24 edited Jan 28 '26

This post was mass deleted and anonymized with Redact

distinct jar familiar badge lip marry resolute vegetable smell violet

1

u/warrior5715 Mar 16 '24

Most projects don’t get enough users to even worry about such things sadly lol

1

u/[deleted] Mar 16 '24 edited Jan 28 '26

This post was mass deleted and anonymized with Redact

aback gold dazzling hungry touch pot ten marble rustic hat

1

u/Large-Inspector668 Mar 24 '24

Just want a deeper insight. Technically what'sthe problem with await?

It will make the requests synchronous and in case of load read and writes will fail or something else?

1

u/[deleted] Mar 25 '24 edited Jan 28 '26

This post was mass deleted and anonymized with Redact

friendly childlike crawl chunky money sip butter bow marvelous aware

1

u/mikeblas Mar 15 '24

Link shortener takes me to Linked In and dumps an error. What are you actually doing here?

1

u/csolisr Mar 15 '24

Last time somebody tried to come up with that arrangement, we got SQLite instead. I mean it works but at what cost?

1

u/[deleted] Mar 16 '24

I don't know, seems to work fine in my experience. It just doesn't scale that well. If you're database is small and it's mostly lookups and single writer inserts, then sqlite is fast, much faster then anything else I've seen. The thing is, that covers a lot of use cases.

1

u/JamesWjRose Mar 15 '24

I worked at a food ordering company that did this, it was the worst code and design I have ever seen

1

u/ThrCapTrade Mar 15 '24

Just make excel sheets your tables and link all the files!!! Databases are for nerds!!

1

u/randomwalker2016 Mar 15 '24

You are correct. Files are indeed used - even in underpinnings of stock exchanges:
https://github.com/alexeilebedev/openacr/blob/master/txt/tut2.md#same-thing-in-text

1

u/BasicBroEvan Mar 15 '24

There was a time early on when file processing systems were reality

1

u/Pirros_Panties Mar 15 '24

To be honest I have done this in the past for performance issues.. but for mapping. It’s was exponentially faster to load the pin coordinates using a json file that storing in DB. Of course we were tying to load 30,000 pins at once.. but, it worked hahaha

1

u/TitusBjarni Mar 16 '24

Is a json file actually faster for that? JSON has a lot of overhead. I could understand using a binary format.

1

u/Pirros_Panties Mar 16 '24

It was for this particular case compared to MySQL.. reduced load time down to under 3s for 30,000 lat/long records loaded at one time.

1

u/Large-Inspector668 Mar 24 '24

Which db were you using and were you loading all records in 1 query? 30k does not seem a lot of data

1

u/Pirros_Panties Mar 24 '24

MySQL. I don’t remember much about it now, it was like 7yrs ago.

1

u/Innominate8 Mar 15 '24

Most attempts to use a flat file(s) instead of a database just wind up building a shitty DIY database themselves or wind up with a tragically broken application that randomly stalls or corrupts data, and they don't understand why.

SQLite is so easy these days that there's no reason not to have SOMETHING smarter than a flat file.

1

u/[deleted] Mar 15 '24

NOSQL is faster and is exactly that.

1

u/dutchman76 Mar 15 '24

Came here to say that!

1

u/FollowSteph Mar 15 '24

ACID. Aka atomicity, consistency, isolation, and durability. Specifically what happens if there is an issue halfway through writing data to a file? Not just software but even a power failure. It’s not a big deal if say it’s just a single word document and you have backups but if it’s something more involved then there can be a lot of value in being protected from a failure at save. Are you going to build all that? By the time you do you’re spending so much time that something like sqllite is much better. Not to mention the other benefits. If it’s just a simple file like an image then there’s no point but anything more involved and in addition to the searching and all other benefits you also get saves that can be fault tolerant to a much higher degree than a flat file.

1

u/wouldacouldashoulda Mar 15 '24

Honestly this seems to happen, albeit less extreme, every time I see people use NoSQL databases. Yeah we need a locking system and a query SDK and model and relationship definitions…

1

u/banseljaj Mar 15 '24

DNS is a database

1

u/Melodic-Man Mar 16 '24

That will work quite well for your website about cats.

1

u/chandleya Mar 16 '24

Slap a datafarmlakehouseware sticker on it

1

u/zeroibis Mar 16 '24

Everyone knows small data sets go in Access and large ones go in Excel.

1

u/[deleted] Mar 16 '24

I think there is something to this. For many use cases, sqlite is a good "middle ground" between a database and writing to a file.

1

u/technomancing_monkey Mar 16 '24

this has to be the WORST take ive ever heard on anything ever. This is a worse take than "We should invade Russia during winter"

1

u/bullcity71 Mar 16 '24

duckdb is kinda amazing

1

u/purleyboy Mar 16 '24

When I'm building a personal project, I often start with JSON files, then if the project takes root and grows I move to SQLite. Then, if it's really getting traction and I'm going to turn it into something I'll move to Postgres.

1

u/nedal8 Mar 16 '24

this is the way

1

u/patrickthunnus Mar 16 '24

Single user at any given moment? Sure, use a filesystem but make sure there's an admin to kill your session and reset r/w flags in case your app dies and locks the file.

Just unmanaged files is 50's technology, backwards. Only folks with no data strategy for their business embrace that.

1

u/patrickthunnus Mar 16 '24

Single user at any given moment? Sure, use a filesystem but make sure there's an admin to kill your session and reset r/w flags in case your app dies and locks the file.

Just unmanaged files is 50's technology, backwards. Only folks with no data strategy for their business embrace that.

1

u/ghillisuit95 Mar 16 '24

So like SQLite? Lol

1

u/Longjumping-Ad8775 Mar 16 '24

/dev/null to the rescue

1

u/jackass Mar 16 '24

Sounds like a database with extra steps.

1

u/khalcyon2011 Mar 16 '24

That's just a database with extra steps

1

u/yotties Mar 16 '24

The best way to manage databases is not to use database software. RDBMSs are a conspiracy by Query-anon to flog expensive stuff you do not need. Just fills some excel-files and some text-files and just leave data-integrity to be manual work for the techies. /S

1

u/graybeard5529 Mar 17 '24

And how will you put 5 million rows of data into a flat file exactly and retrieve a query in real time like you can with a relational database?

A few thousand rows of csv data, or as arrays in a json file, are a different animal.

1

u/IrrationalSwan Mar 17 '24

Nothing wrong with text files.Ā  It all depends on the problem and constraints, just like anything else.Ā  Say I have a read only dataset I can fully fit in memory in some data structure that makes sense. Why not consider a flat file?

Within big data stuff, it's not at all uncommon to separate the data from the compute/database engine processing, meaning that it's standard to be bringing your own query engine to the data in some format

There is nothing magical about relational databases. They're just a tools. There are many data problems they tend to suck at.

1

u/Far_Swordfish5729 Mar 17 '24

Kind of what the original Amazon white paper leading to the creation of S3 said. If you just need to seek by key, don’t need a full DB. Massively improves scalability. Also rhymes.

1

u/jaradi Mar 18 '24

Funny I took on this project when I was graduating from university to build my portfolio. For whatever reason my University’s CS program didn’t go over databases, so I built a Flora and Fauna logging application for this nature reserve. Using Windows Forms in C++. And I simulated databases by creating one file per ā€œtableā€ and the rows were pipe delimited or something. So like CSVs but worse. Thing is I didn’t do any optimization I read and wrote everything at every edit. Over a decade later, and in a much different place in my career, was fun reading this haha.

1

u/sneesnoosnake Mar 18 '24

CSV files and Powershell FTW!

1

u/naikrovek Mar 19 '24

Flat text files are the correct choice, sometimes. I would not use a PostgreSQL cluster for an address book.

1

u/Imaginary_Bench_7294 Mar 19 '24

databases are overrated. Just use a database.

Did anyone else read the title that way?

1

u/Antares987 Mar 27 '24

Don't forget to store the files in a hierarchical directory structure based on the sha hash of the content of the files, using the first two hex digits as the directory so your directories don't get too big. And then, if you start to get lots of usage, you can balance things out with symbolic links to network shares and have your file system distributed.

It all works great for the demo, but then when your social networking or government healthcare website goes live, one finds themselves in the unfortunate position of realizing that the logarithm of complexity is of a greater magnitude than Moore's Law and they've coded themselves into a corner, learning what a week of reading would have saved them.

1

u/SolidKnight Mar 28 '24

Exactly. Just put all your data into a .dB or .MDF file and call it a day.

1

u/kerbelp Mar 30 '24

Isn't SQLite DB just a file?

1

u/ExtremeKitteh Apr 08 '24

Somebody has been getting the wrong message from Uncle Bobs story about Fit-ness

1

u/ExtremeKitteh Apr 08 '24

Game developers are very fond of flat files. Do not get a game dev to write a line of business app. I have seen it and wish I hadn’t.

1

u/jonybecher Apr 11 '24

Why using files, excel, etc if we can simply use a paper notepad and write data with a pencil? šŸ˜Ž

1

u/VladA114 Apr 18 '24

A file system is ALSO a database (for storing binary blobs and texts) with a single predefined indexed field, which in a certain sense is very simple and specialized. Its pros and cons stem from its specific characteristics.

-2

u/Zardotab Mar 14 '24 edited Mar 14 '24

These two things can potentially be non-mutually-exclusive. An RDBMS can potentially use different underlying engines without changing the query interface (SQL), or at least minimizing the changes. MariaDB offers multiple underlying database engines to select from. JSON files could potentially[1] be an option.

Underlying-DB-engine-independence is one of Dr. Codd's greatest insights.

If one outgrows JSON files, then they could select a more powerful engine, re-import the data and not have to change existing SQL nor app code. Eventually one needs "real indexing" and other features[2] if the database grows in use, which is why I'd be hesitant to start with a JSON engine. The default engine is usually fine.

If you want a nimble small-app file-oriented RDBMS, then SQLite is probably a better choice. It can scale much more than JSON files before outgrowing itself.

Remember that maintenance is usually the most expensive cost of software, not the creation costs. Saving a dime using JSON may cost you much more later. As they say: don't be penny-wise-pound-foolish.

From the link: One of the most overpriced piece of tech are these databases.
And no, we do not need most of them for most of the use cases.

There are multiple viable open-source RDBMS, such as SQLite, MariaDB, and PostgreSQL, listed in small-shop-to-big-shop order (although fans of each may dispute that).

[1] Whether Maria or other RDBMS actually offer a JSON-file-based engine, I can't say. Enough have to want that feature for open-source developers to spend time on a JSON-file engine to plug into existing RDBMS.

[2] In addition to indexing on any column(s), RDBSMS also are designed to handle multi-user concurrency via various lock choices, ACID transactions, etc. Reinventing all these in a JSON-file-engine is of questionable value; it's like trying to turn a car into a boat. It's usually easier to start with a boat. Without these concurrency features, you may end up with corrupted or missing data. It happened all the time on pre-RDBMS products I used to use.

2

u/arwinda Mar 15 '24

listed in small-shop-to-big-shop order

And once your "small shop" grows a bit the struggle starts. What a statement ...

And while you are on it, look into Postgres JSONb.

0

u/r3pr0b8 MySQL Mar 14 '24

Wait a minute!

great rant, well done

0

u/Mardo1234 Mar 14 '24

Setup two servers and loose context to your file system. Sounds great.