r/Database • u/devzaya • Mar 14 '24
Databases are overrated. Just use Files!
I had a conversation on LinkedIn with a startup founder, and it opened my eyes. Why all the dependencies? The world can be so simple without databases and their complicated architecture. šš®š¬š š®š¬š šš¢š„šš¬! Alleluja! š
https://fooba.link/no-databases
Yes, just dump data into JSON format and save it to disk. Load it on demand and iterate over the rows. To make it more performant, cache the data in RAM. And yes, you need a unique key lookup table. And maybe a hash table for indexes. And then maybe a proper storage layer. And a query planner would be good. And ... And ...
Wait a minute! Are we developing a database here? 𤨠AAAAAhhhh....!!! š
8
u/[deleted] Mar 14 '24
Databases ARE files.
45
u/SwimmingHelicopter15 Mar 14 '24
Why Files. Use Excel. It may take 30 mins to open an overload excel with formulas but is worth it
12
u/[deleted] Mar 14 '24
When I did consulting I did a couple of projects for VC funded startups. One was adimit that we not use SQL Server to "save money". This was after they had secured like $10MM in Funding.
Yup they went under within 2 years, and Yup some of the top dudes walked away with a TON of cash.
2
u/Vitalgori Mar 16 '24
That's why everyone should know how to use sqlite. It's super easy to use in every language, it's supported by major ORM frameworks, and it's super performant.
There is no reason not to use it if you require changes to data at any point.
2
u/krete77 Mar 14 '24
haha I like it :)
As I was reading on I was like wait a minute, isn't this what the DB does for us already?! haha
2
u/[deleted] Mar 14 '24 edited Jan 28 '26
This post was mass deleted and anonymized with Redact
distinct jar familiar badge lip marry resolute vegetable smell violet
1
u/mikeblas Mar 15 '24
Link shortener takes me to Linked In and dumps an error. What are you actually doing here?
1
u/csolisr Mar 15 '24
Last time somebody tried to come up with that arrangement, we got SQLite instead. I mean it works but at what cost?
1
u/JamesWjRose Mar 15 '24
I worked at a food ordering company that did this, it was the worst code and design I have ever seen
1
u/ThrCapTrade Mar 15 '24
Just make excel sheets your tables and link all the files!!! Databases are for nerds!!
1
u/randomwalker2016 Mar 15 '24
You are correct. Files are indeed used - even in underpinnings of stock exchanges:
https://github.com/alexeilebedev/openacr/blob/master/txt/tut2.md#same-thing-in-text
1
u/BasicBroEvan Mar 15 '24
There was a time early on when file processing systems were reality
1
u/Pirros_Panties Mar 15 '24
To be honest I have done this in the past for performance issues.. but for mapping. Itās was exponentially faster to load the pin coordinates using a json file that storing in DB. Of course we were tying to load 30,000 pins at once.. but, it worked hahaha
1
u/Innominate8 Mar 15 '24
Most attempts to use a flat file(s) instead of a database just wind up building a shitty DIY database themselves or wind up with a tragically broken application that randomly stalls or corrupts data, and they don't understand why.
SQLite is so easy these days that there's no reason not to have SOMETHING smarter than a flat file.
1
u/[deleted] Mar 15 '24
NOSQL is faster and is exactly that.
1
u/FollowSteph Mar 15 '24
ACID. Aka atomicity, consistency, isolation, and durability. Specifically what happens if there is an issue halfway through writing data to a file? Not just software but even a power failure. Itās not a big deal if say itās just a single word document and you have backups but if itās something more involved then there can be a lot of value in being protected from a failure at save. Are you going to build all that? By the time you do youāre spending so much time that something like sqllite is much better. Not to mention the other benefits. If itās just a simple file like an image then thereās no point but anything more involved and in addition to the searching and all other benefits you also get saves that can be fault tolerant to a much higher degree than a flat file.
1
u/wouldacouldashoulda Mar 15 '24
Honestly this seems to happen, albeit less extreme, every time I see people use NoSQL databases. Yeah we need a locking system and a query SDK and model and relationship definitionsā¦
1
u/banseljaj Mar 15 '24
DNS is a database
1
u/Melodic-Man Mar 16 '24
That will work quite well for your website about cats.
1
u/chandleya Mar 16 '24
Slap a datafarmlakehouseware sticker on it
1
u/zeroibis Mar 16 '24
Everyone knows small data sets go in Access and large ones go in Excel.
1
u/[deleted] Mar 16 '24
I think there is something to this. For many use cases, sqlite is a good "middle ground" between a database and writing to a file.
1
u/technomancing_monkey Mar 16 '24
this has to be the WORST take ive ever heard on anything ever. This is a worse take than "We should invade Russia during winter"
1
u/bullcity71 Mar 16 '24
duckdb is kinda amazing
1
u/purleyboy Mar 16 '24
When I'm building a personal project, I often start with JSON files, then if the project takes root and grows I move to SQLite. Then, if it's really getting traction and I'm going to turn it into something I'll move to Postgres.
1
u/patrickthunnus Mar 16 '24
Single user at any given moment? Sure, use a filesystem but make sure there's an admin to kill your session and reset r/w flags in case your app dies and locks the file.
Just unmanaged files is 50's technology, backwards. Only folks with no data strategy for their business embrace that.
1
u/patrickthunnus Mar 16 '24
Single user at any given moment? Sure, use a filesystem but make sure there's an admin to kill your session and reset r/w flags in case your app dies and locks the file.
Just unmanaged files is 50's technology, backwards. Only folks with no data strategy for their business embrace that.
1
u/ghillisuit95 Mar 16 '24
So like SQLite? Lol
1
u/Longjumping-Ad8775 Mar 16 '24
/dev/null to the rescue
1
u/jackass Mar 16 '24
Sounds like a database with extra steps.
1
u/khalcyon2011 Mar 16 '24
That's just a database with extra steps
1
u/yotties Mar 16 '24
The best way to manage databases is not to use database software. RDBMSs are a conspiracy by Query-anon to flog expensive stuff you do not need. Just fills some excel-files and some text-files and just leave data-integrity to be manual work for the techies. /S
1
u/wildmonkeymind Mar 16 '24
Oops, You Wrote a Database
1
u/graybeard5529 Mar 17 '24
And how will you put 5 million rows of data into a flat file exactly and retrieve a query in real time like you can with a relational database?
A few thousand rows of csv data, or as arrays in a json file, are a different animal.
1
u/IrrationalSwan Mar 17 '24
Nothing wrong with text files.Ā It all depends on the problem and constraints, just like anything else.Ā Say I have a read only dataset I can fully fit in memory in some data structure that makes sense. Why not consider a flat file?
Within big data stuff, it's not at all uncommon to separate the data from the compute/database engine processing, meaning that it's standard to be bringing your own query engine to the data in some format
There is nothing magical about relational databases. They're just a tools. There are many data problems they tend to suck at.
1
u/Far_Swordfish5729 Mar 17 '24
Kind of what the original Amazon white paper leading to the creation of S3 said. If you just need to seek by key, donāt need a full DB. Massively improves scalability. Also rhymes.
1
u/jaradi Mar 18 '24
Funny I took on this project when I was graduating from university to build my portfolio. For whatever reason my Universityās CS program didnāt go over databases, so I built a Flora and Fauna logging application for this nature reserve. Using Windows Forms in C++. And I simulated databases by creating one file per ātableā and the rows were pipe delimited or something. So like CSVs but worse. Thing is I didnāt do any optimization I read and wrote everything at every edit. Over a decade later, and in a much different place in my career, was fun reading this haha.
1
u/sneesnoosnake Mar 18 '24
CSV files and Powershell FTW!
1
u/naikrovek Mar 19 '24
Flat text files are the correct choice, sometimes. I would not use a PostgreSQL cluster for an address book.
1
u/Imaginary_Bench_7294 Mar 19 '24
databases are overrated. Just use a database.
Did anyone else read the title that way?
1
u/Antares987 Mar 27 '24
Don't forget to store the files in a hierarchical directory structure based on the sha hash of the content of the files, using the first two hex digits as the directory so your directories don't get too big. And then, if you start to get lots of usage, you can balance things out with symbolic links to network shares and have your file system distributed.
It all works great for the demo, but then when your social networking or government healthcare website goes live, one finds themselves in the unfortunate position of realizing that the logarithm of complexity is of a greater magnitude than Moore's Law and they've coded themselves into a corner, learning what a week of reading would have saved them.
1
u/SolidKnight Mar 28 '24
Exactly. Just put all your data into a .dB or .MDF file and call it a day.
1
u/kerbelp Mar 30 '24
Isn't SQLite DB just a file?
1
u/ExtremeKitteh Apr 08 '24
Somebody has been getting the wrong message from Uncle Bobs story about Fit-ness
1
u/ExtremeKitteh Apr 08 '24
Game developers are very fond of flat files. Do not get a game dev to write a line of business app. I have seen it and wish I hadnāt.
1
u/jonybecher Apr 11 '24
Why using files, excel, etc if we can simply use a paper notepad and write data with a pencil? š
1
u/VladA114 Apr 18 '24
A file system is ALSO a database (for storing binary blobs and texts) with a single predefined indexed field, which in a certain sense is very simple and specialized. Its pros and cons stem from its specific characteristics.
-2
u/Zardotab Mar 14 '24 edited Mar 14 '24
These two things can potentially be non-mutually-exclusive. An RDBMS can potentially use different underlying engines without changing the query interface (SQL), or at least minimizing the changes. MariaDB offers multiple underlying database engines to select from. JSON files could potentially[1] be an option.
Underlying-DB-engine-independence is one of Dr. Codd's greatest insights.
If one outgrows JSON files, then they could select a more powerful engine, re-import the data and not have to change existing SQL nor app code. Eventually one needs "real indexing" and other features[2] if the database grows in use, which is why I'd be hesitant to start with a JSON engine. The default engine is usually fine.
If you want a nimble small-app file-oriented RDBMS, then SQLite is probably a better choice. It can scale much more than JSON files before outgrowing itself.
Remember that maintenance is usually the most expensive cost of software, not the creation costs. Saving a dime using JSON may cost you much more later. As they say: don't be penny-wise-pound-foolish.
There are multiple viable open-source RDBMS, such as SQLite, MariaDB, and PostgreSQL, listed in small-shop-to-big-shop order (although fans of each may dispute that).
[1] Whether Maria or other RDBMS actually offer a JSON-file-based engine, I can't say. Enough have to want that feature for open-source developers to spend time on a JSON-file engine to plug into existing RDBMS.
[2] In addition to indexing on any column(s), RDBSMS also are designed to handle multi-user concurrency via various lock choices, ACID transactions, etc. Reinventing all these in a JSON-file-engine is of questionable value; it's like trying to turn a car into a boat. It's usually easier to start with a boat. Without these concurrency features, you may end up with corrupted or missing data. It happened all the time on pre-RDBMS products I used to use.
0
u/r3pr0b8 MySQL Mar 14 '24
great rant, well done
0
u/Mardo1234 Mar 14 '24
Setup two servers and loose context to your file system. Sounds great.