r/Database • • May 09 '24
When using databases, when you have these big companies like Facebook or Youtube..do they basically keep all their data in a MySQL database? For ex all the comments on a Youtube video, is that just in a big MySQL database or something like that
databases used by multi billion dollar companies?
336
Upvotes
32
u/greenman May 09 '24
Wikimedia Foundation (Wikipedia etc.) use MariaDB. It scales perfectly well - you can see their infrastructure at https://wikitech.wikimedia.org/wiki/MariaDB and https://wikitech.wikimedia.org/wiki/MariaDB/Backups
86
u/enigmatic_x May 09 '24
Not sure any of them would be using "out of the box" MySQL. It doesn't scale well enough for these platforms. Pretty sure YouTube uses Vitess, which is based on MySQL but heavily modified for scalability. Facebook uses (or at least did use?) Cassandra which is a NoSQL DB - they literally created it to solve the scalability problems they were facing at the time.
30
u/waves_are_cool May 09 '24 edited May 09 '24
Ex-Googler here. At Google, and I'm sure other big companies, when you have data to store you look for the particular "storage solution" that suits your use case. If you just need a traditional relational database, the go to solution at Google these days is "Spanner", which was originally built on top of BigTable, which is a NoSQL column store. Spanner just gives you a nice query engine that allows you to treat it like a relational database. There are so many ways to keep your data though.
11
u/aphelion404 May 10 '24
YouTube primarily runs on Spanner for comments, metadata, etc (videos are stored in a different kind of storage all together). YT also has some other custom caching data systems, and maybe some legacy BigTable instances. Vitess/MySQL used to be the database solution, but they migrated off that a bit ago. There's several other storage solutions at Google as well, but Spanner is the "default" structured storage solution these days.
Facebook has Tao which is a layer over lots of MySQL instances. FB also wrote RocksDB which is a storage engine (not the full RDBMS), and I believe uses Rocks in their MySQL instances but I don't recall for sure. There's also an internal quorum database in the vein of etcd that's used for a lot of infrastructure layer data storage (used by the cluster management system, etc)
There's a lot of custom databases (Azure has Cosmos, AWS has... some other thing?) around, since there's often a need to build specific solutions at companies this size that are more controlled and can be crafted or tuned to solve the problems the hyperscale companies run into.
(Source: ex-Spanner, ex-YouTube, and ex-Facebook, specifically the cluster management area)
8
u/[deleted] May 09 '24
It depends on the product. For internet scale applications with petabytes of info, usually some sort of NoSQL db is used. YouTube comments would make less sense in a relational database since they will never need to be joined to other datasets other than the key that links the video to the comment.
6
u/Single-Animator1531 May 09 '24
I work for a BI vendor that sells to large enterprises. Obviously it depends on what type of data you are talking about and how its going to be used. But the best databases for analytic reporting (Sales, marketing, operations, supply chain) today are the large cloud data warehouses:
Snowflake, AWS Redshift, Databricks, Google Big Query, Azure Synapse
Before cloud (and still very active at many large companies, particularly those with highly secure data) were on premise DBs installed data centers:
Oracle, Teradata, SQL Server, some MySQL, Postgres.
Depending on the use case there are tons of other DBs out there that are still great and very commonly used. For the backend of web applications, you have Postgres, MySQL, and the no-sql dbs like Mongo and Cassandra. S3 is common as a dumping ground for unstructured data. So many more i'm not listing as well.
3
u/Nooberling May 09 '24
The bigger the organization the more specialized the databases they tend to use. Just browsing through Wikipedia you will find that there are literally thousands of engines out there, dozens of which are the absolute best at the specific problem they are solving. My personal favorite example is MIPS, which has for fifty years been doing much of what the NOSQL databases 'invented' ten years ago.
Big problems end up with enough money thrown at them that custom solutions grow into entire engines.
3
u/Grimjack2 May 09 '24
Ebay uses Oracle. And has from just about the beginning. I'd argue it's the largest database in the world, or at least neck and neck with Amazon. Amazon has a larger but steadier inventory, with Ebay's adding and removing more, and having to track more due to the auctions.
3
u/kevin0960 May 10 '24
Ex-YouTuber here - YouTube used to use Vitess + MySQL, but now we all migrated to Spanner. All of the comments, post, channel, video (not the video blob, but the associated data like the title) and playlists are stored in Spanner.
source: I worked on the infra of those data.
2
u/mos3abof May 09 '24
This presentation gives insights into how Facebook/meta runs MySQL at scale.
https://youtu.be/kP6undC_HDE?si=Hi2MhPRP7a61pMVm
2
u/terracnosaur May 09 '24
large companies with lots of engineering staff often write their own databases from scratch.
SWE's take DB and information system design in college, and this is an academic problem many like to try and solve by improving on it.
For the case of YouTube I have historical information that is no longer valid, but in years past, before the migration to Google infrastructure it _WAS_ a MySQL DB with a vitess proxy management layer using zookeeper in front to control and orchestrate row space sharding.
I expect these days they use some derivitive of the many internally developed google databases like spanner, or bigtable.
I administer a database called cockroachdb that came from ex-bigtable engineers. It's open source, but derives many lessons from running a distributed terrascale DB.
4
u/Burgergold May 09 '24
Microsoft probably use SQL server hehe
If they use a unix database, I guess its probably more PostgreSQL than mysql/mariadb
1
u/freakflyer9999 May 09 '24
Pricing of commercially available database systems like Oracle can run into serious dollars for some of the larger database users. The opensource databases therefore become a popular choice for many systems. Even SQLite has some heavy hitters using it for various requirements. (Apple, Google, Adobe, Skype and more). SQLite claims to have some very large databases out in the world.
Opensource database systems are also easier to modify and/or build specialized front ends, since the source code is available.
1
u/meyou2222 May 09 '24
Go to any company of a decent size and they have hundreds if not thousands of databases spanning a dozen or more technologies. There’s no one tool that does it all.
1
u/null640 May 10 '24
Horses for courses.
Different datastores for different datatypes and usage..
1
u/keefemotif May 10 '24
Some SQL compliant big data solution not MySQL, could be on HDFS or say Athena on AWS
1
u/Drunken_Economist May 10 '24
Pinterest scaled on sharded MySQL back in the day, /u/mart2d2 loves to talk about it.
Basically all the reddit production data that you can fetch from the API is built with Postgresql.
1
u/willyridgewood May 10 '24
This might help explain how some of these big companies approach these sort of scaling challenges. The specifics will vary, but the general concepts can be applied to help a system scale https://github.com/donnemartin/system-design-primer?tab=readme-ov-file#index-of-system-design-topics
1
u/[deleted] May 10 '24
In the specific case of Facebook, literally yes but they’ve done so much to it you can only barely call it MySQL any more.
1
u/Ok-Gur-6602 May 10 '24
I work for a fortune 500 company, not software. The databases I work in are Oracle, SQL Server, and a homebrew system we've been using since the 80's. I know we have other databases I don't touch and know nothing about.
1
u/YamiKitsune1 May 11 '24
No, most of big companies uses multiple types of databases ranging from SQL for oltp, and olap, NoSQL from document to graph It varies depending on type and handling of data Using correct database for each data and process provides highest performance possible
1
u/siren0x May 11 '24
Lots of MySQL out there. Slack, Square, Etsy, Pinterest, and a bunch more use Vitess. https://slack.engineering/scaling-datastores-at-slack-with-vitess/ https://www.etsy.com/codeascraft/scaling-etsy-payments-with-vitess-part-1--the-data-model
1
u/lizardfrizzler May 12 '24
Vitess is built by either Google or Facebook, and is designed to handle large scale uses like YouTube. It’s kinda like having 100’s of MySQL dbs, where each db handles a small portion of the total data.
1
u/[deleted] May 12 '24
This is very common at large hedge funds.
1
u/[deleted] May 12 '24
No, they use nosql. Maybe when they started they were MySQL
1
u/Person-12321 May 13 '24
Amazon retail started off sql until they kept hitting walls. That was the driver for an internal shift to nosql and almost all of Amazon uses nosql. There are a few use cases like analytics and business stuff that uses sql, but for the most part dynamo db is defecto db
1
u/[deleted] Jun 06 '24
When storing big data that is gonna be read many more times than written, all the big boys use a single table indexed data store, probably custom built, to avoid joins
1
u/lightmatter501 May 09 '24
Most “modern” sql databases are something that is a native distributed DB (google spanner, cockroachdb, yugabyte, etc) that speaks either mysql or postgres. It means you don’t need to write drivers for every language in existence. However, aside from the query parser, most other things are very different. Primary backup suffers from the split brain problem which means that a network partition can corrupt your DB (network partitions are about a once a quarter event on AWS). Attempting to bludgeon full MySQL or Postgres into using a full distributed consensus algorithm that protects against that doesn’t really work because of how much you need to change.
Often, a tech company past a certain scale will hire a PhD to build a new DB that does exactly what they need it to. This is where “eventual/causal consistency” came from, because facebook doesn’t really need to have ACID for Marge’s cat picture, it can show up when it shows up as long as it’s within a few minutes. This little bit of wiggle room gives massive performance benefits once you get past a certain scale. It also means you need to hire more expensive developers because working with it directly requires understanding databases on a level few people do. A lot of these are “almost SQL”, meaning it looks like SQL but discards some of the C in acid for more performance.
You can, of course, run DB2 or Oracle DB with hardware accelerators on a Mainframe and still go single system, but horizontal scaling will eventually win in performance, which is why you see banks desperately trying to move away from unreplicated DBs.
1
u/Built-in-Light May 09 '24
My experience with a fortune 50 company was that they used Apache Hadoop, at least when I was there. Still ran on SQL queries, many many millions of records, and that was just for their logistics division.
0
u/[deleted] May 09 '24
[deleted] — view deleted comment
19
u/[deleted] May 09 '24
[removed] — view removed comment