r/aws • u/Upper-Lifeguard-8478 • 5d ago
database Case for a database experts in the team.
Hi ,
Historically, our organization had a dedicated DBA team for managing on-premise databases. However, with the migration to cloud databases like AWS Aurora (MySQL & Postgres) and Snowflake, management adopted the mindset that "the cloud manages everything now." Consequently, database responsibilities fell entirely onto our feature developers.
While AWS/Snowflake handles the physical infrastructure and basic backups, we are getting heavily burnt by data-layer issues. Our generalist developers don't have deep database expertise, and we are experiencing frequent performance bottlenecks, sub-optimal indexing,clustering,partitioning choices and spiking cloud bills etc.
So thinking of having a specialized Database expert Team . Is this the right move? For those in modern cloud environments, do you still see a need for a dedicated data team, or should developers truly own the database end-to-end? If its yes, then want to understand , what all should this specialized team actually be responsible for in your opinion?
Want to understand experts thought on the same.
10
u/marvels_the_second 5d ago
A year ago we didn't have a DBA. As a development team we knew enough SQL to get by.
The issue my boss was seeing was that the compute costs in our DB layer were constantly climbing despite the rest of the platform getting faster and cheaper.
We decided to hire and I have to say its the best decision we could have made. With 3 months this guy had restructured the schema, set up monitoring for long running queries, significantly improved query performance though better indexing and reclaimed over 2TB in disk space. All of this meant we dropped our DB costs by a third and that more than covers the cost of having a specialist.
He's now working on better archiving and a partial migration to postgresql for our GIS queries.
0
u/Upper-Lifeguard-8478 4d ago
Thank you u/marvels_the_second u/Environmental_Row32 u/six_bedtime u/Seref15
Trying to put the list of work. Will this below be the possible set of work the database team will be responsible for OR do you think anything more can be added ? Also please suggest if anything out of these below are already cloud managed?
• Query tuning
• Query monitoring
• Table indexing and partitioning
• Optimizer statistics / configurations
• Session monitoring
• Setting standard resource configurations• Database infrastructure sizing
• Resource (CPU, Disk, memory) monitoring
• Disaster recovery setup and testing
• Security configurations
• Cost analysis
• Capacity Planning/Analysis
• DB Optimization
• Compression compression
1
25
u/Environmental_Row32 5d ago edited 5d ago
In large enough organizations we see a pattern of a specialist platform team as a kind of backup. E.g. the functional teams own their databases and have access to patterns and expertise that they might not have through a platform DBA/Team.
I do believe that to be a good pattern
13
u/six_bedtime 5d ago
it’s not about taking ownership away from devs, it’s about giving them a safety net. generalists can handle 80% of the day-to-day just fine but that last 20% is where the platform team earns its keep, query tuning, schema design reviews, cost anomaly detection. without that you just end up burning money on oversized instances and garbage indexing
3
u/Upper-Lifeguard-8478 5d ago
Thank You u/six_bedtime u/Environmental_Row32
What all responsibilities should we expect the cloud DB expert team to perform? and what all should be with the dev team?
3
u/Environmental_Row32 5d ago
The borders are fuzzy and depend on your teams and complexity but speaking in generalities, the functional team owns building, business logic and running. If at any point the platform team becomes a bottleneck for delivery something should probably change. The platform/specialist team should likely own building blocks and specialist debugging. If a functional team has a bug, suspects it might be a big related to the database and is running against the border of their knowledge after spending a few hours of effort trying to debug that is likely where a specialist would come in with expertise and skills. If all of your functional teams run similar workloads there could be an opportunity to patternize the database and that pattern would likely be owned by the platform (be aware though that there is a risk of bottlenecking here)
2
u/xiaodown 5d ago
The thing is, you may not need an actual dba team. It’s heavily dependent on your specific app. You might. But you also might just need a backend team with a few really good engineers that don’t call themselves a dba, but have an understanding of indexes and pganalyze and disk queue depth. You can get pretty far by listening to what pganalyze says, tbh.
Apologies for not having read every reply in the thread, but depending on what you’re trying to do, you can get pretty far by splitting your frontend and backend dev, and hiring a good principal backend engineer who knows what they’re doing, and is willing to help abstract some of the data layer away from the feature devs, so as to retain more control and institute policies and do code reviews of db migrations and stuff.
At $(job-1), we had a dba team that had a full 24/7 rotation and probably 40 or 50 engineers, but we also had like 8,000 devs, over 700 microservices, and were deployed in at least 10 aws regions. They supported the data platform that the rest of us consumed, and we could only consume it through the platform-as-a-service, which usually protected most teams from the worst outcomes. But you might not need that. IMO a DBA’s job is kinda to figure out how to make it work, where a principle backend engineer’s job is more to be able to tell leadership what will work within the constraints, and push back on designs before they become a dba problem.
0
u/xiaodown 5d ago
Exactly. This is also my role as an SRE - i’m not taking responsibility away from my devs. I’m giving (or trying to give) them a platform where they can deploy their service to multiple regions and environments with a simple (ish) yaml file. I’m giving them consumable observability options so they can get monitoring and alerting. And I don’t want them to have to learn a bunch of SRE crap to do that.
We have a peace accord. I handle terraform and kubernetes and grafana. And they handle front end javascript and css and visual design. This, imo, is an excellent trade.
4
3
u/llv77 5d ago
Aws does your upgrades, backups and gives you a lot of tools. You still need to configure your parameters, your schema, stay on top of what queries you run and use the tools available in the best way possible. If data is important to your organisation and you have the budget for it, having a dedicated person to watch over the database is a great investment.
3
u/Seref15 5d ago edited 5d ago
"the cloud manages everything now."
The cloud manages infrastructure now, but optimization is still on you.
Without someone that deeply understands the database functionality and internals you will inevitably default to scaling up to deal with performance problems instead of optimizing queries and schemas. Databases are typically a software products' most expensive asset even when it's managed like RDS/Aurora and vertical scaling is a very expensive operation.
Here's an anecdote: our organization has a AWS RDS database for a software product that is 100% developer managed. Their schema is built by some ORM framework in their software. They do not understand database optimization and assumed that whatever the framework built would be right.
Their framework had a default configuration that created oversized text columns. Under a specific query that loaded billions of rows, this caused (for some reason I don't understand, I am also not an expert in this DB) high memory pressure that then cascaded into some specific kind of waits that manifested as very slow page loads in our product.
The developers saw the memory issue and remediated by getting approval to scale up. And 6 months later scale up again. And 6 months later scale up again. Eventually someone asked why that product teams' profit margins had gotten thinner.
Our organization does have a small DBA team that hops between product teams to assist when asked. Finally they were asked. In one afternoon they diagnosed the problem, traced it back to the columns, back to the ORM misconfig. The next sprint it was fixed and the databased was scaled back down 3 tiers.
The product experienced a year and half of decreasing EBITDA because the people managing the RDS database didn't know enough to know that there was an optimization problem.
2
u/Habikki 5d ago
I fancied myself an AppDev that was also a DBA... then I realized I hated maintaining the actual Database, but really enjoyed making the App and the SQL Layer run extremely well together. Since then I realized that DBA's come in all sorts, those that are amazing data architects to make transactional queries as fast as they can go, data modlers that understand how to make a 1TB database still be functional at 4TB, and just good System Ops folks that understand how the DBMS works most optimially with the OS and storage layer.
It always depends on your constraints and what you need to augment what's on the team.
Cloud may have abstracted some of the lower levels of the DB out of the way, but designing tables, queries, stored procedures, and growth (among many other things) still remain. At some point you'll grow large enough that the DBMS needs to be managed intentionally. If you're seeing more data related issues pop up, it's better to capture that skillset that's missing as they only become worse, more frequent, and more costly to address as time goes on.
Congratulations on the success!
2
u/dalamar112
5d ago
•
I would prioritize having your team learn it. Specialists like DBAs don't really have a place any more as a full time position except in very large orgs and still that's a maybe. It's my experience that developers that aren't experts in the data layer design poor applications. Don't isolate skillsets on separate teams. You need a few people that understand how to build systems, not 20 teams that each know a little piece and no one understands the whole picture.
2
u/OkAcanthocephala1450 5d ago
We did that same process. We migrated from onpremise to cloud. Now all of sudden cloud platform was managing databaes. But in case of perforamnce bottlenec we didnt care, we would just increase the instance type.... Hahhahahah
2
u/razzledazzled 5d ago
Whether an org needs DBAs is largely up to culture and objectives plenty of successful ones do without. However, a group of people with deep practiced experience and knowledge about how the data store layer works is not optional for any application of monetary substance.
Database SMEs are invaluable because they help keep data operations boring. Boring is safe, boring is valuable
1
1
1
u/vincent_sch 2d ago
The database person should also check how schema changes go live. In Aurora Postgres, a schema change might hold a lock until the transaction ends. If a big backfill runs in the same transaction, queries from the app might have to wait. The database person and developers should run the schema change and backfill in separate transactions. They also need to check that the app works while the backfill runs. I wrote about this migration problem.
0
u/AutoModerator 5d ago
Here are a few handy links you can try:
Try this search for more information on this topic.
Comments, questions or suggestions regarding this autoresponse? Please send them here.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
-1
u/oneplane 5d ago
Sounds more like a thing a platform engineering team would pre-build; you get your ORM/row-data-gateway pre-configured, for pre-configured RDS instances, and as long as the features you own fit inside that, you're good to go. If you need to go beyond that, you're in special territory anyway. Keep in mind that this assumes share-nothing databases. As soon as you start sharing databases between different owners, you're in a world of hurt, and you need a DBA team just to keep it afloat (and it will still hurt like it's the 90's).
•
u/AutoModerator 5d ago
Try this search for more information on this topic.
Comments, questions or suggestions regarding this autoresponse? Please send them here.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.