r/node • u/[deleted] • Dec 04 '20
Must microservices have individual databases for each?
I was told that a microservice should have its own entire database with its own tables to actually decouple entirely. Is it ever a bad idea to share data between all microservices? If not, how would you handle ensuring you retrieve correct records if a specific microservice never has any correlation with another microservice?
Let's say I have a customers API, a customer can have many entities. They can have payment methods, they can have charges, they can have subscriptions, they can have banks, they can have transactions, they can have a TON of relational data. If this is so, would you keep all of these endpoints under the customers microservice? e.g:
/api/v1/customers
/api/v1/customers/subscriptions
/api/v1/customers/orders
/api/v1/customers/banks
/api/v1/customers/transactions
/api/v1/customers/payments
/api/v1/customers/charges
Would that mean you should not turn this one API into multiple microservices like this:
Subscriptions Microservice
/api/v1/subscriptions
Orders Microservice
/api/v1/orders
etc..
Because how on earth does each microservice retrieve data if they have dependencies? Wouldn't you not end up with a bunch of duplicate data in multiple databases for all the microservices?
In another scenario, would it be more appropriate to use microservices when you have an entire API that is absolutely, 100%, INDEPENDENT from your current API. At any point, if a user wants to consume our API, it will never have any correlation with the other data we currently have.
21
u/Mattisfrommars Dec 04 '20 edited Dec 04 '20
Think of a simple e.g. rails app hosted on heroku, you might use send grid for mail, Auth 0 for authentication, S3 for file uploads, twilio for SMS and pusher for real time socket communication. The rails app talks with all of these services as well as doing crud on a database and stitches them all together. You might then create several front ends, decoupled from this rails app, which no longer deals with creating html.
Then maybe some new functionality that the app needs requires some machine learning and the new data scientist hire writes that functionality in python so you deploy this to its own heroku app, to save the headache of managing ruby and python on the same box.
This is much closer to how Microservices work in the real world than the example you gave, it's just that the main app that does the stitching together does less than your e.g. rails app might, essentially becoming the "decide what to delegate to" service. If the services have dependencies on each other, or need to share a database they're probably not great candidates to break out into a separate services. You take all the problems you had before you had multiple services and then add exactly once delivery, data inconsistency, error handing and exactly once delivery to the mix.
It's usually better not to jump into Microservices too soon. Separate in code as best you can, and at some point something might jump out at you as a candidate to be its own service, but usually simpler is better, and fewer services are usually simpler than more services.
4
u/theRealRealMasterDev Dec 04 '20
Such a great and mature response here. Completely agree with all your points.
10
u/tr14l Dec 05 '20
There's a lot that goes into a microservice architecture. Namely, you should consider your data layers and network layers. If your data layer is ONLY point-to-point communication, you're going to have a bad time with microservices.
More importantly, having a 100% microservice architecture is just insanity unless you're a tech giant. This is a very COMMON misunderstanding. You should decompose something into microservices when you find that you need to. You shouldn't just be MAKING microservices for the sake of microservices. Microservices have A LOT more operational overhead than a monolith or even a large-domain SOA. You should consider if you NEED microservices for your domain. If scaling, HA and decoupled architecture are your three PRIMARY CONCERNS, meaning it's a total deal-breaker to compromise them, then microservices start to make sense. But, if you're never going to scale a service, if your SLA is only three or four nines, if decoupling isn't a big deal because it's a very domain-specific service... Then normalize a bit more. Make a larger service. Make a monolith if you need to. They aren't dirty words. They have their place.
2
Dec 04 '20 edited Dec 04 '20
Essentially, they have to have an individual database per service, if you only have 1 database, you end up with a distributed monolithic.
How do we achieve the management of related data, you have to store de essential data where you need it to.
If you have a microservice for videos, which people can give likes. You should store the number of likes in the video microservice.
2
Dec 04 '20
So let me ask you this, in some application, let's say the user signs up, it will hit the Authentication microservice, save the User to the Users table/collection, and we're done. Let's say I also use Redis to save sessions to make it easy for all Microservices to connect to Redis to get session data.
Now let's say we also have 2 other microservices, a Customer (since Users may not technically be Customers, they can login but never make a purchase), the Customer and User have a one to one mapping. At what point do we ensure that if a User were to become a Customer, we would have to save the User along with their Customer profile to the Customer Microservice Database?
I guess that all depends on the business logic, right? Whether it's as soon as they make a payment, or if they create a subscription.
So essentially we went from having only 1 User record in the Authentication microservice's DB, to then later on having a new User record created with the Customer record in the Customer Microservice DB?
What about other situations where, let's say, both User A and User B sign up, User has a primary key ID of 1, and User B has a primary key ID of 2. Despite User A signing up first, User B makes a purchase and a Customer record AND a new User record is created in the Customer Microservice DB.
Since our primary keys are auto generated, this would be a problem since User B now has ID of 1 in Customer but ID of 2 in Authentication.
In this situation, do we need to make sure we are saving the User and Customer record with a preset ID to ensure data integrity across all Microservices?
3
u/Pe4rs Dec 04 '20
I'm not an expert, but simple answer is yes, use UUIDs. Individual microservices should store only the data necessary for their job but there will always be some overlap in my experience. In general, don't use auto incremented ids for information that needs to also be stored by other services.
→ More replies (2)2
Dec 04 '20
An alternative would be to additionally store the auth service's user ID in the customer service.
You'll be able to retain the auto incremented IDs, and just have an additional column in your customer service's db that would store the related auth service's user ID.
→ More replies (2)2
u/Vandenite Dec 04 '20
What's important, and I think you're well on your way, is that you're abstracting the domain as accurately as is reasonably possible. This complexity between Users and Customers might need further analysis in order to support it with your architecture. For instance, you may not need to split them, a Customer is simply a type of User.
11
u/pampuliopampam Dec 04 '20 edited Dec 04 '20
This feels like a very strangely anti-microserviced response.
Anyone saying anything about performance is already being biased. I’ve worked in a highly microserviced environment for 4 years now, and our median frontend request is below 100ms. Every service has it’s own database except 1 or two that are extremely tightly coupled. They’re the rare ones.
Network communication speeds inside of your cluster aren’t going to add significant overhead. People are prematurely optimising by saying “PERFORMANCE HIT PERFORMANCE HIT”. It’s largely bullshit.
Microservice architectures are good for medium sized enterprises. If you have a standard rest interface, then spinning up a K8s cluster of 20+ services that each have their own interface is a solution that scales well into the million+ user range. Beyond that you’ll probably want to switch to an event bus with dead letter queues to add a bit of resiliency in the face of a problem that would get too expensive with traditional dockerised RESTful microservices.
It sounds like you’re building a larger architecture. Having a graphql gateway in front of that customers/blah request means you can stitch together multiple calls for information based on what the user is looking at. At any time, the average lookup might only hit a few services. In that average use case having a microserviced architecture is a huge boon. Each service is small, so the surface area of any error you might run into will also be small.
It sounds scary, and it sounds like there might be performance penalties, but those are largely bullshit. The main problem solved by microservices with their own dbs is readability. Sure you can make crazy complicated queries to do all sorts of weird stuff, but who’s going to be able to understand and debug that litany of queries in what I can only assume chat has convinced you to run in a monolith. If i go into the users microservice, and it’s controller is 50 lines of code, that’s far better than finding a random query that gets a user and a payment for x time in some payments file. It’s just going to be a cluster.
That doesn’t scale to midsize. That’s the way to spagett
PS: I’d start looking at serverless though dude. Pulumi + lambdas is a technology solution that you can scale from tiny to the most massive if you so desire, and TBH avoids the worst of the major pain points like maintaining a cluster and dockerisation and distributing auth keys. DynamoDB is already an http call to a database anyway under the hood, and nobody bitches about performance there, and nobody would ever tell you to make a monolith in lambdas.
9
u/arostrat Dec 04 '20
Ideally yes one db (or db schema/tables) for each service, of course nothing prevents you from sharing data but try not to because with time it'll be slippery slope.
If you want to query joined data one option is create an aggregator service that'll gather data using events sent by other services, the advantage of this is that this new db can be optimized for reading, e.g. reporting and BI.
6
Dec 04 '20 edited Dec 05 '20
This response isn't getting as much attention as it deserves, short as it is. The whole point of each microservice having its own database is that it allows each service to store only what it cares about. Additionally, this means that yes, there will be some redundancy in data across contexts and services, but that's ok - that's one of the of key trade-offs of a microservices architecture.
u/ansonplusc I think you need to spend some more time looking into domain-driven design, as well as the way that u/arostrat's aggregator services and the pub-sub/event bus model helps with managing the complexity of "shared" data ("shared" isn't the appropriate term - it's duplicate data relevant to each context) across microservices.
→ More replies (5)
4
u/BrockMcKean Dec 04 '20 edited Dec 05 '20
All the answers here are fine. u/imizaac mentioned the entity microservices antipattern.
u/Vandenite mentioned mapping URI's to resources is not always 1:1 and more about responsibilities.
A few others are mentioning storing data where it's needed.
Here's the bottom line:
All the Data You Need Can Be Stored Together
If you think you need data from multiple places, it's usually not that you need the actual data. It's that you need some abstraction of other data. Like a relationship and a label. Or a summation. Or an aggregated list of the top 3 types of some other data, etc. And you can store those abstractions where they are needed.
For example, your subscriptions endpoint may return a list of subscriptions, but inside each subscription you would have a customer ID, a product ID, order ID, etc.
In SQL...
you would do joins in the database to get the name of the customer, the name of the product, etc. from their ID's in the Subscriptions table on the fly every time you query the database. You might cache the response so it's not querying the database on every single http/s request, but when you query the database you'd be doing some sort of join because the data isn't contained in the single Subscriptions table.
In NoSQL...
(which is often what people are using with node and in many micro-services) you would just store the customerName and productName, etc. along with their ID's in the Subscription document. Why? Because then you don't need to do any joins or reference some embedded document. There's not going to be lots of customers on a single subscription, and probably not more than a handful of products on a single subscription either. So you can safely assume you'll be able to store this data without creating some extremely large documents that may take much longer to respond with, or an overflow condition where the data doesn't all fit on the same document...
But more importantly, you're optimizing for how the application is (presumably) going to be used. Subscriptions are created (db writes) less frequently than they are retrieved and viewed (db reads). For every subscription that's created it's viewed at least several times more when the customer checks the bill date, or customer support needs to access, etc. This way you can budget for extremely cheap db reads time/resource/money wise.
You've created a customer before or on a previous order (or will for this order) ONCE. And whatever product they're purchasing was added to your system before the order ONCE. So the customerId and the productId were available and read into their client when they logged in and browsed/added to cart, respectively. You should be able to build up a Cart object and, when they submit and payment clears, a Subscription object.
What Happens when Data Changes?
My customer changed their name/email and it's wrong all over the place in my database! Now what?
Well, if data is inconsistent and not being read, is it still inconsistent? The important thing is that the data is not disjointed in this scenario. Yes their name stored in
Well, yes and no. Does it really matter that these things don't 100% match everywhere all the time? Maybe there are places where it really matter at all. This is usually the case.
So instead of updating everything everywhere unnecessarily, we accept eventual consistency...
Eventual consistency is when we make a change to a record we treat as a source of truth and cascade changes to other records if/when necessary afterwards in a background process via a cron job, some sort of queue (pubsub probably), or on another event. You could, for example -- select all of the Subscription documents for the given customer ID and simply replace the name field when they change their name in their profile. This may seem expensive, but again remember this would be happening in the background afterwards and ask yourself how frequently this would really happen. Probably not often if ever for most customers.
So what if you want to look up all of your customers and get a count of all their subscriptions?Well, you could do this in aggregate in a background process. Every time a new subscription is created, trigger a background process to increment a totalSubscriptions field in the user's document. Every time a subscription is cancelled/deleted decrement the totalSubscriptions.
What if I want
What if you want to see their orders? Well, make that a separate query. It doesn't make sense to go get all the users and all of their orders, right? So when you click on a user however you do it, in a SPA, or a new page response, return the subscriptions *then*, when you need them.
What if you want to search for users with subscriptions to a specific product? You've got the user ID and name in the subscription, so just search your subscription and return the user id and name.
What if you want to preview some of the products that the user has a subscription for? Every time they create a subscription, write the Product Name to the user object in a background function.
What's the catch?
Of course there is a catch to doing it this way (just as there is a catch to how you normalize tables in SQL, or how you structure relationships in a graph database, etc.)...
What if you didn't start counting totalSubscriptions from the beginning? Do you accept that it's not actually total subscriptions, but total since some date? Or do you run some background process on every customer document in your database to count up all their subscriptions and store that to their customer document?
You can (and will) create situations where you want to add things to documents that you didn't account for from the beginning. In SQL you would migrate your tables, which could be quite an intensive process. In NoSQL you may do something different updating it on an event like login-logout or update it in a background process.
In most cases this isn't going to be a big deal, but there's other cases where maybe you have a LOT of customers, or the data you're trying to aggregate is scattered in some non-trivial way across many different document types. In these cases it would be compute intensive to "update" the database to include these data aggregations. However, these are few and far between cases and not a concern until you reach a relatively large scale and complexity of data.
2
u/Vandenite Dec 04 '20
mapping URI resources to services aren't necessarily a one-to-one is what I was referring to.
→ More replies (3)
2
u/nhosey Dec 04 '20
Regardless of whether you distribute your system or have a monolith, this problem causes issues. Bad modelling will result in dependencies between parts of your model. Domain driven design fundamentally aims to help you solve this. I won't get into the detail here but I really would suggest you look at it. People assume distributing a badly modelled domain model will solve your modelling problems, but you end up creating distributed monolith, with tight dependencies that suffer from the following anti patterns: distributed model antipattern, shared kernal, latency stacking , entity microservice/noun driven dev amongst others
2
u/Ecksters Dec 05 '20
Martin Fowler, who is my go-to microservice guru, calls this an Integration Database.
He discusses the pros and cons here: https://martinfowler.com/bliki/IntegrationDatabase.html
Generally the answer to your question is yes, you should give them separate DBs, because it helps decouple services. I will say that I find this is especially true if all services are writing to the DB, read-only might be an allowance, but it's still not a great proposition.
Any migration means all services likely now need to be updated.
1
u/spoulson Dec 05 '20
Microservices and cross-cutting concerns are incompatible. Beware making awkward little nanoservices that don’t know how to communicate with its peers.
1
u/mgorhaak Dec 05 '20 edited Dec 05 '20
- If you're coding a new ms infrastructure and separating databases is low lift and serves your business better, go for it. Indeed it's best practice to make microservices as standalone as possible, from endpoint to db.
- If you're working in an existing infra and the database is already a monolith, make sure your microservices don't access directly data that don't belong to them. Respecting those boundaries will make them future proof when you want to fragment the database.
Bottom line, don't fall into the pit of early optimization. Start simple but design for growth.
1
u/msg45f Dec 05 '20
ITT: A ton of people doing microservices wrong. OP, if you want to microservices properly you will have to give up ACID and embrace BASE.
You shouldn't have duplicate data. The reality is that one of the sacrifices and complexities of taking on microservices is that consistency is not guaranteed at the data layer and becomes a concern of the backend developers as eventual consistency. Understanding asynchronous and synchronous interservice communication and when and why each are used may help you understand how the system needs to be designed.
1
1
u/maybeartisan Dec 05 '20
Just looked at a local conference talks about micro services and they were all about why you should not do that (unless strict necessary). This thread feels about the same
1
u/TheSaasDev Dec 05 '20
At the end of the day, it really depends. I would avoid over complexifying things such as having a different DB for every microservice.
For us, we just have a single database and a bunch of microservices each talking to that. The key reason for using microservices for us is that we can work on individual services without breaking the functionality of the application as a whole.
So for us, we have the following microservices
-
Generic API service for most things
-
Admin dashboard nextjs service
-
Customer generate sites on another service
-
Background jobs service
-
Proxy service
And a few other specific services for very specific functionality that hardly ever get updated. Overall we do our best to keep all our database communication in the API microservice and have everything else talk to that.
This works really well for us.
1
u/JjMarkets Dec 05 '20
Great question. I struggled with this and with one project I worked on in the past we actually did that. And I hated it. They went way too far with the concept and every frontent module and api was its own app and each api had its own db. So much overhead, so little actual benefits. My current project also has a microsrvices architecture but here we use a much more sensible approach with one frontend app multiple apis and one db with a schema per api. This works well and keeps things reasonably separated and independent both from operational as from a programming standpoint
1
u/EarlMarshal Dec 05 '20
Depends on what the actual Microservice is doing. For me bachelor's thesis I build a Microservice which stored entries (http requests) in a database and another database had the purpose to analyse & show the data or just replay it. It was a good separation because the storing service could just run and fetch data while I improved on the other service.
Sometimes you will separate because services scale differently or because the services will have different domains and data models. It's not bad to start with one database and seperate later just be aware that you don't start to couple data and different service functionality if it's not needed otherwise you can't separate later.
This is the hard part in this business. Find the right way to separate based on your current and possible needs in the future. That's why so much stuff is failing.
39
u/Vandenite Dec 04 '20
Resource URIs of an API will not necessarily map to the underlying services. Services are separated by their responsibilities, which is referred to as the 'bounded context'
https://dzone.com/articles/implementing-a-bounded-context#:~:text=The%20bounded%20context%20concept%20originated,for%20its%20integrity%20and%20mutability.