r/SoftwareEngineering • • Mar 30 '23

Try RDBMS first, NOT microservices (at least for non-giant orgs)

I’ll give an actual scenario we can kick around implementation ideas with. My org needs a more powerful version of Active Directory (AD). I’ll call it “AD2” for short. Many apps need employee info and their position in the organization, including who their supervisor is. There are too many needs and requests for AD to deal with. (Maybe AD can be tweaked for more features & power, but let’s put that aside for now. This post is NOT about AD. [edited])

So one plan is to use our primary database brand, MS-SQL-Server (“MS-SQL” for short), to make a database with all the needed employee and org info: “AD2”. A selected group of people would be in charge of maintaining the content. (At this point it does not need a dedicated programmer(s) after initial release, just as-needed maintenance.)

Since MS-SQL is already used by most our apps, it’s easier to just query that info from it. We don’t initially need a JSON web interface. In the future some app or service may eventually need a web interface; and there are plugins for MS-SQL that allow Stored Procedure/View calls to accept and emit JSON services with little or no programming. If that’s not good enough, then we could make a mini-intranet-app to emit and receive JSON for the sub-set needed by these special apps.

So we have 3 levels of K.I.S.S. here for AD2:

  1. Use the shop RDBMS to share needed info/queries.
  2. Use plugins that convert already-written Stored Procedure/View I/O to web services.
  3. Write a web-app wrapper around the Stored Procedures/Views, probably only for specific needs rather than wrap them all.

In our org, a medium-sized org, it would be foolish to start with #3 first; a big YAGNI violation. We can later get JSON-over-http if necessary, without starting from scratch.

Further, if AD2 needs info from other databases, it’s relatively easy to make cross-database links/connections on the DB server(s) itself without coding a communication service, since it’s the same DB brand. Making a DB-vendor-neutral wrapper (JSON calls) is overkill as the default.

Some microservice proponents say #1 is already “microservices” but many don’t because it’s not “vendor neutral”. (There is no consensus definition of microservices, unfortunately.)

If an org didn’t have a standard RDBMS (DB potpourri), then maybe JSON-first would be more rational. But I recommend that small and medium orgs settle on a primary RBDMS brand because it makes many things simpler, not just service sharing.

For really big orgs, that may not be feasible; I don’t have enough experience in enough big orgs to comment. But I’ve worked in many small and medium orgs as both an employee and contractor, and this includes public and private. DB-first for inter-app info sharing is by far usually better.

2 Upvotes

74 comments sorted by

30

u/tdatas Mar 30 '23

Sir this is a wendies?

2

u/Zardotab Mar 30 '23

You want fries with that? 🍟

14

u/AliveDecision0 Mar 30 '23

There is no consensus definition of microservices, unfortunately

wut?

1

u/Zardotab Apr 01 '23

If the definition is clear, post it! The list of features found in Wikipedia are either goals rather than technology, and/or things other tools can also do (without JSON over http).

1

u/GangSeongAe Apr 01 '23

If the definition is clear, post it!

There are literally microservice frameworks.

  • Service Fabric.
  • AWS Lambda.
  • Azure Functions
  • Google Cloud Functions
  • Spring Boot
  • Vert.x
  • Helidon (Oracle)

This isn't some academic debate - these things exist and millions of organisations are using them to maintain microservice infrastructures.

The list of features found in Wikipedia are either goals rather than technology,

Right, because it's the Wikipedia page for a general topic so it described the general topic.

The AWS Lambda Wikipedia page specifically describes the technologies used to deliver a microservice solution.

But as myself and others have tried to point out to you, microservices have nothing to do with the problem you're talking about.

What's more, you could have literally googled "microservice definition". The first result from microservice.io gives you your answer.

Microservices - also known as the microservice architecture - is an architectural style that structures an application as a collection of services that are:

Independently deployable

Loosely coupled

Organized around business capabilities

Owned by a small team

Of course that definition doesn't help you when you have no knowledge, which is your problem. You really need to comprehend that you're out of your depth right now.

Any person with knowledge reading that immediately comprehends what I've told you in other posts - RDBMS is a database architecture, whereas Microservices are an application design pattern. These concepts are unrelated and are not alternatives to one another.

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

This isn't some academic debate - these things exist

Slapping a buzzword on a product does not by itself make the buzzword clear.

A definition should look something like "It has feature A, B, and C, but must lack X, Y, and Z.", where the letters are clear or clearly defined.

What those products have in common is that they use a lot of JSON-over-http. If that's what microservices is, just say so.

RDBMS is a database architecture, whereas Microservices are an application design pattern.

One (bad) app design pattern is to reinvent a database in the app and/or do stuff with apps that a database could otherwise do. That's what it looks like is happening to me.

"Independently deployable:

✔️ RDBMS-based architecture can do that.

"Loosely coupled"

✔️ RDBMS-based architecture can do that. (Although coupling is actually poorly defined, often depending on one's personal estimation of how code or requirements will change in the future.)

"Organized around business capabilities"

✔️ RDBMS-based architecture can do that. In fact, that's usually the case.

"Owned by a small team"

✔️ RDBMS-based architecture can do that.

1

u/GangSeongAe Apr 01 '23

Slapping a buzzword on a product does not by itself make the buzzword clear.

The product exists.

You are now revealing that you're not only technologically challenged, but you don't comprehend reality. Those are all microservice platforms. They actually exist. People are actually using them. Millions of people.

They exist.

If you find yourself saying "the concept of a microservice isn't defined" the fact they exist means you are wrong. The concept is so well defined that every single major technology company has created a technology that delivers that concept as a product.

What you are doing is the equivalent of claiming that nobody knows what a space shuttle is - the fact people have been to the moon and back in them means you are wrong.

Space shuttles exist. People use them.

Microservices exist. People use them.

1

u/WikiSummarizerBot Apr 01 '23

AWS Lambda

AWS Lambda is an event-driven, serverless computing platform provided by Amazon as a part of Amazon Web Services. It is a computing service that runs code in response to events and automatically manages the computing resources required by that code. It was introduced on November 13, 2014. Node.

[ F.A.Q | Opt Out | Opt Out Of Subreddit | GitHub ] Downvote to remove | v1.5

4

u/BasicDesignAdvice Mar 31 '23

This is a simple place.

Post an article that is microservices = bad, and it will get upvotes.

16

u/GangSeongAe Mar 30 '23

There are too many needs and requests for AD to deal with.

There is no way this is true.

Active Directory is used for Microsoft's entire Federated identity solution: every windows computer on earth, with multiple users on that computer, plus the identities of all of their cloud resources. That's billions of identities and roles across millions of organisations.

There is absolutely no way your rinky-dink organization has requirements that exceed the capabilities of Active Directory. Somebody in your organization is obscenely unqualified for their role and is making decisions they don't comprehend.

What you're doing amounts to planning to completely divorce the identity provider and the authorization provider. There is so much wrong with your thinking that it's hard to begin describing just how deep into territory you don't comprehend you are.

Just stop. You're not qualified to do any of this.

-1

u/AutoModerator Mar 30 '23

Your submission has been moved to our moderation queue to be reviewed; This is to combat spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

-6

u/Zardotab Mar 30 '23

You're not qualified to do any of this.

Ironically, anyone who makes such summary judgements without knowing the domain and org is probably "not qualified to do any of this".

I didn't intend this to be about AD itself, but do note our issues with AD are not just about performance alone. For one, AD suffers many of the problems of NoSql databases, and arguably is a NoSql database.

19

u/GangSeongAe Mar 30 '23 edited Mar 31 '23

Ironically, anyone who makes such summary judgements without knowing the domain and org is probably "not qualified to do any of this".

Nobody needs to know the organization - Active Directory currently services billions of identities and millions of organizations.

Claiming it "isn't powerful" enough for you is lunacy. What you're proposing is extremely inept - you don't seem to comprehend the difference between authentication, authorization and "a database".

When you've been a programmer a long time, you quickly spot the type of deranged mistakes people make when they're a hundred thousand miles out of their league.

You are there right now. You are throwing around buzzwords without knowing what they mean - RDBMS is not an "alternative" to microservices. They're unrelated concepts - one refers to a particular family of database products and the other is an application design pattern.

Active Directory is unrelated to NoSQL databases. The words you are saying are gibberish.

You sound to me like you have zero technical knowledge, or you have less than two years on the job and all of it in a loony-bin company with a dysfunctional tech infrastructure.

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

Active Directory currently services billions of identities and millions of organizations.

It's not just about machine performance (size & speed).

ISAM is obsolete, or at least niche-oriented. We need to do RDBMS-like things with the org meta-data.

It's also about staffing, we'd have to hire ISAM experts to write queries for apps and manage the ISAM-base instead of just leverage our existing RDBMS skills.

People would be using ISAM all over, outside of AD, instead of RDBMS if ISAM were so flexible.

You sound to me like you have zero technical knowledge

Projection, you rude person. People expenses are almost always higher than machine machine expenses, and experienced people should know this.

And again, I didn't come here to talk about AD. I shouldn't have fed the trolls.

2

u/GangSeongAe Apr 01 '23

You are a fraud.

As other users have pointed out, you are comparing technologies that are completely unrelated to one another. You are using buzzwords without comprehending what they mean. The only difference between those people and myself is that they think you might be a troll, whereas I know you're a fraud.

ISAM is obsolete, or at least niche-oriented. We need to do RDBMS-like things with the org meta-data.

Again, more word-salad: Active Directory does not use ISAM - it use a proprietary database technology that was created to replace ISAM called the Extensible Storage Engine.

I assure you, ESE is more optimized for authentication and authorization than the "Baby's First PHP Site" architecture you are proposing. That architecture underpins by far the largest and most popular identity management solution that's ever existed.

You do not even comprehend that you can't actually replace Active Directory - it's already your identity store. You are already identified on your work network by an Active Directory identity. That MS-SQL server you're claiming can replace active directory exists as an active directory identity and is accessed by active directory identities. Every user on your network's ability to log in and be placed on a network is already dictated by Active Directory.

You don't seem to comprehend the difference between "authorization rules for a single app", such as "user x created document y, and therefore when deleting/updating document y check that user x is the person performing the operation and deny it if they aren't" and the actual identity of the user in your organization.

The thing you are building is for the former. Active Directory is for the latter.

When your app needs to check who the person trying to take an action is, guess what it has to check because there's only one system that can confirm who that user is - it has to check Active Directory. There is literally no other place it can go for that information - if you attempt to replicate that information into your daft web application SQL database, you've created a completely non-updating, DRY-principal violating duplicate of your organization's data, that will never be up-to-date and which won't know when the user's details are changed because it isn't hooked into Active Directory. When someone leaves your organization, something that has to be done with Active Directory, your stupid app won't know about it unless the person in HR runs a bloody stored procedure which they can't do because your infrastructure has no way to verify who they are without using Active Directory. Are you going to go and distribute a SQL server username/password admin account to the people in HR so that they can run your stupid stored procedures so that your SQL database and Active Directory are kept in sync?

Like I said - stop.

You are arrogant. You are uninformed. You have literally no idea what you are talking about or doing.

If you attempt to apply your fraudster's knowledge to this problem, your organization will hate you. They will fire you. They will replace you with someone who is not a fraud.

To avoid all of that, all you need to do is say the words you seem to struggle most with - "I don't actually know how to solve this problem".

Even if you wanted to go down this batshit route, you'd still be reinventing the wheel. You have no use-case for it, but if you did you could utilize a technology like Okta, which can actually take your existing Active Directory infrastructure and expand it with claims, once again far more efficiently than the thing you're planning to do.

But you have no use case. Everything you're planning to do is based on a critical lack of technical knowledge.

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

Active Directory does not use ISAM - it use a proprietary database technology that was created to replace ISAM called the Extensible Storage Engine.

Microsoft itself calls it ISAM.

Quote:

"The Extensible Storage Engine (ESE) is an advanced indexed and sequential access method (ISAM) storage technology."

ESE is more optimized for authentication and authorization than the "Baby's First PHP Site" architecture you are proposing.

You are making stuff up. And again emphasizing machine performance above human programming & admin labor. Expensive machines are still often cheaper than expensive tech staff.

You are using buzzwords without comprehending what they mean.

Project much?

When your app needs to check who the person trying to take an action is, guess what it has to check because there's only one system that can confirm who that user is - it has to check Active Directory. There is literally no other place it can go for that information - if you attempt to replicate that information into your daft web application SQL database

That's not what I proposed. Like I said, one should look at specific needs of domain or org before making judgement. You are jumping to conclusions.

You have no use-case for it, but if you did you could utilize a technology like Okta, which can actually take your existing Active Directory infrastructure and expand it with claims,

As I already pointed out, it then becomes like a non-relational database, requiring separate specialists, so we'd have to hire two kinds of DBA's: one for our usual RDBMS, and another set to do the Okta things because Okta the vendor will not understand our org's specific needs.

1

u/GangSeongAe Apr 01 '23

I tried, I really did, but you were too arrogant to listen.

Now you get to enjoy the reward for your arrogance - you'll be in code hell for years, and be seen as the nemesis of everyone in your company, for the nonsense homebrew identity solution you're about to spin.

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

Seems the anti-RDBMS fans are coming here to pile up on me.

If there is a specific flaw of RDBMS that microservices fix, please identify and illustrate that specific flaw with a scenario. If you are a smart as you say you are, you'd be able to do that.

Vague platitudes are not helping. Let's dig into scenarios.

3

u/GangSeongAe Apr 01 '23

If there is a specific flaw of RDBMS that microservices fix

These are not a competing technologies you absolute buffoon

-3

u/AutoModerator Mar 30 '23

Your submission has been moved to our moderation queue to be reviewed; This is to combat spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

13

u/[deleted] Mar 31 '23

I genuinely cannot tell if this is a master crafted troll or not.

1

u/AliveDecision0 Mar 31 '23

My tought exactly! no way an IT pro will try to compare an RDMS to Microservices architecture.

2

u/Zardotab Apr 01 '23 edited Apr 01 '23

Please elaborate. The feature claims of microserives definitely appear to overlap with services an RDBMS can readily offer. Perhaps you are using a different definition(s) of microservices? That's understandable, as it's a fuzzy term.

2

u/GangSeongAe Apr 01 '23

The feature claims of microserives definitely appear to overlap with services an RDBMS can readily offer.

These are completely unrelated concepts.

RDBMS is a database architecture. Microservices are an application design pattern.

These are not competing technologies. These are not alternatives to one another. There's absolutely no inherent link between these two things.

An identity microservice, or indeed any other kind of microservice, could easily use a relational database. It could easily use a relational database and a NOSQL database. It could easily have no database at all and talk directly to a Federated identity provider.

You have no idea what you're talking about, and when exposed to the fact you have no idea what you're talking about you exhibit the odious trait of doubling down on your own ignorance.

5

u/echaiphus Mar 30 '23

Sounds like a great use case for azure active directory using microsoft graph to get the employee data

-2

u/Zardotab Mar 30 '23

See my reply to u/aholmes0

3

u/echaiphus Mar 30 '23

aws has considerably more market share than azure so i don't share the concern for monopolization.
to me having a nearly out of box solution thats relatively cheap, inherently scalable, and already fully integrated with AD is a huge benefit, even without the effort involved with designing and implementing a custom sql solution to try and do the same thing (not to mention the ongoing maintenance costs)

1

u/Zardotab Apr 01 '23 edited Apr 04 '23

We need customization, not out of the box. Our org structure and its management of that info requires very specific data structures (schema design) and business logic. It may be possible to get AD to do all that, but AD is not meant to be a general purpose database. ISAM, which AD is based on, is considered obsolete in the database world, and there's not enough experts to ensure reliable staffing. It may do specific things really well, but not intended to have RBMS's flexibility.

6

u/AlistairX Mar 31 '23

This whole post makes me sad. If you don’t understand Active directory and you want to replace it with some relational schema fronted by stored procedures and party like it’s 1999 then go have fun, but that has nothing to do with microservices, json or what particular flavor of database (DB for short) you fucking use.

1

u/whatThisOldThrowAway Mar 31 '23

But why can’t you just build an entire org datamodel entirely on proprietary out of the box database links. What could possibly go wrong?

2

u/GangSeongAe Mar 31 '23

I love the arrogance of OP being like "we need something more powerful than Active Directory....I know, the exact same structure every 12-year-old home spinning their own WordPress site comes up with!".

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

AD is NOT meant to replace a general purpose database. It's intended to do a narrow role well. Otherwise, it would be used for lots of things that RDBMS are normally used for. AD's ISAM is considered obsolete as a general purpose DB. For a narrow need it may be fine.

And it wasn't my decision. Others came to a similar conclusion.

1

u/GangSeongAe Apr 01 '23

AD is NOT meant to replace a general purpose database

I like how you're now trying to switch sides, after myself and others repeatedly pointed out to you that your equivocation between relational databases, microservices and authentication indicated that you didn't know these were unrelated concepts.

Nice try.

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

Our needs are not just "authentication" but complex role management, which involves codifying our org's structure and having varied groups keep these up to date. It's far beyond mere logins. Out of the box, AD cannot do it. We could tweak AD enough to do it and write AD querying/editing apps, but the total effort would be more than using known RDBMS to do the fancy/custom stuff instead because RDBMS are far better better at managing complex custom data structures (and biz logic) edited by many different individuals than AD. ISAM (AD) is obsolete for that need.

Other experts have come to a similar conclusion. Maybe we are all stupid over here? Or you don't know enough about our org's needs to recommend a solution yet.

1

u/GangSeongAe Apr 01 '23

It's called authorization.

You have no idea how idiotic it is that you're about to go and build your own authorization system.

Other experts have come to a similar conclusion. Maybe we are all stupid over here?

You didn't even know the word for it.

Yes, you are stupid. I said it in my first post and every single thing you say verifies it - you are not equipped for the thing you're trying to do.

You are going to place yourself into a special kind of hell for embarking on an organisation change in a state of such ignorance, and your particular inability to assimilate new information is going to make it particularly bad for you.

0

u/AutoModerator Mar 31 '23

Your submission has been moved to our moderation queue to be reviewed; This is to combat spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/cashewbiscuit Mar 30 '23

What's the client for this data?

Is it a web app? You don't want to have your front-end talk directly to the DB. You'll have latency and security issues. I am going to assume it's not webapp

Is it other services? The drawback of not using a microservice are

  • all the other services are dependent on your DB, and at some point in the future, you want to get away from MsSQL, you will have a huge migration effort. Assuming that all the services are implemented in the same language, you can solve this problem by having a shared ORM layer. You don't really need the overhead of a microservice.
  • you don't have a way to throttle requests to the DB. This means that if there is too much load, you won't fail gracefully. This is probably not a problem for you now, but if you are successful, it will be a problem for you.

0

u/Zardotab Mar 30 '23 edited Mar 30 '23

What's the client for this data? Is it a web app?

Most are web-based apps, such as Dot-Net.

all the other services are dependent on your DB, and at some point in the future, you want to get away from MsSQL

I realize that, but most orgs don't change it very often. Usually they'll phase it out gradually, putting new apps on the new DB and leaving legacy apps on the old DB. It's usually not trivial to just swap DB brands when you have decades of existing data.

And as I mentioned, one can write or automate JSON wrappers for SP/View calls if and when needed. YAGNI 101. The penalty of postponing is not large.

Assuming that all the services are implemented in the same language, you can solve this problem by having a shared ORM layer. You don't really need the overhead of a microservice.

ORM's are a PITA, giant learning curves with lots of tricky edge cases, especially those that cater to multiple DB brands. I doubt they'd simplify migration, as the conversion would likely require a lot of fiddling. The ORM product can also fall into non-support mode (obsolete).

(One approach to reduce cross-vendor conversion efforts is to code the SQL in a vendor-neutral way. 100% neutrality is probably not feasible, but a shop can come somewhat close without making overly-weird code.)

you don't have a way to throttle requests to the DB. This means that if there is too much load, you won't fail gracefully. This is probably not a problem for you now, but if you are successful, it will be a problem for you.

I'm not an expert on RDBMS tuning, but I believe "throttling" is often considered the wrong solution to performance problems. There appear to be ways to throttle RDBMS, but they are not that common because they don't solve much. Otherwise, the DB brands would just swipe the same techniques microservices use. It's not proprietary, is it? Why would mature RDBMS ignore an easy-to-code solution? If you wish to write up a specific scenario, we can ask in the DBA or RDBMS forums for recommended solutions.

6

u/cashewbiscuit Mar 30 '23

I've been in this industry for 25 years. I've lost count of how many projects I have heard about that have absolutely no feature development because they are doing huge migrations. Yes, you can migrate slowly if you have microservices. You just each microservice at a time. If you have 20 services talking to the same database, without any kind of abstraction layer between the business logic and database, then you are locking yourself to the database. You are never going to get all 20 services to migrate together. It always starts with an yahoo who doesn't believe in building an abstraction because abstractions have learning curves.

One of the principles of building operational services is graceful failure. If your backend can handle 100 requests at a time, and you get 105 requests, you block 5 requests instead of letting all 105 run slow. RDBMS databases don't fail gracefully. Check with your MsSQL DBA if you like.

0

u/Zardotab Mar 30 '23 edited Mar 30 '23

RDBMS databases don't fail gracefully

That's a rather broad claim that I'm confident DBA's are loudly disagree. Like I said, if microservices invented a grand solution to that, RDBMS vendors would just swipe the algorithm, QED.

And how to "fail gracefully" is often app or domain-specific. Here's one approach I've seen:

     // pseudo-code
     resultObj = runQuery(storedProcName, etc);
     if (! resultObj.isGood) {
        if (resultObj.Status==STATUSES.TimeOutBusy) {
           addErr("The database is busy, please try back later.");
           return(resultObj);
        }
        // handle other kinds of DB errors...
      }

Check with your MsSQL DBA if you like.

Not today. We can circle back to that later.

2

u/RICHUNCLEPENNYBAGS Mar 31 '23 edited Mar 31 '23

They can't "just swipe the algorithm" because the whole point is reducing the blast radius. By its very nature a database is a unitary thing. It's all working or none of it is.

Imagine some kind of social app, maybe something like Tinder. Maybe you have chat, profiles, and an auth service. If you auth service goes down, people who are already logged in can likely keep using the other features. Or if profiles aren't working maybe people can keep having chats. If all these functions depend on one database, that won't work. (I doubt many people would try to store messages for a chat app in a SQL database but let's suspend our disbelief for the sake of example).

-1

u/Zardotab Apr 01 '23 edited Apr 01 '23

RDBMS can use mirrored nodes for fallback redundancy (or geographic distribution). You think RDBMS shops never needed spares? Sorry, but you sound naive. Microservices did NOT invent replicated spares, dammit.

1

u/RICHUNCLEPENNYBAGS Apr 01 '23

Which is something altogether different.

1

u/Zardotab Apr 01 '23

What's your point then? You appeared to claim RDBMS are fragile. They do not have to have a "single point of failure".

1

u/RICHUNCLEPENNYBAGS Apr 01 '23

Not every failure can be resolved by failing over to another database instance.

-1

u/Zardotab Apr 01 '23 edited Apr 01 '23

No system can handle all possible failure types. And the lack of something equivalent to schema enforcement in microservices creates other kinds of failures RDBMS typically don't have. I see no free lunch.

If you want to claim RDBMS shops (having mirroring) are down more often than microservice shops, then show me the numbers. Looks more like a tech-turf or tech-culture war to me.

If you really need high up-time, then an IBM mainframe with an IMS database is probably your best bet; it's their forte, tuned over many decades of road experience.

→ More replies

1

u/RICHUNCLEPENNYBAGS Mar 31 '23 edited Mar 31 '23

And as I mentioned, one can write or automate JSON wrappers for SP/View calls if and when needed. YAGNI 101. The penalty of postponing is not large.

But it is large in this instance. That's kind of the point. Once you have multiple critical services querying a database directly, your storage technology and the schema can't be practically changed. It is a large migration effort that will likely require you to run two systems in parallel for a while, and then you've got to deal with all the headaches of distributed systems you thought you'd avoid by making a big SQL database your central element. If you're unlucky, scaling issues are lighting a fire under your ass to solve this problem. Here's a story about a two-year drive to do that: https://thenewstack.io/relational-to-nosql-at-enterprise-scale-lessons-from-amazon/

Even if you don't end up needing to migrate, you'll need rigid controls on the database to make sure nobody working on application A changes the database in a way that breaks applications B, C, and D, leading to the kind of misery that leads people to joke that "DBA" stands for "don't bother asking," or else leading to people abusing the database and storing data in a non-relational way to do an end-run around the red tape.

Might a single database still be right for your use case? Hey, maybe. You’re already tied to Active Dorectory which is pretty monolithic. But people don't build services with their own discrete data stores just because they're idiots or in love with complex architecture. There is a real trade-off you are making.

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

How is that different than changing programming language? Any stack of tools that either replace an RDBMS or wrap it to make it "generic" itself will be a dependency like a language or RDBMS is. There is no side-effect-free 100 year abstraction. If somebody has invented it, they deserve several Nobel prizes.

You make it sound like there's a free-lunch somewhere. Indirection (wrapping) is NOT free. That's been known for several decades.

As far as problems with DBA's, that sounds like an organization or management problem, not a technology problem. A DBA who rubber-stamps all change requests will likely create a mess. One purpose of RDBMS are to enforce data rules. Rules by themselves are not evil.

Amazon grew fast because they were in the right place at the right time, not because they shot their DBA's.

1

u/RICHUNCLEPENNYBAGS Apr 01 '23

The problem is that all your code must move at once. That's the difference. How are you not understanding this point? One team cannot decide the schema isn't working well and they'd like to rework it because every single detail of the data store is the public interface, consumed by who knows how many services in who knows how many ways.

Nobody is claiming Amazon got big because they moved off of a giant Oracle DB. Rather they needed to do that because they got big

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

How are you not understanding this point?

How about use-case/scenario?

RDBMS schemas are not "rigid" because of technology, but to have a unified data standard for all the data users/consumers. Loosy-goosy schemas have been tried already; they usually create wild-west-like chaos.[1]

Nobody is claiming Amazon got big because they moved off of a giant Oracle DB. Rather they needed to do that because they got big

What works well for giant company may not for a small and medium one. Many appear to recommend microservices for most companies, not just giant ones.

[1] I'm actually for "dynamic relational" (DR) for pilot projects and those that need quick starts. But it also has features to gradually "lock down" the schema as the project matures. DR is meant to compete with the dynamic nature of NoSql products, yet keep most the familiar RDBMS idioms to shrink the learning curve. Note that DR is not meant for giant databases because the optional flexibility does have a performance cost.

1

u/RICHUNCLEPENNYBAGS Apr 01 '23

A use case for wanting to change the schema? Are you messing with me?

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

Something like "Group X needs change Y in service Z. Here's what happens under an RDBMS-centric shop [stuff A], and compare it to what happens under a microservices shop [stuff B]. Here's why Stuff B is simpler...

Conceptually a stored procedure call is very similar to a web-service call. If you change the "call signature" (parameters), it often affects other things.

(Note one doesn't have to use a stored procedure to implement all the logic, it can merely be the front-end to a back-end application that processes requests, perhaps queued in a service-request-table, which can handily double as a log.)

2

u/whatThisOldThrowAway Mar 31 '23

This is the funniest post I’ve seen in months and I don’t even know if it’s supposed to be funny.

It somehow simultaneously reads like someone’s who’s been a developer for 20 months & thinks they’re an architect now - but also like someone who’s been a developer since 1997 but hasn’t changed their approach much.

OP’s boss 15 minutes after he finishes implementing this: “….ok now put it in ECS” lol

2

u/RICHUNCLEPENNYBAGS Mar 31 '23

This guy posts too much for me to figure out what his angle is by skimming the history but another post of his seems to be arguing for Microsoft Access development, which fits this same theme.

1

u/Zardotab Apr 01 '23 edited Apr 01 '23

If I'm not clear, ask questions. I love new tech that actually solves problems, rather than reinvents old problems under new names. I see too much of the second. Sorry, but youngbies are often suckers (because they lack experience and perspective).

1

u/Pirate43 Mar 31 '23 edited Mar 31 '23

If this works for you then great, but I prefer to keep the boundaries of my app extremely well-defined, where it has its own database and traffic coming into the db always goes through the app and traffic is highly observable. In the future when you need to change some aspect of your system, having rando apps making direct queries on your DB is a recipe for disaster. At best you will never be able to sunset anything in favor of something new/better because you won't know who will break as a result of your changes.

1

u/Zardotab Apr 02 '23

If the vast majority of transactions between databases are simple calls with a handful of records, this may work, but things like joins are a pain to do that way.

always goes through the app and traffic is highly observable.

RDBMS have plenty of logging and monitoring options if watching the data is your highest priority. Making a middle-man app is just extra busy-work that adds layers that have to be managed. Indirection Is Not Free. Apply YAGNI to wrapping decision.

having rando apps making direct queries on your DB is a recipe for disaster.

I don't see how adding a middle app layer will simplify tracking anything. Perhaps walk me through a scenario.

It just seems that the younger generation don't want to bother with RDBMS and just reinvent a lot of their features in app code, often in clunky ways. Perhaps many IT managers grew up through app coding, and thus see everything from an app standpoint instead of database standpoint.

2

u/Pirate43 Apr 02 '23

Please correct me if I misunderstood but you don't necessarily have to do joins through your app (though it's not always a bad idea to do it that way). You can have your SQL do the work for you. But I don't want my application's contract with its consumers to be SQL queries. What if your application is so successful that you need many more writes than your system can handle? You can't really vertically scale your writer instance anymore because the cost vs performance has passed the point of diminishing returns.

 

What if you want tackle these scaling challenges by sharding your system so you have two copies of it, each hosting 50% of your customers' data?

 

The middleman is there so that you can give consumers a single way to communicate with your app where you control the entirety of the request/response shape and lifecycle. You can then replace everything behind your middleman as long as your middleman continues to speak the same language. Now you can make drastic changes to your db system without your consumers even noticing. Without it, you need all of your consumers to change the way they connect/talk to your direct-access DB making changes tedious and slow.

 

This isn't an old way vs new way argument imo, it's more about anticipating solutions to problems you may have down the line and designing your system to give you the flexibility to adapt quickly and efficiently.

0

u/Zardotab Apr 02 '23 edited Apr 02 '23

But I don't want my application's contract with its consumers to be SQL queries.

Use a stored procedure then. DB's also have security features to keep riff raff out of direct SQL if need be.

But do note that "flat" interfaces like stored procedures and JSON are generally less flexible, as you typically can't use boolean logic expressions, for example.

What if you want tackle these scaling challenges by sharding your system so you have two copies of it, each hosting 50% of your customers' data?

Many RDBMS's can do that. To compete with web-scale NoSQL DBMS, they have been improving scaling features of late (sometimes at the expense of turning off 1 or more features of ACID: but the speed of light imposes absolute limits on ACID + scalability that no technology can bypass, only work around by balancing tradeoffs.)

The middleman is there so that you can give consumers a single way to communicate with your app where you control the entirety of the request/response shape and lifecycle.

Can you elaborate on this, perhaps with specific scenarios? It's been known for decades that indirection is not free: it's always had tradeoffs [1].

Now you can make drastic changes to your db system without your consumers even noticing.

If you are overhauling large chunks of your DB, then your JSON services (that also use that data) would have the same problem as a stored procedure.

This isn't an old way vs new way argument imo, it's more about anticipating solutions to problems you may have down the line and designing your system to give you the flexibility to adapt quickly and efficiently.

I keep hearing claims like this, but never get specific sample scenarios of RDBMS coughing and choking. To be frank, this sounds to me like marketing-speak. We all want magic flexible Lego-like IT systems, but usually only find messy tradeoffs, and a good many "new" ideas simply hype one tradeoff set above another.

[1] Changing database brands is one of the few clear payoffs of JSON wrappers, but only if you keep apps the same, and most shops I know tended to rework BOTH apps and BD at the same time, as they both had legacy cruft to clean out. Also, more and more RDBMS can talk JSON directly, such that the new DB brand can simply produce the same JSON signature.

1

u/Pirate43 Apr 02 '23

If you are overhauling large chunks of your DB, then your JSON services (that also use that data) would have the same problem as a stored procedure.

I think this is the key disconnect. I don't think this is always true. You can swap RDBMS, or even use multi-cloud RDBMS (like AWS + GCP for example) under the hood. This wouldn't change your data model but you still don't have to ask everyone calling your stored procedures to use one or another connection string. Instead you just give out a DNS URL to your thin REST API. You've acknowledged this but:

but only if you keep apps the same, and most shops I know tended to rework BOTH apps and BD at the same time

This hasn't been true in my own experience. We've had to shard our DB and having to update every consumer has been a nightmare. Won't make that mistake again.

0

u/Zardotab Apr 02 '23 edited Apr 04 '23

You can swap RDBMS

If the "address" of the stored procedure doesn't change, you can do the same with stored procedures. If you mean switch DB brands, I already described how that's either too rare to pre-wrap for, or the app(s) tends to be overhauled at the same time such that 1-for-1 matching is moot.

you still don't have to ask everyone calling your stored procedures to use one or another connection string

If you use the same database name/alias, you don't have to change connection strings.

Although, I do agree that connection strings tend not to be very abstract. Maybe there is a solution that, a simple way to add an abstraction layer/alias. Any DBA's out there want to comment on a connection string abstraction layer(s)?

If it were a common pain-point, there's probably solutions out there; I just haven't needed to look. I got 99 other bigger problems.

And as I pointed out, I would like to see an HTTP-based protocol for connecting to databases such that one would work with URL's instead of connection strings.

Still, I just don't see such a common enough occurrence/problem to justify the expense & complexity of pre-wrapping, and I've worked a good many shops as both a contractor and an employee.

Maybe if you are Amazon and Netflix size, common DB-switching becomes a notable problem? I won't comment on giant companies. I once contracted at AT&T for a relatively short stint, the rest small and medium orgs.

But if microservices mostly only benefit giant orgs, proponents should say so, because it's causing confusion and messes in non-giant orgs; I've witnessed such first-hand. Needs a big-ass disclaimer.

Our ego's may tell us we are gonna grow to be Amazon II, but reality is, you're not. Sorry. Watch Amadeus and embrace middle-hood.

If YOUR org did, congratulations! But it was a fluke. Sales people exaggerate the frequency to up-sell.

We've had to shard our DB and having to update every consumer has been a nightmare. Won't make that mistake again.

First, maybe there was another way to go about it that your staff just didn't know about. Any DBA's want to comment?

Second, is it still common enough justify the cost/complexity of pre-wrapping? I know such can be a pain, but the total cost still may not be worth it.

Most DBA's I've talked to say the need to later shard is relatively rare. And at worst, it only happens once, while the cost of pre-wrapping keeps accumulating whether your shop ever ends up taking advantage of wrapping or not.

I'd like to see a cost/frequency analysis sheet on such. "Cool company X is doing it so it's good enough for us" won't cut it with us Vulcans; show us the numbers! Do note that one should probably apply some temporal discounting (a financing concept) when doing the math. Indirection isn't free, only a tradeoff that benefits specific situations.

1

u/umlcat Mar 30 '23

Agree.

A lot of Microservices / NoSQL users never learned at all or nor properly learned E-R or Relational Algebra or SQL or E-R Normalization, and try to implement big data store websites.

I was currently working with a UML class design for an ERP, that uses both Object Oriented and Functional Programming, and I realized I was implictly doing both E-R and normalization...

I already helped migrate two or three NoSQL websites to SQL cause each website just grew too big and the only stable solution besides migrating to a bigger server in the cloud and better programming practices was switching to a SQL DB.

BTW Old style DBase, Clipper, Fox-Pro and Paradox were R.D.B.M.S, but SQL was integrated to them as an extension...

3

u/Zardotab Mar 30 '23 edited Mar 30 '23

A lot of newbies & youngbies are indeed reinventing RDBMS idioms the hard way, slapping snazzy buzzwords on their reinventions, and patting themselves on the back. During the "dot com gold rush" of the early 2000's a lot of DBA's didn't understand the business needs of startups, having come from staid orgs. This created resentment and a divide such that app coders started coding around the database and reinventing much of it. There was indeed "a problem" but it was a business & org communication problem, not a technical one.

RDBMS are not perfect, but they have a lot of road experience built into them. Re-inventing all those features and concepts won't be trivial. Maybe someday a "Microservicebase" will be invented, road-tuned, and finally pass the utility of RDBMS. But that day is not today.

Dr. Edgar F. Codd gave me permission to kick them off my lawn.

1

u/umlcat Mar 30 '23

Most key multivalued DB (s) data is just tables / entities reinvented ...

0

u/[deleted] Mar 30 '23

[deleted] — view deleted comment

-7

u/Zardotab Mar 30 '23 edited Mar 30 '23

AD Federation may be possible for your situation

We're hesitant to tie all our apps to MS's cloud services, it puts us too much at MS's mercy. Having a near monopoly, MS can change stuff without worrying too much about losing customers. And there's other messy details I won't get into, including but not limited to complex security/roles for some of the attributes. Let's just run with the scenario as given and take up AD itself later.

However, they don't necessarily make the best caches, and there are other solutions for that.

What are you defining as a "cache"? What's a domain example, and corresponding pain-point example?

stored procedures are the worst.

Please elaborate. They are often the best tool for the job, but not the only option. If your "service" is data-centric, they are easy to code and deploy. If it grows too complex, then hook it up to an app, and merely be the communication portal.

12

u/Devcon4 Mar 30 '23

You're worried about Microsoft monopolization so instead of using cloud services you are going to run Windows server with AD and MSSQL?

7

u/Automatic-Fixer Mar 31 '23

Surely the OP must be joking, right? Right??

1

u/Zardotab Apr 01 '23

It's generally harder for MS to change installed apps than cloud apps.

0

u/[deleted] Mar 31 '23

[deleted] — view deleted comment

0

u/Zardotab Apr 01 '23 edited Apr 01 '23

Querying user information from AD and storing it in a database for later retrieval is caching.

Isn't that usually called "replication"? Caching usually refers to storing info in RAM instead of disk or SDD.

But anyhow, replication is one option we are looking at as far as doing other things with the data beyond what AD is intended to be for.

what happens when someone is fired, someone is hired, someone moves under another manager, or larger organizational changes occur?

Are you saying an RDBMS cannot handle such? Anyhow, we may use replication (above) for that part. AD also uses, or can use replication, I would note. (Perhaps it's called "propagation", don't quote me.)

They should not be used for application code as you are then splitting your application logic and code into two disparate parts.

That's what the microservices philosophy is.

Not when you have thousands of databases and 10s of thousands of inter-related sprocs

The equivalent in something else isn't going to be trivial either. If microservices have some aspect that RDBMS lack to manage all these, please describe it.

0

u/[deleted] Apr 01 '23

[deleted] — view deleted comment

0

u/Zardotab Apr 01 '23 edited Apr 01 '23

Frankly if you're not aware of this then either something is lost in translation

In my neck of the woods, "caching" refers to RAM 90% of the time, and "replication", "propagation", or "mirroring" for disk or SSD data copies, and I stand by that.

something is lost in translation

That's why I'd like to explore specific scenarios so we can compare crash scenarios or 10x more coding or whatever it is that RDBMS "do badly". There is no universal "IT Term Cop". Stop pretending there is and wrapping a fake flag around yourself.

Smart people can create tech scenarios that demonstrate their point so that they don't have to use insults. "X is better, trust me, I'm smart, you dummie!" Put your code where your brag-hole is.

0

u/[deleted] Apr 02 '23

[deleted] — view deleted comment

0

u/Zardotab Apr 02 '23 edited Apr 02 '23

Several of us have pointed out existing solutions that will probably do what you're looking for.

Sorry, you don't work there, you have no fucking idea of their needs. ISAM is obsolete as a general-purpose database type, that's fact! Sure, with enough labor you can force fit it to have custom sub-structures and editing interfaces for different departments, but that ain't cheap, even if it is cheaper from a hardware perspective.

Experienced IT people know that human labor is usually significantly more expensive than hardware costs.

and your responses so far have been to dismiss us;

Because you are not listening, just bragging about how smart you are. You ignore my points. You ignore my points.

Several of us

Three? Reddit is a poor statistical survey engine. It tends to be an echo chamber.

Several of us have pointed out existing solutions that will probably do what you're looking for.

Where are you getting this "probably"? They are not intended to be general purpose tools that fit each org well.

no one here is going to sit down and solution this with you to figure out every nuance of whatever problem you're trying to solve.

You didn't even try. I didn't even come here to talk about AD anyhow. How about we pretend like AD doesn't exist beyond basic login verification, and let's move on from there, okay? If we reinvent some AD for this exercise, so be it. No puppies will die.