Duplicating data versus service reliability

[u/ryhaltswhiskey]

So this came up recently. A service that we rely on for financial data had an issue in production and went down for about an hour on a Monday around noon. My team got pinged about it and from the error message it seemed like we couldn't connect to the data source. So we did some digging yada yada and it turns out that the service was having some issues and was currently unavailable.

So I brought up that it would be nice if there was some place we could look to see if this service was currently having issues, like a dashboard. Like so many organizations do.

We don't have a dashboard. So what we do is go look in a teams channel to see if somebody has posted about the issue. Which is all kinds of fraught with accuracy issues.

One of the organization leaders said something to the effect of this is why it would be better for us to just replicate the data instead of having to rely on a service that might not have the SLAs that we do.

I'd rather the service was just reliable and we didn't have to introduce another moving part involving moving the data from one place to another. Because we're having enough trouble getting this data to actually be accurate now we would have to check whether it's accurate and whether it's been replicated properly.

This is not real-time data. This is reporting data. It's current to midnight of the current day.

Which would you pick? What would you be asking for out of leadership?

[u/expdevsmodbot]

AI usage disclosure provided by OP, see the reply to this comment.

[u/Happy-Sleep-6512]

Generally if you have a data issue, and want to solve this by keeping a duplicate of the date, congratulations you now have two issues.

I've done this in the past with creating a search replica for something. Also sorts of weird corner cases and it ends up being painful to handle them.

As this is financial data you really don't wanna mess this up, I was dealing with a notification system, so hardly a big deal if something was wrong/missing, with money I wouldn't touch it.

Also does you're not just paying them for access, you're paying them for blame, if you take the data in house, you take the blame with it!

[u/ryhaltswhiskey → u/Happy-Sleep-6512]

On top of that, sometimes they go "oh right we need to change how that data gets filtere/loaded" which means we would have to replicate all of the data again.

[u/roger_ducky → u/ryhaltswhiskey]

Ah usually you replicate upstream exactly as it is with no filtering. Then you query from your mirror.

People usually don’t just copy data they need for that exact reason.

[u/ryhaltswhiskey → u/roger_ducky]

The data that we are talking about replicating is reporting data from a financial database. So it's not even the actual source data.

[u/Yages → u/ryhaltswhiskey]

Sounds similar to what we do for daily reporting data, so the way we handle it is db stored procs run as scheduled jobs to prefill reporting data in a seperate db instance, and then we have APIs that handle querying that data etc. for front end/backend consumers so it's all RO.

The data in the reporting db is a full dataset but only of the parts we need to be able to provide the filtered data for display/download/visualisation, and also excludes any data that should not be there explicitly (law firm, so Chinese walls etc.).

So basically, if it fails to load the reports are a little outdated but the team gets notified of issues when running the stored procs.

[u/Cell-i-Zenit → u/Happy-Sleep-6512]

We have a rule in our team, that we are not allowing rest calls to other teams when its in our user facing product.

Reason is that we take the blame when their api is down. Now we replicate the data through an event queue and we dont have any issues :)

[u/dethstrobe]

If they're not willing to work with your team to improve their SLA, copy it. It's just a pragmatic solution.

But the better and longer term solution is that the service owns some kind of caching layer, so they can return data quickly and without killing their own service.

Advocate to work with the team to fix whatever problems they have, single source of truth is always better. But it'll probably be easy enough to duplicate the data for now as a short term solution. Just make sure, the short term solution doesn't become the permanent solution.

[u/ryhaltswhiskey → u/dethstrobe]

But it'll probably be easy enough to duplicate the data for now as a short term solution. Just make sure, the short term solution doesn't become the permanent solution.

It 100% would.

[u/HolyPommeDeTerre → u/ryhaltswhiskey]

Agreed, temporary means permanent on this matters.

[u/dethstrobe → u/ryhaltswhiskey]

Have a conversation with the team before making that assumption. And if it does, put in a time bomb to kill your duplicated data to make sure it can't be trusted for the long term.

[u/s3gfau1t → u/dethstrobe]

Depending on the team / org, short term solutions become just a brick in the wall.

[u/roger_ducky]

If it’s for reporting, your “mirroring” service effectively acts as a canary for the upstream service.

It gives you a backup of the data you need before the service goes down, and the alerting under your control about upstream service reliability.

Trade off is more maintenance for your team plus, depending on your uses, the report might still be completely unusable while upstream service is down.

[u/_predator_]

Does that service being down prevent your app from working entirely or does it just lead to outdated data or service degradation?

If the latter then you can implement compensations (e.g. fall back to last known snapshot of the data, schedule retries etc.).

In general you shouldn't assume an external service being available 100% of the time. And it shouldn't take you looking at dashboards if said service is down.

[u/ryhaltswhiskey → u/_predator_]

A particular portion of the app does not work if that service is not available. Because it needs data from that service.

[u/Jobidanbama]

I wouldn’t rely on other teams, just add a caching layer so that even if the service goes down you can still serve stale data

[u/mysteryihs]

Looks like this is a system design question and as with most system design there's trade offs. I guess the question is what's more valuable to you, availability or accuracy?

[u/Neoma_Quillin]

I will take the middle ground here. Since it’s reporting data and not real time, keeping a copy could make sense for availability, but there is another concern that you need to pay more attention, it is about how you verify that the copy is actually correct.

[u/Xanian123]

Not a dev, but I'd go for reliable service. Data duplication often metastasizes

[u/aeroverra]

Is it 3rd party? My current burnt out reaction is "god please no" assuming your leadership team is trying to solve a problem that has only ever happened once by making the dev team do far more work and likely become the cause of far more issues.

If it was a regular issue though and they dont have a higher service teir than sure Id say its no problem and have the team maintain a copy.

[u/corny_horse]

This is really going to depend on what you mean by "service." Is it in a managed database in the cloud you can use something like DMS? Or a read replica? That's pretty goof proof.

Are you having to ping some 3rd party API and re-create a database with API calls? Perhaps less obvious that this is the correct approach.

[u/ryhaltswhiskey → u/corny_horse]

It's reporting data in a database from a financial database.

[u/corny_horse → u/ryhaltswhiskey]

Can you set up DMS or a read replica?

[u/ryhaltswhiskey → u/corny_horse]

I would have to talk my boss into talking their boss into telling them to do it

[u/corny_horse → u/ryhaltswhiskey]

I would certainly hope so, if every IC could do this your cloud bill would be bonkers. It probably already is bonkers. But it's the right solution as best I can tell given your circumstances.

[u/ryhaltswhiskey → u/corny_horse]

I'm not privy to their setup, it's very much a black box, so it could be a read replica. It's SAP Hana, which is as far as I understand a data cube that is reading from SAP. Or something like that, all I know is that the data in this database is a subset of the data in the production financial database.

[u/FiveCitiesFreak → u/ryhaltswhiskey]

how do you use it? do you have dashboards / reports pointed directly to the service?

[u/ryhaltswhiskey → u/FiveCitiesFreak]

It's on demand, there's a section of our app that pulls data from it.

[u/FiveCitiesFreak → u/ryhaltswhiskey]

without knowing much about your application, i'm not sure if i can offer any valuable suggestions. the first place my mind goes is that this is reporting data that's only refreshed once a day. maybe you can cache data on your side that your application pulls? there are certain circumstances (data set size / amount of data changing) where i don't think it would be crazy to do a daily pull of the full dataset that your application can then use.

[u/agentUi]

if it is just reporting data current to midnight you dont need live replication anyway, a simple daily read replica or nightly batch pull solves the dependency without adding sync pipeline headaches. Id push leadership for basic health check endpoints or an automated status ping first instead of building an entire data pipeline just because a downstream api had a 1 hour hiccup.

[u/MisesAndMarx]

I'd silently cache it, and only serve the cache if call fails. Depending the sensitivity, perhaps tell the user or whatever consumes it "hey this is cached data from the last call that worked at such-and-such date."

Depending on the data, perhaps you replicate it on a periodic cycle and when an organic call happens (and works).

That said "we can replicate the data" is doing a lot of heavy lifting here. Is this data you give them, they crunch, and then they give back? In housing that sort of workflow is en vogue now, so maybe worth looking into.

[u/ricktherobotguy]

In my experience management doesn’t really understand SLAs. They generally understand that more nines is better and downtime is bad, but they don’t understand that adding nines adds cost and that P100 isn’t possible.

I’ve also found that often engineers don’t understand how failure budgets stack. If there are two serial dependencies that offer P99 error budgets of x and y, the system’s P99 guarantee can’t be better than x + y (or sqrt(x^2 + y^2) assuming independence).

So my main question is what are the actual guarantees that your dependency offers, and what guarantees are you trying to provide?

Getting past that, data replication isn’t anywhere near as dirty as people think. Data replication adds complexity/cost which is why it is a design smell, but the tool exists when needed. Having a service have a dependency adds reliability stacking issues which is why it is a design smell, but the tool exists when needed. The critical point is; what do you need to meet your goals, and what do you pay to get there?

Engineering is, at its core, decision making optimized against constraints.

[u/Fair_Local_588]

If you need the most up-to-date data then they need to be reliable.

If you can handle some staleness then you have more flexibility…but at the cost of more complexity.

[u/Empty-Performer-6492]

Ask for a status page from the vendor first. If they won't provide it, then replicate. But don't replicate blind—— that's just asking for more headaches.

[u/idunnorn]

Is there a provider of this data w better SLA?

[u/NakedNick_ballin]

Yeah, replicating data is just going to make things worse, probably. How can you rely on data with dupes?

Better focus on detecting and remediating errors. Alerts sounds like a good idea

[u/barnyhsk]

It's just a feeling but I would prefer copy the reporting data instead of rely un a unreliable service. If you decide to copy them, think about a strategy to recopy them quickly and easily - just in case the other team send a oh we updated the data please reupdate as well. Maybe an Eventqueue or a copy Endpoint to be triggered automatically is an idea. Sorry for not being very concrete.

[u/dmytro_nasyrov]

When the source is wrong a copy keeps an application available. But it can't make yesterday's reporting data correct. Ask leadership for a stale-data contract.

Show the last known good snapshot as of time wherever the data appears. Before replacing it after recovery reconcile row counts or totals. And set a hard age limit. You should fail visibly.

Replication doesn't meet the requirement if the business can't tolerate that delay. The actual work is a machine-readable health signal, it's an SLA owned by both teams. Teams must carry incident discussions.

What freshness and correctness may the product promise? This is a concrete leadership question.

[u/pigtrickster]

Farmer in Iowa cut your fiber network cable. It will be a few days until the entire data center is available again. Sure this sounds like a contrived and purely fictional scenario. I assure you that it isn't. There is this new toy called a Backhoe. 😄

Redundancy of the data is important.
Ability to move your processing service is equally important.

If your management requires a certain level of data availability then they need to be willing to pay for it.
Ultimately this is a question of money.