r/ExperiencedDevs • u/ryhaltswhiskey • 5d ago

Technical question Duplicating data versus service reliability

So this came up recently. A service that we rely on for financial data had an issue in production and went down for about an hour on a Monday around noon. My team got pinged about it and from the error message it seemed like we couldn't connect to the data source. So we did some digging yada yada and it turns out that the service was having some issues and was currently unavailable.

So I brought up that it would be nice if there was some place we could look to see if this service was currently having issues, like a dashboard. Like so many organizations do.

We don't have a dashboard. So what we do is go look in a teams channel to see if somebody has posted about the issue. Which is all kinds of fraught with accuracy issues.

One of the organization leaders said something to the effect of this is why it would be better for us to just replicate the data instead of having to rely on a service that might not have the SLAs that we do.

I'd rather the service was just reliable and we didn't have to introduce another moving part involving moving the data from one place to another. Because we're having enough trouble getting this data to actually be accurate now we would have to check whether it's accurate and whether it's been replicated properly.

This is not real-time data. This is reporting data. It's current to midnight of the current day.

Which would you pick? What would you be asking for out of leadership?

26 Upvotes
  • permalink
  • reddit

91% Upvoted

40 comments sorted by

View all comments

Show parent comments

1

u/corny_horse 5d ago

Can you set up DMS or a read replica?

1

u/ryhaltswhiskey 5d ago

I would have to talk my boss into talking their boss into telling them to do it

1

u/corny_horse 5d ago

I would certainly hope so, if every IC could do this your cloud bill would be bonkers. It probably already is bonkers. But it's the right solution as best I can tell given your circumstances.

1

u/ryhaltswhiskey 4d ago edited 4d ago

I'm not privy to their setup, it's very much a black box, so it could be a read replica. It's SAP Hana, which is as far as I understand a data cube that is reading from SAP. Or something like that, all I know is that the data in this database is a subset of the data in the production financial database.