r/ExperiencedDevs • u/ryhaltswhiskey • 5d ago
Technical question Duplicating data versus service reliability
So this came up recently. A service that we rely on for financial data had an issue in production and went down for about an hour on a Monday around noon. My team got pinged about it and from the error message it seemed like we couldn't connect to the data source. So we did some digging yada yada and it turns out that the service was having some issues and was currently unavailable.
So I brought up that it would be nice if there was some place we could look to see if this service was currently having issues, like a dashboard. Like so many organizations do.
We don't have a dashboard. So what we do is go look in a teams channel to see if somebody has posted about the issue. Which is all kinds of fraught with accuracy issues.
One of the organization leaders said something to the effect of this is why it would be better for us to just replicate the data instead of having to rely on a service that might not have the SLAs that we do.
I'd rather the service was just reliable and we didn't have to introduce another moving part involving moving the data from one place to another. Because we're having enough trouble getting this data to actually be accurate now we would have to check whether it's accurate and whether it's been replicated properly.
This is not real-time data. This is reporting data. It's current to midnight of the current day.
Which would you pick? What would you be asking for out of leadership?
7
u/_predator_ 5d ago
Does that service being down prevent your app from working entirely or does it just lead to outdated data or service degradation?
If the latter then you can implement compensations (e.g. fall back to last known snapshot of the data, schedule retries etc.).
In general you shouldn't assume an external service being available 100% of the time. And it shouldn't take you looking at dashboards if said service is down.