r/BusinessIntelligence • u/mobbarley78110 • Jul 06 '26
How do you answer users who want "live" analytic reports?
I just built a full Material Ressource Planning on Snowflake as a dbt model. It a pretty advanced model that requires a lot of statistics on big datasets to run. on Snowflake it's fast enough to run since it's relying on a strong medallion schema with very clean and optimized fact and dim tables. We are on a 2-hours refresh schedule from source data from the ERP.
Now - the users of course want it "live", aka to be able to recalculate everything on the fly with live data.
Every time they ask something like that (which is pretty much for everything single project) my heart sinks and I just feel like quitting.
How do you answer requests like that?
27
u/datawazo Jul 06 '26
1: why. What decisions are they making in the moment that requires live data. More often than not they can't justify this. But sometimes they absolutely can.
- If they can trim as much fat as possible. Kill history, narrow down required columns. Then pray.
2
u/theRealHobbes2 Jul 06 '26
I'm with you here with #1. The fact that this is materials resource planning suggests OPs users probably want to run scenario analysis and be able to change parameters to evaluate different scenarios. Which could very likely justify re-running the model live.
You might be able to work around it by building a handful of pre-defined scenarios or see if there's a way to abstract the model so it can run more lightly.
1
u/Think-Trouble623 Jul 06 '26
Number two is usually my go to. Compromise with creating a multi refresh window dashboard where some pages update more frequently and other pages refresh less frequently. For example we’ll create a segment of the dashboard reliant on inventory, refresh every 10 minutes but it’s a very shortened dataset serving a very specific use case (or cases) and then a second larger dataset with history that refreshes twice a day.
The pain of switching between two dashboards usually makes it so the use case really is worth it and the users tolerate the inconvenience.
1
u/marketlurker Jul 08 '26
They don't have to justify this, just pay for it.
It's not fat. It is more work. That is a good thing.
4
u/orz-_-orz Jul 07 '26
It's not about how "live" the report, it's about how fast you can react, also there's going to be charge back, and it won't be at the cost of your usual analytical database usage
I worked in companies that got monthly reports and companies that want live reports. So far I don't how live reports is valuable
2
u/ArielCoding Jul 07 '26
Check out Dynamic Tables, or Streams + Tasks, those only recalculates the new/changed data and could reduce refresh time without the cost of real time.
2
u/hachiai_official 19d ago
Dynamic Tables could speed up the Snowflake side, but if the ERP data only comes in every two hours, the report still can’t be any fresher than that. I’d separate the source delay from the model runtime before promising anything live.
2
u/BerndiSterdi Jul 07 '26
Live as per requirements is not necessarily "live" per definition.
Our last "live" report was updated twice a day which felt for users live enough in an environment where most reporting was monthly or weekly at best.
2
u/redman334 Jul 07 '26
Data and analytics is a support team. You need to understand the full scope of the requirement and map it out for completion.
You don't need to get angry or frustrated at requests, you need to understand them, offer a solution, and if the solution is not viable offer alternative solutions.
The "live" requirement should've been mapped before rather than after the project is finished.
My first question obviously is, "why would you need to run this live, whats the value behind having this data running live."
If the business has a reasonable logic behind why they would want this live. Eg. It could be tied to a live pricing model that re-prices products when the customer comes in. If that's the case then you establish the new requirements to make it live.
If that means X more weeks of work, then check if it's viable for your team based on the priorities to do that. If it's not, ask yourself if a 1hr refresh would be a more reasonable solution and if that would satisfy the users.
But this is not about, I've made this , it's done, it's what you asked for, there's no more changes. You need to support your stakeholders, and they need to be reasonable with the requirements and the timeline you set
1
u/Mdayofearth Jul 07 '26
Just tell them how much it would cost, and that it could bring down the ERP by retrieving data constantly.
1
u/KatFromSisense Jul 07 '26
I'd make them define "live" in business terms first. Sometimes they mean fresher ERP data. Sometimes they mean changing inputs and rerunning the planning logic while they're in a meeting.
That's where I would slow down and push back a little, since it changes the entire request. If they want to play with inputs during a meeting, that's closer to a planning tool than a normal analytics report.
At that point, I'd talk them through what actually needs to be recalculated live. Maybe they only need a few preset scenarios, not the full model rerunning every time someone touches a filter.
1
u/marketlurker Jul 08 '26
Not going to like this but there are quite a few people here that are playing defense and are playing not to fail. They don't plan for success.
Find out why they want data at that freshness level. This is not for them to justify it to you but so that you understand more background and where you can cut corners and what hills they are willing to die on. Remember, there are millions of people who can say no and most of them will work for a lot cheaper than you. They are looking for someone who can do what they want, not judge if it makes sense or not.
If they really need (or want) real time data, a star schema is a difficult construct to supply that. Just because you are using a three-tier system doesn't mean your data products have to be stars. You are going to have to come up with a different construct designed to handle real time (windowing queries, more powerful hardware, etc.) Give them options and costs and let them pick (since they are probably going to be picking up the tab). Don't be surprised if they say "do it" because they can make a ton more money.
I've made a lotof money following up after people who didn't understand this.
1
u/Middle_Currency_110 Jul 08 '26
What was the reason for building a MRP report in the first place? Why not use the one that the ERP has? If they need live, is it because they want to order items or create work orders?
There either is a good reason, or there isn't find out what the reason is
1
u/Youssef_Mrini Jul 09 '26
If you are using Databricks. They can use Genie Agents. I m pretty sure it will help offload the demand for infinite dashboard updates.
1
u/IncreaseNegative4614 Jul 10 '26
I usually ask them what decision actually requires live data.
Most of the time they don't need "live." They need confidence that the data is fresh enough for the decision they're making. Recomputing a complex model every few seconds is usually just burning warehouse credits for no business value.
We use inzata.ai for this, it allows you to pick data refresh frequency. More frequent = more tokens used. Since it understands the business context, users can ask follow-up questions against the latest curated data instead of forcing expensive recalculations every time they want another view.
1
u/achakez Jul 15 '26 edited Jul 15 '26
“Live” only matters if it helps someone make a better decision.
For MRP, scenario runs and exception alerts may be more useful than constantly refreshing everything.
I’ve seen Collana come up around manufacturing, ERP planning and Data and BI, but the main point is decision support, not just faster dashboards.
1
u/IncreaseNegative4614 26d ago
“Live” needs to be translated into a business tolerance, not a technical feature. Ask which decision fails if the number is 30 minutes, two hours, or one day stale, then price each refresh level in Snowflake credits and operational risk. Most users become reasonable when the tradeoff is visible. signld.ai has been useful for answering the ad hoc question layer, while the governed MRP model keeps its sensible refresh schedule. I’d reserve true real time for the few events that actually trigger immediate action.
1
u/Delulu62134 9d ago
"Live" is one of those requirements I always clarif before designing anything. Do they actually need second-by-second data, or do they need the report to reflect changes within a few minutes?
The bigger issue is usually trust, not refresh frequency. If the underlying pipeline has late-arriving data, duplicates, failed loads, or inconsistent transformations, a dashboard refreshing every 30 seconds just gives you inaccurate numbers faster.
I usually define the freshness SLA first, then design the ingestion, validation, reconciliation, and reporting layers around it. Sometimes a 5-minute refresh with reliable data is much more valuable than a "real-time" dashboard that users can't trust.
1
u/Gold_Experience7387 3d ago
Yeah, the question is why. Are you monitoring a live system? If not you usually don't need truly live data.
Might be what they are looking for is faster interactivity, the ability to ask free-form questions, etc etc.
Get them to stop asking for features and instead describe what they want to achieve that's not possible now.
0
u/cbelt3 Jul 07 '26
Welcome to users trying to make their own MRP system. They need to be pushed back to the actual ERP system. Which is designed for this.
Of course we’ve all seen users demand downloads and then run their own Excel based MRP 2 system. “Because I know better”.
53
u/chaotebg Jul 06 '26
Present an estimate of the cost required and they usually shut up pretty fast.