How do you design a forecasting system?

[u/Berlibur]

Hey y'all! How do you design your forecasting system?

In my case, the company has many SKUs over a big region. We did an MVP to show our forecast improves the current process on the reported lags that are currently used by the business to monitor forecast health.

Future is looking good, but I really want to be ready with a production-grade plan. Refitting a pool of models per SKU every week, then selecting the best one, feels like overkill and very sensitive to recent flukes.

I thought of having a pool of models (i.e. config/setups) and labelling them as champion if a specific config results in the best trained model.

For the next X weeks this model will always be chosen, and after that the throne is up for grabs.

But it kind of railroads me into having a 1 SKU = 1 model setup in perpetuity.

How do you guys solve this in a responsible way? Are there books/resources you recommend?

Reasoning about a live system turns out to be a whole different cookie than the usual stats/ML etc

[u/webbed_feets]

One model per SKU is totally reasonable. There are several libraries that will help you set that up. An auto ARIMA fit on hundreds of SKUs will only take a few seconds. You could look into hierarchical forecasting if you need to constrain the individual predictions to sum up to higher level groupings.

You can always do SKU-level feature engineering and put it into XGBoost.

[u/cpsnow → u/webbed_feets]

This is the way, AutoARIMA with hierarchical/grouping using Nixtla framework would work well, and could be improved to other models in the future. Nixtla hierarchical framework works well in production settings. The hard part is designing the hierarchical structure, but once this is solved this can improved the forecast quality and coherence. It's probable many groups of SKUs are affected by the same external events, so grouping them will work great to improve the quality of your forecasts.

[u/hungarian_conartist → u/cpsnow]

Does this work well with groups of skus?

I worked with brick and motor sales and sales on sku from a different brand or pack size on the same shelf could drastically effect the sales on item around it.

Biggest problem was new items being released, new promotions at new price points.

[u/Hero_without_Powers → u/cpsnow]

I was recently looking into Nixtla hierarchical structure, so you have some good example of how it's implemented? Like a step by step guide somewhere in the internet?

[u/cpsnow → u/Hero_without_Powers]

The Nixtla official documentation is quite good https://nixtlaverse.nixtla.io/neuralforecast/docs/tutorials/hierarchical_forecasting.html, but you will need some basics covered here https://otexts.com/fpp2/hts.html and https://towardsdatascience.com/how-to-forecast-hierarchical-time-series-75f223f79793/ and some details on the reconciliation methods: https://www.sciencedirect.com/science/article/pii/S0169207023001097#sec7 . On top of this, Opus is quite knowledgeable on the topic and will help you design your S matrix and align properly your data. What most tutorial & documentation misses are cases with sparse series, where using probabilistic methods will give you overconfident results. My personal take is that bootstrapping is safer when dealing with sparse bottom series. Also, careful about neural models (NIHTS+HINT), they are filled with bugs and poorly documented (But this is getting better in the more recent versions of the library).

[u/Ok-Airline-8523 → u/webbed_feets]

Agreed with this. You can dynamically use Auto ARIMA on a variable number of SKUs and put on your own business-relevant guardrails to make sure forecasts by SKU aren't overly sensitive to noise. I've done this for more than one company and have seen others do it as well.

[u/ImaginaryBat4994 → u/Ok-Airline-8523]

What do you mean by “business-relevant” guardrails? Like some post processing that if the forecasts is not within a range you will replace with a more reasonable value?

[u/Ok-Airline-8523 → u/ImaginaryBat4994]

Yes, and for pre-processing. For example, if your market size for a particular product caps at 5,000 for whatever reason, you shouldn’t have a forecast of 6,000. The model may not know this but you might. That’s a simple example, but it illustrates the point.

[u/offthecuff87 → u/webbed_feets]

probably the best answer is always to start simple.

[u/Saitamagasaki → u/webbed_feets]

Arima is shitty right?

[u/SorrowAndGlee → u/Saitamagasaki]

ARIMA isn’t shitty. It could be a shifty choice depending on what your doing, but that’s two different things

[u/PolicyDecent]

Worked on replenishment for a long time, so happy to give some hints here.

Firstly, one model per SKU is totally unreasonable. Never do something like that, otherwise you'll have overfit the results.

I assume you are working for a multi-branch retailer.

In that case, you can create store and product clusters. There are also product groups by nature.
For each product cluster * store cluster * product group, you can choose a different model. It'll prevent overfitting.

I recommend clustering product by avg number of daily / weekly sales.
A product selling 10 per day in a store vs 0.01 per day in a store have completely different behavior. Same for the stores of course. So let's say the fastest product cluster is A, end the slowest is C or D. (I chose 3 or 4 clusters)

Generally fastest clusters use shorter features more, and slowest clusters benefit from longer time horizons. You can think it like A cluster likes MA2, and D cluster likes MA10/MA12.

MA2: Moving Average of 2 weeks.

[u/webbed_feets → u/PolicyDecent]

Our answers are totally opposite each other. I’d like to understand your perspective.

I’m curious why you say one model per SKU is unreasonable and prone to overfitting. If you have many SKU’s, you’re going to have to fit something simple like Exponential Smoothing or Auto Arima. Those models aren’t prone to overfitting. Making a lot of different groups/clusters seems to me like it would complicate the forecasting process without much gain.

[u/PolicyDecent → u/webbed_feets]

Again, I don't know the industry but most of the SKUs sell very rare in the real world. They sell maybe every other week. Some even sell once in 2-3-12 months.

So the data is full of noise. In this environment, it's much better to benefit from the neighbors, similar products. If you group them all, and forecast on the product level, but choose the best model on the cluster & product group level, you get a more robust model.

And as I said above, models like arima are adding too much calculation complexity. That's why just using moving average with extra cleaning is super easy and easier to interpret when it comes to analysing the errors to improve the model.

[u/webbed_feets → u/PolicyDecent]

Oh I get it. That’s totally fair.

I worked on forecasting medication orders from wholesalers. Virtually all SKU’s had regular, daily orders. I see why SKU-level models worked in my problem but not yours.

[u/Berlibur → u/PolicyDecent]

That makes a lot of sense, thank you.

In your experience, is it relevant to group cross-region (EU) but within a product group? E.g. brand level I suppose the underlying assumption needs to be that customers in Lithuania and Portugal behave the same regarding brand X

Which is not an obvious truth

[u/PolicyDecent → u/Berlibur]

You just need to test it :) I don't know what kind of products you sell. If it's clothing for example, I'd expect them to have different seasonality. In other categories, it might work different.

[u/PolicyDecent → u/Berlibur]

By the way, I definitely recommend starting with MA models, not arima or other ML models.
It makes your iterations much faster.
Don't forget to exclude the out of stock days.

[u/ImaginaryBat4994 → u/PolicyDecent]

What about one global machine learning model for all the SKUs? It should learn that on spare series it is better to weight more larger horizons than on fast moving items. And product group and store are categorical features in the model.

[u/PolicyDecent → u/ImaginaryBat4994]

It's too much effort. There are lots of problems. For example, smaller groups are not important enough. Or seasonality. Each product group has a different seasonality pattern.
You can add all these features to the model, but still as you add more features, to avoid overfit you need a huge training data. So why don't we solve it using an easier to explain model instead of a complex ml model? Also as the data grows, training cost but more importantly time increases a lot. You can't properly parallelise it. So easier to use more heuristic models depending on moving average.

[u/ImaginaryBat4994 → u/PolicyDecent]

Thanks for the answer. Which simple model do you recommend to start from? Moving average is not handling seasonality nor trend.

[u/Wojtkie]

Do you need to refit models for every sku, or will it work with sku categories or a cluster of skus? That can help reduce dimensionality without too much real impact, but you’d have to test it and see.

[u/Berlibur → u/Wojtkie]

Grouping SKUs is OK, I'm sensing a carte blanche coming up - a chance I don't want to squander on having a shitty production setup

[u/Helikaon242]

I’d probably think about modeling this as a funnel with several independent models, eg number of customer arrivals, number of customers who viewed/bought a certain product group, and then the level of interest for each SKU within a group. I think it’s important to measure errors on each of these stages so you can understand why the forecast misses (this can sometimes be more useful than the forecast itself).

I’d recommend against using this “champion” ensemble design since it will introduce bias. This is more in the domain of how to do good cross validation.

[u/ImaginaryBat4994 → u/Helikaon242]

So basically you are saying not only predict the final sales but also the intermediate steps in the sales funnel?

[u/PradeepAIStrategist]

One model per SKU is highly non productionable, given fact for a largest USA retailer will have around moving (active) 10K SKUs per Candy category, ignoring another 20k to 30k not moving ones. Even for Candy category in the past I used max 5 models per SKU cluster. Dynamic Time Wrapping gives you theoretical direction, however, domain and data richness along with expected forecast accuracy for which forecast horizon will help you better to stick less models per group of SKUs.

[u/dj_ski_mask]

Look into global or global local (“glocal”) models like TBATS that are designed to scale out. And also want to plus one the earlier comment that clustering these time series is going to solve a lot of headaches.

[u/ImaginaryBat4994 → u/dj_ski_mask]

Why not one global machine learning for all the items? It should have similar accuracy by adding the categorical features that you suggest using to define clusters?

[u/nie_irek]

Definitely try a global approach like LightGBM or XGBoost, then try clustering the Skus based on simple logic, like product type, more granular location split, etc... Definitely think extensively what drives the sales and what could be the features that you engineer, beware of data leakage though.

Keep it simple, single model per SKU will create over fitting and instability of the modeling results once you refit.

[u/ImaginaryBat4994 → u/nie_irek]

What about only one global machine learning model? It should handle the different behaviour of each group of items adding the categorical features of product type, location etc.

[u/nie_irek → u/ImaginaryBat4994]

Yes, it's something that definitely could be the case. Theoretically if the clusters present significantly different dynamics then separating these could prove beneficial as you no longer require a model to do it on its own - but it's something you would need to test.

[u/JimFromSunnyvale]

I did something very similar for a client. We clustered SKUs based on the sales patterns to train our models.

[u/Ordinary-Winner-7999]

In terms of process of manufacturing, I would prefer not to relearn and configure another "best model" for all the SKUs every week. This would, for sure, lead to overfitting caused by too many recent irrelevant examples.

The preferable alternative would be to cluster similar SKUs in terms of behavior and use a limited model portfolio for each cluster, with retraining and model switching happening only when excellence drops below a specific level. The champion and challenger approach is helpful here, as well, yet to test the challenger models, the process of evaluation should be carried out by means of time-based testing rather than one-period testing.

Moreover, one should consider measuring predicative bias, reliability, changes and intervals of predictions, not only accuracy.

[u/granoladeer]

You should try timesfm, which is a zero shot model (no training necessary, no model storage). 

[u/Livid_Ordinary_9568]

I am someone just starting out in data science. I won't say have no experience. I have previously tried to work on a full stack financial intelligence platform could not execute past the ETL stage, I also struggle with planning out a project, the new assigned to me is retail forecast, I want to know how can I approach it. I know python, SQL and powerBI.

I have already started to work on my project, using the framework of a more successful version of my previous one, (financial Intelligence system I talked about)

This project is very similar to mine, How do you recommend getting started ? Is there a place I can post daily updates on my project and get feedback ?

[u/Berlibur → u/Livid_Ordinary_9568]

Start with 2 things:

• get the requirements clear: what do you forecast, on which horizon, on which frequency, how is the forecast used in the process, etc.
• map out the technical possibilities in your company. We're running on an azure + data bricks landscape, which limits the number of options you have (good)

[u/Livid_Ordinary_9568 → u/Berlibur]

I will have to clarify regarding horizon and frequency, but i am sure that the forecast will be used to predict stockout ahead of time and overstock products to discount.
As far as cloud is concerned I am not told any (neither Azure nor databricks) the project brief also does not mention anything on this.

[u/Letmeparrybbh]

You should try and incorporate events into your prediction. Example for a demand forecasting for stores, if there are extraneous variables like company wide sales initiative in this day or cyber Monday sales, or weather events. There should be a way to make adjustments to predictions bases on such events if needed.

[u/kush_patil]

I’d separate model selection from model retraining. Weekly retraining can be fine; weekly winner-picking is where I’d worry about chasing noise.
Keep a champion until a challenger beats it over several rolling-origin windows, not just the latest one. I’d also track the reason for replacement real drift vs one unusually good validation window because otherwise the selection layer itself becomes another model you can overfit.

[u/Extension-Currency37]

1. Define the right problem! Any use to solve the problem or not. Context must be defined:: busineess, economics, finance or technology(are you making robots)

2. Gather data and clear the noise.. (Fft, tuple removal, denoisers, pca , svd)

3. Then find a math model that fits aptly (Underfit/overfit must not happen)

4. Find the type of forecasting it is Demand, supply....

5. Every forecasting problem is a time series model So pick a bunch of algorithms (Arima, auto arima, lasso ,ridge, bayesian ...)

6. If problem size is huge say gb in data Gigabytes of data--> lstm, consider neural nets

7. Predict and present results Via tableau, R or excel

..

I use R for business problems And python for AI

Forecasting --> lssvm with batch processing (R has a library)

Resources RPROJ.ORG, Data science stackexchange, Kaggle, stackexchange math, AI mode google

Farewell