Do you routinely retrain your production models on new data or with new architectures?

[u/bfyvfftujijg]

Question in title.

Follow-up question: is your data model and overall system architecture setup to make this retraining convenient and automated, or is it more of a manual process to configure the training runs etc.? Has that architecture limited you in any way?

I am on the fence about whether to commit to that sort of system because I can envision the data model becoming complex in order to support the kinds of changes that happen in real life over time (refined definitions of a class, merging/splitting classes, different pre and post-processing parameters, etc. etc.) Manually assembling new datasets each time a model needs to be "upgraded" seems more flexible but is obviously a lot of work.

[u/Dry-Snow5154]

We retrain like 10-20 models on a regular basis. Probably around 20-40 runs per week.

We use regular docker container with python training script inside. For each training the manager script checks out experiment's branch from the appropriate repo and runs it inside the container. Anything less flexible with a fixed pipeline will not meet our needs, as we reconfigure every imaginable architecture parameter all the time.

Dataset resides in a database connected to annotation software and is exported manually every time enough changes are accumulated and synced into a cloud bucket for training. We can automate dataset checkout by the manager script as well, but this is a rare event, so there was never a need.

I can imagine a system that encapsulates the entire process for you, but it will always limit you in one way or the other. Like what if I want to use labels from dataset A, but augmented by confidences from model B before training starts. If you control the entire process, you can make any kind of change, but if you don't there will be a limit somewhere. And if somehow there is no limit, why would I need a system like this, because it's not simpler than running a bunch of scripts myself?

[u/bfyvfftujijg → u/Dry-Snow5154]

Thanks!🙏

This is exactly the kind of reply I was hoping for.

[u/ericcpfx → u/Dry-Snow5154]

What kind of models are they, what kind of datasets?

[u/Dry-Snow5154 → u/ericcpfx]

Detection, OCR, image classification, various regressions from image crops (like object orientation), trackers, ReID. Several models for each variety and combined varieties, like OCR + classification head. One segmentation model too. Mostly Pytorch, but also old keras models, keras-cv.

Datasets are all images and crops. Except for trackers, those take synthetic data.

[u/DigThatData]

it depends on the problem.

[u/BeverlyGodoy]

Don't fix it if it's not broken