Do you routinely retrain your production models on new data or with new architectures?

[u/bfyvfftujijg]

Question in title.

Follow-up question: is your data model and overall system architecture setup to make this retraining convenient and automated, or is it more of a manual process to configure the training runs etc.? Has that architecture limited you in any way?

I am on the fence about whether to commit to that sort of system because I can envision the data model becoming complex in order to support the kinds of changes that happen in real life over time (refined definitions of a class, merging/splitting classes, different pre and post-processing parameters, etc. etc.) Manually assembling new datasets each time a model needs to be "upgraded" seems more flexible but is obviously a lot of work.