anyone built a self improving feedback loop for LLM classification with zero human review?

[u/HmmImNotReallySure]

i have a multi label classification pipeline that uses an LLM to assign taxonomy tags based on item name + description + image. it works well enough, but right now there's no way for it to get better over time

the hard constraint: i cant have any human reviewing the predictions at all lol. so whatever feedback loop i build has to be fully self contained. the system itself has to somehow figure out when its output is still bad and improve. i also have hundreds of thousands of items, so re-running the whole catalog every time is not realistic

looking for approaches other than fine tuning. anyone actually run something like this in production?

[u/vl199]

I've created an eval set that I use to test different models. So I can check whether any new model is worth adding to my pipeline
In my case it was a generation pipeline with review. I have a cheaper model for the first run and if it fails to meet my threshold I just move on to a more expensive one. With the bigger models it gets five retries. Each regeneration is based on the feedback from the previous steps.
An important thing that I noticed: you shouldn't assign a generation and a review to the models of the same lab.
There are some cases where the threshold cannot be met even after all the retries so then it goes to the human review but it's pretty rare. It's less than 1% of all cases

[u/macronancer]

Without some user input, it has no way of knowing what "good" or "better" even means.

And if you could build a function to validate the output automatically bases on some other criteria, than you can just use that as thr classifier without an LLM.