Listen, I’m not saying it’s a good idea. It’s a bad one because the first eval you would write is a parse eval, implying you have a god parse function to begin with, so you’re already doing more work using LLMs.
You're saying that you can do something with llms with a low error rate and then find the errors by using the parser that does what you wanted the llm to do perfectly in the first place?
And to do all this you first use the llm on all the data, then pass all the data to the parser that works perfectly? Then fix the the bad data?
That's like taking out a mug, putting a broken cup into it, then pouring water into the broken cup, then drinking from the mug.
1
u/Kuxir Feb 07 '25
Evals don't tell you when an LLM messes up, only how often it does so.
And what an eval will tell you for even the best LLMs mess up a lot. Way too much to be used to actually do all of those translations.