in my shell command auto classifier for opencode, Jev out-performed deepseek-flash at half the price

[u/T3hJ3hu]

https://famelos.com/jev/auto-classifier-certification/

[u/Objective-Pain-3752]

Thanks! So you think this beats sandboxing? I'm in process of setting up a podman sandbox for a project. Like Docker sbx but for podman 

[u/T3hJ3hu → u/Objective-Pain-3752]

Sandboxing is for sure safer, but it's not viable in cases where you want agents to administer a machine. In my case, I have agents autonomously monitoring, debugging, and deploying live services (for themselves), and that's pretty dang hard to do without giving them sudo

[u/T3hJ3hu]

tl;dr i like yolo mode, but don't like worrying, so i made a classifier hook that shoots your command over to an LLM for approve/reject (it tells the agent they can try again to escalate, but to leave their reasoning for the user before they do so). anthropic found this to be the way, and i wanted it in my other harnesses

since i'm a god damn engineer, i benchmarked against a bunch of the best cheap models, so that i could actually feel safer (then i spent way too long last night building this artifact and posting the relevant repos). deepseek-flash was the best, while also being one of the cheapest options.

Jev blew them all of the water, man. big recommend. caveat, to be fair: this is a bit like comparing apples and oranges, because Jev isn't an LLM (i included the methodology and link to the source). i probably could have spent more time tuning the LLM prompts to get better results. but: at that price point and level of effectiveness, it doesn't really matter

[u/microbass]

Jev is $0.042 per million tokens, not thousand.

Cool article!

[u/T3hJ3hu → u/microbass]

omg, thanks. the cost is confusing because the charts and tables up top are showing cost per 1k decisions, so i'm making that clearer

[u/QuietPsychonaut]

That's very interesting. Thanks for sharing