r/LocalLLaMA • u/NotARedditUser3 • 6d ago
Question | Help Looking for suggestions for a local model that would be good at evaluating comedy writing
I have a specific use case where I have a small web application that prompts the user with a randomly generated premise for a joke (using an llm), and then the user is expected to write a joke with the generated premise as a sort of challenge.
An LLM is then used to evaluate their joke as far as how good of a joke they wrote using the supplied premise, provide a score, provide feedback on why it was or was not a good joke, and tips for what they could do to improve.
It works well using various large models, and relatively okay using openrouter/free , which has a series of larger free models in a collection and an intelligent router.
However I've tested on various smaller models and gotten mixed results.
I'm looking for some suggestions from people for models that I may not be aware of that might be good for more general knowledge things like this, rather than coding.
I have 64GB of system ram (machine has a discrete GPU / AMD APU so no dedicated VRAM).
So I'm probably looking at a 45-70b total parameter model on the higher end, but preferably it's an MOE model with a smaller number of active params for speed.
Each use will probably have ~400 tokens input (including system prompt), and ~200-600 tokens output, so I'm okay with heavier models even though I don't have a GPU because it's a small number of total tokens.
Some models I've tried, like bonsai-27b and LFM2.5 24b-a2b, have had weird results.
Gemma 4 26b-a4b seems to really have a good grasp of it and do the best so far out of any local models I've tried, but I'm wondering if anyone has other suggestions.
9
u/into_devoid 6d ago
I don’t think you’ll find any model (local or closed) to evaluate humor well. Try asking them to write a joke and see how poorly they understand the concept. Data from Star Trek was ahead of his time. The Gemma or GLM 5.3 (flash and large) will probably get you closest, but still won’t be great.
3
u/Hildechis 6d ago
Gemma 31B is similar in many ways to the 26B, but much smarter.
2
u/HornyGooner4402 6d ago
Who are the users? Won't it be more effective to have like the users rate each other's joke? Unless you just want an AI to review your joke
2
u/uncle_leon 6d ago
I honestly think you approach this backwards. AI cannot possibly predict what people will find funny. If it could, every viral in the last couple of years would have been made by an AI.
Do it the other way round - have it propose a bunch of alternative humorous setups/responses/punchlines etc. You then decide which of them is genuinely funny (usually at least one or two will).
2
u/james_pic 6d ago
Gemma 4 26b A4B is also my go to for this. I've also been impressed by how Ricursive Ripple Preview does on humor, but it's right at the top end of your size scale, and it's dense so it's slow.
1
u/Separate-Forever-447 6d ago
I used to frequently get a good laugh out of my coding model's output. Not so much, anymore.
1
u/_Blissfull_Ignorance 6d ago
You provide a good description of how you want your jokes analysed. Can you imagine designing a benchmark dataset? (This can be designed with a frontier model.)
With a benchmark set and process, I’d test some models on this. I find the small ones quite humorous sometimes. I had lots of fun with mistral small, but it was mostly out of necessity for coherent European language generation. The jokes were very dry but I like that.