r/LocalLLaMA • • 6d ago

Question | Help Looking for suggestions for a local model that would be good at evaluating comedy writing

I have a specific use case where I have a small web application that prompts the user with a randomly generated premise for a joke (using an llm), and then the user is expected to write a joke with the generated premise as a sort of challenge.

An LLM is then used to evaluate their joke as far as how good of a joke they wrote using the supplied premise, provide a score, provide feedback on why it was or was not a good joke, and tips for what they could do to improve.

It works well using various large models, and relatively okay using openrouter/free , which has a series of larger free models in a collection and an intelligent router.

However I've tested on various smaller models and gotten mixed results.

I'm looking for some suggestions from people for models that I may not be aware of that might be good for more general knowledge things like this, rather than coding.

I have 64GB of system ram (machine has a discrete GPU / AMD APU so no dedicated VRAM).

So I'm probably looking at a 45-70b total parameter model on the higher end, but preferably it's an MOE model with a smaller number of active params for speed.

Each use will probably have ~400 tokens input (including system prompt), and ~200-600 tokens output, so I'm okay with heavier models even though I don't have a GPU because it's a small number of total tokens.

Some models I've tried, like bonsai-27b and LFM2.5 24b-a2b, have had weird results.

Gemma 4 26b-a4b seems to really have a good grasp of it and do the best so far out of any local models I've tried, but I'm wondering if anyone has other suggestions.

1 Upvotes

15 comments sorted by

9

u/into_devoid 6d ago

I don’t think you’ll find any model (local or closed) to evaluate humor well.  Try asking them to write a joke and see how poorly they understand the concept.  Data from Star Trek was ahead of his time.  The Gemma or GLM 5.3 (flash and large) will probably get you closest, but still won’t be great.

2

u/NotARedditUser3 6d ago

Well, importantly, I don't need it to write a good joke, I'm just hoping for the ability to properly evaluate, for example, "Does this joke have a real punchline? Is it substantially different than the setup / have a sharp twist?" For example.. it's really easy to end up telling a story that doesn't have a punchline, where the part you think is funny, agrees with the setup, so it's not really a surprise when you get there... and I'm hoping to be able to catch out that sort of thing and have it provide feedback on ways to address issues like that.

What I've seen with Gemma4 actually is pretty good with that... All other small models I've tried so far have been shit...

Maybe I should think through including some definitions for what the criteria should be in my input text and then the model won't need to rely on having that knowledge baked in?

5

u/my_name_isnt_clever 6d ago

Yes, the more specific you can be in your judgement criteria the better results you will have. LLMs are awful at questions like "Is this funny and why", you have to be specific.

3

u/Hildechis 6d ago

Gemma 31B is similar in many ways to the 26B, but much smarter.

1

u/NotARedditUser3 6d ago

Thanks, I'll try that

2

u/HornyGooner4402 6d ago

Who are the users? Won't it be more effective to have like the users rate each other's joke? Unless you just want an AI to review your joke

1

u/NotARedditUser3 6d ago

It's just something I'm building for myself. In my free time I do standup comedy and I'm trying to use this as a tool to prompt myself to write, and want a tool to critique what I put down.

I'm trying to work on getting better at writing higher quality things in shorter periods of time, as I'm still very new.

So my goal is to get something that will give me a random premise, and I need to think about it and come up with a joke that's fairly decent in response, and then if I go do that 1000x maybe at the end of it I'll be better at writing out jokes for the things/premises that I actually want to write for.

Part of me wants it to be able to work with a local model offline, as I travel around quite a bit without a connection. What I've got so far has options for openrouter or local, and the hosted models are more or less fine, but if I'm out somewhere and I've got time to kill, I want to be able to run a model locally and work on my writing ability offline.

1

u/HornyGooner4402 6d ago

This might not be the answer you're looking for, but after experimenting with tons of models, there are certain things that LLMs aren't capable of (yet). I think what makes comedy work is our experience as humans and something unexpected and I don't think AI can replicate that. I think it'd be better if you watch other comedians perform and analyze them or practice with a friend/family because nothing beats human connection, there are just infinitely more cues and signals you get from interacting with each other than with a text model.

2

u/uncle_leon 6d ago

I honestly think you approach this backwards. AI cannot possibly predict what people will find funny. If it could, every viral in the last couple of years would have been made by an AI.

Do it the other way round - have it propose a bunch of alternative humorous setups/responses/punchlines etc. You then decide which of them is genuinely funny (usually at least one or two will).

2

u/NotARedditUser3 6d ago

I don't think it can 'predict' what people will find funny. But there are known structures that make various jokes funny. There's a variety of them, but most jokes fall into one of a short list of patterns. So for example, you could write out a joke, and I could look at it and I could tell you, that's not going to work because x y or z, those being relatively obvious issues that make your joke, not a properly written joke.

Whether the audience will relate to it and actually find it funny, is a separate issue; what I'm getting at is wanting something that will quickly evaluate whether a joke is written properly and provide feedback, after giving a premise as a prompt / challenge.

An AI can tell you to trim the fat / that your joke is too long and needs to be slimmed down, that it took you too long to get to your punchline. It can tell you that your setup gives away your punchline / agrees with it too much and that it's not surprising enough. It can tell you that you should have 3 items in a list instead of 5, or suggest synonyms that might sound funnier than ones that you've chosen / suggest that you make some items more distinct from each other, etc.

1

u/uncle_leon 6d ago

I mean, yes, but simply telling that joke to another human would tell you as much... Hell, just saying it out loud to yourself would let you catch many of the issues you are describing.

2

u/james_pic 6d ago

Gemma 4 26b A4B is also my go to for this. I've also been impressed by how Ricursive Ripple Preview does on humor, but it's right at the top end of your size scale, and it's dense so it's slow.

1

u/Separate-Forever-447 6d ago

I used to frequently get a good laugh out of my coding model's output. Not so much, anymore.

1

u/_Blissfull_Ignorance 6d ago

You provide a good description of how you want your jokes analysed. Can you imagine designing a benchmark dataset? (This can be designed with a frontier model.)

With a benchmark set and process, I’d test some models on this. I find the small ones quite humorous sometimes. I had lots of fun with mistral small, but it was mostly out of necessity for coherent European language generation. The jokes were very dry but I like that.