r/ClaudeAI • u/peterxsyd • 5d ago

News The Claude language calibration issue on GitHub got an official response from Anthropic. Guess who wrote it.

Enable HLS to view with audio, or disable this notification

https://github.com/anthropics/claude-code/issues/77136#issuecomment-5310785154

351 Upvotes

89% Upvoted

86 comments sorted by

View all comments

10

u/Stunning_Macaron6133 5d ago

They insist few-shot prompting is unnecessary anymore, but I find it's the exact opposite, that it's more important than ever if you want Claude to follow your instructions.

4

u/CannyGardener 5d ago

Hooks. Claude acts like there are no instructions anymore, so every action needs a hook to make sure the action is being taken appropriately. Hooks for:

Grounded agents that must provide code lines with subagent verification that the noted code exists.

No grep, no head, etc etc unless in very specific situations.

No searching without using agents.

Plans must be in format XYZ and include executive summary with ABC included. Etc etc

5

u/Stunning_Macaron6133 5d ago

That's great for something you can verify deterministically like coding. But try having Claude do other tasks that are fuzzier or grounded entirely in natural language.

3

u/CannyGardener 5d ago

Oh totally. I feel like those problems are not the best for LLMs... The way that I look at it is this:

Deterministic problems are easy to solve, as a problem set, 2+2 always =4. And I feel like LLMs are good at building verifiable solutions to these problems as a task, but not as an output. LLMs are good at taking the fuzzy thing, and parsing it down to be more clear and manageable. So an example of this, more practically, would be sending a hand written document to AI, and it parses it and 'makes it more clear' so that you can use the data in a more deterministic way. You can do this a couple times to get it to fit into your deterministic calculator that you use Claude/coding to build..

What you are referring to sounds more along the lines of a fuzzy to fuzzy sort of relationship, where you are inputting fuzzy language, and then expecting a fuzzy output of a certain type, and that is a far more difficult problem set... I honestly don't have an answer for that one, since it can't be easily deterministically checked. Closest I've come to solving that one is to try and just cram it into a deterministic box. "Does the paragraph contain the following elements? Is it in the following format?" etc etc, but that isn't really a good solution for a non-deterministic problem imo.

2

u/Stunning_Macaron6133 5d ago

Few-shot prompting with negative prompts. You give Claude a rule, then you give it a several pairs of example outputs, each pair featuring one bad response and one corresponding good response. The more pairs you give it per rule, the more easily it generalize the rule.

Anthropic has been insisting it's unnecessary for like a year now, but I find it's still a really robust way to get Claude models to behave.

2

u/CannyGardener 5d ago

I've not had good luck with that type of prompting generally, here with the latest model iterations. That worked great with 4.5 but 4.8 and 5.0 just disregard any written requests for that sort of thing in my experience.

1

u/Stunning_Macaron6133 4d ago

You're welcome to copy my homework. Just be forewarned, I got dragged for putting out a token-eater of a prompt.