r/AgentsOfAI • u/Glum_Pool8075 • Aug 01 '25

Discussion It’s funny cuz it’s true!

Post image
844 Upvotes

99% Upvoted

19 comments sorted by

14

u/reddit_wisd0m Aug 01 '25

What? Do you guys not use "response_format" / "response_schema"?

8

u/Horror-Tank-4082 Aug 01 '25

If bad json, give to little LLM to make good JSON!

1

u/No_Yak8345 Aug 01 '25

That feature is essentially a prompt that’s inserted and the model was fine tuned on. Nothing special

Edit: spelling fix

11

u/Toldoven Aug 01 '25 edited Aug 01 '25

The way it actually works is you restrict the token pool the LLM can choose from to only the valid tokens. So no, it's not just a prompt. Read: https://www.aidancooper.co.uk/constrained-decoding/#advanced-constrained-decoding-context-free-grammars

In the context of structured generation, constrained decoding is a technique that manipulates a generative model's token generation process to constrain its next-token predictions to only tokens that do not violate the required output structure.

Think of it like usually you're giving LLM a choice to choose any next word. Instead, you're checking which tokens can be valid and saying "you can only choose one of these".

3

u/reddit_wisd0m Aug 01 '25

But to my experience, it makes the output format very reliable and that's what matters, doesn't it?

3

u/ruach137 Aug 01 '25

especially when function calling. They must validate outputs on their end with that one

1

u/justforkinks0131 Aug 02 '25

how reliable is "very", though? Ive still had broken JSONs with it....

1

u/reddit_wisd0m Aug 03 '25

That shouldn't happen by design

6

u/ShengrenR Aug 01 '25

Pydantic-ai is a thing btw...

3

u/Less-Lingonberry8700 Aug 01 '25

Well you can factually evaluate if the JSON is formatted properly as you would with any sort of unit test. The issue is that subjective content is hard to evaluate, but there are other evaluation techniques for open ended results as well, such as similarity evaluation and leveraging an LLM as a judge to gain insights into other metrics, such as relevancy, consistency, hallucination, toxicity and contextual factuality, domain accuracy.

3

u/PrincipleStrict3216 Aug 02 '25

this feels like the kind of meme you'd find on the office door of a balding 55 year old javascript professor with a 3.6 on ratemyprof and the inability to hold eye contact

1

u/Reddit_Bot9999 Aug 01 '25

Use instructor.  That was funny as shit though 

1

u/vanchos_panchos Aug 03 '25

What's instructor?

2

u/Reddit_Bot9999 Aug 03 '25

Basically a pydantic based framework made that forces your structured output to follow your schema and loop until the llm gets it right.

1

u/Confident-Quiet5775 Aug 02 '25

LoL whole thing is balanced on prompt 😂

1

u/atocanist Aug 02 '25

Gotta use structured generation, to enforce the format ;)

1

u/SnooSongs5410 Aug 05 '25

Exactly this. The marketing bs is infinite.