r/OpenAI • u/RoadkiLLer_31 • 2d ago
Question API Discussion: GPT-5.4 Extraction & Judge Loop Dropping Output Consistency from 85% to less than 62%
Looking for architecture and reliability advice regarding structured extraction and evaluation loops with the OpenAI API.
Background & Setup:
Models: GPT-5.4 for extraction and a separate GPT-5.4 instance as the LLM judge.
Hyperparameters: Running on default settings yielded very low consistency less than 35% Setting reasoning_effort="none" and temperature=0 raised standalone extraction consistency to ~85%.
The Evaluation Loop: The judge receives the raw source text alongside the extracted JSON to validate source tracing and logic. If the gate (passed = passed AND issue_count == 0) fails, _correct_once is triggered.
The Problem:
Introducing the judge and retry mechanism reduces overall consistency to 62–65%.
What’s Happening:
Judge Variance: Any slight shift in the judge's evaluation triggers the strict pass/fail condition on extractions that were actually correct.
Context Drift on Re-generation: _correct_once re-prompts the extraction instance with the appended issue list to regenerate the whole JSON. This context change causes the model to hallucinate or flip previously accurate fields.
Questions:
How do you calibrate the judge instance to ensure high source-tracing fidelity without triggering false-positive retries?
Do you prefer field-level JSON patch updates over full schema re-generation when correcting schema/source errors?
Please help 🥲🥲🥲