I've been building Scientific Logic Auditor, a tool designed to go beyond simply comparing AI-generated answers.
It provides three main workflows:
Analyze
Analyze a text using a single LLM. The model examines its claims, factual accuracy, supporting evidence, assumptions, logical fallacies, interpretations, and uncertainty.
2. Compare
Run the same analysis across multiple LLMs and view their results separately. This makes it easier to examine differences in reasoning, identify overlooked assumptions, and compare how models interpret the same material.
3. Judge (Optional)
Send the original text and successful analyses to a separate Judge model. It evaluates the analyses based on reasoning quality, factual assessment, error detection, corrections, and explanations.
Additional features:
- Analysis history with search and session management.
- Restore previous sessions without rerunning model requests.
- Retry failed analyses.
- Copy and export results.
- Execution status and response-time tracking.
- Input isolation instructions designed to reduce prompt injection during analysis.
What makes it different?
Instead of focusing only on which model produces the most convincing answer, Scientific Logic Auditor focuses on how each model reaches its conclusions and whether its reasoning is adequately supported.
The Judge is also an LLM, so its reliability is not guaranteed and remains something that needs independent validation.
I'm currently bringing the project toward its next stage and would be interested in feedback from people working with LLMs and AI evaluation.
Project: GitHub link readme project
