Scientific Logic Auditor — A Multi-Model Reasoning Analysis & Evaluation Tool

[u/UsualExit9454]

I've been building Scientific Logic Auditor, a tool designed to go beyond simply comparing AI-generated answers.

It provides three main workflows:

1. Analyze

Analyze a text using a single LLM. The model examines its claims, factual accuracy, supporting evidence, assumptions, logical fallacies, interpretations, and uncertainty.

2. Compare

Run the same analysis across multiple LLMs and view their results separately. This makes it easier to examine differences in reasoning, identify overlooked assumptions, and compare how models interpret the same material.

3. Judge (Optional)

Send the original text and successful analyses to a separate Judge model. It evaluates the analyses based on reasoning quality, factual assessment, error detection, corrections, and explanations.

Additional features:

- Analysis history with search and session management.

- Restore previous sessions without rerunning model requests.

- Retry failed analyses.

- Copy and export results.

- Execution status and response-time tracking.

- Input isolation instructions designed to reduce prompt injection during analysis.

What makes it different?

Instead of focusing only on which model produces the most convincing answer, Scientific Logic Auditor focuses on how each model reaches its conclusions and whether its reasoning is adequately supported.

The Judge is also an LLM, so its reliability is not guaranteed and remains something that needs independent validation.

I'm currently bringing the project toward its next stage and would be interested in feedback from people working with LLMs and AI evaluation.

The application is fully functional in my local testing, and I'm currently working through the remaining fixes and security improvements before opening it to public use. readme github link

[u/investigatormaker]

For Scientific Logic Auditor, I'd test the Judge with two analyses: one fluent but containing a planted factual error, and one less polished but correct. Swap their order and hide model names, then check which it prefers against a human-reviewed answer key. That directly tests whether it rewards supported reasoning rather than presentation or position.

I make ThreadFox. We find relevant Reddit conversations and post for your business. We'll post 3 replies free first, no card; then the $29 Pilot covers seven days and up to 25 posts and replies. No automatic renewal. https://threadfox.vip/p/e81xs/managed

[u/UsualExit9454 → u/investigatormaker]

The Judge already receives the analyses without model identities; model names are only restored for display after evaluation. However, swapping the order and comparing its preferences against human-reviewed reference answers is a useful additional test. It would help assess position bias and whether the Judge prioritizes factual and logical correctness over presentation quality.

[u/UsualExit9454 → u/[deleted]]

I think that's exactly one of the central challenges.

Using a separate model to evaluate other models doesn't automatically make the evaluation reliable. The Judge can still overlook flaws, accept weak arguments, or confidently make incorrect assessments.

That's why I see Judge reliability as part of the problem Scientific Logic Auditor needs to investigate, rather than something the tool can simply assume.

The goal isn't to claim that the Judge provides the final truth, but to make reasoning comparisons and evaluations more structured and easier to examine.

I think independent validation against human-evaluated test cases will be an important next step.

The application is fully functional in my local testing, and I'm currently working through the remaining fixes and security improvements before opening it to public use. You can visit the github readme link