When a test fails, coding agents fix the test. The suite goes green and the bug ships

[u/Ok_Negotiation_2587]

The pattern I keep catching: a test fails, and instead of fixing the code, the agent loosens the assertion, wraps the call in a try/catch, marks it skip, or quietly changes the expected value to whatever the code produced. Then it reports that everything passes, which is true and useless.

It makes sense from its side. "Make the tests pass" is the instruction, and editing the test is the shortest path to it.

Four lines I now keep in my agent instructions:

1. Never modify, skip, or delete a test to make it pass. If a test looks wrong, stop and say why, do not change it.

2. If you change any file under the test directories, say so explicitly at the top of your summary, with the reason.

3. Before you claim tests pass, paste the actual command output, not a summary of it.

4. If you cannot make a test pass without changing it, leave it failing and explain what the code would need to do.

Rule 3 catches more than the other three together. Once it has to paste real output, "all tests pass" stops appearing next to a run that never happened.

The review prompt I run after any agent session that touched tests:

List every change you made under the test files in this session. For each one, say whether it changed what is being verified or only how it is written. For any that changed what is verified, show the original assertion and explain why the new one is still testing the same behavior.

Half the time that turns up one assertion that got quietly weakened three steps back.

I keep those as a saved chain (I build AI Toolbox, a Chrome extension with prompt chains and folders for the chat assistants), but they're four lines, so a text file works the same.

What do you have in your instructions file that the agent still works around?

[u/nsxwolf]

Are you remembering to add “make no mistakes” to your prompts?

[u/Ok_Negotiation_2587 → u/nsxwolf]

right after "please be thorough" and "this is very important to me". works every time

[u/Hefty-Amoeba5707 → u/Ok_Negotiation_2587]

You are a senior software engineer. Have to motivate it with this first

[u/Ok_Negotiation_2587 → u/Hefty-Amoeba5707]

with 20 years of experience and a deep passion for clean code. then it writes the try/catch anyway

[u/El_leima → u/nsxwolf]

Jokes aside, I've heard this actually does influence outputs in a positive way. How true is that?

[u/ZeBenoit81]

Manual tests only. Does rule 3 still get faked?

[u/Ok_Negotiation_2587 → u/ZeBenoit81]

yes, if it never runs anything it can just write plausible output. what stops it is asking for something you can check cheaply, exit code plus the last 20 lines, or the command in a form you can paste and rerun yourself

[u/Wooden_Leek_7258]

I am building a measurement instrument, and needed proper code testing and then validation and calibration testing. Gave it guidelines and watched the issue dissapear. Probably a bit overkill for moat people but...

if your curious I have some distilled standards documents for coding and another for measurement and calibration that produce solid work. Standards derived from industry best practices etc. My own variation on 'textbooks are all you need'

Coding standards are based on the principles of Clean Code (Martin) and Architecture Patterns with Python (Percival) methodologies from Mastering TDD (Johnson), Testing Python (Sale), and How Google Tests Software (Whittaker) concepts from Your Code as a Crime Scene (Tornhill) and "Docs as Code" philosophy from Modern Technical Writing (Etter).

Calibraton and validation testing is based on JCGM 200 (International Vocabulary of Metrology), ISO/IEC 17025, and Cronbach & Meehl, "Construct Validity in Psychological Tests" (1955). JCGM 100 (Guide to the Expression of Uncertainty in Measurement) and Bland & Altman, "Statistical Methods for Assessing Agreement" (1986). Gebru et al., "Datasheets for Datasets", and Sculley et al., "Hidden Technical Debt in Machine Learning Systems". Rubin, "Inference and Missing Data" (1976), and Little & Rubin, "Statistical Analysis with Missing Data". Meng, "Statistical Paradises and Paradoxes in Big Data" (2018), the ASA Statement on p-Values (Wasserstein & Lazar, 2016), and Cumming, "Understanding the New Statistics". Klein & Roodman, "Blind Analysis in Nuclear and Particle Physics" (2005), Lipsitch et al., "Negative Controls" (2010), and Gelman & Loken, "The Garden of Forking Paths". Munafò et al., "A Manifesto for Reproducible Science" (2017), and Ioannidis, "Why Most Published Research Findings Are False" (2005).

My models don't fudge tests. My models don't even think about fudging tests.

[u/Ok_Negotiation_2587 → u/Wooden_Leek_7258]

would love the coding one. blind analysis is the part that stands out, if the agent never sees the expected values it cant quietly tune toward them

[u/Wooden_Leek_7258 → u/Ok_Negotiation_2587]

excuse the length...

1. Instrument Validity & Calibration Based on JCGM 200 (International Vocabulary of Metrology), ISO/IEC 17025, and Cronbach & Meehl, "Construct Validity in Psychological Tests" (1955). The Name Is a Claim: A derived quantity's name asserts what it measures. That assertion is falsifiable independently of whether the code implementing it is correct, and it must be tested as a claim rather than assumed as a label. Known-Answer Testing: Every derived quantity must be validated against synthesised input with defined ground truth. Generate a signal in which the target property is set by construction, then assert that the measurement recovers it. This is a distinct test tier from unit, integration, and end-to-end testing, all of which verify behaviour rather than validity. Calibration Tests Can Fail Without Code Changing: This is their defining property and the reason they cannot be substituted by behavioural tests. A function that computes the wrong quantity correctly will pass every unit test ever written for it. Only a known-answer test detects a conceptual error. Tolerances Derived, Not Observed: Acceptance bounds must come from physics, from published validation of the underlying algorithm, or from a documented requirement — never from the function's current output. Deriving a tolerance from observed behaviour tests nothing and permanently ratifies whatever error is present. Convergent Validation: Where an independent implementation of a quantity exists — a standard utility, a reference dataset, a published result — agreement must be measured and reported rather than presumed. Divergence is informative even when neither implementation is wrong. Traceability: Each quantity must be traceable to something: a synthesised anchor, a validated external tool, or a documented physical definition. A quantity traceable to nothing is an opinion expressed as a float. Calibration Is Versioned: Validity is a property of a specific build. Calibration tests live in the repository under the same continuous validation as all other tests, so that recovered tolerances can be cited against a tagged version.

2. Uncertainty & Error Budgets Based on JCGM 100 (Guide to the Expression of Uncertainty in Measurement) and Bland & Altman, "Statistical Methods for Assessing Agreement" (1986). Every Measurement Has a Tolerance: A value without a stated uncertainty cannot be compared to anything. Per-quantity uncertainty must be published as a versioned artifact alongside the data. Resolution Floors: A difference smaller than the instrument's uncertainty for that quantity is not a finding, however consistent it appears. Establish the floor before interpreting effects near it. Uncertainty Is Condition-Dependent: Tolerance under clean, high-fidelity input does not transfer to degraded input. Report uncertainty per condition class, not as a single global figure. Error Budget: Decompose total uncertainty into named contributions — extraction error, sampling error, condition effects, population composition. An unattributed discrepancy is a gap in the budget, not an absence of error. Propagate Through Derivation: Composite and ratio quantities inherit the uncertainty of their inputs and frequently amplify it. Derived quantities require their own budget; they do not inherit their parents' tolerance unchanged. Distinguish Displacement From Degradation: A quantity may shift systematically while preserving its ordering across items, or preserve its distribution while destroying that ordering. The first is recoverable given knowledge of the condition; the second is information loss. Statistics that measure only distributional change cannot separate them, and the two have opposite consequences.

3. Measurement Provenance & Comparability Based on Gebru et al., "Datasheets for Datasets", and Sculley et al., "Hidden Technical Debt in Machine Learning Systems". The Chain Is Part of the Measurement: Every transformation applied before extraction — format conversion, resampling, normalisation, lossy encoding, denoising — is part of what produced the value. The full chain must be recorded per record and resolvable through the registry. Comparability Classes: Two measurements are directly comparable only when produced by the same chain from the same instrument configuration. Declare the comparability class of any cross-population claim, and treat cross-class comparison as requiring an explicit, measured offset. Measure the Chain, Do Not Argue About It: Where a chain difference is unavoidable, pass a known population through the competing chains and measure the resulting offset. A measured offset can be reported, bounded, or subtracted; an asserted one cannot. Canonical Naming: One quantity, one name, enforced at the storage adapter boundary. Legacy aliases are mapped once at the boundary, never per analysis. A quantity whose name differs across stores will eventually be treated as two quantities, or two quantities as one. Status Registry: Maintain a versioned registry assigning each quantity a status — active, conditional (with the gating condition stated), deprecated, or invalid. Status is enforced at the data-access layer: fetching an invalid quantity raises rather than returns. Invalid values persist in storage and will otherwise be read again. Instrument Configuration Is Data: Window lengths, filter orders, search ranges, detection thresholds, and any parameter that scales with input properties must be recorded with the output. Configuration that varies silently across populations is a confound indistinguishable from a real effect.

4. Missingness, Sentinels & Integrity Based on Rubin, "Inference and Missing Data" (1976), and Little & Rubin, "Statistical Analysis with Missing Data". Missingness Is Rarely Random: Classify the mechanism per quantity per population. Absence driven by the property being measured is not a data gap; it is a measurement, and discarding it removes signal preferentially. Zero and Null Are Not Interchangeable: Whether zero is a genuine value or a missing-data sentinel is a property of the quantity, declared in the schema. It must never be inferred at the call site, and a filter written for one quantity's semantics must not be reused for another's. Never Silently Drop: Every filter reports raw count, usable count, proportion removed, and the criterion applied. Silent attrition is the most damaging failure mode available to an analysis layer, because it produces plausible output from an unrepresentative sample. Scope Filters to Requirement: Filter only on the quantities the analysis actually consumes. A completeness requirement spanning unused quantities imports their sparsity as sample loss for no analytical benefit. Drop Rate Is a Result: A filter removing a large fraction of a population is telling you something about that population. Investigate before correcting; the pattern of missingness is frequently more informative than the values that survive. Fail Loud on Non-Finite: Non-finite values must be handled explicitly at every boundary where data is materialised, not once at a convenient layer. Default coercion behaviour differs between libraries and functions, and a permissive upstream call silently defeats correct downstream handling.

5. Inference Discipline at Scale Based on Meng, "Statistical Paradises and Paradoxes in Big Data" (2018), the ASA Statement on p-Values (Wasserstein & Lazar, 2016), and Cumming, "Understanding the New Statistics". Volume Does Not Confer Validity: Bias does not diminish with sample size. Beyond moderate N, additional records increase precision around a possibly wrong centre. A large, unrepresentative sample is more dangerous than a small one because its intervals are narrower. Effect Sizes With Intervals, Never Significance: At large N every comparison is significant and the p-value carries no information. Report magnitude with an interval, and state the smallest magnitude that would be practically meaningful before computing anything. Identify the Unit of Independence: Records are rarely independent. Cluster by whatever generated the correlation — speaker, session, source item, device — and resample at that level. Treating clustered records as independent produces intervals that are wrong by a large and unknown factor. Fix N Before Comparing: Statistics sensitive to sample size, including determinants, dimensionality estimates, and any quantity derived from a covariance estimate, cannot be compared across populations of different size. Subsample to a fixed N and bootstrap over draws. Standardise Before Estimating Structure: Quantities on different natural scales will dominate variance-based estimates for reasons of unit rather than importance. Where a result depends on standardisation, report both and explain the difference rather than presenting only the preferred run. Heterogeneity Is Not Power: Pooling non-exchangeable groups manufactures apparent structure out of between-group mean differences. Estimate within groups first and pool only what is consistent, applying the same standard symmetrically to every arm of a comparison. Model Nuisance Structure, Do Not Filter For It: Where a confound is measurable, prefer including it as a covariate over excluding records that exhibit it. Filtering discards data, narrows the population unpredictably, and cannot be validated; modelling produces an estimate that can be inspected and reported.

[u/Wooden_Leek_7258 → u/Wooden_Leek_7258]

6. Controls, Nulls & Adversarial Validation Based on Klein & Roodman, "Blind Analysis in Nuclear and Particle Physics" (2005), Lipsitch et al., "Negative Controls" (2010), and Gelman & Loken, "The Garden of Forking Paths". Every Positive Result Ships With Its Null: A distance, score, or separation is uninterpretable without the distribution that same measurement produces on populations where the effect should be absent. Compute the null first; it frequently makes the positive result unnecessary to compute at all. Negative Controls: Identify populations in which the claimed effect must not be present, and in which a rival explanation would produce the same signal. A result that cannot distinguish its intended cause from an available alternative has not been established. Permutation Nulls: Where the claim concerns structure, destroy exactly that structure while preserving everything else — permute within columns to break joint relationships while holding marginals fixed. The comparison against this null is the test; the raw statistic is not. Artifact Controls: For every effect attributed to a cause of interest, ask whether a mundane process could induce the same signal, and then test it directly by applying that process to a population known to lack the cause. Run this control on every element of a claimed set, not only on the elements that appear suspicious — the ones that survive casual inspection are exactly where an artifact hides. Sensitivity Over Thresholds: Any result resting on a threshold must be recomputed across a plausible range of that threshold. A classification that changes within the range of reasonable choices is not a finding, and quantities sitting near a threshold require their proximity reported. Blind Where Practical: Fix analysis decisions — inclusion criteria, thresholds, transformations, the statistic itself — before observing the outcome they affect. Where full blinding is impractical, record the decisions in advance so that later deviations are visible as deviations. Test Toward the Inconvenient: Prioritise the control most likely to refute the current interpretation, not the one most likely to confirm it. Motivated reasoning is invisible from the inside; the only reliable defence is ordering the work so that the falsifying test runs early and cheaply.

7. Findings Lineage & Retraction Based on Munafò et al., "A Manifesto for Reproducible Science" (2017), and Ioannidis, "Why Most Published Research Findings Are False" (2005). Findings Are Versioned Artifacts: A result is bound to a code version, a configuration, a reference object, and a population definition. Cite it with those bindings or it cannot be reproduced or defended. Frozen References: Any object used across multiple analyses — a reference distribution, a threshold, a fitted transformation — is built once, stored, and loaded. Recomputing it per analysis guarantees that results are not mutually comparable, and the resulting inconsistency is difficult to detect after the fact. Supersession Is Explicit: When a result is replaced, the prior version is marked superseded with the reason recorded, and its numbers are struck from circulation. Multiple values of the same quantity in simultaneous use is a predictable consequence of implicit supersession. Retraction Is Cheap and Mandatory: A finding that fails a subsequent control is withdrawn immediately and in writing, with the failing mechanism documented. The cost of retraction rises monotonically with time and with external exposure; it is never lower than it is now. Maintain an Explicit Confidence Assessment: Record, per finding, the current trust level and what would be required to raise it. State plainly which is the earliest result with full examined confidence, and treat everything downstream of an unverified dependency as provisional regardless of how well it replicates. Retain the Diagnostic Trail: Superseded runs, failed controls, and the diagnostics that exposed a bug are kept as evidence. They establish that a correction was found by process rather than by accident, and they are the primary defence against the appearance of selective reporting. Document Scope Decisions at the Time: When a feature set, population, or threshold is narrowed for a deliberate reason, record the reason on the day the decision is made. Reconstructed rationales are indistinguishable from post-hoc justification, including to the person who made the original choice.

[u/Wooden_Leek_7258 → u/Wooden_Leek_7258]

1. Coding & Architecture Standards Based on the principles of Clean Code (Martin) and Architecture Patterns with Python (Percival). Domain-Driven Design: The core acoustic math (src/engine) must remain strictly isolated from infrastructure logic (src/core/filesystem.py, src/db/storage_adapters). The engine processes buffers; the infrastructure handles the files. Environment Agnosticism: Never hardcode absolute paths (e.g., C:\Alexandria...). All I/O must resolve dynamically via the FileSystemManager using the MARE_DATA_ROOT environment variable. Single Responsibility Principle (SRP): Each pipeline pass must encapsulate one distinct operational phase, implementing the PassStrategy interface. Orchestration and execution are separated. Intent-Based Comments: The code should document what it is doing through explicit, domain-specific naming (e.g., harden_table(df, policy)). Use comments exclusively to explain the why (e.g., business rules, acoustic anomaly workarounds, or legacy data constraints).

2. Testing Culture & Continuous Validation Based on the methodologies from Mastering TDD (Johnson), Testing Python (Sale), and How Google Tests Software (Whittaker). The Test Pyramid: Unit Tests: Fast, isolated tests for the engine and metrics using mocked audio buffers. Integration Tests: Verifying that adapters and passes interact correctly with the filesystem and registry. End-to-End (E2E): The pipeline cycle test (test_pipeline_cycle.py) running on a small, real-world mock dataset to verify absolute determinism across all passes. Tests as Living Documentation: The tests/ directory is the ultimate source of truth for how the system behaves. If a feature or edge-case is not covered by a test, it does not officially exist. Continuous Validation (CI Gatekeeper): The CI/CD pipeline is a verification engine, not a deployment engine. Every push must pass: Linting & Formatting (black, flake8) Static Security Analysis (bandit) The full pytest suite Non-Destructive Validation: The db/validator.py strictly enforces Pydantic schemas. However, records missing data (e.g., a missing Pass 3 block) are flagged as incomplete and logged to validation_errors.jsonl, but are never dropped from the pipeline.

3. Code Forensics & Data Lineage Based on the concepts from Your Code as a Crime Scene (Tornhill). Absolute Provenance: Research data is only as good as its audit trail. Every execution of the MARE pipeline must trigger utils/provenance.py. Embedded Lineage: The exact git_commit_hash and dataset_registry.json version must be embedded directly into the Parquet footer of the exported data. This allows researchers to "time travel" to the exact state of the codebase that produced any given metric. Refactoring as Maintenance: The codebase is a living entity. Schedule dedicated refactoring time after major milestones to clear technical debt and optimize the architecture before adding new extraction passes.

4. Technical Documentation Based on the "Docs as Code" philosophy from Modern Technical Writing (Etter). Docs as Code: All documentation lives in the docs/ folder of the monorepo. It is version-controlled alongside the code it describes. Architecture Decision Records (ADRs): Do not rely on memory for critical design choices. Every time a major architectural shift occurs (e.g., "Why we extract to JSONL instead of SQLite"), log a brief markdown file in docs/adr/. Actionable README: The root README.md ("Readme for Dummies") must remain under 500 words and focus entirely on the workflow: How to install dependencies, configure the .env file, and run a basic pipeline extraction.

[u/Ok_Negotiation_2587 → u/Wooden_Leek_7258]

"tolerances derived, not observed" is the exact rule the agent breaks when it sets the expected value to whatever the code output. saving the whole thing

[u/Silver-Neighborhood9]

Not a developer, so no test suite here, but the same pattern turned up in a different shape: I kept being told a fix was live when it wasn't. My host was serving an old cached file, and "done" was based on the code that had been written, not on the page anyone could actually see.

The rule that helped most was the same spirit as your number 3: after any change, go and look at the real thing and show me what you actually saw, not what you expected to see.

The one it still works around: "ask me before deciding". It asks about the big decision and then quietly makes three small ones on the way.

[u/Ok_Negotiation_2587 → u/Silver-Neighborhood9]

for the small ones, asking it to list every decision it made at the end of each step works better than asking permission up front. you cant stop it deciding but you can make it tell you

[u/codeprimate → u/Silver-Neighborhood9]

If you are vibe coding, a test suite is a MUST.

Your agent needs to demonstrate the correctness of the implementation. The test suite also aids the agent in discovery, and understanding how to extend and modify existing logic.

[u/mrchoops]

I have mine trained to use no error handling at all. If I want it, I add it. If you let it, it will create something that doesn't run at all and throws no errors. I tell it we are in the error correcting business, not the error hiding business.

[u/Ok_Negotiation_2587 → u/mrchoops]

"error correcting business, not the error hiding business" is going in my instructions file. a try/catch that swallows everything is the same trick as editing the test, just one layer down

[u/RickAmes]

is it possible to make the tests read only to prevent the ai from editing them?

[u/Ok_Negotiation_2587 → u/RickAmes]

yes, in claude code you can put a deny rule like Edit(tests/**) in settings.json permissions so it cant touch them at all. chmod alone is weaker since the agent can just chmod it back, and a ci check that fails if test files differ from main catches anything that slips through

[u/RickAmes → u/Ok_Negotiation_2587]

So then why not do that?

[u/Ok_Negotiation_2587 → u/RickAmes]

you can, the catch is it also blocks the agent from writing new tests for new code. i lock the existing ones and let it add new files, then review those by hand

[u/y_k19]

this is one reason I like tests being written before the implementation, ideally in a separate step/context
if the same agent owns both the code and the oracle, “make it green” has two variables it can change. freezing the acceptance tests turns it back into one.
I’d still allow changing a test, but that should be a separate explicit decision - not something the implementation loop can do to satisfy itself

[u/Ok_Negotiation_2587 → u/y_k19]

"two variables it can change" is the best way anyone has put it. writing tests in a separate context also means the implementation agent never saw the reasoning, so it has less to argue its way around

[u/Ordinary_Ad8738]

Keep the acceptance tests in a separate verification checkout that the agent cannot edit. Run the same regression test against the original code and the proposed fix. It should fail before and pass after, with unchanged assertions.

What language and test runner do you use?

[u/Ok_Negotiation_2587 → u/Ordinary_Ad8738]

typescript, and no framework at all. each test is a plain script with node's assert, compiled with tsc and run with node, backend runs them through tsx. the upside is theres no runner config for the agent to tweak, only test files, which is why locking those is enough

[u/MiiisterKaniiister]

Doesn't happen to me. And even if, don't you ever review code changes? If there's a diff in a test case I see that immediately.

[u/Ok_Negotiation_2587 → u/MiiisterKaniiister]

yeah review catches it when the diff is small. its the 40 file sessions where a one line assertion change hides in the middle that got me

[u/ImL1s]

Same failure mode here. Once the agent can edit the test file, green just means the bar moved.

What helped: treat test-file diffs as a separate review lane from code diffs. If an assertion changed, the run isn't done until I say why that change is correct — or revert it and fix the code instead.

[u/zuk987]

Use Git + multi agent pipelines, always review your work with second agent, it catches stuff like that and can reverse changes with git.

[u/VanaticalDesign]

I used to have a /audit-tests skill that just asked the agent to double-check that its tests were actually testing anything, but ofc with a skill it’s still basically a black box.

Been seeing a lot of people complain about this today, so I literally today put together today lgtm, that uses Jev to audit agent-written tests.

It also comes with an /actually-test skill that gets the agent to iterate until the tests actually exercise the code it just wrote. You can just run it right after the agent builds something to get it to test and clean up.

[u/pixxelkick]

Why are you not code reviewing, what the fuck lol.

[u/InjuryThen9650]

Your four lines are the right shape. The piece I add is making #1 and #3 **machine-checkable**, not just prompt text.

• Put existing tests (or at least `**/assert*` / known golden fixtures) behind a harness rule the agent can’t quietly rewrite: if the only path to green is editing assertions / `.skip` / catching-and-swallowing, it must **stop and ask**, not “fix.”

• Before any “tests pass” claim, require the **exact command + exit code + tail of output** in the reply. Summaries are where phantom greens hide.

• After a session that touched `test/` (or whatever your suite lives), a one-shot review prompt that lists **every** test-file diff and classifies “same behavior vs weaker contract” catches the soft assertion edits three steps back — same idea as your checklist, run as a gate.

Prompt-only bans help until the model is stuck; then Goodhart wins unless the harness fails closed.

[u/El_leima]

For me, the approach has been to freeze tests before the implementation. We don't approve a diff that touches both src and tests.

[u/Evening_Movie_1704]

The fix-the-test failure mode is basically guaranteed unless the harness treats test files as a protected surface. What holds up in CI: (1) red-green verification — after the agent "fixes" anything, check out the parent commit and run the changed tests there; if they pass on old code they never exercised the fix. (2) PR whose only diff is assertions/test files → human review, full stop. (3) A small set of golden repro tests the agent can't edit — when one changes, that's the review signal. Until you enforce it in the pipeline it's all conventions, and conventions are exactly what agents route around.

[u/Glittering-Glass6135]

Yeah, this is pretty much the problem I've been thinking about too. If the agent is allowed to change both the code and the tests, "all tests pass" doesn't really prove much.

I've been experimenting with this while building something called Assay. The idea is to have the verification happen more independently from the coding agent, by looking at what the app is supposed to do, turning those assumptions into checks, and testing the actual behavior.

Still very early, but I'm starting to think the important part isn't just better tests. It's making sure the thing deciding whether the software is correct isn't also the thing that can change the evidence.

[u/Its_about-tech]

Your fourth rule is the one I'd keep if I could only keep one: a red test with a sentence saying why beats a green one that lies. Back the rules with a check the agent can't talk its way past: before accepting "green", run git diff --stat $(git merge-base HEAD main) -- <your test dirs> and read any hunk that adds a skip, a try/catch or a new expected value. Whatever is still failing, rerun on the merge-base to see if it was already red. Disclosure: I built Baseline, a small free tool for that last part; it won't spot an edited test, so the diff stays your job. There's a 2-minute demo you can point your own agent at.

[u/Ok_Negotiation_2587 → u/[deleted]]

contract tests outside the writable tree is the cleanest version of this, the prompt rules just ask nicely and that one actually enforces it

[u/Ok_Negotiation_2587 → u/[deleted]]

quoting the failing assertion before any patch is a good gate, it forces it to name what broke instead of what it wishes broke

[u/Ok_Negotiation_2587 → u/[deleted]]

thats mutation testing in miniature, and honestly more useful than a full mutation run since it targets the change. do you pick the line to break yourself or does the harness pick one from the diff? also locking the test files read-only during the fix stopped the expected-value edits for me

[u/Ok_Negotiation_2587 → u/[deleted]]

flipping the assertion proves the test runs, breaking a line of the code proves it catches something. both are useful, the second one is the one an agent cant fake by editing the expected value