How the IDE MVP works
What it is
The MVP runs the plan's core loop end to end on five simulated difference-in-differences studies: the schema rows, a project manifest, a deterministic matcher, verified checks, findings the researcher confirms, Claude's case-specific reading of each fired row, and a ledger of decisions. It runs as a local web dashboard where rows change state live as facts and findings come in.
Scope: Diff-in-Diff only, 11 check rows, simulated data. Staggered rollout is left out.
The schema rows
The MVP uses 18 rows of the schema spreadsheet, copied into schema/mvp-rows.xlsx and reviewed cell by cell; the original spreadsheet is unchanged.
| Rows | Topic |
|---|---|
| I, I.A, I.A.1 | Covariate balance |
| I.A.2, I.A.2a, I.A.2b | Pre-trends and anticipation |
| II, II.B, II.B.1 | Rival policies |
| II.D, II.D.2 | Unit-specific trends |
| II.K, II.K.1–3 | Anticipation remedies |
| II.N, II.N.1–2 | Baseline level differences |
Main conventions adopted: tokens are identifiers spelled identically everywhere ([needs: baseline_predetermined_covariates]); a new Possible findings column lists what each check can produce; conditions are joined with AND and written as tokens, not prose. Every change and every open question is in the change log.
How a run works
- Facts are declared: design facts and the data inventory, keyed by the schema's
[needs: …]names. - The matcher sets each row's state, by rule: doesn't apply, greyed (a needed input is missing), ready (a check can run), or fired.
- The researcher approves a ready check and it runs through the DSL.
- The check proposes findings; the researcher confirms or rejects them.
- Confirmed findings fire the rows waiting on them, e.g. an anticipation finding fires II.K.1–3.
- Claude writes what each fired row means for this project. It sees the row, the manifest and the check results, never the data, and never decides whether a row applies.
The DSL
A check is a plain-English statement filled in from the manifest, e.g. "event study of y by town and year, against the 2014 announcement and the 2015 implementation." Each statement is bound to a tested routine (fixest does the statistics), with the package version it was verified with and a verified-on date. Each run produces a named result carrying its full lineage: statement, binding, the code that ran, input file and time. Nothing writes code during a run.
The five test cases
Each case is a simulated panel of 400 towns over 2010–2019 with one planted flaw, and an answer key listing the findings and the state of each check row.
| Case | Planted flaw | Findings | What they fire |
|---|---|---|---|
| 00 clean | none | pre_trends_pass |
I.A.2a |
| 01 pre-trend | treated towns already growing faster | pre_trends_imprecise_or_violated |
I.A.2b, II.D.2 |
| 02 anticipation | towns react at the announcement | pre_trends_pass, anticipation_pattern_in_leads |
I.A.2a, II.K.1–3 |
| 04 baseline gap | treated towns start higher, same trend | large_baseline_outcome_gap_between_t_and_c, pre_trends_pass |
I.A.2a, II.N.1–2 |
| 05 rival policy | second policy at the same time, no data on it | pre_trends_pass |
I.A.2a; II.B.1 greyed out |
Cases 04 and 05 are traps: a gap that looks alarming but isn't, and a flaw the data can't reveal. The IDE matches every answer key exactly.
What the researcher decides
- Approve each check before it runs.
- Confirm or reject each proposed finding.
- Mark each fired row addressed, or dismissed with a reason.
- Sign off on
[trade-off]rows (II.D.2, II.N.2) before marking them addressed.
The baseline comparison
Claude working without the IDE got the same 18 rows, the manifest, and a tool to run its own R or Python on the data, and was scored against the same answer keys. In one run per case it also scored perfectly, for $0.42 in total. With 18 clean rows and single-flaw cases the comparison shows no difference; the Sushant test's failures came with the whole schema in context, which this setup doesn't test.
Not built yet
- Claude extracting manifest facts from a draft for the researcher to confirm.
- Staggered rollout and the other strategy columns.
Open questions are in the change log.
Using the dashboard
Layout. Manifest on the left. Schema rows in the middle, grouped by critique; each card shows the row's state, its tokens (green when met) and what it is waiting on. Findings inbox, ledger and event log on the right.
Controls. Case picker, Reset, Step (one event), Play (steps until a decision is needed), Claude explanations (on/off), and "show answer key", which marks each row ✓ or ✗ against the key.
A run through case 02 (anticipation):
- Pick
02_anticipationand press Play. The manifest fills in, and I.A.1, I.A.2 and II.B.1 turn ready and run. (A ready check can also be run from its card with "Approve & run".) - Play stops at a red bar, "Waiting for you: confirm or reject…", with two findings in the inbox:
pre_trends_passandanticipation_pattern_in_leads. - I.A.2's "lineage" shows the event-study numbers and the exact code that ran. Confirm both findings.
- I.A.2a and II.K.1–3 fire. Each fired row gets Claude's reading for this case; "what Claude saw" shows its prompt.
- Mark each fired row Addressed, or Dismiss it with a reason. A trade-off row needs a sign-off note first (case 01 has one, II.D.2).
The event log records every step and which rows each one changed.