One Trial, Many Trials

Assumption ledger

• parameterPopulation of N = 2000 students. Score without tutoring Y(0) ~ Normal(50, 10). Individual effect tau ~ Normal(5, 4), independent of Y(0). Y(1) = Y(0) + tau. The true ATE is the mean of tau in this population, about 5 points.
• distributionEach trial draws n students independently with replacement from the population. Exactly n/2 are assigned to treatment, by a random shuffle. The estimator is mean(treated outcomes) minus mean(control outcomes).
• distributionVar(estimate) = v1/(n/2) + v0/(n/2), where v1 and v0 are the population variances of Y(1) and Y(0). This is exact for this sampling scheme. SD = sqrt of that, which is proportional to 1/sqrt(n).
• simplificationNo dropout, no spillovers between students, full compliance, one outcome, and a single fixed population whose potential outcomes are all known (which is possible only in a simulation).
• simplificationThe break-it control makes treatment depend on baseline score: students are ranked by a noisy score of selection x (Y(0) - 50)/10 plus standard normal noise, and the top half are treated. At 0 this is pure randomization.
• dataNo external data. Every number is simulated in the R setup cell with set.seed(2024). Each trial set uses the seed on the page.
• interpretationUnbiased is read as: the mean of the estimator over repeated trials equals the ATE. It is not a claim about any one trial.
• interpretationThe model answer to the exit question uses the setup population's theoretical SD: about 3.3 points at n = 40 and 1.6 at n = 160. A miss of 4 points is about 1.2 SDs at n = 40 (common) and about 2.4 SDs at n = 160 (rare). Real schools do not know the true effect, so the answer also says the school should report a plausible range rather than one number.
• interpretationShared state: every button writes only to a state key (New seed to seed, Check my predictions to reveal, Compare with a model answer to exit_reveal, Lock/Unlock to locked). Both predictions and the written exit answer are listed as responses. The instructor's lock message is derived from locked; only the typed instructor code and a transient 'Code not recognised' notice stay local, because the code is a credential.
• interpretationInstructor lock: typing the code RCT (any case) in the corner Instructor box and pressing the button toggles the boolean `locked` in state. While it is true, the sliders, radio choices, display-size selector, buttons and written-answer box ignore student input. The code is visible in the page source, so this is a soft lock for classroom use, not security. Code-cell editors are platform-owned, so setting them read-only is best-effort.
• interpretationStyle: Chronicle Display is not on Google Fonts, so EB Garamond is used for headings. Neue Haas Grotesk is replaced by Helvetica Neue, Helvetica and Arial. Clay is used only for text, buttons and lines on White, Ivory and Bone, and Bone replaces it on dark backgrounds. Goldenrod appears only as the dashed truth line in dark mode. The instructor box uses Charcoal with white text and a Clay button with a white border.