The benchmark protocol
A reviewer- and product-readable specification of what TargetSpace scores: the forecast unit, the sealed walk-forward loop, the scoring stack, and the reporting contract. This is the protocol a submission must satisfy.
1 · The forecast unit
Every scored item is one forecast unit: a sealed probabilistic prediction about a single future state of a single target, resolved later by a deterministic rule.
forecast unit = ( i , E≤t , q , A , r ) → sealed probability vector over A
Plain-language example: a note-taking app claims its memory helps its users. TargetSpace asks it to forecast — before the outcome exists — whether a user will complete, defer, cancel, or replace a recurring commitment. The forecast earns credit only if it beats population rates, beats that user's own routine, stays calibrated, and fails when scored against another user.
2 · The sealed walk-forward loop
One pass over a target produces one reporting row. The loop is strictly prospective: each forecast is sealed before its outcome exists, then scored against externally observable resolutions.
The four-bar decision rule. A forecast earns target-specific credit only if it (1) beats the R1 population prior, (2) beats the R2 own-routine baseline, (3) stays calibrated, and (4) loses skill under matched target permutation. Miss any bar and the credit is not target-specific — the result reduces to generic prediction or routine replay.
3 · Scoring
Scoring is applied to the full sealed probability vector, not to cherry-picked accuracy. Skill is reported relative to two baselines, gated by calibration and permutation, and attributed across evidence tiers.
Log score (bits)
The primary proper scoring rule over the full vector. Higher is better; expressed in bits so skill differences read directly.
Brier score
A secondary proper score over the vector, reported alongside the log score.
Skill vs R1 (bits)
Bits gained over the population prior. Rules out looking good by predicting what usually happens; the entry condition.
Skill vs R2 (bits)
Bits gained over the target's own routine. Rules out routine replay: skill over R2 is skill the routine does not already contain — the headline number.
Calibration diagnostics
Calibration intercept and slope plus a reliability read. ECE is a coarse diagnostic at small n, not a hard gate.
Permutation specificity gate
Forecasts re-scored against the wrong target must lose their skill. If skill survives permutation, target-specific credit is withdrawn.
Evidence-tier ablation
The marginal lift of each evidence tier, measured by removing streams — what each stream buys, never assumed.
What a score does and does not certify. A high score certifies calibrated prospective predictive skill only — evidence that the system can maintain and use a target-specific predictive representation on sealed calibrated forecasts. It is not evidence of understanding, inner life, or causation, and it is never permission to act on a person.
4 · The hard inclusion rule
A target state is scored only if it has a pre-registered observable resolution rule; otherwise it is evidence, not a scored state. Attention, affect, and inferred goals are always evidence — never scored states. Self-report may be evidence, but it is never the outcome label when it is also a model input. Causal claims require intervention and are outside the default benchmark.
5 · Valid vs invalid target states
The inclusion rule, made concrete. Only states with a deterministic observable resolution rule are scored; everything else is admissible as evidence or excluded.
| Candidate target state | Category | Verdict | Why |
|---|---|---|---|
| P(no substantive reply to message m by r) | Observable outcome | Scored | A deterministic observable rule reads the thread at r and assigns the outcome. |
| Commitment resolves as {complete, defer, cancel, replace} | Observable outcome | Scored | Discrete answer space; each outcome is externally observable at r. |
| “Feels anxious” | Affect | Evidence only | An affective state with no pre-registered observable resolution rule. Usable as evidence, never as a scored state. |
| “Attending to X right now” | Attention | Evidence only | Attention is evidence, never a scored state. |
| “True goal is Y” | Inferred goal | Evidence only | Inferred goals are evidence, never scored states. |
| “Attention causes the switch” | Causal claim | Out of scope | A causal claim requires intervention and is outside the default benchmark. |
| A self-reported goal that is also a model input | Contaminated label | Invalid label | Self-report may be evidence, but it is never the outcome label when it is also a model input. |
6 · The reporting row
Each pass reports one row. A result is the row, not a single number — skill is meaningful only next to its baselines, gates, counts, and status.
A large forecast count from few targets is not a large independent sample. Resolved-forecast count and independent-target count are reported separately for exactly this reason.
Run the loop, or read the schemas
The Version 1.0 reference harness executes this loop on synthetic data as a smoke test and reference path — not a validated leaderboard result. The schema reference defines the on-disk formats a submission must produce.