Gravity Spy

All 100 current campaign subjects are shown. GPT-6 Astra received the campaign tutorial with low reasoning and the optional field guide off. These are direct API responses from one run on 2026-10-01.

Each individual volunteer vote is scored directly against the frozen benchmark target label with the campaign's exact-match scorer. A subject's baseline is the mean over its volunteer items, and the campaign baseline is the subject-balanced mean of the per-subject means, matching how the model's campaign score averages one score per subject.

# Gravity Spy baseline methodology

All baselines use the same 100 expert-reviewed targets and exact-match scorer as
the models. The review was not blinded: the volunteer plurality was highlighted
in the labeling interface. Expert reference provenance does not imply blinded
independence from the volunteer evidence.

## Average volunteer

Score every retained workflow-7766, single-choice T1 response directly against
Mike Zevin's target class. Average within each subject, then average the 100
subject means. Each subject has equal weight regardless of its vote count.

Historical response choices such as `UNCATEGORIZEDGLITCH` remain distinct and
count in the denominator. They are not mapped to `None of the Above` or dropped.
Volunteer responses span workflow versions; selection requires a unique plurality
in the retained current vocabulary, not that every minority vote uses it.

## Volunteer plurality

Find the most frequent choice among all retained votes on each subject, including
choices absent from the expert target distribution. Score that choice against
the expert label and average across subjects. Ties use uniform random tie-break
expectation; no selected subject has a tied plurality.

## Permutation reference

Compute the exact within-campaign permutation expectation using the frozen
expert targets. For single-choice classification this is the sum of squared
target-class frequencies. It is not uniform guessing among the 27 options.

The canonical result is `baselines.json`, produced by the campaign preparer or
`python -m zoobench.baseline --campaign campaigns/gravityspy`. Consumers validate
its input fingerprint. The old 91.9% average-volunteer baseline used >80%
consensus selection and volunteer-derived targets, so it is not directly
comparable to the new cohort's expert-reference baseline.