Penguin Watch

All 100 current campaign subjects are shown. GPT-6 Astra received the campaign tutorial with low reasoning and the optional field guide off. These are direct API responses from one run on 2026-10-01.

The October 4 scoring update awarded partial credit for counts within one. The cohort, model answers, and inference date remain unchanged. The October 15 direct-result archive contains the identical October 1 result file.

Each prepared individual volunteer count vector receives 1 when every count matches exactly, 0.5 when every count is within one of its reference and at least one differs, and 0 otherwise. A subject's baseline is the mean over its volunteer items, and the campaign baseline is the subject-balanced mean of the per-subject means, matching how the model's campaign score averages one score per subject.

# Penguin Watch scoring and baselines

The score for an image is **1** if both predicted counts exactly match the expert:
`adults == GS_adults` and `chicks == GS_combined - GS_adults`. It is **0.5** if
each count is within one of its reference and at least one differs, and **0**
if either count differs by more than one. The tolerance applies independently
to adults and chicks, not to their combined total. For example, an expert pair
of (20 adults, 0 chicks) and an answer of (3 adults, 0 chicks) scores zero.
Missing, extra, negative, boolean, string or fractional count fields are invalid
and score zero. The campaign score is the mean over the 100 images.

This `exact_or_within_one` rule replaced joint exact match on October 4, 2026.
The frozen cohort, labels, volunteer responses, prompts, and saved model
answers are unchanged. An exact-match fraction may still be reported separately;
it is not the primary mean score.

The production scorer is `zoobench/scorers/counting.py`; its explanation trace
shows the error in each count, exact agreement, and the within-one condition.

## Average volunteer

For each image, independently score every retained complete user-image response
against its expert pair and average those 0/0.5/1 scores. Average these 100
image means equally. An image with 57 responses has the same overall weight as
an image with 10 responses. Do not pool all 1,317 responses into one accuracy.
The complete-only filtering and ambiguity rules are in `selection.md`.

Final sample: **35.993415470368%**.

## Published volunteer consensus

The common `majority_volunteer` artifact slot stores the authors' published
aggregate, taken directly from `CS_adults` and `CS_combined - CS_adults` in the
same workbook. Score each pair with the same 1/0.5/0 rule, then average
equally across images. In user-facing reports this is labelled **Published
volunteer consensus**, since it is not a majority vote on the count pairs and
is not a fresh aggregation of complete responses.

The published aggregation can include volunteers excluded from our individual
baseline. Its thresholds were evaluated using gold-standard comparisons, so
this is a descriptive published-aggregation baseline, not an independently
held-out estimate of a newly trained aggregation algorithm.

Final sample: **45.00%** (26 exact images and 38 additional images within one
in both categories).

## Permutation

Use the existing ZooBench exact expectation: score every ordered reference/answer
pair among the 100 expert count vectors, including self-assignments, and average
all 10,000 scores using the same 1/0.5/0 rule. Whole pairs move together; classes are not shuffled
independently, and shuffling is not restricted by camera. This is the expected
score under a random permutation of the observed expert pairs, not random
integer guessing.

Final sample: **5.90%**.

## Canonical artifact

`baselines.json` is built by the preparer, or explicitly with:

```bash
uv run python -m zoobench.baseline --campaign campaigns/penguinwatch
```

It stores the per-image means and response counts as well as campaign means.
Its fingerprint covers the ordered subjects, expert and volunteer labels,
published aggregates, scoring configuration and counting scorer/baseline code.
Consumers reject stale artifacts. `tools.validate_penguinwatch` checks these
three scores against independent count-error arithmetic over the same responses.