# Evaluation set

`examples.csv` is the acceptance test for everything: a model, a regenerated app, a successor.
Each row is one case with the right answer attached by whoever knows what correct looks like. In this
example record every answer was written by the reference rule (`labeled_by`), not by a person.

Columns:

| Column | Meaning |
| --- | --- |
| `id` | Stable id for the case |
| `state` | The input the judgment was made on, or a path to it (keep inputs out of the CSV when large) |
| `label` | The right answer, from the label set in `thresholds.yaml` |
| `labeled_by` | Who attached the label: a person, or in this example record the reference rule |
| `labeled_on` | Date |
| `source` | `declared` (a real case) or `illustrative` (a made-up example; never counted in a miss rate) |
| `note` | Why, when it is not obvious |

Disputes become rows: when a person disagrees with a judgment, the case is added here with the
person's label. The miss rate is computed over `source = declared` rows only.