Start Here
Bring your working prototype and two or three example inputs you can share appropriately. Work in groups of three or four for the first two exercises, then fill in your own workflow card and exchange it with a partner.
Class materials: Lecture slides (PowerPoint) · Eval workflow card (PDF).
Today's running example
An ambient clinical scribe listens to a patient visit and drafts a note the physician reviews, edits, and signs. All clinical material on these pages is fictional.
Exercise 1 · Same tool, different claims
Five minutes: compare who was studied, the task, the tool, and how time was measured in two randomized studies. Write one sentence naming the claim each study supports.
Exercise 2 · Judge the judge
Six minutes: ask a model in Stanford AI Playground whether each of two draft clinical notes is safe to sign, then change one phrase at a time to try to break the judge. Record what flipped its verdict and which model you used.
Exercise 3 · Your eval workflow card
Eighteen minutes: fill in steps 1–4 of the workflow for your prototype (10 minutes), get one question from a partner (5 minutes), and revise (3 minutes).
Terms
We use these words precisely for the rest of the course.
| Term | Meaning | Scribe example |
|---|---|---|
| Eval | A repeatable test: run your system on a set of cases, score each output against stated criteria, and use the scores to make a decision. "Evals" is the whole suite. | Score 20 draft notes on accuracy and completeness before a pilot. |
| Benchmark | A public eval that many teams share. | MT-Bench, SWE-bench. |
| Gold set (golden set, gold-standard dataset) | Cases with expert labels for what a correct output contains: the reference you score against. Gold labels can be wrong too, so have a second expert relabel a sample. | 20 visits; a clinician marks what each note must and must not say. |
| Train set | Examples used to build the system. Keep them out of your dev and test sets: a case that sits in your prompt cannot test the prompt. | Example notes pasted into the prompt; fine-tuning data. |
| Dev set (development set) | Cases you look at while you iterate and choose between versions. | 15 of the 20 gold-set visits. |
| Test set (held out) | Cases you leave untouched until the end. Score them once; that is the number you report. | The other 5. |
Working Norms
- State exactly which users, tasks, and context a claim covers.
- Score several dimensions separately; a critical failure is never averaged away.
- Never paste real patient, user, or interview data into an unapproved model. Use fictional or approved, de-identified examples.
- Treat disagreement between raters as information about the rubric, not as something to average away.