Draft Steps 1–4 · 10 Minutes
Use short phrases. Swap the clinical scribe for your own product as you go.
1. Start from the scenario
- In → out: what goes in, and what comes out?
- Consumers: who uses the output, and what does each one need? List every consumer; they rarely want the same thing.
- Cost of failure: what is the worst plausible error, and who pays for it?
- Baseline: how does the user do this today, without your product?
- Decision:
We will [advance / pilot / ship] if [evidence], compared with [the current workflow].
Scribe example
Visit transcript and patient record in; a structured clinical note out. Consumers: the signing physician, the next clinician, billing, the patient, compliance. Decision: start a 10-physician pilot if documentation time drops by at least 3 minutes per visit with zero critical accuracy failures in expert review.
2. Define the dimensions
List three to five dimensions. Give each one a question, a pass example, a fail example, and a severity tier.
| Dimension | Question it asks | Pass example | Fail example | Critical / major / minor |
|---|---|---|---|---|
- After you read 20–30 real outputs, which dimension will you add or split?
- If you track a proxy (edits, clicks, acceptance), which outcome will you track next to it?
3. Build an initial set of cases
- Common cases: where will they come from, and how many?
- Corner cases: list at least two inputs that would break a naive system.
- Known failure modes: list at least two failures you already suspect.
- Two collections: which cases are a representative sample (to estimate), and which are a challenge suite (to find failures)?
- Gold set and dev/test split: which cases will an expert label (your gold set)? Which will you iterate on (dev set), and which will you leave untouched until the end (test set)? See Terms.
- Groups to report separately: who could your product serve worse? Name the groups you will score separately (for the scribe: visits through an interpreter, pediatric visits).
Write one case out fully: the input, the expected output, and how you will check it.
4. Evaluate in layers
- Layer 1, automated checks: one rule you could check in code on every output.
- Layer 2, LLM judge: which dimensions, and how many human-labeled cases will you calibrate it on?
- Layer 3, expert review: who reviews (do they span your consumers?), and how will you sample: flagged cases plus a random sample?
- Read the results: what threshold will you set before you look?
Advance if ___ · Revise if ___ · Collect more evidence if ___
Partner Review · 5 Minutes
Exchange cards. The reviewer asks one question:
How could your product pass these checks and still fail a consumer or a group?
A group can be users or the people your output is about: for the scribe, patients who speak through an interpreter.
The owner revises the weakest box. If there is time, run the case you wrote out on your prototype.
For Week 4 · Steps 5 and 6
5. Improve your product
What will users change or redo, and how will you capture it? Which expert sample will confirm that a drop in edits is a real improvement?
6. Close the loop
When a check fails, what is the cheapest fix you would try first (prompt, template, pipeline, or fine-tuning)? Which new check will keep that failure from coming back?
Exit Ticket
Submit two sentences at the exit ticket and attendance form (the QR code on the last slide opens the same form). Submitting it also records your attendance.
- "The decision my eval should support is …"
- "The first check I will automate is …"