Prototype · synthetic data · not a medical device
Harness suite
The deterministic reconciler cases run here, in your browser, against the same frozen record and the same lib/reconcile.ts used in production. The model-facing cases are documented below.
Deterministic cases run here and in npm run eval. Model cases run via npm test (mocked) and npm run eval:model (live).
derivedTier: watch
derivedTier: watch
derivedTier: watch
derivedTier: watch
derivedTier: watch
derivedTier: elevated
derivedTier: watch
derivedTier: watch
derivedTier: watch
derivedTier: watch
derivedTier: watch
Not run in the browser. These execute in CI (mocked) and, for the quality set, against the live API.
The wiring around the model, with a fake createMessage. Deterministic, zero network. Proves our code copes with whatever the model returns.
Valid JSON on the first try
Given The model returns a well-formed pre-brief on the first attempt.
Expect Parsed and returned, no retry.
The happy path must not do extra work or mask a clean first-try success.
Invalid then valid retries once
Given The first reply is not JSON; the second is valid.
Expect createMessage is called exactly twice; the valid result is returned.
One self-correcting retry with the parse error fed back, never a loop.
Invalid twice throws, nothing rendered
Given Both replies are unparseable.
Expect PreBriefGenerationError; the route surfaces a clean error, no partial content.
Unparseable output must never reach the screen as clinical content.
Missing provenance is rejected
Given A finding with an empty provenance array.
Expect FindingSchema rejects it; generation fails.
A claim with no source cannot be reconciled, so it cannot exist.
Clinician-only fields are stripped
Given The model includes a status of accepted and a clinicianEdit on a finding.
Expect Both are dropped; the finding comes back unverified.
Only a clinician sets those; the model cannot pre-approve its own output.
Fenced JSON is unwrapped
Given The reply is wrapped in a Markdown-style code fence.
Expect The fence is stripped and the inner JSON is parsed.
Models often wrap JSON in fences; tolerate it rather than fail.
Wrong memberId is overwritten
Given The model echoes a different memberId than the one requested.
Expect It is forced back to the real member id.
The record identity is ours, not the model's to choose.
No key serves the sample
Given POST /api/prebrief with no ANTHROPIC_API_KEY.
Expect 200 with the reconciled built-in sample, generated: false, tray populated.
The fixture flow stays fully navigable offline, still through the reconciler.
Finalised debrief rejects unresolved findings
Given A finalised pre-brief containing a finding that is still unverified or dismissed.
Expect FinalisedPreBriefSchema rejects it (400).
The debrief may only be drafted from findings a clinician accepted or edited.
The advisory judge is downgrade-only
Given The observation judge returns not-supported, or supported, for a claim.
Expect Not-supported moves grounded to flagged; supported never moves flagged back.
An advisory LLM check can add caution, never remove it.
Rate limit maps to 429, other errors to 502
Given The model call throws a RateLimitError, or any other error.
Expect 429, or 502, with a clean JSON body and no stack trace.
Predictable failure surfaces the client can retry on.
One real generation per fixture member. Asserts properties of the output, not exact prose. Run with npm run eval:model; skipped without a key.
Output passes the schema, per member
Given A real generation for each fixture member.
Expect PreBriefSchema parses it successfully.
Shape is the floor; nothing downstream runs on malformed output.
Every finding reconciles to grounded or flagged
Given Each finding from a real generation, run through reconcile().
Expect The verdict is grounded or flagged, never rejected.
On faithful synthetic data the model should copy values exactly; a rejection means it fabricated.
A first-visit member has no deltas and no trend claims
Given A member with a single scan on record.
Expect deltas is empty and there are zero trend claims.
There is no prior scan to compare against; a trend would be invented.
Every claim.metric resolves to a real path
Given Each claim's metric, and each delta provenance metric.
Expect It resolves to an actual value in the record at that scan date.
A claim pointing at a non-existent path is a hallucinated metric.
Observation claims only where there is no number to check
Given Each observation claim in the output.
Expect Its metric is not a plain number that has a reference range.
observation is the unverifiable escape hatch; it must not dodge the numeric checks.