Map the source pattern
Identify chronology, medication conflicts, diagnosis pileup, copied errors, collateral disagreement, missing information, and the clinically important facts buried in repetition.
Private workflow under development
PsychWorkflowBench Case Factory
Turn authorized chart complexity into synthetic cases that expose clinical AI failure.
The case factory converts the reasoning structure of difficult psychiatric records into synthetic benchmark packets, structured truth states, gold notes, scoring rubrics, and private regression suites. It preserves the clinical puzzle while removing the identity and biography of any real person.
Three transformations
Preserve the reasoning difficulty, replace the identifying story, and keep the answer key private.
Build a synthetic twin
Change identity, demographics, facilities, dates, family structure, quotations, and distinctive life events while preserving the same reasoning traps and scoring target.
Create a held out test
Keep the truth state, rubric, critical failure flags, and selected variants private so the evaluation measures system behavior rather than answer retrieval.
Case factory sequence
A reproducible path from authorized source material to a scorable synthetic case.
Inventory and code
Assign source codes, relative chronology, record type, privacy risk, contradiction potential, and benchmark value.
Construct the fact matrix
Separate patient report, staff collateral, observation, prior assessment, MAR data, problem list entries, and copied text.
Generate and challenge
Create a synthetic base case plus controlled variants for medication contamination, diagnosis pileup, and timeline conflict.
Adjudicate and lock
Complete clinician review, privacy review, gold state, gold note, scoring rubric, and private release status.
Synthetic packet design
The packet should feel like a real chart without representing a real patient.
A difficult case needs enough redundancy and contradiction to force genuine synthesis. It also needs a stable gold state so different models, prompts, retrieval systems, or workflow configurations can be compared fairly.
Longitudinal residential records
Several synthetic group home ISPs from different periods, with realistic repetition, changing function, and copied background material.
Competing medication sources
A current MAR, old medication lists, pharmacy history, discontinued entries, duplicate strengths, and one unresolved status conflict.
Accumulated diagnosis history
Supported, historical, provisional, overlapping, and unsupported psychiatric labels that must not simply be copied into the final note.
Current interview and collateral
Patient report, staff report, recent clinician documentation, and at least one clinically meaningful disagreement between sources.
Known traps and missing facts
Seeded stale facts, old symptoms, tempting template fields, and information the evaluated model should identify as unknown.
Case package
Every case should be reviewable, scorable, and usable without reopening the real source record.
Source inventory
A coded map of record types, chronology, clinical domains, contradiction potential, benchmark value, and privacy risk.
Fact matrix
Every clinically relevant fact linked to provenance, temporal status, reliability, conflict group, and current support level.
Synthetic packet
A new case that preserves the reasoning traps while changing identity, biography, dates, facilities, and distinctive narrative details.
Gold clinical state
Current medications, discontinued medications, supported diagnoses, historical diagnoses, symptoms, contradictions, unknowns, and safety facts.
Gold note
A concise current psychiatric note written only from the structured truth state, without invented examination findings or obsolete chart content.
Scoring rubric
Case specific scoring criteria, forbidden elements, critical failure flags, and clinician review requirements.
Quality control
The benchmark must be difficult for the right reason.
Case review checks whether the dates remain internally consistent, medication doses remain plausible, intended contradictions survived synthesis, the gold note contains no invented findings, and the task can be scored without access to the real source record.
Privacy review remains separate from clinical review. A synthetic case is not treated as approved for public release merely because it contains no direct identifier.
Privacy and research boundaries
Clinical access does not automatically authorize benchmark use.
Real records should be processed only inside an approved environment with appropriate authorization, contracts, data controls, and human review. Initial inquiries must not include patient records or protected health information.
Public release, publication, external research, client distribution, and regulatory use are separate decisions. Each requires its own governance, privacy review, permissions, and validation plan.
The synthetic case should preserve the trap, not the person.
A private case suite can begin with one narrow workflow and a small set of clinician reviewed cases before expanding into a larger benchmark or continuous regression program.