Real workflow pressure
Cases are built around longitudinal records, conflicting sources, old medication entries, accumulated diagnoses, copied documentation, and incomplete current information.
Clinical AI red team testing and benchmarking
Stress test the system before the psychiatric chart does.
I evaluate behavioral health AI systems against realistic longitudinal records, adversarial clinical scenarios, and clinician authored scoring rubrics to identify hallucinations, stale chart propagation, medication errors, diagnostic incoherence, and safety failures before deployment.
Private custom evaluations are available while the public benchmark methodology is under development.
A clinician built evaluation layer
General benchmark scores do not reveal whether a system can survive the actual psychiatric record.
Clinician truth state
Each case separates current supported facts, historical facts, contradictions, stale entries, uncertainties, and information that the model must not invent.
Decision ready findings
The result is not merely a score. Every important failure is linked to source evidence, severity, likely root cause, remediation, and retest criteria.
The core stress test
Can the system reconstruct the present clinical state from years of chart sediment?
The difficult task is not recalling a DSM fact. It is deciding what remains current after multiple group home ISPs, hospital records, progress notes, medication administration records, pharmacy lists, collateral reports, and legacy diagnoses are placed in the same packet.
A strong system should compress that packet into a useful current representation without resurrecting discontinued medication, treating historical symptoms as present, copying every inherited diagnosis, or manufacturing a complete mental status examination.
Multiple group home ISPs
Repeated background material, different dates, changing staff observations, and copied functional descriptions.
Medication lists with fossils
Duplicate strengths, old prescriptions, discontinued medications, and disagreement between the MAR and the chart list.
Diagnosis accumulation
Schizophrenia, bipolar disorder, schizoaffective disorder, depression, anxiety, and unspecified psychosis carried together.
Conflicting patient and staff reports
Current denials, recent collateral, historical symptoms, and uncertain timelines that require attribution rather than silent selection.
Unsupported note completion
Pressure to fill appearance, eye contact, speech, orientation, memory, insight, judgment, and risk fields that were never observed.
Failure modes
The output can be fluent and still be clinically untrustworthy.
Red team cases are designed to reveal where polished language conceals temporal confusion, inherited chart error, missing information, or unsupported certainty.
Legacy chart propagation
Old diagnoses, discontinued medications, copied errors, and stale history are promoted into the current note as though they remain active.
Medication reconciliation failure
Duplicate strengths, discontinued medications, obsolete pharmacy lists, and conflicting MAR entries survive the synthesis process.
Diagnostic incoherence
Accumulated labels such as schizophrenia, bipolar disorder, schizoaffective disorder, and depression are repeated without a defensible current formulation.
Temporal confusion
Historical symptoms, old collateral, and past hospital findings are written as present tense facts without source or date boundaries.
Unsupported documentation
The model invents mental status findings, risk statements, family history, adherence, or other details that were never supplied.
Clinical compression loss
The output becomes shorter and more polished while omitting the facts that actually matter for safety, continuity, and decision making.
PsychWorkflowBench
A working framework for longitudinal psychiatric reasoning, documentation, and safety.
PsychWorkflowBench is the working name for a clinician authored evaluation framework under development. The framework is designed to test real psychiatric workflow performance rather than exam style medical recall.
Record synthesisMedication reconciliationDiagnostic coherenceTemporal fidelityDocumentation fidelitySafety and escalation
Custom private cases can be built around a client system now. Public benchmark publication, validation, and any research use remain separate later phases.
Review the methodologyWhat gets measured
A score should reveal the anatomy of failure.
Overall performance is useful only when the buyer can see which clinical capability failed, how often it failed, and whether the failure matters for the intended use.
Temporal fidelity
Whether current, historical, discontinued, and uncertain facts remain correctly separated.
Medication accuracy
Whether the final output identifies the active regimen and excludes stale or duplicate entries.
Diagnostic coherence
Whether the formulation is supported by the supplied record rather than inherited from the problem list.
Contradiction handling
Whether conflicting sources are surfaced, attributed, and resolved only when the evidence permits.
Unsupported assertion rate
How often the system adds clinical facts, findings, or certainty that do not exist in the source material.
Legacy error propagation
How often deliberately stale or incorrect chart content survives into the generated note.
Clinically relevant recall
Whether the important current symptoms, risks, medications, and uncertainties survive compression.
Critical failure rate
The frequency of errors with plausible safety, treatment, handoff, or deployment consequences.
Evaluation services
From one focused audit to a private regression program.
Benchmark audit
Run a system against a fixed psychiatric task suite and return dimension scores, failure cases, and a prioritized remediation map.
Best for an initial reliability baselineCustom red team audit
Build adversarial cases around the actual product, workflow, prompts, data format, and known risk surface.
Best for predeployment stress testingModel selection evaluation
Compare candidate models and configurations on the same clinical workload instead of relying on general benchmark scores.
Best before procurement or architecture decisionsRemediation retest
Rerun the same cases after prompt, workflow, retrieval, or model changes and quantify which failures improved or regressed.
Best after engineering changesContinuous clinical evaluation
Maintain a private regression suite for major model, prompt, workflow, and knowledge base updates.
Best for production systemsFounding red team pilot$1,500
Limited to the first three completed pilot engagements.
A fixed scope first test with a concrete engineering decision at the end.
- One defined behavioral health workflow or up to 20 deidentified outputs
- One model, prompt configuration, or existing system under test
- A clinician authored scoring rubric tailored to the intended use
- Sentence level findings linked to source evidence
- A severity ranked failure map
- A concise remediation plan and retest criteria
- A forty five minute findings review
Normal delivery is seven business days after complete materials are received. Custom multi model comparisons, larger case suites, deployment validation programs, and formal research studies are scoped separately.
Discuss a pilotProcess
Define the risk, build the cases, map the failures, retest the fix.
Map the intended use
Define the users, input material, clinical purpose, system boundary, and decision the evaluation must support.
Build the truth state
Create structured current facts, historical facts, contradictions, uncertainties, and prohibited unsupported claims.
Run and classify failures
Score outputs, trace errors to source evidence, assign severity, and identify likely prompt, retrieval, model, or workflow causes.
Prioritize and retest
Deliver remediation priorities and objective criteria for confirming that the next version improved without creating regression.
Boundaries
Independent risk discovery, not decorative certification.
Initial inquiries must not contain patient records or protected health information. Evaluation material should be synthetic or properly deidentified unless appropriate contracts, secure systems, permissions, and data handling arrangements are in place.
The service provides clinical and operational analysis. It is not legal advice, regulatory certification, formal validation, or a guarantee that a system is safe, compliant, or ready for independent clinical use.
The question is not whether the note sounds clinical. The question is whether every important sentence survives contact with the record.
Send a concise description of the system, intended use, material being processed, current concern, and the decision the evaluation must support. Do not attach patient material to an initial email.