AI Standardized Patient Training & Assessment
Consistent simulated clients. Training feedback with traceable evidence.
Turn case knowledge into configurable characters, multi-turn dialogue and reference evaluation, connecting learner practice, teacher review and institutional management.
My role: Project lead | Stage: Teaching/training product and early administration design
This case uses the initial administration prototype to explain case, character, evaluation and training-management design. The actual product subsequently evolved with requirements.
300+ learners · 1,800+ hours of simulated training
Counselling practice requires recognising problems, developing understanding and applying skills in a concrete situation. A case document alone is not a usable simulated client. The character needs a consistent background and expression, and needs to reveal information as the conversation develops.
I planned case simulation and training evaluation from the ground up, designing character configuration, multi-turn behaviour, feedback reports and administration, alongside algorithm, engineering and psychology specialists.
Learners need to understand the task and feedback. Teachers need to choose and organise practice around teaching goals. Institutions need to maintain cases, roles, knowledge and training content.
The product connects these tasks:
- 01Configure training
- 02Enter the task
- 03Practise multi-turn counselling
- 04Review reference scores and feedback
- 05Replay evidence, reflect and practise again
Diagnostic practice: form a judgment as information emerges
The diagnostic mode hides the diagnosis. Learners gather information and identify symptoms and context rather than being given the answer first.
The AI needs to remain within the case while expressing information according to role and conversational conditions. It should not summarise the entire condition in clinical terminology.
Skills practice: apply methods in a defined situation
The skills mode makes the condition or training stage explicit, focusing attention on the use of counselling methods. Character behaviour needs to correspond to the training goal, and feedback to observable performance.
The modes change the task, information conditions and evaluation emphasis, rather than just a label in the interface.
The early administration design separates personal background, case information, course and treatment history, response rules, training stages and applicability instead of putting everything into one character-description field.
Case fields include diagnostic codes, severity, duration, core and accompanying symptoms, behaviour, triggers and relieving factors. Age, occupation, education and personality connect specialist case information to a particular person.
Response configuration covers opening lines, reference questions/answers, expression and behaviour. Learner level, orientation, session structure and difficulty allow “who the person is” and “what this trains” to be adjusted separately.
Character example: express anxiety through everyday experience
The prototype's preset case connects work worries, poor sleep, attention difficulties and avoidance, alongside speech rhythm, avoidance of family topics and consistency requirements.
Reference answers turn symptoms into everyday language. The sleep example describes lying in bed thinking about work and fearing mistakes, instead of reciting diagnostic criteria. It is a writing reference for maintaining the character, rather than a counselling transcript.
The early design also generates an editable prompt draft. It checks required personal, case and opening information, assembles entered fields, and leaves further editing to the content maintainer. Structured configuration therefore connects with a detailed prompt while retaining manual authoring.
Having the whole case does not mean revealing all of it in every reply. The initial design separates what the role knows from when it expresses it.
Different information needs different conditions
Stage configuration includes triggers, disclosure rules, progress indicators and retreat conditions. Field examples distinguish surface information, details revealed when asked, and material requiring more support before discussion.
Maintainers therefore need to specify what can be volunteered and what requires conversational conditions, rather than only writing “do not reveal everything.” Reference answers also distinguish core and extended content and can be associated with a stage.
Progress needs retreat conditions too
Each stage can specify an openness range, entry conditions and completion indicators. Retreat conditions describe reactions to unsuitable interaction. Examples such as interruption, premature explanation and judgmental language prompt maintainers to define how the character changes.
The interface also plans an intervention-response matrix linking intervention categories, patient reactions and immediate/delayed changes. It specifies behavioural relationships to configure; values and generated behaviour require character design and testing.
The character does not have to agree with everything
Behaviour-field guidance discourages sudden insight, perfect cooperation with every intervention, total change within one session, and language inconsistent with the client's background. Consistency includes facts, response intensity and language.
The case supplies background; stages and disclosure conditions constrain expression; behavioural boundaries constrain responses. Conversation checks are still needed to establish whether the model follows them.
The early design treats a scoring standard as a managed object. It specifies applicable diagnosis, scenario, orientation and learner level, then dimensions, weights and positive/negative conditions. Each standard has a defined purpose.
Translate “empathy” into observable behaviour
The editor starts with empathy/presence, technical accuracy, pacing/timing, therapeutic alliance, and ethics/professional boundaries. Each dimension has positive and negative conditions.
Field examples make empathy observable: allowing silence after which the character continues; or interrupting and rushing emotional expression. They guide maintainers toward behaviours rather than “good/bad empathy.”
Weights are editable and must total 100% before submission. This is a configuration and validation requirement in the initial interface, not proof of an independently implemented deterministic evaluation engine. Adopted rules and evaluation validity need the relevant product version and expert review.
Examine key moments and changes over time
I also planned critical-moment checks, an openness arc and additional items. Dimensions give an overall view; key moments address particular behaviours; the arc supports review of how a session progresses.
Return to the conversation, beyond the score
Training details combine role, learner, duration, turns and dialogue history with dimension scores and written feedback. The dialogue separates speakers and also represents tone and emotion beside character replies.
This puts evaluation and replay on one review path. An administrator can revisit the conversation before reviewing or adjusting evaluation. Summary reporting plans filters by time, learner and patient type, alongside dimensions and weak areas. Preset values in the early prototype represent interface content.
University teachers reported that some AI clients used overly formal symptom descriptions or medication names, sounding clinical and written rather than like a client.
I worked on case-knowledge organisation and input, character prompts and behaviour rules, added everyday expression and dialogue examples, and coordinated changes to model/generation configuration.
Teacher and internal trials then checked case consistency, language appropriate to the background, and coherence across turns. This documents a feedback-led refinement process; public material does not include raw before/after dialogue or a quantified gain.
- 01Teacher feedback
- 02Locate knowledge/expression issues
- 03Revise inputs, rules and examples
- 04Coordinate configuration
- 05Trial again
The early design manages roles, scoring standards and tasks separately, then combines them in task configuration. A task selects its audience, patient case, scoring standard and time limit, plus opening times, attempts and pause conditions.
This allows the same role to support different tasks and evaluation to follow the learning goal. The task defines who trains under which conditions instead of duplicating character content.
Initial administration flow
- 01Create/edit patient
- 02Configure evaluation
- 03Assemble task
- 04Set publication details
- 05Select learners
- 06Review records/results
Publication separates configuration and learner selection. Time and subject are checked before proceeding; final publication requires recipients. Search and selection let administrators verify content and scope.
This early design connects roles, standards, tasks, people and results. The actual administration product subsequently evolved with development and use requirements.
I planned differentiated solutions for university teaching and counsellor training, coordinating with algorithm, engineering and psychology specialists. The product reached more than ten universities and counselling institutions through collaboration or trials, supporting 300+ learners and 1,800+ training hours.
Conversational AI quality involves more than sounding human: case fidelity, appropriately timed disclosure, evaluation tied to observable behaviour, and teacher maintenance and review together determine its usefulness for training. Scores are teaching references, not formal professional certification or clinical diagnosis.