Student Journey Gap Analysis

Case study · AI Tools & Strategy · UX Design

A written persona is a description. A synthetic student is that persona’s traits instantiated as an agent that actually goes and does the task. I designed this layer of the study so a skeptical reader can see it is legitimate engineering rather than a model imagining a student.

Fifty specialists and one orchestrator

The system is fifty specialist agents plus a single orchestrator, not one generalist agent trying to be every student at once. Each of the fifty personas is its own agent, a specific student carried whole and never broken into traits, lenses, or sub-models. The orchestrator sequences the journey and routes the runs.

That architecture is a deliberate choice. A single agent asked to play fifty students blurs them toward an average, and the average student is exactly the one who never has trouble. Keeping each persona whole is what preserves the edges where services actually fail.

What each agent does

The prompt scaffold

One template, filled from each of the fifty persona briefs. No model is fine-tuned and no agent is hand-built from scratch, which is what makes fifty agents maintainable by a committee rather than a lab. The scaffold has three parts.

Who you are. Name and age, home college, language and reading level including where the student slows down, what the student knows and does not know down to the exact terms they would not recognize, device and technology comfort, life constraints, emotional state, and the one goal for today.

The hard rules. Use only this student’s knowledge and show genuine confusion at unfamiliar terms. Behave with this student’s device, patience, and reading speed. Start from the need, never from the office. Think aloud in the first person throughout. Do not be more helpful, thorough, or clever than the persona.

The findings schema. Persona identifier and name, college, task, outcome (found, partial, not found, or gave up), path taken, where it broke, a one-line first-person quote, suspected barrier, a severity candidate from 0 to 4 for a human to confirm, a device note, and effort.

What one run looks like

The record below is a constructed example, not a study result. The study is in progress and no findings are published here. This is shown only to make the output format concrete, so a reader can see what an agent actually returns and what a human rater would be confirming.

The point of the example is not the result. It is the shape: a persona’s traits, run as an agent, returning a specific, page-level finding that a human can go and check.

The evidence this method rests on

I chose this method on the research, and the research cuts both ways. Reporting only the encouraging half would be the same failure the study exists to correct.

What the literature supports. LLM agent frameworks now run usability testing directly on live web designs, simulating a user who browses a real interface and reports where it breaks, which is exactly the live-site task asked of these agents (UXAgent). In a paired study, AI personas detected roughly 73 to 77 percent of known usability issues, with about 75 percent attitudinal correlation and 92 percent theme agreement against real users. Other work finds LLM agents complement human testing and overlap broadly on issues that are not identity-specific.

Where it fails. One evaluation found GPT-4o reached about 84 percent agreement with human evaluators on issue detection, but that severity judgments were unreliable and required human oversight. In education specifically, a recent review of simulated-learner studies found nearly half report no formal validation at all.

What that dictates about the design. The literature points to one shape: lead with synthetic students for reach, then gate on human agreement. So severity, the judgment the evidence says AI is least reliable at, is the one thing the agents never decide. They propose a candidate; human raters confirm it. A validation sample of roughly ten testers, about one per campus, runs the same tasks the agents run, and coverage scales only if agreement holds. If it does not hold, the synthetic layer does not get to expand.

That validation step is also the study’s clearest methodological contribution. Against a field where half of comparable studies skip validation entirely, building the gate in from the start is what makes the findings defensible to an institutional research office.

Division of labor

Synthetic students carry breadth. They walk the public, no-login paths across all ten colleges and produce a first map of every task at every campus quickly, which is a volume of walkthroughs no volunteer committee could staff. Humans carry judgment: the in-person walk-ins, the authenticated-navigation sample, and the validation runs that check the synthetic findings. They are the check on the synthetic layer, not the coverage.

The burden on volunteers is deliberately low. One scenario, one sitting, about fifteen minutes, because that is what people with full day jobs can actually give.

Sources

• UXAgent, an LLM-agent framework for usability testing on live web designs. arxiv.org/html/2504.09407v2
• AI personas detected roughly 73 to 77 percent of known usability issues, with about 92 percent theme agreement with real users. dl.acm.org/doi/10.1145/3788149.3788233
• Evaluating LLM issue detection and severity: about 84 percent agreement with human evaluators, but unreliable severity judgments requiring human oversight. arxiv.org/pdf/2506.16345
• LLM agents complement human usability testing and overlap broadly on non-identity-specific issues. dl.acm.org/doi/10.1145/3801978
• Simulating students, a review finding nearly half of simulated-learner studies report no formal validation. arxiv.org/abs/2511.06078