Michelle Blomberg

Student Journey Barriers Study

AI Strategy · Adoption · Case study

A written persona is a description. A synthetic student is that persona’s traits instantiated as an agent that actually goes and does the task. The whole setup is a multi-agent system: fifty synthetic students, what UX researchers call synthetic users, one orchestrator, and one research aggregator.

Fifty-two agents in three tiers

The system is fifty specialist agents plus two agents above them, not one generalist trying to be every student at once. A single agent asked to play fifty students blurs them toward an average, and the average student is exactly the one who never has trouble. Keeping each persona whole is what preserves the edges where services actually fail.

The three tiers of the agent system
TierWhat it does
Fifty synthetic studentsStay naive on purpose. Attempt one task, in character, at one college, and report honestly where a student like them struggles. They never judge whether a problem is fixable, which is what keeps their findings real.
One orchestratorAssigns the combination that makes a run: which persona, which task, which college. Collects the returned findings and watches for saturation so the study stops when new runs stop producing new barriers.
One research aggregatorThe only place in the study where every run is looked at together. It merges all logged runs into one register, collapses repeats into a single row with an instance count, ranks by candidate severity, and types each barrier by whether AI can close it. It never runs a test and never generates a finding.

Keeping judgment out of the persona tier and analysis out of the testing tier is what makes the corpus inspectable. A barrier can always be traced from the register back to the run that produced it, and from that run back to the page.

What each synthetic student does

How each synthetic student agent operates
PropertyHow it works
It runs autonomouslyGiven one persona and one task, the agent browses the live college website itself, reading pages, clicking links, and navigating toward the goal the way any member of the public would. No special access, no back channel. For public findability it does not log in, because a prospective student cannot either.
It is constrained to the personaThe craft is in the constraint. An AI is too capable by default and would breeze through a form a real student abandons. Each agent is held to the knowledge that student would have, flags unfamiliar terms the way that student would, and behaves with that student’s device, patience, and reading speed.
It thinks aloudThe agent narrates in the first person the whole way through, so the reasoning is visible rather than inferred from an outcome.
It returns a structured findingEvery run produces an inspectable record: outcome, path taken, where it broke, a first-person quote, a candidate severity, and device notes. Every finding traces back to a specific page.
A human validates itEvery finding is a candidate until a person confirms it. Severity is proposed by the agent and confirmed by human raters. The agent finds; it does not decide.

How one run works

One run, end to end

  1. Orchestrate. One orchestrator agent sequences the journey, from finding the college through to work, and routes each run to the right persona at the right college.
  2. Walk. A persona agent walks one real task on the live public site, constrained to that student’s knowledge and habits, thinking aloud, and stops where a real student would give up.
  3. Return a candidate. It returns a structured finding: outcome, the path and clicks, where it broke, and a severity guess from 0 to 4.
  4. Hand to people. A human rater confirms severity before the finding counts. The Method tab covers how.

What one run looks like

The record below is a constructed example, not a study result. It is what an agent returns and what a human rater confirms.

Outcome
Gave up
Path taken
College home page, searched “talk to someone about classes,” no clear match. Tried “Become a Student,” found “Advisement,” unsure that is the same as help picking classes. Opened Advisement, found a “Connect with an Advisor” button that opened a form.
Where it broke
The page offered a form and said the team would reach out, with no way to book a time and no sense of how long a reply takes.
Quote
“I filled out the form but I don’t know when someone will answer, and I work tomorrow. I don’t know if I should just show up.”
Barrier
No online self-scheduling, a form plus an unstated wait, and the word “advisement” did not obviously mean help choosing classes to her.
Severity
3, for a human rater to confirm
Device note
The form was long on a phone.
Effort
Several dead ends, about ten minutes before giving up.

What the research says

I chose this method on the research, and the research cuts both ways.

What it supports

LLM agent frameworks now run usability testing directly on live web designs. In a paired study, AI personas detected roughly 73 to 77 percent of known usability issues, with about 92 percent theme agreement against real users.

Where it fails

Severity judgments were unreliable and required human oversight. Nielsen Norman Group found interview-style AI participants too agreeable, and concluded their output should be treated as hypotheses to test with real users.

So severity, the judgment the evidence says AI is least reliable at, is the one thing the agents never decide. They propose a candidate; human raters confirm it. This study uses agents for task walkthroughs, not opinions. A validation sample of roughly ten testers, about one per campus, runs the same tasks the agents run, and coverage scales only if agreement holds.

How the study stays safe

The study’s safeguards and what each means
SafeguardWhat it means
Synthetic testers, real people by consentThe scaled testers are AI agents built from the district’s aggregate enrollment demographics, not real people. The signed-in tasks run on a single sanctioned OIT-controlled test account that is no real person and holds no real student record. The small group of real people who validate the findings take part knowingly and with consent.
The model stays inside the districtThe method is model-agnostic and built to run on a small language model kept inside district control. Data-governance and security decisions are worked directly with the Strategy, Security and Governance domain (Domain 1).
Humans make the judgment callsAt least three people score each barrier’s severity independently and the reported severity is their mean. The AI expands reach; people decide what any finding means and what to do about it.
Staff always know a test is a testWhen a run will submit a form or message that reaches a real staff inbox, the offices that will receive it are notified in advance. Every such message carries a clear line stating it is an AI usability test, not a real student.
Access is phased and gatedPhase 1 touches nothing beyond the public web. Phase 2 is cleared by Domain 1 before it begins. Phase 3 waits on both the Salesforce platform being live and Domain 1’s sign-off.
Human-subjects careThe study will confirm with the college and the district whether the work requires Institutional Review Board (IRB) review or qualifies as exempt usability testing, and it will complete that determination before any human testing begins.
No one is replacedEvery fix is designed to take routine, repetitive work off advising, financial-aid, and support staff so their time goes to the students who need a person.

Synthetic runs never replace real students. They locate and rank candidate barriers so the limited hours of real student and staff testing are spent confirming the findings that matter most.

Sources