Student Journey Gap Analysis

Case study · AI Tools & Strategy · UX Design

A ten-college study of the district student journey, from finding a college through to work, mapping where students hit barriers reaching support. I designed the study and I lead it as co-chair of the Student Support and Success domain, working with advisors, financial aid specialists, and student-facing staff across the district.

The student journey across nine milestones, from Find the college through Apply, Enroll, Pay, Take classes, Get support, Feel connected, and Graduate to Work, with campus support services drawn faintly behind the arc

Goal

Set a baseline of how many students use each service now, then find and rank where students hit barriers reaching support by how many students each gap touches, and hand staff located, severity-rated evidence they can act on. A successful AI tool answers routine questions without a booking or an office visit, so help is counted through the tool even as office traffic falls. Student teams build the tools, not the staff workflows, which the ARC team and the department that owns the service lead.

Audience

The Student Support and Success domain of the district AI Resource Center, made up of advisors, financial-aid, and other support staff across all ten colleges, plus the college leaders and faculty who help run the study and recruit the student build teams.

Process

Build fifty demographic-grounded personas as specialist agents, add one orchestrator agent that sequences the journey and routes the runs, and walk the path from finding the college through to work. Each service is walked at each of the ten colleges until added personas surface no previously unseen barrier, the saturation stopping rule, so the study tests to saturation rather than to a fixed quota; Nielsen’s five-user estimate is the planning figure, not the stopping rule. The walk runs in three parts by access: Part 1 needs no login and runs now, Part 2 is the signed-in tasks on one sanctioned test account, and Part 3 waits for the district’s incoming Salesforce platform. Log what breaks and how badly with more than one rater, rank the highest-reach gaps, decide where AI fits first, recommend and pilot the most promising fixes to prove they help before anything scales, then have student teams build them as human-in-the-loop tools that collect no student data. Humans decide what to fix. No one is replaced.

Across the district, ten colleges each run advising, financial aid, basic needs, and other support in their own way. Before a student can even apply, they have to find the college, and from there the path from a felt need to the right person is full of dead ends. This study walks that whole path on purpose, at scale, so the domain can see the gaps clearly and decide where an AI tool, a shared staff workflow, or a person is the right answer. It covers the student-services journey up to and around the classroom, not inside Canvas or the coursework itself, since the in-course experience belongs to the Teaching and Learning domain, Domain 2.

Technology

What the AI actually is. The fifty “synthetic students” are AI agents. Each one is a specialist agent instantiated from a single fixed persona profile, its situation, constraints, language, and one goal, and it autonomously walks a college’s live site the way that student would, narrates what it is looking for, and returns a finding. One orchestrator agent sits above the fifty: it sequences the journey, decides which persona, task, and college runs next, routes each run to the right agent, and records what comes back. The AI does the walking, at a scale no team of people could reach.
How the AI is kept honest. The agents are held to standard agent-testing discipline: a growing “golden set” of known barriers every agent version is re-run against, full task simulations rather than one-off prompts, several runs of each scenario to measure consistency, and versioned persona and agent templates so every finding traces to the exact build that produced it.
Structured output, not prose. Every run returns its finding in a fixed JSON schema: the barrier, its location, the path taken, the local service name, a severity flag, and the persona and campus that produced it. The study’s output is therefore a structured barrier dataset the committee can sort, count, and compare across all ten colleges, not notes to re-key.
The data-collection form. Human and AI runs are logged on the same Google Form, the Barrier Log built with Apps Script, so the two are directly comparable. The tester tool hands each tester one scenario and pre-fills the first fields (scenario ID, persona, campus, task, tester type, mode); the tester then records whether the task was completed, a 0 to 4 severity, time on task, what happened, and a suggested fix. The form collects no email and no student data.
How the gaps get decided. A coverage dashboard reads the scenario bank and shows how much of each college and each journey stage has been walked. The logged barriers are then analyzed in three passes: affinity mapping so themes emerge from the data, severity-by-reach (each theme placed on a matrix of how severe it is against how many of the roughly 140,000 students it touches, so the highest-reach barriers rise first), and journey mapping plus service blueprinting to show exactly where a student is lost and which routine, repetitive staff steps an AI tool could absorb.

Fifty student agents

The reach of this study rests on a method I designed: fifty demographic-grounded personas instantiated as specialist AI agents, coordinated by a single orchestrator. Each agent carries one whole student, browses the live college website itself, and is deliberately constrained to what that student would actually know, so it gets stuck where a real student gets stuck instead of breezing through. Every run returns an inspectable record that traces to a specific page.

Agents carry breadth; people carry judgment. Severity is the one call the agents never make, because the research is clear that it is where AI is least reliable. Findings are candidates until human raters confirm them, and coverage only scales while agreement holds. How the agents work, and the research behind the method.

Outcomes

The study produces a prioritized, severity-rated map of where students hit barriers, ranked by how many students each gap touches, alongside a service crosswalk across all ten colleges and a human-in-the-loop AI roadmap routed to the right owners. Part 1’s first runs are logged as of 27 July 2026; Parts 2 and 3 have not started. The measures are defined and the instrument is proven.

Status

A built prototype now in early fieldwork. The fifty persona agents and the orchestrator are built, and Part 1, the public tasks that need no login, has begun, with the first runs logged on 27 July 2026. Parts 2 and 3, the signed-in tasks and the Salesforce-dependent tasks, wait on district approval, the data-governance review, and the single OIT-controlled test account. No student data is collected at any point.