Student Journey Gap Analysis
Case study · AI Tools & Strategy · UX Design
How the study stays safe: no real student data is ever collected, the agents drive only a sanctioned test account, and the real people who validate the findings take part knowingly and by consent. A person makes every call the AI is not qualified to make.
No student data, and real people only by consent
The scaled testers are synthetic: AI agents built from the district’s aggregate enrollment demographics, not real people and carrying no real student’s record. The public findability tasks run on public web pages with no login. The signed-in tasks run on a single sanctioned OIT-controlled test account that is no real person and holds no real student data, staged and reset by OIT. No real student account is ever used at any point. Because agent-based usability testing counts only when people back it up, a small group of real people validates the findings, knowingly and with consent.
The model stays inside the district
The method is model-agnostic and built to run on a small language model kept inside district control, so student-facing testing never sends data to an outside service. That is the district’s data-sovereignty position. The study is led by the Student Support and Success domain (Domain 5), and its data-governance and security decisions are worked directly with the Strategy, Security and Governance domain (Domain 1), which owns that side.
Humans make the judgment calls
AI findings are always provisional. At least three people score each barrier’s severity independently and the reported severity is their mean, because the research shows AI is least reliable at judging severity. A small human sample re-runs some of the same tasks as a spot-check, compared on percent agreement about whether a barrier is present and a kappa on severity; synthetic runs scale only if that agreement holds, and if it is weak the study leans on the human sample instead. The AI expands reach; people decide what any finding means and what to do about it.
Transparency to staff
When a run will submit a form or message that reaches a real staff inbox, the offices that will receive it are notified in advance, close to the run and coordinated through the ARC; runs that only read or navigate public pages contact no one. Every such message carries a clear line stating it is an AI usability test from a sanctioned test student account, not a real student, and asks staff to answer as they would for a real student. Staff always know a test is a test.
Access is phased and gated
Access deepens only with permission. Phase 1 touches nothing beyond the public web and runs under the tri-chairs’ approval already in hand. Phase 2, the first time a district system is touched, is cleared by Domain 1 before it begins. Phase 3, the deepest access, waits on both the Salesforce platform being live and Domain 1’s sign-off. Between phases the method is audited and corrected on the safest data before it ever escalates.
Research permission and human-subjects care
The study secures permission at each level before it acts: Phase 1 runs under the tri-chairs’ approval, and any district-system access is cleared by Domain 1 first. For the human-facing parts, the small online validation sample and the in-person office walk-ins, the study will confirm with the college and the district whether the work requires Institutional Review Board (IRB) review or qualifies as exempt usability testing, and it will complete that determination before any human testing begins. Human participants give informed consent, and the in-person visits are conducted openly, so staff know a study is underway. The intent is to do this by the book, so the findings hold up and no participant is caught unaware.
Minimal data, no PII, no FERPA exposure
The study records counts and rates, never personally identifiable information and nothing FERPA-protected, since the personas are research instruments and the test account holds no real record. The barrier dataset lives on the district GitHub account the domain co-chair runs, with ARC access, and departments own their own numbers: the study aggregates and never exposes.
No one is replaced
Every fix is designed to take routine, repetitive work off advising, financial-aid, and support staff so their time goes to the students who need a person. Automation absorbs the four-hundredth identical question and the callback-triggering booking form, not the human judgment and connection staff provide. The study also weighs the compute cost of what it recommends, favoring the lightest option that solves the problem.