Panel review: the method on its own build

Course Design · Case study

The method’s claim is that a panel of AI agents, three agents I prompt to review work from different angles, can hold work to a standard with a human owning every decision. Here it is tested on a real artifact: after building the graduate data-science simulations, I pointed the panel back at them. It caught a statistical error and a chart that argued the opposite of its own point, both before any student saw them.

What this is

Every other tab describes the method. This one shows it working on something concrete. I built the four playable simulations and the defense practice tool in the Authentic Assessment suite, spanning model evaluation, clustering, and AI governance, then convened the review panel and asked each agent to critique the build. What follows is what they returned and what I changed in response. The panel widened what one builder could see.

The panel

The subject-matter seat is filled by whatever discipline the course needs. This build happened to be a graduate data-science course, so the expert here is a data scientist. For a nursing course it would be a nurse educator, for a welding course a certified welding inspector. The instructional-designer and student seats stay the same from course to course; only the subject expert changes.

Subject-matter expert: the discipline expert the course requires, here a data scientist, PhD-level, 20+ years.Checks technical accuracy and whether each artifact is faithful rather than misleading, here the statistics and the data visualizations.

Instructional designer: a learning-experience designer, PhD-level, 20+ years.Checks alignment, clarity, cognitive load, accessibility, and consistency across the suite.

Synthetic student: a graduate STEM learner.Reads each page as the person who has to use it, and flags what is confusing, unlabeled, or unclear.

What the panel caught

Subject-matter expert (data scientist)

A real technical error, caught before a student could. One page credited PCA with keeping a single feature from dominating the distance metric. That is what standardization does (scaling the features so no one of them can dominate); PCA does not, and on unscaled data PCA is itself dominated by the highest-variance feature. Split into two accurate statements.This is the strongest evidence for the method: the expert agent caught a genuine mistake a single builder had missed.
A misleading chart, corrected. The governance cover’s “overall” line sat below most of the bars, which shows the disparity rather than hiding it. A real size-weighted aggregate sits high, near the majority groups, while one small subgroup collapses far below. Redrawn so the visualization tells the true story of disparate impact (one subgroup harmed) concealed by an aggregate.
Fidelity fixes to the other data visuals. The clustering scatter’s ambiguous points now sit between the clusters (matching the “how many, really?” premise, and its own alt text); the capstone time series’ drift marker now lands exactly on the break in the line.

The catch, before and after.Before: “Standardizing features and justifying PCA so no single column dominates the distance metric.” After: “Standardizing features so no single column dominates the distance metric,” and, as a separate point, “Justifying PCA for decorrelation and dimensionality reduction, not as a scaling fix.”

Instructional designer (learning-experience designer)

Orient the learner first. Add a one-line label to every simulation stating its module and whether it is a graded assessment or a preparation tool.
Make the throughline explicit. State plainly that the governance simulation carries forward the very model the student audited in the model-evaluation module, so the sequence reads as one story.
Lower the vocabulary load. Expand “NIST AI RMF” (the NIST AI Risk Management Framework) to its full name on first use, standardize the repeated framing across pages, and cut redundancy.

Synthetic student

Give me a first step. Landing on a graduate simulation with no visible “start here” is intimidating; add a clear entry cue.
Do not assume the acronyms. The framework name read as a wall of letters before it was spelled out.
Tell me what counts. Distinguish the practice tools (the capstone prep and the defense rehearsal) from the graded assessments, so I know what is being assessed.

What changed

Result

Every finding above was implemented before the suite reached a student.

Why it matters

The panel is not a rubber stamp. The expert pass caught the failures a single builder stops seeing after enough hours in the work, and the instructional-design and student passes then closed the gap between “correct” and “usable.” None of it replaced my judgment; I accepted, adjusted, or declined each note.

See the reviewed simulations in the Authentic Assessment suite, or how the panel drafts a course from the start in The Build Skill.