Dial Your Course
Case study · AI Tools & Strategy · Learning Design
A browser-resident course review instrument: a faculty member drops in a Canvas course export, nineteen checks run against it, and the approved fixes come back as a package ready to reimport.
1. Summary
Online course preparation contains a substantial deterministic component that is currently performed by reading, and therefore performed inconsistently: alternative text, heading structure, contrast, deadline distribution, competency coverage, workload against the credit contract, and the presence of mandated language. These are checkable properties of a file.
Dial Your Course parses an IMS Common Cartridge export (.imscc), computes nineteen independent checks from the parsed structure, reports each finding with the figures that produced it, and, on the instructor’s approval, rewrites the package and returns it for reimport. All computation occurs in the browser. Nothing is transmitted.
A second problem compounds the first. The Canvas rich content editor sanitizes HTML on save, discarding <style> blocks, external stylesheets, and class-based CSS. Only inline style attributes survive. The instrument writes styling inline for that reason, so a designed, accessible template arrives intact.
The instrument reports, it does not certify. A check that cannot compute does not fire. No finding is emitted without the figures behind it, and no standard is reported as satisfied unless it was verified. Twenty-three of the fifty OSCQR standards concern conduct during the term or require a judgment of quality; those are enumerated with the established method for evaluating each, and are not scored.
2. Goal
Reduce the mechanical component of online course review to a single pass over the course package, and return a remediated package rather than a checklist.
The driver is seat time. The Higher Learning Commission has flagged seat hours in online courses at this institution. A credit carries a fixed allocation of student learning hours and the syllabus commits to them, but in an online course there is no classroom to observe, so the course package is the only evidence the hours were delivered. It is also the artefact a reviewer will read.
Three questions follow from that, and the instrument answers all three. Does the work fit the hours the credit allows? Are the objectives written in verbs a student can be observed performing? Is every outcome the course claims actually evidenced by something it grades? Assessment validity follows from the third: which graded items could a model complete unaided, and what process evidence does the rubric already collect?
The instrument did not begin with a rubric. It began with the checks its author performed by hand, cycle after cycle, as a peer and lead reviewer for Quality Matters and OSCQR. It automates the countable part of that review and declines to automate the part that takes judgment.
3. Users and context
| User | Context | What the instrument gives them |
|---|---|---|
| Faculty | Preparing an online course for delivery or for formal review. No technical fluency assumed. | A private rehearsal for review, run on demand, on their own machine. Findings tied to the standard and the location in the course, and a corrected package returned. |
| Instructional designers | Supporting many courses across a term, with the countable checks consuming the hours that consultation should have. | The mechanical pass done before the conversation starts, so their time goes to design judgment rather than to counting alt attributes. |
| Departments and programs | Multiple sections, multiple instructors, one approved course record. Preparing for program review. | Consistency across sections, and outcome coverage that can be shown rather than asserted. |
| Reviewers | Formal Quality Matters or OSCQR review, once a year for most courses. | A course that arrives already correct on everything establishable mechanically, so review time is spent where professional judgment is required. |
Formal review is thorough, valuable, and for many faculty exposing, because the first close reading a course receives is frequently the one that evaluates it. Small and entirely fixable problems (a missing alternative text attribute, an outcome never attached to a rubric, a module whose workload has drifted past the credit contract) surface during review rather than before it. This instrument is a rehearsal, not a substitute. It reports to the instructor and to no one else: no score is submitted, no record is retained, no administrator receives the result.
4. How it works
Export the course from Canvas, drop the file in, choose the checks. The parser constructs a course object from the cartridge, and every check is a pure function of that object. Each finding names the standard it comes from and where in the course to fix it. The instructor approves the changes, the instrument writes them back and repackages the cartridge for reimport, and it prints a short list of the few things only Canvas itself can do.
Does the course hold together
- Canvas setup: outcome objects, rubric bindings, assignment groups, and the weighting scheme, reported as one task with a sequence rather than as separate defects.
- Outcome to assessment alignment, presented as an alignment matrix.
- Course competencies resolved to the assignment and criterion that assess them.
- Measurable objectives (OSCQR 9), tested against a verb bank, with observable replacements at the appropriate level of Bloom’s Taxonomy.
- Assessment validity under generative AI, and the AI resistance audit.
- Seat hours and workload against the credit contract.
Can a student use it
- Universal Design for Learning (the CAST framework, distinguished from accessibility, with which it is routinely conflated).
- Course overview and information (OSCQR 1 to 10), and technology and support (11 to 15).
- Design and layout (16 to 28): alternative text, descriptive link text, table header cells, typeface.
- Content and activities (29 to 37): captioning and PDF accessibility.
- Interaction (38 to 44), and assessment and feedback (45 to 50).
Remediate and reimport
- Visual consistency: counts distinct typefaces, sizes, and colours, applies a selected palette inline, and verifies contrast against WCAG AA at selection, so a failing pair cannot be applied.
- Due dates: flags dates carrying more than two deadlines, and graded items with no due date, which appear in neither the student To Do list nor the calendar.
- Student data scan of course Files, which flags and never deletes.
- Export round-trip.
Not scored, enumerated for human review
Ten standards cannot be verified from a file, including Regular and Substantive Interaction (34 CFR 600.2) and the think-aloud protocol (Ericsson and Simon, 1984). Each is presented with the reason a parser cannot evaluate it and the established method for doing so, with a checkbox retained in the browser. The default is unchecked, because nothing has yet verified them.
Seat hours
Total student hours equal credits multiplied by forty. Load hours alter how that total is split between direct instruction and outside work; they never alter the total. The figure that matters to an instructor is the outside-work allowance, because that is the portion the course package spends: the reading, the video, and the graded work. It sits on the approved course record, is not recoverable from the export, and is therefore requested from the instructor.
The instrument estimates hours for every graded item and every page of reading and video, reconciles the aggregate against the allowance at module and course level, makes every estimate correctable, and writes the confirmed figure into each module overview, which satisfies OSCQR Standard 2. A student who knows what a module costs plans for it.
An estimate is not evidence. Video duration is assumed, because the cartridge stores links rather than durations, and effort per graded item is inferred from point weight, which is a proxy. A wrong number in a module header is worse than no number, because students plan around it. The remedy is not a better estimator but measurement: one question at submission, answered by the people who did the work, converts the published figure from a guess into a claim with data behind it.
Assessment validity and the AI resistance audit
Two checks operate here and they are deliberately distinct. The first evaluates each graded item against two conditions: whether a generative model could produce a passing response within sixty seconds, and whether the item carries consequential weight, defined as at or above the median point value of the graded items in the course. Only the conjunction constitutes a finding. A low-stakes formative check is rehearsal, not measurement, and a model completing it costs the course nothing. Where an item fails both conditions, the finding is framed as a threat to construct validity rather than as academic integrity, and a substitution at equivalent cognitive demand is proposed.
The second inverts the question. It reads the rubric criteria of every graded item, not the assignment titles, and reports which of seven kinds of process evidence each one already collects: physical capture, process artefacts, critique given, revision from a named critique, articulated rationale, situated content, and file-level evidence. Each measured assessment is banded (strong, adequate, exposed) and returned with two or three specific additions. Low-stakes items are listed and explicitly not judged.
Authenticity of task does not confer resistance to generative substitution. A model will write a portfolio case study, a client rationale, and a reflective essay, all of which are authentic tasks. What resists substitution is the record of making: iteration and drafting, documented response to critique demonstrably received, situated local context, and live defence of a decision. The artefact never protects the assessment. The record of its making does. The audit therefore measures what a rubric collects, not what an assignment asks for.
Accessibility, and the boundary of what a parser can see
The instrument verifies the structural properties recoverable from the package: alternative text on every image (OSCQR 36), descriptive link text (37), header cells on every table (24 to 26), typeface (23), and captions required on video (35). It does not open PDFs and does not claim to. It counts them and routes the instructor to YuJa Panorama, which the district already licenses and which scores files in place, performs text recognition, tags headings and reading order, and generates alternative formats. A check that reimplements a tool the institution already owns adds no capability and one more thing to maintain.
Two boundaries are stated rather than papered over. OSCQR 23 (typeface) is SUNY’s editorial standard, not an accessibility requirement: neither WCAG nor Section 508 specifies a typeface family, so the instrument reports the departure without characterising it as an accessibility defect. And caption accuracy is not verified by automated checking anywhere, including here. Automated checkers confirm that a caption track exists. Automatic speech recognition runs at roughly 70 to 90 percent accuracy and degrades most on domain terminology, which in a technical course is frequently the content being taught. An unedited track can satisfy a compliance check while remaining difficult to use for the student who depends on it.
5. Data, privacy, and governance
A course export is instructor content, not student content. Canvas intentionally excludes enrollments, grades, submissions, quiz attempts, and student-authored discussion replies from a course export, and Instructure omits them for FERPA compliance. The cartridge carries course content only, so the instrument processes no student data and no FERPA-protected record, and requires no policy governing either, because it receives neither.
Nothing is transmitted. The cartridge is opened, read, rewritten, and repackaged inside the browser tab. There is no server, no account, no vendor intermediary, and no network request carrying course content, and the claim is verifiable in the network panel during execution. The privacy position is architectural rather than contractual: there is no processor to contract with.
The only thing stored is the set of human-review checkboxes, in localStorage, on the reviewer’s own machine, so a review can span sessions. No course content is retained.
Residual risk in course Files
Course Files travels with every export and every course copy, and is published by default. Instructors accumulate gradebook spreadsheets, rosters, and letters of reference there. The instrument scans filenames and types, flags candidates, and does not delete, because removing an instructor’s material without instruction is a worse defect than the exposure it addresses.
| Security review question | Response |
|---|---|
| Where is course data transmitted? | Nowhere. All processing occurs in the browser. Verifiable in the network panel. |
| Is student data processed? | No. A Canvas export excludes enrollments, grades, and submissions by design. |
| Is authentication required? | No. The instrument holds no state and no account. |
| What is retained? | Human-review checkboxes in localStorage on the reviewer’s machine. No course content. |
| Third-party dependencies? | JSZip, MIT licensed, loaded from a CDN and vendorable beside the file. |
| Accessibility of the instrument itself? | Semantic markup, keyboard-operable disclosure controls, AA contrast throughout. |
6. Build and portability
A single static HTML document. Vanilla JavaScript, no framework, no build step, no server-side component, no database, no runtime configuration, no authentication. The one external dependency is JSZip, loaded from cdnjs.cloudflare.com; where an institutional Content Security Policy prohibits third-party script sources, JSZip is vendored beside the HTML file and the script tag repointed. That is the only modification any hosting environment is expected to require. Deployment consists of placing one file where a browser can reach it, which is what makes district-scale hosting a question of a web request rather than of budget: no per-seat license, no inference bill.
Built with a model, shipped without one
Both claims are true and they are separate. The instrument was designed and written in sustained collaboration with a language model: the Canvas cartridge schema was reverse-engineered against real exports that way, and the checks were drafted, adversarially tested, and corrected that way. What ships contains no call to a language model, no credential, and no network request. Every finding is produced by deterministic parsing and a decision table. Where the report proposes a remediation, that proposal is a rule with an unmet condition, not a generated sentence.
This is a design decision with three consequences a generative implementation could not offer. It cannot hallucinate a finding or a standard. It cannot leak, because nothing is transmitted. It costs nothing to run. The deterministic checks are also better for this class of problem: verifying that mandated language appears verbatim is a string comparison, and a model performs worse at it because it accepts paraphrase as equivalence.
The cost is symmetrical and should not be obscured. The decision table cannot notice an outcome its author did not consider, and it will propose an outcome for a course that merely uses a trigger word in passing. A model would perform better at that and considerably worse at inventing nothing. For an instrument whose only asset is that a faculty member believes its findings, that is the correct trade, and it remains a trade.
Is the model swappable
Today there is no model to swap: version 1 is deterministic end to end. Version 2 surfaces five capabilities the instrument already detects and cannot resolve without generation (draft alternative text, rewritten objectives, a situated assessment substitution, caption accuracy, and prose clarity), and they are currently disabled in the interface.
The generative layer is specified as model-agnostic. It sends only flagged passages, never the course package, and it receives text that arrives as an editable draft subject to the same approval as any other change. Any chat-completion endpoint satisfies that contract: Claude, Google Gemini on Vertex AI, or a district-hosted model behind an internal endpoint. Swapping the model means changing one endpoint and one request shape, not the checks, because the diagnostic work is deterministic and stays that way.
One constraint governs all of it: a credential cannot reside in the client. The instrument is a static document served to the browser, so any embedded credential is a published credential. This is a property of the delivery model and is not mitigable by obfuscation. Three options exist.
Bridge, with no infrastructure
The instrument composes a prompt containing only the flagged passages. The instructor pastes it into the model the institution has already sanctioned and pastes the reply back. No credential, no server, and no request originates from the page, so only what the instructor personally pastes leaves the machine. It is inelegant, it is available immediately, and it is the recommendation for the pilot.
Instructor-supplied key
Each instructor obtains a developer key and stores it in their own browser, and the instrument calls the model API directly. This does not scale: each instructor must obtain and fund a key, and a key held in a client application is a recognised exposure class that institutional IT is unlikely to sanction. Not recommended.
Institutional proxy
A minimal server-side endpoint holds one institutional credential (a service account or a Workload Identity binding), authenticates instructors through existing single sign-on, forwards the flagged passages, and returns only the generated text. No credential reaches the browser. This is the only path that scales past a pilot, and it requires a cloud project, an endpoint, and a data-handling position. It is the institutional ask.
What the institution would provide for the proxy: a sanctioned endpoint reachable from a proxy the college controls (Vertex AI if the district has a Google Cloud project; a developer API is a prototyping surface, not an enterprise control plane, and Gemini in Google Workspace is an end-user surface that exposes no programmable endpoint, which is a common source of confusion in procurement); a server-side credential, rotated on the standard schedule, never in the client; and a data-handling position covering flagged passages only. That last point is what keeps the request document-scale rather than corpus-scale, the inference cost negligible, and the review tractable.
Could it run on district infrastructure, or inside Canvas
| Stage | Method | Requires | Suitable for |
|---|---|---|---|
| Local | The file is opened from disk | Nothing | Individual evaluation |
| Present hosting | GitHub Pages, the current arrangement | Nothing | Pilot with named colleagues |
| Canvas | Uploaded to course Files and linked, or embedded in a page | Instructor rights only | Departmental use without an IT request |
| College web server | A static file under a college path | Web team, one file | College-wide availability under an institutional URL |
| District | Static hosting under a district domain | District IT, security review | District-wide adoption |
It runs inside Canvas today, as a linked or embedded static file, with no LTI registration and no district action. An LTI placement would only change how it is launched, not what it does, because it holds no state and needs no roster. Moving to district infrastructure means copying one file and, if a Content Security Policy applies, vendoring JSZip.
District infrastructure now in progress (roadblock today, the intended path)
The pieces this section describes as options are becoming concrete. A pilot protocol has been requested through the AI Resource Center to reside at the district, and the district plans to stand up a small sanctioned model instance (Claude, and possibly a second such as OpenAI) that approved pilots can call. That is where any AI feature would run, so no instructor key and no personal credential is needed. A district GitHub account has also been applied for, which would host the tool under district control in place of a personal account. Until those are live the tool ships without a model and uses the bridge, and hosting stays on the present arrangement. Once they land, the institutional-proxy path and district hosting above stop being hypothetical. This is a roadblock today and the intended path forward.
7. Version 2, from detection to redesign
Version 1 diagnoses and stops. The assessment validity audit flags which graded items a model could complete unaided and which rubrics already collect a record of the student’s thinking, then hands the finding to the faculty member. It does not say what to do about it. Version 2 answers that question: for a flagged item, it proposes a more authentic version of the assessment, and the faculty member decides whether to use it. The instrument moves from telling a faculty member their assignment is exposed to offering them a redesign that is not.
The premise is the one the audit already rests on. When a model can produce the artefact, the artefact stops being evidence of who produced it. That is a validity problem, not an academic-integrity problem, and it is not solved by a harder prompt or a detector. It is solved by changing what the assessment collects, moving the evidence from the finished artefact to the record of making it, at the same level of cognitive demand the original asked for.
How it works with the AI model
Version 2 is the generative layer specified in section 6, and it is deliberately model-agnostic. For each flagged item, the instrument sends only the flagged passage, the assignment prompt and its rubric criteria, to a chat-completion endpoint. It never sends the course package, and it never sends student data, because none is present. The model returns a proposed redesign as an editable draft, which the faculty member approves, edits, or discards under the same approval gate as every other change the instrument makes. Nothing is rewritten without instruction.
The endpoint is a contract, not a vendor. Claude, Google Gemini on Vertex AI, or a district-hosted model all satisfy it, so moving from the Claude the tool was built on to the Gemini the district is standardising on is one endpoint and one request-shape change, with the deterministic checks untouched. The one hard rule is that a credential cannot sit in a static page served to the browser: for the pilot the instrument composes the prompt and the faculty member pastes it into the model the institution has already sanctioned (no credential, no server), and to scale, a minimal institutional proxy holds one server-side credential behind single sign-on. The detection stays deterministic and free; only the suggestion calls a model.
The suggestions are discipline-aware
Authentic process evidence is not the same in every field, so the redesign a flagged item receives depends on what making the work actually looks like. A studio critique leaves a trail of iteration and response to feedback, a lab leaves a notebook, a programming task leaves a commit history, and a writing task leaves drafts, sources, and decisions. Version 2 reads the rubric criteria and proposes the evidence that fits the discipline rather than one generic fix. The hardest case, and the one worth working through, is the one every program has: the paper.
The hard case, English 101 and the essay
A first-year composition course is built on the essay, and the essay is precisely what a generative model produces best. The artefact cannot carry the assessment, and no amount of prompt engineering makes an essay a model cannot write. Version 2 does not pretend otherwise. It does not propose an “AI-proof” essay or a detector. It proposes moving part of the grade onto evidence a model cannot generate on the student’s behalf, so the paper stops being the only thing assessed.
- Anchor the prompt to something only this student encountered. A specific class discussion, a local source the student found and annotated, a personal artefact, a text read in this section this week. A model does not know what the student saw, so it cannot ground a response in it.
- Collect the record of making. An outline committed before drafting, the revision history, an annotated bibliography in the student’s own words, and a short paragraph on what they cut and why. This is the trail a finished essay does not contain.
- Add a brief defence. Three questions generated from the student’s own specific claims, with follow-ups. A student who did not do the thinking cannot sustain “why not the other reading?” on work they did not do.
- Grade the disclosure, not the absence of AI. Ask the student to mark what the model produced and what they wrote, and to revise the model’s draft against a named critique. Grade the delineation and the revision. This is the only move that assesses the student who used AI well, which is the student the course actually wants to graduate.
The boundary is plain: none of this makes a paper impossible to write with a model, and that is the point rather than a failure of it. The goal is not an unbeatable artefact, it is an assessment where the artefact is no longer the only evidence, so the grade rests on a reasoning trail the student can show and defend. Making that shift assessable without surveillance, by asking for the process as a deliverable rather than monitoring the student, is the whole design problem, and it is the same one every writing program in the country is now facing.
The alternate-assessment menu
The English 101 case is one path through a general menu. Version 2 carries a categorised bank of authentic, generation-resistant substitutions drawn from the current two-lane consensus: a secured lane that validates individual achievement, in person or oral and supervised, and an open lane where AI use is scaffolded and the process, not the product, is graded. For a flagged item the instrument proposes two or three substitutions at equivalent cognitive demand, selected by what making the work leaves behind: process and portfolio evidence, iterative drafts against a named critique, an oral defence or walkthrough, a supervised in-class performance, a prompt localised to what only this student encountered, an applied project with a real client or constraint, reflection tied to a specific revision, applied identification in place of a recall quiz, or disclosure-graded AI use where the student marks what the model produced and revises it against a critique. Each is grounded in the reference library rather than generated from nothing.
Universal Design for Learning, as recommendations rather than a label
Version 1 names UDL and delivers structural accessibility, which is not the same thing. Accessibility asks whether a student can reach the content; UDL asks whether there is more than one way to engage with it, perceive it, and act on it. Version 2 makes UDL checkable against the CAST 3.0 guidelines and their three principles, engagement, representation, and action and expression. It flags content that exists in a single modality and proposes a second, an assignment that accepts one submission format and proposes an alternative means of expression, and a module with no learner choice, no self-assessment, or no activation of prior knowledge, and it offers the specific addition rather than a citation to the framework. The distinction from accessibility is stated in the report, because the two are routinely conflated.
The RSI writer
Regular and Substantive Interaction cannot be read from a package, which is why version 1 enumerates it and stops. It is also the federal line (34 CFR 600.2) that separates distance education from correspondence and protects Title IV eligibility, so an empty result is a poor place to leave a faculty member. Version 2 turns the dead end into a deliverable. It generates a drop-in interaction plan and RSI statement the instructor edits and pastes into the syllabus and course: a weekly instructor-authored teaching announcement, a stated feedback schedule, an instructor-facilitated discussion, a published response-time commitment, and a monitoring-and-outreach policy that uses Canvas analytics by hand and collects no student data. The draft is checked against the OSCQR interaction standards and the two-of-five substantive-interaction test the regulation requires, direct instruction, feedback on coursework, answering content questions, facilitating discussion, or an institution-approved activity.
The instructional-design lens
The same findings can be read through the model a faculty member was trained in. A checkbox reframes the report through a chosen lens without changing what was found. Backward Design asks whether every objective has evidence and every graded item serves an objective, which is the instrument’s native logic. ADDIE surfaces the missing evaluation loop. Merrill’s First Principles flags a module that jumps to application with no worked example or real problem. Gagne’s nine events read as a per-module checklist. Dick and Carey emphasise where one component has drifted out of alignment with another. Connectivism and personal learning environments, the author’s own theoretical floor, value real audiences, networked tools, and a public artefact, and flag a course that is a closed loop. The instrument does not impose a model; it offers the vocabulary the instructor already thinks in, and a course does not use every model at once.
Builder mode
Everything above audits a course that exists. Builder mode scaffolds one that does not. It borrows the pattern from Wayfinder: the page does the structured work, and the model, through the same bridge, does the open-ended reasoning. The instructor states the course objectives, the lesson objectives, and the topic, and the builder returns a starting scaffold at the right level of Bloom’s Taxonomy: a module outline with a stated time cost inside the seat-hour budget, an assessment matched to each objective at equivalent cognitive demand, an RSI plan, and UDL options for engagement and expression. It produces a starter package to import and, where a suggestion needs generation, a prompt to hand to the sanctioned model. The ask is deliberately small, objectives and a topic, so the useful first output arrives before any file exists.
Why the builder takes objectives by hand
Builder mode asks for the objectives directly rather than reading them from a package, and that is a decision the export forced. On the test course, verbed outcomes visible in Canvas were absent from the export, reproduced twice, with the outcomes file not written at all rather than written empty. The leading explanation is a cached has_outcomes flag on the course record that the CSV importer never clears, and a separate confirmed defect makes account-level outcomes serialise as references that cannot resolve on reimport. Both are documented in the outcomes-export research (section 11) and both are unfixable from a package. Two cheap tests are queued to learn whether the drop is universal or particular to that one course: re-save the course settings to bust the cache and export again, and failing that, bind one outcome to a rubric criterion and export again. Until they resolve, any feature that needs objectives, the builder most of all, asks the instructor for them rather than trusting the export to carry them. If the tests show a normally built course does export its outcomes, the builder can offer to read them; the manual path stays as the fallback either way.
One tool, four front doors
Version 2 presents as tabs over one instrument that shares one reference library and one approval gate. Audit is the version 1 checks. Builder scaffolds from objectives. Syllabus brings in the syllabus checker, currently a separate Apps Script tool, so the required-language, seat-hour, and RSI checks sit beside the course audit. Report is the findings, exportable. Rolling the syllabus checker in removes a second tool to learn and lets the seat-hour figure and the RSI statement move between the syllabus and the course without retyping.
Export in more than one format
Version 1 exports one thing, a Canvas cartridge for reimport. Version 2 keeps that and adds report and rebuilt-course exports that need no server: a standalone HTML report, which is the page serialised with its styles inlined and satisfies the rule against linking a downloadable document; a print-to-PDF stylesheet with an optional one-click PDF; and a Word export for an instructor who wants a rebuilt course or a report offline. The cartridge stays the primary course export; the rest are for reading and sharing.
The reference library
The suggestions are only as good as what grounds them, and a model left to invent them produces plausible, generic, and occasionally wrong advice, which is exactly what an instrument whose whole asset is trust cannot afford. Version 2 grounds every suggestion in a curated reference library held beside the tool as versioned files: the authentic-assessment menu, the CAST 3.0 UDL guidelines, the RSI practices and the regulation, the instructional-design models, and an OSCQR-to-Quality-Matters crosswalk. Retrieval is deterministic, the finding type selects the reference block, and the model writes only the last sentence against material it did not invent, the same line the tool already draws between a decision table and a generated phrase. The library is authored and kept current at build time, not at run time: a build-time pass queries the open scholarly indexes, OpenAlex and ERIC, both free and keyless, the author vets what comes back, and the vetted claim and its citation go into the library. The running tool never calls those indexes, so there is no latency, no rate limit, and no unvetted text in a trust-critical path. NotebookLM is a useful workbench for that authoring; it is not the tool’s engine, because it exposes no stable endpoint to build against.
Status: version 2 is design only. Its capabilities, the alternate-assessment menu, UDL recommendations, the RSI writer, the instructional-design lens, builder mode, the rolled-in syllabus checker, the four-tab shell, multi-format export, and the reference library, are either detected by version 1 today and disabled, or specified and not built. Nothing here is validated with faculty, and it ships only after the version 1 pilot and a faculty review of whether the suggestions read as credible to the people who teach the subject. The line holds throughout: detection stays deterministic and free, the model writes only last-mile suggestions against the grounded library, every suggestion is an editable draft under the same approval gate, and no finding, binding, or outcome is ever fabricated.
8. Pilot and testing
The pilot takes routine checking work off faculty and instructional designers. It does not replace either. Every finding goes to the instructor, who decides; the instrument revises nothing without instruction, and the judgments that require professional expertise are not automated at all, they are enumerated with the method for evaluating each and handed back.
| Element | Plan |
|---|---|
| Who | Three to five faculty across two disciplines, starting in Digital Media Arts, plus the instructional designers who support those courses. |
| What | One live online course each, run end to end: export, check, set the seat-time estimates, approve, export the corrected package, reimport, and verify every change landed. |
| Period | One term, followed by departmental adoption across all Digital Media Arts sections for a second term. |
| Measured | Every finding known to be false, recorded by severity. Which findings were acted upon and which were dismissed. Whether the corrected package reimports cleanly. How long the pass took against the same review done by hand. What the instructor had to correct in the workload estimate. |
| Reported | To the Teaching and Learning Centre, with pilot data rather than a demonstration. |
A false positive on a high-severity finding is the failure mode that destroys confidence in an instrument of this kind, and surfacing it is the pilot’s purpose. A finding that is universally dismissed is either incorrect or badly worded, and both are defects.
9. Definition of success
Version 1.0 means the pilot passed and the instrument is cleared for rollout. The thresholds are stated in advance so the decision is not made by impression.
| Threshold | Condition to clear |
|---|---|
| Correctness | No false positive on a high-severity finding survives the pilot uncorrected. Any finding a pilot participant can demonstrate to be false is fixed or withdrawn before rollout. |
| Round-trip integrity | Every corrected package reimports into Canvas with the file count preserved, all XML byte-identical, and no styling discarded by the editor. |
| Usefulness | Each participant acts on findings they would not otherwise have caught, and no check is dismissed by every participant. A check dismissed by all is rewritten or removed. |
| Seat hours | Each pilot course publishes a confirmed outside-work figure per module, corrected by the instructor, and at least one course closes the loop by asking students at submission whether the published budget matched the work. |
| Trust | Participants report they would run it again unprompted, on a course of their own, without being asked. |
Nothing here is in production, and no figure above is claimed. They are the conditions under which the prototype would become a 1.0.
10. Rollout
- Departmental adoption. Digital Media Arts, all sections, one term, with the findings log kept.
- Present to the Teaching and Learning Centre with pilot data, and hand them the instrument to offer as part of course preparation support.
- College hosting. One static file, one request to the web team, JSZip vendored in anticipation of an institutional Content Security Policy.
- District hosting under a district domain, following a security review. The questions that review will ask are answered in section 5.
- Generative layer, if the pilot supports it. Bridge first, at zero cost and zero risk. The institutional proxy is requested only with pilot evidence behind it, never in advance of any.
Explicitly out of scope
| Capability | Why it is out of scope |
|---|---|
| Batch analysis across many courses | Reports to an institution rather than to an instructor. That is a different instrument, with a different consent model. |
| Modality equivalence, an online section against its in-person master | Requires two exports and an authoritative course master. Feasible, and a reasonable later version, but it answers an institutional question rather than a self-check. |
| Reporting to an accreditation or assessment office | The instrument’s usefulness depends on faculty trusting that it reports to them alone. Routing results elsewhere would end that. |
| Student engagement monitoring | Requires continuous student data, which crosses the no-student-data line the whole design rests on. Canvas New Analytics already serves that need. |
These are recorded because they are the predictable next requests. Declining them is a design decision rather than an oversight.
11. Open questions and risks
Outcomes do not reliably survive an .imscc export. This is a known Canvas limitation and a real constraint on rollout. On the test course, five outcome groups and fourteen verbed outcomes are visible in Canvas, imported by Outcomes CSV. The export contains none of them, and course_settings/learning_outcomes.xml is absent rather than empty. Reproduced on two separate exports. Until outcomes reach the export, the outcome-alignment half of the instrument cannot run on that course. Seat time and objectives still run, because assignments do export.
The leading explanation is a cached has_outcomes? boolean keyed on the course record: the CSV importer creates the outcomes without touching the course row, a stale value persists, and no file is written at all, which matches the symptom exactly. Two tests are queued, cheapest first: re-save course settings to bust the cache and export again; failing that, bind one outcome to a rubric criterion and export again. A separate, confirmed defect will bite on reimport regardless: account-level outcomes (the institutional learning outcomes) serialize as references, not copies, so on import the identifier cannot resolve and the criterion arrives as a plain criterion with no outcome attached. No tool can fix that from a package. It is two minutes each, by hand, in Canvas, which is what the post-export to-do list exists for.
| Open question or risk | Priority | Notes |
|---|---|---|
| Does any campus role hold account-level Outcomes rights? | High | This single fact determines whether program outcomes are a two-week problem or a two-year one. Program outcomes are currently absent from Canvas at every level, and entering them by hand per course would create one unrelated object per course, which Canvas cannot aggregate. The unblock is one administrative action: create a program outcome group at the account or sub-account level, after which Find Outcome works for program outcomes exactly as it does for institutional ones. |
| Workload is an estimate, not a measurement | High | Video duration is assumed, effort is inferred from point weight. The figure is a starting position for a curriculum conversation, and it becomes evidence only when students confirm it at submission. |
| Assessment validity is a pattern match | Medium | It identifies items exhibiting the structure of a model-completable assessment. It does not establish that one is. |
| Course information is detected by keyword | Medium | Where that information lives in Simple Syllabus rather than in a Canvas page, it will not appear in the export and the finding will be wrong. |
| The district course record cannot be retrieved | Medium | The Curriculum Office does not serve its pages to scripts from another origin, so competency text must be compared by the instructor. No browser-resident tool can do otherwise. |
| Content Security Policy | Low | JSZip is loaded from a CDN. Vendoring it beside the file is a one-line change and should be done before any institutional hosting request. |
12. Roadmap
| Milestone | Target | Notes |
|---|---|---|
| Parser and nineteen checks | Built | Operational against Canvas Common Cartridge exports, across all package states, including outcomes absent, present and unbound, and bound. |
| Export round-trip | Built | File count preserved, all XML byte-identical, output reduced to ASCII with numeric entities, no <style> block emitted, package reimports. |
| Sample report and style guide | Built | Published alongside the instrument. |
| Outcomes export tests | Next | Cache-bust test, then the rubric-binding test. Determines whether the alignment check can run on a course whose outcomes came in by CSV. |
| Departmental pilot | Scheduled | Three to five faculty, two disciplines, one term. Every false finding recorded by severity. |
| Vendored JSZip | Before hosting | In anticipation of an institutional Content Security Policy. |
| College hosting | Post-pilot | One static file, one request to the web team. |
| Generative layer, version 2 | Post-pilot | Bridge first. Institutional proxy requested with pilot evidence behind it. Five capabilities, all detected today and disabled: draft alternative text, rewritten objectives, situated assessment substitutions, caption accuracy, prose clarity. |
| District hosting | Post-pilot | Following security review. |
OSCQR, the SUNY Online Course Quality Review Rubric, is made available under a Creative Commons Attribution 4.0 International License by the State University of New York. This instrument applies the rubric and is not affiliated with or endorsed by SUNY.