Michelle Blomberg

Dial Your Course

Three questions about the course, answered from the course package itself.

  • Is it aligned? Every stated outcome is assessed by graded evidence.
  • Is it consistent? A single visual system across every page, retained through the Canvas editor.
  • Is it correctly sized? The required work fits the student hours the credit load allows.
What it scores against, and what it will not claim

The instrument checks three bodies of criteria. Constructive alignment: whether stated outcomes are carried by graded evidence, and whether the gradebook matches the syllabus. The SUNY OSCQR rubric: 27 of its 50 standards can be verified from a course file; the remainder depend on instructional practice during the term and are listed separately as guidance, not scored. Criteria no published rubric yet addresses: workload against the seat-hour budget, assessment validity under generative AI, and Institutional Learning Outcomes.

A faculty-facing self-review instrument. It reports to the instructor and revises nothing without instruction. The course package is opened, read, and rewritten in the browser; the revised package is returned as a download, ready for reimport. No data is transmitted to any server.

Step 1 · Add your course

In Canvas: Settings → Export Course Content → Course. Studio and YouTube videos export as links; no video files are required.

✓ A course export contains no student data, and nothing is uploaded

Why. Canvas builds the package from course content only. Enrollments, grades, submissions, quiz attempts, and student discussion replies are excluded by the exporter, for FERPA. There is nothing here to collect, and this tool could not collect it if there were.

Nothing is uploaded. The file is opened, read, and rewritten inside this browser tab. There is no server. Nothing is retained when you close it. You can confirm that by watching your browser’s network tab while it runs.

The one real exception, and it is worth knowing: the package does include everything ever uploaded to course Files. If a roster, a gradebook spreadsheet, or an annotated example with a student’s name on it was ever put there, it ships with the export. That is not a Canvas defect and it is not a flaw in this tool. It is a thing people do. This tool scans for it and flags it, and it will not delete anything.

Step 2 · Choose what to check

1 · Does the course hold together?Outcomes, evidence, and workload. Nothing downstream survives a bad objective, so this runs first.

Canvas setup Verifies that learning outcomes exist as Canvas objects, that assignment groups are defined and weighted, and that rubric criteria are bound to outcomes. Each finding names the setting and where to change it.
Measurable objectives  OSCQR 9Flags unmeasurable verbs ("understand," "be introduced to"), suggests observable replacements at the right level of Bloom’s Taxonomy, the standard framework (Anderson & Krathwohl, 2001) that orders learning from Remember through Create, checks the level fits the course. Uses a verb bank, not a model. “Understand” is a valid level of Bloom’s Taxonomy and an invalid objective verb. The level names what happens in the learner’s head; the verb has to name what you can watch them do. If you cannot observe it, you cannot assess it.
Outcome → assessment alignment Builds the alignment matrix: every outcome, and the graded evidence that substantiates it. Flags outcomes that are claimed but never measured, and graded work that carries no outcome.
Competency alignmentLists the competencies the course carries and opens the district record beside them for comparison. The board-approved text is the contract; the syllabus paraphrase is not. Course your institution’s curriculum office
Authentic assessment  OSCQR 31, 45 · and beyondAsks one question of every graded item: could an AI do this, correctly, in under a minute? If yes, it is no longer measuring your student. Flags multiple-choice quizzes, definition matching, take-home essays, and generic reflection prompts, and offers a replacement at the same cognitive level. Not a lecture. A defect report with a fix. A caution: authentic does not automatically mean AI-resistant. A student can have AI write a portfolio case study. The artifact never protects you, the process does. Drafts, iteration, response to real critique, live defense, situated context.
AI resistance  Process evidenceReads the rubric criteria of every graded item and reports which of six kinds of process evidence it collects: physical capture, process artifacts, critique given, revision from a named critique, articulated rationale, and situated content. Names what to add to each. An artifact never protects an assessment; the record of its making does.
Seat hours and student workloadEstimates each module, writes the total into the module intro (OSCQR 2), checks the course against its seat-hour budget, and verifies the approved seat-hours language. Try the sliders → Credits  Outside-work hours Total student hours = credits × 40, always. The outside-work allowance is the part of that total the course package spends, and it is on the approved course record. Neither figure is in the export; supply them.

2 · Can a student use it?The course as a learner meets it: structure, accessibility, clarity, interaction, feedback.

Institutional Learning OutcomesReports which ILOs the course already carries on a rubric criterion, and recommends where the remaining ones fit. A course contributes to the institutional record only through an attached ILO.
Universal Design for LearningTests the three CAST principles against what the course package can evidence: whether learners have more than one way to demonstrate learning (Canvas records the accepted submission modes), whether content is offered in more than one modality, and whether the course affords meaningful choice. UDL is not accessibility; conflating them is the common error.
Overview & Information  OSCQR 1-10Welcome and Getting Started content, module-level overviews, findable syllabus, printable syllabus, campus policy links, learner success resources, stated modality, technical requirements, measurable objectives instructor contact.
Technology & Tools  OSCQR 11-15Requisite tech skills stated and scaffolded, unused tools removed from the menu, privacy policy links for every third-party tool tools meet accessibility standards.
Accessibility  WCAG 1.1.1, 1.3.1, 2.4.4, 3.2.5, 4.1.2Ten structural checks, each located to the page. Alt text that exists and says nothing. Bold used where a heading belongs, which a screen reader cannot navigate. Links that open a new tab without saying so. Raw URLs as link text. Untitled frames. Tables with no header cells. Most accessibility defects are structural, and structure is what a parser sees.
Design & Layout  OSCQR 16-28Consistent navigation, white space, contrast clear instructions, spelling, heading structure, no blinking text, sans-serif at 12pt minimum table captions and header rows, slide titles.
Content & Activities  OSCQR 29-37Variety of resources, higher-order thinking, authentic activities, OER and low-cost materials, copyright and licensing stated text in accessible formats, alt text and captions understandable without color, descriptive link text.
Interaction  OSCQR 38-43 · RSIRegular and substantive interaction, stated expectations for all interaction channels, instructor presence, class community, learner-to-learner collaboration, shared resources. Mostly human judgment.
Assessment & Feedback  OSCQR 44-50Grading policies including late work, authentic assessment, rubrics on every graded item self-check opportunities, timed-response notice, working gradebook, course feedback channel.

3 · Fix, and prepare to reimportApply the style, correct the dates, and check what should not be shipping with the course.

Style guide  OSCQR 16, 18, 21, 23Makes every page look consistent and designed, instead of like a wall of text. Pick a color and a typeface once, and the whole course matches. Colors are checked for readability as you pick, so you cannot accidentally choose something a student cannot read. Works even if you have never touched HTML. Try it → These three typefaces reliably show up for students. Fancier web fonts often do not load in Canvas, and students see something else entirely.
Due dates and term calendar Campus-closed days are checked against the official campus calendar. A due date on a day the campus is closed is the most common avoidable complaint in an online course, because students reasonably assume nothing is due when the college is shut. GCC Academic Calendar ↗ Finds dates hardcoded into page text (they break on every course copy), checks that due dates fall on a consistent day and time, and flags any due date that lands on a campus holiday or break.
Student data in course FilesA Canvas export contains no enrollments, grades, or submissions by design. But it includes everything ever uploaded to course Files, and that is where a roster or a gradebook ends up. Those files are published by default, travel with every course copy, and ship with every export. This flags them. It never deletes them.
Before you upload your revised course · import it into a sandbox, never a live section

Never import a rewritten package into a live course. Canvas import is additive, so it will not erase your course, but re-importing pages that already exist produces duplicates and can overwrite content in place. On a section with students in it, that is not recoverable by undo.

  1. Preview the changes here first. Every page is shown before and after. You never have to go to Canvas to find out what this tool did.
  2. Import into a sandbox course, never a live one. Request an empty shell, or use the one you build in.
  3. Inspect the sandbox. Then, when it is right, use Course Copy to move it into the live section.
  4. Do not check “Delete existing content.” It is the one option in the Canvas import screen that is genuinely destructive.

The tool never touches your live course. It reads a file and writes a file. Canvas is not contacted at any point, and it has no ability to be.

Preview page:
Before · as it is now
After · what will be written

Step 3 · What it found, and what it fixed

of 27 machine-checkable OSCQR standards
OSCQR is one of three bodies of criteria checked here
The score covers OSCQR only, because OSCQR is the only rubric here with a fixed denominator. Alignment, workload, assessment validity, and ILO coverage are reported as findings and are not scored. The remaining 23 OSCQR standards concern instructional practice during the term or require a judgment of quality. They are listed at the foot of this page with the established method for evaluating each.

Assessment validity in the age of generative AI

Could a generative model complete this, correctly, in under sixty seconds?
Why the answer matters, and what it is not

If the answer is yes, the instrument is no longer measuring the learner. This is a matter of construct validity, not of academic integrity: an assessment that something other than the student can satisfy has ceased to measure the student.

Where this sits relative to the published rubrics

OSCQR addresses authentic assessment explicitly. Standard 31 calls for “activities that emulate real world applications of the discipline.” Standard 45 calls for “frequent, appropriate, and authentic methods to assess the learners’ mastery of content,” and advises against relying on one or two high-stakes multiple-choice examinations, directing instructors to offer learners choices in how they demonstrate mastery.

Neither OSCQR nor the Quality Matters Higher Education Rubric, Seventh Edition, contains a standard addressing generative AI. OSCQR’s academic-integrity guidance concerns proctoring and examination security.

This section therefore asks a question the published rubrics do not: not whether authentic work appears somewhere in the course, but whether the remaining assessments still measure what they claim to.

It sits alongside the standards rather than among them, because this tool does not attribute a finding to a rubric that did not make it.

A caution · authentic does not entail AI-resistant

The artifact never protects you. A model will write a portfolio case study, a project rationale, a reflection, or a personal narrative. A polished final artifact is evidence of nothing about who produced it. This is the mistake most faculty make immediately after being told to assign “authentic” work: they assign a real-world product and consider the problem solved.

What resists substitution is process evidence. A model produces a finished thing, not a trail.

  • Drafts, thumbnails, version history, working files. The trail is the deliverable.
  • Documented response to critique that a specific human gave to this specific student. A model cannot respond to feedback it never received.
  • Live, unrehearsed defense of a decision. A walkthrough, an oral defense, a critique.
  • A situated local context. A real client, a campus problem, a place the student went, a dataset they collected.
  • A physical object. Paper sketches, a photograph they took, a printed piece.
  • Presence in a community. Critique given and received, among named people.

The rule: assess the process, not only the product. The product can be fabricated. The process cannot, at least not cheaply enough to be worth it.

The replacement ladder · what to assign instead

This section never reports a broken instrument without offering a replacement at the same cognitive level. A finding that only says “this is bad” produces resentment, not revision.

Broken instrumentWhy it fails nowReplace with
Multiple-choice knowledge quizAnswered perfectly, instantly. Measures nothing.Applied identification. Do not ask what hierarchy is. Ask the learner to locate hierarchy in a classmate’s actual poster and name the element carrying it. Same concept, unsearchable, and it feeds the critique.
Definition matchingSame.Vocabulary in use. Require the terms inside a critique post. The assessment becomes whether the learner used the word correctly about real work.
Take-home essay or research paperA competent one is produced in thirty seconds.A decision memo with evidence. “Here were three options. I chose this one. Here is what changed after critique.” Requires a process to have occurred.
Reflection promptA model writes beautiful, empty reflection.Reflection anchored to artifacts. “Identify the specific change between v2 and v3 and the critique that caused it.” Requires v2 and v3 to exist.
Summative final examBroken by definition.Portfolio and live walkthrough. The learner presents their own work and defends one decision, unrehearsed.
“Describe the process”A model knows the process.Show the process. Submit the working files, the thumbnails, the version history.
Canned case studyA model has read every published case.A real, local, current problem. A campus client, this term. No published answer exists.
The other half · designing AI in, not only out

Everything above is defensive, and defense alone is a losing design. A course hardened so thoroughly against generative tools that students never touch them is a course that graduates students who cannot use the defining instrument of their field. Both things are true at once, and the tension does not resolve: we need learners to produce their own work, and we need to produce AI-literate graduates.

The resolution is not to pick a side. It is to make the use of the tool itself part of what is assessed. When AI is designed in deliberately rather than prohibited or ignored, the course gains three things it cannot get otherwise: engagement, an opening to teach digital and ethical literacy, and, counterintuitively, stronger assurance that the submitted work reflects the learner’s thinking, because the thinking is now documented rather than inferred from a finished artifact.

Four moves

  • Document the process. Require the learner to show what they did, not only what they produced. This is the same process evidence that resists substitution, so the defensive and the integrative design converge here.
  • Require transparency. Ask the learner to delineate what the model produced from what they produced, and to point to specific revisions they made to its output. Grade the delineation. This is assessable without detection software.
  • Discuss the risk openly. Ted Chiang’s analogy is the one that lands with students: using a model to write for you is like having a forklift lift the weights and then calling yourself strong. The writing is the training, and the learner is the one who does not get stronger.
  • Set explicit expectations. Per assignment, not per syllabus. A blanket course-level AI policy is too coarse to be useful, because the correct answer differs between a brainstorm and a final critique. Wake Forest’s AI syllabus decision tool is a usable starting point.

The instrument this produces is a process-disclosure component, scored alongside the artifact. It rewards a thoughtful, specific account of how the tool was used and what was revised, and it fails a submission that is silent about the process. It requires no detection software, it does not accuse anyone, and it is the only approach here that works on a student who used AI well.

Framing in this section draws on “Authentic Assessment in the Age of AI,” Steve Kaufman, M.Ed. University of Akron, presented by Quality Matters.

How this tool scores it · six signals, and one thing it refuses to do

Each graded item is tested against six signals. Three or more and the assessment is probably doing its job. Zero or one and it is decorative.

  • Does it require a personal or local context a model could not know?
  • Does it require process evidence, not only a final artifact?
  • Does it require responding to feedback a specific human gave this student?
  • Does it require a live, unrehearsed defense of a decision?
  • Does it produce something with an audience beyond the instructor?
  • Would the learner be able to use it after the course ends?

The one thing this section refuses to do is flag a low-stakes formative quiz as a defect. A retakeable vocabulary check worth four percent of the grade is not an assessment of mastery; it is a study aid, and it is supposed to be easy. The check is therefore stakes-aware: it asks both whether a model could complete the item and whether the item carries enough weight for that to matter. A tool that scolds you about a practice quiz is a tool you will stop opening.

And the deeper judgment is not automated at all. Whether a task genuinely replicates the contextual demands the discipline places on a practitioner (Wiggins’s criterion) is a professional judgment. This section surfaces it for review. It does not score it.

Institutional Learning Outcomes

Your college tracks whether every student can do these before they graduate. A course contributes to that record only when an ILO is attached to a rubric criterion.

Attach three to six. Two minutes each. The steps are below.

Why they are worth doing

An ILO is how a college answers the question a degree implies: can our students communicate visually, think critically, use information responsibly? Attainment is tracked across a student’s whole time in college, first semester to last.

An outcome assessed only within the major yields data describing majors alone. Students who take a single course in the discipline generate no record, so introductory courses carry disproportionate weight in the institutional dataset.

ILO scores are not grades. They do not affect a student’s grade and are not used to evaluate a student or a faculty member.

The scoring standard

Every ILO begins “by the end of their college experience…”, so you score against a graduating student, not the one in front of you.

Work that is proficient for your course may be emerging for the ILO. Emerging is the expected rating at the introductory level. Uniform Proficient ratings in introductory courses render the longitudinal data uninformative.

How to attach one · about two minutes

You do not add ILOs to the Outcomes list. They are already in Canvas at the institution level. You attach them to a rubric criterion, so the rubric has to exist first.

  1. Open a graded assignment → its Rubric → the pencil to edit.
  2. Click Find Outcome.
  3. Open the domain folder in the left column, Communication Visual, Creative Thinking, Career Goals, Information Literacy, and so on.
  4. Choose the outcome, read its description to be sure, and click Import.
  5. It becomes a criterion on your rubric, scored 0 to 4. Save.

One ILO per criterion, three to six per course. Canvas calculates mastery as Highest Score, so a single attachment is sufficient. Attach late in the course.

Where to ask: your institution’s assessment office · see Importing ILO rubric items into Canvas.

You can do this after you import the restyled course

Unlike course competencies, ILOs are not a before-export step. They live on rubrics, and the rubrics are in your course either way. Export, restyle, import, then attach them. Nothing is lost.

What a machine cannot check, and what to do instead

The standards no parser can verify · and the established method for evaluating each

Roughly half of OSCQR concerns what exists in the course file, which a parser can read. The remainder concerns instructor conduct during the term, or requires a judgment of quality. Neither is visible to any tool. The standards below are listed with the established method for evaluating each.

So instead of a fake score, here are the lessons. Each row below is a standard a machine cannot verify, why not, and the specific thing to do this week to satisfy it. Do them in order of how much they will change your course, not how easy they are.

Why a parser cannot verify it. RSI is instructor behavior across a term, not an artifact. A course package contains no record of it.

Established method for evaluating it

Audit against the federal definition. 34 CFR 600.2 requires interaction that is instructor-initiated, regular, and substantive, the last defined as at least two of five named activities (direct instruction, assessment feedback, providing information, facilitating discussion, other approved). Evidence each one from Canvas Course Statistics and Gradebook History. WCET’s RSI guidance is the reference practitioners use. A claim you cannot evidence is a claim an auditor will not accept.

Why a parser cannot verify it. A parser detects missing steps. It cannot detect that a novice is unable to execute them.

Established method for evaluating it

Run a think-aloud usability test. This is a formal protocol, not an opinion poll. Recruit participants representative of the novice learner, give them the task, and instruct them to verbalize their reasoning continuously while attempting it. Do not assist, and do not answer questions; every hesitation is a defect in the instructions, not in the participant. Nielsen’s well-replicated finding is that five participants surface roughly 85 percent of usability problems, which is why this is tractable rather than academic. The underlying method is the think-aloud protocol (Ericsson & Simon, 1984). For a lighter-weight pass, the cognitive walkthrough (Wharton et al. 1994) asks, at each step, whether the learner will know what to do and will recognize that they have done it.

Why a parser cannot verify it. Counting media types is not a measure of engagement, and “engaging” is not a property a file can carry.

Established method for evaluating it

Evaluate against the multimedia learning principles. Mayer’s cognitive theory of multimedia learning supplies testable criteria: the coherence principle (extraneous material depresses learning), redundancy (identical on-screen text alongside narration depresses learning), signaling, and segmenting. Audit each resource against them. For the engagement construct itself, use Hidi and Renninger’s four-phase model of interest development, which distinguishes situational interest, which novelty can trigger, from individual interest, which only sustained value-relevance develops. A course that only triggers is a course that does not retain.

Why a parser cannot verify it. The package stores video references, not video files. There is no media in it to inspect.

Established method for evaluating it

Verify against the applicable standard, then remediate by ownership. The governing requirement is WCAG 2.1 Success Criterion 1.2.2, which mandates captions, not transcripts, for prerecorded video. Quality is specified by the DCMP Captioning Key, the field’s reference standard for accuracy, placement, and speaker identification. Auto-generated captions do not satisfy either; they average 70 to 90 percent accuracy and degrade most severely on discipline-specific terminology, which is precisely the content being assessed. Remediation depends on ownership, and the two cases share no steps. See the full procedure in the Content & Activities finding above.

Why a parser cannot verify it. Instructor presence is a learner perception. The existence of a biography page is not evidence that it was formed.

Established method for evaluating it

Measure it with a validated instrument. The construct is teaching presence within the Community of Inquiry framework (Garrison, Anderson & Archer, 2000), which models the online learning experience as the intersection of teaching, social, and cognitive presence. It is measured by the CoI Survey (Arbaugh et al. 2008), a 34-item instrument that has been validated across multiple institutions and is freely available. Administer it at midterm. This produces a defensible measurement of presence rather than an impression of it.

Why a parser cannot verify it. Whether an instrument measures the outcome it claims to measure is a question of construct validity, and it is a judgment.

Established method for evaluating it

Apply constructive alignment, then test for validity. Biggs’s constructive alignment requires that outcome, activity, and assessment cohere; the tool’s alignment matrix reports the structural half of this, and the interpretive half is yours. Wiggins’s criteria for authentic assessment supply the test: does the task replicate the contextual demands the discipline places on a practitioner? For scoring, the AAC&U VALUE rubrics are validated, published, and free.

Why a parser cannot verify it. A parser detects a survey. It cannot detect whether the survey changed anything.

Established method for evaluating it

Use Small Group Instructional Diagnosis (SGID). A facilitated midterm protocol (Clark & Bekey, 1979) in which a colleague, not the instructor, convenes the class, elicits consensus on what supports learning and what impedes it, and reports the themes back. It substantially outperforms an anonymous form because it produces consensus rather than outliers. The mechanism that makes it work, and the step most instructors omit: report back to the class what you changed. Feedback that visibly changes nothing trains students not to give it.

Why a parser cannot verify it. Gradebook configuration lives in course settings, not in the content package.

Established method for evaluating it

First, reconcile the gradebook against the syllabus. The syllabus states a weighting to the student, for example Projects 30%, Quizzes 20%, Critique 25%. The gradebook enforces the weighting in assignment group settings. When the two disagree, the syllabus is the promise and the gradebook is the reality, and the student discovers the gap at the worst possible moment. This tool reports the discrepancy automatically, because both the stated weights and the enforced weights ship in the course package. Confirm that the enforced weights sum to 100 and match the published figures exactly.

Then inspect it as a learner. Canvas provides Student View and a Test Student for this purpose. Confirm that a mid-term total is interpretable rather than misleading, and that no ungraded item is silently depressing the displayed grade.

Why a parser cannot verify it. Navigation configuration is not fully represented in the export.

Established method for evaluating it

Inspect the course in Student View and hide every menu item the course does not use. Each dead link is an unnecessary decision imposed on the learner, and it degrades trust in the navigation that does work.

Why a parser cannot verify it. These typically reside in Syllabus+ or a linked document rather than in the course package.

Established method for evaluating it

Verify against the institutional requirements directly. Confirm the presence of campus policies, late-work consequences, accommodation procedures, the approved seat-hours statement, and, for online sections, the approved RSI statement. This is a verbatim-language check rather than a judgment: the approved statements must appear word for word, because a paraphrase of mandated language is not the mandated language.

Do not perform this check by hand. It is fully automatable and it is the one place where a human reader reliably fails, since the eye accepts a paraphrase that the requirement does not. Run the Syllabus Checker → It verifies the district required elements, the approved seat-hours statement, and the approved RSI language against the current source document.

The rule this tool runs on: it will never report a pass on a standard it did not verify. A check that cannot be computed is listed as unverified, with the method for checking it by hand.

OSCQR the SUNY Online Course Quality Review Rubric, is made available under a Creative Commons Attribution 4.0 International License by the State University of New York through SUNY Online. oscqr.suny.edu. This tool applies the rubric; it is not affiliated with or endorsed by SUNY. Use of OSCQR does not guarantee course quality, accessibility, or compliance with any accrediting standard.

A prototype, in testing. Applies the SUNY OSCQR rubric, openly licensed under CC BY 4.0, alongside constructive-alignment and workload criteria of its own. Not affiliated with or endorsed by SUNY.

Nothing is uploaded anywhere. Your course package is opened, read, and rewritten on your own machine. There is no server, no account, and no vendor in the middle. A Canvas export contains no enrollments, grades, or student submissions by design, so there is no student data for this tool to receive.

Every estimate is a draft for you to confirm, and nothing is written into your course without your approval.