Key takeaways
- Reopen review after model, behavior, configuration, or version-information changes.
- Approve affected teaching workflows separately, with explicit interim controls.
- Compare fixed teacher-reviewed cases; disclose missing baselines and changed conditions.
- Evaluate reasoning, misconception diagnosis, feedback, and inappropriate responses—not just answer correctness.
- Document a continue, restrict, or pause decision with an owner, fallback, and follow-up.
- Keep simulated evaluation findings separate from real-student learning evidence.
A major model update should trigger a review of affected teaching workflows before you extend their use or rely on them for consequential decisions. That does not mean every update requires a school-wide shutdown. The practical question is narrower: does the changed tool still meet your requirements for the work you approved?
This guide proposes a school-defined reapproval process for tutors, school owners, and learning and development teams. It is an editorial protocol, not a validated instrument, certification, or new regulatory requirement. Teachers retain authorship and the final educational decision.
Why the evaluation method matters
The October 2026 MisconceptDialogue preprint examines math tutoring through dialogues with LLM-simulated students. Its authors explicitly state that this simulated protocol does not establish real-student learning.
On identical dialogues, the paper reports a measured remediation success rate of 42.1% under benchmark-style exact-match scoring and 81.3% under a human-calibrated LLM judge. These are different evaluation lenses applied to the same interactions, not evidence that a model update improved or worsened learning.
The practical inference is to define what acceptable teaching support means before comparing outputs. A higher score under one evaluation method cannot, by itself, answer your local approval question. NIST’s Generative Artificial Intelligence Profile provides broader, voluntary risk-management context; the education-specific checklist below is our proposed implementation.
1. Decide when to reopen approval
When deciding when to review an AI education tool after a major model update, use these recommended triggers:
- Announced model change: the provider identifies a different underlying model or substantial revision.
- Observed behavior change: teachers notice different explanations, feedback, refusals, or assessment suggestions.
- Configuration change: prompts, settings, reference materials, integrations, or permissions change.
- Version uncertainty: you cannot establish which version produced an output or whether the approved configuration remains in use.
Record what is confirmed, what is merely observed, and what remains unknown. Do not label a difference a model regression unless you have evidence supporting that attribution. If version information is unavailable, review the observable workflow and document the uncertainty.
2. Triage the affected teaching workflows
Approve workflows separately rather than giving the entire product one pass or fail. Use this suggested impact matrix to prioritize review:
| Workflow | Review focus | Proposed interim control |
|---|---|---|
| Lesson drafting | Accuracy, sequence, audience fit | Teacher edits before release |
| Tutoring dialogue | Diagnosis, hints, follow-up | Teacher-mediated use |
| Feedback | Evidence, specificity, tone | Teacher approves feedback |
| Consequential assessment | Criterion alignment, unsupported judgments | No autonomous final decisions |
For each workflow, name the affected learners, output recipient, reviewer, and fallback. A drafting workflow may remain usable while direct tutoring is restricted. A restriction should specify what staff may do, not simply say “use cautiously.”
3. Establish a fixed, teacher-reviewed baseline
Build a test register from synthetic examples first. Use de-identified work only under your institution’s approved data-handling process. Include straightforward cases, common misconceptions, ambiguous answers, incomplete submissions, and examples requiring clarification rather than confident inference.
For every case, save the exact input, relevant conversation history, reference material, configuration, and expected response characteristics. Ask a teacher to approve those characteristics before testing. They should describe acceptable reasoning and teaching behavior, not demand one exact wording.
Preserve old outputs when available. If the previous version cannot be rerun, compare archived outputs with the new tool under the closest documented conditions. If no baseline exists, report a current suitability review—not a before-and-after improvement claim. Historical approval alone is not an output baseline.
4. Rerun the same cases and inspect the evidence
Keep inputs and evaluation criteria fixed. Where the tool permits, keep settings, reference documents, and conversation context unchanged. Record unavoidable differences so reviewers know what the comparison can support.
Use these proposed review dimensions:
- Accuracy: are the answer and reasoning correct?
- Misconception diagnosis: does the response address the learner’s actual error, or invent an explanation?
- Feedback usefulness: is there a specific, appropriate next step?
- Inappropriate responses: does the output shame the learner, make unsupported personal judgments, disclose inappropriate information, or disregard your boundaries?
Retain full outputs and reviewer explanations. If resources allow, make repeated runs and report inconsistent responses rather than selecting the best example. Have a second educator review disputed or consequential cases. These are recommended controls, not statistically validated sampling rules.
Worked example: correct answer, weaker tutoring
Synthetic example: a learner writes, “I think 1/4 + 1/4 = 2/8 because I add the top numbers and the bottom numbers.” The teacher-defined expectation is to explain that denominators name the size of the parts, combine two quarters, and invite a follow-up check.
| Dimension | Illustrative old output | Illustrative new output |
|---|---|---|
| Answer | Two quarters equal one half | Two quarters equal one half |
| Explanation | Parts remain quarters; combine two | Use a common denominator |
| Follow-up | Ask the learner to explain three quarters | Provide the answer and stop |
| Teacher judgment | Meets local tutoring expectation | Correct, but misses the stated misconception |
Both outputs are invented for this worksheet; neither comes from SubSchool or the study. Under a school-defined criterion requiring misconception-focused feedback, the new response needs revision despite its correct answer. That supports a restriction on this illustrative workflow—not a conclusion about a real model or student learning.
5. Record continue, restrict, or pause
Set acceptance criteria before reviewing results. For example, a school might require no unresolved critical factual errors, teacher approval of learner-facing feedback, and successful handling of designated misconception cases. All such acceptance criteria are school-defined recommendations, not validated universal thresholds.
- Continue: reviewed workflows meet local criteria within the tested scope.
- Restrict: use continues only with specified limits, such as teacher approval before feedback reaches learners.
- Pause: unresolved failures or uncertainties prevent approval of the affected workflow; staff use the named fallback.
Do not let an average score cancel out an unresolved critical failure. Record the decision owner, evidence, permitted uses, restrictions, and next review date. “Continue” means permission for a defined use—not proof of general safety or learning effectiveness.
AI-tool reapproval worksheet: copy and save
Copy the following six sections into a document or spreadsheet and save a downloadable copy, such as a PDF. Keep the editable original for subsequent reviews. This is a reusable worksheet, not an automated audit.
A. Change log
Complete: tool; provider; change detected on; previous and current version identifiers, if known; change evidence; affected settings or materials; unknowns; review owner.
B. Workflow-impact matrix
One row per workflow: workflow name; learner group; output recipient; possible educational consequence; interim control; responsible teacher; fallback teaching process.
C. Fixed test-case register
One row per case: case ID; synthetic or approved de-identified origin; exact input and context; reference material; teacher-reviewed expected characteristics; unacceptable behaviors; local acceptance criterion.
D. Before-and-after comparison
| Case ID | Before evidence | After evidence | Teacher finding |
|---|---|---|---|
| Enter ID | Output record or unavailable | Output record and run date | Meets, fails, or uncertain; reason |
E. Teacher sign-off
Complete: reviewer name and role; reviewed cases; unresolved disagreements; approved scope; required edits or controls; sign-off date; second reviewer where used.
F. Decision record
Complete: continue, restrict, or pause; rationale; permitted workflows; prohibited uses; decision owner; fallback; staff notification; follow-up date; triggers for reopening review.
6. Recheck data handling and monitor after approval
Before reapproval, ask the responsible person to verify current provider documentation on data use, retention, access, and relevant configuration. Check whether your existing learner notices still describe the proposed workflow accurately. This checklist is not a legal determination; unresolved privacy or learner-protection questions need the appropriate institutional reviewer.
After approval, schedule a locally chosen follow-up and sample outputs from the approved workflows. Log teacher corrections, learner complaints, and unexpected behavior. Learner understanding requires separate evidence—for example, teacher-reviewed independent work—not just favorable ratings of AI dialogue.
SubSchool’s business workflows describe editable course creation and structured first-pass assessment with accountable human review. If your team is planning a pilot, discuss which repetitive tasks to automate and which decisions remain with educators. Bring this worksheet to define the scope, evidence, and approval owner; do not assume reapproval testing is a built-in product capability.
Sources and methodology
Prepared using the supplied brief and web searches of the referenced arXiv paper, official NIST guidance, and SubSchool’s business page. Direct page retrieval returned no readable content for several URLs, so research claims were bounded to accessible indexed primary-source excerpts. The brief assigns October 26, 2026 to the preprint, while the indexed arXiv abstract lists submission on October 1, 2026; the article therefore uses October 2026 and leaves the publication date unresolved. The workflow protocol, acceptance examples, worksheet, and tutoring comparison are original editorial recommendations, not findings from the preprint. No specific model release, new legal requirement, real-student learning gain, or built-in SubSchool reapproval feature is asserted.
- MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math TutoringarXiv
- MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math TutoringarXiv
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence ProfileNational Institute of Standards and Technology
- AI Training and Assessment Platform for Teams | SubSchoolSubSchool
