Build an Assessment Blueprint Before Writing Questions: A Worked ML Example

A learner can recognize a sound experiment plan without being able to implement it. Another can execute a supplied workflow without explaining why its methodological choices are appropriate. Before writing assessment questions, decide which of those performances your course actually requires.

This is the starting point for how to build an assessment blueprint before writing questions: specify the learning claim, choose evidence that could support it, and only then commission questions or tasks. Carnegie Mellon’s Eberly Center recommends aligning assessment tasks with learning objectives and instruction, rather than assessing factual recall when the intended outcome is analytical performance.

1. Start with the decision the assessment will support

Write a purpose statement before listing topics. For this worked example: At the end of a beginner machine-learning unit, use separate outcome scores to identify whether learners need further practice with methodological knowledge, experiment decisions, planning, or implementation.

This is an editorially constructed course example, not a qualification assessment or a reproduction of a research instrument. Learners have practiced classification using a supplied teaching dataset, a starter notebook, and explicit instructions. The assessment does not establish readiness for unrestricted professional ML work.

Record what resources learners may use, whether they work individually, and what assistance is allowed. If a starter notebook supplies the data-loading code, do not claim that successful completion demonstrates independent data-loading ability. Keep the intended inference as narrow as the evidence.

2. Separate knowing, choosing, explaining, and doing

Replace “understand ML experiments” with observable performances. The University of Michigan’s guidance explains that the verbs in learning objectives help identify suitable assessments. For this example, use four planning categories:

• Knowing: identify the roles of training, validation, and test data.
• Choosing: select an appropriate methodological action in a specified situation.
• Explaining: justify the choice and describe the consequence of an alternative.
• Doing: implement the required workflow and produce inspectable outputs.

These categories are an editorial organizing device, not a validated taxonomy. Choosing and explaining can share one task, but their scoring criteria should remain distinguishable.

A research illustration, not proof of this blueprint

A November 2026 preprint on support for beginner ML experiments reports two within-participant studies with 18 predominantly technically trained participants. After using structured guidance, participants produced better high-level workflow designs, while scores on a quiz about specific methodological choices did not improve significantly.

The useful editorial inference is limited: performance on one kind of assessment should not automatically stand in for another. The result does not establish that quizzes are ineffective, demonstrate no learning occurred, or validate the worksheet below. Its small, predominantly technical participant group also warrants caution when applying the finding to other learners.

3. Match the outcome to evidence before choosing item counts

Use a quiz when recognition or selection is the performance you intend to observe. A scenario-based quiz can also ask learners to choose between methodological alternatives. However, if you need their reasoning, request a short explanation; if you need implementation, request inspectable practical work. The Eberly Center’s examples of assessment formats for different learning objectives include objective items for recall and labs or performances for execution.

For the ML example, use this technical constraint: split the data before fitting learned preprocessing, and fit those transformations on training data rather than test data. The official scikit-learn documentation explains how training-only fitting and pipelines help avoid preprocessing leakage. Restrict the exercise to this taught case rather than claiming to assess every possible leakage mechanism.

4. Completed assessment blueprint worksheet

Editorial planning aid—not a validated measurement instrument. The following linked tables form one worksheet. Outcome IDs connect evidence decisions to allocation, scoring, and ownership without creating an excessively wide table.

Part A: Outcome-to-evidence map

Part B: Allocation, criteria, and review

The 100-point allocation is illustrative. It gives greater emphasis to decisions and practical performance because those are priorities in this fictional unit. It is not a recommended universal weighting, and five knowledge questions should not be treated as comprehensive evidence of methodological knowledge.

For O3, check five features: a stated prediction target, data separation, training-only preprocessing, model selection using validation evidence, and a reserved final evaluation. Do not award implementation credit for drawing those steps. For O4, inspect both the notebook and its outputs; do not score only the final model metric.

Part C: Evidence-mismatch check

Ask for every row: Can this assessment format actually demonstrate its assigned outcome? Then ask what it leaves unobserved.

• O1: selected responses reveal recognition, not an independently produced explanation.
• O2: the explanation reveals stated reasoning, not whether the learner can execute the choice.
• O3: the diagram reveals a plan, not a working experiment.
• O4: inspectable work reveals performance under the stated support conditions, not necessarily unaided execution.

If a row fails this check, revise the evidence format or narrow the outcome before drafting anything.

5. Blank reusable blueprint worksheet

Editorial planning aid—not a validated measurement instrument. Copy the outcome section for each intended performance. Replace the prompts with your own decisions.

6. Turn approved rows into writing briefs

A blueprint row becomes a specification, not simply a request for “ten ML questions.” For the completed example:

1. O1 question brief: draft five questions covering training, validation, test data, preprocessing order, and recognition of a supplied leakage example. Provide one defensible key per question and a rationale for each distractor.
2. O2 scenario brief: describe a learner fitting a scaler before splitting the dataset. Ask for the next action, its justification, and the consequence of retaining the original procedure. Specify the expected reasoning before writing the prompt.
3. O3 task brief: request an annotated workflow using the five planning checks. Accept different diagram layouts when the methodological sequence and relationships are clear.
4. O4 practical brief: supply the practiced dataset and starter notebook. Require separated data, training-only preprocessing, evaluation outputs, and a brief explanation of an inspected execution step. Publish the allowed support conditions.

Draft performance descriptors alongside the tasks. Carnegie Mellon describes rubrics as explicit criteria, descriptors, and performance levels. Replace vague judgments such as “good explanation” with observable features, such as identifying which information entered preprocessing and why that matters for evaluation.

7. Pilot coverage before release

Treat this as a small editorial pilot, not statistical validation. Ask a colleague and representative learners to work through the instructions, then review:

• Ambiguity: are multiple answers defensible under the stated assumptions?
• Scoring consistency: can two reviewers apply the descriptors to the same sample responses?
• Accessibility: do diagrams, interfaces, and submission requirements introduce barriers unrelated to the outcome?
• Coverage: does each required outcome still receive direct evidence after revisions?
• Inference limits: could a combined score hide weak implementation behind strong quiz performance?

Report outcome-level results where useful. Do not treat item counts alone as establishing reliability, or this worksheet as establishing assessment validity.

Once the blueprint is approved, explore SubSchool’s AI homework generator for editable exercise drafts based on saved lesson context. Use the approved specifications to guide your review. SubSchool can support repetitive drafting work; teachers retain authorship, approve the assessment, and make the final educational decision.