Standardised rubrics: Yay or nay?
Before we talk about AI generated rubrics, we need to talk about standardised rubrics. As an Educational Designer, a question I frequently hear is “Do we have a bank of rubrics I can access?”. While this would be easy to collate and disseminate, I have always resisted. The reason? Constructive alignment and course/unit/program contextualisation. Rubrics are complex and the makeup of a rubric: the criteria, the descriptors and the weightings can vary based on many different considerations:
- The year of study, the AQF level, and the prior experience level of students.
- The fundamental skills demonstrated in the task.
- The unit of study: core, elective, project-based, honours, placement.
- The task weighting relative to other tasks.
- The structure of the task and whether it diverges from norms.
- Most importantly: How the task aligns with the learning outcomes.
There are also practical matters to account for
- How many markers will use the rubric and what is the level of their expertise?
- What inter-rater reliability and moderation practices are in place?
- Most critical: How much time can be devoted to rubric creation?
I get it. It’s hard to think up criteria and descriptors for every assessment task. It takes hours, loads of feedback and many iterations to get the rubric to a point where it aligns with the learning outcomes, evaluates students in a valid way and decreases bias and subjectivity. I describe ways of dealing with this in my blog post: Two common problems working with rubrics. I can appreciate the need for efficiency where possible. So, while we still need to create bespoke rubrics for each assessment task, there are ways in which we can improve efficiency.
Step one: Ensure your assessment task is clear and aligned with your learning outcomes
If the alignment is off within the task, the alignment will be off in the rubrics. Focus on the learning outcomes, and ensure the assessment task directly and explicitly achieves each learning outcome it is aligned to. This may mean fleshing out the task instructions to ensure the students achieve each learning outcome.
Learning outcomes:
- Analyse the needs of priority populations and select strategies for effective support interventions.
- Communicate effectively in oral, written and digital formats.
In weeks 3, 5, 7 and 9 you will be provided with a case scenario on a priority population.
Task one: Analyse the case and identify the needs of the priority population.
Task two: Select two support strategies that align with the needs of the priority population. Use evidence from the literature to support your rationale behind these interventions.
Step two: AI generated rubrics: prompt AI to create a base rubric
Once you have our clear overarching criteria, aligned with the learning outcomes and the task, take this information to your local friendly AI, and get it to write a rubric. You will get a basic and highly subjective rubric, but it’s better than a blank page.
Prompt: Create a rubric for a case study assessment task. The criteria are: 1. Issue identification and analysis, 2. Intervention choice and rationale, 3. Evidence selection and integration, and 4. Academic writing. Create a three-scale rubric with the grades: Not yet achieved (Less than 50%), Competent (51-74%), and Exceeds expectations (75%-100%).
Rubric: Generated by Copilot on 1/8/2024
| Criteria | Not Yet Achieved (<50%) | Competent (51-74%) | Exceeds Expectations (75-100%) |
| Issue Identification and Analysis | Incomplete or inaccurate identification of key issues. Limited or no analysis of issues. | Identifies most key issues accurately. Provides a basic analysis of issues. | Accurately identifies all key issues. Provides a thorough and insightful analysis of issues. |
| Intervention Choice and Rationale | Inappropriate or unclear intervention choice. Weak or no rationale provided. | Chooses a generally appropriate intervention. Provides a basic rationale. | Chooses the most appropriate intervention. Provides a strong, well-justified rationale. |
| Evidence Selection and Integration | Little or no relevant evidence selected. Poor integration of evidence into analysis. | Selects some relevant evidence. Integrates evidence into analysis with some effectiveness. | Selects highly relevant and credible evidence. Integrates evidence seamlessly into analysis. |
| Academic Writing | Numerous errors in grammar, spelling, and punctuation. Poor organization and clarity. | Some errors in grammar, spelling, and punctuation. Generally clear and organized. | Virtually no errors in grammar, spelling, and punctuation. Clear, concise, and well-organized writing. |
If this rubric looks great to you, then it’s worth reading my previous blog post, Two common problems working with rubrics, to help you identify a few of the many issues with this generated rubric.
Step three: Evaluate the AI generated rubric
Evaluate the rubric language, paying particular attention to the language used to differentiate between grades. Use these questions to help you:
- Do the grade descriptors clearly outline what needs to be demonstrated in unbiased, easily understandable language?
- Do the grade descriptors differentiate clearly between levels of achievement?
Here are some key issues identified in the generated rubric:
Little definition between the grades
The grade scale descriptors above vary by one or two words, so the difference between these grades relies entirely on the marker’s definition of those words. This is an easier issue to resolve in a three-scale descriptor as there are fewer incremental levels of language needed to define between the scales.
Lacks detail of performance
Consider the difference between ‘some relevant’ or ‘highly relevant’ evidence. What does that look like in practice? Is the evidence aligned with the needs of the priority population? Is there direct evidence of previous implementation that has been successful? What does ‘seamless integration’ look like? Is the evidence located close to the idea and building on the argument, or is it plonked on the end as a tick box? What does that all look like in practice?
What is ‘academic writing’?
I used the criterion ‘Academic writing’ because I see it frequently and it never really encapsulates what is needed. Is it about coherence and cohesion? Spelling and grammar? Writing for an intended audience? Using discipline-specific terminology accurately? Is it about the development of the argument throughout the writing? There are many facets of academic writing, but usually, this criterion only assesses precisely what Copilot has produced. Is this what you’re looking for in your students? Is this how they are showing themselves to be capable in their future careers? In alignment with the learning outcome, what does effective communication look like in an academic writing context?
Step four: Rewrite the rubric to add specificity and reduce subjectivity
| Issue Identification and Analysis | The issues are incorrectly identified, baseless, or unnecessarily profiling. The analysis is overly broad or lacks fundamental knowledge of the priority population and their needs. | The key issues are identified, though some subtle or nuanced issues may be lacking. The analysis indicates an understanding of the reasons behind the issues and the needs of the priority population. | The issues are accurately and comprehensively identified. The analysis is sophisticated, showing a high degree of insight into the needs of the priority population. |
While not perfect, the criterion rewritten above has a clearer indication of what performance looks like at all three levels.
AI tools can offer support for educators seeking to streamline rubric creation, providing efficient starting points for assessment design. However, the true value lies in the thoughtful human refinement of AI generated rubrics. Educators ensure fairness and pedagogical integrity by critically evaluating rubrics for clarity, specificity, and alignment with learning outcomes. The four-step process: defining tasks, generating with AI, critiquing outputs, and rewriting for precision, empowers teachers to create high-quality, context-specific rubrics that enhance both grading consistency and student understanding. When used responsibly, AI becomes not a replacement, but a valuable partner in effective assessment design.