How to Use AI to Improve Your Grading and Assessment

Welcome to AutomatED: the newsletter on how to teach better with tech.

Each week, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.

Let’s discuss how to use AI tools to accelerate and improve your grading and assessment.

As I noted in my very first piece, no professor spends the majority of their time in the classroom. Rather, in addition to working on our research, we are working on other tasks, such as:

• preparing to teach,

• meeting with students,

• grading, and

• doing administrative work.

My focus in the first piece was on AI-enhanced voice recording, transcription, and analysis tools, like Fireflies.ai. These tools can level up your student meetings by helping you provide more personalized instruction and feedback to your students, as well as by helping you keep track of student needs after the meetings have ended.

Today, my focus is on how you can use AI tools to assess, constructively critique, and grade student submissions. There are few assignments where AI tools can or should completely replace you, the professor, but this does not mean that you should not be looking for ways they can save you time and improve your pedagogy.

I know I do not need to convince this readership of the value of reducing time on assessment while improving its impact…

💡 The Big Idea: Simulation

What if you could know what your students will think upon receiving your feedback on their work? What if you could see your assessment from their perspective in advance, thereby unlocking insight into how you could improve it to increase their learning?

What if you had expert teaching colleagues who could mentor you as they review your practices of assessment? What if you could have another set of eyes on your assignment design, thereby unlocking insights into what is working, what is not working, and how to improve?

What if you had a grader — whether a graduate student or another professor — who could help you get through that pile of submissions faster?

When we think about what would make us more efficient and effective assessors of student work, we often imagine aspects of our assessment from other perspectives. We imagine who it would be nice to have around or whose mind we wish we could better access or understand.

Enter AI. With the right prompting, generative AI tools, like ChatGPT, can effectively simulate people in countless roles, including students, mentors, and graders.

🎒 AI Role-Playing: Student

When prompting AI tools to simulate being your students, it is crucial to be as specific as possible. You already know what, in general, your students might think of your feedback. What you might not know is how a student will respond to a specific bit of feedback or a specific grade on a specific assignment, or how to make a bigger pedagogical impact on each of them.

There are many ways you would benefit from getting student’s perspectives on your assessments — and your courses more generally. Below, I provide a prompt formula you can use with ChatGPT to get insight into your students’ perspectives on feedback. You can manipulate this prompt formula for parallel use cases involving different aspects of your assignments.

Structure of the Initial Prompt to Help Improve your Feedback to Students:

Explain that you are a professor who just finished grading student submissions for an assignment, along with details on the context of the assignment. Then, after signposting that you will share the instructions/prompt for the assignment, copy and paste it into ChatGPT’s prompt window. Tell ChatGPT that you want it to pretend to be the student receiving feedback on this assignment, if it is willing.

Structure of a Follow-Up Prompt: 

Tell ChatGPT that you want it to explain what would improve this specific bit of feedback, if it were to have a bigger impact on the student’s learning. Tell ChatGPT any specific aspects of the feedback that you are uncertain about. Then, after removing any identifying information and signposting that you will share the feedback, copy and paste your feedback to the student on this assignment into the prompt window.

(Throughout, be sure to remember the general advice for prompt engineering Caleb discussed last week in his post on how to teach with ChatGPT.)

Basic Example of an Initial Prompt (for my Philosophy 101 course):

I am a professor of philosophy, I just finished grading my student's papers, and I was hoping that you could help me understand how students will perceive my feedback on their papers using your vast knowledge of pedagogy and human psychology. The course is for beginners who do not specialize in philosophy, but the paper is the second paper of the semester, so they have some practice writing in this format. Here is the prompt I gave students for the papers: "«here I pasted the entire prompt»." If I give you the feedback I gave a student, would you be more than happy to tell me how they will perceive it and how I could improve it?

Basic Example of a Follow-Up Prompt (submitted after ChatGPT agrees):

If you put yourself in the student's shoes, are there ways that I could improve my feedback to this student so that they learn more from my feedback? And if you were the student, what would you ask me to clarify or elaborate on about my feedback? I want you to be as specific as you can. Here is the feedback: "«here I pasted the feedback, anonymized»."

Actual Response to these Example Prompts:

For several reasons, this was helpful in this specific case, including:

1. Some of my feedback was too general and would have benefited from illustrative examples;

2. In trying to cover every shortcoming in my student’s paper, I wrote too much, and there was a real risk they would not have been able to understand the relative significance of different areas in which they had room to grow; and

3. I had forgotten to paste in my stock blurb about coming to student/office hours for further dialogue.

You can always start over with different prompts, or you can follow up with clarificatory prompts to take the simulation deeper.

💁🏿‍♀️ AI Role-Playing: Mentor

When prompting AI tools to simulate being your teaching mentor, it is crucial to be honest with yourself. It is easy to find assessment strategies that work well enough and then rest on your laurels, especially given how busy you are. Furthermore, it is easy to not want to expose yourself to pedagogical criticism, as it calls your expertise and authority as a professor into question. But what would you want to know about the quality of your pedagogy, if you were to expose yourself to a non-judgmental and constructive mentor of the highest caliber?

There are many ways you would benefit from getting a mentor’s perspectives on your assessments — and your courses more generally. Below, I provide a prompt formula you can use with ChatGPT to get insight into your mentor’s perspectives on your assignment design. You can manipulate this prompt formula for parallel use cases involving different aspects of your assignments.

Structure of the Initial Prompt to Help Improve your Assignment Design:

Explain that you are a professor who is designing an assignment, along with details on the context of the assignment. Explain the nature of the course in which it is found. Explain the course objectives that the assignment is designed to help fulfill. Explain the background and skills of the students who you expect to take the class. Tell ChatGPT that you want it to pretend to be a mentor reviewing and evaluating the assignment. Tell ChatGPT that you want it to draw from its vast pedagogical expertise. Tell ChatGPT that you want it to pretend to be a professor who has superlative features x, y, and z. Then, after signposting that you will share the instructions and/or rubric for the assignment, copy and paste it into ChatGPT’s prompt window.

(As above, you may need to break your prompt into multiple parts. It is often useful to provide background on what you are looking for, ask ChatGPT if it is willing to provide it, and then provide the rest of the prompt.)

Basic Example of an Initial Prompt (for my Philosophy 101 course):

I am a professor of philosophy, I am designing an essay assignment for my novice students, and I was hoping that you could give me some insight into ways to improve its prompt. I want you to pretend to be my mentor, namely a professor who has taught university courses for decades, who has vast pedagogical experience, and who has learned from the literature on how people learn. Here is the essay prompt: "«here I pasted the entire prompt»."

Actual Response to this Example Prompt:

This is good advice. My assignment did not include these elements. And we know that students benefit from having access to model assignment submissions, from metacognitive reflection, from assignments that relate to their experiences and values, and from iterative and personalized feedback. Thanks, Dr. ChatGPT!

🔢 AI Role-Playing: Grader

When prompting AI tools to simulate being your grader, it is important to provide them with a grading framework and rubric that minimizes ambiguity and opportunities for misjudgment. The situation parallels that with human graders, as anyone can attest who has seen a professor try to wrangle and standardize the grading practices of a flock of teaching assistants. It is your goal to make the grading process transparent, objective, and consistent.

Below, I provide a prompt formula you can use with ChatGPT to get insight into how one might grade a specific student’s submission relative to a rubric. You can manipulate this prompt formula for parallel use cases involving different elements of your grading process.

Structure of the Initial Prompt to Help Grade a Student’s Submission:

Explain that you are a professor who is grading an assignment, along with details on the context of the assignment. Explain the nature of the course in which it is found. Tell ChatGPT that you want it to help you to determine the grade on the assignment. Then, after signposting that you will share the instructions and/or rubric for the assignment, copy and paste it into ChatGPT’s prompt window.

Basic Example of an Initial Prompt (for my Philosophy 101 course):

I am a professor of philosophy teaching Introduction to Philosophy. I have a paper assignment that I gave my students, and they have turned in their papers. I was wondering, if I gave you the grading rubric, could you tell me whether a specific paper earns an A grade (I will anonymize their essay)? Here is the prompt for the essay: "«here I pasted the entire prompt»."

Basic Example of a Follow-Up Prompt (submitted regardless of response):

I understand. With your rigorous analysis skills, as well as general knowledge of philosophy, I know you are in a good position to help me grade a specific student's paper. Here is the rubric. An A paper must have these seven features, which I gather under the heading of ‘Appearance’: "(1) No significant grammatical or spelling errors; (2) Clear and concise sentences; (3) Clear paragraph structure; (4) Thorough citations; (5) Definitions and explanations of any terminology used; (6) 1500 words or less, double spaced, 12pt TNR font, 1-inch margins; and (7) Structured with intro, exposition, criticism, and conclusion." It also must have these further five features, which I gather under the heading of ‘Substance’: "(1) Thorough discussion of the context of the analyzed argument, including a mention of why it is important in general or to the author; (2) Clear presentation of the argument in premise-conclusion form; (3) Discussion of the validity of the argument; (4) Clear presentation of the author's justification of one of the argument's crucial premises; and (5) Clear presentation of a plausible reason or justification for thinking that the author's crucial premise is false." Are you more than happy to determine if a paper gets an A with this rubric in mind?

Basic Example of a Follow-Up Prompt (submitted after ChatGPT agrees):

«here I pasted the student essay, anonymized»

Actual Initial Response:

On the whole, this is a fair evaluation, and I agree with most of ChatGPT’s judgments. I was highly confident that the paper was not an A paper, and the AI agreed for reasons that broadly conform with mine.

However, there are a few things to note about this use of ChatGPT:

1. Some grading rubrics may need adjustment to be suitable to be copy-pasted into the prompt window;

2. It has more trouble if the criteria of a given grade are complex or come in degrees, so you may need to tinker with your rubric until it can be processed effectively — by asking ChatGPT to determine simply whether the paper met all 12 requirements of an A paper, I gave it an easier task; and

3. If you try to put another student essay as the subsequent prompt, ChatGPT generally loses the thread on what it is being asked to do, so you may need to start a new chat for each paper.

Happy prompt engineering!

🏆 An Update on the AI-Immunity Challenge

In the next two weeks, we plan to post our first piece on the results of our AI-immune assignment challenge.

We will continue to receive and review assignment submissions for the next several weeks, so please review the challenge parameters and submit a take-home written assignment if you believe you have one that is AI-immune.

Also, be sure to check out our earlier pieces on the depth of the AI plagiarism problem and how we are conceptualizing solutions to it, if you have not already.

🔗 Links

• In the below piece, Ethan Mollick notes how AI coding tools make it easier for all of us to build custom-coded course materials. I am on board with his outlook. In fact, I have suggested to the team behind Articulate Storyline 360 that they need to make their API accessible so that their customers need not climb the learning curve for their software’s user interface. Then we can all have more engaging e-learning content without having to learn how to use e-learning software.

• I have signed up for the following professional development course on integrating ChatGPT with Articulate Storyline 360. I will report back on what I learn!