A Guide: How Professors Can Discourage and Prevent AI Misuse
The past six months of our research is gathered into a guide for professors for the fall semester.
Graham Clay
August 07, 2023
[image created with Dall-E 2 via Bing’s Image Creator]
Welcome to AutomatED: the newsletter on how to teach better with tech.
Each week, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.
In this week’s edition, I present a guide for how to discourage and prevent AI misuse.
Over the past six months, we have been testing take-home assignments for our AI-immunity challenge — experimenting with them to see if we can crack them with AI tools alone in an hour or less. We have been testing our own assignments, experimenting with all sorts of AI tools, and reading about others’ methods online and in our learning community. We have been researching AI detectors. We have been thinking seriously about how to design assignments to encourage AI use, as well as how to design them to prevent AI misuse. We have been consulting with professors about how to build their courses for the fall. We have been collaborating with educational researchers to learn more about best practices for oral assessments and in-class dialogue.
In this piece, I build on these experiences to present a comprehensive guide to discouraging and preventing AI misuse by university students.
I am often told by professors that they would appreciate a zoomed-out take on this issue, rather than a piecemeal or partial approach. Here it is, with the main options available to professors, at least as I see things here in the summer of 2023.
🖼️ The Big Picture
There are six broad strategies you can take to discourage and prevent AI misuse by students on a given assignment:
1. Motivate students to not misuse AI in completing the assignment.
2. Require students to complete the assignment without access to AI.
Because AI can be accessed easily on a device connected to the internet — and even on those that are not — this leaves two device-free options:
Develop an in-class handwritten version of the assignment.
Develop an in-class oral version of the assignment.
However, in some contexts, secure online proctoring is available as an alternative, as I will discuss below.
3. Allow students to complete a (more) AI-immune version of the assignment with access to AI.
Developing an assignment to be more AI-immune is a complex and ever-evolving process, but there are two broad categories of options:
Develop an assignment that is AI-immune due to its format.
Develop an assignment that is AI-immune due to its content.
4. Pair the assignment with another assignment that students must complete in an AI-free zone, such that they are incentivized to achieve the learning objectives in both cases.
Conceptualize pairing like you conceptualize students’ ability to rely on their peers’ expertise at their dorms and in the dining hall. Sure, they can ask their clever friend about how to solve a problem or write an essay, but then they need to come to class and perform with those skills and that knowledge internalized (and not merely memorized).
5. Do nothing.
6. Some combination of the above.
Let’s discuss these in sequence.
1 - 🍎 An Ounce of Discouragement
…is worth a pound of prevention.
Or so Professor Paul Blaschko argued in a recent interview at AutomatED. According to Blaschko — and plausible in its own right — one of the central goals of the educator is to convince students of the extrinsic and intrinsic value of what they are learning. A teacher’s job is not just to convey facts, information, or knowledge, but rather to inculcate in their students the motivation to care about their subject matter, to find it interesting, and to value learning it the right way — both for what it is in itself and what it will bring them. The result of a professor successfully completing this motivational task will be students who do not want to misuse AI or, more generally, cheat.
While there are some students, some professors, and some contexts where this strategy may be less feasible than others, it is surely the first line of defense for many professors. Besides, we should be working to motivate our students in these ways more generally, AI misuse aside.
One way to alter the motivation paradigm entirely is “ungrading,” which has slowly gained some popularity — and notoriety — in recent years. In short, ungrading is a broad family of pedagogical practices that deemphasize or remove graded assessment. You can read about it ad nauseum elsewhere (see here, here, or here). In whatever way you implement ungrading, the core idea would be that it would discourage AI misuse because it discincentivizes students from wanting to cheat or plagiarize in the first place, as Emily Pitts Donahoe has discussed. The concern, of course, is that it disincentivizes them in bad ways, too, but that’s a debate beyond the purview of this piece. Nevertheless, it is worth considering as an option.
Another option would be to have an AI detector policy — a policy that is conveyed to your students that, among other things, informs them that their submissions for your assignments will be evaluated by AI detectors. (We have recently been discussing a policy like this in the AutomatED learning community.) Even if you cannot get students to disvalue AI misuse by encouraging them to value what they learn/gain from being honest and earnest, you can at least use the risk of getting caught to disincentivize their AI misuse.
As we have discussed, there are some issues with this latter approach. First, AI detectors are a “black box” of sorts, just like AI tools themselves, in that they cannot provide the same sort of transparent evidence for their claims that prior forms of AI plagiarism detection tools could. Second, AI detectors vary in their reliability, both at a time and in general, and it is not clear which ones are reliable. Third, you need to remain cognizant of and compliant with institutional policies governing AI detection at your university. You may not have the freedom to develop your own policy.
2 - 📝 An AI-Free Zone
Moving to the second option on the list above, I will now discuss ways to require students to complete assignments without access to AI.
There are three broad methods here:
Online Proctoring (Software and Human)
There are some options for online assessment that attempt to limit students’ access to AI tools — and/or the internet — during remotely administered online assessments. Some of them involve software/AI solutions to prevent AI misuse, while others involve proctors watching via Zoom or reviewing recordings. One benefit of the COVID-19 pandemic is that it forced many universities to more seriously experiment with these solutions, and there are now many insightful research studies analyzing the results.
For instance, in a large-scale study of Tel Aviv University’s efforts during the pandemic, authors Patael et al. survey the literature surrounding online assessment and provide a thorough discussion of the opportunities and challenges (see section 2 for a summary; for other useful discussions, see here and here). Even after noting the significant concerns about online proctoring in general, they describe Tel Aviv University’s efforts and conclude as follows:
This study described the procedures for online-proctored examination implemented by TAU University in the Fall 2020/21, and reported its assessment by TAU students and faculty. The described procedures were born out of necessity triggered by the COVID-induced lockdown. Nevertheless, it appears that the protocol employed for conducting proctored assessments remotely can serve as a fertile foundation to provide a direction for a viable alternative to traditional university models of fully on-campus examinations, as the institutions for higher education move towards normalizing online assessment in the post-pandemic environment. We demonstrate that even in a challenging learning context—such as the one triggered by the COVID-19 pandemic—online videoconferencing technology can be used effectively as a quality assurance mechanism to carry out proctored examinations remotely. […]
Moreover, this research demonstrated potential advantages of a well-designed use of the Moodle and Zoom, for conducting proctored examinations remotely, allowing us to provide some key recommendations for institutions, examiners and students. […] The good news is that the institutions for higher education should not throw the baby out with the bath water and revert to the fully f2f mode of proctored assessments once the pandemic-related restrictions are lifted and the students are allowed back on campus. In fact, paradoxically, the COVID-19 pandemic might have acted as the catalyst to create a disruption form the prevailing model.
If you are interested in this option, its viability in your case depends on whether and to what degree your institution has the technology and policies in place to accommodate your plans, as well as the fit between your assignments and the technologies required.
In-Class Oral Assignments
A second option is to move assignments into the classroom where students cannot access their devices (at least during certain times).
If students lack access to their devices, then — setting aside performative assignments like those involved in athletics and dance — there are two ways they can complete assignments: orally and via handwriting.
We have already discussed at some length the pros and cons of oral exams in a prior piece (see other discussions here, here, and here). I have also recently co-authored a piece in Inside Higher Ed with experts from the Constructive Dialogue Institute on best practices and ideas for in-class dialogue-based assessments.
But one question I have received after we posted the piece on the pros and cons of oral exams was this: supposing I wanted an oral assignment to play a central role in my class, what do you recommend?
My answer revolved around my own experiences with oral assessments which are always paired with written assignments, so I will wait to discuss them below in section 4.
In-Class Handwritten Assignments
This option is self-explanatory — and others have argued for its viability in other contexts (see, e.g., here) — but there are a few considerations worthy of your attention.
First, remember that many of today’s students write primarily with their devices, so completing assignments by hand can be relatively foreign method of composition. Second, there are issues with handwriting variability, including issues with objectively assessing the content it expresses. For instance, there is some evidence that graders assess a sentence, paragraph, and paper differently based on the legibility of the handwriting used to express it. Joseph Klein and David Taub report that “there are significant differences in the manner of evaluation of compositions written with varying degrees of legibility and with different writing instruments” (see here).
The only systematic and recent literature review of handwritten examinations I have found is that of Cecilia Ka Yuk Chan, who concludes that there are four “unambiguous advantages” of typed exams: “alignment with industry practice,” “easing examination anxiety," “legibility,” and “efficiency in grading and feedback.” Likewise, there are five “unambiguous disadvantages“ of typed exams: “[un]fairness,” “unsuitable for all disciplines and examination types,” “accessibility,” “[learning] environment disturbance,” and issues related to “technology adoption and training.” And there are many “ambivalent factors” that Chan notes, as well as issues with stakeholder’s perceptions. Chan concludes with a mixed and cautious conclusion as follows:
In conclusion, the shift from handwritten to typed examinations carries both opportunities and challenges for educational institutions. By thoroughly assessing the advantages and disadvantages of each format, carefully considering the needs of various stakeholders, and adopting a proactive and well-planned approach to implementation, higher education institutions can maximize the benefits of typed examinations while minimizing their potential drawbacks. This, in turn, can contribute to a more efficient, fair and inclusive assessment environment for all students and faculty.
If you are interested in this option, it is worth a look at Chan’s paper, in order to consider each of the factors Chan discusses (and the literature Chan cites in connection with each) in relation to the assignment you are interested in.
3 - 🛡️ AI-Immunizing Assignments
If you cannot sufficiently motivate students to not misuse AI in general and you cannot move a given assignment to an AI-free zone, then one option is to increase the AI-immunity of that assignment. (Another option is to leave it susceptible to AI misuse — were it standalone — but pair it with a second AI-immune assignment, an option which we will consider next.)
There are two broad strategies here:
Develop an assignment that is AI-immune due to its format.
You could have students handwrite their submission. You could have them write a paper with a word processor, print it, and mark it up. You could have them record their submission orally. You could have them make a video. You could have them screenshot their sources (note: ChatPDF and other PDF plugins for the LLMs can locate source texts with some reliability, but most cannot provide screenshots of them without a lot of extra work).
One popular way to carry out this strategy would be to require students to turn in submissions that allow the grader to review the version history of their submission in Microsoft Word or Google Docs. What this version history will reveal is a pattern of creation. If, for instance, a student pastes in a swathe of an essay from another source, then it is more likely that this bit was produced by an AI tool (or another source). In short, an honest and earnest submission is created by a different sort of process than a dishonest one — and your job as a professor would be to differentiate the two to distinguish AI misuse. (For discussion of using Google Docs’ version history in this way, see here or here.)
There are some issues with this way of carrying out this strategy. First, students can forget to turn on version history, misunderstand the functionality, have technological issues, etc. Second, all of these format-based strategies can be beaten by a student transcribing AI tools’ outputs. Whether version history is turned on or not, I can have a separate tab open on my browser with ChatGPT’s outputs, and I can type them at an irregular pace into my Google Doc, deleting periodically, etc., thereby simulating an honest process. Third, there are serious issues with the standards of detection — namely, lots of grey area cases where it is not clear whether a student was dishonest or not. Fourth, it takes a lot of time to review the tangled web of tracked changes or the version history of a Google Doc.
With that said, remember that the core idea is to make students submit their response to the assignment in a format that disincentivizes them from misusing AI. AI-immunity is not binary; it is a spectrum upon which assignments can be placed, from less to more. So, even if this strategy is vulnerable to some methods of circumvention, most of them involve the student working harder to be dishonest. Thus, they increase the AI-immunity of the assignment.
Develop an assignment that is AI-immune due to its content.
There are many takeaways from our AI-immunity challenge, which focuses primarily on the content of assignments that make them more or less AI-immune. (By ‘content’, I mean the subject matter that the assignment concerns, rather than the manner in which it is presented.) In the case of the Clinical Research exam we tested, several aspects of its content made it more AI-immune:
It required students to meet high “field specific standards” — that is, standards that are unique to its discipline that require specific skills or knowledge to meet. Since the exam was a Clinical Research exam, students were required to provide sufficient and accurate “clinical detail” that displays “thoughtfulness” about very technical medical experiments, which is something that AI tools sometimes have trouble doing because of the broad nature of their training data and their broad applicability. Here is what I wrote about the relevance of this more successful aspect of the assignment in my conclusion:
Professors should be thinking about analogs to “clinical detail” and “thoughtfulness” — namely, field-specific standards that are challenging for AI tools to meet in completing tasks like those their assignments involve. Crucially — and we cannot emphasize this enough — you should experiment to see if you can get tools like ChatGPT to generate responses that are sufficiently sensitive to your field’s standards to receive passing grades on your take-home written assignments. It is very hard to predict how AI tools will perform from the armchair (hence this challenge).
Furthermore, the Clinical Research exam was more AI-immune because it required students to engage with unique content. This is content from the Clinical Research domain that was not easy for AI tools to engage with because of its specificity, format, and availability. Specifically, it required students to engage with UC Davis’ IRB guidelines, which are not presented in a single document or simple online source. The assignment would have been even more AI-immune if it had required students to cite content that the AI tools cannot have in their training data (or be supplied via upload), including this sort of unique content, or if the content it relied upon had been more unique, like the professor’s oral presentations in class.
In the case of the Economics project we tested, its content was hard to crack with AI tools for similar reasons. The AI tools we used had trouble meeting the Economics professor’s high field-specific standards. For instance, they struggled to consistently distinguish between concepts like economic growth and economic opportunity, as well as to make inferences and claims that were appropriately sensitive to nuances surrounding them.
The Economics assignment also had a few other features that made its content more AI-immune:
It required students to find obscure journal articles to support specific economic hypotheses, which can be a bit of a challenge for even those AI tools dedicated to this sort of scholarly endeavor. As I wrote at the time:
It worked for the professor to demand students to find relatively obscure works — scholarly literature — to support their hypotheses. After all, generic LLMs like ChatGPT3.5 (and ChatGPT4) hallucinate responses about this sort of content, and AI-enhanced search engines like Consensus struggle with the nuances specific to certain fields. Without guidance from a student who is informed and engaged with the assignment, it is hard to find the right journal articles.
It also required students to complete a long list of steps that built on each other in very complicated ways. This feature of the assignment effectively “compounded and amplified” any AI-generated snafus from earlier steps in the submission creation process.
Why this increased its AI-immunity is best illustrated by contrast. AI tools are generally quite good at simplistic iterated assignments. For example, if an assignment requires the student to write a paper first and then revise it in light of their instructor’s comments, they can write the paper with the help of the AI tools, get the instructor’s comments, and then feed the paper and the comments to the AI tools and ask for a revised version. This contrasts with the Economics assignment we tried to crack, which had many complex layers of iteration, several of which introduced a significant likelihood of AI-generated error.
We will be releasing a piece on another submission to our AI-immunity challenge next week — a Philosophy assignment — so stay tuned for more information on this front.
To conclude this section, the main thing I should emphasize about the AI-immunity of specific format and content requirements is that professors should experiment with them in connection with their specific assignments. We cannot emphasize enough the importance of professors doing the experimentation themselves, as we are consistently surprised by what AI tools can and cannot do when put to specific tasks.
4 - 🍷🧀 Pairing
As a warmup to the next option for discouraging and preventing AI misuse — what I call “pairing” — consider the fact that no assignment is ever given in isolation in the university setting, with very few exceptions. Each assignment is assigned as part of a course, module, or program, which are themselves composed by a range of other activities and assignments.
With this in mind, “pairing” enters the optionspace. After all, if the assignment that you are worried about — the assignment that is open to AI misuse — is part of a whole that itself disincentivizes AI misuse, then that may disincentivize students from misusing AI in completing it.
The key thing is to link the assignment you are worried about with another assignment that cannot be completed with AI or which, more generally, students will complete honestly and earnestly. Crucially, the link should be a motivational one. You want to pair the two assignments in such a way that students must complete the AI-susceptible assignment honestly and earnestly in order to succeed at the AI-immune assignment (or the assignment which is completed in an AI-free zone).
Pairing requires the professor to find a second assignment or task that the student must complete in connection with the first assignment that effectively reveals, when completed, whether the student plagiarized the first with the help of AI tools.
For example, you could allow students to complete a written assignment in class with their laptops that is not AI-immune at all but then you could immediately thereafter query them verbally one-on-one about the content of their writing (or about the subject-matter that their writing concerns).
Alternatively, you could have students present their take-home written work to their classmates in a way that requires them to engage at length with their classmates' organic and spontaneous comments and questions. ("Flipping the classroom," as this sort of method is sometimes called, is widely seen as pedagogically effective in many contexts anyway.)
You could have students apply their written submissions' content to a real-world case that is revealed to them only in class, and then create a case study during class time.
After a student completes a pair of assignments, the professor needs to have a rubric or set of criteria by which they judge the student's submissions. This rubric should award grades that are – in some way or another – sensitive to mismatches between the quality of student's submissions.
A more specific example of pairing is as follows (this is the oral one I promised above):
In my Philosophy classes, students often need to write take-home essays on specific philosophical topics or figures. But undergraduate Philosophy essays like these are especially vulnerable to AI misuse. Although I do various things to try to make them more AI-immune, I also pair one of them per semester with an oral exam. Rather than give my students written feedback on the paired essays, I meet with them one-on-one in a tutorial setting (on Zoom or, ideally, in person) to give them oral feedback on their papers, and then we transition into an oral assessment of the positions they developed and defended in their essays. They get an independent grade on the take-home essays, but they are also graded on their performance in the oral exam. If they blow off the essay or cheat on it (with AI or otherwise), they perform much poorer on the oral examination, which is weighted high enough that this disincentivizes such poor performances before they occur.
To address an objection: no, I do not end up spending much more time grading because the time I would have spent writing feedback on their essays is dedicated to scheduling and running the oral exams (wherein I give them feedback on their essays). Sure, they get less detailed feedback on their writing, but other assignments can fill that role. (And the scheduling is handled via Calendly.)
5 - 😴 Do Nothing
The second to last option to consider is this:
Do not try to discourage and prevent AI misuse.
This may sound silly, but all of us will do it to some degree. It is not as if all of our decisions as professors will be or should be guided by an attempt to discourage and prevent AI misuse. We will and should often make our assignments more vulnerable to AI misuse in order to make them more pedagogically appropriate for students who are not trying to be dishonest. We will and should often not dedicate our efforts to being “prison guards” or “police” — punitive watchkeepers of the ivory tower. And that’s fine.
6 - ➕ Some Combination of the Above
I have presented the above options as distinct paths that a professor can take. However, the best strategy is likely a combination of these options.
We should work to motivate our students to do the right thing, as well as value the subject matter and their mastery of it for its own sake and their sake.
We should require students to perform some assignments in AI free zones.
We should explore using oral exams and other in-class methods of assessment.
We should try to AI-immunize our take-home assignments or pair them with others — especially if there is no loss to pedagogical appropriateness or impact.
And we should, at some point in the course development process, rest content with having done enough.
🔗 Links
🔊 AutomatED’s Subscriber Referral Program
We're sure you're not the only one in your network intrigued by the rapidly evolving landscape of AI and its impact on higher education. From the optimists who foresee AI revolutionizing education by 2030 to the skeptics who view it as mere hype — and everyone in between — we welcome diverse perspectives.
Why not share AutomatED with them?
Once you've made two successful referrals, you get membership in the AutomatED learning community. This platform is designed to thrive on the participation of engaged and informed members like you.
To make a referral, simply click the button below or copy and paste the provided link into an email. (Remember, if you don't see the referral section below, you'll need to subscribe and/or log in first).