Is Google's New "Gemini Advanced" Better Than Bard?
It isn't clear. For one, it randomly speaks Russian, unprompted.
Graham Clay
February 12, 2024 • Estimated Reading Time: 14 minutes
[image created with Dall-E 3 via ChatGPT Plus]
This issue brought to you by Packback
Welcome to AutomatED: the newsletter on how to teach better with tech.
Each week, I share what I have learned — and am learning — about AI and tech in the university classroom. What works, what doesn't, and why.
In this week’s piece, I present the learning objectives and agenda for Saturday’s “Build Your Own GPT” webinar, I try to explain what is going on with Google’s AI offerings, and I express my surprise that Google’s new Gemini Advanced speaks to me in Russian unprompted.
Table of Contents
📝 Sign Up for Our
“Build Your Own GPT” Webinar
Our 2-hour "Build Your Own GPT" webinar this coming Saturday!
It is a Zoom-based learning experience for anyone who wants to incorporate custom GPTs into their pedagogy this year. With evidence growing of the effectiveness of this sort of pedagogical intervention, now is the time to jump on the opportunity afforded by recent advances from OpenAI. We will also cover productivity use-cases.
We have only several slots left in the first cohort. We want to keep each webinar small to maximize value for all attendees.
Here are the learning objectives. You will learn…
what custom GPTs are, how they can be used, and what they are analogous to (and thus where else you can deploy the skills you develop in this webinar);
the main components of successful GPT instructions, including those that reliably protect your GPT against misuse and those that enable it to handle complex tasks;
how to add knowledge to your GPT, format it so that your GPT can parse it better, and instruct your GPT to rely on it in the correct contexts;
how to implement and test your own GPT, all while receiving hands-on feedback from Graham and your cohortmates as you carry out these tasks during the webinar; and
about advanced techniques, including integrating your GPT with other applications.
Now, you might be thinking: “wait, what are custom GPTs?” In brief, they are AI chatbots fueled by ChatGPT4 that you supply with instructions and files, thereby empowering them to effectively play a unique role. Graham has discussed using GPTs for in-class activities, as they can enable you to coach your students at scale. Graham uses two in his current classes at UNC Chapel Hill, and his students report that they find them incredibly useful.
The webinar will occur on Zoom this coming Saturday, February 17th, from 12pm to 2pm Eastern Standard Time. It will be run by me, professor of Philosophy and co-founder of AutomatED.
The price is $99. All Premium subscribers ($5/month or $50/year, with prices going up soon) get a 10% discount code, included immediately below. You will need ChatGPT Plus to participate in the activities.
To sign up, click this link or the below button:
❓Question of the Week:
What is Going on at Google?
Last week, I disentangled the 4 different Microsoft Copilots before I set out to explain how to use Copilot for Microsoft 365 to answer student questions addressed by your syllabus, catch students up, and create lecture slides.
This week, with the big shake-up of Google’s AI offerings, I need to do some more disentangling. Here is the short story…
First, Bard — Google’s browser-based chatbot, which competes with Microsoft Copilot (formerly known as ‘Bing Chat’) — will now be called ‘Gemini’. Nothing from Google will be called ‘Bard’ now.
Second, Gemini now has two modes: Gemini and Gemini Advanced. The former is akin to ChatGPT3.5 and is freely accessible at https://gemini.google.com, while the latter is paywalled and is intended to compete with ChatGPT4 (although it currently lacks a fair bit of ChatGPT4’s added functionality, like the ability to upload a range of files or create/use custom AI agents).
Third, you can get access to Gemini Advanced by purchasing a new “Google One AI Premium” plan. It starts at $19.99 per month.
Fourth, Gemini Advanced is powered by a new model called ‘Ultra 1.0’ that Google claims “is the first to outperform human experts on MMLU (massive multitask language understanding), which uses a combination of 57 subjects — including math, physics, history, law, medicine and ethics — to test knowledge and problem-solving abilities.”
Fifth, Duet AI, which is how Bard can be used inside of the Google Workspace apps (Gmail, Docs, Sheets, etc.), is still operational for those users whose companies or institutions have paid for it, but “Duet AI will become Gemini for Workspace, and soon consumers with the Google One AI Premium plan can use Gemini in Gmail, Docs, Sheets, Slides and Meet.”
Next week, we release a Premium deep dive into ways to use Duet AI or Gemini for Workspace as an educator in higher ed, with a focus on Gmail and Slides.
For now, let’s take a look at whether Gemini Advanced is an improvement on Bard with respect to handwriting analysis.
🧪👀 Still Early Days:
Using Gemini Advanced for
Student Handwriting Analysis
When the model underlying Google’s Gemini was first released this past December, it was promoted as being multimodal, “which means it can generalize and seamlessly understand, operate across and combine different types of information including text, code, audio, image and video.”
This piqued our interest, especially given that many professors evaluate handwritten student work. In fact, Google’s own documentation of Gemini’s capabilities included an evaluation of a handwritten student answer to a physics question. Here it is:
there is a lot of debate whether Gemini offers anything unique with respect to multimodal capabilities. It looks like it is comparable to GPT4 Vision (GPT4-V), according to analyses of their respective capabilities. To make matters worse, the initial rollout by Google featured a demonstration video that misrepresented Gemini’s capabilities to make them seem more impressive.
Indeed, Gemini struggled to follow the instructions in the above image, even when provided with a further prompt (“As I say in the image, the handwritten part of the image is the student's solution. I want you to reason about the question step by step. Did the student get the correct answer? If the solution is wrong, please explain what is wrong and solve the problem. Make sure to use LaTeX for math and round off the final answer to two decimal places, if the student got it wrong”). Gemini made several mistakes in its first attempt, although it did correct itself after I informed it of its mistakes.
With the release of the new Ultra 1.0 model, I decided to test to see if Gemini Advanced was more reliable in completing this same task. It is…and it isn’t.
Here was its first response:
There are several oddities here. First, there is a sentence in Russian inserted midway through Gemini Advanced’s explanation of where the student made a mistake. Apparently, it means “[the mistake] is to use the total length of the slope L to calculate the potential energy.” So, it is expressing the appropriate assertion, just in the wrong language. I did nothing to indicate that I wanted an answer in Russian — and I did not want one, as I cannot understand Russian (thanks, Google translate).
Second, while Gemini Advanced recognizes how to calculate the solution, it divides only the right side of the equation by m in the second step, rather than both sides, before inexplicably removing the m from the left side during the third step.
Third, in the final few steps, when multiplying 2g × 40 meters, it correctly multiplies 2 × 40 = 80, but drops the meters unit, leaving 80 × 9.81 meters/s², only to multiple by meters again in the second to last step (to return to m²). Admittedly, this is a confusing example because it involves m, which refers to mass, and meters, which is abbreviated as ‘m’.
In its second attempt, Gemini Advanced did better on these three fronts without me correcting it in between, but made a mistake in its final calculation to arrive at the answer:
2 × 9.81 × 40 = 784.8. The square root of this sum is 28.01. So, while it is an improvement on the prior answer in several dimensions, there is still a mistake that indicates that Gemini Advanced has not ironed out all of the wrinkles with reasoning — especially if professors are to rely on it to help them evaluate student work.
Next week, in a prelude to our Premium piece on Duet AI and Gemini for Workspace, we will compare Gemini Advanced to Bard and ChatGPT4 when it comes to lesson planning.
📬 From Our Partners:
An AI Writing and Grading Assistant
Packback is an AI writing assistant for every student and an AI grading assistant for every instructor. Packback leverages Instructional AI to improve student motivation, writing skills, confidence, and critical thinking through consistent AI-powered one on one feedback.
Our AI acts as an extension of the teacher by coaching and teaching students as they write, allowing them to learn and iterate as they go.
Our AI-powered discussion platform Packback Questions and our AI-powered writing platform, Packback Deep Dives have been proven to create more rigorous classroom discussion and improved writing skills, leading to more A’s and B’s.
For educators, Packback’s built in Digital TA and customizable assignment rubrics have been proven to reduce grading time by up to 30%. To learn more about Packback or to schedule a demo, click here.
📽️🎤 Two Interviews with Me at OneHE.org
On Getting Good Responses from ChatGPT
On the Basics of Custom GPT Creation
Our “Build Your Own GPT” Webinar leaves you with the skills to create your own custom GPT and deploy it with success in your classroom (or to help you save time).
Late in the fall of 2023, we started posting Premium pieces every two weeks, consisting of comprehensive guides, releases of exclusive AI tools like AutomatED-built GPTs, Q&As with the AutomatED team, in-depth explanations of AI use-cases, and other deep dives.
Our next Premium piece will be released on February 21st and is an AI use-case deep dive focused on Google’s Duet AI for Workspace (soon to be called “Gemini for Google Workspace”).
If your college or university uses Google Workspace, you won’t want to miss it.
So far, we have five Premium pieces:
To get access to Premium, you can either upgrade for $5/month, $50/year, or get one free month for every two (non-Premium) subscribers that you refer to AutomatED (note: we expect to raise prices this spring, so now is the time to lock in a lower rate).
To get credit for referring subscribers to AutomatED, you need to click on the button below or copy/paste the included link in an email to them.
(They need to subscribe after clicking your link, or otherwise their subscription won’t count for you. If you cannot see the referral section immediately below, you need to subscribe first and/or log in.)