Putting ChatGPT’s Study Mode Through Its Paces

For better or worse, ChatGPT just might be the world’s busiest teacher. Since launching in late 2022, the chatbot has been a go-to source of knowledge on school-related questions for millions of students globally, more than 70 percent according to one estimate.

The problem with this (even the OpenAI team agrees) is that the standard version of ChatGPT tends to operate as an “answer machine”—spitting out facts or writing a paper for you, without helping to impart knowledge as a teacher or skilled tutor would. That’s part of why OpenAI unveiled Study Mode: a version of ChatGPT designed for college students in collaboration with teachers, scientists, and other education experts. The company claims that Study Mode answers this question: How do we ensure that AI is used to support real learning and doesn’t just offer solutions without helping students make sense of them?

Students who engage with Study Mode are “met with guiding questions that calibrate responses to their objective and skill level to help them build deeper understanding. Study Mode is designed to be engaging and interactive, and to help students learn something—not just finish something,” the company said in a post to its website.

According to Leah Belsky, vice president of education at OpenAI: “When ChatGPT is prompted to teach or tutor, it can significantly improve academic performance.” Early research does suggest that AI tutors have the potential to be powerful teaching tools, with the right guardrails in place. But has OpenAI really figured out how to scale up the positives from these types of small studies?

To begin to answer this question, I’ve been experimenting with Study Mode since it launched. Because students are likely already using or experimenting with the tool, I recently took a deeper dive.

What ChatGPT in Study Mode Gets Right

The more specific my question, the better Study Mode was at helping: often providing instantaneous examples, questions to help immediately reinforce that learning, and pointing out areas where I went astray. This made it helpful for specific math questions. When I asked for a general sense of what scientific notation is, for example, Study Mode explained succinctly and correctly, “Scientific notation is just a way to write very large numbers and very small numbers… without writing a ton of zeros.” Then it provided this solid illustration for the rules around scientific notation:

When it asked me to put the number 0.000084 in scientific notation, I intentionally wrote it incorrectly as 8.4 to the negative 7th power. ChatGPT recognized this mistake and provided step-by-step guidance on how to answer the question correctly.

Because of Study Mode’s ability to immediately “see” and correct student misconceptions, I can understand why many educators think a tool like this has so much potential power.

Where It Can Use Improvement

Despite the benefits, Study Mode is still not ready to ace its test as an effective replacement that rivals a skilled human tutor. For more general and open-ended queries, Study Mode struggled to provide grade-appropriate feedback.

When I was asking about photosynthesis—despite saying I was preparing for a high school test—the questions felt a bit basic.

Study Mode asked: Which molecule absorbs light energy during photosynthesis?

• A. Glucose
• B. Chlorophyll
• C. Oxygen
• D. Carbon dioxide

This is a good starting point, but Study Mode never guided me to more complex questions about chemical reactions (unless I explicitly asked it to). The AI tutor also really latched onto the fact that I’d mentioned preparing for a test, hyper-focusing on what common questions might appear, which distracted me from actually learning the concepts I’d need to answer them correctly.

Additionally, the tool has a tendency to throw out so much information at once that I found myself struggling to keep up. And during a few sessions, the AI tutor kept implying that I had mastered the material when I clearly had not—almost giving me the impression that it had somewhere else to be. Study Mode did work better when I asked it to slow down and ensure that I understood the concepts. That’s something a more experienced learner could (and would) do, but it’d be ideal for the tool to recognize pacing issues on its own.

With math, the more involved the question got, the more ChatGPT’s tendency to throw information at me became overwhelming. When using the tool to study for a hypothetical calculus class, it did provide some important key concepts, but not without firing off the equivalent of two pages’ worth of dense textbook-style language at me (none of which was particularly helpful, nor responsive to my individual understanding of the topic).

Where It Goes Off The Rails Entirely

The biggest problems occurred when I asked ChatGPT to help me write an argument essay. I shared a college writing prompt with it that asks students to pick a side about whether social media is good or bad for society. Study Mode started by telling me it would not write my paper for me but would help me write. At first, it lived up to this pledge and did a great job encouraging me to choose a side while guiding me through the process. However, when I said I wanted to focus on the negative impact that social media had on society, it started to go astray.

Rather than encourage critical thinking by asking why I chose this stance or what I thought some negative aspects of social media were, Study Mode provided a list of common negatives associated with social media and asked me to choose. It may not have been writing my paper for me, but it certainly was steering it. This continued when Study Mode started to guide me through the thesis-writing process. Again, rather than initially asking me to write a thesis statement and building its suggestions from what I wrote, it suggested the following formulaic thesis statement with parts left blank for me to fill in:

• “Social media platforms have a mostly negative impact on society because ___, ___, and ___.”

Once again, this isn’t the same as doing all the work for the student, but it also isn’t leaving enough room to encourage creativity.

Later in the process, when I submitted my opening paragraph, Study Mode offered a fair critique but then provided a rewritten version as an “example.” The reality is, many students would copy and paste this example to try to pass it off as their own. In other words, even though Study Mode kept telling me it wasn’t going to write my paper for me, that’s exactly what it did. Arguably just as bad, it took a very flawed but still human-sounding paragraph and turned it into that Muzak-with-words AI writing so many educators have come to dread in the age of AI:

• My original paragraph: Social media has made the world a more polarized, less social, anxious and less educated place. I hate it. This paper will look at research linking social media to the rise in anxiety the decrease in reading time and the spread of misinformation.
• Study Mode’s rewrite: Social media platforms have had a mostly negative impact on society by contributing to increased anxiety, reducing sustained engagement with reading and literature, and accelerating the spread of misinformation.

Great Potential, But Not Quite There

I love where ChatGPT’s Study Mode is going, but even though I think it can be very helpful in certain instances, I wouldn’t be comfortable recommending it to my college writing students in its current form. As the tool evolves, I do expect my position to change.

To be more effective, ChatGPT’s Study Mode needs stronger guardrails against being tricked into writing for students, plus some work with its tone and response length. I found it at its best when you have a specific question to ask. In other words, it’s a wonderful answer machine, which could be useful to many students. However, this is exactly what it claims not to be and is not all that much better than standard ChatGPT. Sure, Study Mode will make you work a bit more to get that answer, but that’s not quite enough on its own.

Anyone who has ever tutored students knows that not giving away answers is merely one of many best practices. Part of why students turn to tutors is for help learning exactly what it is they need to work on. In other words, they don’t just need help working through questions they have; they need to learn the right questions to ask. With this, even in Study Mode, ChatGPT isn’t helping much. Nonetheless, I’d still encourage other educators to explore this tool for yourselves. There are certainly use cases, classes, and subjects where I can imagine it being helpful.