Generative AI Beats 75% of Students in Medical Informatics—A New Era of Education?

William Hersh, M.D., a longtime professor at Oregon Health & Science University of Medical and Clinical Informatics, became interested in the expanding field of artificial intelligence. He pondered the performance of AI in his course.

And so he decided to experiment.

In an online version of his well-liked introductory course in biomedical and health informatics, he examined six types of generative, large-language AI models, such as ChatGPT, to see how they performed in comparison to real, thinking students. The response, according to a study that was published in the journal npj Digital Medicine, is better than that of up to 75% of his human students.

Also Read: How to Edit ChatGPT AI-Generated Image with DALL·E Editor within Chat?

“This does raise concern about cheating, but there is a larger issue here,” Hersh stated. “How do we know that our students are learning and mastering the knowledge and skills they need for their future professional work?”

Hersh is particularly tech-savvy because he teaches clinical epidemiology and medical informatics at the OHSU School of Medicine. Hersh stated that the use of technology in the classroom is nothing new, citing his personal experience switching from slide rules to calculators as a high school student in the 1970s.

However, the move to generative AI is a quantum leap ahead.

“Everyone should have some kind of foundation of knowledge in their field,” Hersh stated. “What is the foundation of knowledge you expect people to have to be able to think critically?”

Also Read: ChatGPT Usage Limit: How Many Questions Can You Ask ChatGPT in an Hour?

Large-language models

The knowledge evaluation results of 139 students who enrolled in the introductory course on biomedical and health informatics in 2023 were retrieved by Hersh and co-author Kate Fultz Hollis, an informatician at OHSU. They used course assessment materials to stimulate six large-scale generative AI language models. AI performed in the top 50th to 75th percentile on multiple-choice questions used in quizzes and a final exam that required brief written answers to questions, depending on the model.

“The results of this study raise significant questions for the future of student assessment in most, if not all, academic disciplines,” according to the authors.

This is the first study to compare students taking a comprehensive academic course in the biological field with large-language models. Unlike more participative academic courses that assist students in developing more complicated skills and abilities, Hersh and Fultz Hollis observed that a knowledge-based course like this one may be particularly well-suited for generative, large-language models.

Also Read: How US Universities can benefit from using ChatGPT for Students, Faculty, and Campus Operations?

Hersh recalls his time as a medical student.

“When I was a medical student, one of my attending physicians told me I needed to have all the knowledge in my head,” he explained. That was a reach even in the 1980s. The human brain is not able to retain the vast amount of information that medicine possesses.”

Preserving the Human Element

However, he thinks there’s a thin line separating the wise application of technological resources to enhance learning from an over-dependence to the point where it impedes learning. The ultimate objective of an academic health institution such as OHSU is to train medical practitioners who can both optimize the use of patient data and information in the real world and provide quality patient care.

In that sense, the human touch will always be necessary in medicine, he said.

Also Read: How to use the ChatGPT App on iPhone and Android?

“There are a lot of things that health care professionals do that are pretty straightforward, but there are those instances where it gets more complicated and you have to make judgment calls,” he stated. “That’s when it helps to have that broader perspective, without necessarily needing to have every last fact in your brain.”

Hersh stated that he is not concerned about cheating because fall classes will shortly begin.

He stated, “I update the course every year.” Large-language models may not always reflect the most recent developments in any given scientific subject because discoveries are made regularly. This only means we’ll have to look at newer or more subtle tests where you won’t get the answer out of ChatGPT.”

Also Read: How to Create or Generate AI Images for FREE using ChatGPT (Easy Steps Here!)