Using artificial intelligence as a tool for assessment in medical teaching

Using artificial intelligence as a tool for assessment in medical teaching

It’s my great honor and pleasure to be here. I wish to thank the organizers of this excellent meeting because they asked me to give you a couple of highlights about artificial intelligence, maybe ChatGPT, which is part of a solution I will tell you about, and [to explain] how we can use this technology for the assessment of students of medicine and in medical teaching.

Of course, this topic is so fascinating, at least for me, and so complicated that probably I can discuss it for a week but I have only 20 minutes.

Let’s start from a small disclosure because I am a beneficiary of a grant from the National Centre for Research and Development and as a result of this grant, we develop the application and I will show you today a part of the results of this development to support a couple of ideas we have about how to use AI in medicine to improve the outcomes.

First of all, we should ask why we should use artificial intelligence in teaching of students of medicine or in general, in medicine. But I don’t think that this is a correct question because we’ve already used artificial intelligence in many fields. For me, the actual interesting question is “Can we say no to artificial intelligence?”

And hopefully everybody agrees with me that it’s not possible because it’s too tempting and it gives us too many opportunities. We are not allowed to just not use artificial intelligence.

Another huge problem is related to the future. Of course, I don’t want to make any predictions about the future because it’s impossible, but we can have some wishes and we can think about the future of medicine as medicine affordable for everybody and medicine that is available, so [that] medical procedures will be available for everybody. We expect the highly professional health-care system.

Reaching this goal in the ideal world would be easy, but in our reality it’s not so easy, especially that we face huge problems.

The problem is related to the demand for physicians, which grows much faster than the actual demand and supply [provided] by the universities.

And I think that the situation reflects not only the United States—however, I’ll show you some data that come from the United States—but I think it affects all the developed countries, including Poland.

As you can see, the bad news is that we’re facing a huge shortage of physicians. The numbers you can see on this slide are actually valid only in the situation when we do not expect any progress of medicine. I mean that we don’t expect, eg, any improvement in the quality of medical services. But if we want to make some improvement, it means that just in the United States they will lack ~180,000 physicians by the next 10 to 11 years.

So, it’s a huge number. I mean, additional physicians.

What to do with such a problem? There is a lot of possible solutions. Some solutions have been already implemented.

One of the possible solutions is, for example, to increase the physician effectiveness. But how to do that?

Of course, we can equip the physicians with modern technology, which will help them, for example, to [serve] patients, like the technologies that have already proved to be very efficient, eg, telemedicine, remote consultation, which especially improve the availability of a service system for very remote locations.

But that’s not everything, because now we have more and more sophisticated tools, possibly tools equipped with artificial intelligence, which means that we have new opportunities to help the doctors to be more productive and more effective. And of course, it’s only one of the possible solutions, but it’s not enough to solve this complex problem.

Another obvious solution is to increase the number of physicians. But it’s much easier to say than to do. And it means that we have to increase the number of medical students.

Now another question arises because if we increase the number of medical students, it means that we increase the admission level, for example. Have the universities been already prepared for this increasing number of students?

Actually we are facing huge problems related to that because teaching medicine is not only theory, we need to teach practical clinical skills and how to do it? At the same time the progress and advances of medicine make, for example, an average hospitalization time shorter and shorter. More diseases are treated in the outpatient clinic, so it means that there are not enough patients who are available for teaching practical clinical skills.

And there is another problem because more patients are not willing to have students participate in the process.

So, there are problems we are facing at the universities. And I think that the problems are not only related to the problems we’re facing in Poland, but I think that the same problems can be expected in all developed countries.

At the same time we have other problems. Rotations are mainly provided in the teaching hospitals, so it means that the reference level of teaching hospitals is pretty high. We can expect that there will be a huge number of professionals who are dedicated to teaching, but it creates some biases, like a disease representation bias. (…)

It means that we have the overrepresentation of some of the rare manifestations of diseases, and at the same time we have insufficient cases of common diseases.

And another problem, which seems to be a paradox: we have an under-representation of rare diseases, so somebody can say “Okay, rare diseases are not a big deal because they are rare,” but unfortunately it’s not true.

Just in the European Union we observe ~36 million people who are suffering from a rare disease. Those are rare but there is plenty of them. We have to do something with that and one of the possible solutions is just to improve the recognition/visibility of rare diseases.

But how to do that if an average student has no chance to [have] contact with such a patient during the study period?

There is some solution, of course.

I’m proud to be employed by the Medical University of Silesia in Katowice and we were the first university in Poland that actually opened the simulation center for teaching our students. It was 11 years ago and at that moment it was one of the biggest simulation centers in Europe and definitely the most modern simulation center.

From that time, the program was so successful that now in Poland we have >90 simulation centers of very high quality. But it was 10 years ago and I think that now it’s a good time to make a step forward and to introduce a new technology, which is based on artificial intelligence.

Actually, we now have a very good time for artificial intelligence, because, as you can see, most of the doctors think that it can really improve the availability of the health-care system.

Many doctors know that it has already improved the quality of the health-care system. About two-thirds of us think that it can eventually, in the future, diagnose the patients and help us to treat the patients.

Of course, it can create some problems, but, as you can see, the problems seem to be minor. For example, we’re afraid [that] artificial intelligence can ultimately displace physicians. I don’t think it’s true. But fortunately most of us think that actually it can create new positions and increase employment.

I don’t think we should be really afraid of AI. We rather should know how to cooperate with new technology.

And now we are facing the problem because we can actually use AI for simulation of contact between the students, doctors, and patients, so we can provide some kind of simulations, which we generally call virtual patients.

The idea is absolutely not new. As you can see, I was able to trace it down to 1971, so it’s at least almost 55 years old. But till 2007, the virtual patient, such a concept was implemented in <25% of universities. Ten years later, the situation was a little bit better. But now, after the pandemic situation, we observe a rapid acceleration of transformation of these universities toward such technologies.

Actually, what virtual patients can offer us?

We can use actors and everybody knows about it because it’s pretty old... I cannot say technology, but.... A pretty old way how we can simulate the patients, but virtual patients have some advantages. For example, they never get tired, they are open. Any time the student wants to learn, they can just log in to the system and start solving another puzzle, another case.

It’s very tempting and it can be very supportive for medical education.

But now it means that it’s the responsibility of the universities to promote some of the problems and to promote the understanding of the promises and possible limitations of AI.

We have to prepare our students for the future of medicine.

Of course, we do that with help of different organizations, which can issue some policies, like the policies mentioned on this slide. Now we try to implement these new technologies in our teaching techniques and the teaching process.

I strongly believe that it will be able to combine the digital competencies of our students with the continuing medical education, and we will be able to prepare our graduates, future doctors for any possible new situation in the future and then the possible new technologies.

Benefits.

One of the possible benefits is that in comparison to the traditional way how we can learn or how we can teach the students, which is rather a linear model, we can make some branched, complex clinical cases.

It looks like here. For example, our students can be in the position of the doctor. They can actually make the decision. He or she can choose different options, with other options that are possible connected results. After getting the results, the student can reconsider the options and, for example, proceed with this medical case.

Let me give you an example. The example comes from the applications we actually developed and use at the Medical University of Silesia with a great success.

This is just a very simple case. I prepared this case for this presentation, so it’s not the perfect example of how we perform the interview with the patients, but as you can see now I simulated an actual patient, so I can ask any possible question.

And I’ll try to ask her very simple questions. (…) Just to pretend that I am an unexperienced student who wants to learn something. I can now freely discuss with the patient. And as I told you, I intentionally repeat some questions and I intentionally do something wrong, because later I wish to show you the assessment of my conversation, which is done automatically by AI.

Now I can simulate my contact with the patient. Actually, the whole patient is the creation of artificial intelligence. This is one thing and we can do much more because now we can proceed, for example, to a physical examination. Actually, this is not the strong point of this particular simulation because no simulation can reproduce in reality the contact with actual patient, the texture of the body, and so on. But still, I can get a lot of information and I can actually, which you will see, include some kind of multimedia.

It means that now the student can see, for example, the very real symptoms and feelings, which will introduce them to the problem and, of course, it can be everywhere. And now, if you wish, you can see, you can hear some cardiac sounds.

Oh, actually we can’t. But it’s a local problem. So, actually we can see that you can embed the multimedia in the case. It is another opportunity for the student.

We can proceed with that, because there are some other examples how we can embed, for example, multimedia, to show the very real symptoms, sometimes symptoms the students have no chance to face during the clinical rotation.

You can see that it creates some benefits for us, but it’s not everything. We can use such tools for student assessment. For example, we can assess the time that a student spends on diagnosis and treatment. We can evaluate the actions, we can review the actions and the decisions. Later we can discuss every problem with the students. We can calculate the cost and time effectiveness of the student’s decisions, which is actually very interesting for the students.

And this is how it’s done. For example, proceeding with the physical examination, I can actually order any kind of tests. But I think that, again, we have a small problem, maybe not. Sorry, it’s my fault.

As you can see, the student has an opportunity to order any kind of diagnostic testing. What is very important, because of AI, it doesn’t necessarily means that the student has to order the tests we already predicted, which are valid in this particular case. The student can do anything he or she wants.

For example, the student can order the tests that are directly connected to the case or I can order any kind of test I wish just to try what will happen if I do something wrong.

Of course, there will be no penalty at this [point], no immediate penalty. As you can see I can embed a video on which there can be a still image. I can mark some features that I feel that are important for the student in this particular case. But the student is not limited. They can do something wrong. And actually the student was me, so I tried to do something wrong. For example, I ordered a test that is not necessary, like breast US.

And as you can see we can embed some results from this.

This is generated by AI as well. Now the students can see "Okay, no problem. I can order any kind of test and later evaluate it," at the end. but it will influence the assessment of the students.

Let’s move ahead. Another very important part is that I can propose the diagnosis and treatment for this particular patient.

I can choose anything I wish. I have plenty of options. What is very important, in my opinion, now the student is playing the active part. There is not something like they can only choose the predetermined options. The students can actually do anything they want and that’s why I think it’s the strong advantage of using AI because it will greatly improve possibilities. For example, this was my proposal for this particular patient. But it’s still not enough, we can proceed.

You can see that now because of a simulation, I can assess a different aspect of the student–patient interactions, but I can do much more. I can assess something that is very important, something like soft skills and soft competencies.

I will show you my final assessment, which will probably be not very great.

Another very important aspect is the standardization of this case, because AI generates a different case every time the students log in, so [cases] are not repeating. It means that they are not boring. Every case is a little bit different, but the basic principle is still the same. So we can use such an approach for standardization.

This is my final assessment. As you can see, I wasn’t very good. My overall result was 69%. Well, okay... But what is...

Yeah, I passed, okay? I expected that I should be better. It means that I have to train a little bit more but forgive me, it was just for the purpose of this presentation. But what is very important, I have a full assessment, so I have full information what I’ve done wrong and what I’ve done right. I can immediately see my strong and weak points. It means that next time I will take this patient, I can proceed much better because I learned something. At least I’ve learned what I’ve done wrong.

But that’s not enough. Okay. Sorry, I cannot skip it.

So, for example, the supervisor has full access, so the supervisor can check any actions of the student, for example, the interview, sequence of ordered tests, and so on. But we have much more information.

For example, in the assessment I can see what I’ve done correctly, what I’ve done wrong. But I have information about my soft competencies, for example, how I performed during the interview with a patient, which is fantastic, because the students really need this information back to improve their skills. As you can see, I don’t know if you can read it, but there are highlighted my strong points—what I’ve done correctly, for example, during my interview, what I’ve done wrong. And this is in the form of a description. Actually it’s very informative and I think it’s valuable information for the student.

What is the most important thing, till this moment no teacher is involved in this process of assessment. Everything is done automatically by AI. Basically we use ChatGPT because it’s been already introduced in this version of the application.

Of course, this application is fully integrated with our university computer system, so actually we have an access to all students. We can, for example, enroll them automatically to the groups, and so on, and so [forth]. Of course, there are many more options. We probably have no time to discuss all of them.

But as you can see, such tools based on AI... And this is just the tip of the iceberg, we’ve just started this program and I think the results are already pretty impressive, but it can really help in our medical education and preparing our students for being better doctors in the future.

Such an approach can be used in different situations.

We can use such an approach in problem-based learning (PBL) if you wish. We can use these simulated cases in individual learning, remote learning, and we can use them in the assessment, which is very interesting for us.

In my personal opinion, if such tools will be well implemented, they will create a very friendly environment for education and they can possibly fill the gap between the theory and practice, and can solve at least some of the problems related to obtaining practical clinical skills by our students, at least at the initial stages of clinical education.

To me, the very important thing is the opinion of our students. We made a quick survey in different students’ groups and, as you can see, the results are really impressive.

Most of the students… As far as I remember, maybe there was only 1 or 2 persons who said “I’m not interested.” Most of the students, something like 98% to 99%, are really interested in these solutions. They are so interested that they think that it should be introduced not only at the university, but they want to use it at home.

They found these clinical cases very interesting and, what is very important, they feel that now they are in the position to make a decision that they feel like they are making the real clinical decision, which increases their confidence in near future.

The question is: what do you think, did we achieve our intended goals? Probably not yet, but we are still improving.

And I would like to introduce at least a part of a team who helps us to create this tool. Some members of the team are present here in this room, so after this session, if you wish, I will be available for you. If you have any questions, I would be more than happy to give you an answer.

Thank you very much.