Explainable AI to increase hospitals' use of AI

AI (artificial intelligence) is gradually gaining ground in assistive medical technologies such as image-based diagnosis, where artificial intelligence analyzes CT scans with superhuman precision. AI, on the other hand, is rarely designed as a collaborator for healthcare professionels.

In a new human-AI project EXPLAIN-ME – supported by the national research center DIREC, AI researchers together with medical staff will develop explanatory artificial intelligence (Explainable AI – XAI) that can give clinicians feedback when training in hospitals training clinics.

“In the Western world, about one in ten diagnoses is judged to be incorrect, so patients do not get the right treatment. The explanation may be due to a lack of experience and training. Our XAI model will help the medical staff make decisions and act a bit like a mentor who gives advice and response when they train,” explains Professor at DTU Compute and Project Manager Aasa Feragen.

In the project, DTU, the University of Copenhagen, Aalborg University, and Roskilde University collaborate with doctors at the training and simulation center CAMES at Rigshospitalet, NordSim at Aalborg University Hospital, and oncologists at the Department of Urology at Zealand University Hospital in Roskilde.

Ultrasound scan of pregnant women


At CAMES, DTU and the University of Copenhagen will develop an XAI model that looks over the shoulder of doctors and midwives when they ultrasound scan ‘pregnant’ training dolls at the training clinic.

In the field of ultrasound scanning, clinicians work on the basis of specific ‘standard plans’, which show different parts of the fetus’ anatomy to make it easier to see and react in case of complications. The rules are implemented in the XAI model, which is integrated into a simulator that gives the doctor ongoing feedback.

“It would be great if XAI could help less trained doctors to do scans that are on a par with the highly trained doctors.”
Professor and Projekt Manager Aasa Feragen

The researchers train the artificial intelligence on real data from Rigshospitalet’s ultrasound scans from 2009 to 2018, and it is primarily images from the common nuchal scan and malformation scans that are offered to all Danish pregnant women approximately 12 and 20 weeks into the pregnancy. When the XAI models will be ready to use at the training clinic, first they have to check whether the model also works in the simulator, since the EAI model is trained on real data, while the training doll is artificial data.

According to doctors, the quality of ultrasound scans and the ability to make accurate diagnoses depends on how much training the doctors have received.

“If our model can tell the doctor during the scan that a foot is missing in the picture, the doctor may be able to learn faster. If we get the XAI model to tell us that the probe on the ultrasound device needs to be moved a bit to get everything in the picture, then maybe it can be used in medical practice as well. It would be great if XAI could help less trained doctors to do scans that are on a par with the highly trained doctors,” says Aasa Feragen.

Research associate professor and head of CAMES’ research team for artificial intelligence Martin Grønnebæk Tolsgaard emphasizes that many doctors are interested in getting help from AI technology to find the best treatment for patients. Here is explainable AI the way to go.

“Many of the AI models that exist today do not provide very good insight into why they come to a particular decision. It is important for us to become wiser on that. If the model does not explain why it comes to a given decision, then clinicians do not believe in the decision. So if you want to use AI to make clinicians better, then we need good explanations, like Explainable AI.”

Ongoing feedback on robotic surgery


Robotic surgery allows surgeons to perform their work with more precision and control than traditional surgical tools. It reduces errors and increases efficiency, and the expectation is that AI will be able to improve the results further.

In Aalborg, the researchers will develop an XAI model that supports the doctors in the training center NordSim, where both Danish and foreign doctors can train surgery and operations in simulators on e.g. pig hearts. The model must provide ongoing feedback to the clinicians while they are training an operation without interfering, says Mikael B. Skov, professor at Department of Computer Science at Aalborg University:

“Today, it is typically the case that you only get to know if you should have done something different when you have finished training an operation. We would like to look at how you can come up with this feedback more continuously to better understand whether we have done something right or wrong. The feedback should be done in such a way that the people learn faster and, at the same time, make fewer mistakes before they have to go out and do real operations. We, therefore, need to look at how to develop different types of feedback, such as warnings without interrupting too much”.

Image analysis in kidney cancer


Doctors often have to make decisions under time pressure, e.g. in connection with cancer diagnoses to prevent cancer from spreading. A false-positive diagnosis, therefore, could cause a healthy kidney removed and other complications to be inflicted. Although experience shows that AI methods are more accurate in assessments, clinicians need a good explanation of why the mathematical models classify a tumor as benign or malignant.

In the DIREC project, researchers from Roskilde University will develop methods in which artificial intelligence analyzes medical images for use in diagnosing kidney cancer. Clinicians will help them understand what feedback is needed from the AI models to balance what is technically possible and what is clinically necessary.

“It is important that the technology can be included in the hospitals’ practice, and therefore we focus in particular on designing these methods within ‘Explainable AI’ in direct collaboration with the doctors who actually use it in their decision-making. Here we draw in particular on our expertise in Participatory Design, which is a systematic approach to achieve the best synergy between what the AI researchers come up with in terms of technological innovations and what doctors need,” says Henning Christiansen, professor in computer science at the Department of People and Technology at Roskilde University.