AugMedicine: AR for medical diagnosis, Lung Cases

This collaboration began with the following inquiry: How can doctors make a correct diagnosis from a simple complaint like shortness of breath? As a doctor, it is vital to be able to accurately diagnose patients that have breathing complaints. This process is challenging to learn, since clinical reasoning and auscultation (the process of listening to internal sounds of the body using a stethoscope) require a high level of both cognitive, psychomotor skills. To overcome this learning challenge, the Centre for Innovation helped create an innovative learning tool in Augmented Reality. Our goal was to to explore if this technology would provide a missing link between theory and practice in the diagnostic process.

How it works

The HoloLens is used to place a 3D model of an anatomical virtual torso on top of the body of a real person. By using a real stethoscope that is  positionally tracked by the HoloLens, students can interact with the virtual torso. As they touch the hologram with the stethoscope, an audio sample of a lung sound that was recorded from a real patient is played back. The type of audio recording that is played back is determined by the position at which the stethoscope touches the hologram. Through this, students learn to distinguish sounds of various lung diseases and learn how the sounds they hear relate to the position at which they place the stethoscope on the body. Since the application is networked, it can be used in a social learning setting where groups of students collaborate under the supervision of a teacher.

In addition to developing the app, the CFI created a gamified scenario for integrating the app into the medical curriculum. In this scenario, students meet an actor in the role of lung patient. The virtual torso is projected on the actor’s body. By interviewing the actor and requesting additional information sources, such as radiology images and a patient file, students have to come up with the correct diagnosis and formulate a plan for treatment of the patient. By this, students can practice the full process of diagnosing lung diseases in a simulated environment that integrates many of the social, embodied and multimodal components of a real-world setting.

Augmented Medicine: the value of AR for teaching clinical reasoning skills

Besides studying lung cases in AR, there is immense opportunity for this technology to teach medical interns clinical reasoning skills with 3D models and holograms. Within the Internal Medicine programme, we developed an application that helps translating 2D viewing of CT imagery to 3D perspective. This application will be released later in 2019. As a next step in our collaboration with LUMC, we have the ambition to offer students the possibility to study clinical cases of the abdomen through a new HoloLens app. The CFI is working to guide the medical faculty in their further explorations to use and implement the HoloLens in educational programmes. Besides using these applications in the Internal Medicine Masters programme, LUMC is simultaneously researching the effect of the Hololens apps on teaching and learning. The research is supposed by professor Marlies E.J. Reinders of internal medicine.