DAM-Symposium 2024

Overview of Posters

AI4MED – individualized ePortfolio and lab experiences

Poster concept & design: Raphael Cera*, Nina Peltzer*, Chiara Hortmann, Salome Flegr & Jochen Kuhn

*contributed equally

Chair of Physics Education Research, Faculty of Physics, LMU Munich

Abstract

In the introductory physics courses of their studies, medical students face major challenges in terms of physical content. At the same time, students only receive limited feedback on their learning progress and hardly any profession-related content. In the AI4MED project, an AI-based, individualized ePortfolio is being developed to support students of veterinary, dental and human medicine during physics lectures and practicals by providing weekly short tests and subsequent AI-based feedback. This also promotes clinical reasoning and critical thinking. The ePortfolio then supports students in the physical laboratory practicals in combination with a virtual reality (VR) environment tailored to the target group. Taking into account the students' abilities and learning processes, individualized VR visualizations are offered to promote concept understanding, CT and CR.

Automating Qualitative Content Analysis of Evaluation Data: A Methodological Approach Using Large Language Models

Alexander Schmidt, Johanna Huber, Mara Müssigmann, Matthias Stadler

Institute of Medical Education, University Hospital, LMU Munich

Abstract

Qualitative content analysis of open-ended responses in evaluations requires significant time and resources. Large Language Models (LLMs), such as GPT-4o, have proven capable of automating various text processing related tasks. This study aims to develop a method for automating structured qualitative analysis of evaluation responses using LLMs.

Open-ended responses from the Practical Year evaluation at LMU Munich were analyzed using Kuckartz’s structured qualitative content analysis, comparing manual and LLM-based coding.

LLM-based coding achieved a Kappa of 0.75 to 0.8 according to Brennan and Prediger, compared to manual coding. Discrepancies in coding coincided with contradictions in the LLM's rationale, particularly affecting unitizing and category assignment.

The study demonstrates that LLMs can perform key steps in qualitative content analysis. However, further refinement of prompting strategies and model adjustments are necessary to improve consistency with manual coding and to better understand both discrepancies and the model's underlying rationale.

Collaborative Diagnostic Reasoning: A Multi‑Study Structural Equation Model

Laura Brandl(1), Matthias Stadler(1, 2), Constanze Richters(1, 2), Anika Radkowitsch(3), Martin R. Fischer(2), Ralf Schmidmaier(4), Frank Fischer(1)

1 = Department of Psychology, Ludwig-Maximilians-Universität München, Munich, Germany

2 = Institute of Medical Education, LMU University Hospital, Ludwig-Maximilians-Universität München, Munich, Germany

3 = IPN Leibniz Institute for Science and Mathematics Education, Department of Mathematics Education, Kiel, Germany

4 = Medizinische Klinik und Poliklinik IV, LMU University Hospital, Ludwig-Maximilians-Universität München, Munich, Germany


Abstract

In medical diagnosing, like other knowledge rich domains, collaborative skills are crucial. The Collaborative Diagnostic Reasoning (CDR) model highlights the importance of high-quality collaborative diagnostic activities (CDAs; e.g., evidence elicitation and sharing), influenced by content and collaboration knowledge as well as more general social skills, to achieve accurate, justified and efficient diagnostic outcomes (Radkowitsch et al., 2022). Yet, the CDR model has not been empirically tested, so the relations between individual characteristics, CDAs and diagnostic outcomes remain largely unexplored. This study aimed to evaluate the CDR model by analyzing data from three studies involving 504 intermediate medical students diagnosing simulated cases collaboratively using a structural equation model with indirect effects. Our results revealed various stable relations between individual characteristics and CDAs, as well as between CDAs and diagnostic outcomes, highlighting the multidimensional nature of CDR. While both content knowledge and collaboration knowledge were critical for the quality of CDAs, no individual characteristic was directly related to diagnostic outcomes. The study suggests that CDAs play an important role for successful collaborative diagnosing, especially in simulation-based environments. Content and collaboration knowledge strongly influence CDAs, highlighting the importance of understanding the knowledge of collaborative partners. The CDR model should be revised to emphasize collaboration knowledge over social skills. Training programs should prioritize the development of CDAs to enhance CDR skills.

CRDI-T: Developing and Validating a Model for Collaborative Clinical Reasoning

Constanze Richters(1), Frank Fischer(2), Martin Fischer(1), Vitaliy Popov(3) 

1 = LMU Munich, LMU University Hospital, Institute of Medical Education, Munich, Germany

2 = LMU Munich, Department of Psychology, Munich, Germany

3 = University of Michigan Medical School, Department of Learning Health Sciences, Ann Arbor, Michigan, USA; University of Michigan, School of Information, Ann Arbor, Michigan, USA

Abstract

Clinical reasoning in medical practice includes diagnostic reasoning (DR) and intervention reasoning (IR), often requiring team collaboration. Acute care teams, such as those handling cardiac arrest, exemplify time-critical, high-stakes collaborative scenarios with rapid cycles of diagnostic and intervention activities. However, research on IR and the interplay between diagnostic, intervention, and collaborative activities remains limited, with a primary focus on individual DR. Our goal is to develop and validate a process model for collaborative clinical reasoning, focusing on both DR and IR, using data from a VR simulation for cardiac arrest resuscitation (M-TEAM). As a first step toward model validation, we coded diagnostic, intervention, and collaborative activities in five transcripts from simulated expert-led and novice-led teams to identify differences in activity patterns. We analyzed how diagnostic and intervention activities are distributed and used Ordered Network Analysis (ONA) to assess the frequency and types of connections, including self-connections, within collaborative activities. Initial results from two cases show that the efficient expert-led team demonstrated targeted collaborative activities, with frequent connections, primarily focused on interventions. In contrast, the novice-led team exhibited less efficient patterns, characterized by fewer connections and a greater emphasis on diagnostic activities. These findings suggest differences in how DR and IR are coordinated collaboratively between expert-led and novice-led teams. Next steps include analyzing additional transcripts to understand directional relationships between diagnostic, intervention, and collaborative activities, and identifying instructional support needs to enhance novice performance and foster more effective collaboration.

Design and Development of an AI-Enhanced Collaborative Chat Platform for Medical Education

Tarkan Üsküdar(1), Carolin Thiel(2), Daniela Yildiz(2), Albulene Grajcevci(1), Anish Singh(1), Saroj Sharma(1) & Armin Weinberger(1)

1 = Department of Educational Technology, Saarland University, 66123 Saarbrücken, Germany

2 = Experimental and Clinical Pharmacology and Toxicology, PZMS, Saarland University, 66421 Homburg, Germany


Abstract

Scaffolding collaborative learning builds on run-time process analysis, which can be realized by GPT-based Conversational Agents (CA). A case study within medical education fostering collaborative clinical reasoning (CCR) about simulated patient cases in the PaFaSi environment is presented to demonstrate how an iterative design process involving subject matter experts can improve the performance of GPT-based CAs. Preliminary results of the pilot study show that the PaFaSi CA provided learners with adequate feedback as well as scaffolding for their CCR in most cases.

Development of Domain-Specific Critical Online Reasoning

(DOM-COR) Skills in Medical Students During Their Preclinical Studies

Poster concept & design: Jan Zottmann(1), Anna Horrer(1), Jochen Kuhn(2) & Martin R. Fischer(1)

1 = Institute of Medical Education, University Hospital, LMU Munich

2 = Chair of Physics Education, LMU Munich


Abstract

In light of the advancements in medical information, medical education is challenged with equipping students with the skills necessary to effectively navigate this information landscape and make sound judgments (Berndt et al., 2021). Developing critical online reasoning (COR) skills as part of medical studies has become essential – such skills are critical when students use the Internet to retrieve information about patient cases and solve professional problems (Wiblom et al., 2017). A lack of domain-specific (DOM-)COR skills may lead to misconceptions or misapplication of knowledge, ultimately compromising patient safety (Mamede et al., 2019). Thus, our project aims to develop valid assessments to (1) adequately describe DOM-COR in medical students, (2) predict DOM-COR development during academic studies, and (3) predict learning outcomes. Scenario-based tasks are utilized to measure three DOM-COR cognitive skill facets (Molerov et al., 2020): searching for online information (OIA facet); critical evaluation of the information (CIE facet); evidence-based reasoning and synthesizing of the information (REAS facet). A longitudinal study over the first three years of medical studies will examine effects of the medical curriculum on the development of DOM-COR skills based on a random sample of medical students and a comparison group of physics students. We evaluate the medical curriculum by analyzing Internet-based tasks and Internet-like simulations. Our goal is to identify the impact of courses and research activities on student performance. We also consider other key influencing factors such as individual learning prerequisites and interactions with specific properties of DOM-COR tasks or online materials used.

Evaluation of the Application and Relevance of Digital EPAs in a Simulation Software for Spinal Surgery: Perspectives of Experienced Neurosurgeons and Orthopedists on Training Optimization

Maximilian Domann(1), Matthias Witti(1), Fabiana Bertram(2) & Matthias Stadler(1)

1 = Institute of Medical Education, University Hospital, LMU Munich

2 = Brainlab AG, Olof-Palme-Straße 9, 81829 München, Germany


Abstract

This study investigates the application and relevance of digital Entrustable Professional Activities (EPAs) within a simulation software designed for spinal surgery. The primary aim is to optimize medical training for neurosurgeons and orthopedists through the integration of EPAs into digital simulations. Qualitative interviews were conducted with nine expert spinal surgeons to gather insights into their experiences with BrainLab’s simulation software. The findings reveal that digital EPAs hold significant potential in enhancing surgical education by improving task management and patient-centered training. The software is seen as beneficial for bridging the gap between theoretical knowledge and practical surgical skills. However, concerns were raised regarding its user-friendliness and the playful design, which some participants found not reflective of real-life surgery. Additionally, while digital EPAs were found to streamline certain aspects of training, there were limitations in terms of motor skill development. The study concludes that while simulation technology shows promise in medical education, further research is necessary to refine its usability and better integrate it into clinical training. This includes focusing on balancing theoretical concepts with practical applications and assessing the long-term impact of such technologies on surgical competency and patient outcomes.

Making Diagnostic Decisions Count: Design and Development of a Virtual Patient Environment for Fostering Medical Education

Saroj Sharma(1), Carolin Thiel(2), Daniela Yildiz(2) & Armin Weinberger(1)

1 = Department of Educational Technology, Saarland University, 66123 Saarbrücken, Germany

2 = Experimental and Clinical Pharmacology and Toxicology, PZMS, Saarland University, 66421 Homburg, Germany


Abstract

Virtual patients (VPs) are found to be efficient tools for building diagnostic competence and clinical reasoning in medical students. Here, we present an approach that includes additional game elements, namely counters that keep track of various costs of the diagnostic decisions. The initial pilot studies show unanimous acceptance of the platform as a learning tool among medical students with varying degrees of prior knowledge. In qualitative interviews, students refer to the counters as “good reflection,” “increasing the feeling of responsibility” and evaluated it as an incentive for intense group discussion. This has shown to be helpful in designing instructional tools to foster medical education, also allowing medicine instructors to create their own VP environments and patient cases, tune the respective instructional design, and launch VPs for specific learning purposes.

Mapping the Reasoning-Identity Nexus: A Theory-Integrating Review of Clinical Reasoning and Professional Identity Formation in Medical Education

Julius Josef Kaminski(1), Susanne Michl(2), Harm Peters(1)

1 = Charité – Universitätsmedizin Berlin, Dieter Scheffner Center for Medical Education

2 = Charité – Universitätsmedizin Berlin, Institute of the History of Medicine and Ethics in Medicine

Abstract

Background
Clinical reasoning (CR) and professional identity formation (PIF) are fundamental aspects of medical education typically studied separately. This review explores the connections between CR and PIF in medical education, bridging these crucial domains of physician development.

Methods
We conducted a theory-integrating narrative review, analyzing literature and theoretical frameworks through ten distinct lenses, ranging from cognitive to sociocultural and ethical perspectives.

Results
Our analysis revealed significant commonalities and some divergences in the theories and models underlying CR and PIF. We found that CR and PIF are deeply interconnected processes that mutually influence each other throughout a physician's career. The reflection of both concepts through diverse perspectives led to the derivation of a novel theoretical construct: "Clinical Reasoner Identity Formation" (CRIF). This construct integrates the theories of clinical reasoning and professional identity formation.

Discussion
As CR is increasingly understood as more than a mere cognitive activity but embedded in wider systemic contexts, opportunities emerge to connect CR and PIF in teaching and to research their relations. PIF inherently involves how one thinks as a doctor, while clinical reasoning forms a core part of professional identity. This integrated perspective offers new approaches to faculty development, curriculum integration of PIF and CR, and teaching strategies. By conceptualizing CR and PIF as intertwined processes, we suggest avenues for further research and the development of educational interventions that can more holistically support medical students in learning to "think, act, and feel" like a doctor.