Q&A with the community
An inside look at the important work being done by graduate students and researchers across the country at the intersection of immersive technologies and healthcare.
Nhu Q. Nguyen
Biomedical Engineering Researcher at Sunnybrook
Nhu is an experienced Biomedical Engineer with a demonstrated history of working in the medical device industry. She is skilled in Human Factors, Image Processing, Computer Vision, Extended Reality (AR/MR/AV/VR), with strong interest in Machine Learning.
Q: One-liner. Tell us a little bit about what you're currently working on.
A: Determining how we can use AR in the neurosurgical operating room to improve the workflow from a human factors perspective.
Q: Do you remember your first-ever AR experience? Where was it, and what was it?
A: My first AR experience was with the Microsoft HoloLens at Ryerson University, when I first started my research. I had convinced my supervisor, Dr. Victor Yang, to use research funding to purchase one. I remember my excitement mounting when I first opened the box and started running around the hallways being endlessly entertained by the spatial mapping meshes forming around me. There’s something so stunning in seeing something virtually unfold around you and literally transform your space.
Q: How did you first become interested in using AR in your particular healthcare setting?
A: What really inspired my interest in using this technology was definitely Marvel movies! It’s funny how movies really spark your imagination - even when it seems like the technology is impossible. Iron Man has always famously used a head mounted display within his suit to help him out! In Black Panther- there’s even a scene where one of the main characters uses a mixed reality system to repair another person’s spine. Sometimes I feel like a child pretending to be Iron Man- but I don’t think it’s a bad thing at all!
Let's talk about your work
Q: Why is right now an exciting time to be involved in using AR in a clinical setting?
A. It’s so exciting being a part of this medical mixed reality community right now. The technology has come such a long way, research projects are huge collaborative interdisciplinary efforts, and society is at an all time high for innovation. This technology was something that used to be so unattainable and inaccessible. It’s mind boggling how we can recreate some of these experiences from our ubiquitous cell phones just sitting in our pockets. Mixed reality is so versatile and highly scalable it makes me wonder why anyone wouldn’t want to be a part of this field of research! There’s so much potential to change so many lives around the world with a mixed reality platform.
Q: What is it about AR in particular that makes it a good fit for what you’re doing?
A. AR from a human factors perspective is a very interesting use case, specifically in the neurosurgical operating room. There are many technologies which are incredibly accurate, but are not routinely used during surgery because of poor workflow integration. The operating room workflow already has numerous people, systems, and so much information being exchanged. By using AR, we propose a new method of information transfer during surgery such that it does not take away from the surgeon’s focus and reduces their overall cognitive load. AR allows users to see the real world as well as some virtual information. From this perspective, surgeons can view the surgical site as well as view other patient specific information on an as-needed basis. We can maintain the focus of the surgeon on what matters most and make information available, but not mentally exhausting.
Q: What do you wish your peers and others knew about AR that might be commonly misunderstood or misleading about the technology’s capabilities?
A. One of the biggest things that I wish that other people knew about was that mixed reality has varying degrees of intelligence, and some demonstrate none at all! Some systems may have very good facial recognition abilities, but it does not necessarily mean that it can understand what a nose even is. Similarly, for my purposes, humans are very good at recognizing surfaces and depth- but extended reality systems? Some have a good idea of approximating the space, but still not as smart as us! With that in mind, there are a number of limitations that this technology experiences, especially in an environment like the operating room. Although these are set-backs of the technology in its current state, it only serves as research motivation to improve it in the future!
Q: Are there any aspects of AR/VR that you've found to be underwhelming, over-hyped, or that the industry as a whole tends to over-promise and under-deliver on?
A. One aspect of mixed reality that I think is underwhelming is the “realistic” aspect of it. Often times, in advertisements and reviews, they mention how immersive and convincing the experience is. Our eyes have a very good understanding of space- when we recreate this in extended reality, and try to trick our eyes into believing what they see, users can get dizzy and experience discomfort quite quickly. This nausea is a large contributor to the lack of uptake as well as negatively affects the immersive experience.
Q: Any teams or individuals stand out whose work has inspired some of your thinking?
A. In the context of surgical navigation, there are a number of research groups who are doing some interesting work with soft tissue deformation tracking in order to inform the AR model in real time. It’s an old concept, but wow is it interesting! There are generally some tracking issues in terms of robustness, but the idea is something more companies should look into addressing. There are a number of computer vision algorithms which have so much potential and use in the healthcare industry that aren’t being tapped into right now.
Onward. What might the future look like?
Q: Do you think there are any underserved areas within healthcare where VR has the potential to make an immediate impact?
A. I think that mixed reality as it stands right now can play a larger role in information transfer and reducing mental fatigue in the operating room. Human factors in mixed reality is often overlooked as something that can be worried about later. Mixed reality is in essence, only made for humans, and we may as well use that to our advantage! Displaying information intuitively to any medical personnel, whether it’s patients, doctors, surgeons, clinicians, researchers, etc. can have such a large impact on the healthcare system as we know it! We admire and thank our healthcare teams endlessly for caring for our sick loved ones. Imagine what our hospitals could do if our teams experienced reduced levels of exhaustion!
Q: It seems like mass adoption is eternally just around the corner. Are there any catalysts that you think might help get us there a bit faster? (regulatory, hardware, new industry entrants, payers, geography, etc)
A. I think that hardware and software improvements can really propel the technology forward. With smaller head mounted displays, no nausea, better ergonomics, and robust software solutions, the technology would be unstoppable! It’s these improvements that require the most research, time and energy. However, they can also provide the largest impact. Mass adoption in my eyes is a usability issue- I think when we change the issues that people experience with the technology subconsciously, mass adoption can be pushed further.
Q: Okay, here’s the ‘there-is-no-box’ question: what’s your vision for the potential of AR’s role in the future of healthcare?
A. I think that mixed reality overall has huge potential for the future of healthcare! Once we get over some of these hurdles that the technology has right now, there are so many applications that we can address and really bring to life. Healthcare is a field with such large breadth that it’s hard to imagine a technology like mixed reality not having a place in it. Specifically in my own research, seeing medical personnel with a hands free solution to communication and information transfer would be absolutely incredible.
Q: You made it to the end. You also might have inspired some people out there who are now reading this. How should they get in touch with you?
A. Hello lovely inspired people! Feel free to reach me on LinkedIn or by email. I promise I don't bite (virtually)!