GigXR transforms healthcare training with Microsoft Azure OpenAI standardized patients

Scientific knowledge and medical proficiency are critical to the success of caregivers and first responders, but those are only part of a practitioner’s required skills. Equally important is learning how to communicate with patients in real-world environments. Through practice and simulation, clinicians hone their skills at verbal assessment and history taking—not to mention listening and empathy.  

Traditional patient-interaction simulation requires hiring, training, and scheduling actors who then play specific patient roles. It can be difficult to educate the actors about diseases so they can accurately mimic underlying symptoms. The trainers expect them to provide the right amount of detail and respond naturally in their roles. Such training and hiring are expensive. Results are unpredictable and inconsistent, and live actors are not available on demand in all locations 24 hours a day, 7 days a week, 365 days a year. Overhead for programs with live actors is another cost factor.  

The Gig Immersive Learning Platform by GigXR, which delivers a growing catalog of holographic content for healthcare training, brings together healthcare providers, educational institutions, government agencies, defense agencies, and technologists into an ecosystem that furthers the use of immersive technologies and AI. The company uses mixed reality to blend physical and digital elements to enable hyperrealistic and collaborative 3D learning environments around holographic patients, medical equipment, anatomy, and medical imagery. Together, its global network of subject matter experts shares best practices to inform the medical integrity of each application and support the collective success of GigXR’s growing platform.

An AI-based solution 

GigXR covers the broad spectrum of healthcare training, so it continues to create better ways to enable clinicians-in-training to communicate with patients. “We’ve learned from our experience that the realism afforded by mixed reality matters. We believed that training OpenAI models on Azure could produce highly realistic dialogues that are more consistent than the human actors our customers currently use,” says GigXR CEO Jared Mermey. “The early feedback we are getting is proving we were right.” 

GigXR has developed a solution to create intelligence for specific AI patients using Microsoft Azure OpenAI Service Large Language Models (LLMs). The architecture capitalizes on Azure AI Speech for speech to text and text to speech, Azure OpenAI Service LLMs, and real-time communication sessions on Azure using data stored in Azure Cosmos DB and SQL Server. GigXR also uses numerous other Azure services including Application Gateways, Azure Web App, and more.  

“New technology can be a gamble,” says Stephen McIver, VP of Product Strategy at GigXR. “Microsoft was a natural choice for us because it offers the full package: maturity of its mixed reality solutions, rapidly evolving AI products, government certifications, and the agility to help us scale quickly.” Microsoft Azure has qualified under the Federal Risk and Authorization Management Program (FedRAMP). FedRAMP is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services. With this “FedRAMP Ready” designation for Azure already in place, US government agencies that intend to use the GigXR solution can shorten the formal agency authorization process. 

Since the beginning of GigXR’s work with holographic patients, institutions have sought the ability to talk with them. Before deploying Azure OpenAI Service, the only available dialogues required educators to be the “voice” of the holograms, much like using a microphone to voice patient mannequins. This exercise required each learner to exercise tremendous imagination. Without Azure and Azure OpenAI Service, it was difficult to produce free-form dialogue without educators providing it. “Our ability to create standardized patients that are available on-demand underscores both the strength of our continued collaboration and the underpinning technologies,” notes Mermey. “We can build patients that are intelligent and can facilitate dynamic conversations very quickly.” 

With Azure OpenAI, all the elements of AI were available—including the ability to stitch together the right collections of services to facilitate speech-to- speech. In this case, speech-to-text sends a prompt to Azure OpenAI, which then turns the response into voice using text-to-speech. “Mimicking real patients was important to us,” says McIver. “Qualified doctors and nurses trained the AI patients, and they also continue to learn. For instance, we repeatedly update the model within a conversation, so the AI patient can quickly recall previous discussions.” 

“We figured out a rubric that can scale, and now what we're doing on our end is combining the technical infrastructure that leverages Azure to enable multiplayer conversations and the intelligence that we've created for the patients,” says Mermey. “We think of OpenAI as an embedded technology built into our platform. There’s a meaningful degree of proprietary training built into our solution. From our perspective, not every layer has to be proprietary. We’re also exposing our work to other developers through a software development kit so they can build their own applications on top of our work that take advantage of Azure services.” 

The magic of AI patients 

Without any training or customized setup, medical students, nurses, first responders, and healthcare professionals can talk to the AI patients in free-form dialogue. Users can train in a multitude of languages and accents—and eventually in as many dozens of languages as Azure AI Services make available. GigXR has already developed 65 different and diverse patients for scenarios such as meeting for the first time, performing physical examinations, and giving bad news.  

The AI patients represent a more economical alternative to live actors and enable schools, hospitals, and government agencies to do their training virtually, with no need to send actors to other locations. Technology also gives the students access to a more diverse group of patients than they would otherwise have seen and provides a consistent experience for all the learners. 

Equally valuable has been the ability of the AI patients to provide training to remote areas and during health crises. “Medical training has tended to be more available for well-funded institutions, tending to favor urban areas and universities with large endowments,” says Mermey, “Our tool democratizes this essential teaching to help rural and underserved communities improve their training.” 

Find out more about GigXR on Twitter, Facebook, and LinkedIn.