3D Telecommunications goes open source

Bringing Holoportation™ from the lab to the field

A decade ago, researchers from Microsoft unveiled Holoportation™, a provocative new technology that could virtually teleport (opens in new tab) a person from one place to another in three dimensions and in real-time. Using multiple cameras and a HoloLens-augmented reality headset, people could visit with loved ones from a great distance and enjoy a replay of that visit much like they might watch a video.

In the years to come, the 3D capture technology was upgraded, enabling high-quality 3D models of people to be reconstructed, compressed, and transmitted anywhere in the world. Microsoft Research provided the 3D Telecommunications (3DTC) technology that enabled doctors from NHS Scotland to consult with clinicians and patients at Korle Bu Teaching Hospital (opens in new tab) in Ghana, allowing them to collaborate on life-changing surgical procedures in real-time over a live Teams call. This groundbreaking system allowed doctors to see these patients as if they were in the same room, even though they were thousands of miles away. 

Having demonstrated its real-world capabilities, 3DTC is now ready to move beyond the lab. To facilitate future development by external researchers and organizations, Microsoft is releasing the technology under an open source (opens in new tab) license, allowing anyone to freely use and modify it.

“This is a rare opportunity for us to take a technology that we’ve developed and bring it to the public to use. We’re really hopeful that open sourcing it will enable external groups to pick it up and run with it,” said Spencer Fowers, a principal researcher who works on the Special Projects team at Microsoft Research. That’s the team that launched and nurtured Holoportation^TM from a research project through multiple 3DTC deployments.

Fowers and the Special Projects team are tasked with pushing the boundaries of new technologies and proving their value outside of a research environment. This includes work like Project Natick, which showed that datacenters can run efficiently under the ocean using renewable energy. Once the pure research phase of a project ends, the team can move on to incubating additional frontier technologies.

“The 3D Telemedicine project was extremely gratifying to work on. Now, we’ve pushed that technology about as far as we can go. We’ve streamlined it down to the best technologies available and the best price point to allow people to keep moving forward with it. We’re ready to hand it off.”

Improving the system and driving deployments

Holoportation^TM has changed considerably since its first research paper: Holoportation: Virtual 3D Teleportation in Real-Time, which introduced it as an “end-to-end system for augmented and virtual reality telepresence.” Like many new systems, the underlying code was unrefined and a bit buggy, according to Fowers. In addition to continuously cleaning and simplifying the code, the researchers have drastically reduced the system’s operating costs. The initial setup used expensive research grade cameras that required multiple graphics processing units (GPUs) to process and transmit imagery.

Today, the cost of the latest 3DTC system in use in Ghana has been reduced by about 90%. This system is also mobile (opens in new tab)—it fits in the back of a van and can travel to remote locations, making it easier for patients to get treatment. It uses just two standard computers and off-the-shelf imaging components. The researchers also reduced the bandwidth requirements by 97% without compromising quality.

“The initial system required up to three gigabytes of bandwidth. They had to run fiberoptics from one location to the other just to run the demo. Today, it runs on Wi-Fi. We ran it literally in a van using 4G cellular networks from Ghana all the way to Brazil, Rwanda, and Glasgow, Scotland, all at the same time,” Fowers said.

Doctors at Korle Bu Teaching Hospital (opens in new tab) in Accra, Ghana, will continue to use this system to train reconstructive surgeons. To date, doctors in Ghana have seen 30 patients using 3D telecommunications (3DTC). A second system has been donated to ECL Global (opens in new tab), an IT provider in Ghana that has agreed to support Korle Bu, continue to improve 3DTC, and explore additional deployment opportunities. Doctors from NHS Scotland plan to continue to conduct international multi-disciplinary team consults using the 3DTC system to train and assist the surgeons in Ghana.

“We’re building a mobile, locally owned version to deliver lifesaving reconstructive care for cleft lip, burns, noma, and cancer in low-resource settings—and to reshape remote care across Scotland,” said Steven Lo, a surgeon at the Canniesburn Plastic Surgery and Burns Unit (opens in new tab) in Glasgow, UK.

Other external organizations have shown interest in taking the next steps with the technology, including large healthcare providers. Potential applications include providing medical attention for workers at remote job sites such as mining operations, or increasing the reach of specialized healthcare in the rural U.S. Physical therapists could deliver in-home coaching via 3DTC. Non-medical possibilities include entertainment, such as experiencing a Broadway show on stage, or sitting in the dugout with your favorite baseball team. 3DTC could also enable online shoppers to virtually try on clothing, support immersive 3D meetings, or create avatars in real time for virtual and augmented reality.

The research team hopes that open sourcing the codebase will spark new ideas and experimentation that leads to new applications.

That’s actually how the collaboration between the Microsoft Special Projects team and the doctors at Scotland NHS’s Canniesburn Plastic Surgery Unit got started — a speculative email to the Microsoft Special Projects team in late 2019 ultimately led to the 3DTC launch. Lo and his colleagues would travel to Korle Bu Reconstructive Plastic Surgery hospital in Ghana to treat patients, but important consultations and post-operative care proved challenging.

Patients might need to travel for three days just to get to the clinic. Could 3DTC help address this problem?

Work on the 3DTC system soon pivoted to deploying a remote care system in Scotland in response to the COVID pandemic. That initial system in Glasgow was followed by an installation in the hospital in Accra, Ghana, and ultimately the van-based system.

“I’m really excited to see what other non-medical applications people come up with,” Fowers said. “Having access to the code and not having to start from scratch gives people a huge leg up to iterate on what we built and make it even better.”