Canary Speech analyzes conversational speech in near real time, innovating healthcare with Azure

Giving a voice to medical assessment

Assessing people’s physical and cognitive state and detecting disease hasn’t been as efficient as it could be. What if, by tapping into the power of AI and machine learning, a brief conversation between a patient and their doctor was enough to accurately identify and monitor anxiety, depression, fatigue, diseases, and other forms of cognitive impairment? The evolving demands of patient care require innovation, and health technology company Canary Speech is ready to respond.

With the American startup’s proprietary vocal biomarker technology, a 40-second voice sample can match the data density of an MRI, producing thousands of markers of speech and up to 12 million data elements. These outputs are not only a rich source of information but an objective one, according to recently expanded Current Procedural Terminology (CPT) codes—the uniform language used by health professionals to code medical services. “If you’re looking at other medical biomarkers such as behavioral health interviews, patients are reporting about themselves subjectively, and tests can take hours to conduct and assess,” explains Nate Blaylock, Canary Speech’s Chief Technology Officer. But as the company’s cofounder and Chief Executive Officer, Henry O’Connell, shares, “With Microsoft Azure, we can process conversational speech and return scores within three seconds, offering real-time information to doctors and clinical team members on patients’ welfare and medical state.”

Since adopting Azure, Canary Speech’s approach is straightforward and streamlined. With informed consent, conversational speech can be recorded noninvasively and without specialized equipment in a clinical setting or remotely using a phone or Microsoft Teams. Once a speech sample is processed in Azure, clinicians receive a vocal score they can use to track the presence and severity of diseases and impairment. That result can then support decisions to refer high-risk patients to specialists for optimal treatment and care.

What makes the company’s approach to speech so unique is that while other products listen to what people say, Canary Speech focuses on how it’s said. “You can say today is the best day of your life, but our algorithm looks at how you’re saying that and picks up on filler words, pauses, and other cues to assess your mood and overall health,” says Blaylock. “It complements the rest of what’s happening in AI in speech today because what people say and how they say things are both very important in a lot of different applications.”

Engaged with top-tier health companies, including the CDC and Intermountain Healthcare, a Microsoft and Nuance partner, Canary Speech’s technology reaches hundreds of thousands of patients in the healthcare space. “All you need to use Canary Speech is some way to capture audio and then send it through Canary’s API,” says Blaylock. “After you get the assessment information back, you can start delivering actionable care.”

A high-tech, API-first experience on Azure

Innovation has been at Canary Speech’s core since day one, with its cofounder and several executives having led teams that worked on pioneering speech-to-text and far-field speech technology. Born in the cloud, making the decision to migrate to Azure unlocked a range of performance improvements and growth opportunities. “We essentially wanted to replace the engine that’s running the car while the car was still running,” explains David Brown, Vice President of Engineering at Canary Speech. “We looked for architectural equivalency and similar features and along the way, we found things that we could improve with minimal impact.”

Canary Speech stood up GPU-based virtual machines in Azure, running its own custom model training, processing, and tooling on top. Backed by Azure infrastructure, the Speech to Text feature in Azure AI Services gave it highly secure real-time audio transcriptions—the stepping stone to its speech analysis—and support for multiple languages. As it built out its full environment, Canary Speech deployed highly integrated managed application services using Azure Kubernetes Service, which it has since identified as the most critical piece of its Azure puzzle. “As we start to see new and exponential growth, we’ve taken special care to make sure we can scale through our architecture as much as the business needs,” says Blaylock. “If our growth means going to hundreds of millions of transactions a month, we can do that with Azure Kubernetes Service, which is especially important for us as a startup.” Adds Brown, “We really need the economy of scale that Azure can bring, which we can’t build ourselves.”

Early in its journey, Canary Speech teamed up with an external technology-support provider and was able to migrate and fully transition all of its traffic to Azure in six months. “Our amazing account team and account executive were fantastic in making this process straightforward and helping us understand the initiatives going on within Microsoft that align with our values and what we’re trying to accomplish,” says Brown.

For customers, Canary Speech’s Azure environment translates to a simple API-first experience that supports countless use cases in and beyond traditional healthcare. “Using the same API that made it possible for us to integrate our app with Microsoft Teams, our customers can also take this to their own solutions and applications very quickly,” says Blaylock.

A highly secure environment and enterprise approach for global expansion

Canary Speech is as much a healthcare company as it is a technology enterprise—it develops all its solutions within the rigors of the healthcare environment, facing a full range of peer reviews, protocols, and institutional review-board approvals. The company uses this level of scrutiny to improve and maximize the quality and accuracy of its models. On the technology side, Canary Speech particularly values its Azure foundation and the Microsoft commitment to augmenting its cloud platform for the healthcare industry. “Microsoft’s support for clinical and hospital work and the deployment of new technology is a big deal for us because it’s a partnership of like minds,” says O’Connell. Adds Blaylock, “This is also really key for our customers because they’re now moving to the cloud and there’s a lot of insecurity, but Microsoft delivers confidence and a more enterprise-wide approach in Azure.”

For its own security, Canary Speech relies on Azure Application Gateway, Azure Web Application Firewall, and Azure Front Door to protect its API and application services. The company has also started to lean heavily on Microsoft Sentinel and Microsoft Defender. “We recently implemented and adopted a full security-assessment posture through Defender and incident management and detection through Sentinel,” explains Brown. “That all fits into more immediate goal that we’re trying to achieve in the next few months of high trust certification.”

Working with sensitive medical information, a highly secure, trusted environment affords Canary Speech added credibility with industry partners and customers, not only in the United States but Europe, Japan, and other future expansion areas. “When we were building out in the United States and preparing to pass our audits for security, data handling, and data management, it was natural to also do that in Europe and Asia with the global reach of Azure,” says O’Connell. “This was much easier with Azure than our previous platforms, and it was far quicker than building the environment out ourselves.”

Navigating the future with “a key partner in healthcare”

Committed to growing with Azure, Canary Speech is ready and equipped to continue scaling easily and in a highly secure way while adding new and next-level Azure tools, such as Microsoft Fabric and OneLake. “Our DNA has allowed us to build a tech stack on the speech AI side that’s really not available anywhere else,” says Blaylock. The result: being there for clinicians whenever patient assessments are underway. Adds Brown, “The model training that we do in Azure for a new disease or medical scenario can happen within a couple of months, which is pretty unprecedented in the medical field.”

In addition to underpinning its solution, Canary Speech credits its relationship with Microsoft for shifting its perspective on positioning for the future. “Moving to Azure not only provided us with a web service, but a key partner in healthcare,” says O’Connell. “Working with Microsoft, as it’s committed to innovation and approaching the market with solutions that can change the nature of healthcare, has been everything to us.”

Find out more about Canary Speech on Facebook and LinkedIn.