NHS gets well-architected in response to the pandemic

Scaling traffic during a pandemic is problematic

Established as one of the major social reforms following the Second World War, the NHS provides health services for more than 63 million citizens in the United Kingdom. There are two telephone services British citizens can call as healthcare needs arise. 999 is the number for emergency medical treatment, and 111 is for people who need advice or medical treatment quickly and cannot wait for an appointment to see a doctor. 111 offers a digital version called 111 online, where people can go through a series of questions that direct them to the help they need, whether it be articles on self-care or connection to a healthcare professional.

In the case of COVID-19, 111 online was used as one of the frontline digital services for the government’s response. “Initially, as the case definition was forming, defining symptoms of the virus, people were given generic health information about distancing, washing hands, etc.,” says Ste Nelson, Head of Delivery for UEC Digital Services, former Head of Delivery for 111 online, at NHS Digital. “This moved on to a simple triage flow, offering more help for those with symptoms, but became more complex over the following days as increasingly more detail was added to the government guidance. We were shipping content and changing logic with updates on our site, often three or four times daily, often with very tight deadlines.”

The need for the NHS Digital team to respond rapidly put a strain on the system, but the real problem was the sheer number of people going to the site to get information. “Prime Minister Boris Johnson would make an announcement, and as soon as that happened, the public would go online to find out how it affects them and what they might need to do,” continues Nelson. “We were being used as one of the frontline defenses. People would be trying to access our service because we had the guidance to give the population the information that they needed.”

With the website acting as the hub of COVID-19 information during the pandemic, 111 online experienced such an enormous surge in traffic that severe outages and performance issues occurred due to unprecedented demand.

NHS partnered with Microsoft to re-architect

Hit with these unexpected outages, the NHS Digital team needed to act quickly to understand the root of the problem. Looking through the logs, they noticed that traffic suddenly surged to 25 times the previous peak load at 250 requests per second. Initially, they thought it might be a distributed denial of service cyberattack. However, once they discovered that the surge occurred directly following the COVID-19 news announcements, they realized this was no attack. The site started to become unresponsive and unavailable because of the overwhelming number of users hitting 111 online all at once.

“That’s when we started to question every aspect of our system,” says Craig Bond, Azure Infrastructure Engineer, NHS 111 online, for NHS Digital. “We realized quickly that maybe we had a few too many eggs in one basket. We could tweak certain components, but we would always run out of RAM. Our service was specified according to the number of users we were expecting, not for the unprecedented levels of users hitting our website. There just weren’t enough resources to go around.”

With the pandemic continuing to surge and no end in sight, it was clear this was going to be a problem they had to solve, and fast. Microsoft was alerted to help identify issues and triage solutions using the Azure Well-Architected Framework to re-architect a system that could scale and perform to meet user traffic demands to the site.

“The AlwaysOn solution that Microsoft gave to us did a whole lot more than just fix the actual problems that we faced in March due to COVID-19,” says Neil Mukerji, former Infrastructure Delivery Manager for NHS 111 online. “They certainly addressed those, but they also gave us a better way of working across the whole development process. The framework that Microsoft helped us build enabled us to split our web services far more intelligently across computing resource. We can now scale out our resource with good accuracy where it’s needed, enabling us to provide a better national service. We can also develop more features at once because we can replicate that framework really easily and accurately. Moving our infrastructure into code also enabled us to do things easily like stand up a replica of production to do load tests against, make miniature versions to do integration testing on, and rapidly implement new features.”

The NHS is there for its citizens now and in the future

The result was a completely re-engineered/re-architected system with significantly improved performance, reliability, and security, as well as far superior operability. As 111 online continues forward, the NHS Digital team is far more prepared for unexpected things that come their way.

“111 online is an ongoing program at the very forefront of the national response, not just for COVID-19, but in general for national health,” explains Mukerji. “We plan to continue to develop and modernize it using Azure to prepare for the future where we might use the same framework again for similar services.”