TalkTalk's journey to the Cloud
TalkTalk has committed to make broadband, TV, mobile and calls simpler, fairer, and more affordable. To support this, we need to make fast data-driven decisions, quickly adapt to customer needs and enable best in class repair. These ambitions, along with new GPDR legislation, prompted us to begin the journey to enhance our data capabilities and increase our data maturity. We recognised the opportunities of cloud infrastructure; scalability, resilience, agility and security and wanted to leverage PaaS solutions, benefit from PAYG cost-optimisation and focus on data innovation instead of managing infrastructure.
Although TalkTalk's connection with Microsoft goes back a number of years, when TalkTalk acquired blinkbox in 2015, it also inherited a long history of collaboration with Microsoft and use of the Azure platform. When we were ready to consider approaches to modernise our data platform, we naturally looked to Microsoft and its partners to help us. In 2016, TalkTalk joined forces with London-based Microsoft Gold Partner, endjin, to build an anomaly detection system for TV Set Top Box data using Azure Machine Learning. The solution processed 15GB/day and provided Power BI dashboards showing anomalies and trends.
Following the success of the anomaly detection project, the TalkTalk TV Data Team worked with endjin in early 2017 to build out the foundations of our modern data platform. Using Azure Data Lake, Data Lake Analytics and Data Factory, endjin quickly developed real time and batch data pipelines while introducing security best practices and DevOps processes.
The introduction of GDPR in May 2018 brings many welcome benefits for customers, but also presents significant challenges for our business around data lineage, deletion of data and tracking customer consent. We wanted to incorporate GDPR into the modern data platform at the very core rather than retrofitting the capabilities as an afterthought. With endjin's help, we prototyped PII masking APIs and data pseudonymisation approaches.
Using the momentum gained from endjin, the TV Data Team continued to nurture their Azure development skills; creating, deploying and managing new data pipelines. As their capabilities and delivery grew more consistent, predictable and efficient, they were asked to replicate and scale their approach across the wider business. As part of this, a recent organisational change has seen a number of TalkTalk data teams change their reporting lines into the Chief Architect to help the business put data at the forefront of our architectural decisions.
Once again, we're now looking to Microsoft to take us on the next stage of our journey; scaling the modern data platform, introducing real time and event-driven ingestion and processing and migrating on premise systems to the Cloud. New products and services are constantly being added to Azure and we look forward to the exciting opportunities that lie ahead.