{"id":9548,"date":"2023-02-02T09:00:00","date_gmt":"2023-02-02T17:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/insidetrack\/blog\/?p=9548"},"modified":"2026-04-08T06:54:26","modified_gmt":"2026-04-08T13:54:26","slug":"enabling-advanced-hr-analytics-and-ai-with-microsoft-azure-data-lake","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/insidetrack\/blog\/enabling-advanced-hr-analytics-and-ai-with-microsoft-azure-data-lake\/","title":{"rendered":"Enabling advanced HR analytics and AI with Microsoft Azure Data Lake"},"content":{"rendered":"\n
\"Microsoft<\/figure>\n\n\n\n

We periodically update our stories, but we can\u2019t verify that they represent the full picture of our current situation at Microsoft. We leave them on the site so you can see what our thinking and experience was at the time.<\/em><\/em><\/p>\n\n\n\n

We\u2019re on a mission to transform our human resources systems here at Microsoft. To make it happen, we\u2019re upgrading the way we use analytics and AI.<\/p>\n\n\n\n

Our digital transformation has been a twofold journey.<\/p>\n\n\n\n

First, we upgraded our core processes, providing efficient and effective self-service portals for our employees and powerful tools for our HR team using SAP SuccessFactors. Those processes include the nuts-and-bolts applications associated with human capital management (HCM): the employee portal, rewards, payroll, and other essential HR functions.<\/p>\n\n\n\n

With the core processes in place, our team at Microsoft Digital\u2014the company’s IT organization\u2014had everything they needed to revolutionize the data at the center of HR.<\/p>\n\n\n\n

The architecture we chose? Microsoft Azure Data Lake.<\/p>\n\n\n\n

Building a modernized HR data estate<\/h2>\n\n\n\n
\"Samuel,
The Microsoft Digital HR Data and Insights team, including (left to right) Johnson Samuel, Harsh Raj Singh Thakur, and Mithun Manganahalli Goud, were instrumental in implementing a new architecture for HR analytics and business insights.<\/em><\/figcaption><\/figure>\n\n\n\n

When data is scattered across disparate systems, it\u2019s difficult to provide agility, insights, and advanced analytics through AI. In today\u2019s world of big data and predictive intelligence, these capabilities aren\u2019t just a luxury. They drive talent conversations, workforce planning, and an improved employee experience that affects business outcomes.<\/p>\n\n\n\n

But when an enterprise\u2019s data is siloed or fragmented, those outcomes are out of reach.<\/p>\n\n\n\n

\u201cWhat happens when you don\u2019t have a modern data architecture?\u201d asks Harsh Raj Singh Thakur, principal software engineering manager on the MDEE HR Data and Insights team. \u201cYou have a tedious and drawn-out process before you can retrieve your metrics. It\u2019s a cumbersome task, it\u2019s expensive, it\u2019s not easy to maintain, and there\u2019s a lot of cost to get it all done.\u201d<\/p>\n\n\n\n

To make HR insights more accessible and insightful, we first had to assemble a unified and accessible data estate. Our SAP SuccessFactors implementation<\/a> for core HR processes helped lay the groundwork by streamlining external and operational data to make them more organized and available for processing.<\/p>\n\n\n\n

With modern core processes in place, Microsoft Digital engineers could turn their attention to data.<\/p>\n\n\n\n

The journey to data transformation<\/h2>\n\n\n\n

Like all large-scale transformations, this one involved a great deal of complexity and multiple touchpoints. Microsoft Azure Data Lake provided the modern analytics platform that would not only enable the team to ingest, store, transform, and analyze the data, but also deliver simpler data discoverability, maintain data security, and ensure compliance.<\/p>\n\n\n\n

HR Data Lake business coverage<\/h3>\n\n\n\n
\"A
The HR Data Lake\u2019s business coverage delivers value across Microsoft\u2019s entire people analytics ecosystem, from employee-facing, self-service utilities to large-scale, future-oriented planning.<\/figcaption><\/figure>\n\n\n\n

Unifying the data<\/h3>\n\n\n\n

Considering the wide array of HR systems at Microsoft, it was important to bring all the data together to give HR an end-to-end view of the employee lifecycle and the moments that matter in an employee\u2019s journey. At the same time, the team took efforts to reduce redundant data copies across the enterprise.<\/p>\n\n\n\n

\u201cEnabling connected insights which are trusted and secure through a modern data platform in Azure Cloud was a key goal as we set out to drive the digital data transformation in the HR ecosystem,\u201d says Johnson Samuel, principal group engineering manager for Microsoft Digital\u2019s HR Data and Insights team.<\/p>\n\n\n\n

Multiple systems make up the HR ecosystem: Employee Central for core HR, iCIMS for applicant tracking, listening systems, rewards, CRM, employee learning, and more. While each of these systems serves an important purpose, the potential to unlock insights by unifying all of their data is immense.<\/p>\n\n\n\n

\n

\u201cThe ease of use from actually having everything collocated in an Azure Data Lake makes it easy to build out connected insights,\u201d Raj Singh Thakur says. \u201cIt\u2019s the foundation of our modernization journey.\u201d<\/p>\n<\/blockquote>\n\n\n\n

Azure Data Lake Storage Gen2<\/a> serves as the common storage layer, which ingests data through Azure Data Factory<\/a>, messaging systems, and other sources. By properly defining storage structures and models, the team had made the first step toward a more modern data platform.<\/p>\n\n\n\n

Expanding the data footprint with new metrics and scorecards<\/h3>\n\n\n\n

Ever-increasing volumes of data illustrated the need for advanced analytics. They were no longer a choice\u2014they were a necessity.<\/p>\n\n\n\n

\u201cThere are many lines of businesses within HR, like Global Talent Acquisition, Talent and Learning, and HR Services who manage HR operations,\u201d Samuel says. \u201cWe\u2019ve enabled new capabilities for each of these different HR functions.\u201d<\/p>\n\n\n\n

Key metrics across the ecosystem include the recruiting funnel, workforce, headcount, employee engagement, learning and development, and other functions across HR. The analytics apparatus uses a combination of Azure Synapse Analytics<\/a>, Azure Analysis Services<\/a>, and Power BI Shared Datasets<\/a>, while Microsoft Power BI<\/a> is responsible for visualization.<\/p>\n\n\n\n

This powerful combination of technologies helped build complex analytics and drove consistency across teams. It also unlocked the ability to bring disparate metrics together to help determine correlation and causation between different factors.<\/p>\n\n\n\n

Data governance<\/h3>\n\n\n\n

Next, the team needed to ensure that engineers and end users could access data in the lake safely and securely. Good governance keeps data access compliant because users can only request information that\u2019s relevant to their roles. Driven by the HR Privacy team and enabled by a home-grown security and governance platform, Microsoft Digital established column-level security (CLS) on the Data Lake.<\/p>\n\n\n\n

\u201cWhen an HR team requests data, they get access to only the specific data set,\u201d Raj Singh Thakur says. \u201cSo if you\u2019re looking for an employee\u2019s name and alias but your role doesn\u2019t require you to know their salary, gender, or other aspects of their identity, you won\u2019t get access.\u201d<\/p>\n\n\n\n

This approach makes sure we respect our employees\u2019 privacy and that we comply with local laws that regulate how we use our data. Data governance also includes data discoverability, quality, and lineage functionality, which the team established through Microsoft Purview<\/a> and in-house solutions to support more complex scenarios.<\/p>\n\n\n\n

Modern engineering<\/h3>\n\n\n\n

Microsoft Digital also developed key platform capabilities that ensure high-quality and trustworthy data across the estate and drive engineering efficiency.<\/p>\n\n\n\n

\"Klinghoffer
Modernizing our data architecture is expanding what the company\u2019s HR teams can do, says Dawn Klinghoffer, vice president of People Analytics at Microsoft.<\/em><\/figcaption><\/figure>\n\n\n\n

Whether the metric is headcount, performance management, employee learning, or any other area, each of them follows the architectural pattern of a Data Lakehouse, a system where all information resides in the Data Lake, without the need to build separate data marts. It allows our engineers to scale storage and compute independently for greater efficiency.<\/p>\n\n\n\n

Between telemetry dashboards that help engineers understand system health and continuous optimization across code and infrastructure, this new architecture has helped save significant Azure costs\u2014a reduction of around 50% over two years. Meanwhile, enabling agile development and DevOps is helping the team deliver iteratively and realize business value faster.<\/p>\n\n\n\n

But the real value lies in the insights that unified, normalized data empowers.<\/p>\n\n\n\n

\u201cWe\u2019ve normalized the data by leveraging a company-wide taxonomy that we can use across other projects very easily,\u201d says Mithun Manganahalli Goud, principal software engineer on our HR Data and Insights team. \u201cSo from a data-delivery service standpoint, we can provide information to a wide range of downstream systems and data consumers.\u201d<\/p>\n\n\n\n

Building a platform for the future<\/h3>\n\n\n\n

While the new architecture is actively meeting current reporting needs, Microsoft Digital also looked toward the future.<\/p>\n\n\n\n

The platform is capable of enabling deep insights that leverage machine learning. While today\u2019s focus is on descriptive and diagnostic functions, the team is working toward predictive and prescriptive analytics through AI and machine learning.<\/p>\n\n\n\n

\n

\u201cWe\u2019ve created a rich content system where we can manage emerging requirements with the current data and metadata, so it\u2019s future-ready,\u201d Manganahalli Goud says. \u201cWe already have the process in place, so we won\u2019t have to go back and reinvent the wheel.\u201d<\/p>\n<\/blockquote>\n\n\n\n

When our HR team takes the next step into AI-driven insights, the foundations will already be in place.<\/p>\n\n\n\n

Driving human-centered innovation with Microsoft Azure Data Lake<\/strong><\/strong><\/h2>\n\n\n\n

Our modernized data architecture has enhanced the HR teams\u2019 capabilities. Better data immediacy means data pulls that used to take 24 hours now get done in a fraction of the time\u2014around four to six hours. Similarly, the time it takes to enable self-service access for bring-your-own-compute data processing is rapidly falling.<\/p>\n\n\n\n

But the most powerful outcomes are the cross-category, cross-disciplinary insights that unified and accessible data provides for HR leaders.<\/p>\n\n\n\n

\n

\u201cOne of the most unique and forward-thinking outcomes is that we’ve been able to combine qualitative with quantitative data,\u201d says Dawn Klinghoffer, vice president of People Analytics at Microsoft. \u201cWe’re able to create data models with our survey information as well as more quantitative data like attrition and diversity, then combine them in an aggregated, de-identified way to understand broad insights.\u201d<\/p>\n<\/blockquote>\n\n\n\n

For example, by combining sentiment data with de-identified calendar and email metadata, we\u2019ve been able to quantify the impact of blocking focus time on employees’ perception of work-life balance.<\/p>\n\n\n\n

\"Pelland
Focusing on self-service gives HR practitioners important flexibility, says Patrice Pelland, partner group engineering director for Microsoft Digital.<\/em><\/figcaption><\/figure>\n\n\n\n

\u201cMaking data available to all people in a self-service, consumable way gives them the opportunity to ask the questions they don\u2019t even know they have,\u201d says Patrice Pelland, partner group engineering director for Microsoft Digital. \u201cThe more people interact with the data, the more it will lead to deeper questions and better insights to drive their business or Microsoft as a whole.\u201d<\/p>\n\n\n\n

Those questions and insights have already led to human-centered improvements and innovations. One example is the wide adoption of team agreements that empower employees to collectively self-determine the work modes that serve them best. HR\u2019s work has even informed some of the \u201cnudge\u201d product features for employee experience tools like Microsoft Viva<\/a>, which recommends focus blocks to improve productivity and overall work-life balance\u2014a metric that\u2019s currently on the rise across Microsoft.<\/p>\n\n\n\n

Ultimately, the more people who have access to high-quality, trustworthy data, the more we can provide a world-class experience for all employees.<\/p>\n\n\n\n

\n

\u201cThere’s a lot of envisioning based on the services that we’ve been building that people didn’t even think could exist,\u201d Pelland says. \u201cWe’re building the foundational layers to offer things that will be truly transformational for the HR business. Whatever size your organization is, and whichever HCM you use, with Azure, you can do what we\u2019re doing right now.\u201d<\/p>\n<\/blockquote>\n\n\n\n

\n
\"Key<\/figure>\n\n\n\n

Here are suggestions for using Microsoft Azure Data Lake to do more with your HR data at your company.<\/p>\n\n\n\n