![\"An](\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/01\/DataHeader.jpg\")
<\/p>\n<\/p>\n
With Azure Purview, Microsoft launched a unified data governance service that automates data discovery, cataloguing, building lineage, and classifying sensitive data, enabling users to get a holistic view of the data landscape. Today, Data Lake is a strategic investment if you\u2019re in a data driven organisation. Here\u2019s our shared experience on why Purview should be a key component in your Data Lake architecture.<\/p>\n
Most data professionals have worked with tabular data stored in databases because that was the way applications and reporting systems were designed or built.\u00a0 This meant there were only a finite number of tables\/databases to search for relevant information. With the advent of newer technologies such as IoT devices, and a plethora of web and mobile applications, data professionals must not only process and secure structured, semi-structured and unstructured data but also discover the pertinent data from the huge amount of data accumulated by the organisation to solve business problems.<\/p>\n
Many organisations are now moving to a more complex architecture that has data stored as flat files in storage such as Data Lake, unstructured data in non-relational and tabular data in relational databases. Managing the life cycle of data from creation to archival has been extremely challenging and requires massive investment from an organisation including training people, optimising existing or creating new processes, and adopting new technologies. Despite all these efforts, it remains the nightmare of many CDOs and CSOs. As Gartner suggests<\/a>, over 60 percent of challenges to the data management practice arise from data quality and supporting data governance and security.<\/p>\n Many of our customers have a petabyte-scale Data Lake, and it doesn\u2019t take long for data governance to become extremely difficult. The purpose of the Data Lake is to accelerate data projects by having the entire organisation\u2019s information in one place. The average time taken to identify and retrieve the relevant data from the Data Lake is somewhere between 5-6 weeks. We might have heard of the old adage \u201ctime is money\u201d, and it\u2019s true for our customers. The time taken at the beginning of the project to discover relevant information is unacceptable because the project team is to be paid during this phase, and the business gets frustrated as no progress is made on the project. In the meantime, the competitor would have shipped the latest product when the project team was searching for relevant information to kick start the project.<\/p>\n The user journey for data discovery and mapping can be summarised as shown below:<\/p>\n We just covered the complexities of a modern data estate, and the key challenge is the lifecycle management of data. Data governance is not just about data quality but also about metadata, policy, and identity management. Broadly speaking, we can divide the people accessing data into four categories:<\/p>\n At a high-level, metadata management provides the data about the organisation\u2019s data. For example, it provides the physical location of the data you are trying to find, identifies the different columns in a file and applies appropriate classification for the column within a data asset. Finally, it also provides the data lineage, which indicates the source for the information, the transformation applied on the source information, where it\u2019s getting used etc. The personas interested in metadata management are producers and consumers. The consumers leverage the search functionality to find the relevant data in a huge Data Lake or other data sources.<\/p>\n Policy and identity management is of interest to the data security officer persona. For example, they need to ensure finance information is accessed only by the finance department, define how long data should be retained for compliance purposes and so on. Our colleague, Pratim Das, has previously covered how to build and deliver a data strategy<\/a> in great depth.<\/p>\n You can get more from data sources with Microsoft Azure Purview. Azure Purview is a fully managed cloud service that serves as a system for registration and system of discovery for enterprise data sources. It provides capability that enables any users, from analysts to data scientists and developers, to register, discover, understand, and consume enterprise data sources.<\/p>\n A data catalogue lifecycle will register <\/strong>the various sources, discover <\/strong>the shape of the ingested dataset, understand <\/strong>and trace the data as it flows through intermediate layers, and finally enable analysts and engineers to consume <\/strong>those data by downstream applications.<\/p>\n The USP of Data Lake architecture is to break the problem of data in silos and gain valuable insights from TBs and PBs of data, which was not possible five years or even a decade ago. Not only that, a Data Lake expedites getting insights in near to real time. As different heterogenous data is ingested in the lakes \u2013 whether the traditional relational database, semi-structured JSON, XML-type data, HDFS-like data (Avro, Parquet etc) \u2013 it\u2019s important more than ever to have a central place of metastore.<\/p>\n The ability to connect to diverse data sources is also a vital capability. Azure Purview allows you to register not only cloud native storage and databases such as Azure Data Lake, Azure CosmosDB, Azure Synapse, but also on-premise databases, e.g. SQL Server etc. You can also logically group assets based on a project or an entire data asset in the organisation. An example is shown below:<\/p>\n Let us take the next step. You have TBs of data in your Data Lake connected to 100+ data sources. Is that good enough? No. Data lying undetected or unexplored is as bad as, if not worsethan, not ingesting large data in the analytics domain. Hence, the ease of discovering data empowers the end users or business users and this capability will determine the adoption of the Data Lake asset in an organisation.<\/p>\n In so many meetings, I have found even the most technology-minded decision makers use \u201csingle source of truth\u201d and \u201csingle version of truth\u201d interchangeably. <\/strong>But there is a clear difference.<\/p>\n Single source of truth <\/strong>means that the data comes from a specific source all the time. You have a CRM system running on SQL Server, then the data coming from that is the single source of truth. However, in some other scenario, say weather information, data is pulled from public datasets and also from the IoT sensor data that lands in your Data Lake. In such cases, there is no single source of truth.<\/p>\n Any transactional or temporal data system will change multiple times over its lifecycle, and it can upset an operation or delete an operation. The single version of the truth is to ensure you get the most consistent version, not necessarily the latest version, of the data that is available in the Data Lake.<\/p>\n Your organisation might be trying to solve the problem \u201cHow to get the single version of the truth on Data Lake\u201d, and the answer is in the lineage.<\/p>\n The ability to trace and track the data as it goes out of the data source and then goes through multiple layers in the Data Lake – usually raw, curated, enriched etc – allows an environment of transparency in the data organisation and encourages self-service analytics in the ecosystem. Every time the data is manipulated or transformed, the shape and the meaning of the data changes.<\/p>\n Azure Purview offers the lineage capability all the way from the source to the destination. This feature answers several questions from consumers when considering the data in end-user products.<\/p>\n Examples of these questions are:<\/p>\n A glossary acts as a glue between the business decision users and the technical engineers, and is a very common setup.<\/p>\n Take a hypothetical situation. You\u2019re talking to your parents and during your conversation you use the terms \u201cYOLO\u201d, \u201cFOMO\u201d etc. Even though these are English abbreviations, your parents may have never heard of them.<\/p>\n Similarly, in the IT world, if you\u2019re from an IT background and use the terms \u201cmaterialised views\u201d, \u201cdata factory\u201d etc, business users may not understand. Also, business users from the supply chain might use functional jargon such as \u201cAPS\u201d, \u201cB2C\u201d, \u201cMRP\u201d etc, and IT engineers may not have the knowledge to interpret it.<\/p>\n In addition, different terms mean different things, depending on context. \u201cAPS\u201d in the technology world means Analytics Platform System, whereas \u201cAPS\u201d in the retail space refers to Advance Planning and Scheduling.<\/p>\n A glossary solves this problem by enriching the catalogue, adding business terms and definitions and linking those terms with the datasets or Data Lake assets.<\/p>\n In Data Lake architecture, the idea is to cross pollinate the data from one business area with another to get a holistic picture. \u00a0The datasets contain raw and KPI driven measures that are relevant to a specific business function. However, cross-skilling SMEs, developers and analysts is not always possible. A glossary can act as one place to manage those business terms and give control to the data owner to publish the data and bring consistency across the various business units in the organisation.<\/p>\n Earlier, we spoke about how the data discovery phase takes more than 5-6 weeks. When Azure Purview was presented as a solution, it was looked upon with scepticism. With the reincarnation of Azure Purview, Microsoft has injected a lot of features into a genuinely stable productthat is here to stay. The scan of the entire Data Lake, which was over 1PB, took less than two days to complete. Customers can speed up this operation if the integration runtime is scaled from one to four. These scans can be run on a schedule and as frequently as required.<\/p>\n By default, Microsoft provides a list of classification to which the data is classified during each scan. Similarly, the glossary terms can be associated with assets discovered during the scan. Any existing documentation pertaining to the data on web pages or a SharePoint site can be linked to an asset via the glossary. Many of these activities are automated and can be used as soon as the Azure Purview is provisioned. You can also use the Apache Atlas API to extend any existing feature.<\/p>\n After the scan and tagging operation, one of our customers was able to complete the data-discovery journey for various projects in 1-3 weeks rather than 5-6 weeks per dataset. This is a marked improvement, as the catalogue setup itself is easy and is a PaaS service.<\/p>\n This product is just starting its journey outside private preview and has an exciting vision and roadmap.\u00a0 So please give us your feedback<\/a> to help us shape the product in future.<\/p>\nLinking to the real world<\/h2>\n
![\"The](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/Data-discovery.png\")
<\/p>\nWhat is data governance?<\/h2>\n
\n
![\"Data](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/Data-platform.png\")
<\/p>\nWhat Is Azure Purview?<\/h2>\n
![\"Azure](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/Azure-Purview-1024x572.webp\")
<\/p>\nThe role of a data catalogue in Data Lake architecture<\/h2>\n
1.\u00a0\u00a0\u00a0\u00a0\u00a0 Register<\/h3>\n
![\"New](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/new-scane-rule-817x1024.webp\")
<\/p>\n![\"Image](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/Group-assets-based-on-project-1024x541.webp\")
<\/p>\n2.\u00a0\u00a0\u00a0\u00a0\u00a0 Discover<\/h3>\n
![\"ContosoPurview](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/ContosoPurview.png\")
<\/p>\n3.\u00a0\u00a0\u00a0\u00a0\u00a0 Understand<\/h3>\n
Lineage \u2013 Single source of truth vs single version of truth<\/h4>\n
Single version of the truth<\/h4>\n
Why lineage is important<\/h4>\n
![\"Image](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/Sales-fact-1024x561.webp\")
<\/p>\n\n
4.\u00a0\u00a0\u00a0\u00a0\u00a0 Consume<\/h3>\n
![\"Image](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/Glossary-terms.png\")
<\/p>\nLinking to the real world \u2013 continued<\/h2>\n
![\"The](\"https:\/\/www.microsoft.com\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/12\/User-journey.png\")
<\/p>\nConclusion<\/h2>\n