{"id":48305,"date":"2021-04-15T11:09:43","date_gmt":"2021-04-15T10:09:43","guid":{"rendered":""},"modified":"2022-02-10T21:04:22","modified_gmt":"2022-02-10T20:04:22","slug":"creating-a-metadata-driven-processing-framework-for-azure-data-factory","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/","title":{"rendered":"Creating a Metadata-Driven Processing Framework For Azure Data Factory"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"attachment-full size-full\" src=\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2019\/06\/WindowsVirtualDesktopHeader.jpg\" alt=\"The Azure logo, with a drawing of Bit the Raccoon looking up at it.\" width=\"1920\" height=\"700\"><\/p>\n<p>This open source code project delivers a simple metadata driven processing framework for Azure Data Factory and\/or Azure Synapse Analytics (Intergate Pipelines). The framework is made possible by coupling the orchestration service with a SQL Database that houses execution batches, execution stages and pipeline metadata that is later called using an Azure Functions App. The execution batch, stage and worker pipeline structures allow; concurrent overall batch handling with inner stages of process dependencies executed in sequence. Then at the lowest level, all worker pipelines within a stage to be executed in parallel offering scaled out control flows where no inter-dependencies exist.<\/p>\n<p>The framework is designed to integrate with any existing set of modular processing pipelines by making the lowest level executor a stand alone worker pipeline that is wrapped in a higher level of controlled (sequential) dependencies. This level of abstraction means operationally nothing about the monitoring of the orchestration processes is hidden in multiple levels of dynamic activity calls. Instead, everything from the processing pipeline doing the work (the Worker) can be inspected using out-of-the-box Azure monitoring features.<\/p>\n<p><a href=\"https:\/\/mrpaulandrew.github.io\/procfwk\/overview.png\" target=\"_blank\" rel=\"noopener\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/mrpaulandrew.github.io\/procfwk\/overview.png\" alt=\"A simplified overview in an image of how the framework works.\" width=\"819\" height=\"461\"><\/a><\/p>\n<p>This framework can also be used in any Azure Tenant and allows the creation of complex control flows across multiple orchestration resources and even across Azure Tenant\/Subscriptions by connecting Service Principal details through metadata to targeted Tenants &gt; Subscriptions &gt; Resource Groups &gt; Orchestration Services and Pipelines, this offers granular administration over any data processing components in a given environment from a single point of easily controlled metadata.<\/p>\n<p>&nbsp;<\/p>\n<h3 id=\"why-use-procfwk\">Why use procfwk?<\/h3>\n<p>To answer the question of why use a metadata driven framework, please see the following YouTube video:<\/p>\n<p><a title=\"Why you need a metadata driven processing framework\" href=\"https:\/\/www.youtube.com\/watch?v=rVlc-GBpNnc\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" src=\"https:\/\/mrpaulandrew.github.io\/procfwk\/youtubeheader.png\" alt=\"YouTube Video\"><\/a><\/p>\n<p>&nbsp;<\/p>\n<h3 id=\"framework-capabilities\">Framework Capabilities<\/h3>\n<p>Below is a simplified list of profrwk&#8217;s capabilities &#8211; be sure to check out <a href=\"https:\/\/mrpaulandrew.github.io\/procfwk\/contents\" target=\"_blank\" rel=\"noopener\">the framework&#8217;s contents page<\/a> for a full explanation of each capability and how they can be used:<\/p>\n<ul>\n<li>Interchangeable orchestration services.\n<ul>\n<li>Azure Data Factory<\/li>\n<li>Azure Synapse Analytics (Intergate Pipelines)<\/li>\n<\/ul>\n<\/li>\n<li>Granular metadata control.<\/li>\n<li>Metadata integrity checking.<\/li>\n<li>Global properties.<\/li>\n<li>Complete pipeline dependency chains.<\/li>\n<li>Batch executions (hourly\/daily\/monthly).<\/li>\n<li>Execution restart-ability.<\/li>\n<li>Parallel pipeline execution.<\/li>\n<li>Full execution and error logs.<\/li>\n<li>Operational dashboards.<\/li>\n<li>Low cost orchestration.<\/li>\n<li>Disconnection between framework and Worker pipelines.<\/li>\n<li>Cross Tenant\/Subscription\/orchestrator control flows.<\/li>\n<li>Pipeline parameter support.<\/li>\n<li>Simple troubleshooting.<\/li>\n<li>Easy deployment.<\/li>\n<li>Email alerting.<\/li>\n<li>Automated testing.<\/li>\n<li>Azure Key Vault integration.<\/li>\n<li>Is pipeline already running checks.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h3 id=\"complete-orchestrator-activity-chain\">Complete Orchestrator Activity Chain<\/h3>\n<p>The following offers a view of all pipeline activities at every level within the processing framework if flattened out onto a single canvas. This applies regardless of the orchestrator type.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone\" src=\"https:\/\/mrpaulandrew.github.io\/procfwk\/activitychain-full.png\" alt=\"A view of all pipeline activities at every level within the processing framework if flattened out onto a single canvas. This applies regardless of the orchestrator type.\" width=\"6726\" height=\"4544\"><\/p>\n<p>&nbsp;<\/p>\n<h3 id=\"deployment-steps\">How to Deploy<\/h3>\n<p>For details on how to deploy the processing framework to your Azure Tenant see <a href=\"https:\/\/mrpaulandrew.github.io\/procfwk\/deployprocfwk\" target=\"_blank\" rel=\"noopener\">Deploying ProcFwk<\/a>, which covers the basic steps you\u2019ll need to take. Currently these steps assume a new deployment is being done, rather than an upgrade from a previous version of the framework. In addition, a reasonable working knowledge of using the Microsoft Azure platform is assumed when completing these action points.<\/p>\n<p>&nbsp;<\/p>\n<h3>Useful Links<\/h3>\n<ul>\n<li><a href=\"https:\/\/github.com\/mrpaulandrew\/procfwk\" target=\"_blank\" rel=\"noopener\">Check out procfwk on Github<\/a><\/li>\n<li><a href=\"https:\/\/mrpaulandrew.com\/\" target=\"_blank\" rel=\"noopener\">Visit Paul&#8217;s blog for more info on procfwk and other projects<\/a><\/li>\n<li><a href=\"https:\/\/docs.microsoft.com\/en-us\/learn\/browse\/?products=azure?ocid=AID3020565\" target=\"_blank\" rel=\"noopener noreferrer\">Get stuck into more Azure on Microsoft Learn<\/a><\/li>\n<li><a href=\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/03\/31\/new-azure-features-and-functionality-for-march%e2%80%8b-2021?ocid=AID3020565\" target=\"_blank\" rel=\"noopener noreferrer\">New Azure features and functionality for March 2021<\/a><\/li>\n<li><a href=\"https:\/\/docs.microsoft.com\/en-us\/learn\/tv?ocid=AID3020565\" target=\"_blank\" rel=\"noopener noreferrer\">View live and on-demand sessions on Azure and more on Microsoft Learn TV<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>This open source code project delivers a simple metadata driven processing framework for Azure Data Factory and\/or Azure Synapse Analytics (Intergate Pipelines).<\/p>\n","protected":false},"author":430,"featured_media":34758,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","footnotes":""},"categories":[594],"post_tag":[128,636,519],"content-type":[],"coauthors":[1688],"class_list":["post-48305","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technetuk","tag-azure","tag-data","tag-technet-uk"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Creating a Metadata-Driven Processing Framework For Azure Data Factory - Microsoft Industry Blogs - United Kingdom<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Creating a Metadata-Driven Processing Framework For Azure Data Factory - Microsoft Industry Blogs - United Kingdom\" \/>\n<meta property=\"og:description\" content=\"This open source code project delivers a simple metadata driven processing framework for Azure Data Factory and\/or Azure Synapse Analytics (Intergate Pipelines).\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/\" \/>\n<meta property=\"og:site_name\" content=\"Microsoft Industry Blogs - United Kingdom\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-15T10:09:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-02-10T20:04:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"450\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Paul Andrew\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Paul Andrew\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 min read\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/\"},\"author\":[{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/author\/paul-andrew\/\",\"@type\":\"Person\",\"@name\":\"Paul Andrew\"}],\"headline\":\"Creating a Metadata-Driven Processing Framework For Azure Data Factory\",\"datePublished\":\"2021-04-15T10:09:43+00:00\",\"dateModified\":\"2022-02-10T20:04:22+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/\"},\"wordCount\":543,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg\",\"keywords\":[\"Azure\",\"Data\",\"TechNet UK\"],\"articleSection\":[\"TechNet UK\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/\",\"url\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/\",\"name\":\"Creating a Metadata-Driven Processing Framework For Azure Data Factory - Microsoft Industry Blogs - United Kingdom\",\"isPartOf\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg\",\"datePublished\":\"2021-04-15T10:09:43+00:00\",\"dateModified\":\"2022-02-10T20:04:22+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#primaryimage\",\"url\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg\",\"contentUrl\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg\",\"width\":800,\"height\":450,\"caption\":\"The Azure logo, with a drawing of Bit the Raccoon looking up at it.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Creating a Metadata-Driven Processing Framework For Azure Data Factory\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#website\",\"url\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/\",\"name\":\"Microsoft Industry Blogs - United Kingdom\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#organization\",\"name\":\"Microsoft Industry Blogs - United Kingdom\",\"url\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2019\/08\/Microsoft-Logo.png\",\"contentUrl\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2019\/08\/Microsoft-Logo.png\",\"width\":259,\"height\":194,\"caption\":\"Microsoft Industry Blogs - United Kingdom\"},\"image\":{\"@id\":\"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Creating a Metadata-Driven Processing Framework For Azure Data Factory - Microsoft Industry Blogs - United Kingdom","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/","og_locale":"en_US","og_type":"article","og_title":"Creating a Metadata-Driven Processing Framework For Azure Data Factory - Microsoft Industry Blogs - United Kingdom","og_description":"This open source code project delivers a simple metadata driven processing framework for Azure Data Factory and\/or Azure Synapse Analytics (Intergate Pipelines).","og_url":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/","og_site_name":"Microsoft Industry Blogs - United Kingdom","article_published_time":"2021-04-15T10:09:43+00:00","article_modified_time":"2022-02-10T20:04:22+00:00","og_image":[{"width":800,"height":450,"url":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg","type":"image\/jpeg"}],"author":"Paul Andrew","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Paul Andrew","Est. reading time":"2 min read"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#article","isPartOf":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/"},"author":[{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/author\/paul-andrew\/","@type":"Person","@name":"Paul Andrew"}],"headline":"Creating a Metadata-Driven Processing Framework For Azure Data Factory","datePublished":"2021-04-15T10:09:43+00:00","dateModified":"2022-02-10T20:04:22+00:00","mainEntityOfPage":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/"},"wordCount":543,"commentCount":0,"publisher":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#organization"},"image":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#primaryimage"},"thumbnailUrl":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg","keywords":["Azure","Data","TechNet UK"],"articleSection":["TechNet UK"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/","url":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/","name":"Creating a Metadata-Driven Processing Framework For Azure Data Factory - Microsoft Industry Blogs - United Kingdom","isPartOf":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#primaryimage"},"image":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#primaryimage"},"thumbnailUrl":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg","datePublished":"2021-04-15T10:09:43+00:00","dateModified":"2022-02-10T20:04:22+00:00","breadcrumb":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#primaryimage","url":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg","contentUrl":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/06\/AzureThumb.jpg","width":800,"height":450,"caption":"The Azure logo, with a drawing of Bit the Raccoon looking up at it."},{"@type":"BreadcrumbList","@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2021\/04\/15\/creating-a-metadata-driven-processing-framework-for-azure-data-factory\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/"},{"@type":"ListItem","position":2,"name":"Creating a Metadata-Driven Processing Framework For Azure Data Factory"}]},{"@type":"WebSite","@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#website","url":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/","name":"Microsoft Industry Blogs - United Kingdom","description":"","publisher":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#organization","name":"Microsoft Industry Blogs - United Kingdom","url":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2019\/08\/Microsoft-Logo.png","contentUrl":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-content\/uploads\/sites\/22\/2019\/08\/Microsoft-Logo.png","width":259,"height":194,"caption":"Microsoft Industry Blogs - United Kingdom"},"image":{"@id":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/posts\/48305","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/users\/430"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/comments?post=48305"}],"version-history":[{"count":0,"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/posts\/48305\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/media\/34758"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/media?parent=48305"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/categories?post=48305"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/post_tag?post=48305"},{"taxonomy":"content-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/content-type?post=48305"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/wp-json\/wp\/v2\/coauthors?post=48305"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}