{"id":53937,"date":"2022-01-14T12:40:30","date_gmt":"2022-01-14T11:40:30","guid":{"rendered":""},"modified":"2022-02-10T20:35:47","modified_gmt":"2022-02-10T19:35:47","slug":"cross-tenant-data-ingestion-without-whitelisting-ip-addresses-with-azure-synapse-pipelines-azure-data-factory","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-gb\/industry\/blog\/technetuk\/2022\/01\/14\/cross-tenant-data-ingestion-without-whitelisting-ip-addresses-with-azure-synapse-pipelines-azure-data-factory\/","title":{"rendered":"Cross-tenant data ingestion without whitelisting IP addresses with Azure Synapse Pipelines\/Azure Data Factory"},"content":{"rendered":"

\"An<\/p>\n

While general guidance suggests operating under the same tenant, there are instances where, for various reasons, it’s not possible. In this blog, we\u2019ll provide guidance for situations where organisations cannot operate under one tenant due to specific constraints or requirements. Such companies may operate independently and have their own Azure tenant, subscription, and potentially have deployed their resources in separate regions.<\/p>\n

While waiting for in-place data sharing to be available, the requirement is for Operating Company A (OpCo A) to ingest data from Operating Company B (OpCoB) into their Data Lake Gen2 storage for further processing by Azure Databricks or Synapse Spark pools. Security is top of mind for these companies, so the firewall is enabled in the storage account to manage access control for the public endpoint. The development team in OpCo A completed the implementation of their ADF pipelines to ingest data from OpCo B, using a self-hosted integration runtime and by whitelisting the IP address of the virtual machine in the ADLS Gen2 storage account set up by OpCo A. The team, however, did not get approval from the security team to move to production because whitelisting the IP address of the Self Hosted Integration Runtime in their ADLS Gen2 storage account is against their security policies.<\/p>\n

We\u2019ll detail the solution we recommended to OpCo A in the next paragraph, to securely ingest data from OpCo B without having to whitelist any IP addresses, and also ensuring that no traffic goes through the public Internet. We\u2019ll use Azure Data Factory to illustrate the approach, but the process is similar if you are instead using Synapse Pipelines. For this example, we assume you’re already familiar with Azure Data Factory or Synapse Pipelines.<\/p>\n

\"The<\/p>\n

Cross-tenant data ingestion with Private Links (No IP Whitelisting)<\/h3>\n

To ingest data across tenants without using a SHIR or whitelisting an IP address using ADF, you will have to create your Azure Data Factory with a managed virtual network (vNet) and create an Azure-managed integration runtime in that vNet. To access the source and target storage accounts you will need to create managed private endpoints to each storage account, as the firewalls of the storage accounts will block public network access. If you\u2019re using Synapse pipelines, you will have to create the Synapse Workspace in a managed virtual network.<\/p>\n

Create Azure Data Factory in Azure Manage Virtual Network<\/h3>\n

As mentioned above, you will need to enable the Azure managed vNet while creating your Data Factory. This needs to be done during its creation, as you will not be able to add a managed vNet to an existing ADF. To enable the managed virtual network while creating the ADF, check the \u201cEnable Managed Virtual Network on the default AutoResolveIntegrationRuntime” option in the Network tab.<\/p>\n

\"Enabling<\/p>\n

Create private endpoints to the source and target storage accounts<\/h3>\n

We assume that the Data Factory is in the same tenant as the source (OpCo A). Open your Data Factory and go to the \u201cManage\u201d hub and select “Managed private endpoints”. Select \u201cNew\u201d to create your managed private endpoint. You can also click on \u201cCreate Managed private endpoints\u201d.<\/p>\n

\"A<\/p>\n

In the new window, select Azure Data Lake Storage Gen2 and click on the Continue button. Enter a name for the private endpoint and provide the details of the source storage account before clicking on create.<\/p>\n

\"A<\/p>\n

After a few minutes, the status of the private endpoint creation will change to \u201cSucceeded\u201d. However, the approval state will show as \u201cPending\u201d until the owner of the storage account approves the private endpoint.<\/p>\n

\"A<\/p>\n

To approve the private endpoint, go to the Security and Networking section of the storage account and select Networking, then the Private Endpoint tab. Hit the refresh button if the private endpoint you created from ADF doesn’t appear with a pending status.<\/p>\n

Select the private endpoint and click on the \u201cApprove\u201d button.<\/p>\n

\"A<\/p>\n

After few minutes, if you refresh the managed private endpoint window, you will see that the \u201cApproval status\u201d has been updated to Approved.<\/p>\n

\"A<\/p>\n

The next step is to create a private endpoint to the target. The approach will be slightly different as the target storage account is in a different tenant.<\/p>\n

In the managed private endpoint menu, click on the New button. Enter the name of this new private endpoint and, in the account selection method, choose \u201cEnter manually\u201d.<\/p>\n

Provide the resource ID of the target storage account. You can find the resource ID by selecting the endpoint’s property in the settings section. The rest of the process remains the same as when the Data Factory and the storage account were in the same tenant.<\/p>\n

\"A<\/p>\n

Create the Linked Services and Authentication methods<\/h3>\n

Source Linked Service<\/h4>\n

There are no specific considerations for the creation of the source linked service, as it’s in the same tenant as the Data Factory. Refer to this documentation<\/a> if you need more details.<\/p>\n

Target Linked Service<\/h4>\n

The target storage account and the Data Factory are not in the same tenant. That means there are some considerations to make regarding authentication methods:<\/p>\n