DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security

1. Section Introduction

Hi and welcome to this section. When it comes to security now, security is a very important aspect and in Azure there are a lot of security aspects that are available for Azure based services. So when it comes to Azure data Lake we’ll be looking at role based access control, we’ll be looking at the access control lists.

When it comes to Azure synapse, we are going to be looking at aspects such as data masking, we are going to be looking at encryption. We’ll be looking at row level and column level security and other aspects which are important from an exam perspective. So let’s move ahead.

2. What is the Azure Key Vault service

Now, in this chapter, I just want to talk about the Azure Keyword Service. So we have seen this service earlier on when working with Azure databricks. There, in order to store the storage account access keys, we were making use of the Azure Keyword Service. We were trying to create a databricks scope wherein we were storing a secret. And there we made use of the Azure Key vault. And now in this section of security, I just want to explain what is the purpose of the Azure Keywall Service because we will also be seeing it in other videos as well. So the Azure Keyword Service is a managed service that is used for storing and managing the life cycle of your certificates, of your encryption keys and your secrets. So instead of a company actually investing in hardware or investing in software to maintain their certificates, their encryption keys and their secrets, they can securely manage all of this with the Azure Keyword Service.

So I’ll give you some examples on where you can actually use these different parts your certificates, your encryption keys, and your secrets. So normally, applications, when they want to connect onto a database, the application needs to establish a connection onto the database by using the database password. So one method is to embed the database password in the application itself. But this is not a secure practice because then in some way or the other, you are exposing the password of the database. So what you can do is that you can actually store the database password has a secret in the Azure Keyword Service. Then, when the application wants to connect onto the database, it will make a secure call onto the Azure Keyword Service, fetch the database password, and then connect onto the database.

Next, a use when it comes to your certificates, let’s say you have an application that wants to implement SSL that secure sockets. So it wants to ensure that all traffic goes over STPs. Then the application needs to make use of certificates. Now again, if you want to manage the lifecycle of your certificates, instead of actually maintaining this locally, again, you can actually make use of storing and managing the life cycle of these certificates in the Azure Keyboard Service.

And then finally, your encryption keys. Let’s say an application wants to encrypt data. In order to encrypt data, it needs to make use of an encryption key. Now again, ends up storing that encryption key locally in the application. It can actually call the encryption key or reference the encryption key in the Azure Keyword Service to encrypt the data. When it actually comes onto your Azure Virtual Machines. It can actually encrypt the data on the disks that are attached onto the Virtual Machines by using a customer manage key that is stored in the Azure Keyboard Service. So there are many scenarios in which you can actually make use of the certificates, the encryption keys and the secrets that are stored in the Azure Keyword service.

So I thought in this chapter, for those students who are not aware on the purpose of the service, I’ll explain what the service is all about. Obviously in this particular course we are only going to be looking at referencing whatever is required in the Azure Keyword service when it comes to security when it comes to data engineering on Azure. But I just want to give an upfront introduction onto the Asia Keyword service. We had already created the keyword earlier on and we’ll make a reference on to the same keyword whenever required.

3. Azure Data Factory – Encryption

Now, in this chapter, I want to go through encryption when it comes on to Azure Data Factory. So Azure Data Factory already encrypts Data address, which also includes the entity definitions and any data that’s cached the encryption is carried out with the help of Microsoft Manage keys. But you can also also define your own keys using the Azure Keyword service. Now, if you are using the Azure Keyword service, then you have to ensure that the Soft Delete feature is enabled and the setting of Do Not Perch is also enabled. By default, when you create the keyword, these settings are already in place.

And also you need to grant the Azure Data Factory the required permissions of get UnwrapKey and wrap key. So I’ll show you how you can actually perform encryption for Azure Data Factory. Remember, this is for encrypting whatever definitions and data that’s cached in Azure Data Factory. Now, here I have my Azure Keyword in place. Now, when it comes to encryption for Azure Data Factory, you have to ensure that there are no resources defined in Data Factory.

So I’ve gone ahead and create a new Azure Data Factory resource in the same region as my Azure Keyword. So if I go on to it, let me open up azure Data Factory studio. So this should be empty. You should not be having any data sets, any pipelines, any activities, no Linked service as well, if you have a link service, you have to delete that link service before you can enable encryption. This is if you want to enable encryption. Remember using customer managed keys. By default, the encryption is done using Microsoft Manage keys. But sometimes organizations have this security protocol in place wherein they have to manage the encryption keys. And in that case, they might be ensuring that they have the encryption keys defined in the Azure Key Vault. So here I have to go on to the Manage section. Here. I have to go on to customer manage key.

Now, before I add the Customer Manage key, I have to go on to my key vault and I have to give the required permissions onto my keyword for my new Azure Data Factory. So for that, I have to go on to Access policies. Then I need to add an Access policy here, in terms of the key permissions, remember, as per the slide, it was the Get permission. It was the unwrap and the wrap key permission. I will select my principal. So the name of my factory is Life Factory 1000. So I can search for that. So I can see I do have the identity in place. I’ll choose it. I’ll hit on select. I’ll click on Add. And here, let me click on Save. So we are ensuring Azure Data Factory is given the required permissions. Now I’ll go back onto Azure Data Factory and let me add the key. Now, here it’s asking us to directly give the Azure Keyword URL which also includes the key name and the key version.

That means we should be having a key already in place. So let’s define one. Let’s go on to the Azure Keyword service. I’ll go on to Keys here. I’ll click on Generate here. Just give the name and let me hit on Create. I’ll go on to the key. I’ll go ahead and click the current version. And here we have the entire key Identifier. Let me copy this onto the clipboard. I’ll go back onto a data factory. I’ll add the key here. I’ll paste in the entire key URL. I’ll hit on save. And once this is done now, the encryption is based on the customer manage key that you have defined in the Azure keyboard service. So in this chapter, I just want to show you how we can use a customer managed key when it comes to encryption for Azure Data factory.

• Category: Uncategorized