Category Archives: Uncategorized
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security Part 3
7. Lab – Azure Synapse – Auditing And welcome back. Now in this chapter I want to go through the auditing feature which is available in Azure Synapse so you can enable the auditing feature for an Aziosql pool in Azure Synapse. This feature can be used to track the database events and write them… Read More »
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security Part 2
4. Azure Synapse – Customer Managed Keys Hi, and welcome back. Now, in this chapter, I quickly want to go through the concept of being able to do encryption of your Azure Synapse workspace with the help of your keys defined in your Azure Key vault. So if you want to use Customer Manage keys… Read More »
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security
1. Section Introduction Hi and welcome to this section. When it comes to security now, security is a very important aspect and in Azure there are a lot of security aspects that are available for Azure based services. So when it comes to Azure data Lake we’ll be looking at role based access control, we’ll… Read More »
29. Delta Lake Introduction Hi and welcome back. Now in this chapter I just want to go through Delta Lake when it comes to Azure databricks and we’ll see a couple of examples on Delta Lake in data bricks itself. So with the help of a Delta Lake, you get some more features when it… Read More »
24. Autoscaling a cluster Hi and welcome back. Now, in this chapter, I just want to give some notes when it comes to auto scaling. So I’ve shown you that when you create your cluster in Azure databricks, you have this auto scaling concept. So when creating an Azure databricks cluster, you can specify the… Read More »
22. Lab – Azure Data Lake Storage Credential Passthrough Now in this chapter, I want to go through a scenario wherein you can actually make use of something known as your ad Credential pass-through. So earlier on we had seen that if we wanted to fetch data from a data lake gen two storage account,… Read More »
19. Lab – Getting data from Azure Data Lake – Implementation So in the previous chapter we had created our keyword. We create a key in the keyword. Now it’s time to create something known as a databricks scoped secret. So for that I need to go on to my Databricks resource. So here, here… Read More »
17. Lab – Structured Streaming – Streaming from Azure Event Hubs – Implementation So where did we leave off in the last chapter? So, we had run our notebook to read events from the Event Stream, but we were getting the body of our event in some sort of encoded format. So I have now… Read More »
12. Lab – Filtering on NULL values Now in this chapter, I just want to quickly show you again, how do you filter on null values? Again, something we had seen earlier on when it came to working with Azure Synapse, the Spark Pool. So just one thing that I want to confirm. Let me… Read More »
7. Lab – Reading a CSV file Now, in this chapter, we’ll see how to process our log CSV file. So, one of the most important aspects when it comes on to any service that we have seen. So, from the exam perspective, you should be able to understand how to go through CSV files,… Read More »
