DP-203 Data Engineering on Microsoft Azure – Monitor and optimize data storage and data processing Part 5

13. Azure Data Factory – Pipeline Failures

Now in this chapter, I just want to give a quick note when it comes to your pipeline failure runs. Now, what I’ve done is I’ve created two pipelines, pipeline one and pipeline two. So if I go on to pipeline one here I just have a simple activity of setting a variable. So you can set a variable, a value for a variable variable that can be used later on in your pipeline. For that, firstly you have to choose your pipeline. So click on the canvas. And here you need to create first a new variable. So I create a variable of status of the type Boolean and I’ve given a default value.

Once you have defined the variable, then click on set variable the activity. And here you can go on to variables. And here you should be able to see your variable in place and then set a value. So that’s how you use the set variable activity. But again, the purpose of this is not to show the set variable activity but just a point when it comes to your pipeline runs. So here I have two activities in place and here in the set variable I’m setting the value of one which will actually result in an error because this is a Boolean type variable which accepts values of either true or false and not 10 or any sort of numbers. So this will actually fail.

Now, next I have another pipeline that’s pipeline two. Now, in pipeline two, what I’ve done is if I go on to general, there is something known as an execute pipeline activity. So I drag that onto the canvas. And here if I click on this in settings, I told please invoke pipeline one and wait on completion of pipeline one. So in a pipeline you could also execute other pipelines as well.

So now let me trigger this pipeline and then I’ll go on to the monitor section. And now here I can see that both my pipelines have failed. So my pipeline one would definitely fail because I made an error in the first activity itself. But then that also triggered pipeline two to fail as well. Now, the main reason for this is you might get a question on the exam that actually tries to ask you about pipeline failure runs. So when you have one pipeline embedded in another, what is the pipelines that will actually fail? So here in our use case scenario, pipeline two depends on pipeline one. And since pipeline one fail, pipeline two will also be marked as a failure.

14. Azure Key Vault – High Availability

Now, in this chapter, I just want to give a quick note when it comes to the High Availability for your Azure Keyword service. So we have seen the usage of this service in certain chapters. And when it comes to this particular exam, I have to give this note when it comes to the High Availability for this particular service. Now, when you create a keyword, so let’s say you create a keyword in the East US location. Now, the contents of the keyword are automatically replicated within the region and to a secondary region that is defined by something known as Azure paired regions. So there is a pairing definition or pairing table already in place where a primary region is paired onto a secondary region. So when it comes to east us, the pairing region is the West US location.

And here the data is replicated automatically from the East US location onto the West US location. So you don’t need to do anything on your own. This replication is already in place. So if the primary region goes down for any reason, then your request can actually fail over onto the secondary region. It will take a few minutes for the failover to take place. So if you had a net application, for example, that is referencing this key vault, if this region goes down automatically, your application can actually reference the vault in the secondary location.

The only difference is that during the failover process so during these few minutes, the vault will be in read only mode. That means you can’t create any new secrets, any new encryption keys. You can’t delete anything. You can read the keys, you can read the secrets, you can read the certificates, but you can’t make any changes. This is just during the failover. After the failover occurs, then you can make changes has desired. So, just a quick note when it comes to High Availability for your Azure Keyboard service.

15. Azure Stream Analytics – Metrics

Hi and welcome back. Now in this chapter I want to go through the metrics that are available for streamatics. So first is your backlog input events. So these are the number of input events that are backlog. So a nonzero value for this metric implies that your job isn’t able to keep up with the number of of incoming events. For this, one of the options is you could decide to scale the number of streaming units. Then you have the data conversion errors here. It could be that the number of output events that could not be converted onto the expected output schema. So for example, let’s say that the input, one of the values of the input, the type is not matching the column type in, let’s say a table in your dedicated SQL pool. Let’s say you are taking the data from the stream antics job from, let’s say as your event hubs and then streaming it onto a table in a dedicated SQL pool.

If there is a mismatch between the type of data you are trying to insert into the table, then you can get a data conversion error. You also have early input events. So these are events wherein the application timestamp is earlier than the arrival time by more than five minutes. Now, please note that I do have a chapter wherein I explain this time concept which is important for Azure Stream Attics. At this point in time, I just want to go through the metrics that are available. Then we have the late input events. These are events that arrive later than the configured late arrival tolerance window. Then we have out of order events. These are the number of events received out of order that were either dropped or given an adjusted timestamp based on the event ordering policy. Then we have something known as the watermark delay.

This is the maximum watermark delay across all partitions of all outputs in the job. Now, if the value of the watermark delay is greater than zero, this could be due to many reasons. This could be inherent processing delay of your streaming pipeline, or it could be because of the clock skew of the processing node generating the metric. Or if there are not enough processing resources in your stream antics job. Now, don’t worry, I’m going to go through some examples wherein we are going to explore these metrics or understand these metrics. So we’ll do this in subsequent chapters. At this point in time, I just want to let you know about the metrics that are available, the important metrics when it comes to the exam perspective.

16. Azure Stream Analytics – Streaming Units

Now in this chapter, I just want to quickly go through the streaming units concept. So when you define that stream at its job, you actually define the number of streaming units to assign to the job. So these streaming units will actually determine the amount of compute resources that are actually allocated to execute the job. To ensure low latency when it comes to stream processing, all of the jobs actually performed in memory. If the streaming units reaches 100%, then your jobs will actually start failing. So always ensure to monitor the streaming units that are being consumed. Now here I am just showing you a snapshot of the pricing page from the Microsoft documentation.

So here you can see the amount per hour when it comes to the streaming units. If I go on to our existing job, which is in the running state. If you go on to the overview, if you scroll down. So here in the mounting section. So here you can see the resource utilization when it comes to the streaming units. Here we can see that we are roughly around seven 7%. So no issues over here because we don’t have that many number of events that actually consume our stream antics job. But once you start having a lot of events so let’s say you have a million events that need to be processed at that point in time, you have to look at the resource utilization of your stream antics job.

17. Azure Stream Analytics – An example on monitoring the stream analytics job

Now, in this chapter, I want to give an example when it comes to the metrics. One of the metrics that we had seen in the Theory Chapter for the Azure Stream at X. So here I have Matic job in place. Here I am reusing one of the queries we had seen earlier. This query was actually being used to take data, data from the energy flow logs. Remember, these are the flow logs of my Azure Virtual Machine, which is flowing into my Azure Data Lake Gen Two storage account. And then from there we have the job that is actually processing these requests. Now let’s take this data and move it onto a table in our dedicated SQL pool. So, the first thing I will do is to actually create a table based on the data that we want to push in. So, I’m going to take the statement to create my table, so I have the time, the source IP, the source port number, the direction and the action.

So here in SQL Server Management Studio, let me right click on my dedicated SQL pool and hit on a new query so that we have this table in place. So I’ll execute this query. So we have the table. Let’s just have the select statement in place first. So this is done. Now, next I’ll go on to my outputs. And here let me add a new output. So I’ll choose Azure Synapse, give the name of the alias here. I’ll choose the authentication mode as the connection string. I’ll put in the username. I’ll put in the password here for the table name. So I can take this entire table name and I can place it here and click on Save. So it will add the output, it will test the output as well.

Once this is done, I’ll go on to my query. And here, let me scroll down. So I’ll put the into clause, and here I’ll put network flow. So I have the output here. Let me save my query. Then let me go on to the overview and let’s start the job. Now here I’ll choose a custom start time. So let me start in the past so that we have some data that will be inserted into our table. So here let me choose a past time of just an hour. And let me click on Start. So let’s wait till the job is in the start of state. Now, once a job is in the running state, so let me see if I now have data in my table. So here I can definitely see. Now I have data in my table. If I do? Select, count, star, hit on, execute.

So I have 588 rows currently in my table. Now here, if I scroll down, there is something known as the job diagram. So this job diagram actually gives you a good view when it comes to your entire flow in your stream antics job. Since we just started the job. That’s why we don’t have any information in place. So let me do one thing. Let me wait for around three to four minutes and refresh this page and then come back. Now I’ve just come back after some time, just a couple of minutes and I hit on Reset View. Now here I can see I do have some information in place. The first thing I’ll do is just to reduce the zoom. And here we can actually see the metrics either for the entire job or for each stage. You can see it for your input, or you can also see it for your output.

So let me just choose the output for now so that we have a better view. Here we can see all of the stages. If you hover around any stage, let’s say NSG, you will see what is the metrics at that point in time. Here we can see that we don’t have any sort of input events in place at this point in time. So again, let me go back on to my network flow. So here I can see initially when it came to the number of output events, you can see it was 588. So because we had started the stream antics job in the past 1 hour in the past, yet it had taken all of those events and then inserted it into our table in our dedicated SQL pool. So that’s when it came on to the output events.

Here you can see something now known as the watermark delay. At this point in time, when I hover around here, it’s around twelve. But if I go over, let’s say on to the last extreme point. So this is at this point in time, you can see the watermark delay is 204. So the watermark delay is increasing gradually. Now, why is this? So, as per our theory, this could be due to multiple reasons here. The main reason is because there are no input events that are coming in onto our stream antics job. So remember that this job is looking at the data that is coming in from. Remember that network Security group logs in a container in a Data Lake Gen Two storage account.

So if I go on to my Azure data lake, Gen Two storage account. So I’ll go back onto data lake. Yeah, this is the container when it comes to the network security group logs. So let me drill down into the container. So Microsoft Network network Security groups, my demo VM, let me go on to the 23rd, that is the latest day. And let me go on to the most recent R. That’s R is equal to eleven. Yeah, in terms of the minute that’s zero, zero and the Mac address. So here we have one file in place. Now, this file hasn’t been processed till date.

So if you go on to every hour, so let me go back on to the day that’s today 23rd, then I’ll go on to the R of ten. That was the last R. If I go on to the minute, so you can see that in every folder, in every R there is only one folder for the minute and that’s equal to zero zero. If you go on to that folder we have the Mac address and we have one file in place. So this was the file that was last processed. So mostly all of the file 88 events probably came from this single file. If I go back onto the day and if I go back onto the most recent R, so this file is being updated continuously by that feature of that Network Security Group flow log. So that feature so if I go on to another tab, if I go on to Network watcher, so if I go on to the NSG flow logs, so remember we had enabled this.

So now this feature is continuously writing onto that JSON based file and for this last file it will keep on writing to this particular file for the entire R till 04:00. It’s only after 04:00. Then it will create a new R. So it’ll be R is equal to twelve and again there will be another subfolder and then it will start writing onto that JSON based file and then as the Austrian antics will actually get access on to this particular file once the log has been released. So the reason I know that this is how it is working is because I done research on this, I had understood that this file currently it’s only one file and the log data is continuously being written on top of it.

That is why Azure Steam antics now has a watermark delay in place because no events are coming in onto the stream antics job. And the Azure Steam antics jobs needs to understand this. It needs to understand how the events are flowing because time is a very important aspect when it comes to a stream antics job. If you wait for an hour, if I go back on the job diagram, so we’ll see it is continuously increasing. When the R is actually complete, you will see a change in this particular diagram.

So let me come back after some time now I’ve come back after some time and now you can see there is a dip in the watermark delay. So this was the point in time at which the events were now taken in by the Steam antics job. So then there were some output events. So if I go on to seeing the number of rows now in my network flow table, so here I can see it has increased slightly. If I go on to my Insights folder, we will remember in R number eleven and now we can see R number twelve and here we’ll have a new file in place and again at the end of this duration, probably around 05:00 p. m. , then this file will be processed.

• Category: Uncategorized