AZ-700 Microsoft Azure Networking Solutions – Overview of Monitoring Networks

1. Overview of Monitoring Networks

In this section of the course, we’re going to cover monitoring. Now, within Microsoft Azure, as you can imagine, we don’t have the same exact tools that you would have in a physical network to monitor the packets, to do deep packet inspection. But it’s actually quite interesting how Microsoft is able to provide us some tools to watch traffic going back and forth.

And so in this section we’re going to cover things such as alerts and logging. We’re going to talk about connection monitor, we’re going to talk about network watcher and some of these issues around monitoring a network. Now Microsoft Azure has a tool called Azure Monitor. That is their central dashboard for monitoring all sorts of things. It is represented by this speedometer or gauge if you will. Or you can go up to here and just start typing Monitor and you will see it in the list of services.

Now Azure Monitor is again a centralized dashboard and so it does contain a lot of the logs for virtual machines, storage accounts, containers, but there is a networking tab. In order for data to be pulled into Azure Monitor, you do have to do some configuration. So if you’re experiencing issues with your network, it’s probably not enough just to go in and start examining the logs because if you haven’t configured them in the first place, then you’re going to have to then turn those things on in order to start collecting data.

So we’re going to go into Azure module, we’re going to skip right into the networks section. I’m going to minimize this left panel here so we can see more. Now you can see here we have a bit of a dashboard showing the various things that are network related. I still have my application gateway running and it is green.

There are also firewall policies, network interface cards for all the various VMs, the virtual networks themselves, IP addresses, network security groups. We can sort of see a dashboard here. Now I don’t have any alerts configured and so obviously there’s no alerts that have been triggered. Now your view might be different than mine because I’m only seeing the resources that are active on my account and you might have more resources or fewer resources.

And so you’re going to see different things. If you click on any of these, you get sort of a more focused view where you can see the application gateway. In this particular case that we have running. Now, the reason that I can see my application gateway is I have the diagnostics set up and we’ll look at that when we talk about setting up diagnostics. If I didn’t set up diagnostics, I wouldn’t be pulling data from it, obviously. Now you do have the ability in a very busy subscription to search for resources.

And so if you wanted to filter based on that, you could. You also have these filters at the top where you can filter on specific subscription a specific resource group so that I can filter on my AZ 700 related resource groups and by type, etc. To so if this does become too busy, you can use this interface to trim down the amount of information being shown. If we switch over to the connectivity tab, we haven’t created yet a connection monitor and so we’re going to have to do that in order to see data on this tab. The third tab here is traffic. And traffic traffic is going to be based on what are called flow logs and traffic analytics. And so we’re going to right now I’ve been focusing mostly on the West US region. We’re going to dive into this in this section and look at how traffic is analyzed using Azure monitor.

2. Enable Network Diagnostics

So before we can actually monitor things, we actually have to start collecting data. And so the way we’re going to do this, I’m going to open up this menu again for Monitor and we’re going to go down to Diagnostic settings under Settings to make this simplified. This is all of the resources on my account I’m going to filter first on the AZ 700 course resource group. Now we can see all of the resources sources that are relating to this group. Yours could be different, obviously. And the only one that has Diagnostic settings enabled is the application gateway. And so when I was setting up the application gateway, I actually turned on Diagnostic settings. I consciously did that. And we can see here that there’s a diagnostic settings called Service and it stores its data in a storage account. And so that has been connected. In fact, if I go back up to the application gateway under Diagnostic Settings, we’re going to see the same view here where we can see the diagnostic setting name, storage account. I’m going to say Edit setting.

And we can see that for each resource it’s going to be different. In the case of the application gateway, we are keeping all of the access logs, the performance logs and the firewall logs for 30 days. This is being stored in a storage account that was created specifically for this purpose. I am not keeping the performance metrics. And so in terms of minute by minute look at to how it’s performing, I’m not keeping that. But in terms of access logs and firewall logs, I am keeping that. Now, besides the storage account, there is a thing called a Log Analytics Workspace, which also can be pulled into Azure Monitor. Okay, so that’s the second option. I can also stream this. So if I have a solution listening for these things, I can hook it up to an event hub and then have something on the other end that is in real time doing something with this data.

And recently Microsoft added these third party partners that can handle some of these analytics and performance analysis. So this is fine. So for this application gateway, it’s archiving and stuff to a storage account. Let’s go back to Azure monitor into diagnostic settings. Once again. I’m going to filter on the group that matters and we’re going to see there’s a bunch of other things that don’t have their diagnostics enabled. Now the Virtual Machine has its own diagnostics, but the Network Interface Card is a separate resource for the Virtual Machine and it does not. And so if I wanted to get some data and information from the Network Interface Card, I go into it and I see Add Diagnostics settings.

So this is the Nick diagnostics one. I got to spell that right. And the only option I have for the Network Interface Card is the metrics. So I’m going to click on that and I can send this also to the same storage account, right? And I can have a retention policy. Let’s say I also want only 30 days worth of metrics. Maybe that’s even excessive. I can do 14 days worth of metrics. I’m going to send that to the same storage account as my Azure application gateway and then I can click Save. Now, this isn’t retroactive, so I can’t go back in time to start pulling data. But going forward, the network interface card is now going to start sending its metrics data into the storage account. And then we can start maybe doing an alert on it or looking at some charts so we can graph the performance to specific network interface cards.

All right, so I’m going to select all again, so we can see here that the virtual machines themselves are not actually in this list. So the virtual machine scale set that we are using as the back end of the application gateway is not in the list. Neither are the virtual machines that we were testing earlier in the course. We can have the network interface cards, we have the IP addresses, security groups, but we don’t have the virtual machines themselves. So the way to get access to that is we go under the Virtual machine tab of monitor. Now, it’s probably beyond the scope of this course to go through monitoring of virtual machines or monitoring of storage accounts. But this is where you would go. You could find that there are six VMs on my account that are not monitored and I can enable them from here. Some of these VMs aren’t even running, so I can’t enable them.

The reason why is because it actually has to install some software on the VM which is called an agent. And the agent is what reports the data back to Azure Monitor. So I’d have to start the VMs and then enable monitoring in order to get the data storage accounts. Similar thing where we can sort of see the storage accounts we have and if we’re not getting the data from them, we can play with the monitoring. But again, that’s a bit outside the scope of this course. So I’m going to go back to diagnostic settings here, just for some variety. I’m going to turn on monitoring at the IP address. I can see the application gateway has a public IP. It is not enabled. So I can go in here, add diagnostics settings, IP diagnostics, and you can see the distributed denial of service attack protection. So Microsoft Azure comes with a basic level of denial of service attack protection.

And so we are going to be able to see some of this from here. Now, I’m going to put this into what is called a log analytics workspace that is different than a storage account. Storage account. We can actually download the files. These are stored in file form, download them to our local, view them in Notepad, use them for other purposes. Log Analytics Workspace is basically a data storage that’s specific to Azure monitor here. And then we can use these logs, we can query them, we can run reports against them, etc. You so there’s sort of a different use case for Log Analytics. So I’m going to store, for the difference sake, all the IP data into Log Analytics Workspace and click Save.

And then I’m going to go into the Network Security group and similarly we’re going to enable diagnostics at the NSG level and we’re going to get whatever they have here, put that into the workspace as well. So hopefully at this point forward we’re collecting diagnostics that from the network interface card, the NSG, the IP, IP address and the application gateway itself. And we can then start to see some of this data coming in. We can run various reports.

3. Connection Monitor

So before we move on, I want to show you a pretty cool thing. If we go under Network Health, there’s this little diagram icon on the left of the application gateway, name called Dependency View. And if you click it, it’s a whole new window. We can see the metrics for it. I’m going to close that out. You can actually see a visual representation of your application gateway. We have two listeners, two rules, and one back end. Obviously, if you had multiple back ends and multiple rules, you could see lines showing one path versus the other path. Then it becomes interesting too, as you can add filters and search for things like so if I did search for rule one, it then removes everything else. I can see that single path.

So in order to debug your complicated application gateway, if you’ve got multiple applications, multiple listeners, multiple rules, multiple front ends, you can start to simplify it in this visual diagram. I thought that was pretty cool, and I just discovered it. So close that out. I want to talk about connections. So there is this connectivity tab, and there’s no results here because we don’t have a connection monitor. So connection monitor is part of the overall broad network watcher set of tools. And so I’m going to get out of here and I’m going to look for Network Watcher. So now we’re getting into some serious monitoring of our connectivity. And this is one of the impressive things about Azure and the Cloud to me, is that they’re able to allow us to inspect the traffic between resources that are basically allows us to see our traffic and blocks out anyone else’s traffic and vice versa.

So think about the trillions of network connections that are happening per day, and somehow they’re not leaking data all over the place. This is actually amazing. You can see Connection Monitor here on the left. There’s a classic connection monitor. Well, that’s being deprecated. You have until you can’t create any more of these until after July 1. So it’s already passed. You have until 2024, beginning of 2024, to migrate any of your old connection monitors to the new one. So we’re going to go under the new Connection Monitor, and we see that we don’t have any.

So we want to set up basically a monitor of this connection, of these resources. So I’m on the network watcher connection monitor tab. I want to say create. Like everything. You give it a name conman is as good as any to me. I choose it for my own subscription. And you want to make sure you’re putting this in the same region that you have your resources, so you can monitor where that is. Now it’s going to basically create its own workspace for monitoring this data.

You could force it to use one, or you could allow it to create its own. This is a default workspace that I’m going to let it use so we’re going to say next, now this is where we would set up any test groups if we wanted to do that. So we’ll give it a name called Test Group One. And the sources are going to be sort of these are the machines and they can be in Azure, Azure Virtual Machines within Azure. Or you could even use on premises servers as your sources. So once again, we’re installing software which are called agents onto these machines. And they’re going to be giving us their connectivity data.

And it doesn’t have to be even specifically within Azure. So we’re going to say add sources. And we can see here that we’ve got the choice depending on the level that we want to add, we can add the virtual network level as a source. We can add the subnet level as a source. We can add any of those machines that are installed on the subnet as sources. So I can choose the whole front end. So you’re seeing here an error message saying that we have to install the extension in order to enable these machines. That makes sense. Like I said, you do have to install software in order to use this connectivity. So what we’re going to do is I’m going to open the portal in a new tab so we can do two things at once here. Go into this LB server.

One, it is running. And what we’re going to do is go down into extensions and we’re going to have to add the network watcher extension. So as I view it here, I can see there’s a number of extensions to choose from. And what I’m looking for is the network watcher extension. So there’s network watcher agent for Windows. So this is the extension that we’re going to install on this machine. So it’s going to go off and do that. All right, so the networking watcher extension was successfully installed. Now I’m going to go, I should have done this while I was doing this, but I’m going to go and turn on this virtual machine and I’m going to install it as well there. The reason I’m doing this is because to do a connectivity test, I’m going to see if one virtual machine can talk to another.

And that’s going to be the check between the connectivity is from A to B. And I need both an A and a B to be able to do that. So I’ve added the network watcher agent to two of my virtual machines. And now what we can do is we can set up these machines as being tests, right? So I can go to, for instance, this is the AZ 700 VM, which is a public facing website, as the source. And then I can basically set up the other machine as the destination. And then we can sort of set up a test between them to see what is the connectivity between them. And that becomes a thing that we can report on and we can do alerts on and things like that. So I’m going going to pause the video. When we come back, we’re going to finish setting up a connection monitor between these two endpoints.

4. Testing a Connection Between Endpoints

So now we’re ready to add a test configuration to basically run a test between our sources and our destinations. So we’ll give it a name. I’m going to call it Http test. Now the test is going to be in the Http protocol. We could go down to the TCP or ICMP, which is the Ping protocol. But that layer seven Http protocol is fine for our purposes. Http runs on port 80. Now I can choose how frequently I want this test to occur between these two endpoints. I’m going to choose every minute. I’m going to call a get method up to the route, and I’m expecting a 200 status.

Okay, back now depends on your application and your situation. Now, what percentage of failures am I willing to accept in order to continue to think this is successful? I mean, I think 25% is a fairly high number. 10% of failures is, I guess. Okay. And the other thing you’re testing is the amount of time it takes, 90 milliseconds being this. So either of these fails, it’s a failure. So I created the test and now I want to create an alert based off of this test.

And so if the failures are more than 10% or it takes more than 90 milliseconds to do the connection, I want to be notified. So I can say create alert, select action group. I do have to create a new action group, and one of the prerequisites is to enter this email address. And I can just for the sake of this test, I’m going to enter an example email address.

Maybe it’s better to say example@gmail. com, assuming nobody owns that. So I can say create action group. And so this email address will be notified if the connection was to ever go down. So it’s a pretty straightforward it’s just going to notify me when there are failures in this connection monitor. So I can say review and create. Everything looks good. So I can just say create. See, it estimating me a cost of $0. So it’s going to create, first of all, the connection monitor, and then it’s going to create the alert to say when something failed. Now I gave it a couple of minutes, and we can see that right now it takes three to five minutes for the first set of monitoring.

Remember, I’m running this every minute. And when I hit refresh, I can see all right away that I’m already passing, that those two machines can connect with each other. And I’m within the 90 milliseconds. And the web page on port 80, the root is being retrieved. So if we go back to Azure monitor under the Networking Settings under Connectivity tab, we can see that we are no longer saying that there’s no connection monitor. Now we have a connection monitor, and any alerts will show up here on this tab. So we’ve successfully created our connection monitor, and now it’s successfully monitoring the connection between those two machines. You can obviously set this up to monitor all of your machines connected to each other and various setups.

5. Traffic Analytics

So let’s look at the traffic tab. We’re in Azure Monitor under networks and we’ll click over to see traffic. Now Azure Monitor does know that we have those two regions, west U s. And West US two that has a couple of NSGs in each. But we can see that the flow log configuration has got a red X on it and it’s telling us that it’s not currently enabled. In fact, if I click one I can to see that it says disabled here. So we’re going to have to enable the flow log. So let’s get out of here and go back to the home page and we’re going to go into the resource group and we can see the network security group from here. I’m picking one called LB Server one. I was using it for testing earlier and I’m going to go into NSG flow logs and say Create flow log. It’s already prefilled in the information in terms of the subscription. The name of the flow log has a predefined value and I have to answer how many days do I want to retain this traffic? And I think about 14 days is about right.

Well keep the default, which is version two. But I actually do want to enable this traffic analytics. So traffic analytics is what’s going to allow me to run some reports on it, geo mapping, hotspots, things like that. For the purpose of this demo, I want to get traffic analytics pretty frequently. So I’ll say every ten minutes. And this is getting fed into in log analytics workspace. So we can just go ahead and say review and create and click Create and we’re going to then start collecting traffic logs from that network security group. We’ll let that run, it shouldn’t take too long. So what I’m going to do is I’m going to go to the web server, I’m going to grab the IP address and I’m going to open it up basically in the web browser and I’m going to send some traffic to it.

So I’ve actually turned on the flow logs for both the LB Server one and the VM from the front end. So we’ve got two flow logs running. And even just for some extra traffic, I’m going to RDP into one of the machines. Because RDP is also traffic that travels over the network security group, it has to be allowed to come in. So let me just log into RDP quick and that will count I guess as traffic being logged by this flow log. Now I don’t need to actually do anything, I’m just again creating traffic. So if we go back to Azure Monitor under networks and this time when we switch over to the traffic tab, we’re going to see something different. We’ll actually see some green under West US and that the two of the three VMs are now logging traffic. We can see the green check marks being positive signs for that.

Now if we go back to Network Watcher and scroll down to traffic analytics, we’re going to see that it actually does take some time for this data to get analyzed and collected. So it’s going to come up with a little message saying it’s going to take 20 to 30 minutes. So let’s give that some time. So if we come back more than half an hour later and go under logs, traffic analytics, we can remove this notification. But we can see we have quite the dashboard here. If I scroll down a little bit, it’s breaking out port numbers, NSGs, the machines themselves that were involved, and basically some information about how my environment is set up, where my networks are, the deployed regions, the NSGs subnets, all of this sort of dashboarding information. Now we could obviously filter if we wanted to. We could say, well I’m not interested in some of this stuff. I’m really interested in just those two resource groups. I can filter based on so many hours, so many days, et cetera. Now obviously these machines are not used. I’m just recording some demos here. So fifty seven k in, inflows, 4. 5K outflows. We’re not talking about big amounts of data here.

There’s a couple of malicious requests here, two blocked malicious requests. Click to go to log search. Now we haven’t really talked about log analytics and log search too much, but this log search uses this format called Acoustic Query. It’s similar to SQL in terms of doing searches, but it’s basically got a log of data and then you can just run some where clauses and select statements and group stuff together. So we can see, well, a couple of IPS I don’t recognize that somehow had found my public IP address and were attempting to do something. And so other than RDP and port 80, a lot of traffic is denied.

So this deny statement got triggered. Let’s see, port 1433, I think that’s the sequel port. So somebody was trying to find a SQL server running on this machine and of course there’s not, and it wouldn’t be public if there was. So nice try, but that was very interesting discovery. So we can see here that the traffic analytics allows us to go further. We can write our own queries, modify our own queries, and even just like I said, get an overview of what happened, which IP receiving the traffic, which subnet was receiving the traffic and which is malicious, which is coming from, which was blocked, et cetera. So some dynamic reports here, pretty basic stuff from India. So interesting.

• Category: Uncategorized