AZ-304 Microsoft Azure Architect Design – Design a solution for logging and monitoring

1. Group Resources Using Tags

So in this section of the course, we’re going to be talking about designing, auditing and monitoring strategy. Now again, this is a very strategic thing as opposed to the technologies. One thing you have to understand about auditing and monitoring is that it’s things are getting a lot easier if you have a good organizational structure of your resources. So if you can imagine having an Azure subscription with 1000 different resources in it, everything from databases and virtual machines, network cards, everything in between, how do you organize all that?

So that you can do reporting, billing, the alerts, get put to the right person, the permissions are handled in the right way and all that stuff. So there are a few ways of organizing resources within Azure. One of the underutilized methods is called tagging. Basically within the Azure Portal or within PowerShell or CLI, you can assign tags to resources. If we switch over to the Azure Portal, we can demonstrate this. Now this is a resource group and even a resource group can have tags. So if you want to have some type of metadata that is specific to this project, specific to your company, you can add it as a tag.

You can also go into the resource level and you can see that the overview screen has access to add and change tags. There’s even a separate settings panel for tags. Now I’ll give you an example of how tags can be used. Now one of them could be, let’s say you have an internal billing system and every project has a particular code to it.

And so you can assign the billing codes to your resources. And that’s one way you’re going to basically track what the costs are and how those internally get associated with projects. Okay, we can save that. Another way you can do that is by having some other type of metadata. And I could put my name against it so I could say this resource was created by Scott. Now of course you can go into access control and go back in time and see who’s the responsible party. But having a person that a company employee that can be contacted about any particular resource that’s giving trouble might be very beneficial. You can also talk about various environments.

I know in many environments we have dev, test, staging multiple levels of those things and production. And so certainly tagging things by environment is good for billing and for other purposes as well, even understanding the disruption of like if you’re going to reboot a resource, who it affects. So these are all examples of custom created metadata. Now this, Azure didn’t give this to me. I created these and values that I assigned to them. But what good is that, right? Let’s go back to the homepage. I’m going to go into the subscription settings here. I’m going to pick my subscription and go into cost analysis for my subscription. Now right now this is a test account and it’s not accumulating very much in costs, but I could certainly break out my costs based on those tags.

So if I said add a filter and I picked tag, and then which tag am I going to filter on? Could be the building code, could be the environment, let’s say I want to see all of those. Okay, so now the graph has been updated. I haven’t accumulated any costs for these environments, but if I did, I would be able to see broken out by color the costs for various projects. Even these graphs that are broken out down here, we can change them. Instead of seeing the cost by service, we can see the cost by billing code, and it will break out, break out that. So the graphs and the reports can all be modified to filter based on tag.

And so tag is one way that you can basically organize your resources specifically for billing, but also for auditing and other purposes. Now you can also organize your resources into resource groups. If you’ve taken any of my courses on this Azit 300 or other courses, then you will see that we’re constantly creating resource groups. Every time we create a resource, you have to create a resource group. And so if you’re intelligent and strategic about which resource groups you create, how you organize resources, then that’s one way it’s going to make your life easier.

Again, in terms of the billing, in terms of who getting alerts, the security and things like that. Finally, at the top level, you can actually break out resources into different subscriptions. Again, if we go back to an organization that might be so disparate, that has different offices, different departments, people don’t necessarily have to be forced to talk to each other in order to create resources, then you might want to break out into multiple subscriptions. And even within Azure settings, you can open up VNet, peering and other things. So resources within subscriptions can talk to each other with the right settings. But it’s another way of organizing resources in a way that they don’t interfere with each other. The building becomes clearer, and again, all the settings that you do are separated. Now the next thing that we need to talk about is Azure policy. So once you have your tagging policy set up, we can go into Azure policy.

We can basically set that as a rule so that no one will be able to create resources without assigning tags or forcing some type of default tagging onto your resources. And if we switch back to Azure Portal and we go into all services and we type policy, we can see Azure policy is here, and we can basically go and choose what kind of policies we want. We go under definitions. For instance, we want to find a tagging policy. I’m going to enter the word tag. And we can see that there are some enforcing tag and its value on a resource or on a resource group is an option and applying a tag is an option. So let’s say we want to enforce a tag on all resources. Now we go into here, we can see there’s a JSON that basically says based on the, if a tag name is not present then it will deny the creation. If somebody’s trying to create resources but not giving it a particular tag, it’ll basically just fail. Okay, so we can basically go into the assigned section safe for this entire subscription we want. So choose a subscription, pay as you go. I want all the resource groups in there.

You can assign this to a specific resource group and you’re going to say enforce a tag and its value. I want the tag to be environment and I want the value, the default value to be dev. And it will basically deny the creation of the resource if this tag is not present. Okay, that’s an example of creating a policy. And so if I say a sign, then from this point forward the tag and the value have to exist. Now if we change this to say the tag must exist, we can also do that and that would be sort of a slightly different rule.

But assigning a tag and its value as a default is certainly something you can do with an Azure policy and that’s another use of tags. And that’s one way you can enforce your corporate policy within the technology. And as we saw earlier in this video, you can go into the billing and the reports and basically break things out by tag or break things up by resource group. And that’s how you’re going to be able to sign costs. That $500 goes to this team and $200 goes to that team based on the resources all having tags and the tags indicating who’s responsible for it.

2. Introduction to Azure Monitor

Alright, so let’s talk about resource diagnostics. Now, resource diagnostics can really be broken into two distinct components. One is the collecting of these diagnostics and the other is going to be the querying and the gaining insights from them. Now just briefly, I’m going to show you a tool called Azure Monitor. See, it’s represented by this speedometer type icon and I put this on my menu. If you don’t have it, you can go under Azure All Services and you can start to type Monitor and it should come up as very easily to recognize link. So Azure Monitor is effectively the central location where so many of your Azure resources can send their metrics and logs.

And then you can analyze those metrics and logs in one place. So it’s sort of like a way to build dashboards for you to monitor your resources. Now, in order for this to be useful, you have to actually have resources that are creating logs and metrics and pushing them into something called a Log Analytics Workspace. So Log Analytics Workspace is the storage for the log files that get queried by Azure Monitor. Now, as we’re in Azure Monitor, what I’m going to point out is under Insights you can see the various resources that Monitor is designed to work with. This is constantly being improved.

This is rolled out first a couple of years ago and they’ve been adding resources ever since. We can see that it can work with Azure App Services. And so if you have App Services, you can basically add these applications to your log Analytics Workspace and be able to query their performance from here. Virtual Machines storage accounts, all of the container services like AKS networks themselves. So any kind of virtual networks that you have, they are now in preview mode for SQL Database, which is not obviously covered by the exam. Cosmos DB can be tracked here. Key vaults.

Azure Cache for redis. Now we can see I’m going to expand this a little bit, or can I? Azure. Data explorer clusters. These are in preview. Log Analytics Workspaces, that’s a preview screen as well. Azure stack service bus and all the available insights in an insights hub. So a lot of these preview things you don’t have to worry about for the exam. But in this section of the course, we’re going to go through sort of one by one these resources, how to enable these insights, and then finally we’ll get into how to query the insights from these resources using Azure Monitor.

3. Monitoring App Services with Application Insights

Now perhaps you can recall when creating an Azure App service that you get into the monitoring tab and you have this Enable Application Insights option. Now when I do my demos, I usually turn it off because I’m not really debugging these applications, I’m just demoing stuff. But application insights is what feeds feeds into the Azure monitor and so I’m going to say yes this time for creating this web app. And it’s going to, in this particular case, create me a brand new Application Insights container to store these insights. And because my app is in central, this container will be in central. But I can choose another container if that’s my wish. So I can say review and create. Now, just for interest, I’m going to create a brand new razor app here welcome to my web app. Thanks.

And test it real quick, all right? And then I’m going to deploy that into this app service in Azure that I just created. All right? So that published to Azure. Now what we want to do is remember we turned on Application Insights. So if we go back into the app service and we scroll down, we can see Application Insights under Settings. It is enabled. Now we have not added any instrumentation to the app like in. NET or in these other languages, you can actually push things into the log. So if you have counts or internal performance metrics or some type of thing that you would like to query on within Azure Monitor, then we could obviously add that to our code. There’s a way of changing our code to push stuff into the log. So let’s say we don’t have that intention. Okay, so this is going into our Application Insights. All right, let’s go down into monitoring. Now we’ve got obviously alerts and metrics that we can choose from. Now metrics has to do with things like your memory and CPU. Let’s open up here. We can see CPU time, data in, data out.

So the networking stuff, garbage collection, the types of errors that people are receiving. So any kind of successful view of the web page should add to this metric, right? And I’m summing them up. So it’s just going to go higher and higher. So Metrics is pretty straightforward in terms of the performance of your app. The alert says you can actually create yourself an alert. So let’s say you want to have an alert when the CPU metric exceeds 50%. That might be something that you’re interested in, creating a scaling environment or just letting yourself know that the application might be slower for some users, et cetera. So you can set yourself alerts under here. Now the part that is going to be interesting and we’ll talk about in another video and a couple of videos from now is running queries, actual written Kousa queries on the app service log files. Okay, if we go under Diagnostic Settings now this is currently in preview mode but this is kind of separate from what we’re just talking about with Application Insights that go into Azure Monitor.

In logs, a diagnostic settings you can see is actually pulling out antivirus scans, the web logs. So some of these logs in the application Services that we saw, file audit logs, audit logs, IPsec. So these are very specific logs that can get stored into a destination and then you can download that. So let’s create a diagnostic setting. So I’m going to give this a name calling app service diagonal. You can see there are four current destinations, so it used to be three. And now they’ve added this partner solution. So we can push these diagnostics into Log Analytics. Again, log analytics we can query from Azure Monitor. We can put this in a storage account. Now, once it’s in a storage account, then we can access it so we can programmatically go to the same storage location and our applications can access that programmatically. Vent Hub is basically a real time platform where you can move an event which happens to be a page read or an error message that goes through the event hub and can be ingested into another solution. So maybe you’ve got a database or an Azure function or some other listener who is interested in receiving these events.

Okay? And as you saw, we’ve got this sort of partner solution option for potential partner integrations. And if you click on the link, you can see that right now you’ve got Apache, Kafka, you’ve got Datadog and you’ve got Elastic. And so these are three external partners who work with Microsoft to ingest these log files from your applications. So we’re not going to enable this right this second. We can also extract besides the Log stuff metrics, but we have Application Insights running and that should allow us to get access to some of these things from Azure Monitor.

So we will do that. Now, the last thing I’ll mention is that Application Insights for App Services is its own section. I’m not going to cover it here. I’ll clear a video about that. But besides Azure Monitor, we can use Application Insights to get very specific data, graphics information about the running of our app services. And this is something that’s available for apps, that is not available for virtual machines or other resources besides being inside of Azure Monitor. So that’s one of the benefits of app services is application insights. I’ll create a video on that. But you can see that Application Insights grabs your data from your application, allows you to view it from within Azure Monitor. And Application Insights is its own thing as well.

4. Monitoring Virtual Machines

Now, next up, we’re going to talk about monitoring virtual machines. So if you go under Azure Monitor and you look at the Virtual Machines tab, it could be a little bit confusing at first. You go in here and it says, Zero, no results. But there is this tab metaphor that says, I do have one virtual machine on this account running that does not have Azure Monitor or any kind diagnostics being extracted. So it’s under the not monitored tab. Now, you might also see under Monitored that there is out of date agents, essentially. And so you have to do an upgrade. This is an interesting situation, but if you have an old enough virtual machine, sometimes you do have to upgrade your agents. And so upgrading is required. I’m going to flip out of Azure Monitor, go into the Virtual Machine.

So I have this virtual machine called my minor and right on the Overview screen, if I look at monitoring, I can see there is activities going on, CPU and network, et cetera. But I can get visibility with various things, including Insights. So if I go down under Monitoring, we’ll see an Insights tab. And so in order for this to work with an Azure Monitor, we do want to enable Insights on this. Okay, so if I do click this enable button, what it’s doing? Is it’s going to do a deployment to deploy Insights to this resource group? And of course, this virtual machine needs to be modified in order for its monitoring to be sent to Application Insights. So let’s wait for that deployment to happen. So now we can see that Insights installed and we’re starting to get some information. This happens to be a map which is going to show me the server, various ports, et cetera. Interesting view. We can also get to see the performance of the CPU and the network, et cetera. Now, notice there’s a banner at the top that says, we have released a new version of Azure Monitor.

Please upgrade. So this is an interesting thing because we just enabled Azure Monitor three minutes ago and it’s already asking us to upgrade. Let’s go back to Azure. Monitor under virtual machines. And we can see here now this machine has moved from Not Monitored under Monitored, and it says Upgrade Available. And there’s a link to why. So obviously, Enabling Monitor enables one version, but it’s not the latest version. So we can obviously upgrade it from here, or we could have upgraded it from that previous screen. I don’t mind doing the upgrade. I want to get the latest in terms of Microsoft’s technology of Azure Monitor and alert capabilities. So I’ll do that upgrade. All right, so that is now upgraded. We can refresh the screen anyways. I’m pretty sure it’s upgraded. Oh. Guest VM. Health not enabled. Let’s go into here and we can see again, Azure Monitor is here. We do have the ability to look at resource health various machine properties, properties of the VM itself. Remember, we’re going to talk about in a second going into Azure Monitor and running Queries.

And so now we can see these are the kinds of log events that have been tracked in terms of performance, the machines, heartbeat, stuff from insights itself, et cetera, inbound connections, processes that are running so we can get this insights when we run the query. Also, I guess one thing to look at is the concept of workbooks which is predefined reports, right? So I can go into this performance workbook that Microsoft has created and obviously it’s got various reports on it waiting for me. All right, you can customize this, create your own workbook. And we saw that there’s a gallery of workbooks and so there are again, other people are going to create public templates, your own templates, et cetera. Basically, this is the monitoring for virtual machines that you get from Azure Monitor.

• Category: Uncategorized