Amazon AWS Certified Machine Learning Specialty – ML Implementation and Operations Part 2

3. SageMaker On the Edge: SageMaker Neo and IoT Greengrass

Next, let’s talk about living on the edge with Sage Maker on the edge and how that works. By the edge, we mean actually deploying your Sage Maker models, your train models out to edge devices. So maybe you have an embedded computer within your self driving car that you want to deploy things to. Maybe you have a smart camera, like an AWS deep lens that you want to deploy your trained model to, things like that. Neo is a way to compile your inference code to edge devices and make them optimize device for different devices that might be embedded somewhere so you can train once and run anywhere. As the promise of Sage Maker Neo, it supports multiple architectures for edge devices, including Arm, intel, and Nvidia processors. They’re all in the space of building little specialized deep learning enabled edge devices. And again, you can embed that in whatever device you can imagine, like your car. So you don’t necessarily have to be running this inference in the cloud and going over the Internet. You could actually deploy this stuff to run locally within the machine that it is.

So in applications where you care a lot about latency, like for example, a self driving car, you don’t want to be waiting several hundred milliseconds to get a response back from the Internet somewhere to decide whether or not to slam on the brakes, right? That needs to happen immediately. So you want that logic of that train model sitting within the car itself. And Neo is a way to compile that code into a format that can actually be run within that embedded device. That’s what Neo is all about. How do you remember this? Well, again, I’m not sure if this is a cultural reference that holds up around the world, but Neo was a character in the movie The Matrix, and The Matrix is built around this premise that the world is a simulation. It’s all just a computer simulation, and code is surrounding you everywhere, so everything is code.

So the metaphor there is that Sage Maker Neo is deploying the code of your trained machine learning model to the edge devices that are surrounding you in day to day life. Okay. Anyway, it can take any code that you’ve written in TensorFlow, MXNet, PyTorch onyx, or XGBoost and optimize that to these specific devices. And that opens up a whole lot of possibilities. Right? It’s actually pretty amazing that you can just take arbitrary TensorFlow or MX net code and recompile that toward these different architectures, and it just works. And XGBoost, if you don’t want to be dealing with neural networks necessarily, that’s probably the hottest algorithm out there right now for doing machine learning outside of the world of deep learning. So very good variety of technologies available for you there. Neo consists of both a compiler and a runtime library.

So the compiler is what recompiles that code into the bytecode expected by those edge processors. And obviously there’s a runtime component as well that runs on those edge of edge devices to consume that Neogene rated code. Neo pairs well with AWS IoT green grass. So remember, the exam is really all about putting these components together in interesting ways, right? So neo and green grass like peas in a pod, basically. So you can take a recompiled model and you could just deploy that to an Https endpoint. So I could host a recompiled model on a C five m five, m four, p three, or p two instance. But it has to be the same instance type that I use to compile that.

I mean, obviously, if I’m using neo to compile my model to a specific device, I need to make sure that I’m hosting on that same kind of device, right? But that kind of defeats the purpose of Neo, right? If I just want to host it on an endpoint on some sort of a web server, some service somewhere, why not just do that in whatever language I started with?

So it gets interesting when you pair a neo with IoT greengrass. Green grass is the mechanism whereby you actually deploy code to an edge device. So that’s how I’m actually going to get my train model to an actual edge device using inference at the edge, using local data, using a model that was trained in the cloud. So I train my model in the cloud on Sage Maker using training instances, but then I compile it with Neo and deploy it to my actual edge devices using IoT green grass. Greengrass uses lambda functions extensively for inference applications. So again, you might see Neo, IoT greengrass, and lambda used together in a whole entire system. So that’s how it all fits together, basically. Again, neo compiles your model, your trained model into specific architectures that might be deployed to the edge. IoT green grass is what gets it there.

4. SageMaker Security: Encryption at Rest and In Transit

Let’s talk about security in the context of Sage Maker. As we know, most AWS exams focus a lot on security, so you definitely want to pay attention during this lecture. Let’s start by reviewing some general best practices with AWS security that also apply to Sage Maker and using Amazon’s Machine learning services. First of all, you want to be using the IAM service as much as possible that’s identity and access management. Not only can you use that to restrict the permissions of the different services that are talking to each other within your larger system, you can also use it to set up user accounts for AWS and make sure that those user accounts only have the permissions that they need. So if you need to only have permission to run a Stage Maker notebook, you can restrict permissions to setting up Stage Maker notebooks and ABS Three to getting data in and out of it. You want to use multifactor authentication whenever possible. So you want to make sure that your administrator accounts, and certainly your root account, is using MFA to make sure that we’re sure that someone’s not just guessing your password and breaking in.

That can be a lifesaver, and I’ve learned that the hard way in the past. Also, use SSL and TLS when connecting to anything. Fortunately, it’s hard not to do that in the case of EMR, which is about the only case where I can think of where you’d be connecting directly to a server. In the world of machine learning, you’d have to try pretty hard not to do that in a secure manner, but it is possible, so don’t get any ideas. Make sure you’re using SSL when you’re connecting to those servers. Also, you should be using Amazon’s Cloud Trail service to log any activity to the APIs that you’re using and any user activity. So if you do, God forbid, have a breach, you’ll at least have an audit trail to look at to try to get some forensic information about what happened there and how to undo the damage. Now, remember the difference between Cloud Trail and Cloud Watch.

This tends to be a favorite question on Amazon exams in general. Remember, cloud trail is for auditing. It’s leaving a trail of activity, a log of what everyone did, whereas Cloud Watch is monitoring log data and issuing alarms when things are going wrong. Use encryption whenever appropriate, and especially with PII that’s personally identifiable information. So if you’re sending around things like people’s names or email addresses or addresses, or especially things like Social Security numbers or credit card numbers and things like that, make sure you’re encrypting that data both in Rest and at transit. Let’s dive into more detail on how to do that with Sage Maker. So at Rest, you can use Amazon’s key management service for pretty much everything in Sage Maker. Any Sage Maker jobs or notebooks will accept a Kms key to encrypt all the data stored by those notebooks or jobs.

So you can provide a key managed by Kms to actually perform encryption of that data at rest. That includes artifacts coming from training, parameter tuning, batch transforms, and coming from your endpoints. Also your notebooks, the notebooks themselves, and everything under Opt, ML and Temp in your docker containers can be encrypted with a Kms key as well. There’s also the problem of securing your training data and the data used for hosting your model. So you can just use standard S Three encryption techniques for that. S Three supports various kinds of encryption at rest and in transit. So you can just use the capabilities of encrypted S Three buckets for your training data and for hosting your trained models back to Sage Maker. S Three can also use Kms optionally, and it can use its own keys as well. In transit. Basically, all traffic supports TLS and SSL within Sage Maker. You can also use IAM to assign roles to Sage Maker to give it permissions to access specific resources. So in general, you want to follow the principle of least access. So only give Sage Maker itself permissions that it needs to access the data that it absolutely needs and reject everything else.

IAM can let you do that. Also, you can optionally encrypt the inter node training communication as well. So if you actually have training happening across multiple nodes, you can even encrypt that traffic too. It’s kind of hard to imagine how someone could actually intercept that traffic, but you might have some sort of a regulatory or requirement to actually encrypt that too. If you’re dealing with particularly sensitive information, this does come at the cost of increasing training time and time. Time is money when it comes to training. So if you’re doing a very complicated deep learning algorithm that’s spread out across multiple nodes, that encryption will come at a very real cost. But sometimes it’s required. This is also known as intercontainer traffic encryption. You can just turn this on via the console when you’re setting up your training job or through the API for training or tuning, for that matter.

5. SageMaker Security: VPC’s, IAM, Logging, and Monitoring

There are some specific concerns around VPCs with Sage Maker that you need to know about as well. So when you run a training job in Sage Maker that’s running within a virtual private cloud of some sort. Now if you really need extra security, you could even use a private VPC for that. You can specify that when you’re launching your training job. However, that creates some problems, right? Because Sage Maker depends on having access to S Three to get its training data and also to get the training and model artifacts for when it’s actually deploying your model. So we need to make sure that we at least have connectivity between Sage Maker and S Three within a private VPC. That means you’ll have to set up S Three VPC endpoints to enable that communication. And you can still use custom endpoint policies and bucket policies within S Three to keep that secure.

But you do need to set that up for a Sage Maker to work at all. Another consideration with security and Sage Maker is that Sage Maker notebooks are internet enabled by default. So by default your notebook can go out to the Internet and download data and do whatever it wants to it. That can be very convenient if you’re downloading public data sets or what have you, but it can also be a security hole. So you can disable that while you’re creating your Sage Maker notebook. But again, you need to get out to S Three somehow in order for Sage Maker to work at all.

So if you do disable internet access on your notebooks, you’re going to have to set up an interface endpoint that’s also known as Private Link or maybe a Nat gateway and set up outbound connections to actually allow your training and hosting to work. So remember, if you disable internet access from your notebooks, you’re going to have to set up Private Link or a Nat gateway to make sure that you can get through your VPC to get the outbound connections that you need for training and hosting to work. Also, your training and inference containers are also internet enabled by default. It is possible to enable network isolation on those as well, but again, you’re going to lose S Three access by doing so. So you’ll have to work around that somehow. It is a security concern in general that Sage Maker has to get to S Three still somehow, so you can tighten that up using these techniques, but you still have to make sure that Sage Maker has the S Three access that it needs. We mentioned using IAM with Sage Maker, the identity and access management service. Here are a list of some of the specific permissions you can set that are related to Sage Maker.

So you can set up specific permissions for creating training jobs, creating models, creating endpoints, creating hyper parameter tuning jobs, creating notebooks. You can lock these down or enable them to whatever degree makes sense for a given user. So if I want a user that I just want to be able to create notebooks but not actually deploy them, or create notebooks, but not actually create a huge training job that might cost a lot of money, or I might just want to lock down hyper parameter tuning jobs because those in particular can get really expensive. You can do that through user permissions with IAM and Sage Maker. There are also some predefined policies you can use, such as Sage Maker read Only access, Sage Maker Full Access, Administrator Access, and a data scientist policy as well that you can use just out of the box, a brief mention of logging and monitoring. Again. So again, it’s a good practice to use Cloud Watch. In general.

Cloud Watch can log monitor an alarm on your endpoints for actually doing inferences, and it can monitor the invocations of those endpoints and the latency of those endpoints as well. Very important for runtime performance, right? The health of the nodes themselves, the CPU available memory, things like that can also be monitored. And you can also use the Cloud Watch with Ground Truth, which is kind of neat. So if I want to monitor how many active human beings, how many workers are actually working on my job, and how much work they’re doing, I can actually monitor that as well through Cloud Watch. So, kind of interesting twist there usually think of Cloud Watch as monitoring machines, but they can monitor the performance of humans doing your labeling tasks. And Cloud Trail is again used for auditing. So that’s going to record actions from your users roles and services. Within Sage Maker, the log files containing that log information are delivered to S Three, where they can later be used for auditing purposes.

• Category: Uncategorized