Amazon AWS Certified SysOps Administrator Associate – Advanced Storage Section

1. [CCP/SAA] AWS Snow Family Overview

Okay, so now let’s do a little bit of a deeper dive into how the multipart upload works. So this is allowing you to update a large object in parts in any order. And it’s recommended when you have a file that is over 100 megabytes and it must be used for the files that are over 5GB. This helps you paralyze the upload. So this helps you speed up the transfers and also be safer because in case of a failure of one part you can retry it. So the Mac maximum amount of parts you can have in your upload is 10,000.

And the idea is that you have in your Amazon Sree bucket and your big file, you split it into parts and it could be up to 10,000 parts. Then you upload all these parts in parallel so you can speed up by maximizing your network bandwidth. But you can also retry a specific part upload in case it has failed.

Then your files are uploaded into Amazon Street and then once they’re all there, you can finish the upload by doing a complete request. And this complete request is going to concatenate all the parts and all the files into your bigger file back into Amazon is free. So with the failures you can restart uploading only the failed parts which will improve performance and improve the amount of time you have to do retries.

And then if you want to automate delete the old parts, you can use a lifecycle policy to delete the unfinished uploads after X number of days. For example, in case you had a network outage or an application shutdown or that kind of stuff. You can upload using the CLI or the SDK to take advantage of the multipart upload. So if you go into management, I just want to show you the lifecycle rule that is applicable. So you can create a lifecycle rule and I call it demo multipart and you can apply it to all objects in the bucket.

That’s fine. And then the lifecycle rule action would be to delete expired delete markers or incomplete multipart upload. And as you can see, you can have a lifecycle rule to deal with incomplete multipart uploads. And now you just need to say hey, I want to delete incomplete multi part uploads after three days to clean up anything you want and this can come up in the exam. Then you create this rule and you’re good to go. So that’s it. I hope you like this lecture and I will see you in the next lecture.

2. [CCP/SAA] AWS Snow Family Hands On

So let’s have a look at the snow families. So, services in the snowfamily, which is large scale data transports. So in here we’re able to either process data at the edge or migrate data in and out of AWS. So two different kind of use cases and we have snow cone, snowball and snowmobile. So let’s order one. So we have to plan your job. And so these are the use cases for the snow families. So either we want to import data into Amazon is free, in which case we get an empty device that will be shipped to us and then we transfer data onto it and we ship it back or we can export data from Amazon is free. So this is the other way. It will put data onto the device, send it to us and then we send back the device for erasing. Or if you want to do edge computing, we could select local compute and storage only.

So let’s have a look. Let’s first look at import data into Amazon s three. In this case we have to specify an address. So I’ll just call it example. An example the company is Example. The country is the United States kingdom. Whatever. This works. Example this is just a fake address just so that we can not applicable for division. The zip code is ABCDE and the fourth number is 0000. Let’s hopefully hover. This work works. And then let’s do Express shipping or standard shipping. And this basically defines how fast the device gets to you. Okay, so my address has been created.

Now I’ll just create a job so I can call this one test job. So test job. And as we can see, we need to choose our snow device. So do we want a snowball edge storage optimize, snowball edge compute optimize or compute GPU? And I believe that in this region, in the London region, we don’t have access to snow cone devices, but they would be here if we had access to it. So maybe I’ll just switch regions very quickly, go to Ireland and we should start seeing some snow cone as well. So the same job and then we’ll just do example everywhere. So I’ll be very quick here and maybe it’s also an availability based on the country. So let’s choose Ireland and then whatever state example and then ABCDE and a bunch of zeros. Here we go. Express shipping. Click on next. And here we go.

So now we have access to snow cone as well. So here’s my test job. And as we can see here, we have access to snow cone but also the three types of snowball edge. So storage optimized, compute optimized or compute optimizer GPU. So if we were to transfer a small amount of data, we would choose an eight terabytes snow cone device. But if you want to transfer a large amount of data, maybe getting a snowball edge storage optimize is the best way to go. Next, we need to choose a s three storage we want to load data onto so we can just, for example, choose this bucket or create a new bucket to import data once we’ve loaded onto the snowball and send it back to AWS. And then click on Next we could do some encryption, we can create some service roles and then let’s click on Create service role just to have service access. Yes, this is done.

Click on next. We can create an SNS topic to get notified, so call it Snow Family Notifications and for example@example. com, which is a fake email address. And as we can see here, now that we have the Snowball devices shipped to us, we should download AWS Apps hub to manage your snowfamily devices and have a UI that allows us to transfer data into the devices or launch instances and so on. And to get Upsub, we just click on this link and then we’ll see that Upsub is something that needs to be installed for Windows, Mac or Linux. And there’s a down button right here. Okay, so this all makes sense. Then we can review the job. And please don’t click on Create because this will cost you a lot of money. This is not a free service, so I’m not going to click on Create job. If you do, you’re going to have a very heavy AWS bill, so please don’t do that. But if you order the final device for another use case, which is edge computing. So let’s do it for edge computing.

So I can choose local compute and storage only, which is the edge computing. And as we can see here, we can create a cluster if we wanted to. This is optional, but if I were to enter a cluster name, then we would have a cluster, but I’m not going to do it. So this is not a cluster. Click on next. We’ll use the same address as before, express shipping and test edge computing. As we can see here, we have the same snow devices available to us. So snow cone, edge for syruptima and so on. But maybe because we want to run a compute job and there’s going to be a heavy compute job, we can choose the third option and we don’t need a GPU maybe for this time. So let’s not use a GPU. So we’ll choose this option. This looks great. I will scroll down again. An S free storage if you wanted to transfer back data into an S three bucket. But here, as we can see, we can compute using EC two instances, and this is optional, but here we can load EC two amis onto our Snowball edge device. And this is going to allow us to run computations using this EC Two instance.

And this is where you would customize it. The rest of the options are exactly the same. So this really shows you the power of the Snow family and you don’t see mobile here because I guess it’s only for the United States, but you really see that you have different kind of devices for different kind of use cases and so on. So I hope that helps, and I will see you in the next lecture.

3. [SAA] Storage Gateway Overview

AWS is starting to push for hybrid cloud. And what is hybrid cloud? That means that part of your infrastructure is going to be on the cloud of AWS, and part of your infrastructure is going to stay on premises. And this can be due to multiple reasons. Maybe you have a long cloud migration, maybe you have security requirements or compressed requirements. Maybe it’s part of your strategy to only leverage the cloud for elastic workloads, but to keep a lot of stuff on premises. So we have some services that we really like in AWS, such as Amazon S Three, which is a proprietary storage technology which is unlike EFS, which is an NFS compliance file system. So how would you expose, for example, the S Three data on premises, and the bridge between this S Three and your on premises infrastructure is going to be AWS storage gateway. So if we look at the storage cloud native options on AWS, we have block storage, which is Amazon EBS or the EC two instance store.

We have file systems such as Amazon EFS or Amazon FSX, and we have object level storage such as Amazon is Free or Amazon Glacier. So storage gateway is going to bridge your on premises data. And the cloud industry, the use cases of storage gateway can be multiple, for example, disaster recovery, backup and restore doing tier storage. And there are three kinds of storage gateway you need to know about going into the exam. So there is the file storage gateway, the volume storage gateway, and the tape gateway. And the storage gateway will then and we’ll see those in details in a second, import this data into either Amazon EBS, Amazon S Three, or Glacier. So at the exam, you need to know the differences between all three types of gateways, and there’s a lot of scenarios on it. So the file gateway is for you to have configured S three buckets being accessible using the NFS and SMB protocol.

And it supports the kind of storage classes such as S three standard, S three IA, and S three one zone IA, and then the bucket access using im roles for each file gateway will be secured this way, and the most recently used data will be cached in the file gateway. So the file gateway can be mounted on many servers on premises. And finally, something to note for the exam as well. If you need user authentication, you can have integration with Active Directory on premises to perform that user authentication. So let’s do a diagram to make it more clear. The application server will be talking using the NFS protocol v three or V 4. 1 to the file gateway, and then the file gateway will do the bridge to the average cloud. So we have S Three standard or Sri, IA, or even Glacier available in the cloud.

Okay, the file gateway will be interfacing and talking with Https to these buckets in the cloud. Now, as I said if you want user authentication and you have set up Active Directory, then there is an integration between Active Directory and the file gateway to provide authentication at the file gateway level. So we can see here the files are going to be accessible and extended to Amazon Is free. So this gives you a lot more storage for your file system. And on top of it, the most commonly used files are going to be cached onto the file gateway, which is going to be helpful to have low latency access to some files. So this really gives you a way to expand the size of your NFS on premises by leveraging Amazon history in the back end. The second kind of gateway is the volume gateway.

And this is block storage using the iSCSI protocol backed by Amazon S Three. And the idea is that you will have your volumes being backed up by EBS snapshots, which can in turn helps you restore on premises volumes in case you need to. So you have two types of volume gateway. You have the cached volumes to get loaded and see access to the most recent data or stored volume where the entire data set is on premises and there is a scheduled backup to Amazon S Three. So here our application server needs to be backed up. And so using this protocol, we’re going to get a volume gateway and the volume gateway will create Amazon EBS snapshots backed by Amazon S Three. So the same logic here, but here the goal of the volume gateway really is to back up your volumes of your on premises servers.

Tape gateway is that if you have some companies that have like for example, a tape backup system using physical tapes, then with the tape gateway you do the same process, but the tapes are going to be backed up in the cloud. And so this virtual tape library or VTL is going to be backed by Amazon S Three and the Glacier. You’re going to backup existing data using tape based process and using the iSCSI interface. And then this is going to work with leading backup software vendor. So diagram you can expect the corporate data center has a backup server which is tape based. The tape gateway will do the interface into the cloud by storing the tapes into Amazon Is free or in Amazon Glacier. Finally, if you as you can see in all these diagrams from before, the gateway has to be installed on your corporate data center. It has to run within your corporate data center.

But sometimes you do not have virtual servers to run this additional gateway. So an option for you is to leverage hardware from AWS. So it’s called storage gateway. Hardware appliance. So if you don’t have virtualization on premises, you can use a Storage Gateway hardware appliance and you can order it literally on Amazon. com. And then once you install this hardware appliance for this mini server, into your infrastructure then you can set it up as a file gateway, a volume gateway or a tape gateway. And this is really something physical. You have to install and will have the enough CPU, memory network and SSD cage resources to function correctly. So this is very helpful for example for daily NFS backups in small data centers where you don’t have virtualization available. So as a summary for storage gateway read the exam question really well it will really hint at which gateway to use. So if you need to have a bridge between on premises data to the cloud, storage gateway is going to be the answer.

Now if you need a network file system with optionally some user authentication with active directory think file gateway and the data will go in s three in the back end. If you need volumes block storage and iSCSI backup use the volume gateway. EBS snapshots will be created and they will be backed by Amazon is free. And finally if you need a tape solution for backups use the tape gateway and if you don’t have any onpremises visualization systems then you order in the hardware appliance from source gateway and install it in your data center. So that’s it for this lecture. I hope you liked it and I will see you in the next lecture.

4. [SAA] Storage Gateway Hands On

Okay, so just a quick handson with storage gateway. We’re actually not going to set up one, but I want to show you how it works in behind the scenes and just so you can see how you can create one. So you would get started and then you would have an option to choose the gateway type. You could choose a file gateway, a volume gateway or a tape gateway. So the idea I wanted to show you here is that file is to store a file as objects in Amazon s three and there’s a local cache for loading and see access. Volume gateway will be for block storage in Amazon s three with pointing time backups as EBS snapshots. And you can choose either a cache volume or in stored volume.

So this is exactly what I showed you before. And then tape gateway to backup your data to Amazon is three and glacier using your existing tapebased processes. And so for each of these, basically, if you were to create a file gateway, you click on Next and then you would need to select a host platform where you would actually download an image and run it on premise. Or you can even use EC Two. And then there is some set up instructions for EC Two.

We won’t do it right now, but the idea is that once you do all these things, you note the IP address of the new instance, you click on Next and you write the IP address right here. If everything is connected correctly, if it’s all configured and you click on Connect to Gateway, then you’ll have to activate it and then configure local disk and you’ll be done. But we won’t do it right now. All I wanted to show you is that we could have three options file gateway, volume gateway could be cased or stored. And finally, tape gateway. Okay, that’s it for this quick hands on. I will see you in the next lecture.

• Category: Uncategorized