SAP-C02 Amazon AWS Certified Solutions Architect Professional – New Domain 5 – Continuous Improvement for Existing Solutions Part 6

42. AWS Simple Workflow Service

Hey everyone and welcome back. In today’s video we will be discussing about the AWS Simple Workflow service. Now, AWS Simple Workflow Service, also referred as SWS is basically a workflow based execution service. Now this service basically makes it easy to build applications, specifically the applications which requires coordination among multiple distributed components. So the overview about the workflow can be seen in this diagram where you have the input data here and then you have various decision and activities and then you have the output data over here and this decision and activities forms a workflow and this entire chart becomes a workflow execution.

Now, there are two important components here, one is the decision and second is the activities. So the decisions are made by a component called Decider. Now, Decider basically defines the order of the execution of processes. It also defines the flow of input which can be between multiple processes. It handles the conditionals as well as concurrencies and there is always one decider per workflow. So let’s go ahead and understand more about Decider with the help of simple flowchart based diagram.

So this is a simple workflow execution here. So as the input you have a Hello World and at the output you have a Hello World with HS capital and WS capital. So this is what a workflow is all about. So any user who puts the input, you need to convert that input so that the first character becomes capital. So if you look into the workflow activity, everything begins with a start, then the word gets divided and there are two activities which are present. One is first, two Upper and second is Rest to Lower. Now then after both of these activities complete, you do a concatenate on the results.

First result you get from this specific program, second result you get from the second program and from there you concatenate, you go into the finishing stage and then you have the output. Now this specific activity, so these are referred as activities. So there are three activities over here, one is first to Upper, second is Rest to Lower and third is Concatenate. So these are the three activities and you need to have a program or some kind of a script which can execute in this block. So this again is a logical block which is formulated in terms of flow chart just to understand in an easier manner. So the activities are basically the actual processes which are taking place. So activities might be expensive to the system resources.

So if you look into the first to upper activity you have a simple function call as first to Upper where input is the string and all it is doing is it is returning the input with the upper case. So this is the first program which is executed with this activity. Again, with the second block you can have its own program here and for the third block you can have its own program here. So this entire flow becomes a workflow execution. So from the input to the output, the entire flow is a workflow execution, among which there are two important components that we were discussing. One is the deciders and second is the activities. Decider basically decides the workflow like, all right, so first the data comes from input. It needs to go to first to upper and rest to lower.

From there, it needs to go to concatenate. So all of the decision is being taken care by the decider. And then you have the actual activities which contains the programs, which execute when it gets the input. So, this is the highlevel overview about the Simple Workflow service. Again, Exam will not really ask you in detail related to how exactly the Simple Workflow service works. You need to understand the high level overview on what the SWF service is. Now, during the exam, you might get questions related to SQS versus SWF. And which one will you prefer? The first is that AWS SWF offers a task oriented approach. However, SQS is more of a message oriented approach. Now, second is AWS SWF ensures that the task is assigned only once and is never duplicated. However, in SQS, you need to handle the duplicate messages. S WF also allows us to handle manual inputs. So SQS is something that cannot really work there. So these are three important pointers that you need to remember.

Along with that, typically in exams, if you come across questions where there is a domain workflow, activities, workers. So those type of questions are more to do with the AWS Simple Workflow service. It’s not related to SQS. So if you have implemented SQS, you might have known that SQS does not really have something like domain or workflow. Those are more to do with the Sdwf. So if you see over here, you have a domain called as actions. So all of these are specific to the Simple Workflow service, and it will help you prioritize whether SWF is required or SQS.

43. AWS Step Function

Hey everyone and welcome back. In today’s video we will be discussing about step functions. Now step functions are generally used in organizations specifically for the orchestration of the serverless functions like the functions that you run in lambda. Now one of the questions that comes is specifically when you are using lambda for your application is how can you turn a serverless into an application by itself? So let’s say that within this there are two lambda functions and what you want is once the lambda function one executes and completes successfully, then you want the lambda function two to be executed. So only after lambda function one gets executed, then the lambda function two will be called for. So it’s similar to once your payment is successfully done then only the order will be shipped. So the lambda function first should be executed successfully.

Then lambda function two would need to be run. So let’s take few more examples where you might have two lambda function which are running in parallel and output of both. The lambda function should be sent to the lambda function three. So only after lambda function one executes and lambda function two executes successfully, then the output of both of them should be sent to the lambda function three. Similar to that there can be a selection like let’s say lambda function one executes and depending upon what the output data is, it should send it to either lambda function two or lambda function three. So this is also one of the important functions that you generally see as specifically organization which are based on serverless and definitely coordinating lambda function is also important where you have lambda function one calling two, calling three, three calling four, four calling five and so on. And this is an entire workflow and here a lot of thing happens, like it might happen that lambda function two has an issue and it does not successfully run.

So what happens is that the workflow will break. So after lambda function two does not run, it will not call the lambda function three. So what you want is you want something like a try catch block where okay, so lambda function two did not run again, run the lambda function two again or if the lambda function two did not run, go back to the lambda function one, execute the lambda function one, call the lambda function two again. So all of those try catch blocks, error handling everything is very important typically when you are using a lambda for your entire application and this coordination, this coordination between multiple components. So here you see there are multiple components, each lambda function would be for a specific purpose and this entire coordination along with error handling, tricatch block, if you try to do everything inside code it will become much more larger and difficult to maintain. So this coordination among multiple distributed components is handled by the step function. Now one great thing about step function is that it also provides the visual workflow.

So if you look into the workflow over here the workflow starts from the submit job so this can be a lambda function. It waits for x amount of seconds then it goes to the get job status, then it goes to job completed. If the job did not complete it goes to the job failed lambda function. If the job completes it goes to the get a final job status and if it gets the final job status then the application logic completes. From here you can even go back to the weight status. So all of this is actually a code which is written in step function however, this code is also shown in terms of visualization. So let’s understand this with a simple example about these step functions practically. So I am in my EC two and let’s go to step functions. And within the step functions, if you see over here you have the entire workflow. So all of the JSON that you write in step function, the JSON logic, the logic might be that first you should execute the lambda function one, then execute the lambda function two.

So whatever logic that you write in JSON, the step function will give you that in a visual manner. So that it really becomes easy for you to understand on how exactly the workflow is. So in order to start our first step function you need to click on get started and basically there are certain sample projects. We’ll use the sample demo that you have over here which is the hello world. So let’s give this the name of hello world and if you see over here you have Start, you have hello world, you have End. So what basically means is that start will go to hello world. Now the result over here is the type is passed and the result is hello world and then it goes to end. So let’s go ahead and do a next. Now you need to create an im role also so let’s give it a name of say hwstep function and we can go ahead and do a create state machine.

Now once the state machine is created what you can go ahead and do, you can do a start execution and just give it a name and do a start execution here and once your execution has started you know the output is going to be hello world because it was part of the JSON document. However, one important part to remember over here is the visualization workflow. So currently since the execution has completed successfully you are getting a green over here. In case the execution had failed you would get a red over here. So let’s say that you have instead of hello world over here you can even link to a lambda function. And if the lambda function fails then you will get a red color over here so that you can easily detect that already this hello and lambda function has failed. And this really becomes easier for a developer to understand on where exactly the workflow has failed. And this is the basics about steps function that we need to.

44. Understanding AWS Storage Gateways

Hey everyone and welcome back to the Knowledge Pool video series. And in today’s lecture we will be speaking primarily about the storage gateway. Now, Storage gateway is a nice hybrid based solution which is generally used in on premise based servers. So let’s go ahead and understand more about it. So AWS Storage Gateway is a hybrid storage it service that allows on premise application to easily use cloud storage. Now, let me give you one of the examples. Let’s assume an on premise application wants to store certain files in EBS or store certain files in s three. Then what you need to do is you have to put the entire logic related to AWS integration of SDK to the application. So application should be able to connect to the s three bucket and then store data in that x three bucket. So the entire logic needs to be written. Then comes the issue of putting the AWS access key, AWS secret key rotation policies and so on. So that is one way. Now, second way is to use Storage gateway. Now, Storage Gateway is a very nice solution. So all credit goes to the AWS webinar for the images that I’m using. I thought I’ll not reuse recreate the entire image which is already created. So let’s understand on how storage Gateway works where you have an application server. Now, this is customer on premise. So you can consider this as a data center or the local on premise server which a client might have. So on the left hand side you have a customer on premise, and on the right hand side you have the storage solutions in AWS like s three Glacier or EBS. Now, the thing which is connecting both of them is the storage gateway. So you have a storage gateway appliance on the customer side and you have a storage gateway back in on the AWS side. So what basically happens is the storage gateway appliance will create a disk and that disk can be used by the application server to store the data.

Now, before it gets confusing, let me actually show you on how exactly that might work. I have one disk over here which is icecase. Remember we had discussed about this when we were speaking about EBS portability and the network attached storage lecture. So I have a disk over here called ascazi disk. Now this disk contains a file called test TXT. Now, I can easily create any number of file over here, let’s say kplabsmo TXT. Now what happens in Storage Gateway is storage gateway appliance. So this specific appliance will create a disk similar to this like idrive and whatever application that you create, you can store all the application related data or all the backup files within this specific drive. Now, whatever data you store in this drive, this specific drive, the storage appliance will upload that data over internet. Because storage gateway appliance can be connected to one of the storage volumes which is S three glacier and EBS snapshot. So whatever data that you upload in that specific drive, the storage gateway will upload that data in a compressed manner to either AWS, S three glacier or EBS, depending upon the configuration that you have done on this storage gateway appliance.

Now, this is the reason why there is a direct connect which also comes in case you have a lot of data, like terabytes of data which you’re storing in your disk drive, then internet is not a viable option. That is where the direct connect really comes into picture. So let’s look into the storage gateway configuration. I hope you understood the basic on what a storage gateway basically can do. Now, one of the beauty about storage gateway is that it works on the local on premise server. So I don’t really have to worry. All have to worry is that I have to store data in this specific drive and Storage gateway will take care of the entire backup related functionality over here. So let’s go back to the presentation and look into the various configuration related parameters. Now, one important thing to remember is that the storage gateway appliance uses standard storage protocols like NFS or Ice Kazi to mount the disk. And on the other end of the storage gateway, which is the right hand side, the storage gateway connects to the AWS storage services like S three glacier or the data can be stored in form of incremental EBS snapshot. So with this set, there are three type of storage gateway configuration that one can have.

One is the gateway stored volume, second is the gateway cached volume and third is the gateway virtual tape library, also called the VTL. Now, you need to understand each and every one of these as form of a use case which will be presented you in the exam. So let’s start with each one of them. Now, gateway stored Volume what happens in gateway stored volume is that whatever data that you store in that disk drive will be stored locally. However, every time interval there will be an asynchronous backup of data to AWS like every one week, the gateway volume will take the backup of the entire drive and it will store that data in the cloud storage. So this is what is the gateway stored volume.

Now, this is why you have an asynchronous backing up of data. Now, this time interval can be changed according to what we require. Now, if you look into the AWS storage gateway over here, you have a volume storage and you have the upload buffer. So volume storage is present within the on premise itself. That means all the data will be stored in this volume storage and whatever data that needs to be uploaded will be part of the upload buffer. And this is connected to the storage gateway service on the AWS side. And then you can store it in a specific cloud storage option which is available to us. Now, gateway stored volumes are very beneficial when customer wants to access large amounts of data at a very fast rate. So this is one of the use case of gateway stored volume. Now, the second configuration type is the gateway cached volume.

Now, what happens in gateway cached volume is data is stored primarily on AWS S three with cache of recently read or written data stored locally in the on premise server. So whatever data that you store or whatever data that you put in the storage gateway, that data will be uploaded to the S three. So nothing will be stored on premise except the cache of recently read or recently written data. And this is the reason why if you see over here there is no volume gateway, let me open up the gateway store volume. There was a volume storage over here and the upload buffer. However, for gateway cached volume, there is no volume storage. So data is actually not stored locally, it is directly backed up to the S three storage. And this is one thing which is very important to remember. So in certain cases, like exam will present you certain use cases where an application or a client wants to store large amount of data, but he will be accessing only certain portions of data in a day to day basis.

So what type of configuration of storage gateway should you use? Should you use a gateway stored or gateway cache? And then your answer should be gateway cache. So I hope you understood the difference between gateway stored and gateway cache. Now, there is one more configuration parameter called as gateway virtual tape library. Now, this is one thing which is quite important to understand because virtual tape library is something that most of you have not familiar with. So let me show you one of some of the advantages of tape based stories. So generally, all we are aware about is we are aware about the portable hard disk, we are available about the SSDs which are quite famous and the traditional hard disk drives which contain certain platters. However, tape drives are something which many of you might have not seen. So basically, tape backup is a very common practice of periodically copying data from primary storage device to a tape cartridge. So data can be recovered if there is any data thrash or failure on primary device.

So basically, tape solutions are used for storing backups and archivals. Now, one of the benefits of tape based storage is that it remains very cost effective. And this is the reason why the data centers, a lot of big organization, they store the data in form of tapes within the organization. So this is called as the tape based storage and this is where the virtual tape term really comes into picture. So what happens in virtual tapes is that virtual tapes are stored in S three with frequently accessed data on premise. So this is something like a cached volume however, the data that will be stored will be in form of virtual tapes now, remember, the term is called as virtual tapes and not physical tapes so what is a virtual tape? So virtual tape is basically hard disk drive beneath. However, the way in which you write the data and the way in which things work in virtual tapes are quite different from a traditional hard disk it tries to follow. The configuration of a traditional tape based storage.

So in s three, you have the data which is stored in terms of virtual tapes. So when you talk about virtual tapes, you see you have a backup server and you have a tape gateway. Now, this data is stored via Https and it can be stored as virtual tapes in s three or archived state stored in Amazon Glacier. So this is something that we can store our data into. Now, you don’t really have to worry more about how it really works and the practicality aspect because this is something which is far, far away. The practical aspect is far away from what you are expected in the exam session. Now, only thing that you need to remember is what is the storage gateway? How exactly it really works in a high level overview along with the three configuration parameters like gateway, stored volume, gateway, cache volume and the virtual tape library that we have looked into.

• Category: Uncategorized