MCPA MuleSoft Certified Platform Architect Level 1 – Designing Effective APIs Part 3

4. Backend Systems Abstraction

Hi. In this lecture, let us discuss about abstracting the back end systems. Okay? So the best practice that we should follow as part of this API three layer architecture is we should always abstract the back end systems. So all the components should fit in the three layers, and the back end system should be kind of agnostic and irrelevant and to the functionality or the behavior. Okay, so who actually performs his job? I mean, which layer performs his job? As we already discussed, the system API layer is the one that will be abstracting the back end systems.

So the system APIs usually mediate between back end systems and process APIs by unlocking data in these back end systems. So the questions that may arise are like should there be one system API per back end system, or many? Or how much of the intricacies of the back end system should be exposed in the system APIs in front of the back end system? In other words, how many to abstract or how much to abstract from the back end system model into the data model of the APIs? Okay, because based on that decision only, we have to come up with the various data models in the system APL area.

So if we need to, like, bring up all the intricacies of the functionality of backend system, we will end up with lots of API data models. But if the organization or enterprise decides the back end system is huge, but we are using only some XYZ functionalities, then the IP data model will be less. So these kind of questions may arise. Okay? So there are some general guidance here as well that we can follow. Okay? The first one system APIs, like all the APIs, should be defined at granularity that makes business sense and adds to the single responsibility principle.

So it is therefore very likely that any nontrivial background system must be funded by more than one system API. Okay? So if an enterprise data model is in use, then the API data model of system APIs should make use of rana types from that enterprise data model only. Because the organization already has established enterprise data model, then the corresponding API implementation should translate between these data types from the enterprise data model and the native data model of the back end system.

Okay, so the input that comes to system API will be the entrepreneur data model. And when the system API talks to the back end system, it will talk in the native language of the data type of the back end system data structure. If no Enterprise data model is in use, then each system API should be assigned to a boundary context. Okay, like we discussed, one API to one boundary context.

The API data model of the system API should make use of data types from the corresponding boundary context data model only. This is a known thing like we discussed before, and the corresponding APA implementation should translate between these data types from the bounded context data model into the native data model of the back end system. Just like how we discussed for the Enterprise data model, whatever is the organization’s data model, AP data model is whether Enterprise data model or pounded context data model end of the day, the APA implementation should be obviously converting to wind flow between those data models to the native back end systems data structure. Yeah.

So in this scenario, the data types in the boundary condition data model are defined purely in terms of their business characteristics and are typically not related to the native back end model of the back end system. So because it’s the designer of the It or the API who have come up with the different bounded context based on the functionality or the business knowledge shared with the designer. The bounded context data models for the system A might be different from the back end native data model could be like a subset of a huge native data model because there is a limit of functionality being used but the field should be enough that day mappable or transformable to work to and fro. But the back end model could be very huge or very different mega instruction compared to the business context bounded context data model. But if no entropy data model is used and the definition of a clean bounded context data model is considered as too much effort, this is the third scenario. One is if you have Enterprise data model, we can go with that. If you don’t have go with the bounded context data model, establish all the boundary conduct data model and translate within the system layer. But there is a third scenario, third category of customers or people who for whom there is no enterprise data model. As well as defining this bounded context data model is too much of effort for them, then the APA data model of the system API should make use of the data types that approximately mirror those from the back end system.

Okay? So we can just take the data structure that the back end system has as it is. But whatever is the technology that is diagnostic, whatever is the technology, we’re talking about the structure here, take the structure, the field names, data types, et cetera, and prepare your boundary contrary data model as a mirror of It with same semantics and meaning and same naming as in the back end system and same data types, okay?

And for those data types that fit for those data types that fit the functionality of the system API, you apply those data types. And if there are types, some relatives are very negative to the back end system. If it is a DB, for example, if it is a very old system with a DB, and DB has some special data types like wirecare, et cetera and all, then you translate them with a similar high level data types like string, et cetera. Okay? So this way you expose all the fields needed for the given system view functionality somewhat or very much similar to your back end system.

So this is the third approach. Mirroring your boundary context data model with the back end system when you don’t have too much time for the boundary context as well as you don’t have an enterprise data model in your organization. Okay, this last approach which is exposing in system API is an API data model that mirrors that of backend system. Actually, if you think it does not provide some satisfactory level of isolation from back end system because we are talking about abstracting the back end systems here. But again we say this third approach allows you to kind of use the same data structure and all. Yes, it does not give a satisfactory obstruction or isolation from back end systems, but still you are not tightly coupled. Okay? Even if tomorrow and the system replaces the system, you can very well go to the data model and make alterations like add extra or make the fields optional because you have dedicated boundary context data model in this scenario.

Okay? If we are not like importing or reusing the back end system schemas here, correct. So in particular it will be typically not be possible to swap out a back end system without significantly changing the system API data model, right? That is true. It is accepted. That is why it is the last option. Okay. Wherein if there are no other choices, then organization should go. So therefore the APA implementations of all process APIs that depend on those system APIs will also be kind of impacted because there will be mappings inside the orchestration layer to call the system API. If the back end system changes and the data fields the data model again changes, then it’s a big effort.

So this should be something of feedback. As an architect, you should be giving to your organization. If they blindly say let us go with the last model which looks like a simple one where we can just mirror the back end systems data model, then you should suggest that this may have implications and complexities with respect to orchestration. Complex orchestrations. Okay? So that is why it is not desirable for a long life API. Because there is enough chance that whenever you change back end system you have to again revisit your whole API orchestration. So it is not desirable for a very long life long living APIs. If you think so, you have to be immediately suggesting this to your organization. So that is why the API data models of system APIs following this particular approach must therefore change when back end system is replaced. But on the other hand, it is very easy approach that adds compared to the very less overhead. Because in the mapping and all because it’s a similar structure, it will be straight mapping.

It isolates the APA clients from the back end system functionality to the outside data world, okay? Because the experience layer and the process data models might be very different. So still we achieve a smaller fraction of the abstraction that the APA client level. They won’t be knowing what is the back end systems data model, for example, what protocol the backend system uses, what kind of authentication connection pooling, network addresses, formats, et cetera. That is still abstracted, okay? The clients don’t know. It also allows the usual APA policies to be applied because the system APS are on new soft and we can create them on the APA manager. All the APA policies can be applied on the system APS, right?

So still we can enforce an effort and it also makes the APA data model for interacting with the back end system explicit and visible by exposing it in the Ramble definition of system APA. So although it’s an error, we are having our own copy array so it is alterable or we can have a scope to change it in future and further isolation from the back end system data model. It does occur in the APA implementation of the process API layer if you want to orchestrate, right? So these are the different types wherein organization has to choose how or what kind of abstraction they want to achieve from the back end systems. So if you compare it like if you take our logistics and supply chain example. So the things that we need to do for designing the system APIs are like many system APIs. They unlock data and functionality of the ERP that we have. We already know that. And as we discussed, if we have to come up with a bounded context, then our purchasing or purchase orders, sales orders, written orders and buyback are like different bounded contexts, right?

We already discussed that. And then the same principle applies to system of the purchase order, sales order. Also, like we said, the item line, item level, if we take the data model for item might be different for the purchase order, it will be inside the scope of that boundary context and the item data model for the sales order would be different and it will be sitting in that particular boundary context. So all these should be individually captured in the RAML fragments as something like sales order item, purchase order item with its own data type and published to the exchange.

And if there is no enterprise data model is in use and if your organization of the plane goes through the effort of defining a few clean, business oriented boundary conduct data models, then this is the approach they have to take. If they have the entrap data model, then all good. Okay? So this is what is about the abstraction of the back end systems from the system API. As an architect, you should be knowing all three models and suggesting your It management accordingly. During the architectural discussions and all right. Happy learning.

5. API Invocation Patterns

Hi. Let us now move on and learn something about different API invocation patterns. With respect to APIs in this particular lecture. And when we talk about API invocation patterns, we will specially concentrate on the various ways of Asynchronously calling the APIs. We have already seen a way and change our business scenario for the API which we are designing the Create Sales Order with respect to this Asynchronous nature. Right? What we have done was we have made the process layer API bit different. We redesigned it by making it asynchronous so that when the request comes from the API client to Experience layer and when Experience layer forwards the request to the Process layer creates Sales Order, we have modified it such a way that it just validates. And then if validation is successful, then we are acknowledging back with a unique order number which is generated within the process layer. And we are ending the call from the Experience layer there itself so that it is faster.

And we are putting that particular original request into a messaging system, could be an enterprise messaging system your organization has. Or we also discussed that we can use any point MQ or VM technology which we have in new software, either of them. So that was one way. Okay. But there might be scenarios where the APA clients may want to even follow up on their initial request. So what we discussed was a purely asynchronous one where we were giving the temporary or a unique order number internally created, but not the original order number. Or in other way to tell the actual status of the order creation is not being responded in the actual request. Right? We are just giving a temporary reference number back to the clients.

But there could be many things happening. There could be a chance that the actual request, after we post it on the queue, any point MQ or a VM may fail in the Erpe letter due to a number of reasons. But we already give the reference number. So the clients may want to query the status. So depending on the scenario, the requirements may change. So we are going to see a few more ways of Asynchronously executing the APIs, how differently we can implement them. The first one is asynchronously executing AP invocations with the polling. Okay, so what is this? Http has native support for asynchronous processing of the work triggered by the Http request. So this feature is mostly that is why applicable for the Rest API but not for the non Rest APIs and all, okay? Not for the RPC and all those kind of things.

So let us discuss more about how this synchronously calling the APA invocations with polling works actually. So what you are seeing in front of you is a flow diagram or sequence diagram with different states. So if you see the journey, what happens is the APA client first sends a Http request that will trigger Asynchronous processing. Okay, so the APA implementation how it should be implemented is it will just accept the request and validates the request. And if the Http request is not valid, then as usual Http response with the 400 category 400 series code will be sent back as a response code to the client.

Okay, so the 400 could be if it’s a bad request, then 400 bad requests saying the validation is not proper for maybe our customer ID or item ID or location. It could be a four, not four kind of things not found or four, not one unauthorized anyone. So 400 category error will go back to validation phase. But if Http request validation succeeds, then the API implementation triggers the Asynchronous processing in the background. Just like how we discussed before red using an any point NQ or a VM or any other messaging system and it returns a Http response with the Http two naught to accepted response status code. Okay, so why two? Naughty? That is what so like we discussed, Http has this native support, okay? So it has a meaning for each of the codes that were set in the Http. So when we say 200, okay, it means it’s kind of saying your request is fully processed and it is okay. But to acknowledge some clients saying okay, it is accepted, meaning the request is accepted but not fully processed. It is a kind of asynchronous behavior to convey through a technical Http where that it is an Asynchronous API and your request has been accepted but still being processed. You have to send a status code got two not two not two is Http. Two not two code is accepted status code.

So then the clients will know that, okay, our request is successfully accepted but still being processed in the background. And while sending this two not to Http response code back, we have to or the APA implementation have to set a response header as well http response header called location. And this location will contain the URL of a resource to poll for progress. Okay, so it’s like a Polling URL kind of thing. So the APA client then regularly sends Http get requests to that particular polling resource using the Polling URL. The APA implementation will return the Http 200 okay, if the Asynchronous process is still ongoing but returns Http response with the Http three not three. Okay, three not three generally stands for Http zero, three stands for C.

Other redirect status response code okay, meaning there is another URL for you where you have final status. Okay? That is the meaning of Http three note three. And again it will come with a response header called location. And in the location there will be this time a URL of the result final result. Okay? It is called final result. URL. Now the APA client then sends the Http get request to that last result URL to retrieve the final result of the now completed asynchronous processing. Okay? So this means that most of the coding for this will be residing in the APA client side. So though some part of coding is required from the APA implementation side, like sending the Callback Polling URL and sending the result URL. So there should be some Polling URL generated for the client to query and then the final result URL to be created for them to get the final result from the implementation side.

Client side has some bit more logic. Like once they get the two not two and the Polling URL, there should be a logic to continuously poll on that URL and once it gives back a three not three response code, then again they have to implement a logic to retrieve the result URL from the Location Response header and call the final result URL. So their whole solution will be a different approach, right? So the fact that the Http based Asynchronous execution of AP invocations is used should be documented in the RAML definition. That is why because see, this is not all when we discuss on this lecture seems straightforward, but for a client this all details should be reflected straight in the RAML or in the documentation that we are providing could be exchange or notebook. Right? So all this should be documented well in the RAML definition and in the exchange.

Okay? So let us go on and see the second way of Asynchronous execution. The second way is asynchronously executing APA invocations with callbacks. The first one was with Polling. This one is with Callback. So whenever, say, there is a situation, like we said, different projects has different scenarios and situations, right? So if the client is not ready, say to implement the logic for Polling, to continuously write poll logic to check on the Polling URL and all could be there, they don’t want to waste resources and all to necessarily poll, then there is an alternative approach. So an alternative to the Http supported polling based racing execution is the use of Http callbacks. Okay, maybe you might have heard the term called web hooks. This is very popular term web hook these days. So this web hook is nothing but a Http callback. Okay? So this requires the original API client to be reachable by the Http request from the API implementation. So this approach is different than the previous one. So here what’s happening. The API client is telling that, okay, I am calling you the API implementation. I know that upfront is asynchronous API. So what I’m doing is I am putting a Callback URL for you in my request only.

Okay? So this is my original request and in the header I am giving the Callback URL also. So now it is telling the app implementation that you process Asynchronously. I don’t care, but whenever you are done, I will not be pulling or doing anything. Whenever you are done, you call me back on this callback URL and you give me the final result. Okay, so if you see the diagram again, this time for the callback method, the APA client sends a Http request that will trigger asynchronous processing. Same as we discussed before. So with the Http request this time if you see it sends the URL of a resource of the APA client that will receive the callback.

Okay, the Callback URL can be sent as a URL query parameter or as a custom Http request header. It’s up to the agreement or contract between the APA client and the AP implementation, but could be a query parameter or request header. Now what happens? As usual, the API implementation accepts the request, validates it, and if the Http request is not valid, then as usual again Http response with the Http 400 series client response code error response code will go back. But if Http request validation succeeds, then the API implementation triggers asynchronous processing in the background. Till here it is almost same like polling and returns the Http response with the two not to accept it immediately. So, same as what we discussed before, okay? Now once asynchronous processing is completed, the API implementation must send a Http post request to the callback URL containing the final result of the completed request. Okay, so then this is like a way like a push model, right? So APA implementation is calling and telling the client that based on the callback URL, using the callback URL, this is your final result so that the client did not pull.

So now the question is how will the client know that this callback, whatever request that is coming in the later point of time belongs to which particular request it was sent before? So for this reason, along the request, when the client makes a request with the callback URL, along with the response which is to not to accept it, the AP implementation should send a correlation ID as well kind of correlation ID or a reference ID. Okay, then like this is also we discussed before, right? A temporary number or a unique number is generated in the process layer and sends back. Now dealing the call back, the AP implementation will provide that particular correlation ID also or the unique reference number also in the callback details with the final result so that the client will know that, okay? For this particular reference number, this is the final result.

Okay, so this same same thing again, all these details should be properly documented in the RAML and the exchange. Okay, so this is how we can even further improve our asynchronous behavior. So what we discussed in our Create Sales order was we just at high level discussed that, okay, we are going to change this to asynchronous nature by letting the process API called Create Sales Order accept the request by validating the details and then post it to a queue. But we never discussed whether API client wants to receive the response back or not. If that is required, then one of these models can be picked based on the agreement. If the APA client is okay to do the polling and all, then the synchronous behavior with the Polling URL is the best fit. But if the API client says no, but API client is okay to provide a web hook or a Callback URL, then the API implementation can implement that way. So this is the best way to do some asynchronous nature in the API. And these are also the invocation patterns that you should remember during the projects.

Now, in this there is a concept called state handling. So what is the state handling means? So whatever is the approach that we discussed before, whether it is asynchronously executing through polling or the callback situation, if you close the unities, it requires the APA implementation side to maintain the state of each asynch invocation. So it therefore makes APA implementations themselves stateful indirectly. Because if you see let’s say if it is a Polling URL concept, then the AP implementation after accepting the request has to provide back a Polling URL, correct? That means internally to provide that Polling URL, AP implementation should create a unique URL for every request that is accepting, correct? And it has to store somewhere because why it has to store at latter point of time, when the APA clients query on that Polling URL to show the status, the AP implementation has to respond back saying okay, whether it is done processing or not. So for that reason, it has to maintain that URL somewhere until the final request comes. Once the final request comes, then it can remove that or diminish it, right?

Similarly, once the final result URL comes, that also has to be temporarily stored. Not temporarily actually. It has to be stored permanently until the client actually calls it and gets the request correct. Because we do not know when the client may have to client may have to call the result URL unless again there is agreement that okay, once the result URL is given to the client, it is stored for 24 hours or something, it has to be stored temporarily or permanently. So there is a state again here, if you go to the callback mechanism where the state is nothing but the correlation we discussed about, because in Callback URL pattern there is no polling or final result to be stored. But whenever the processing is done, AP implementation will call back the original client using the Callback URL. So for this, the Callback URL has to be stored temporarily across the layers.

And second thing is the correlation ID has to be also stored to link that final result to the correlation ID while sending the request response back in the Callback URL, correct? So no matter what, there has to be some state in any of these patterns. So how can this state be maintained? Okay, so this also at high level we have discussed in our previous lectures in the previous section where we said, okay, there are many ways the mule software recommended way or mules are out of the box way is to use the Object Store. So using Object Store, the state can be maintained as key value pairs, where key can be a request ID and the value can be correlation ID and all. Similarly for polling URL or result URL. And if you don’t want to use Object Store, then we can use any external database as well. Okay? And for messaging, we know it can be any point MQ or Vmqs or some external message brokers.

All right? So if any of these will we can achieve the API invocation patterns through Mule out of the box way or through the external systems like external message broker or external databases. All right, so hope you understood the API invocation patterns. Let us move on to the next section. Sorry, next part in this section, which is the Caching part. Okay. We although we discussed about Caching a little bit at high level in the NFR section, how to improve the performance and all. We will go in with more details in this next lecture about the Caching like which http methods are safe and all right, happy learning.

• Category: Uncategorized