Google Associate Cloud Engineer – Asynchronous Communication in Google Cloud with Cloud Pub Sub Part 1

1. Step 01 – Understanding Need for Asynchronous Communication

Welcome back. In this section let’s look at decoupling applications with Pub sub. In this step, let’s get started with need for asynchronous communication. Why do we need asynchronous communication? Let’s consider synchronous communication. Applications on your web server make synchronous calls to the logging service. You can see that the logging service is writing the logs to the database and the web server. Whenever it needs to write any logs, it makes a synchronous call to the logging service. Now consider a scenario if your logging service goes down, what would happen will your application, the web server go down to? If you have not written it well, there is a chance that your application also can go down. Also consider the situation.

What would happen if all of a sudden there is high load and there are a lot of logs coming in? The logging service might not be able to handle the load and it might go down often. That’s where you can decouple your application from the logging service. You can decouple by creating a topic. For example, you can have your application put log messages on the topic and the logging service would periodically pick up messages from the topic and process them whenever it’s ready. The advantages of this approach is decoupling. The app which is publishing to the topic is the web application. This is also called the publisher. The publisher don’t care about whoever is listening. They keep sending the messages to the topic. Availability also improves.

Even if the logging service goes down, the message will be in the topic and whenever the logging service is ready, it can pick up the message and process it. So publishers remain up and they can keep sending the messages even if the subscriber the logging service is down. The other important advantage is scalability. If you have a lot of messages in the topic, then you can actually scale the number of logging service instances. You can increase the number of logging service instances so that you can actually process them quickly and store them to the database. Durability of your messages is also improved. Even if the logging service is down, a message is not lost. Whenever it comes up, it can start processing the messages from the topic. As we can see, there are are a number of advantages in going for asynchronous communication. Let’s see how to do that with Pub sub. Starting the next step.

2. Step 02 – Getting Started with Cloud Pub Sub

Back in the last step, we understood why we need Asynchronous communication. In this step, let’s talk about Pub. Sub. Pub sub is reliable, scalable, fully managed asynchronous messaging service. It is a backbone for highly available and highly scalable solutions. In Google Cloud, you can have up to a billion messages processed through Pub. Sub every day. And the great thing is you don’t really need to worry about scaling your server. Pub. Sub would autoscale another advantage of Pub sub is low cost. You would pay for use. You’ll pay for the number of messages which are coming in. Some of the use cases where you can consider pub. Sub is for event injection. If there is a lot of events happening, you can put them into Pub. Sub and your consumer can consume the message from Pub. Sub. The other use case is delivery for streaming analytics pipelines.

So if you have a lot of data which is coming in for analytics, you can put it in Pub. Sub and slowly you can take it out to the final data store wherever you’d want to store it to. If you look at other cloud platforms, for example AWS, you have SQS and SNS. SQS is simple queuing service. SNS is simple notification service. If you’d want to do pull messaging, you’d go for SQS. If you want to go for push messaging, you’d go for SNS. However, Pub. Sub supports both of them. Pub. Sub supports both push and pull message deliveries. Let’s now see a quick example of how Pub. Sub works. The first thing that you do is create a topic. And to this topic, a number of publishers can send messages and a number of receivers or consumers can listen onto the topic and process these messages through subscriptions.

So a publisher is a sender of the message. Publishers send messages by making Https requests to Pub. Sub googleapis. com. So they’ll specify the topic name and make a Https request to this specific URL. The subscriber is the receiver of the message. There are two types of subscriptions. One is pull subscription. This is when the subscriber pulls messages from the topic whenever the subscriber is ready. How would they pull the messages? They would actually make a Https request to the topic. Again, they’ll send a Https request to the Pub sub Google aps. com. The subscriber would ask are there any messages to be processed in the topic? And the topic would respond back yes or no. This is called pull.

The other option is push. As soon as a message is sent by the publisher to a topic, the message is automatically sent out to all the subscribers. So messages are automatically sent out to subscribers. When a subscriber is creating a subscription, they would actually provide a web hook endpoint. They would provide an endpoint and the end point would be invoked as soon as there is a message on the topic. When there is a message on the topic, a https post request is sent to the web hook endpoints pub. Sub supports a variety of publisher and subscriber relationships. You can have one too many one publisher, many subscribers, many publishers, one subscriber or many publishers and many subscribers. So it is very very flexible in the step.

We got a high level overview of Pub. Sub. We saw that pub. Sub helps you develop asynchronous applications. You have publishers sending the message and you have subscribers who are receiving the message. There are two different models which are supported pull and push. In the case of pull, the subscribers pull messages when they are ready. They ask for messages when they are ready. In the case of push messages, as soon as there is a new message on the topic, the message is sent out to the endpoint which the subscriber registered with. Pub. Sub is really, really flexible in the sense that it supports multiple publisher and subscriber relationships. One to many emails need to one and many too many. Let’s talk more about pub. Sub in the next step.

3. Step 03 – Exploring Cloud Pub Sub – Publishing and Consuming a Message

Welcome back. In the last step we got a 10,000ft overview of Pub sub. In this step, let’s go a little deeper and let’s see what is involved in actual message processing with Pub sub. Let’s take a step by step approach and look at the different steps involved in creating a topic, creating a subscription and then receiving a message. The step one is to create a topic. So you would start with creating a topic. Step two is to create subscriptions on the topic. That basically means subscribers register to the topic. Each subscription represents a discrete pool of messages from a topic. So if I have ten messages, all these ten messages would be sent to all the three subscriptions. So each subscription represents discrete pool of messages from a topic.

Important thing to remember is if you have multiple clients who are pulling from the same subscription. So if you have multiple clients who are attached to subscription one, then the messages would be split between the clients. However, if you have different subscriptions, each of the subscribers would get all the messages that are sent to the topic. So multiple clients create a subscription each. Each client will get every message. So we now have a topic and subscriptions are ready. Let’s look at the different steps in sending and receiving a message. First, publisher sends a message to the topic. Next, the message would be individually delivered to each and every subscription. Subscribers can receive the message either by push that’s basically Pub sub sends the message to the subscriber or by pull.

The subscriber can say I will pull for messages whenever I am ready and that methodology is called pull. Once subscribers process the messages, they will send an acknowledgement back. They’ll send an acknowledgement saying I have successfully processed the message. Now, it is very important to remember that there might be multiple subscriptions and Pub sub would have individual message queues for each subscription. The message would be removed from that specific subscription queue only when acknowledgment from that specific subscription is received. This would ensure that the message is successfully delivered and processed by all the subscriptions. In this step, we looked at the steps involved in creating a topic, a subscription and also sending and receiving a message.

4. Step 04 – Demo – Playing with Cloud Pub Sub

Welcome back. In this step, let’s play around with Pub Sub. If you just type in Pub Space Sub, you’d be able to get out to this specific service Pub Sub. And over here we’ll start with topics. What we’ll do first is we’ll create a topic and then we’ll create a subscription. And then we’ll put a few messages into the topic and see what would happen. Let’s get started with creating a topic. So create a topic. Check my first topic and I’ll say create topic. It would also create a default subscription. So this is a checkbox which is check by default and a default subscription. So this would create the topic as well as a default subscription. So a topic and a new subscription has been successfully created. And once the topic is created enough, you can see all the details of the topic. You can see the topic name if you want.

You can export to BigQuery or export to cloud storage. You can see a few metrics around it. And you can also see all the subscriptions which are present. What we’ll do is we’ll create an additional subscription. So I’ll say create additional subscription in addition to the existing one. We’ll create another subscription. I’ll call this subscription two. When we are creating a subscription, you can say whether it’s a pull subscription or a push subscription. Pull subscription is the consumer would pull the subscription. Whenever a consumer wants to process a message, he’ll ask for it. The other option obviously is push. When we want to use a push, we need to configure an endpoint URL. As soon as there is a message on the topic, this endpoint URL will be called with the message I don’t really want to worry about the push.

I’ll go with pull. You can also configure the message retention period for this subscription. How long should unacknowledged messages be stored for? Over here the default is seven days. So the message can remain in the topic for up to seven days without getting processed. The next thing you can also configure is the subscription expiration. If there is no activity on the subscription for a specific number of days, then the subscription would automatically be deleted. You can also configure an acknowledgement deadline. The default Acknowledgment deadline is 10 seconds. If I receive a message. If I pull a message, how long should I process it within? That’s the acknowledgment deadline. I would need to send an Acknowledgment back to the topic within 10 seconds.

Otherwise the message processing will be marked as failed. You can also configure a filter to say I am only interested in specific type of messages. I don’t want to worry about all the messages which are coming in here. Only messages of a specific characteristic I’m interested in. You can also configure ordering for your messages. You can also configure a deadlier topic if the processing of a message fails more than a configured number of times. You can send it out to a deadlater topic and you can also configure a retry policy. Let’s not worry about all of them, let’s just say create. So we have created a topic. One subscription was automatically created for us and we are adding in another subscription now.

You can see the subscription details in here. Let’s go back and if you go back, you would see that there are two subscriptions my First Topic Sub and Subscription Two. And if I go to topics, there is one topic. What we want to do now is we would want to put a message to the topic. So you can go ahead and say Publish message. And let’s say I would want to publish one message for now, let’s just say I would want to publish my first message and say publish. Let’s add additional messages as well. So let’s add another message and let’s say second message and publish and publish one more one message again, third message. So we have sent three messages out to our topic. What we’ll do now is we’ll go to the subscriptions.

I’ll open up each of the subscriptions in a separate page, my First Topic Sub and Subscription Two. And I’ll have topics also present in here. So I’ll open up the my first topic in here. So this is my first topic subscription. This is subscription to and this is the topic over here. You can click View messages in the Myfuss Topic Subscription and you can say pull. What would happen? I’m not getting any messages. If you say any delay, try and pull it a few times and you should see the messages come up. You can see that when I say pull, I see the messages which are coming in. I’ll go to the other one subscription to and let’s say view messages and let’s do a pull in here as well. Am I getting the messages in here? Yes, the messages are in here.

If I do a pull here, so you can see that I can pull the messages in both the places and each time I’m pulling, I’m getting all the messages back, right? So each time I’m doing a pull, you can see that I’m getting all the messages back again and again. Now, what is the ideal way to process a message in a subscription? The ideal way to process a message is to first do a pull, then return an acknowledgment back. So I’ll say my first message processing is done, then I would send an acknowledgement back. And now that message is considered to be processed by that specific subscription. So subscription Two has successfully processed the first message.

That is indicated by the fact that I have sent an acknowledgement back. If you do a pull now, you’d see that in the pull of this subscription to the first message will not appear again. What I’ll do is I’ll send an acknowledgement for the second message as well. So the Subscription Two has processed two messages now. So if I do a pull NAV, you’d see that there is only one message. You can see that there is only one message in this subscription which is not processed. However, if I go back to my first topic subscription, if I do a pull, you can see that I can still see the three messages here. The Acknowledgment was sent only by this subscription, so the message has been successfully processed only by this subscription.

The other subscription has not processed the message successfully yet, so it keeps receiving the messages until it successfully acknowledges them. So nav, I would go ahead and send Acknowledgment for my first message and let’s do a pull again. Looks like the Acknowledgment did not go through well, so let’s acknowledge it now the second message as well, and let’s try and do a pull again. You can see that NAV, there is only the third message which is not processed by this specific subscription. So both these subscriptions have not successfully processed the third message yet. Let’s now do a pull again in my first topic subscription and say acknowledge enough. This particular subscription has processed all its messages.

So when it’s trying to do a pull, it does not find any messages anymore. And this one, if it tries to pull, there is one message still to be processed. The important thing to remember is when you create a topic and there are multiple subscriptions on the topic, the message will be sent out to all the subscriptions. The expectation is that the message will be individually successfully processed by all the subscriptions. So if I have two subscriptions like this, the expectation is that each of the messages will be processed by each of the subscriptions. In addition to creating topics and subscriptions and creating messages and playing with them, you can also create snapshots.

Snapshot is point in time status of a specific subscription. Remember that it’s related to a subscription, it’s not related to a topic. So what is the state of processing in a specific subscription? That is what snapshot indicates. You can select a subscription and you can give it a name and you can take a snapshot. Let’s not worry about it right now. The other things you would also see in here are something called light topics and light subscriptions. The Pub sublite is a Zonal messaging service optimized for cost. If low cost is your primary consideration, and you are not worried about the fact that you are using a Zonal storage, then Pub Sublite is recommended. However, the recommendation is to use Pub sub for most of the use cases.

So Pub sublite is very, very similar to Pub Sub, except that the replication here is Zonal and therefore it’s very cheap. In this up, we looked at Pub Sub. In Pub sub you can create a topic and you can create subscriptions for the topic. When a message arrives on a specific topic, the expectation is that the message would be successfully processed by all the subscriptions. You can create either pull Subscriptions or push Subscriptions. You can also take a backup of the state of a specific subscription by creating a snapshot. If you want to keep your costs low, then you can go for Pub sublite. I’m sure you’re having a wonderful time and I’ll see you in the next.

• Category: Uncategorized