Steef-Jan Wiggers introduces the concept of Serverless Messaging and walks through the evolution that resulted in Serverless. He then goes into detail about the various applications for Messaging in Serverless supported by multiple demos.
Integrate 2018, June 4-6, etc.venues, London
I’m here to introduce our next speaker, a great friend. So we’ve known him for a few years now. Do you remember, Steef-Jan, the first Integrate, I think, two years back, we were picking up all the boxes and…
Woman: Yeah? So we pretty much didn’t have a great venue like this and the organization team and I think our association goes quite a long time, long, long years back. So Steef-Jan Wiggers has 15 years of experience as a technical lead developer, application architect and consultant. He’s a very common face in the community. He runs a lot of events. He’s a big speaker and Steef-Jan is very active in the BizTalk community as a blogger, Wiki author, editor, forum moderator, and I’m sure you must have seen and heard him on the “Middleware Friday.” So for these efforts, Microsoft has recognized him as a Microsoft MVP for the past five years. His other face is from the Codit side. So he’s also working for Codit now. So I am happy to present Steef-Jan Wiggers.
Steef-Jan: Okay. Thank you. I just also want to show you my shoes. So I saw a tweet that there is quite some people from Holland and Sweden, is that correct? So who’s from those countries? Wow. Well, you know, one of them is not now going to Russia. All right, so I got joke out of away so you can’t pester me on that.
Right, Serverless Messaging with Microsoft Azure. So, already introduced, [inaudible 00:01:59] form Codit. I’m also a MVP for quite some time, used to be BizTalk then we got moved into the integrations queue and now we’re in Azure. I quite…currently doing quite some Azure for the last couple of years and also this writing for InfoQ. So that’s a website where I’m able to also investigate other technologies on other platforms like Google, IBM, AWS, and also Azure. So I quite frequently write about all kinds of stuff.
So what can you expect in the session? So I’ll also use the fine word “serverless” so you would get pretty tired of the word or you might even get very excited about it. But I’ll dive a little bit into that. And then we’ll do some stuff around messaging and Azure. Look in some of the messaging scenarios and also, we’ll do some demos for you.
So let’s look at serverless. So you’ve heard this term quite a while. I did agree with someone, yeah, you would drink a whole bottle of tequila every time I would say this, every time you took a shot, so you’re going to be drunk.
Right. So look at this as evolution to serverless. So if someone told me, this is another session of Christian [inaudible 00:03:06], this was interesting, the evolution to serverless. So you go from VMs to containers to orchestrators to PaaS to serverless. And basically what you do is that you provide more trust into Microsoft, letting them manage stuff. What you also heard in the previous talk and some other talks is like you trust them more and basically you talk to a bunch of APIs. That’s kind of what you do. So you kind of have that shift all the way up to you give the complete trust of your infrastructure and the services to Microsoft, in this case, for that platform.
So if you define serverless, so if you look at, for instance, there’s always, you know, definitions when back in the day with SOA and Microservices, there’s definitions. And there’s also a definition for serverless. And basically, it’s an execution model, dynamically manages your resources and basically the pricing, and it’s kind of utility computing, that’s what it kind of is. So you can do your rapid development. You can focus on your logic. You have this micro billing, so you pay what you move, so it’s kind of more commoditized, similar to what you now have with electricity, or with your phone, you pay basically by the second. A lot of skills being managed for you so if you need more resources, you will get more resources, or it’ll scale down. And basically it’s that abstraction of notion or abstraction of your servers. So that’s a nutshell of how you could define it.
And if you’re thinking about serverless, it’s like there’s no infinite scale of resources. If there is, then you need kind of atomic workloads for it. And that’s kind of where the truth is. There’s always something finite. Even with our natural resources, they’re not like forever. And if you look this bunch of services running within your environments, you always need some kind of messaging. You also need some type of decoupling and transfer state or manage the data you’re processing. So that’s always kind of the case. You need kind of reliability and stuff as well.
So then, if you look at the term serverless, and we can go all out, right, serverless is functions, serverless database, you could look at Cosmos DP. You have serverless events in Event Grid. You got workflow, which is Logic Apps or Durable Functions. You can look at serverless IoT, which is Event Apps or IoT Hubs. You can even look at serverless analytics, which would be then Application Insights. So you can just keep continuing.
Well, what about serverless messaging? Right? How do I explain this? So I did tell you that I’ll feature my puppy in this presentation, so I could give the message down and stay and I will tell him to come, right? So he gets focused, he gets the command out, and he’s coming up to me, serverless messaging because there’s no servers involved. And he does listen, although he also wrecks my home quite a time.
So serverless messaging. But what I actually mean is that…and you’ve got kind of options in Azure. There’s multiple options to do this messaging for you and some I’ve already mentioned.
So let’s look at messaging and Azure. So what do you do with messaging? Yeah, even in Azure, you have it in over multiple branches if its financial services, logistics, IoT and telemetry, I, currently, for a reinsurance company, there’s messaging going on. Either you’re at order processing, you do some logging telemetry, you might, in the IoT space, have to deal with connected devices, with event driven systems. There’s just a lot of messing around. And even you guys, even from the BizTalk space to where you are maybe now, you always have to deal with messages and with messaging.
So if I look at messaging in Azure, you kind of have four kind of options or services. So I don’t know who’s familiar with the Daltons and Lucky Luke? It’s a more European thing. Those four folks or they’re this, like, in a rest [SP] and timeless sphere is like they always collaborate together and same with these services, they’re not competing. They work together. So you kind of have four type of building blocks with regards to either messaging or eventing and that’s been explained already by Dan Rosanova and Clemens Vasters what kind of what an event is and what our messages with known intent and [inaudible 00:06:59] facts. So you have your Service Bus for your enterprise messaging; for big data streaming, you have Event Hubs; for simple task, you use the Storage Queues; or you can use the Event Grid for kind of reactive programming.
So if we look at enterprise messaging, it’s been clear that it’s about state and transferring from one to the other. And you know a lot of things ahead, whether it’s a schema of this message or you have to process. So you know a lot of stuff beforehand. This is pretty rich service you could use either with topics or queues or subscriptions, or even a relay that’s a part of this. There’s some competitive solutions, so even if I write about some of the stuff, there’s like SQS in Amazon or Amazon MQ or like Google Pub/Sub.
Then on the other end, you have your eventing. So it’s like cross app service, it’s light weight, low cost, it’s got some features, and you don’t know everything. So it depends on what you wanna do with the event.
So if I put these two together, you get kind of this cloud pub/sub model. But one is more like push-pull while the other is push-push. So that’s kind of the difference. It’s not versus. It depends on your requirements and your use case, which of the eventing or messaging you want to use.
Then you’ve got task queues. So those are like for simple task. So you can either have a lot of VMs that have to process some of the workload and you can either just set up all these queues and every queue is read by one of the virtual machines and then it does its task. It’s pretty simple, straightforward. And you can have as many queues if you want.
Then on the other notion, you have something like queues in a service bus. So again, if you put those two together at least, then you have Service Bus Queues or you have Storage Queues. So if you want more of versatility in cross platform, you can use Service Bus. And, well, it’s simple, again, with a simple REST interface, for instance, and you can use just Storage Queues.
And then you have something like events hubs for big data streams. So then you come also in the realm of big data or eventing or IoT. And then you also can handle loads through different protocols. You can feed data. There’s even like, for instance, in this big data with event hubs, there’s Kafka Endpoint, which we’ll have a demo in a second. And again, there’s some competitive offerings also in other cloud space.
So let’s look at this Kafka Endpoint demo. So I was trying this out in Java when I’m in kind of competitive in our little WhatsApp group between MVPs like, “Can we build this in .NET?” Well, I almost got there and we were talking with Wagner and Wagner just found it out. So it was bought by Wagner in this case. So that’s how we’ll go. So if it doesn’t work, you can blame him.
Right, let’s start the Consumer and let’s start the Producer. Okay, now just fingers crossed if this will work. So it’s making this connections and stuff with Kafka. And basically, this Kafka Endpoint mimics Kafka itself, but you don’t have to manage anything. That’s done by Microsoft. So basically, it’s a kind of a facade. So it’s kind of let Kafka clients think that it’s Kafka itself. So let’s see. Yes, all those events have been sent. So it’s about 100 messages. And then we just have to wait what happens on the consumer side. Fingers crossed again. It does all have to go all the way to West U.S and back. So it has to cross the Atlantic because the availability of the Kafka Endpoint was only in the West U.S and the East U.S. Although I’ve learned that it also now is available in Europe.
It has to say something like…okay, it does something, but come on, give me the messages, come on. And there we go. See, “welcome at Integrate.” So those were the messages I sent through thre producer. So this demo worked, cool. Right. So this is kind of what I wanna tell you and demonstrate to you. And in case the demo didn’t work, I could show you that it does work.
Okay. So you’ve seen maybe the Java code yesterday. So this is how the .NET code looks. And basically, it’s all about that part that was a little bit hard to understand. It was all about the certificate locations. It needed kind of set up this SSL. And this is kind of the trick. So it’s been extensively explained in the blog by Wagner, but this just how the .NET code looked and this a little bit how that configuration looked, which is a bit different than the one you saw yesterday.
Okay, so let’s look at some of the messaging considerations. You have to think about protocol, about what are the formats. Are you going to send XML, JSON, CSV, or Bionomic format. You have to think about the size in the end, when it comes to messaging or events. Security, the frequency, so the volume and how many times you going to send data, the reliability. Monitoring, so you just learned that there’s a new product called Serverless360, so you can do some of the monitoring with that product or either using Azure Monitor or Application Insight, depending on what your requirements. And of course, you have to think of networking, even also in a hybrid configuration.
So in the end, depending on which service you can use for…based on your requirements, you can either choose one of these messaging solutions.
Okay, so we talked about serverless, we talked about messaging in Azure. So let’s look at some of the scenarios. So this is one of the first scenario. So this has been taught, at least someone from Mike’s [inaudible 00:12:44], Jeffrey Ali [SP] old about the specific use case of a tollbooth where cars are passing and these cars have been pictured or at least a picture of the license plates have been taken and these are being stored in the Azure Blob Storage, which then generates this blob created event.
And this just triggers this bunch of…the services you see here, a bunch of the functions. And one of the functions will get, basically, or subscribes to that event, and will push the blob, at least download that blob and push it to Computer Vision API which does some OCR capability, that’s optical character recognition. It pushes back that data and then you can continue.
So let me demonstrate this. I’ll just push a few license plate, blobs to Blob Storage. And then these will be sent off to find a function to this Computer Vision API. And then the result is being pushed back and these are kind of the license plates. So the function looks as follows, I hope this is visible. So it gets the data and then it processes through this function and…or at least a part of, within this function, to analyze it, just post that payloads towards the endpoint of that cognitive service and just to showcase you how these would look, and this will be talked about yesterday as well, some of those capabilities like you have topic or at least properties that have to be in at that event.
So this is the event that comes into the function including the URL of that license plate, which is some sort…currently in that function being downloaded and pushed towards that endpoint. So for instance, maybe it’s also visible in the logs as you can see. I’m logging this too so this is kind of what happens.
Okay, so this is one of the scenarios where Event Grid really plays a role in it, by blob creation, leveraging OCR capability. Basically, license plate, this is what I already explained and this is the solution I just showed you.
Another thing I’d like to show you something more…Change Feed about events taking place. So this is part which could happen in Cosmos DB, so you have a set of data and if that data is being updated, then it will raise an event which you could use or trigger functions or an app service or you can do some stream processing or you can do some data movement.
So let’s look at a serverless home automation demo. So let’s say you have devices set up in your house in different…in your living room, in your bathroom, etc., and also as events are being pushed though Event Hub or via IoT Hub through Event Hub to a function, which then subsequently stores this in the Cosmos DB. And let’s say there’s changes to the temperature in your house, which could then trigger a message towards a logic app would send out a notification using a trigger like Trilio to send out an SMS or maybe an email, and then through your app. For instance, you can say, “Okay, I want to lower the temperature.” Let’s have a look at that.
So what I will mimic is…this is Cosmos DB, this is some of the data, this is from one certain house, but it could be multiple houses. So let’s take the… Okay, let me take bedroom 1, change this to…this should give you a view. Change the temperature to 29.5. So let’s say this event comes through the IoT, through Event Hubs to this function, and would update a database. So now your change event has taken place, And subsequently, this would trigger or this will go for that function. If it’s above 25 Celsius, it will then just push out a message to the queue, which subsequently will trigger a logic app and the logic app on its turn…and this has already a run and already feel something buzzing in my back pocket. So that means there will be an email.
Just a moment, this is just where you can use, for instance, a logic app for really simple notification, you can even use Flow, right? It’s just push out a notification and say, “Hey, it’s too hot.” So if I just look into my Gmail, already done this as a few times, okay, just refresh, there you go, bedroom 1 is too hot. So I get this kind of notification that my bedroom is too hot. And then I can subsequently, for instance, with my app, start Flow that would say, “Okay, send that message all the way back to that IoT device, saying, ‘Okay, lower their temperature.'” So it send out a command.
Right, so serverless home automation demo, using Cosmos DB, the serverless database in Azure where you can use this based on the change feed. So you can find this also online in a Microsoft documentation, how you can handle change feeds.
Let’s look at Pipes and Filters. So this is one of the cloud patterns you can find. So this is kind of where you chop up a workload in several parts. So you could send a message to a queue, which then will trigger a, let’s say, a microservice, a function, or something else that has to process it, and then the result of that is being stored somewhere, intermediate, you trigger another message or send a message to another queue, which [inaudible 00:18:36] is being be listened to by another service, which would pick it up and just continue the processing.
So I’ve done this for our customer where we do this in a kind of a microservices way where we had a gateway, an API within a container where you could send a request to which subsequently would send a request to a queue which was listened by a WebJob. We initially tried to do this also with .NET core in a container but somehow, we were not able to provision storage and SQL instance in ElasticPool and put those connection string into key vault. So we’re using kind of a WebJob, which is continuously running similar to a, let’s say, window serverless or window to be triggered to set off a provisioning of, let’s say, a storage account, an SQL pool instance and some connection strings in key vault. And this was done because this was a multi-tenant application, they needed to have these resources set up for us because they couldn’t load the data in.
So what I can do in this case is another example where this is a cow, the cow has been milked and that data of cows being milked is in EDI format. So this is from another scenario of another customer. And this could trigger, let’s say, a message to say, “Okay, hey, I’ve got this workload, and you need to process it. You need to parse that data from, let’s say, EDI to JSON.” And then subsequently later, maybe trigger another step to parse that data or store that data, let’s say, in a database. So let me demonstrate this.
So let me go to ServiceBus360. To kick this off, let’s say we do this. So I’m sending this message to the queue to kick start a process. Let’s say microservices that runs incognitos. So, what I’ll try to do, and this is not our fingers crossed, I will try to tunnel into my Azure Cuba natives cluster running or managed Cuba natives glossing and running an Azure. So that’s Azure Kubernetes cluster running…managed Kubernetes cluster running in Azure. So that’s Azure Kubernetes Services. So this is where I deployed this microservice that is listening, so it’s called the Service Bus listener. So it listens to a message that’s coming in, and it will do this processing of making that EDI file into a JSON file. There’s not much to see other than that you just saw the cluster and let’s look at the logs and see if we can find that message, or at least that payload back, if it will load up for me.
And basically, what it will show and here it is, in this you could see 778, which I just sent to Service Bus, went all the way up to AKS, and then it did some processing. And if I go to my Storage Explorer, you can see this is just the EDI message that’s being dropped there. Now subsequently, there will be a JSON file here. Wait for it. Just to give you a good rundown how does this EDI look like, it’s more like this is not really readable but it’s from certain farm, and they milk certain cows. And when I process this data, it gets more into a JSON and I even could do a translation and other microservice and then store it. This is kind of way we did it. But we didn’t use microservice in that particular use case, but you could. And as you can see here is that JSON file.
So microservice processing, again, way you use some of the messaging to kick start, let’s say, in this case, a container, at least a microservice is running in the Kubernetes cluster in Azure. And this is how what it would look like.
So the other thing I want to say is that…or at least want to express to you guys that you can use some of these services also together. So for instance, you could use Event Hubs an Event Grid together to do, let’s say, ingestion of data and push that full to…into a SQL data warehouse. So let’s say you have wind data, you have all these wind turbines, generating energy and pushing that data, all through Event Hub. So you do this capture of the event, and it will be in an Avro format, then the event’s being captured. This will then trigger a function that will retrieve that file and push that to SQL Azure Data Warehouse.
So let me demonstrate that. Start the WindTurbineDataGenerator, so this is also an example you can find online. But it shows you that you can also use some of those service in conjunction or in collaboration with each other. So you can do fan in ingestion with Event Hubs, and fan out using Event Grid. So this might take some time to start sending that data into Event Hub. Subsequently, that data will end up in my data warehouse.
Let’s go to the data warehouse, let’s see if this already has some data. There might be some lagging. That’s just because of networking or latency. It’s pretty slow. Let’s see, on time, I still… We have some time. Okay, let’s see. Well, there’s already data in there. Okay, so let’s have a look.
So this is where you could ingest data and then push it through to a data warehouse, let’s say…or you do to another type of storage, where you do want to do some machine learning, for instance, or you want some further data processing. As you can see, there’s some wind data in there. So another example of messaging where, you know, in this case, two of those messaging service in Azure can work together to do this data ingestion and data migration into a SQL data warehouse.
So to summarize, if you look at these services in Azure, you have to think about, “Okay, what’s the most suitable in my use case and for my scenario.” It could be one service you want to use, depending if you’re dealing with messaging or events, or you might want to use two of them, like for instance, in that previous demo.
You have to look at workloads. It’s not per se that the workloads are affected, or at least that your messaging service cannot handle it, but it’s more about those services behind it. So let’s say you’re pushing a lot of events and you run them through logic apps. There might be not…no sensible way to do it. Either it could be [inaudible 00:25:57] some limitations, or it’s just that you run into high costs. So in the overall architecture, where you put your messaging, you also have to think about what’s my workload, and how I’m going to land on those services, and how do I have to set those up, and also the services behind that, that have to deal with those messages and have to process those.
Then this thing around security and compliance, so you can look in different ways and the data itself, where does it land, has to be, for instance, now GDPR compliant or when you look at security in other way, who can access that data or who’s monitoring then data and then, for instance, might need some product like Serverless360.
They used to think maybe, of course, platform, is your data or messaging coming from other platforms, or vice versa. And for instance, Service Bus is platform agnostic, so that’s a good choice for it.
Then in the end, looking also back to the workloads, you also have to think about cost in the end of your overall architecture, even though messaging might be a small component of your architecture, maybe small cost, but you have to think about cost in the end, also when you have to manage the solution by itself and having qualified people, etc., etc.
And this also refers back to the DevOps if you want to roll out your messaging or your overall architecture to other subscriptions, to, let’s say, to a test subscription or to a production subscription, etc. So there’s a lot of things you have to factor in when you look at messaging.
So in the end, I want to say to you, happy messaging. So it was nice to talk to you, show you some stuff around what you could do with messaging. So I had some quick demos. and I would say, well, thank you. So I think I’m on time, right?
Fill the form below to get all the presentations delivered as a single zip file in your mailbox.
byJon Fancey & Matt Farmer
byMicrosoft Integration Team