Mattias Logdberg draws up a scenario where a web shop is integrated with an ERP using BizTalk and explains how, based on business needs, to enhance the capabilities in this scenario by using a combination of Azure technologies.
Integrate 2018, June 4-6, etc.venues, London
Mattias is a solutions architect from iBiz Solutions, and he’s going to be talking to you about refining BizTalk implementations. Over to you, Mattias.
Mattias: Thank you very much. Thank you. Wow. Yeah, yeah, as he said, I’ve been jumping around on crutches. You’ve seen me. That’s all good. I mean, I’m a first-year MP, actually, and my first year at Integrate, so I had to come up to something that you will made you remember me. Didn’t I did good?
That was a nice one. I even managed to get on a welcome page from Saravana. That’s a nice one. That’s a nice one. My PR [inaudible 00:00:44].
But today we’re going to talk about refining BizTalk solutions. And I’m guessing that we did this title, we can do, I don’t know, how many sessions it could be. A lot of them at least.
But the thing is that we’re going to talk about today is that we’re going to talk about a story. I’m going to bring up a story. So this is actually a customer case that we have done. And I will actually try to kind of take the timeline and draw it up for you. Like, what we have done, how we have come to these conclusions, and, yeah, what problems we have encountered, and what path we have changed, and so on.
So hopefully, with all these great sessions that’s been a couple of days, you will be able to kind of connect the dots and see what we have made wrong before we even done it.
So everything has an evolution. So everything needs to be changed. We’ve been talking about this a few times earlier. And I mean, in all of these cases, in all of these implementations we are doing, we often is going to have a change.
The funny part is that, if we kind of apply this one to, like, integration and integration space and we say that, I don’t wanna do this, but if the monkey is BizTalk and if the guy with the cell phone is Logic Apps, this is kind of the change that’s going to happen.
The funny part is that, if you challenge this one, is the monkey really faster? Is the monkey really not as good as the man with the cell phone? Not really. I mean, make them to climb a tree. The monkey is going to win every time. It’s going to be really fast with it. So basically, there are different challenges, different needs. And I think you have always seen this one.
So another contexting that I wanna bring up is one of my favorite one is that this kind of challenge that is out there, like the IT needs versus the business needs. So I mean, when I see this one, when I see, I don’t know how you see it. But my initial kind of thoughts about it was like a man…you know, the business like running, and the IT guy is going, he’s like, “Wow, stop. Not so fast.”
I’m not going to do that with my foot. But that’s one way of seeing it. And I think that is…I don’t know. Is that how you see it? A little bit, right? A little bit, right? But the thing is that, what we need to kind of do in this kind of scenario is kind of change that view. Because in the end, we are not going to be holding them back. We are actually going to be able to say, “Hey, mate. You wanna run one more lap? Here, get a bottle.”
Because we’re going to be able to do the things that they can do things faster for them. Because if we do things in a way and we visualize them and give them the tools, they’re going to be able to run much faster. We’re going to build them a path.
And hopefully, you will kind of see this in the end, that we are doing this. Because we are enablers. They mention it all these three days. Integration hero. It’s been there all the way. We are enablers.
So this thing that’s going to talk about today is the evolution of a WebShop, actually. And when I was kind of preparing this one, this session, I was actually thinking, “How should I do this one? How would I do this in a good way? Kind of explain this one?”
So I kind of ended up in a workshop with a customer. And they had planned for, I don’t know, five-hour workshop, I think it was. And I was invited with, “Oh, you don’t need to prepare. Just come along. Just come along.” And after 20 minutes, the same man says, “Oh, table is yours.” And I had four and a half hours. And I had no preparations at all. That’s fun.
So I’m going to do the same thing as I did then. I’m going to actually use my favorite tool, which is the pen. So what is the first thing that we architects do? Well, we get a whiteboard and we get a pen.
Man: A box.
Mattias: A box, yeah. A box. It’s always a box.
Man: [inaudible 00:04:58]
Mattias: Arrows is in there as well. I promise, arrows. So in this kind of case, I got a challenge or not a challenge. I got a mission. We’re going to integrate our ERP with the WebShops. So basically, when we start, as I said, it’s a journey. So we will start this back in 2010. This is when this started. And we will end up in today. But we will start drawing from that point.
So basically, I’ll draw a box, obviously. So we have the ERP and we have the WebShop. Hopefully, you can read what I’m writing. So the ERP has something called articles, and it has customers. And the WebShop has or wants to have orders.
So basically, this is what we have. So we need to kind of move this data along. So we need to send data and we need to receive data. And remember, this was 2010. What should I use?
Man: BizTalk server.
Mattias: BizTalk server. Perfectly. Thank you. Now, I’m going to remove these arrows because I don’t wanna do point to point. I wanna have BizTalk in the middle. So no, not a red one. Let’s add a blue one. That’s fun. So BizTalk. And then we can connect our ERP, and now we can connect our WebShop.
And depending on the school that you have been doing, I mean, canonical format, not canonical format, which path should we go? Canonical format. Perfect. Thank you. Much better when you say what I want you to say.
So then you kind of build this kind of line in the middle. So we our own format in the middle. Maybe we should do it, like, more like…oh, I should’ve brought a wider pen. Like this area is our common ground. So this goes on. We implement this solution. And back in 2010, what did the business people do? They gave us a mission, and then they…
Man: Changed it?
Mattias: Disappeared. They didn’t wanna know us. We just delivered something, and then they were happy, and that was it. And now, this has changed as well, but yeah. Let’s keep it simple. They just disappeared. We delivered. We build this one.
So a few years back, I think it was, like, 2015, they wanted to change this WebShop. Because now the WebShop is too old. And it’s also needs to be changed. So let’s draw the kind of environment. So this is my beautiful network zone for on-prem. So this is my firewall. You’ll get that one. And then, I will do a DMZ as well because that’s fun.
So let’s add…we got WebShop, obviously, in the DMZ. We wants people to get access it. We got our BizTalk and we got our ERP. So let’s pick some colors into this one. This much better, I think. So the ERP, wow, I’m not so steady, are pushing out articles. And the ERP are also pushing out…is that a good color, customers?
And as we said, we had orders in, so let’s add orders as well. So let’s add orders. Cool. And as we said, we had this kind of… Let me use this one. We have this nice area here where we were using our canonical formats. I will just add it down there just for some kind of reference stuff.
And so, articles goes in. Customers goes in. And orders goes…sorry. And we got orders going down. So this is our case. And as I said, they want to change this one. And what do they do when it’s 2015? They go to the…
Mattias: Cloud. Thank you. Of course, you say. So they go to the cloud. So let’s draw a cloud up here. So I’m still not picking that color right. Let’s draw a cloud. That’s a cloud, right?
Man: [inaudible 00:09:37]
Mattias: Ass. Don’t listen to them. So up here, they’re going to buy up. They go out. The product goes in shopping, they buy a WebShop. This is quite similar to our old WebShop. So we just need to kind of provide a data there. They have everything in the store. So the easy part is kind of pushing the data up, isn’t it? I mean, it’s easy to push up our articles and it’s easy to push up our customers.
But now, we’re facing a new kind of problem because how do we get in? You have the answers for me. It’s been there all day, haven’t you? So any suggestions now I want to get in?
Man: Service Bus.
Mattias: Service Bus is one way. I do not want to use Service Bus, actually, because in this case, the requirement is that my order needs to be verified by the ERP. So that’s a business route.
Mattias: Relay is one thing. I’m actually not using Relay because I want to use something more cooler. I want to use API management. Much more fun. Sorry, Wagner[SP]. I know Relay is fun, but…
So I’m going to add in API management. But it’s not going to connect us right away. So what we’re doing is actually doing an IP filter down here. I know, it’s not the best, but what do we do while we wait for ExpressRoute, right? We need to start somewhere.
So this is Scenario 1. And every time you kind of change a WebShop, there’s always some new requirements, right? You need to change the system. Then they want to improve it. And then, they come with new requirements that’s not there from the beginning.
So our requirement in this case was that we had our prices calculated and the order was done in the ERP when the order was created. And this is not so kind of, you know, friendly for the customer. They often want to know what is going to pay in the end.
So they wanted to push the prices out. And why they haven’t done that before is because it has been quite a complex model. So they wanted to add that. So they added a new service. Luckily for us, we had API management, we just add a new service, right? So we’re just adding a new one down here. So and then we go here, and then we go prices. Is this good? No comment? Haven’t you learned anything today?
Mattias: Latency is one thing. What’s going to happen when you have, like, 10,000 customers in your WebShop? It’s going to go crazy. And you know what, maybe the funniest part, I don’t know if it’s fun, but it’s actually something. One of the thing is that, when we’re doing this, when we implemented this, it took one and a half seconds to get a response.
Put that on a web page, and then, you’re going to go for every, each article, and you’re going to wait one and a half second. Yeah. Go back to the 2.5k modem. Something like that. So that’s not going to work. And, I mean, the load on the ERP is really bad as well. So if we get a lot of users up here, we’re going to end up bursting our environment as well.
So that is not a good one. It’s not safe, it’s not good. Rapidly, just, no, no, no. Prices must do something different. So what we did was we actually added storage. So this is actually Table storage. And we added a small, in other word, microservices, to get that data behind the API management.
And on the ERP side, we could actually… I’m using the wrong color, aren’t I? Let’s do that one. So and the ERP could now push prices instead. Much better model. And in BizTalk, what can do BizTalk, obviously, we can push.
So how we do this one, then? We’re going to push up. This customer has calculated, it’s 3 million prices. And due to constantly changes, they need to update it every night. So they need to push 3 million prices every night into our storage. That is quite a lot.
So we need a microservices, obviously, and we need to push that into storage. We started off with like this. We actually pushed, like, bunches from BizTalk to these microservices on the app service plan, so the web API. We actually manage and keep control to make sure that we’re not flooding the storage.
So it works quite fine. And what happens then? The business comes and says, “We don’t want that WebShop. It’s not good enough. We want something new.” Right? Yeah. Obviously. It’s the business.
So what I do is that they kind of scrap this one up. Let’s just remove that one. And remove that one. So we scrap that one, and they go in another new one, much more fancier one.
So I intentionally draw a much smaller database there because this is one of the new ones where you have…I don’t know the English word for it. I don’t know if it’s a good word, but like service injection. So basically, when I need some data, then I wanna store it, then I want an API to call and get it. So I need to build it. So something else need to keep the data. And obviously, everything is going to add up into API management.
So now we going to have, like, all of this going down. They need services for all of this. So it’s easy. We say we have API management. That’s nice. And we have this whole price as well.
The problem is now that we don’t wanna go down to the ERP and ask questions because we don’t wanna go into that part again. We don’t wanna end up with these kind of problems in the ERP and we don’t wanna put all the load on the ERP as well. Because again, every user that logs in, it’s going to go collect data, collect data.
So we need to be able to push this data up instead. I mean, that is what we have done all the time. We have pushed it from the ERP. It’s going to need to use that storage a little bit more.
So we end up building a small microservice, and we’re pushing it. And we have another microservices. And we have…good. I mean, still, we haven’t changed much in BizTalk. All of this process, we have done quite a lot. All this, this has been intact. We have added prices, yeah, but everything else has been…it’s been as it has been since 2010, actually. So that’s nice.
And now, we have built something that actually the WebShop can use and go down to and build storage. Oh, might need to add these kind of arrows from this one as well, but you already get the one, don’t you?
So, now we have built ourselves a microservice layer, both behind API management and from BizTalk and up. And this is to be able to kind of, you know, push data in and out of storage, more or less. Have you guys used Table storage? A little bit? Yeah? Yeah? Then you know it’s quite static. The only kind of partition possibilities you have is on PartitionKey and RowKey. That is a fast thing to get something out.
So the easiest to get an article out…that is easy. And it’s easy to get, for us, a price out, because it’s an article and a customer. But the problem is when you kind of end up in…I need to search for something else. I need to find my customers who have least orders. I need to find something…I need to do something more complex. This story may be a bad one, but I need to do a complex thing.
And then, the storage becomes a little bit too small. It’s like you have RowKey and PartitionKey. You obviously have other things to filter as well, but you don’t have the flexibility.
So what we are doing now is actually moving even a bit further away. So we are actually implementing a DocumentDB. And the reason for this one, it’s actually not because of the articles, it’s actually because of the pricing. Because in this Table storage model, you will get a bunch of prices. You will get, like, a campaign price, you will get the customers, based on the distribution list they’re on, on the discounts, and so on. So you get a few prices.
And it’s really hard to do something better in Table storage. But the requirements from the business evolves to that the WebShop shouldn’t need to be able to kind of handle this logic. Why should a WebShop use this logic?
I mean, picking 100, 90, and/or 95, so this is my prices I get back. Sure, it’s not that hard. But it’s still logic that we push out to the WebShop. Then the funniest part is that the API is called best price. Yeah. I wanna give them only this one. So that is why we are now moving along to DocumentDB because then we can do more queries, we can do more complex scenarios.
And yeah. Another thing that we are also moving to is that, in this part, BizTalk is actually pushing data up to microservices. Is that really a good way? You’re quiet. Okay. I’m going to say it. No, it’s not. Because you have seen it earlier also, but if we get, like, a massive update, so someone brings on a fun, you know, doing some scripting in the ERP, and we get, like, 10,000 articles or 100,000 articles get updated, our, like, bus is going to go like this. And that’s not so good.
And it’s going to be very expensive if you go to DocumentDB. It’s going to be really expensive because then we’re going to need to scale it up really good. And so now, we’re also kind of changing this one. So then, we started to change. So we kind of remove this one.
And earlier, it was HTP [SP] posts. Now we’re adding a new layer. So much fun, so much layers. So we’re using Service Bus and topics. I know, I’ve learned. I should always integrate. Yeah.
This was before that, I would say. Sorry. And so, now we can have this one. We still need to have our microservices over here, but we get a little bit more loose architecture. So now we can actually be able to comply to our loads.
And then, our DocumentDB is going to be much more happier. Because DocumentDB will otherwise…have you played with document DB? No? It works like this. If you have, like, partitions…so if you have partitions, it’s going like this. And you put too much load on one, it’s going to go bananas, and it’s going to go red. So when we get this burst, it’s going to go red. And then, we’re just going to get 429s back. And then, we need to handle that.
And that is okay. We got Service Bus and all of that, but we can…probably before would be nice. It wasn’t that easy when BizTalk was shooting because, you know, BizTalk, it gets data, and it’s massively going crazy on the target.
So we need to be able to be in control. We need to be able to kind of smooth these peaks out so that our services are working in this load instead. More like this, much lower pace. And when it’s needed, it’s going to go active.
Yeah. That’s about our journey. That’s kind of where we are at the moment. And as you’ve seen, we’ve traveled from our ground. BizTalk is still a really core, vital component to this one, and that’s the fun part, I would say. And now we can actually shut this one down. And we can have a burst from all directions. We have a lot of users up there.
So let’s head back. I’m not going to go into topics. I’m going to use this one. As you can see, my foot is getting better. I can walk a bit, so that’s nice. What we’re trying to achieve here is actually kind of bringing this down from monolithic applications. So our ERP in the beginning was our monolithic application. Our WebShop as well.
So we need to kind of bring it down to smaller pieces…to loosely coupled entities. And this is done via the microservice layer, more or less. So that’s how we do it. And BizTalk is helping.
So even if we might not have seen BizTalk earlier as an API provider more or less. But thanks to that… If you noticed the yellow line, thanks to that, we actually have an entry point. We often kind of forget that one, I think, when we’re developing. That is a good one to actually make sure that we have our path, our API, whatever you wanna call it.
The other part is that we want to kind of remove all of this unnecessary logic. So if you have an API called Get Best Prices, let’s return best price. Don’t let all of the users, all of the apps return. It kind of calculate the best price based on the prices you are returning.
And we also need a kind of a framework. So kind of move the complexity from these big things to smaller parts. Like, Functions and Logic Apps and all of that, instead of building complex logic inside of ERPs.
I kind of forget that one, but that is also one of the tasks is moving business logic out to the ERP as well. But you still need robustness from it.
The reference architecture for this solution looks like this. It might not be… So hopefully, you saw similarities in my drawing. But the thing is that we need to front something up, and we use API management because that is how we gain security. That way we can handle throttling, we can handle a lot of things, and we can handle onboarding as well.
You’ve seen a lot of it before, so I’m not going to go in too deep. And we need some sort of, like, microservice layer. So where we use Functions, WebJobs, Web APIs, Logic Apps, you name it. This thing, we will use to connect to different things. In our scenario now, we didn’t any SaaS stuff, we didn’t use any IoT or whatever it could be. But in the end, this is where we kind of connect these things.
And then we have supporting parts, which is storage and Service Bus. We also have connection down to our on-premise. As we said, we used firewalled opening and the DMZ proxy to get the traffic down.
And our BizTalk is our connector or adapter to all the on-prem environment. So going to the ERP or going to a file or whatever it could be, we use the BizTalk for that.
If you need to add something, we have a perfect way of doing that now. So we kind of invested all the money to build something for our WebShop. So we can reuse that into a mobile app, we can reuse it in an advertisement app, whatever we feel is suiting this one.
And this is kind of where business comes in and say, “Hey, this is much more fun than when you were doing BizTalk. I can see something. I can see the APIs.” So that’s something you should keep in mind.
Because one of the things that we are doing with this kind of architecture and kind of setup is creating possibilities for innovation. So if we need two APIs to create these websites, and then we do another couple of APIs to create this one, and we need a few more to create this one. This is actually from another customer case, though, but still a customer, who had…we built it like this. We built up some APIs and we started receiving and chatting and so on.
And then, it came along and said, kind of looked into, “Wow, we have this. We have a new problem.” “Oh, what’s that?” “Well, they’re turning down our internal product viewer, so they wanna shift this into SaaS service instead, and it’s all connected to on-prem services.” And they were like, “Okay, what were you using? What were you using?” “Look, we’re using these three APIs.”
That is exactly what we’re using. The only difference is that you have connected it to the SaaS services but you are kind of…we are, as the integration team, are moving to the SaaS. So we are actually changing the connections.
Which meant that they kind of brought in a…I think they brought in a web developer and, like, 40 hours later, they had an app. They did go quite big announcement, actually, because they don’t do apps much. They are rental company. They rent stuffs. They don’t do apps.
But still, this kind of app went viral, both because, internally, they could now use this internal again. They lost it, so they gained back. That was fun. But because we put it outside of our firewalls, it was a web app, they could share it to the customers.
So the kind of thing was that, before, the salespeople could go in and kind of, you know, enter the product code and get up the documentation. And they could get, like, small documentation. They couldn’t get all they wanted to. They needed to go to three systems, I think it was, in the end to get all the information.
But when we sat there and kind of built this one, in the end, they kind of landed up with building a new app. And they could get a lot of more information and could send it out to others as well. And in the end, this was also powered by ERP data, but push out to this one.
So summary for this one. I would say that BizTalk is just as vital today as it was yesterday and will be tomorrow. The only difference is that we might use it as a tool in a toolbox rather than the only tool in the toolbox.
Because, as you have seen in the couple of days, again, everybody’s talked about…a lot of people have talked about the right tool for the job. And BizTalk is definitely one of them when it comes to the on-prem environment. And it’s a good way of connecting things.
We also need to have respect for our customers or for you guys who have invested a lot of time and money into these BizTalk solutions. Because it’s not like every single solution will be easy and better in the cloud. And we could, in this case, we use things that were done eight years back. Still works the same, still same logic, still same stuffs. So we only needed to redefine our solution into a solution that better fitted the landscape of today.
And I say as many others, better together. I really think we should use this kind of stuff more and more together because it makes life much easier.
I will also kind of say that there have been a lot of talk about it, and I think Kent Weir [SP] and Michael Stevenson was two get inspirer to this kind of stuff. To kind of be able to grow, be the hero, be the one who helps business, be the one who helps ops teams, be the one who helps non-IT people to understand this and able to build stuffs better. And enable ultimate processes that has been a pain in the ass.
And I mean, just one case, when we talked about this app that we’re doing, I was presenting that on another conference, but that was focused on that company and that kind of stuff. And then, I was talking to one guy afterwards, and he called me on…I think he called me on the business day after. And he called me and said, “Thanks, mate. This was really…I was really excited. So I had a use case, and it looks just like your stuff. We needed to do something like this. I just went into product and said, “Hey, I know what we should do. We got like this. I wanna be into this one and I wanna be helping you to build this one good.”
Product loved him. He called me back, he was really happy. And now, they’re working together to actually build this from much earlier point than being the one who’s getting everything dumped down in the knee that he was before. So if you aren’t moving up there, start doing it because you can do that.
Because it’s then we help the business evolve. That’s what I say, at least. Good. And I think I’m up, actually.
Fill the form below to get all the presentations delivered as a single zip file in your mailbox.
byJon Fancey & Matt Farmer
byMicrosoft Integration Team