Saravana Kumar explains his new product Atomic Scope and it’s capabilities regarding security and end-to-end business visibility for BizTalk and Hybrid integration scenarios, which is followed by Atomic Scope demos.
Integrate 2018, June 4-6, etc.venues, London
Saravana: This is a slightly different context to the previous presentation. So far the products what we have, both the BizTalk 360 and the Serverless 360, is more technically focused products. So it was designed for, you know, like enhancing the experience of operation people and support people, more technical people. Like somebody, you know, instead of going to Azure portal, they feel comfortable going to some other portal but it all still technically powered tools. But with Atomic Scope, we are trying to take slightly different directions. It’s basically we are designing something for keeping business users in mind, complete non-technical people and we just wanted to cover the bunch of technologies that falls under this under this [inaudible 00:01:03]. Since people are coming more from integration background, let me start with, you know, a couple of example scenarios so that you get the context of why we need a tool like this.
So this is a typical, you know, like, end-to-end healthcare scenario. So you have a middleware platform in the middle. It could be BizTalk and now it could be hybrid and you have Logic Apps and things in the middle. So you are dealing with a bunch of healthcare providers like, you know, Symphony and Botnet and Cerner and there are tons of external healthcare providers. And a lot of communication that goes through the system, like admitting patients, transfer, discharge and a lot of things back to back. So if you look at it it’s a, kind of, an end-to-end application. And let me take one more example like a typical supply chain, which I think a lot of the European customers with, you know, EDA background might relate to. So there’s any purchase order system, you send a purchase order, you send an acknowledgement back, and then proof of delivery and invoicing and it’s a sequence of steps that goes back and forth to make something meaningful out of it.
So here are some of the, you know, the common challenges what you’ll see. Again, you know, you need to keep it from a business user’s perspective. So, how are you going to manage, you know, that kind of, you know, end-to-end solutions? So, people deal with, you know, exception handling, you know, edit and resubmitting messages and message archiving, security, and monitoring and business processes. So addressing, you know, these kind of challenges using custom solutions requires significant effort. Currently, what’s happening is, you know, lik,e people build something like BAM solutions, you know, coming from a BizTalk world. You know, they use BAM portal to give it to business users. And majority of the time, you know, like, that’s not enough and you end up with some kind of, you know, like a custom-tracking solution, instrumentation solution, and some web-based application. So how many of you got some kind of Asp.Net web application for business?
I think the remaining people are bit shy. I’m sure, you know, like, given the experience, you know, everybody build something like that. You pull data from tracking. You pull data from BAM. You got some of your own custom databases and you pull it and then you have some kind of end-to-end visibility. So, typically in a way, you know, 20% to 30% of any integration solution, you know, like, you spend building this kind of infrastructure to support your business users. Because when you have a health care product or when you have, like, end-to-end supply chain, so you need to see whether the purchase order was sent and did you receive the reply back. So here we are talking, you know, more and more on a more very functional level and on a business process level. So even after spending that, you know, 20%, 30% of the time, you still you won’t get a really polished a solution because it’s not your core competency.
And it always takes a backseat, and you have some kind of a web solution that’s been a user .Net developer to ask him to knock out Asp.Net web application to put a face to your data you collected. This is all, you know, like a common problem. And that’s the problem what we wanted to tackle with the Atomic Scope. You still need to do some kind of instrumentation. This is different to our previous products because the other products are basically plug and play. You just plug your stuff and you get visibility. But here we are getting into more runtime and we expect some kind of development efforts to happen. But the value proposition here is, you know, instead of you spending, you know, like a lot of time in providing this kind of solution, you can use the components and solutions what we provide, and you can reduce time to significantly get the value out of this thing.
So this is how it works. So we have this Atomic Scope solution. It comes with a web portal. So you define your business processes. It’s just a very abstract term. You define it. And the business process can have multiple stages and multiple transactions and you have the properties to track that’s coming from various places. From your side of the things, like, we provide out-of-the box solutions to provide some kind of instrumentation. You can use BizTalk or Logic App or Functions and App Services, whatever. You simply instrument your data just to call the methods in a required way and everything shows up on the portal automatically. So, out-of-the box, we ship these kind of components. So we provide a bunch of BizTalk pipeline components, .NET SDKs and REST APIs and we even ship some Azure Functions. You take the technology that’s relevant for you, maybe if you’re still on the BizTalk side, you can plug in the BizTalk pipeline components and automatically the things will get instrumented.
If you’re on Logic Apps, you can use Functions to you know, record your start activities and end activities and things like that. If you are using custom solution, you can use a .NET SDKs or REST APIs. So this is a very high level architecture how things work. So you can see that it’s a combination of both the BizTalk solution and then the Azure Solution. So all those green dots is where you start instrumenting the code. So you don’t need to do the full solution. You want the instrument stuff, what’s required at each one of those points, and then all the data collection and then visualizing the data in a nice portal. Everything comes as part of Atomic Scope. So let me go through quickly all the features what we have. The first is, you know, end-to-end business process visibility. So the portal gives you, like, the complete visibility of your end-to-end business process.
So this could be, you know, like a combination of both your BizTalk and Azure or it could be just BizTalk or it could be just the Azure side of things. So it’s a very abstract term and, you know, like, it covers both on-premise on the Azure and the hybrid solutions as well. So it’s a fully configuration-based solution. So you define your business processes, transactions, etc. And then you just change the configuration in terms of, you know, what you wanted to collect at each one of the points. So you can say, for example, I’m interested in a purchase order number. That purchase order number could come from either the BizTalk context properties or it could be a JSON or XML in the payload. You can provide the export, the JSON port or export, from where you can fetch it or it would come from HTTP header as well. So it’s all completely configured in the portal. So we provide a powerful search mechanism, message archiving, and exception management. So you can think of, you know, like a USB portal and BAM portal and all those things are getting combined.
And, you know, like you can search…your business user can say, “Where is my purchase order in the system?” And people can, you know, like, they can log in and figure out where it is. So support for things like debatching. Maybe, you know, you have, like, a big batch file coming in and you debatch it and that’d go into multiple little transactions. So we provide support for those kind of advanced scenarios as well. So, authentication and user access policy, it’s, again, it’s at the business process level. So you can say, for this video processing or healthcare YDD processing, you can provide access to specific business departments or business users, and they can only access those kind of things. On top of that, you get a good monitoring story for your functional messages as well. So if somebody didn’t get a purchase order, you wanted to notify, like, within a time period you didn’t get basically like the acknowledgement for your RPO or something like that.
So as I said, you know, this is a slightly different context. So the product is basically designed for keeping business users in mind. So you can think of it like BAM, V Next, so which is scattered for the end-to-end scenarios, both the on-premise and then the server and then the hybrid model. Since this is a fairly new concept, you know, maybe there are some of the technologies that are fairly new, we wanted to show a demo, like, instead of me showing, like, sample application, so this is in production, we have customers using it. So we invited one of our partners from Belgium who’s using this solution in production, and we’ve been developing this for last 15 months and very closely with some of the partners. So we had a lot of feedback and let me introduce Bart from integration team who is going to show how this is really used in a customer scenario.
Bart: Hello. Thank you, Saravana, for having me here on the stage to let the users here show your new product over there. So a quick word to who am I. Just ask Google “BizTalk Bart” and you will find me out and this is, no kidding you, the first link you’ll receive if you do it and some other links you will find me back there as well. Where do I work for? I’m working for Integration Team, which is a consultancy company in Belgium. And we are focused on integration scenarios on top of the Microsoft stack. We have a lot of people not doing anything else than integration with BizTalk with Azure on top of Microsoft. And these days, I’m now working, most of my time, for a customer called Van Moer, which is a logistics provider based on the Port of Antwerp. And they are doing a lot of logistics for their customers. So the business scenario and the challenges we now have at Van Moer, they are having a lot of trading partners that they need to deal with. And these trading partners are sending a lot of data in many different ways.
Van Moer is growing fast and we need to have on a regular basis, we need to add more and more partners to the platform and they are now heavily investing in a stable and reliable IT platform that is supporting all these business scenarios and all these cases. So what do we have today? Today, we have this typical scenario where we have an on-premise BizTalk server and all these trading partners are connecting to this BizTalk server via the known protocols, FTPmail, Ace 2, whatever, you know, like the old days, let’s say. So, because we are doing this and because Van Moer is that much growing fast and they have a need for more SLAs and for tighter SLAs and we need to deliver them in a more high available environment. So we are thinking how can we do that? And we came with the solution that we want to go in a hybrid scenario to gain of the higher SLAs of the Microsoft Cloud to support at least all our public facing endpoints and all our endpoints that are now used by all these customers that at least that one have the advantages of the very good SLAs on the Azure cloud.
So, that said, the hybrid scenario, it’s very feasible, but what will we do with the view for the business users on all the data that is coming in and out in the system? So we are thinking, what will we do? We will go further on with BAM, or will we built a custom solution? What will we do? And actually, our journey here, and with here I mean here in London and probably not so far from here last year, where I met Saravana and his team and I was telling them about our challenges over there.
And actually they then tell me a little secret on that moment of time. And they said, “Oh, actually, we are building such thing. We are working on top of this.” And they requested me, “Do you want to go with us in this journey to make it a product?” So we decided to join the effort and we decided to go with the story together with 360 team. So from our architecture point of view, where we have already made the choice to go in the hybrid scenario, we now made the decision to add Atomic Scope on top of this hybrid scenario to be able to give the business their view on the data, also not only in the on-premise mindset, but also in the full end-to-end visibility of the hybrid solution we are building.
So what’s for me the added value from Atomic Scope comparing with the current BAM solution we have? For sure, the number one is, for me, the hybrid, that it’s capable of doing whatever BAM is doing in the BizTalk scenario, that’s capable of doing it in a hybrid scenario too. So it’s also delivering these endpoints and the events are being visible that have been tracked in the Azure components. Atomic Scope, as I will show you in a table later on is also a lot better in handling with tracked messages, because that was also a point of comments we already have from our business using the BAM portal of 360 today.
We really want to see the message that is coming in from our trading partner. We really want to see the full content of this messages. Another topic is the business exceptions. If something goes wrong, you can track it BAM, that’s not a point, but to visualize it in a more visual way that something is going wrong and the huge batch of the messages are going right. Also on that topic, the team of 360 has done a good job to really make it really easy to see in the huge bunch of messages, the few messages that are going wrong. They also built some dashboarding on top of it to make it even visible in one simple view.
Next one is the easy user security. As you all know, probably, if you’re using in a BAM scenario, you can give users access, but it’s more hard to do to get it all right, and in Atomic Scope, as you are now already used in the BizTalk 360 product, there is also a lot of attention to how to give access to specific users, to specific processes. And then the next one is in the portal itself. A lot of you guys work with BizTalk and a lot of you will know the BAM portal of BizTalk out of the box. It’s not changed a lot in the years. So we are now even already using BizTalk 360 BAM portal to have it in a more modern way and also the Atomic Scope portal is built in the same ideas and the same stuff.
So what’s my scenario? I want to show you right now today. It’s an logistics import scenario where a training partner of our customer is actually requesting with a message, meaning EDI Message, “Okay, I want to have an order where some stuff needs to be transported from the customer side,” for example, “to the warehouse locations at the Van Moer site.” So the EDI Message will come in and there must be an order created in their ERP system called iTos and transport must be created in the ERP system so the business at Van Moer will be able to see in their own ERP systems what needs to be done.
So is it visible? Saravana also told us it’s now a different way of working. We need to do something. It’s not the plug-and-play thing anymore. So we need to create first our business processes on the Atomic Scope instance. And for example the for the demo we’re using here, I created an iTos logistics import business process that’s capable of doing all these trackings. And as you can see, here is the process, and I have here defined a lot of stages, a lot of transactions that I’m expecting as a developer of this business process that should be happened.
So when we have done this, we also need to configure here all the stuff we need to be able to see, all the stuff we want to see in the portal, all the stuff that we want to track. So all this configuration is done right here in the portal of Atomic Scope. This is also easily being changed. If you compare it again with the BAM, if BAM wants to add another column and to deploy this to all your environments, it might be a hell of a job. Even just renaming one fielding BAM might be a hell of a job. Here is just some configuration changes in the portal and from that point off, the new interfaces that are being processed will take this change into account. So what I’ve prepared over here, I have here, I sent over EDI messages. And just to show you that all these new stuff that you are talking these days in hybrid scenarios and in Azure, they’re still capable of delivering the old stuff, let’s say. What you see here on the screen is just artifact messages that have been already been there from the early days.
So even that one is still working. As you can see, I’m now expecting from my trading partner that they post this artifact message into…actually API management as my public-facing endpoint. As you can see here, my BizTalk environment, I’ve now stopped my receiving host, so nothing will be processed by BizTalk anyway. So let’s now imagine that BizTalk is not doing anything at all because it’s just not available. But, as I said, the customer is expecting us to have a high available system and they want to send us messages anyway.
So let’s kick here, the send button and hopefully something will work here. So now this artifact message is being sent to Azure API management. And from there, I get here to 200. So that’s already nice. And if I go in now to my Atomic Scope instance, then I can see that indeed the message is already visible in our tracking system. If I want to scope my tracking environment, Atomic Scope, only to the BizTalk environment, I was not able to do this, because BizTalk has not processed anything at all.
This message is just only arrived at Azure API management and the event is pushed out from there to the Atomic Scope instance. So I can already see something has arrived in my system. If I click on the details, you can see only the first step is already being executed. I have received something from Azure. So let’s now start up BizTalk again. And now this message will be processed further onto the processing chain. And then we can see that some more stuff will happen. Now you see here, it’s becoming orange. So this specific instance is now indeed in progress of processing some more stuff to the ERP system, and that’s undone by the BizTalk environment. So if I click again, you can see here that indeed the order is received and this is a tracking point in my receive location, and the order is being processed as we speak to the ERP system further on in the chain. And at the end, we hope it will be fully completed, of course. So what’s more in here, like I told you, one of the other values for me, Atomic Scope versus BAM was also the tracked messages.
And as you can see here, if I click on a specific stage and we scroll down a little bit, then we can see indeed here just one click away view me the message body, and as you can see here it’s exactly the same artifact as I’ve shown before in the post [inaudible 00:24:34], indeed that party has sent this artifact message to our system. So now the full process is done. And as you can see, all these different steps from the Azure API management to the BizTalk reciprocation to some different steps in the orchestration at the end send port to the iTos environment, they are all being correlated and they’re all having these boxes in all these different steps that have been executed. Also the messages that then has been sent to our ERP system, also that one can be tracked as full track messages and you can see this artifact is converted to an XML and it’s sent that way to the ERP system further on the system.
Man: I have a question. Can you also change the file off-stage and release it [inaudible 00:25:32]?
Bart: The question was, can we change the file and then…
Saravana: The hand written resubmit is not there at the moment, but that’s in the pipeline.
Bart: So I also talked about the view and exceptions. So I prepared here some more stuff. So in case something goes wrong, you immediately see already here, the one that are in red has some business exception. So if we go to the details here, something went wrong with the specific case. If we then click to this specific step, then you can see here the exceptions that has occurred. So on the header level, it’s just saying, “Okay, order number has failed.”
But if you go to one step further on, you can see the full API exception that occurs on the ERP system, and we can really see here in more detail why this message has been failed. So that’s really nice. Also on the first overview, you can see here how many messages we have received and you can drill down to the one that fails. You can drill down to whatever you want. Back to the slides for a while. And as a partner, of course we want to give some feedback to the product team also. And I’ve also listed here all my future improvements that I really want to see happening in the product later on if they are developing further on Atomic Scope.
So there is an alerting engine also available in the product where if something goes wrong and you can see it on these red lines, that you can set alerts on top of it. So if something goes wrong, that you immediately get an email that something went wrong and that you can then go to the portal and actively have investigation what might be wrong and how you can handle it. Another topic is that the alerting engine today is more a basic one, just monitoring on whatever what’s going wrong.
But actually, I would like also to see some more data-monitoring concepts like we have seen also in BizTalk 360 and in Service Biz 360 that we can also have alerts if something is not happening. If we do not receive any message today, that’s also a problem. So some more rich stuff in the alerting engine. As of today, it’s not possible to have a tracked message in whatever is going on on the Azure site. In my case, on the API management or also in the Logic Apps there are custom connectors available from the from the 360 team that you can just drag and drop in your Logic Apps to enable all this tracking.
But you cannot track the full message body from there as it is of today. You can do it in the BizTalk pipeline components and that’s a result you have seen in the demo. And also the dashboard I have shown, where you have the overview from what’s going well, what’s going wrong, and how many processes executed from which business process. It’s now more basic and it should be nice to have also created our own stuff, our own…that we can give answers on questions from the business users itself that you can have it as a widget, import it on the dashboard and then give the business in that way a more dynamic view on their business processes. So to end it, if you have got any questions, feel free to find us. Maybe also Saravana can come back here to answer together with me all the questions that might be there.
Saravana: Sure. Thank you very much. And hopefully, you know, we’ve given a picture of, you know, what we are trying to address with the Atomic Scope. And now I think probably, you can correlate yourself, like, this is a common problem and we all build our own custom tools to achieve it. So it took us a long time. It’s about nearly 15 months’ effort. So as Bart was saying, like, when he was speaking to us in the last integrate, that’s when we already spent about three months on the initial POCs and prototypes and how we wanted to achieve that. And then, we heard feedback from few partners like this. And we worked very, very closely with not only partners but with some big customers as well. Like, throughout the year, we’ve done with numerous workshops, presentations.
And we’ve tweaked the product, and we already have done two major releases. So the first version was released in March, and then we released a version just before integrate, version 2.0 is before integrate. So now the product is in a pretty usable state, usable state in the sense that, like, it covers a wide variety of use cases but we are, you know, aggressively now we have the foundation right. Now we are adding more and more features, like what Bart was mentioning on the last slide plus features like edit and resubmit. And those are all real business problems, what we people from integration world expect on a daily basis. So that’s what it is and if you want to have, like, you know, more in depth presentation or if you wanted to discuss, you know, you can come and speak to one of us outside here. Thank you very much.
Fill the form below to get all the presentations delivered as a single zip file in your mailbox.
byJon Fancey & Matt Farmer
byMicrosoft Integration Team