Sandro Pereira shares valuable tips and insight drawn from his experience as a BizTalk Server consultant, which relates to making use of patterns, naming conventions, logging and tracing.
Integrate 2018, June 4-6, etc.venues, London
Good afternoon, everyone. I’m Rochelle Saldanha and I welcome the client relation and technical support teams. And I’m here today to introduce Sandro Pereira, who really needs no introduction, in between his busy activities like work, blogging, traveling, speaking at events, parenting a newborn, and most recently, battling a stomach ailment, too. How does he do it all? He has prepared a talk today called BizTalk Server Lessons from the Road, and without further ado, let’s listen to what he has to say as he chronicles his experience as an external BizTalk Server consultant.
Sandro: Thank you.
Rochelle: Thank you, Sandro.
Sandro: So yeah, again, my name is Sandro Pereira. I will not speak too much about me. You have time to read my slide, but yeah, I’m this guy. This is my blog. You probably already have been there. And I work at DevScope, a small company in Portugal that I managed to be a sponsor of this event, a small one, a [inaudible 00:01:09] sponsor, but yet, still a sponsor. What I like to do at DevScope is I work not only with integration but the full Microsoft stack. And we actually are four MVPs at the moment in the company and one occasional director. So I can spread my wings and talk with other guys, other products, and have a better view of challenge that our client faces.
And of course, I also bring gifts. Last year was stickers, this time I will have a free license of PowerBI Robots and Tiles. They are two products that you can connect your PowerBI reports and dashboard and share it with your teams in Outlook, Powerpoint, or in the big screens. So Tweet and you will get a chance to win a license.
My talk is about real lessons from the road, some experience that I have during these years. And I actually created a Windows Form asking what should I speak about, with a nice flow behind, and connecting Windows Form with PowerBI using Microsoft Flow. And you guys decided my talk will be about real case implementations, okay? So I prepared eight stories for you. The first one is, again, we have this sentence in Portugal that if your favorite tool is a hammer, everything that you can see, problems that you can face, you will try to solve it with nails, no?
The same applies to technology, you know? Sometimes we go to a client and the client asks BizTalk, and they say, “We need to do everything with BizTalk, or Logic Apps,” no? You have to do everything with Logic Apps or everything with BizTalk. And that’s not really the case, you know? There’s a lot that you can use in an integration scenario, you know? For example, do not do ETL process in BizTalk, okay? You don’t have performance there.
And sometimes we say BizTalk is bad in low latency. I’m working a lot with API open banking, and we are using API Management. We are using BizTalk components, and we are trying to use a lot of integration stack. Then we are also starting to use Logic Apps inside. Logic App is even worse than BizTalk in low latency scenario, so we have you choose another option, and you went to API apps, for example. Logic Apps is still using there, but in some…most non low-latency performance implementation or process, no?
If we are speaking about open banking, expect, like, 200 milliseconds response time. It’s difficult to achieve that with Logic Apps, and we actually achieve that using an API Management and BizTalk components. So depending on the scenario you should use the right tool for the job, you know? We actually have glanced at who was using ETL process with BizTalk. Of course, they have spent, like, four hours. We are putting in SQL and it does the job well done and faster.
You see that in the first day John asked a customer to be explaining they are using Logic Apps in their case, a good case for them. It was nice. In the other case they are using Service Bus and BizTalk. It’s also working nice, you know? Depending on the challenge you should use the right tool for the job. It’s not BizTalk or Logic App, it’s the integration layer. The integration layer is a huge amount of pieces, and you should have this in consider, no?
Second, if you really believe that these tools are the best that you can deliver for your clients, you should run and keep pushing from them, okay? Believe in yourself. Believe that your decisions are right for your customer, but do not go, like, “I am Microsoft. I will deliver all the tools from Microsoft to the client.” No, for me, at the end it’s client that is paying my work. I need to deliver a good option to the client. If it’s Microsoft tools I would be happy because I like and I’ve only used Microsoft tools. But in some cases if I need to go outside and use an external tool, we use it, okay?
But if you like these tools you should fight for them. And fortunately, Microsoft did good work, you know? They created this concept of enterprise integration, and now we are in the leaders of the business, no? And some of our competitors need to find another excuse for not being there, no?
So lesson three, GDPR, you know? This is killing Europe. Everyone is concerned about GDPR, and this is all about communication, consent, access to the data, rectification, profiling, and warning, and erase data, and sensitive data, okay? Paul onstage told that BizTalk with Feature Pack 3 it will be GDPR compliant. That’s only marketing stuff, because BizTalk, in its essence, is already GDPR compliant. Message will come in, message will go to MessageBox, be consumed, go to the external server and it’s gone, okay? We don’t keep information. We can keep tracking but the jobs will clean that data after a while.
So if a client wants to erase the data they will be erased automatically because BizTalk will not keep the data. We are the middleware, no? We are not in the business of keeping the data, so BizTalk, in its essence it is GDPR compliant, okay? But your applications are not, okay? This is a different story, okay? So we are in the middle. We are subject to middle-man attack, no? And if you are subject of that someone can impersonate another person, no? So we need to be careful. Stop configure password in clear text, no? Everyone sometimes is using config files or BTS config file to use passwords, and these can lead to have access to sensitive data. So be careful, don’t do that.
Reduce the number of people that can remotely access to the machines, because if they remotely access the machine they have access to your environment. They can check for [inaudible 00:09:24] logins, blah, blah, blah, and a lot of things. So if you remove that access, limit to the access of the persons, you will be better prepared in terms of GDPR. And you can actually use tools like BizTalk360 to allow you to do that. You completely remove the access to the environment, no? Stop exposing services in an anonymous way. Who here has internal service in BizTalk that’s anonymous? Almost everyone, no? Don’t be afraid, me too. Stop doing that. At least do some authentication, because it’s anonymous. I can [inaudible 00:10:08] requests and I will get access to sensitive information, no?
If your process already has sensitive data, reduce message body tracking. So we will not track that data so you will have less problems, no? Avoid using files, FTP adapters, and start using secure channels. For example, I have a client that use everything is a file, file, file. If I have access to the remote machine I have access to the files. Use a queue. You can implement some security. Use other stuff that it can be secure. File is not really secure. FTP is not really secure, no?
So SQL supports Always Encrypted. If you are consuming or pulling data from SQL you can actually encrypt the columns and you can access through BizTalk with that feature, no? So BizTalk will have all the features for you to actually implement secure applications. And please, stop logging everything. Every time I open a project I see some custom logging to a log file or to event viewer with full message logging, you know? It’s like that concept, what do you want to log? Log everything. Well, then we will see later.
Stop doing that, because if you are doing that you will probably break some…you will log some sensitive information, and people have access to that. You will probably be in a tough situation. So BizTalk is GDPR compliant but your application needs to be careful about that, you know?
Lesson 4. So I asked, how many of you are installing Feature Packs? And most of the response was yes. And this is a good thing. I really like Feature Pack. This is the new roadmap for BizTalk. We don’t actually need the new BizTalk version because we actually are, each time, three, four months, okay, this is BizTalk and Microsoft is releasing a Feature Pack. So this will bring new capabilities. You don’t need to wait another version of BizTalk to have the new features.
Microsoft tells that these are non-breaking features, okay? Sometimes they will break your applications at least. Sorry, Microsoft. So this is a real case scenario, you know? Behind this scenario there are two things, one bad, one good. So I don’t know if you already face this problem. I have a recent project that was integrating Siebel, and this is the aspect of the Siebel JSON message. They put a lot of empty space in the names, or in the elements of the message, no? I don’t know why. The problem is, like, out of the box with default installation BizTalk will not like that. It will fail to generate the file because of the empty space. And by default installation I actually have factored the Microsoft code, and Microsoft is pointing to the Newton JSON framework but it is an internal Microsoft Newton JSON DLL, sorry. And they are porting for version 4.03. That is, of course, in GAC, okay?
So for me to be able to process that JSON file I have to go and recreate that component with some escape for removing the empty space or replace the empty space so I was able to consume and process that JSON file. With Feature Pack 2, Microsoft did a great thing, you know? They removed the JSON framework…DLL, sorry. They are now creating a BizTalk.CommonJSONExtension, and they are using the generic Newton JSON DLL. And this is more than likely code that they are using, so very simple stuff. And it actually solved that problem.
Now you can process and generate that JSON schema file, but unfortunately, Microsoft removed the version 03 from GAC. Now they are using version 010, so my code was also using not the Newton JSON DLL that Microsoft was using so my code start to fail. I just had to copy and paste the DLL and everything was working again. So please, Microsoft, don’t remove nothing from GAC. GAC is there for that. Just create a new version and add that there, no? Just keep it and you are safe. They are known breaking, though. So be careful. A Feature Pack is great. Again, don’t get me wrong, they are great but sometimes they aren’t really non-breaking stuff. They will probably break something. So test it in your queue, in the environment, every environment, and then go to production safe, okay?
Lesson number 5. Another question. Do you normally install Cumulative Updates, you know? And most of you are saying yes, and that’s extremely good, no? So some people ask installing Cumulative Updates is good or bad? They will break something or not. Feature Packs already broke my code. Cumulative Updates never broke nothing to me. So I’m safer installing Cumulative Updates than Feature Pack. Again, don’t get me wrong, you should install everything, okay, because I really like new features, okay?
So Cumulative Updates are good, and I will only give you one example. Everyone know what is orphan instance, no? They are there in BizTalk. They have never gone away. They appear sometimes. I don’t know why, but they appear, it’s like a mystery in BizTalk. And two, remove it. By Microsoft conditions you should stop your environment, run the terminate tool that is now BizTalk Health Monitor, and execute some procedures, and you are gone, you are safe, no? Done. Orphans deleted until they appear again, no?
So I need to stop again my environment. This, for me, is horrible. I don’t want to stop my entire environment just because I have [inaudible 00:18:02] service. I will never stop it. I use other strategies, but that’s me, okay? I will not share it because it’s not official. But lucky for us, Cumulative Update 1 from BizTalk Server 2016 or 2013, R2 of 2013 already fixed that problem for us. So you no longer need to stop your environment to kill DTAs, orphan instance, okay? They will be gone from time to time. It’s actually quite good, no?
So I don’t know why Microsoft didn’t make any buzz about this. They were saying, like, “Yeah, now we have new features, like app inside, blah, blah, blah, blah, blah, blah.” Great stuff, don’t get me wrong, but I prefer that one, okay? At least it’s safer for me. So yeah, you should always, always install Cumulative Updates. Be careful, again, for safe side install in dev environments, Q&A, and then go into production. Again, I never faced a problem with a Cumulative Update before.
Lesson 6. So if you think about electronic invoice, or EDI, or an invoice, you think about B2B, EDI, electronic message, and so on, and so forth. And that is really the perfect scenario, when you are laying down on the beach, drinking margarita, something like that. Because my clients use paper, PDF, you know? I have a lot of them using EDI, don’t get me wrong, again, but asking to a small company that they need to deliver, I mean, my message in the EDI format, they will not do that because they will not have probably the money to do that. And sometimes even big companies need to work with small clients, and they use a PDF.
So I went to a client that it was processing invoice, a PDF invoice inside a pipeline component of BizTalk. Let’s say that’s not really a good story, no? BizTalk is not an OCR platform. So let’s see them now. How do we, as DevScope, implement and solve the problem of this client? This was something that they were processing, and so normal invoice or that people…or try to do it with BizTalk where most of the time we fail, or they are processing manually, insert in another system to be generated. We actually have…you can use, of course, any OCR client program that is in the market.
Fortunately, in DevScope we have, like, a small and powerful OCR that is called SmartDocumentor, and basically it’s more like BizTalk in the generic terms. You can set where the message will be, where the file is coming, from scanner file, or other stuff like that. And you can create a flow saying, what’s the process of that file, okay? Go to revision, go to automatically integrating the system, and so forth, and so on.
So in this case I’m just putting my PDF inside the file. It’s not that one, it’s this one. And it will be consumed by the OCR client. It will process and try to generate the data that is inside. Of course, you can also go to the scan document station, and it’s open, and you can actually go from the Scan Station itself and the process will be the same. So there are, like, multiple [inaudible 00:22:53] locations in terms of BizTalk language. And after that it will go for a review station, and the file that I already processed, it automatically detect all the field that I can specify. Of course, for being the safe side I only specify little ones, but these can be more, and more, and more. Of course, if it’s a real case scenario we’ll have more data inside.
And you see that I have an invoice number selected and it’s highlight what it’s picking of that file. It’s important when you have an OCR system to have, like, a revision. Of course, then you can create some rule saying that if you have 99 confidence of that success of that OCR, it can automate and skip that review station. And again, we can process and say, “Okay, it’s fine,” and that’s it. In this case we’ll go to a PDF XML file in the folder.
Of course, I don’t have BizTalk installed here so I pick up to a different machine. So basically this will be the aspect of my file, something like that. And of course, I can then pick up, choose whatever I want. This can be a [inaudible 00:24:32] server, whatever, no? I have it digital in an electronic way. I can then send to my internal systems, okay?
So don’t try to do crazy stuff in BizTalk. BizTalk is not a tool for every scenario, you know? If you do, like, OCR, you should use other tools and then communicate with BizTalk, and then BizTalk will process that message, okay? It’s better. You give more and more quality to your clients. And of course, we see just a little of that. This is more a real case that we implement.
So you know that we have more detail, more lines. You can customize, you can do that stuff, and you can process that stuff. I also was able to implement something like that. We have a PDF file with a list of records, and unfortunately, there was a guy looking to do that PDF and insert it manually, each line in the system. We end up create, for example, a CSV file, just because a CSV file was easy to, in Excel, validate every lines and the totals. And then we can actually process that file, okay?
Lesson 7. Sequence Convoy. How much of you have already implemented Sequence Convoy? That’s a lot. I never implement the Sequence Convoy, never, not an aggregate and not a single time. I always try to move from that patterns. They pose a lot of problems in BizTalk. They will probably produce orphan message. You will always have an active orchestration, never stop it. It’s always there. You produce latency problems. I don’t see any benefit of having these type of patterns, to be real honest with you guys.
And you may think, “So how can I implement Sequence Convoy?” You just need to look and try to redesign in a different way. For example, I implement Sequence Convoy with a design that I call a [inaudible 00:27:20] process design, or a sequence process design. And with this type of approach I actually can create sequence in BizTalk. I can scale. I don’t no longer have a long-running orchestration that keeps alive forever, and ever, and ever. I don’t have zombies, or other crazy stuff like that.
So let’s see, this is just an example, okay? So this is the aspect of my Sequence Convoy orchestration. It’s not the sequence, right? It’s okay, nice, awesome. Thank you, BizTalk. But you see that it’s a normal orchestration that receives a message, process a message, and send to message box. I then will have other [inaudible 00:28:22] orchestrations that consume each type of message that I want to implement and do the right job for that process, okay? Safe, yeah? Yeah, there you go.
So you see that actually this is receiving from SQL. I can process different types of messaging, here’s XML, JSON. It can be text file, you name it, okay? I will implement two, this was a proof-of-concept, and then I process and I send to message box, directly to message box, no? If I see that you have here opened my port, SndToMsgBox, and you see that is a Direct how to message box, okay?
Please don’t be like me, you never name it. Operation one, give proper names, okay? This is a proof-of-concept. It’s not an excuse, I can do better than that, but don’t do that, okay? You see that I actually have good names, Catch Persistent Exception, Promote, StartOrchestration. Keep it that way, okay? Good for documentation, no? And again, I will have something like, for example, CreateClient. That’s, again, don’t do these kind of orchestrations, okay? Just construct and send it, don’t do that. Get proof-of-concept, and don’t do the expression one also. But yeah, you know the name. I’m consuming from the Message Box. I’m doing something and send to other [inaudible 00:30:12], no?
What [inaudible 00:30:15] my strategies, I have…let me close this one. I have a concept of a queue that is, in this case, is a SQL table. And I will actually have some metadata that I will add it in that table. So what is the process? What is status of the project? This channel and entities are different concepts. It can be different, okay? I actually have the body of the message inside here. This should be encrypted, but, again, it’s proof-of-concept, create_time [SP]. data_time [SP], number of [inaudible 00:31:03], and so on, and so forth. So I can really do a lot of stuff here.
And I have a status here. The N status is new message. The P is processing, and the I is integrated, and the U is undefinitive, unknown. I don’t know that process so if I process I will mark it this, I don’t know this type of message for someone to configure that process or know why the hell it’s there, no? So if you see that I have, like, 1, A, 2, A, this is a CreateClient, and this is up to date. The one is the type of the message, the entity is, like, the main owner. So in this case I want to process the creation of the message, then the update, and then the second update, and then the second update in a sequenced way.
But you know that I have the 1B and 2B, so I want also to process at the same time 1B, and afterward 2B because they are different entities. So I want to go sequence by entity, okay? I can do that. So let’s, again, my port, stop it. So let’s Enable. Let’s go through [inaudible 00:32:45] and then you see that? Two message [inaudible 00:32:49], it’s processing. This take a while. Second one is being used, processing. It’s already integrated. And then only missing two, no? Third. And then last one. I could improve my pooling, but yeah, done, okay, in a sequenced way without any convoy scaling. Awesome for me, no?
And if someone complains, “Hey, Sandro, I didn’t get the last update.” No worries. Check, let me check. Edit. It should be this one. New. Processing, integrated. Sequence, re-entry, no…let me check my BizTalk Admin Console, F5, always F5. It’s there, no? Nothing is there, no? Nothing is live. My environment’s clean, okay?
So sometimes you need to be smart. You need to sync in a different way to implement these common patterns, okay? It’s not because I wrote in a mapping book that is the pattern you need to use, you should follow. You need to think outside, and you need to think for yourself. Someone wrote a long time ago that Singleton Orchestration was the way to go, and now everything is falling. So sometimes you need to be smart and say, “No, I will do different ways,” okay? And it actually can do and improve.
Last lesson. BizTalk REST support. Who here is using REST in BizTalk? Sorry to tell you guys, need’s support also. It’s there. You can do good stuff there. You can do basic stuff there. If you want to really do stuff in a proper way go and customize, okay? For example, it has an optional parameters, or query parameters, no? It cannot do in BizTalk. By default you cannot. Being empty is not the same of being optional. Being empty is mandatory but it can be empty. It’s not optional. If I have 20 query parameters that I can have a REST, I don’t want to be 20 empties. I went them to be short. I want the right data to be sending in URL, and this is impossible to do.
And you know that about resize windows in BizTalk? Yeah, we have it. They forgot to resize the important stuff. That is binding, no? Yeah, thanks, Microsoft, again. You will see my email complaining about that. They are nice guys, they will do it in the Feature Pack 4. I know for sure, otherwise I will be next year here telling the same story. So actually I’m working in…I have five minutes, okay? I’m working at the moment in a project that is, most of them, JSON to REST, REST to REST, JSON. And some case scenarios it’s like a passthrough, others have orchestrations inside. And we end up creating a lot of pipeline components in the receive side and in the send side.
It’s a smart proxy. Then we specify the catalog of the request that we have inside SSO. So every request that come from BizTalk, it goes to SSO. It has heavy cache implemented to be fast. We check if they are allowed to consume, and we have a static port on the send part with the dynamic behavior inside the pipeline component, okay? Why? Because it’s crazy to create a dynamic port in REST. It’s crazy. The amount of work you have, it will fail.
So you can actually create a static port, specify [inaudible 00:38:14] , there’s security, whatever you want. And then in the pipeline say, “This is a dynamic port through,” and then specify the URL. And then you can do awesome stuff with C#, no? You can actually create empty or optional parameters. You can modify the [inaudible 00:38:40]. You can add [inaudible 00:38:41]. My solution is all about pipeline components and REST capabilities, and it’s working like a charm. I have, like, 200 milliseconds, 100 milliseconds response time, and it’s awesome. So if you complain about low latency with BizTalk you are doing something wrong because BizTalk is good also in low latency, no?
Last question I asked to the community was, “What’s your favorite ‘piece?'” And BizTalk is there, no? I was surprised. I was expecting more love from Service Bus, so it’s BizTalk, Logic Apps gaining a lot of love. Functions, somewhat both Flows also. It’s great, yeah, it was you again, no? No, actually it was about [inaudible 00:39:41] Flows. I had response two times Flows and BizTalk.
I love BizTalk, okay? It’s a nice platform. You can use it, of course, on-premise. I will use it definitely on-premise. I will extend my BizTalk to the cloud using Logic Apps. API Managers is an awesome tool. I can now expose my REST service or my SOAP service. You know that when you expose a SOAP service to the cloud you have this annoying network team saying, “I cannot open connection. They have the security,” blah, blah, blah. Now I can expose to the API Management. I don’t need to talk again with these guys. I just need to configure the API Management. I can expose my BizTalk service to the cloud, okay?
If I want to connect SalesForce, Dynamics CRM, SharePoint, I will use Logic App for sure, no? Logic App is good for many stuff, not for all. API Management is good for a lot of stuff, not for all. BizTalk is good for a lot of stuff, not for all, okay? Just be sure. And that’s it. Thanks, and I hope you enjoyed the session.
Fill the form below to get all the presentations delivered as a single zip file in your mailbox.
byJon Fancey & Matt Farmer
byMicrosoft Integration Team