Paul Larsen and Valerie Robb introduce what’s coming in BizTalk Server 2016, revealing when support will end for BizTalk Server 2013 and BizTalk Server 2013 R2. They highlight how the BizTalk Server Migration Tool will make migration to BizTalk Server 2016 easier.
Integrate 2018, June 4-6, etc.venues, London
Good afternoon. I’m Bhavana from BizTalk360. I work as a Customer Relationship Manager. I’m here quickly to introduce the next two speakers. So we have Paul Larsen and Valerie Robb. So Paul Larsen is a Principal Program Manager for the Microsoft Corporation in Redmond. I’m sure all of you know him very well. Paul leads the host integration engineering team to deliver and support Azure BizTalk SQL Office and Windows connectors to existing IBM systems.
Today, he’s also joined by Valerie Robb. Hi, Valerie. Valerie has worked with the Host Integration Server product since 1998. That’s a long time ago. She has experience working with HIS as a support engineer, escalation engineer, tester, and program manager. Her passion is helping the Enteprise customers use the Windows platform to gain access to programs and data on IBM mid-range and mainframe computers. She also loves writing Cobol programs.
Valerie: Yes. I do.
Bhavana: So together they present to you today Hybrid Integration with Legacy Systems. So please welcome both of them. Thank you.
Paul: So a little bit about introductions. Probably, I should have said something. Hey, you know, change is important and we embrace change at Microsoft. And in fact, Valerie and I have embraced change for eight months now. It seems longer. Maybe, it’s nine months. We’ve been co-owning…oh, thank you very much…both the BizTalk Server and Host Integration Server, and the line of business adapters, and Enterprise Single Sign-On. So all of the stuff that is tied to the BizTalk Server on-prem license, that’s Valerie and I now. So we’re the PMs for BizTalk. Woohoo.
The way I see BizTalk, BizTalk is the integration server from Microsoft. It’s an on-prem solution, and so it connects to all the Line-of-Business systems that matter. In fact, the Microsoft IT, as you might know if you’ve attended this event in the last couple of years, is one of the biggest Microsoft customers as well. We’ve got a lot of large institutions, organizations that run BizTalk for their mission-critical Enterprise Tier 1 applications. And many of our customers are running BizTalk Server and Azure IaaS/VM infrastructure, as well as the Amazon web services and other clouds like IBM’s using infrastructure.
It connects to the top five Line-of-Business systems. Microsoft, of course, like SQL server, is a data store, also, IBM, TIPCO, SAP, and Oracle. And one of the key things that it offers is technologies at all levels of the stack. So it has orchestration, kind of a workflow with the rules engine. It has a great tracking of messages, has adapters for all the important systems, and it has XML document interchange.
But we also have built-in to BizTalk a pipeline for JSON to XML, XML with JSON. So it’s modern as well in terms of the message format that it supports. And then, key at the very bottom there in the center are some accelerators that are combinations of templates, XML templates, and adapters, healthcare, [inaudible 00:03:31], and Swift, things that you actually can get anywhere else.
We made a big investment in BizTalk Server 2016, and I’ll call out a couple of things here. One in particular is an add-on to BizTalk Server, which is the Logic App adapter. I used it earlier in the demo in that first phase of the keynote demo to get a credit card authorization and hold. We used that adapter. Also, I’ll call out there a bunch of work we did around security in terms of new encryption schemes for authentication and encryption.
Another large aspect of BizTalk Server license is Host Integration Server for IBM platform interoperability. It has a set of bidirectional components or services or adapters. Some of the adapters run in BizTalk, some run in SQL Server, others are just .NET clients and servers. And where we differentiate in terms of Host Integration Server and their adapters, was this third party things that you might get from companies such as MillSoft, is we actually implement the IBM adapters all the way down to the wire. So we write our own protocol clients and servers either by devising the protocols using IBM documentation or license or using industry standards like we do for DB2.
And there was a big investment in HIS 2016 as well. In particular, I’ll call your attention to the right side. We developed our own MQ client, and that was primarily because we couldn’t get a licensed MQ client for IBM that can work in Azure for Logic Apps. So Logic App connector for MQ uses our own Microsoft-devised MQ protocol client. We took that same client and made it available to your use in .NET framework, and also as an underpinning to the new not quite fully but substantially rewritten MQSC adapter. One of the most popular adapters in BizTalk is the MQ client.
So can use IBM client or the Microsoft client now. And then, for DB2 and [inaudible 00:05:30], we wrote an entirely new adapter as well. And it’s what I’m using today in my demos.
Really important to know. Lifecycle. Approximate lifecycle for BizTalk Server 2013 R2 and 2013, the two of them, is not very far away. Just over a month away, so about five weeks. So it’s really, really important that we talk to you as Enterprise customers and partners and make sure we’re starting to plan if not move your workloads from the older BizTalk to the new BizTalk.
In order to make that possible, our MS IT department, who I said is one of our largest BizTalk customers, has developed a toolkit, a migration tool. And we have it available in a built version on one drive and also the source code in GitHub.
It really does help to migrate from an existing BizTalk Server 2010, 2013, 2013-R2, to 2016 installation. It’ll dehydrate the artifacts, move those to a new target server, rehydrate the artifacts, get you up and running a lot quicker.
In the audience today and here all week, we have a couple of development managers, Sanjeev and Shem. Are you guys here? You wanna stand up for a little bit? There’s Sanjeev and Shem’s in the back. Hello. And collectively, they own BizTalk Server,Enterprise connectors, key portions of Logic Apps. And so they’re our engineering guys. They represent the engineering team. Also, we have some professional field engineers. Samuel? You out there?
Paul: Here’s Samuel Casper. Do we have any other of your co-workers like [inaudible 00:07:03] here? Any of the other guys? So we have some of our support engineer… Yeah? We have some of our support engineers here as well. Thank you, guys. And really the focus for the product team and the PMs and our field and our partners is just this. Helping customers migrate their existing workloads and deploy brand new workloads on BizTalk Server 2016, Host Integration Server 2016, Line-of-Business adapters. So be sure and talk to us and tell us how that’s going and see if we can help you.
Similarly, Host Integration Server has a lifecycle. It’s a little bit later because it shipped after BizTalk Server 2016 or 2013 rather, and it’s due out just in January of next year. Important thing to note. You can use the HIS adapters for BizTalk like MQ, DB2 with the new BizTalk server or the old BizTalk server. We have customers doing that as well. But it only gives you like another six months.
All right. So what have we been doing since we shipped BizTalk Server 2016? Continual improvement, internet speed, ship, ship, ship, ship,ship. About every six months, we do a Feature Pack. Last year, we talked about Feature Pack 1, where we offer the first look at defining and deploying an application lifecycle management through Visual Studio Team Services, VSTS. We also talked about the ability to use alternatives to BAM for analytics, Application Insights, and Power BI, really cool features. And I mentioned this in the fall conference. Some of my favorite features are the analytic features over at BizTalk now. And then also an issue standard web interface for administration. And those were really well-received.
So we followed those up in the fall with a Feature Pack 2 for BizTalk Server 2016. Let me just call out here in the upper right side, new functionality for BizTalk like these Feature Packs between releases. These are license for customers with Enteprise Assurance (EAs) that allows you to get the new updates as part of your license to BizTalk, or if you’re deploying BizTalk Server as VMs in Azure and you’re using an Enterprise agreement for Azure, you get to use the Feature Packs. If you’re not licensed in BizTalk in one of those two ways, you can’t use the Feature Packs. It’s outside the license. Just what our guys want us to do.
And Sean, you’re here. Where’s Sean at? I think he’s back in the green room. We have one of our licensing experts here with us this week. So if you’d like to ask some questions or you wanna challenge him on our approach for this, I’ll introduce you, although I do support what Microsoft does here.
All right. So in Feature Pack 2, we added support for, in the adapter for ServiceBus, support for ServiceBus Premium and Partitioning, and that’s actually what I was using today in my demos. ServiceBus Premium and Partitioned queues. It’s also had support for TLS 1 and 2 for encrypted OAuth connections. You can publish orchestrations through API managements. We’re now participating in API management from BizTalk directly.
And then lastly, support for EventHubs both as an adapter to send or receive messages to EventHubs, as well as for sending analytics to EventHubs. And so in that last keynote scenario, where Matt and Derek and I were processing the order, we actually could have had BizTalk Server return the response directly to EventHubs. We were returning it to ServiceBus, which then passed it onto EventHubs. So we could have done that from BizTalk as well.
All right. So announcing today that we are developing BizTalk Server 2016 Feature Pack 3. Woohoo. The focus is a little bit different than the previous Feature Packs. We’re going to look at it from three aspects. One is compliance. Big for our UK and EU customers, those of you who out here, is support for GDPR so we are stating that BizTalk Server with Feature Pack 3, and then the follow on suite which I’ll talk about it in a moment, is compatible with GDPR. Also, the U.S. correlation of that is FIPS, and then, we also have support for the U.S. accessibility. So we’re at the base level accessibility when we deliver Feature Pack 3 in the follow-on cumulative update.
And then, we also have on the right side a little bit of administrative work in terms of configuring endpoints, where we’ve got support for the final feature set for advance scheduling all the way down to hours, minutes, and seconds. So we have the ability to configure BizTalk much as you’re familiar with Logic App actions. But the key focus of Feature Pack 3 is new adapters, new adapters for Office 365. If you look at our user input through user voice, one of the top 10 asked is connectivity with Office 365. And so now, in Feature Pack 3, we’re building in support for Office 365 Mail, Contacts, and Calendar, and we’ll give you a preview of that.
So this morning, we were doing a couple of things with order processing. We were setting an initial order, where we would update the system of record which is an IBM mainframe. And we would process an authorization hold. So the humungous size of the T-shirt is much more expensive than a little tiny T-shirt that we had in stock. So we had to put a hold on Matt’s account. We couldn’t do it backward without that. So we did. And we did that by sending a message down from Logic Apps using the connector for BizTalk. Look at that up here.
This is an example of one that I setup that’s a one-way, but we also have the ability for Logic App to make a request reply down the BizTalk. And I wanna point out for those of you who haven’t used the Logic App connector for BizTalk, it has a couple of key really cool features. One is you can take a JSON message and you can convert it to XML using a BizTalk schema that you’ve uploaded to the integration account.
So you can take your BizTalk schemas and your pipelines, out of the BizTalk Server, and use them in Logic Apps. It’s really, really important and very powerful. This is a schema that I generated in Visual Studio for the BizTalk adapter for DB2.
Also, it has the ability to take a response message from BizTalk and convert that from XML to JSON and again, using a schema as input,aschema that I’ve uploaded to my integration account. Okay? So really important to note that very cool capability.
Let me just process a new order here for a second, and we’ll go in and we’ll do that first initial processing. The order ID is 20002. Let’s see if it’s done yet. Okay. It’s processed through the system. So I’m making an HTTP call in this case to a different Logic App than were used in the demo earlier. This is going to be received, trigger HTTP in Logic Apps. It’s going to use the BizTalk adapter for or, excuse me, the connector for BizTalk, convert from JSON to XML, send the XML message down to BizTalk, and we will receive that down on the BizTalk Server receive location for Logic Apps, and send that over through orchestration to the BizTalk adapter for DB2, Solicit-Response send port and process the store procedure on the host side. If I rerun the query, you should see another order here. Key stroke, key stroke. It really actually does work. Okay.
And then, what I was having problems with earlier, because it was the Mac. I was trying to use a function key without the function key with, but a function key is the F11 key to scroll to the right. Gotta love the mainframe, right? And there’s that 007 user account.
All right. So this is in pending status. And then just to show the other part of the demo we had earlier and then to get to the new stuff is I wanna take that order 20002 and just complete that order. So again, I’ll call an HTTP trigger in Logic App. It’s going to post a message to ServiceBus. BizTalk Server is going to go up and read that message from ServiceBus using this receive location that I have defined in a BizTalk Server. Again, map it to a different Solicit-Response send port for DB2, process a different store procedure, and it’s in complete status. Great.
So the other thing that we did is rather than rely on Logic App to send a confirmation email to the customer, Matt, is we used… Hang on. I don’t see the email? We used the new BizTalk adapter for Office 365. And here, I just took the raw XML response from the DB2 adapter and sent it out in email to this user. Valerie, sent it to me. Thanks, Val.
Paul: Cool. So how do we do that? Let me turn the mic over to Valerie and let her give you an early look at our development and process of these Office 365 adapters. Val?
Valerie: So what I’ve done here is on this machine I’ve got an initial build of Feature Pack 3 installed. So it has the Office 365 adapters. So you can see that if we hit down here, you can see there’s the Outlook Calendar adapter, the Contact, and the Email. Now, the Calendar is both a send and a receive. The Email is both send and receive and for Contacts, it’s to create a contact, and we send that to the email and a contact will be created for you.
So what I have today is I’ve got a demo here with both the email send and receive. And I can show you what the Calendar configuration looks like, but I wasn’t able to get this beta version working to actually get my calendar entries.
So we’ll start with the sending email here. And for sending the email, my Send port is here. And when I use the drop-down box, you’ll see that I could choose the Office 365 Calendar, Contact, or Email. And then into the configuration, there’s a sign-in that’ll allow me to pick an account now. I’ve done this before so it’s already asked me is it okay to use your account? So the screen doesn’t come up that says, “Hey, BizTalk is going to be using your account. Is this okay with you?” And you have to go through okay-ing that. So the default’s here, I’m going to send this to Paul.
Paul: So as Valerie is showing, we’re using OAuth to get into Office 365, and we’ve implemented this through a new service that we’ve attached Enterprise Single Sign-On. It’s used to configure the endpoint with a static OAuth configuration. At some point in the future, we would like to improve the BizTalk adapters that do know OAuth like this adapter and allow you to use Single Sign-On and map credentials for each call.
This is like the first time in a long time that we’ve shown you some preview technology like this for BizTalk, right? That’s how quickly we’re moving, right?
Valerie: That’s right.
Paul: Maybe not so quickly. It’s getting harder and harder now as we add more and more functionality through this Feature Pack model as opposed to shipping new releases every couple of years. It’s actually kind of challenging. In the integration testing when we get the features working together, and then trying not to break anything. Right? I remember Jim Harrer mentioned this last year. “The Feature Pack is designed to be upgrade in place, don’t break any existing functionality.” And it’s been a challenge, I’ll admit that. And then as you’ve noticed, we also come out with a Cumulative Update soon after the Feature Pack and then reship the Feature Packs. Again, that model seems to work, but our challenge is to not break things when we’re doing that.
Valerie: So I’m going to go ahead and send the email here and it went out. Were you hooked into your Outlook account on this box?
Paul: Yeah. No, on this one. You’re right. Don’t read anything from these advertisements. I have no idea where they come from. You know, unless my wife is using my email. Stuffs like it has nothing to do with me. I don’t really. I have real air conditioning. I don’t think a ductless mini split air conditioner. So I don’t know what that is.
Valerie: But you can see that the email went to out to him. And then, the other one is receiving emails and that one’s actually running. So if I go in here to email received, you can see I’ve got a lot of emails here. This is set up to receive email off of my own Microsoft account. So if we look at the latest email that was received, it should be that test email.
It’ll only show the body of the email when I’m using the file adapter. You can get all of the other envelope information and the subject if you do this through an orchestration. You get that other information. But this was what the body of the email was that I sent out because I dropped this file here. This is the notepad that just the text file that I had dropped in to that email.
Paul: That’s really cool. Hey, thanks, Sanjeev, to the engineers for making that work. Sending and receiving emails. Thank you.
Valerie: Sanjeev’s engineers have been working on this, and I kept pressing them to supply me with a version that I can show you guys. We’ll take a look at the send port for sending the email. And so for this, it sets up some defaults of who you’re going to be sending it to, and then the subject of the email. And then, in the message part itself is where when I drop the .TXT file, and then for receiving the emails…
Valerie: …the configuration… Oh, good.
Paul: That’s by design. You wanna make sure you know it’s not ready for you play with right now.
Valerie: That’s right.
Paul: You have to have a few of those in there. When you say data, it’s alpha. It’s not beta. If it’s beta then you’d already have it right?
Valerie: That’s right.
Paul: So it’s pre-release then, end column.
Valerie: That’s right. Let’s see here.
Paul: Why is the natural inclination to start dancing right now? Singing or dancing.
Valerie: So more than one way to skin a cat. This is what the receive email configuration looks like. So if we’d gone into the existing one, it would’ve already shown my account there. And I wait for just a second, and I can tell it which folder I want to receive from. By default, it’ll start from right now, but I could put this back away so that I could start it from a half hour ago or whatever. And I can also choose whether or not to do unread mails only, and then there’s post actions there, where we can mark the email as read or we can delete the email. There’s a question?
Male: The emails, will that still exist in your mailbox in your Office 365?
Valerie: Yes, it’ll still be in my mailbox. Because here, that’s exactly what this post action is, is if I choose delete, then it’ll delete the email from the mailbox or if I choose mark as read, if it was an unread email, it will do that. I mean, I have chosen to take all my emails not just the unread emails, but there’s all these different settings. Now, here the folder shows up that by default, it goes into the inbox folder. But you can go down here and you can choose whatever folders that you’ve got setup in your email. I’ve got lots of folders setup here, so.
Male: And how does it manage which email has to read from the [crosstalk 00:25:42]?
Valerie: It does keep track of that so that if I stop this and then I start it up again, it’ll start up from the last email.
Male: If there are 500,000 emails, it will store all of those somewhere [inaudible 00:25:55]?
Valerie: Yeah. We’re going to have to do some testing on that. So the question was if it was 500,000 emails, exactly what’s going to happen there?
Paul: Sanjeev knows.
Sanjeev: So I think the question is whether there will be duplicates that you will be getting this start the process. I think there is enough logic that [inaudible 00:26:15]. And yes, BizTalk [inaudible 00:26:23]. But in the end, it’s not always guaranteed here. [inaudible 00:26:41] connectivity and things like that [inaudible 00:26:48].
Paul: So the short answer is that, yeah, sorry. It’s similar to the FTP receive side, right? We have this guaranteed read-once mechanism. And yeah, we’ll do some scale testing, and we’ll certainly document the limitations, and then we’ll see how it works in practice. Right?
Valerie: And then lastly, we’ve got the Calendar adapter. So for the Calendar receive, it’s like real simple. So I put into, sign-in to the account again. That should look similar. And then, it’ll come up if I have other calendars, I could choose those. I mean, I’ve just got the one calendar set up. And then, it’ll return back things that are starting within 15 minutes. And then as this runs, the Calendar event that is occurring like at 2:30, it would show up then at 2:15. It gets brought in by the adapter. And you can set it up…well, here it shows up to one week. So that’s what those configuration panels look like.
Paul: Cool. Thanks, Valerie. Really neat stuff. And thanks to Sanjeev and Valerie for getting us an early preview. Good job. You would be surprised just how many separate engineers are working on these features. We’re in the phase now where they’re trying to complete their test case development and run a full test pass each, and then integrate all these stuff together. It’s really, really challenging as you start integrating them together. And just in case the demos didn’t work at all, maybe connectivity problems, back to our VMs. Your VM was running in the local UK south? Did you say that? site?
Paul: Okay. A lot of our VMs now, unless they’re physically in their office, or on one our machines, they’re also on Azure, but the Azure data centers in the U.S. So we’ve been running everything that we’re showing you using infrastructure to service much like the bulk of your customers are doing as well. And I wanna talk a little bit about that in a moment, like right now.
So following FP3 and concurrently right now, we’re working on Cumulative Update 5. So we’re talking about what we’re currently working on. We’re working on Feature Pack 3. We’re working on Cumulative Update 5. We’ve gotten some of these work done already such as the GDPR support, and then corollary to that in the U.S. it’s FIPS and then U.S. accessibility. So there’s a lot of UI changes we’ve made. There may be some issues remaining, which we’ll document for the U.S. government, but we’ve got the basics done there so we can say we’re accessible.
With regard to running clusters of BizTalk Server, groups of BizTalk Servers and also running it in Azure IaaS/VMs is the support for cost effective deployment into groups. And that is through running multiple databases per SQL server instance. That’s something we couldn’t do before. Limitations in SQL Server 2016 for the most popular cluster and mechanism, not the fail over cluster instances that’s been around since before the turn of the century, but the SQL mirroring and the new capability that’s replaced that called Always On availability groups.
There’s a key limitation on Always On availability groups, and that is you had to have…because of the cross database transactions that we have in BizTalk server, you were limited to one database per SQL Server instance. That’s infrastructure cost. It’s also real license cost as well. And so I’m proud to announce that the engineering team is delivering support for SQL Server 2016 SP2, and that allows you to run multiple databases per instance of a SQL server. That’s really going to help you, our customers, and partners working with our customers to deploy more cost effectively BizTalk Server.
And then lastly, CU5, another key thing in here is transport layer support 1.2, TLS, transport layer security 1.2. Again, a requirement for U.S. government FIPS certification but also something that most of our customers are rolling out as a requirement by the end of this calendar year.
Similarly for the separately packaged but combined license with BizTalk is HIS 2016 CU3. Again, accessibility, FIPS, GDPR, SQL Server 2016 SP2 for Enterprise Single Sign-On. We packed Enterprise Single-Sign-on and ship it. We ship Enterprise Single Sign-On with BizTalk and with HIS package but it’s packed separately. We’ve done some testing for this.
One of our resident experts in terms of SQL server deployments and VM deployment is Samuel Casper, who we introduced earlier. He’s my go-to guy. And he’s here this week so we can talk about some of the additional challenges to running this in Azure and how you handle clustering and limitations of D2C, that sort of thing. We’ll cover those offline.
And then also for HIS 2016 CU3, we have the support for the new IBM platforms, SAP Oracle, TIPCO, Microsoft, IBM, they don’t ship when BizTalk Server ships. You know, they’re not even synchronized with our Feature Packs. And so we add support for new foreign platforms whenever we can. In this case, it’s CU3, the new DB2 in the mainframe and QV9 which is about for just about a year now and then it Kicks V5R4 in the mainframe.
Something important to note. If you go to the website for marketing, you won’t find a whole lot up there but that’s okay. Documentation still remain really good. We miss Mandy Olanger. She’s not here this year, our great doc writer-editor. She’s still with us in spirit, and she does work with us from time to time. We have three PMs in the BizTalk HIS Line-of-Business adapter team. We also work on Enterprise connectors for Logic Apps. I mentioned the MQ and DB2 connectors in particular. Chris Hauser is principal PM, Valerie, and me. So catch Valerie and I here this week and any time, you can send us email. It doesn’t matter what’s it about. Question?
Male: My question is about TLS 1.2.
Male: What’s the new [inaudible 00:33:22], new Feature Package plus the…
Paul: So TLS 1.2 support…
Valerie: The additional stuff.
Male: [inaudible 00:33:30]
Paul: No. Sorry. The question is what about TLS 1.2 for BizTalk Server 2016. Yes, you’re right. We actually deliver TLS 1.2 for BizTalk Server 2013 R2 first and now we’re adding it for BizTalk Server 2016. Yes, good question and I actually had the answer to that. You should always have the answer to everything as a PM after 23 years in Microsoft.
With that, I wanna thank you very much for your time today. I was supposed to get you back on schedule. You have an extra minute and a half before lunch starts. Thank you very much, and I look forward to talking to you this week.
Fill the form below to get all the presentations delivered as a single zip file in your mailbox.
byJon Fancey & Matt Farmer
byMicrosoft Integration Team