Integrate 2018 key speaker John Fancey introduces the world of the Microsoft Integration platform by walking through the evolution of IT from the 80s to present day. He discusses how technology has driven the change of speed in completing every-day processes, using the example of transferring money.
Integrate 2018, June 4-6, etc.venues, London
Jon: Great to see everybody. Thanks for coming. This is an amazing event. And Saravana gave in the, kinda, history of this event, and I was looking back at this as well. This is the seventh year in various guises that we’ve had this event. And how many people were here last year? How many people were…? So most people. How many people were here back in 2012 then? How many people have been, you know, every year? Hardly anybody. So I was here in 2012 as well. I was at the Microsoft Office, I think, at Cornell Place, which don’t even exist anymore in Victoria. And it’s just gone from strength to strength to strength, and it’s all down to you. It’s all down to the community around this, the amazing people who work in this integration space on our technology which we appreciate every single day. So you know, I’ll just extend Saravana’s welcome and just thanks to everybody for taking the time out for the next three days to come to this event. And you know, I really hope you all enjoy. So let’s get started.
So, for those of you who don’t know who I am, I’ve been in the integration space like a really long time. And for those of you who do know me, you already know that anyway. I was an MVP before joining Microsoft about three, three and a half years ago. MVP, I’m kind of BizTalk Server and Azure, and just always saying this integration space for like a decade I think I was an MVP. It has been an amazing experience joining Microsoft, and sort of seeing things from the inside and then sort of driving products. So I’m responsible for Logic Apps. We have a large number of the team here, and I’ll kind of introduce people later on, but there’s over 20 of us here this year. I think it’s the most number of people that we’ve had at this event ever. So you know, most of the front row here are kind of Microsoft people. And I’d encourage you to kinda meet people and talk about integration in the time that you have here for the next three days.
So, let’s get started. So, I really wanna talked about digital transformation but I don’t wanna talk about it in those terms because I think they are really overdone. What I wanna talk about is change. I wanna talk about, you know, what drives change and what’s really driving digital transformation. Because, you know, these two things are really synonyms to one another. Change is really, you know, the one thing that doesn’t change, right? Other than, kind of, death and taxes, you just have to assume, in our industry, that things are gonna change. And the rate at which things are changing is only accelerating. We’re in this continual state of flux. You know, we’re just seeing more and more and more innovation. You know, since BizTalk 2000 nearly 20 years ago, you look at the amazing pace of innovation and change in the industry.
And even kind of before that, and I’ll come onto that in terms of my own history, that things just do not stay the same for very long. You know, it’s getting worse. The acceleration, the pace of change, just doesn’t abate. But it’s up to you how you define what worse is because your worse is somebody else’s better. Right? It’s like if you fail, like you’re being disrupted, if you feel that change is being forced upon you, then you really need to sit back and think about that and how you can reverse that equation. Because you’re always better off in the position of dictating change, driving your own industries, and making change happen in a positive way to get things done for the better. Because this is not gonna change, right? So, point number one, it’s just gonna keep going.
So now, my own personal history, and this is like, you know, this dates myself immediately. Who, sort of, remembers these stuff? So I started off in the ’80s, in the first period, I would say, of computer disruption, right? I mean, we talk about, kinda, digital transformation is this new thing, right? But it’s been going on for a really long time. We talk about citizen integration, we talk about the, you know, democratization of our industry, but this has been happening for a really long time. I remember vividly, you know, back in the ’80s, I got my first home computer. This is actually a Commodore Plus/4 over here. And I started writing games for this thing when I was at school.
You know, this is really the advent of the so called bedroom programmer, right? The ability to actually have a computer in your home is not new. That was not invented for the internet, it was invented 20 years earlier than that. And there’s a huge amount of innovation in this space. You know, everybody from, like, you know, Apple and Commodore and Sinclair and others kinda got into this space, even people like Mattel who are toy manufacturer with their Aquarius, kinda, got into this space because they’d like, you know, have things up to date. Nobody knew where is this gonna go. Nobody could predict the future and what was gonna change with the advent of creating these home computers.
So I kind of, you know, had a bit of fun, you know, some people might love the money. I wasn’t one of them. These are some of the games that I wrote. But what I thought was interesting is that, you know, this era, I think you, sort of, look back and it feels like, “Wow, it was a very short period of time.” It was only 20 years. And the first home computers came on the scene in the late ’70s. And the Commodore 64, which were one of the bestselling home computers that there was, didn’t seize manufacturing till 1994. That was the year before Windows 95, which has really gotten Microsoft’s, you know, huge consumer play, you know, to get computers into the home, get PCs into the home because, you know, the period was 20 years. So change, kinda, happen slowly over a period of time. But sometimes it feels like things are changing really quickly, and sometimes, you know, it feels like they’re moving much more slowly.
You know, so my first home computer was this thing here. Did anybody here have one of these? Anybody here as old as me? Beautiful. So I kind of spent my summer holidays when I was a kid, and, you know, I poured over the manual and the back of the manual that came with this thing. It was along with, like, there was like 200 pages of how to switch this thing on and plug it into your TV. And then there were two pages at the back where be opcodes for like the 6502 processor, and I spent the summer holidays, like, learning this stuff and actually learning machine language. And this was the cartridge you plug into the back of this thing, that lets you actually program in assembly language on this computer in the full 3.5 [inaudible 00:05:36] that this thing had. It was glorious. You know, like, the best part of life, you know, maximum 2,000 instructions because most opcodes were two bytes, right? So you fill this thing out pretty quickly.
But things moved on and, you know, we’ve actually moved away from home computers in the ’90s and really, gaming consoles took over because at the same time, stuff like this, you know, Windows 95 was driving computers into the home in a much broader way than it happened in the past. You know, back when we had home computers, it was really, as I said, nobody knew what these things were for. People are buying them for their kids because they were, kinda, educational and, you know, playing games on these things and companies was springing up and creating word processors and databases and spreadsheets. And you could use the same thing for all of this stuff, but it pretty much sucked depending on which one that you chose. And really, the PC really kind of changed all of that by providing, you know, platform for businesses to actually build on top of this stuff.
The great thing about the ’90s was that these things were, you know, moving so faster that there were pretty much redundancies. Like, four of the computers that I had in the ’90s, they lasted about a year, year and a half before it was just junk and you have to replace it with something faster. You know, the big tower thing, and this was really impressive, was the 133 megahertz Pentium. You know, the first one, I think it had 16 megabytes of RAM. So things have moved on a little bit in the last 20 years. The other thing that happened to me was that I started work. I started off by working for the Southeast in Electricity Board in the U.K. so you know, this is the electricity provider for the southeast. They, kinda, gave me this choice. They said, “Well, you know, we have these mainframes and all the tools.” The intake of junior program is it’s like just bringing people on and teach them COBOL and CICS and all of this stuff.
And they’re like, “You could do this or you could, kind of, get one of these things,” it’s like we have a couple of these kicking around in the office. We’re not really sure what to do with them. So I, kinda, went down the path of kinda doing the mainframe thing, and then kinda moved on to PCs later on. And what I started to do was actually sort of frontend, and screen scripted this stuff like many other people to actually have this sort of hybrid integration architecture platform, right, spanning between different worlds, between his IBM world, all these green screens and being able to drive that is really what we were trying to do and what Microsoft was trying to do was to make computers easier to use. You know, this was hard stuff. It gets people who had no familiarity with computers, they’re only in the offices using kinda 3270 terminals and things to actually get their job done was a non-trivial problem. When you look at this now, and kids in school learn how to code and they learn how to use Office and everything else from a very young age, it was not like that 20 years ago.
So things moved on, right? We got Internet Explorer in the ’90s. We got Visual Studio 97, it’s had its 21st anniversary last year. And, of course, BizTalk Server came along in 2000. It’s, you know, a fantastic product. It is the 10th release of this product, and just have kept going strength to strength. The iPhone came along, it really changed a lot of this. I’ll just keep moving on. And that wave where we are right now is this next wave of disruption. From cryptocurrencies to IOT to, you know, Logic Apps, of course, the invention of iPads, integrations of platform, all the way through to bot framework, and other things, just the intelligence with Cortana, the ability to now talk to computers which was science fiction 20 years ago, and now very much a reality. Of course, you know, my seminal moment was joining Microsoft three years ago and actually working on these products themselves.
And I say that, you know, time, you know, always drives positive progress, but this is better. You know, we moved from one to the other, and we kinda moved between these decades and things just get better and better. And I really hope, you know, that that continues because everybody likes better, right?
You know, one example I’ve used, because I think it’s a very interesting one. When you look at disruption, you look at digital transformation, you look at the amount of change that has occurred in industries of the last 10 years or so, one of the ones that seems really, you know, gone through significant change just in the last five years is banking. You know, banks started in a formal way in the 1300s. In Italy, the first banks were created. And their objective is really around keeping physical assets secured. The problem was that, like, people had money and they needed to, and in many cases, it was in some other asset form like gold, and they needed it protected. And that was the job of the bank, to protect that money and to record it so that you could rely on going back and being able to take this money out, and it was actually still there.
You know, Wells Fargo who’s going through their own kind of transformation right now is the picture of this sort of stage coaches in the 1850s because they have the same problem. You know, 600 years later, that problem hadn’t been solved in the U.S. where they’re just trying to move money from A to B using stage coaches. They did this 150 years ago. Things like, you know, wire transfers were really, you know, enabled through telegraphy, right? So there’s no simple way of moving money around other than physically moving it from A to B. It’s very broad and centric with ledgers in each branch, so the ability to even move money around in your own bank was much more difficult than it is today. But this all changed, obviously, with the advent of computers, and mainframes particularly, honestly. You know, the first ATM, the picture in the bottom here is of Barclays Bank in 1967 with, you know, the crowds just fascinated by this new ATM, the first one in the world. I mean, who wants to see a crowd like that, a queue like that at their local ATM, right?
But things, you know, changed very quickly after that, and it was around banking efficiency, how to enable the bank to do more to scale up, right? As the number of customers grew, they needed things like automated transfers, BACS between payments, SWIFT for automated transfers of money between banks, with this SWIFT networks, excuse me, created in 1973, you know. So, but like what happened since then? You know, we had the ability to, like, move money around between banks, inside our bank like 40 years ago. And again, things slowed down, right, and kinda went through this period again of very little growth or change in this industry, very little disruption, until recently, until the internet came along and PCs got installed in everybody’s homes and everyone started getting online. Now, this is a much more balanced equation.
There was something here for the banks, sure. You know, you see the extinction of branches that’s happening globally in terms of, you know, there’s always one of these double-edged swords, right, where, “I enable people to do things online and then they visit their branches less and they just gradually die off.” And the banks are obviously driving that because they’re incented to do so, but it’s actually about convenience for the customers as well. It’s about that customer experience and improving that by providing more options for them to be able to do things with their money. It was into the bank manager, you know, as well. But, you know, where we’re at right now is even greater disruption. The last few years have brought multiple initiatives to drive banking to the next level. Things like PSD2, which is driving APIs, banks opening themselves up to be able to build new platforms on top of these banking infrastructure APIs.
Blockchain, of course, you know, is just changing what a bank actually means. I think I read recently that 50% of global banking customers have used a non-traditional banking service last year. So that means they didn’t go to a bank to move money around or change currency from one to another, they use P2P lending or some other mechanism online instead of using a traditional bank or a traditional lender to do that. One in 10 Americans actually switched bank accounts last year. So people are moving to where the technology is and where they get the most value out of this. And of course, you know, things that like the cloud is, like, one you can hear a lot about in the next three days.
So if I kinda wanna try to break this down. I think there’s, sort of, three phases of disruptions. There’s probably more than this particularly. I’ll keep this a short list. Three phases of disruption. I think the first one is really denial. In any industry, there’s this period of very slow growth, very slow change, and everyone’s comfortable with the way the company’s doing, the organization is making money, it’s all good. But you’re really in this denial stage because you’re not trying to anticipate what the next thing you can do is, you’re not really trying to drive disruption yourselves, you’re really just getting comfortable, right? And the next thing that typically happens is this sort of period of questioning. You can start to see changes arrive on the horizon. You’re sort of, maybe, getting new people in your organization with new ideas, and that’s what starts to drive the second phase of this, which is like, “Why do we do things the way that we do them? Why is it always being none like this? You know, why it’s like 50 years of doing it like this? There’s another better way of doing it.
And then you reach the stage of enlightenment where you start actually trying to disrupt the way you do things. Often, inside your organization, as I said earlier, it’s like that’s the best way to do this, and I’ll talk a lot about this because the last thing you want is to wake up one day and realize you have been disrupted from the outside, okay? Industries that have been heavily disrupted and caught people by surprise with the application of technology and the things that the hotel industry with Airbnb or the transportation industries with companies like Tesla, or Uber, or Lyft, or whatever, you know, I don’t need to go into those examples because you’re all super familiar with that.
Here’s what usually happens if innovation actually plateaus. So you go through this period of rapid innovation, rapid growth, rapid changes in the way things get done. And then the ecosystem, kinda, builds around this when you get an established business, and a support network community grows around it, the technology matures, the people who know how to use that technology, leverage that technology, make it effective, and make it work, and make it predictable, honestly, right, so you can implement at a predictable cost and a predictable risk. And it, kinda, plateaus. It just kinda levels off, right, until the next, sort of, wave of change comes along, and then it, kinda, drives it up again. And so you go through that cycle again and it repeats. So it’s not like this is some straight line or exponential. It’s always this kinda lumpy thing. If you look at any industry over time, you’ll sort of see this.
You know, here’s a quote from a guy called Lewis Mumford. You know, I find this intriguing because this guy was like 1946. He called himself not just a writer and a historian and an architect, but he’s actually a technologist in 1946, before the first, at least, commercial computers existed. You know, I’ll just point out a few things in this. You know, he talks about the fact about this innovation plateau, kinda, problem. The continuities, the actual just evolution of something very rarely drives progress forward in a non-linear way. You end up with inertia where things just slow down, the progress of change stops. And then you know, you have to reach this period of mutating, which I just love this because, like, I meet people in genetics and IT kinda use these words in a regular basis with technologists in the 1940s. So you have to make a step change. Things have to change significantly to provide durability for the future, to build a new platform, to invent something new which you can all move to to continue that growth. Otherwise, you would’ve just flat-lined years ago, right?
Okay. I am not saying this is easy, this is in fact very hard. A lot of companies have strived creating innovation centers and, you know, centers for excellence on this and that, just getting a group of small people together in their organizations and trying to figure out what their company, what their industry should look like 5, 10 years from now. It’s difficult, right? You can’t just put people in a room and say just come up with new stuff and make it impactful, right? That’s actually pretty difficult. But you know, the hard cold facts are that it’s estimated that 10 years from now, 40% of the Fortune 500 will no longer exist. And if you look back over the last 60 years, 90%, sort 88% of the Fortune 500 that existed in 1955 are gone. Only 12% of these world’s largest companies, of these America’s largest companies, no longer exist. Right? The odds are not good if you stayed the same if you continue to do things the way you’ve always done them.
We don’t have a choice, right? Because if you don’t disrupt, somebody will disrupt you, and this is honestly what digital transformation is all about. It’s just a pretty marketing way of kinda dressing this up to make it sound exciting. It is exciting for sure but it’s also scary, right? But it’s a problem that you’ve got whether you like it or not because change is coming. But here’s the good news. You know, we’re all in this together. You know, none of you, no industry, not even us at Microsoft is immune to this. We’re all being disrupted, we’re all having to face the challenges of learning new technology on a daily basis. You know, so many people including, you know, a lot of… we hear this from our customers all the time, struggle to keep up with the pace of change of technology and the new services and things.
You know, we feel these too even inside Microsoft. It’s almost impossible to keep up with everything that we’re doing. And of course, we try to keep up with everything that you’re doing and everything our competitors are doing and everything that everybody’s industries are doing. This is a tough problem. But we’re all in this together. You know, Microsoft, customers, ISPs, and our partner, all of you, as global community of integrators, it’s how we can succeed, by leveraging one another’s experience and just figuring this out.
You know, this is a quote I really like from the American comedian, Tina Fey. She says it very succinctly. I learned a long time ago that it’s easy to commit to something and work out how you’re gonna do it afterwards. Because if you never commit, it’s actually very difficult to get anything done. It’s very difficult to change. You have to get consensus and you get everyone to buy in to do something to make that change. Because if you don’t do that, you just kinda drift, right? So say yes, and you’ll figure it out afterwards. You know, why? Well, because if you say no and you’re like in the IT Department or wherever you sit in the organization, it’s like, “Why would anyone ever ask you again?” Right? So if you gotta, you know, use some service or go to a store or buy something and they just say no, are you gonna go back? No, you’re gonna go somewhere else.
You know, inside the organization, things are really no different to this. They’ll just go someplace else. And you’re being disrupted into action anyway, so you don’t have a choice. Things are gonna change whether you like it or not. So say yes. It is the right thing to do, and figure it out afterwards. It’s true, right? I mean, change is great. And it’s, you know, the reality of where we live today and what we do, but it’s gonna be really annoying at some point of time, right? Other people in the organization or other organizations, you know, industry are going to disrupt you, and it’s gonna be annoying. You also have this human nature problem that change just takes time, whether it’s cultural change or organizational change or technology change, it’s hot. And I’ll talk about, you know, in the organizations that I’ve worked and part of Microsoft, that talks a lot about this in one particular company that you can try really hard to get things done, you can grow great teams of people to do things, but you cannot remove this from the equation, that change takes time.
As soon as you feel like you’re fighting human nature, and human nature is to resist change, it’s an uphill battle, right? It just takes time. Because there are differences, you know, between the real world and the online world. You know, the… One way I kinda look at this is that it is very similar to this old adage that, you know, “In theory, theory and practice are the same thing. In practice, they aren’t.” You know, when you look at the way disruption happens today, it’s very different to the way disruption happened 20 years ago. You know, if you look at, like, in the physical world, you’re walking around and maybe you bumped into somebody and you maybe apologized. In fact, you know, here in the U.K. it’s like everybody apologizes, right? You know, you bump into one, “Oh, I’m really sorry.” Because there’s a personal connection, right?
But as soon as you kinda get on the freeway or wherever you’re driving around and somebody cuts you up, so you got the steel box around you, you feel this isolation from that physical world. So maybe the other guy apologizes, maybe they don’t. When you’re online, nobody apologizes. You know, did Uber apologize to the taxi industry for, like, moving their cheese, right? No. Because there’s this disconnection that technology creates that empowers you to do things without asking for permission, without asking whether the other guy likes it or not. You’d do it anyway. The big difference, the other guy doesn’t care if you like change or not. He’s gonna make things change, and you’re gonna have to deal with it.
So what if you don’t like change, which is, you know, most people, right? If we don’t like changing, you should do more of it. Honestly. So you need to get out of your comfort zone and do things you don’t like to do and change things that are hard to change. How do you know when to change? Well, you know, if you don’t see things changing, then experiment and change something. Change some more. You know, I think the, you know, the advent of DevOps and how that has grown and how that has matured in terms of the application, the execution of what people do is a great indicator of how you can improve.
You know, it wasn’t so many years ago when the organizations would typically deploy new applications every month or every three months or even every year, right, because of large application deployments. Very high risk, very high effort. You know, the functionality typically gets squeezed to hit the deadlines, right, that you’re committing to the business. So you need to change the model. You need to do it more, you need to look at organizations who have innovated in this space who deploy every minute or every couple of minutes or every second, right? There is no reason why that’s something you can’t do. It turns out that if you do move to that model, the risk go down significantly because the amount of change you’re introducing at a time is much less. And this is true for organizations as it is for IT systems, honestly. Change a little enough, because you can always change it back, right? If you screw up, then, you know, maybe you can change it back even before anybody notices, right? If you’re really quick.
You know, this is a great quote. This is, you know, Henry Ford. He said, “If I had asked my customers, what they wanted, they would have asked for a faster horse.” Right? He actually didn’t say this. Who knows this quote? Because, you know, even his grandson, it’s such a powerful quote even his grandson thought he said this back in the ’50s. And he’s like, “He never said it, but it’s a great quote.” Because it’s like, how do you know what you need to change, right? It’s like you always hear it all the time, and we hear it all the time ourselves. It’s like, “Ask the customers. Look out of the windows. See what’s going on.” Right?
But when you ask customers, they don’t know necessarily what they want. None of them is plugged into the industry as you are. Because that is human nature problem. You need to gain experience as quickly and as broadly as possible to understand what’s going on inside your own organizations and your own constraints, what’s happening on technology in that space and how you can apply that to your industry, and what your competitors are doing so you can start to predict the future, so you can start to see trends emerging in spaces that may not look directly relevant to what you’re doing and figure it out for you customers.
Man 1: Also, best sausages in town.
James: Nice. Does it say anything about parking? We’re just spinning. It’s the green light.
Man 1: She was like my ma.
James: Stop. She shut her eyes. Ah, here we are. Now, that’s how you park in this town. Table for two, please. Two winners.
Man 2: Drivers win at confused.com. Number one for car savings.
Jon: So it’s my very great pleasure to introduce Matthew Fortunka. He is the Head of Development for Car Buying at Confused.com. Thanks, mate.
Matthew: Jon. Cheers. So, I’ve been lucky enough to work in the Confused.com for the last 10 years. In that time, I’ve worked pretty much in every part of the business. I’ve been part of the Partner Integration Team working with, kind of, insurance partners. I’ve been working with our international cousins producing similar sites all across the world. And more recently, I’ve been looking at new products and new ways that we can expand this in new markets.
Confused.com is a technology company. It may seem like we’re a marketing company, but actually, at the heart, we’re a technology company. To live about our story is to take you back 18 years, back to the year 2000 and to think about what the process of like what was getting car insurance back in the year 2000. So, you would typically get a recommendation from a friend or family, and you would buy the policy, and then a year later, the renewal will come through and it will be slightly increased, and you would pay for it because the process of going through and looking at 150 different insurance providers to get a cheaper quote was just not worth the hassle.
So back then, Admiral Insurance was a relatively newcomer to that market. And they were looking to, “How do we break into this market when no one changes?” So they looked for technology solutions. And they looked to say, “How can we give people the option to compare? And how can we then make it easy for them to switch to us?” So a separate company was spun off from Admiral, and in 2002, we became the first U.K. car insurance price by comparison site.
Now, in the early days, those integrations, the way that we worked with partners was crude. We’re talking VB5, VB6, 30, 40 servers that will literally be filling in forms on insurance websites and then scraping the prices from the screen out of the market. Sounds crude but the result was industry changing. Now, the majority of the traffic to most of the insurance companies comes from the big four price comparison sites. You may have heard of some of the others. So the price comparison business is a highly competitive one. As you can see from the amount of time that they spent on advertising, not a ton of money that’s spent on advertising. We’re constantly on the lookout for an edge, something that will make us stand out from the competition. This could be new products, it could be new advertising campaigns, hopefully, or approaches to technology.
Now, five years ago, Confused began a journey to move our existing infrastructure into Azure. We’d already spent some time in the year before moving to a service-oriented architecture based on .NET and the Microsoft stack. But everything was on fixed hardware, which meant that we were priced up according to our peak capacity, which was typically during Coronation Street on a Monday night when an ad went up. The cloud offered us an opportunity to make a significant cost savings. In fact, we estimated it something like a sevenfold decrease in our cost. While also, at the same time, it gave us new tools to play with. So as part of this migration, we moved away from these 30, 40 servers onto BizTalk.
Now, to give you some indication as to what it was like before, when I first joined the company, we were using a SQL database as a kind of orchestrator. We had the frontends writing requests into tables. We’d have some primitive load-balancing technology that would then farm those requests out to 100-150 partners. And then, those results would then be written back to the database, which the frontend would then be polling for. That’s a locking nightmare. Particularly, when you factor in the fact that we’re talking about thousands of requests an hour, 100-150 HTTP requests out to partners with every single one of those. You’re looking at two billion requests going out to partners every year. With BizTalk hosted in Azure, we were able to replace those backend servers into standardized on the way we did our transformations of communication with partners. What I meant was that our developers were now freed up to work on new integrations with new partners to increase our panel size and to look at new products.
The ability to define in Azure, our infrastructure as code, meant that we could actually now react to demand even on BizTalk. So we could scale up and we could add new notes to our cluster in anticipation of demand. Still, there’s a bit of a problem with managing load. So we were a very early adopter of the Geo Service Bus. The frontends are talking to BizTalk via service bus queues. That way, we get to, kind of, smoothen out that load, and we don’t introduce that much latency at peak times. And then to get policies back, we’re using Service Bus topics and the frontends are subscribing to these topics and literally having their messages pushed back from BizTalk all the way back up to frontend, no more polling.
So this is my baby. Having transformed the car insurance industry three years ago, we started to look at new markets. And Confused have always had this kind of drive to be on the driver’s side or driver’s win or whatever the marketing message is this week. But we were looking at it in terms of, “What’s the pain points along the process?” And actually, the process of buying a car right now is pretty horrible. No one wants to go into a dealer. By the time most people get into a dealer, they’ve already established the car that they wanna buy, they wanna sign, maybe take it for a test drive, done. No one wants to be pressured into buying unnecessary finance products. The test drive, whilst useful, is awkward because, as everyone knows, it is another sales opportunity. It’s that moment for 10 minutes when the dealer has you exclusively their attention.
So, Confused set up a team, and over the last three years, my team and I have been working to make this a reality. We were given a bit of a remit to experiment, to find new and interesting ways to solve problems, and to feed that learnings back to the core business. For Confused, this will be the first time we would create a brand-new product that was unshackled from the legacy. It would be totally cloud-native. So what do I mean by cloud-native? Well, the site is built from off the shelf products available in Azure. You can see a selection of them here. Some of them have been tested in production and then thrown away because they weren’t meeting the purpose.
Now, the integration challenges on a project like car buying are very different from car insurance. Our challenges are mostly centered around this idea of, “How do we get stuff to sell?” Every day, we receive updates on 10,000 vehicles from a dozen dealer groups. In principle, this is an easy problem to solve. We get a file in, we write it to the database, we can sell, everyone’s a winner. But as I’m sure we all know in this room, problems like that are never that simple. Right now, the file comes in, the data has to be cleaned, we have to apply pricing rules, we have to submit requests off the CRM systems, we have to notify parties that new feeds have come in, new data has come in.
Now, we could have extended our existing BizTalk Cluster for this or we could have used as a size packages. But for what was…actually in comparison to car insurance, a relatively small job, adding more notes to BizTalk cluster would probably have been cost prohibitive. And it went against this remit of, “We should be trying new things, we should be experimenting.” As a team of developers, we were skeptical, to say the least, of Logic Apps. Cries of, “Why wouldn’t I write this in C#?” were ratted in the office. But we persevered, we put together the workflow using the Visual Designer. Snapping actions together was easy and the range of actions available was more than enough.
The product tone that could get involved and understand the process because it wasn’t that far remove from the original flow diagram they’ve given us in the first place. What’s more is the template was active, which meant we could test changes to the process really quickly. The most rewarding moment during the development phase was seeing a group of cynical devs around a single screen asking questions like, “Can we run those actions in parallel?” And then a quick drag, trying it, moments later, we can see the results. After the first couple of templates have been produced, the cries have changed to, “Why would I write this in C#?”
So with the product tone is in dev team now sold on the idea, all we needed to do was to sell it to the people signing the checks and the people monitoring production. Selling to the bill payers is easy. Using the online tools, we’d estimated that to process about 10,000 vehicles a day, vehicle updates a day, it would cost us about £30 a month. By the time we’d finished tweaking that process, that figure would roughly halved, and we suspect there’s probably more savings to be had in there as well. When the Ops team started playing, they were amazed by the amount of diagnostic information they could get from the Azure portal. The visual nature the designer meant it’s always clear when any issues occurred, and individual runs can be re-submitted with a single click.
So what’s on our radar for the second half of this year? As with most businesses, there’s an interest in the data that we produce. We’re looking to make a lot of our data available via our APIs, but the nature of those APIs is they probably won’t be heavy used, they’ll be occasionally used. And we, thanks to Logic Apps now, we’re quite smitten with this idea of the pay for what you use model. So we’re now looking, as Azure Functions as an alternative for this, as a way of porting across some of our existing codes that we have in some of our APIs and we don’t have to spin up VMs to make that happen. Now, once we’ve got this collection of APIs, we’re gonna want a consistent way of managing the users. And we’re gonna want a consistent way of billing and throttling and all those kind of wonderful things that we could build ourselves. But, frankly, API Management has it all covered for us.
So to give you a quick recap, Confused is, at heart, a technology company. There are some debate within the company as to whether we’re a marketing company or a technology company. Without the technology, none of the business would exist. We’ve used as much of Azure as we can get our hands on and we’ll continue to do that. And to echo what Jon said, the importance of experimentation, to use a quote from Jimmy Carr, “Cats have nine lives. Makes them ideal for experimentation.” If you need any more information about what I’ve spoken about today, please come and find me. And enjoy rest of your conference.
Jon: Awesome stuff, right? This is incredible. This is the application of technology and the changing of business and the disruption of an industry all wrapped up into one which, of course, is even better using Microsoft technology. You know, let’s move on. I think one of the things that I think is fascinating about integration, the way integration is changing and where it’s going is this network effect that occurs. For those of you who aren’t familiar with, kinda, network effects and how they work, you know, this is a term coined in the ’70s as the first telephone networks, kind of, really started to grow. You know, the problem of having a network that’s relatively small is like there’s relatively small value in it, right? The guy who buys the first telephone is taking a bit of a punt on this technology being successful, right? And even when he does that, the second person comes along, he’d really, kinda, like to know who the second person is, although he’s probably not gonna wanna talk to him anyway. His kinda wasting his money.
So what happens is that over time, the network grows, and the amount of value that gets added as each node, you know, telephone, whatever application system, in this case, gets added, the value grows exponentially relative to the cost growing linearly in conjunction with that. So as you start to connect different systems together, and Logic Apps and our Cloud platform make this super easy to do this, so you don’t just solve those traditional problem of, “How do I move data around between my different parts in my organization, and maybe my partners, and how I do things like invoicing and all these other, you know, very common processing patterns?”
No, you move on from that and you’ll start to be able to apply other services to those processes and start to get insights into that data, start to understand what’s going on from an organizational level, start doing things like fraud detection or anomaly checking to see whether things are out of pattern and react and respond to that in real time. You can start to play machine learning and AI to this to get genuine insights into your organization to understand it better so that you can create more integration to drive more insights, to drive more change and more disruption in your industry.
So, you know, what I wanna introduce now is a company that we created called Contoso Retail. So Contoso Retail are an outfitter for every occasion. Contoso Retail are a traditional bricks and mortar company who are being disrupted by the online world. You know, they always prided themselves in customer service which, to them, always means they wanna provide the best experience possible both in store and online. And, of course, that means they never wanna lose a sale, they never want their customer to leave empty handed and unhappy and dissatisfied because they’ll just go someplace else, because didn’t say yes, right?
So let’s have a look at what they did. They took their traditional processing and looked at how they could augment that with the cloud, how they could introduce new technology to change their business. So the first thing that they did was they looked at the inventory checking and back-ordering process, and introduced PowerApps in stores so that the sales assistant could interact with the customers so that when somebody comes along and says that they can’t find what they’re looking for, they can drive that through putting up APIs in front of their backend systems, SAP in this case, to do inventory checking and order management and sales order creation so that they can automate these processes.
Derek: My name is Derek. I am your personal shopper here at Contoso Retail. Now, Contoso Retail have equipped me, a personal shopper, with this awesome PowerApp in which I can not only use to check inventory from a customer but even place a back order if they are looking for something that we don’t have in stock. It looks like we have just the customer that I can help with.
Matt: Hello. I was browsing your store and I love this T-shirt. But it’s maybe a size or two small for me. So, have you got any other larger ones in the shop?
Derek: Well, sure. Let me check that for you really quick. So all I need to do is to point my camera at this barcode. It should be recognized instantly.
Jon: Yeah, the bar code is bigger than the T-shirt.
Matt: We went to lengths to make this as realistic as possible.
Derek: So a call is being made to our backend system. It should take a few seconds for us to retrieve the results. Let’s see. Well, Sir, it looks like we have a few small, medium, and large one available but I think you are looking for something different.
Matt: I’m looking for humongous.
Derek: All right. So, we do have this humongous size in our inventory system but unfortunately, we don’t have that in the store. But do not worry, if you like, we can actually place a back order for you. It should arrive in the store within three to five business days. You would even get an email when that arrives, so you can come and pick it up.
Matt: That sounds fine.
Derek: So let’s start with the backorder process where I have the item ID and the size and quantity already populated, so all I need from you is the customer ID, if you remember that.
Matt: Yeah, it’s 007.
Derek: All right. Customer 007.
Matt: I’ve been thinking awful a lot of myself.
Derek: So, it looks like we already have all the information that we need, your address, your credit card, and your CVV number on file. So with your permission, I can just go ahead and place your order for you.
Matt: That sounds great. Thank you very much.
Derek: All right. So, we’ll take another few seconds. We’ll make a few calls to our backend just to make sure the credit card is still valid, and then we’ll create the sales order in our ordering system. And then, you can be on your way and come and pick it up once that is ready. All right, there it goes. So the order is received and we will get an email and you can come and pick it up once we have that.
Matt: Thank you very much.
Derek: So now, we have showed you the customer’s experience in checking inventory and placing back order. I want to show you what happened behind the scene. So in this implementation, we have two Logic Apps. The first in which calls into customer’s DB in which we use for inventory management. But the second one is actually the interesting piece and the one that I want to show you.
So I’m gonna switch into the Agile portal. The second Logic App we call the Backorder Item is a Logic App that runs in integration service environments. With this VNet connectivity, we first… Let me refresh the page. We first make a call into BizTalk Server and check to make sure the customer’s credit card information is still valid and contains sufficient funds that we need to complete the purchase. The BizTalk will return a value telling us whether that is indeed the case and we will proceed if that is valid and true. If the customer’s credit card is good, we’ll, again, using VNet connectivity provided to us with integration service environments making a call into SAP server. Again, this on-prem installation that will connect you without the need of on-prem’s data gateway. So we’ll create a sales order in SAP which will then be picked up in a later demo and send it to our suppliers.
So let’s recap really quick. What we have done here? We have two Logic Apps in which we used to integrate with cloud on on-prem systems. Here we used Cosmos DB for inventory that lives on the cloud, we have BizTalk Server for payments processing, and we have SAP for our ERP system, all connect to from Logic App through VNets. So Logic App make this integration very easy. Both Logic have used request and response pattern to expose itself as an API, which is then secured and monitored by API Managements. The API Management endpoint is then exposed to Power Apps and with its rich set of built-in controls, just like barcode scanner that you’ve seen, make it really quick for us to do application development and make it possible for personal shopper like me to better help with our customers. So now, the sales order is in SAP, let’s see what happen next.
Jon: Okay, let’s do that. Thanks, Derek, and thanks, Matt, as well.
Okay, let’s look at the second part of this process. Here, like, once we’ve collected this data that customer…looks like they got what they need, it’s in the warehouse, we need to create an order that we can send from SAP to the supplier. So that’s gonna be done for our Logic App using a new SAP trigger to actually take the changes that happen in SAP and actually then invoke a Logic App and make that happen. Divya, why don’t you show us that?
Divya: All right. Thank you, Jon. All right, so I want to start with the SAP system. As a next step, what we are going to do is we have created a purchase order in response to the sales order that we received from the customer. So as a next step, I want to send this purchase order to my supplier and I’m going to hit this button here. And let’s close this window. So we want to send…we want to process the purchase order created here before we send it to the supplier and we want to do all the processing in the Logic Apps. Today that… So today, if you want to do this today, this is not possible, right? Like, you cannot receive a message from SAP into Logic Apps. This is changing from today.
I’m very excited to announce the private preview of SAP trigger that would allow you to receive messages from the SAP system into the Logic Apps. So let’s look at the run that… All right. Before we look into the Logic App, I do want to mention that this is a big milestone for us when it comes to integration between Logic Apps and SAP, because now we have bi-directional connectivity between Logic Apps and SAP. We already had it from Logic Apps to SAP, but with trigger, it completes our story with bi-directional connectivity. Another thing that I want to mention that the trigger, SAP trigger, that we have added is not a polling based trigger, which means that you do not have to poll all the time to check whether a message has been…whether an event has been generated in SAP system. This trigger is based on WebHook, which means that whenever an event is generated from SAP, the trigger gets it via a call back that has been registered at the gateway.
So let’s look at the Logic App. In the Logic App, I have the trigger where I have received the XML message from my SAP system. As a next step, I want to do some B2B processing on the message. So I’m transforming the message that I received from SAP into an EDI purchase order document here. Next, I’m including the document using the X12 agreement between the Contoso Retail and Fabrikam, which is our supplier. And finally, I’m sending the message to the supplier over their SFTP server. If disconnected and looks new to you, well, this is another thing that is coming pretty, pretty soon where VR, the current SFTP connector has a message limitation of processing up to 50 MB. The new SFTP connector is going to process at least 1 GB when it is introduced, probably more. So, with that, let’s quickly recap on what we saw today.
We basically, received a message from SAP into Logic Apps via the new SAP trigger that we are announcing today. And then we did some B2B and XML processing on the message using artifacts from integration account which is our secure container for artifacts. And then finally, we sent the message to the supplier using the newly introduced, again, SFTP connector, which would support large message sizes up to 1GB and more. Thank you.
Jon: Great stuff. Thanks, Divya. Awesome.
So let’s move on. We’re halfway through. So we had the ability for our customer, Matt, to be able to come to our store, interact with our sales agent, a personal shopper, because everybody has personal shoppers in the States, right? And could find what he wanted and then order that through SAP to the supplier. So let’s look at the process now of actually handling the delivery, the supplier telling us when they’re gonna ship that item to the store so that we can let the customer know that it’s there and ready to arrive by sending him an email. Let’s have a look at that.
Matt: Okay. So, yes, we’re going to look at the process. Once the order has been fulfilled by the logistics company, they’re going to tell us that the order is there and we’re gonna process it. So let’s see if I can use my Mac skills here. There we go. All right.
So the first thing I’m going to show you is if we head into our Service Bus instance. So, something that was announced at Build this year is a much easier way to integrate with Azure Event Grid. And we see in the Events tab here on the Service Bus instance, we can easily set up a new Event Grid Subscription. If you go to the Get Started tab, you can see it’s very easy for us to create a brand new Event synch that we then use to process the Event Hub trigger when it runs. You don’t have to go and set this up yourself, you just click on the button and provision a new Logic App function or you can set it up as a WebHook if you’d like.
So let’s see that working for real. The way I’m gonna do that is I’m going to use my API Management service. Now, like all good modern companies, Contoso does a lot of their partner interaction by provisioning HTTP APIs at the edge so their partners can call then tell us what’s going on in manage business processes. So, we would, obviously, use the partner’s application, but we’re gonna cheat a little bit today by just going in and running the Account Management Test Console to create the item arriving. So, I’m gonna hit Send now, and send in my customer order. So that will carry on and run. And we’ll go and have a look here at the Logic App that’s processed.
So, what’s happening is the API Management service API is popping a message onto the Service Bus queue. When the Service Bus message arrives, Event Grid is waiting for messages to appear, it will then trigger and alert the Logic App to say something’s happened. I’m hoping in the Run History we’re gonna see that it ran, but here’s what I prepared earlier just to get around with a little demo. I suspect what will happen is this will run and it will trigger an email. And I’ll feel it pop in my pocket in a second. But what happens is the Event Grid Trigger runs now. Now, this is all set up for you automatically when you use the Events tab that I just showed you.
So, this is actually what happens when the Event Grid Trigger runs, passes in the body of message, then we use the peek-lock method for Service Bus to go and have a look and see what message is on the queue. Now, it could be one or more messages. So what we need to do then is run a For Each loop around those messages on the queue. Oops, run those run as…oops, and there we go. Mac skills. Check those messages on the queue, process each one of them, pass the JSON out of those messages, and then complete the message. And what we do is send an email. And that’s the sort of email I just received. So that’s the Event Grid Trigger running in Logic Apps.
Let’s go back and show you exactly what we did. So, we used the API Management as our public interface for partners to tell us about business processes that are going on. The logistics partner told us that the item ordered had arrived. It then creates a message on the Service Bus queue for us to consume. And because there’s an Even Grid Trigger setup on that Service Bus Queue, it runs and Logic Apps consumes it. We go through all the messages in that queue and then perform an action, in this case, it was setting an email, sending to the customer, but it could be performing lots of other actions. Jon.
Jon: Great. Thank you very much, Matt. All right.
So just one more step to go. So how do we handle the pick up? How do we deal with that? How do we deal with the customer coming in store and, kind of, presenting themselves rather than having to go through the process of finding somebody and taking out their order number and all the rest of it? What we’re gonna show here is using Agile Functions running on a Raspberry Pi, which is a device right here on the table, which is running a machine learning model which is running facial recognition so that the customer can just walk straight up to the kiosk where this is installed, be immediately recognized, trigger a Logic App to complete the backend processing, which is actually gonna get the BizTalk Server on premise to do the bank interaction with the credit card company to do the completion of the payment from the details that we’ve collective earlier, you know, that credit card check to make sure everything was valid. Now, we’re gonna execute on that, and back to the backend using BizTalk on premises to complete the process, get the payment, and let the customer leave a happy bunny.
Derek: All right. This is really a cool stuff. So what I have here is a Raspberry Pi, but it’s not any Raspberry Pi, it has Agile Function running right on top of it, and within the Agile Function, we’re running machine learning algorithm to do facial recognition. With Contoso Retail, every customer, when they sign up, just like 007 here, we actually take pictures and with the help of those pictures, we can train the model so that later on, when they walk into the store, their face can be instantly recognized and we know that that’s the customer that’s here to pick up the backorder item.
So what do you see here? I have an array of LEDs and the moment it recognizes a face, it knows, it will turn green and show a green checkmark. So, let’s try that out really quick. So, I’m just gonna be pulling that out towards Matt. There you go. So, there is that green checkmark, if I orient it this way. Once the face is recognized, we will drop a message into Service Bus via Logic App which will then be transferred into BizTalk, which is our payment system, if you remember from the earlier demo, and that will finalize the payment and the customer can go on their way. And here, we have a T-shirt for you.
Matt: All right. My humongous T-shirt. There we go.
Derek: All right. So with that, I’m gonna hand this over to Paul who will be showing you the BizTalk version of the demo.
Paul: Hey, everyone. So let’s see. Back in the ’80s, I was using real PCs, the first Apples including Macintosh, and then IBM systems. So hopefully, I won’t have problems Mac. I am here to prove that you can, this is real again, kind of like Divya’s part of the demo. And also that you can reuse your existing infrastructures. So Contoso has invested in BizTalk Server for integration on premises infrastructure as well as their financial applications are running on IBM mainframe in this case. We’re going to reuse those.
So BizTalk Server is gonna receive from Service Bus using the BizTalk adapter for Service Bus on their C port, and then it’s gonna process that message into a message that can be understood by the BizTalk adapter for DB2. We’re gonna call our store procedure because, again, we’re doing two things. In this case, we’re gonna go ahead and process the order. But earlier, we were doing two things, we’re checking the balance of the credit card, check in the credit card, check and make sure the credit card number and CVV, all that stuff was accurate, expiration, and then putting an authorization hold. In this case, we’re gonna process authorization hold till it completed. And then lastly, BizTalk Server is gonna send, again, using the BizTalk adapter for Service Bus, the message back to the Service Bus for the Logic app to receive that and complete the whole end of the process order on that Azure side.
So, real quick, just to prove this all works. Got the mainframe that’s locked me out since I was last in there. Okay, so the point of this is also to get that message of change. You don’t wanna be using these backend systems forever, okay? You wanna change. You want to use… Here you go. It’s case sensitive. You wanna use modern on-prem and integration platform as a service to get away from these systems as much as possible. In fact, we have some of our Microsoft field people here that if you are looking to migrate off of this SAP and IBM systems, we’d love to have a conversation with you. Oops. And this is about as graph we can get in our mainframe system. Multiple colors.
Jon: That’s something I just think is great about running this on an Apple or Mac. I just think that the colliding of worlds.
Paul: All right. So, I’m just gonna make a real quick query to make sure that that order have been processed and should have been in completed state. Let’s take a look here. Ta-da. That was done earlier today this morning, and it is in complete state and is for the customer. Scroll to the right. Scrolling here is so much fun. 007. Well, trust me, it’s for that customer. Great. It’s so much fun. I love that whirl, especially on a podium with a Macintosh. Thanks, Derek. All right.
So, just to recap, we used the internet of things with intelligent devices on the edge of machine learning. And then with that, all that was running in the context of a server-less compute model with Azure Functions. We’ve wrapped and protected and published our APIs with API Management. The actual application in the cloud is a Logic App workflow. And that then, is connecting through a VPN connection to BizTalk Server on prem. It turns out it’s also running in Azure data center in the U.S. and then that’s communicating across the wide area network to our mainframe in Redmond, Business Class E12, and you’re able to reuse your existing infrastructure while taking advantage of the latest and greatest technologies in cloud, machine learning, really cool stuff.
Jon: Stuff. Thank you very much, Paul.
Okay. So let’s just do a quick recap of what you saw. One of the first things that you actually didn’t see, and you will see later, is that the Logic Apps that were running in the first couple of demos are using our new Integration Service Environment. This is a new capability which is in private preview. So if you’re interested, come and talk to us. It gives you the option of creating your own private, dedicated, isolated, set of compute in Azure that you can deploy Logic Apps into to be able to give you VNet connectivity that are on premises without using the data gateway providing, you know, isolated storage and other sort of actions to that to give you predictability in the performance and for a fixed price per month.
The second thing you saw was a new SAP Trigger. The ability to just trigger directly from SAP without having to mess around with doing sort of gymnastics that you’ve had to do in the past and make that super easy. We will go into that in a lot more detail this afternoon. The next thing you saw was a quick glimpse of our new SFTP Connector. So this is gonna be released alongside the existing one, and we’ll sort of phase out it over time. But we heard, you know, consistently over the last year or two that you really needed better support for large file processing. This will bring you that. Initially, a gigabyte, but there’s no reason why we can’t go much higher than that. And so we encourage you, when this is released, hopefully, in the next couple of weeks, to try this out and give us your feedback and tell us the cool things that you’re building with this.
The next thing you saw was the Event Grid Trigger for Service Bus. This is a big deal because the Service Bus is one of our most used connectors and most used services with Logic Apps, people are implementing, you know, ESP type pans and other sort of pops up type things. And the ability to actually trigger off of new messages arriving in queues and topics for subscriptions was a big deal because you’re constantly having to poll previously to using your Event Grid for this. And now you’ll just get fired immediately notified that our new message as a process and pick them up without having to sort of hammer the Service Bus endpoint all the time using the connector just because you are waiting for this message and wanted to process this as quickly as possible.
The next thing you saw, which is obviously, you know, super cool, is the ability to run functions on the edge, the ability to run functions anywhere on the Raspberry Pi, running a whole bunch of cool stuff, whether it’s machine learning or other function code being able to then, you know, inside of that doing facial recognition. And the final thing that I wanna call out is this seamless integration between all of those, not the WiFi network notwithstanding, the ability to put all of this together and build that end-to-end holistic solution.
You know, I wanna give thanks to Matt and Divya and Paul and Derek for putting this together in very short time schedule. This is not a lot of work to kinda create these things, and it just, you know, speaks to the power of the integration platform.
Okay. So let’s sort of to wrap this up. As I said earlier, you know, we are all in this together. So, you know, this is the opportunity, you know, all 400 of you to introduce yourselves with your peers here and get some of the other people. Please, come and talk to us. Tell us the challenges you’re experiencing, the success stories that you’ve had. You know, it’s great to see Matt Fultanka talking about Confused, the real, you know, born in the cloud, native, disrupting innovation in a very established industry like insurance. You know, this is our, sort of, products in action if you like. So you know, please have fun. You know, this is your event, this is your community. Dig in and dive in and talk to people.
As I said earlier, we have more people here than I think we’ve ever had before, so I get 21 people here. So here’s all the sort of faces and names behind everybody. I’ll just leave this up here for a couple of minutes so you can memorize it. All the teams that we have in Logic Apps to Functions to API Management, Service Bus, Event Grid, etc., we are all here from the integration teams back in Redmond. So, you know, please talk to us.
Okay, I’ll leave you with this. You know, I think this is a great quote from Satya recently, “Every company is a software company.” You know, Matt said this from Confused, it’s true, right? If you’re not a software company, if you’re a company that’s not built around the innovation and the technology that you’re using, it soon will be. Or somebody else will do it and disrupt you and make you a follower rather than a leader. It’s really you thinking about your own future as a digital company. So as we approach the end of this decade, which is kinda scary in itself and we head towards the 2020s, you know, what are you gonna change to create the future? Let me leave you with that. And thank you very much, and I hope you have a great conference. Thanks.
Fill the form below to get all the presentations delivered as a single zip file in your mailbox.
byJon Fancey & Matt Farmer
byMicrosoft Integration Team