Integrate 2021 Day 2

Integrate 2021 Remote – Day 2 Recap

Published on : Jun 2, 2021

Category : Events

BizTalk360

Author

Table of contents

1. Microsoft’s Event Bus in the Cloud – Event Grid the missing piece in Azure Integrations Services
2. New Scenarios with Azure Logic Apps
3. Integration beyond point to point; Implementing Integration Patterns using Azure
4. Build and release your BizTalk applications with Azure DevOps
5. Azure and BizTalk 2020 – Better Together
6. BizTalk360 v10 – Not just a Phase lift
7. Service Bus Updates
8. Lightning Talks
9. Azure API Management Updates
10. Functions: Event driven Compute Patterns and Roadmap

With Day 1 of the event rolled up, the platform was all set for the next day of the event.

Day 2 of Integrate 2021 has been a roller-coaster experience for the attendees with some amazing sessions, power packed demos, and certainly, loads of updates from the Microsoft Product Team and MVPs.

Fasten your seat belts as we’ll take you through all the significant updates and highlights on Day 2 of INTEGRATE 2021 Remote.

Microsoft’s Event Bus in the Cloud – Event Grid the missing piece in Azure Integrations Services – (Steef-Jan Wiggers)

Steef-Jan Wiggers working as a Technical Integration Architect at HSO who is also an Azure MVP for the past 12 years, gave an informative session on Event Bus concept and the Microsoft Azure Event Grid service in this regard. He covered the following concepts in his sessions.

  • Event Bus and some real time scenarios where Event Bus can be used.
  • Event Grid terminologies.
  • Real time use cases for Event Grid.
  • Points to be considered while using Event Grid.
  • Competitors of Azure Event Grid.

He started the session with a D365 FSCM Integration solution which leverages Business events. The architecture consisted of Event grid topics, Storage account, Service Bus, Logic Apps and Key Vault.

D365 FSCM Integration solution

Event Driven Architecture

He explained about the event driven architecture where the events from the source are sent through the Event Bus to the required target systems using the required APIs.

When to use Event Driven Architecture?

Event Driven Architecture can be used when the events are real time that must be processed by multiple event processors with minimum time lag. This can be used to process high volume of data with high velocity.

Benefits of Event Driven Architecture

  • Efficient Development: Speed and cost. All events are processed through the central event bus.
  • Better user experience: Think and build around events.
  • Lower running cost: No need of polling
  • Resiliency: Service fails, it can restart and replay events from the Event Bus.

Event Grid Architecture

Event Grid consists of Event grid topic and Event grid subscriptions. The events are sent from the event sources to the event grid topics and the events are then forwarded to the event subscriptions. The events are then received by the event handlers.

Event Grid Architecture

Event Grid Concepts

  • Events: What happened
  • Publishers: Where it took place
  • Topics: Where publishers send events
  • Event Subscriptions: how you receive events
  • Event Handlers: the apps or the services reacting to the events

Scenarios where Event Grid can be used

He explained an Azure Integration scenario – speech to text conversion, where event grid can be used. He also explained a complete Serverless solution which uses Event Grid.

Scenarios

Points to remember in Event Grid

  • Enable dead-lettering of events to storage container.
  • Use Event Grid viewer for trouble shooting.
  • Use Event Grid Domains to manage custom events sent to multiple consumers or endpoints.
  • Maximum size of an event is 64 kb and the maximum size of an event batch is 1 MB.

Competitors of Event Grid

He listed the competitors of Azure Event Grid. They are as follows.

  • Event Bridge from AWS
  • EventArc from Google
  • Every Bridge from TRIGGERMESH

New Scenarios with Azure Logic Apps – (Wagner Silveira)

Wagner Silveira, digital practice Lead at Theta, New Zealand, handled the session on “New Scenarios with Azure Logic Apps” at INTEGRATE 2021. Azure Logic Apps has got a new runtime in GA now, an improved designer tool, different models of execution, and so on. This enables scenarios that were not possible or complex to be implemented in the past. In this session, Wagner demonstrates some unlocked scenarios and how they will affect some of your previous and future design decisions.

  1. Identity-Based Authentication
  2. State * Speed
  3. Logic Apps anywhere
  4. Private Network Integration

Wagner started the session with Identity-Based Authentication, where he explained how it was earlier, what is new and how it is done. He demonstrated Inbound Authentication using OAuth.

Inbound Authentication using OAuth

The findings of the demo were,

  • Inbound Auth is not yet available in Standard
  • Disabling shared access keys is not possible. It is still the preferred method
  • The operator on policies is not available
  • Managed Identity is not enabled for selected connectors

The next demo was on State * Speed which is quite new with Logic Apps. Like the previous demo, he explained how it was earlier, what is new and how it is done. He demonstrated a comparison of the Stateful and Stateless Logic App.

Stateful and Stateless Logic App

The findings of the demo were,

  • Each Logic App has a single state
  • Managed connection triggers and variables are not available in stateless

Followed by that, the third demo was based on Logic Apps anywhere. This demo covered how Logic Apps choice of deployment and choice of packaging has evolved. He demonstrated Logic Apps Standard on Docker.

Logic Apps Standard on Docker

The findings of the demo were,

  • Choice of Storage
  • Management plane
  • Usage of Built-in connectors and Managed connectors

The final demo was based on Private Network Integration. This demo highlighted how both inbound and outbound integrations are done. He demonstrated a scenario on Accessing SQL Server using Hybrid Connection.

Accessing SQL Server

The findings of the demo are,

  • To use hybrid connections, you should use built-in connectors.

This session was engaging with demos like,

– VNET Integration

– On-premises execution

– Development and debugging

– OAuth authentication

All these were put up into different scenarios pitching in the fact that Logic Apps is bringing workflows closer to app development.

Integration beyond point to point; Implementing Integration Patterns using Azure – (Eldert Grootenboer)

In this session, we had Eldert Grootenboer, Technology Lead – Azure & Integration, who has a wide knowledge and experience working with Azure Integration Services.

Eldert introduced a book named “Enterprise Integration Patterns”, which discusses different Integration patterns and standards to be followed while designing an integration. If you don’t have a book, check out the below website with all the patterns covered in this book.

Reg: https://enterpriseintegrationpatterns.com

He started the session by defining Integration Pattern and Why should we use the Integration Patterns. He also gave a quick head up about different Azure Integration Services

Azure Integration Services

Following Integration Services, he shared the advantages of implementing patterns with Azure Integration services that listed below:

  1. Compose using Different services – Different Azure services supports Integration patterns by design.
  2. Various Options – Azure offers various options; say, if the need is to set up a router, it is quickly achievable using Azure Service bus Topic.
  3. Azure Architecture centre – There are enormous integrations readily available for reference in this repository that can be implemented in real time.

To make the session even more interesting, he took us through a simple scenario that transfers the inventory data to the external API to process it.

Simple Integration

In the Demo, he introduced various patterns listed below and how we can implement those patterns in the inventory management system.

  1. Process Manager (Logic App) – Pattern that defines the workflow of a small chunk of tasks that can accomplish a big job at the end.
  2. Channel Adapter (Connector) – can access the application’s API or data and receive messages and invoke functionality inside the application.
  3. Message Channel (Service Bus) – connects the application where one application writes information to the channel and reads that information from the channel.
  4. Message Endpoint (APIM) – Connect an application to a messaging channel using a Message Endpoint, a client of the messaging system that the application can then use to send or receive messages.
  5. Message Router (Topic) – which consumes a Message from one Message Channel and republishes it to a different Message Channel depending on a set of conditions.
  6. Pipes and Filters (Function App) – divide an enormous processing task into a sequence of smaller.
  7. Publish/Subscribe channel (Event Grid) – delivers a copy of a particular event to each receiver.

Build and release your BizTalk applications with Azure DevOps – (Samuel Kastberg )

Samuel Kastberg, Senior Customer Engineer at Microsoft who has years of experience in the Microsoft BizTalk Server. He was also a part of the conference for multiple years. He gave an information session on setting up, build and deploy BizTalk projects using Azure DevOps.

He set the context for the session:

  • Tips and Tricks handling BizTalk projects
  • Demos on things you can do with both Azure DevOps and other tools
  • All scripts shared in Github
  • Don’t Expect the perfect process as it depends on the various business requirements

Demo

During the demo, he mentioned what is needed to deploy BizTalk Applications like

  • VM’s which are IaaS
  • Git for repos
    • TFVC
  • Agents on the servers
    • Local execution
  • Consider one server primary
    • Secondary, GAC
  • Secrets in Release variables like the key vault

A Scenario of how it works

Scenario

Build Server

He explained the process of building the server and the prerequisites.

  • Updated windows
  • NET 4.8 Targeting package
  • Visual Studio 2019
  • BizTalk Server Developer Edition
  • BizTalk Server Visual Studio Extension
  • Azure pipeline agent

Once all the prerequisite conditions are met, next he introduced the artifacts library in the demo where he emphasizes the importance of creating a new feed and connect to feed in VS options.

Build Server

BizTalk Application Project in Visual Studio 2019

Here he mentioned the packages that are available in VS 2019, how users bind files with parameters and how to parameterize the values with tokens. Also, during the build, the file is updated with tokens in bindings.

BizTalk Application Project

Deploy

He mentioned users to validate before starting the deployment like if there are any instances active or do, they have some backreferences, etc.

Deploy

Scripts and resources that you can look for!

In the end, he added few links for the attendees to have a look at the GitHub for BizTalk developers

Scripts and resources

With the Demo on the above-mentioned, Samuel completed the presentation, and the session ended with a Q&A session with a lot of insightful answers.

Azure and BizTalk 2020 – Better Together – (Stephen W. Thomas)

Stephen W Thomas, Azure integration Architect at Kendall Technologies, Inc who is also an Azure MVP (Most Valuable Professional) for 16 years gave a session on “Azure and BizTalk 2020 – Better Together.”

He covered the below topics

  • Why BizTalk and Azure together?
  • How to implement?
  • Roadblocks / Pushbacks

He started the session with a short introduction about him and shared an announcement about his courses on Logic Apps, Azure, and BizTalk Server on the Pluralsight and moved to the main agenda of the session.

Phase 0 – Bonus

He showed the practical application of Logic Apps as a bonus to receive notifications in email and mobile when his son awakes and needs help. The below image shows the notification received in the email and mobile when his son awakes and needed help.

notifications in email and mobile

Phase 1 – Why BizTalk & Azure?

He emphasized that the demand is higher for integrating Azure with the BizTalk and gave insights on the stepping stones to Azure which is shown in the below image

stepping stones to Azure

Benefits of leveraging Azure

  • Increased security.
  • New scenarios like new connectors, processing non-traditional messages like images.

He added some practical scenarios for the need of new connectors in social media monitoring, customer communications, cross-team communications, and web-centric connections.

Phase 2 – How?

BizTalk server 2020 has built-in adapters like Event Hub, Service Bus, Blob Storage, and Logic Apps which facilitates migration to Azure.

He added some general issues we might face when migrating such as the personal account not supported by BizTalk server 2020 and TLS issues while making the HTTP request by the HTTP adapter.

Next, he presented a demo using built-in adapters like Event Hub adapters, Service Bus adapters, Blob Storage adapters, and the HTTP adapter and showed its configuration and uses.

He showed a demo on passing a sample message on the Service Bus Queue from the Service Bus explorer, sent the video through the Blob Storage send port and showed the result in the Blog Storage container, and passed the sample in Logic Apps send port to find the sentiment by the sample Logic App.

Phase 3 –Roadblocks

He cited few roadblocks moving to Azure.

  • New investments.
  • Sensitive data like credit cards, etc.
  • Azure improves frequently- making easier, new features, and JITA (Just In Time Architecture).

BizTalk360 v10 – Not just Phase lift – (Saravana Kumar)

Saravana Kumar, Founder & CEO of Kovai.co and the creator of BizTalk360 started the session with a small introduction to BizTalk360 and how it can be efficiently used to manage and monitor the BizTalk environment with the different operational tools bundled within BizTalk360. BizTalk360 being the one-stop monitoring tool to monitor the BizTalk server helps in overcoming the challenges faced with the BizTalk server.

Challenges in Microsoft BizTalk Server

The BizTalk360 v10

The session mainly highlights the features and improvements that have come up with BizTalk360 v10.

After a year of hard work of the team, now we have the complete picture of BizTalk360 v10 with its new look and feel. Saravana highlighted some of the important features and how the look has been changed for more user-friendly operations.

Some of the highlights of BizTalk360 v10:

Landing page

The first page that you see when BizTalk360 loads is the landing page which gives the complete overview of your BizTalk environment.

BizTalk360 landing page

Business Rules Composer

XSLT is used to transform rule conditions (XML Data) using XSLT. Now, the UI design has been improved.

Tracking Manager

The new look of the Tracking Manager now makes it easy to enable/disable tracking for the ports without logging on to the BizTalk admin console.

Tracking Queries

With the new UI, it is easier to view the complete message flow and drag and drop the artifacts in the flow which is not possible with the actual BizTalk admin console.

Tracking Queries

Governance and Audit capabilities

Audit data representation is improved to tabular format in Excel export and support’s PDF download option.

Governance and Audit capabilities

Topology

The Topology diagram represents the pictorial representation of BizTalk Servers and SQL Servers in a BizTalk Group. This helps in understanding the architecture of the BizTalk Group.

Monitoring Dashboard

Monitoring being the core functionality of BizTalk360, the monitoring dashboard gives you the complete view of the state of the artifacts that are being monitored. The new version has a completely new dashboard with the different filtering capability added and with the full screen mode capability with dark themes supported.

Monitoring Dashboard

Analytics

Analytics feature being the third pillar of BizTalk360 has also got its makeover in v10 with a lot of enhancements and performance improvements added in viewing the performance related information of the BizTalk and SQL servers.

Analytics

And much more. Each feature has its own advantage of the new cleaner and nicer UI. To be added, there are new features as well into the product such as

  • Monitoring SQL server availability
  • Support for nSoftware adapters

We have mainly concentrated on the UI and performance improvements added to this version.

UI/UX Improvements

BizTalk360, with its new UX, now becomes much more efficient for the admin and support teams managing their Microsoft BizTalk server environments.

Service Bus Updates – (Clemens Vasters)

Clemens Vasters, Principal Architect – Azure Messaging team at Microsoft, entertained us with an informative session on “Service Bus Updates”. 

He covered the session with the following agenda:

  • Service Bus
  • Java Message Service (JMS) 2.0 support
  • Azure SDKs
  • Large message support >1 MB
  • “Durable terminus” connection recovery
  • Sneak peek: “Replication tasks.”

Service Bus 

Service Bus is a “swiss army knife” for messaging driven workloads.

Service Bus is an enterprise standard message broker that handles messages in sequential order with high reliability. 

He emphasized using Service Bus along with other message-oriented services like Event Grid, Event Hubs, and Relays based on the architectural need.

Service Bus comes with a couple of tiers, namely Standard and Premium, with capabilities to support appropriate use cases.

Service Bus JMS 2.0 Support in Premium

The product team decided to invest in JMS 2.0 because Java dominates in many industries verticals that traditionally have used message queue technology.

This support will help to support the ongoing life & shift wave of large enterprises moving their core workloads dependent on JMS into the cloud.

“Service Bus JMS 2.0 support is the biggest single feature update since 2011.” 

It comes with the following capabilities: 

  • Service Bus is a JMS 2.0 compliant message broker
  • JMS 2.0 for Service Bus is an OSS and standards collaboration result
  • JMS compliant message selector syntax
  • Conditional receive operations on a queue
  • JMS transactional model support 

Service Bus JMS 2.0 – What’s new?

  • Temporary Queues & Topics – Exclusive receive access by creating a party
  • Ad-hoc, receiver-created topic subscription
  • JMS message selector syntax
  • Implicit coordinator entities for transactions 

New Azure SDKs: In case you missed it

He also highlighted the updates in the recent past.

  • 4th generation Service Bus API, fully integrated with Azure SDK, is now available
  • Terminology alignment and full access to AMQP message metadata
  • C#, Java, Python, and JavaScript are all first class with the same level of commitment and feature parity
  • New Azure Functions and Logic Apps bindings are coming this year. 

Soon: Support to Large Message

Service Bus Premium will support up to 100 MB per message using AMQP (including JMS). It is going to be an opt-in feature for each queue and topic. 

Soon: Durable Terminus

Connection drop happens much more frequently today than they used to be in the past. With most brokers, once the connection is lost between client and broker, all the states associated with it will be lost.

With Durable Terminus, it automatically recovers connections even before you notice it. 

Sneak Peek: Replication Tasks

This is very similar to the Logic Apps Automation Task that we had seen in one of the previous sessions. You can use pre-defined templates to automatically replicate messages from a source Service Bus to a target or perhaps automatically monitor dead letter queue when queue size exceeds.

Service Bus updates

Lightning Talks – (Derek Li)

Derek Li, Senior Program Manager for Logic Apps, and his colleagues from various departments like Engineering, Content Designing, UX designing joined us for a lighting session.

Derek invited Sonali Pai, Workflow Designer at Logic Apps team, to present the UX improvements. She took us through the significant updates made in the logic app designer.

logic app designer

The above picture represents the new designer with the following improvements:

  • It has a modern look and feel.
  • The cards are no more clunky filled with information.
  • The edges and card design are much cleaner.
  • The designer itself is more compact and easier to visualize.

Separated Graph visualization

Separated Graph visualization

Graph visualization is now separated from the configuration of the cards using a configuration panel.

The panel has the card configuration split into nice tabs that users can toggle in between as they edit their workflow.

Integration with Dagger and JS Plumb

With the new integration to the above two open-source java script libraries for rendering nodes and building connectivity in apps respectively, now logic apps support complex graphs and parallelism.

Next up – Xiaowei Jiang joined us from Design & Research team to represent the design-oriented updates. With the new standard logic app, the design team has refreshed the designer.

Designer Refresh

The difference between the old designer and the new one is how you configure the actions into the logic apps.

Designer Refresh

Now the actions to be configured on the right panel, and the workflow stay intact. There are also other improvements like

  • Dark theme support for designer
  • Run after configuration refresh

Next, Laura Dolan from the Documentation team took the online stage. She demonstrated how you could contribute to the open-source Logic Apps Documentation and the step-by-step process to commit your request through GitHub.

Parth Shah, Program Manager, joined us followed by Laura. He took us through a quick demo on Automation Tasks which is in-resource automation, by abstracting away the complexity of Logic Apps.

Automation Tasks

For instance, you can delete the old blobs simply using the templates in the Tasks tab without having to provision a logic app workflow. See all the available templates.

Finally, Henry Liu, Program Manager, concluded the lightning talk with a demo on one of the Logic Apps capabilities – Inline code. Given logic apps standard is built on top of functions; the team has included inline JavaScript as one of the actions.

Inline code

Henry showed us orchestrating a simple HTTP triggered logic app using the new inline JavaScript action.

Azure API Management Updates – (Vlad, Mike, Miao, Elizabeth)

In this session, we had three presenters from the Microsoft Azure API Management team to deliver all the latest updates related to the API Management service.

Elizabeth started the session with an overview of API Management and explained the 3 main components in the service which include the Developer Portal, the API’S, and the Gateway. All these components are Azure hosted and fully managed by default but the option to self-host the gateway in the developer portal is also available.

She mentioned that the self-hosted gateway enables to efficiently and securely manage APIs hosted on-premises and across clouds from a single API Management service.

API Management's architecture

The lifecycle of an API was also briefly explained by Elizabeth,

lifecycle of an API

After which, Miao took up the stage to explain the newly announced WebSocket APIs which is in public preview. It allows to manage, protect, and expose the WebSocket APIs through API Management alongside the REST APIs.

Then, he showed a detailed demo of configuring Web Socket APIs in the Azure Portal and explained about a system operation called OnHandshake which initially allows to catch the WebSocket handshakes and deploy policies to it.

The below image shows the output obtained by testing the WebSocket APIs in the Azure Portal

testing the WebSocket APIs

Next, he showed how to configure the same in the developer portal and this is how the output looks like

Configure

Self-hosted gateway on Azure Arc (Public Preview)

  • Expands support for hybrid and multi-cloud environment
  • Enables customers to securely and efficient manage APIs running anywhere

Self-hosted gateway on Arc for Kubernetes (New)

  • Deployed as a cluster extension
  • Unified and consistent operation experience across all Arc-enabled Services
  • Support for both AKS and Arc-connected clusters

Miao also showed a very quick demo for enabling Azure Arc on self-hosted gateways.

As the final part of the session, Mike presented about securing your APIs with validation policies. He mainly focused on the policies that were added to API Management in the last few months which would help to validate response and requests against the API Schema.

The image below represents the list of validation policies and the most common vulnerabilities from which your APIs should be protected.

list of validation policies

He also gave a detailed explanation with a demo for using each of those validation policies for overcoming different types of vulnerabilities.

Mike then moved on to the topic “API Portal” which is built on top of the developer portal in API Management.

API Portal

He concluded with a demo by showing how to spin up the API portal, where he also shared the link for accessing all the information and source code which is available as a public open-source project on GitHub.

Finally, the session ended with all three of them answering a lot of questions related to Azure API Management.

Functions: Event driven Compute Patterns and Roadmap – (Eamon O’Reilly)

Eamon O’Reilly, Lead program management team on Azure Functions at Microsoft gave an insightful session on Azure Functions. This session covered the event driven compute patterns in Azure Functions and Functions Roadmap.

Eamon started the session by explaining how native apps in Azure traditionally and slowly moved to the advancements it has made. Impacts that Azure PaaS service has made in the cloud space was also touched in the session.

A turn-key platform for Applications

Azure Function was developed to make coding easy and effective.

Patterns for Functions

Premium Plan in Azure Functions

Azure Function team has spent most of their time past year to make this premium plan publicly available.

Along with these advancements, Durable functions now uses code first approach when many challenges will be taken care natively.

Some of the important improvements that premium plan can provide you is,

  • Scalability – when the configured instances cannot holdup the load, the instances are smart enough to scale automatically to process the load and scale backdown after processing.
  • Cold Start Controls – This premium plan can solve the cold start issues that current function has. Functions in premium tier used pre-warmed instances.
  • premium plan

Event-based Automation

Event-based Automation

When an event is triggered, automation can be made with full flexibility using Azure functions as the tooling in function offers common tooling platform that every team can use. Also, Azure functions supports nearly all popular languages, and each team or team member can choose the preferred language and make the automation.

Azure Functions ARC

Now while deploying an Azure function, you can use a Custom region to deploy you code. These custom regions are the servers that your organisation has. You can connect and deploy your code to any location that you have access for.

Roadmap
  • Functions with Azure Arc generally available
  • Azure ARC features are now available in Durable functions also.
  • Full integration with Azure Ecosystem
  • Storage providers for Durable functions generally available
  • Inner loop and outer loop from portal / IDE / GitHub / DevOps

That’s the wrap!

We are now closing this exciting day 2 with a ton of informational content from the Microsoft Product Group and Integration community while keeping our spirits high for the final day.

Stay tuned with #INTEGRATE2021

This blogpost was prepared by:

Ranjith
Amritha
Nishanth
Christhini
Naveen Shanmugam
Nadeem
Modhana
Sri Hari
Praveena