Scenario based BizTalk Server Monitoring using BizTalk360

Published on : May 31, 2014

Category : BizTalk360 Update



In a real world BizTalk server environment typically you’ll have one or more set of BizTalk applications deployed. Examples include Real Time processing, Batch Processing, SAP Integration, Oracle Integration, Legacy Integration etc. Each set of BizTalk integration scenarios will have different application monitoring requirements, even though all of them are deployed and running in the same environment. The infrastructure monitoring aspect like disks, CPU, memory, etc. will be common between these different BizTalk applications, but their application monitoring requirement will differ from one another. Let’s take look at a scenario here and see how BizTalk360 can help to create a bespoke monitoring/notification pack for a specific integration application.

Scenario: Dynamic CRM – SalesForce Integration

In this example you have built a set of BizTalk applications for a Dynamics CRM to SaleForce integration with SAP lookup. You are taking data from Dynamics CRM, making some look up for customer information in SAP system, enhancing the message with additional information present in your SAP system and then sending that data to the Salesforce system. BizTalk Server DynamicCRM - SAP - Salesforce Integration When you build this integration system using BizTalk Server, typically you would have custom built some orchestrations , and configured the corresponding receive ports, send ports, maps, pipelines etc. For illustration purpose, let’s assume your solution is structured as shown below, you have 4 BizTalk applications (referenced one another and built in a modular way) with corresponding BizTalk artifacts.
BizTalk Applications Receive Ports Send Ports Orchestrations
ACME.CRM.Integration.Common ACME.Orch.Common
ACME.CRM.Integration.SAP ACME.TwoWay.SAP ACME.Orch.SAP.Customer
ACME.CRM.Integration.DynamicCRM ACME.OneWay.DynamicCRM ACME.Orch.Dyn.Customer
ACME.CRM.Integration.SalesForce ACME.TwoWay.SalesForce ACME.Orch.SF.Customer
Once the solution is developed and deployed into your production environment, then it becomes critical to keep an eye on the health of various things that are relevant to these applications (all related to single integration application Dynamic CRM to Salesforce), which will include the following
  • Monitoring BizTalk Artifacts State
  • Monitoring runtime service instances
  • Monitoring process/transaction volume
  • Monitoring supporting components
  • Monitoring Infrastructure

Monitoring BizTalk Artifacts State

As illustrated in the above table, there are bunch of BizTalk artifacts (Receive Ports, Send Ports and Orchestrations) that all work together to make this integration seamless. Even if one of the artifact say for example the ACME.OneWay.DynamicCRM receive port is down for some reason (ex: wrong credentials), then the whole system will stop functioning. So, it’s critical to make sure all the artifacts are in a healthy running state.

Monitoring runtime service instances

Once all the artifacts are deployed, configured and running correctly, there will be 100’s of messages flowing through the system (example for every customer record created in Dynamic CRM). Each message flow will create series of service instances in BizTalk (pipeline executions and orchestration executions). For a healthy working system, all these services instances must complete processing successfully and clear up. There should not be instances hanging in error states like Suspended (Resumable), Suspended (Non-Resumable) etc. Even too many of the services instances in successful running states like Active, Dehydrated, Read to Run etc can highlight some problem

Monitoring process/transaction volume

Based on your business requirement you’ll probably know the volume of message you need to process every hour or business day. In this case the requirement may be you are expecting to process minimum of 200 customer records every hour during the business hours 9am to 6pm, Monday to Friday. If the volume is not met, then you probably will know there is a problem with some up stream systems like web application which is inputting records into the Dynamic CRM system.

Monitoring supporting components

An integration solution like this will most probably will have some external dependencies. Event Log Sources: You may wanted to keep an eye on any events that’s been raised with event sources “Salesforce”, “Dynamic CRM”, “SAP” etc in the windows EventLog. Business Rules Service: If in case you have used the BizTalk rules engine for some business rules, you will need to make sure the rules engine NT service is running correctly. SQL Jobs: You may have configured some custom SQL Jobs to do some data clean-up, it will be important to keep an eye on the healthy of those SQL jobs execution BizTalk Host Instances: All of your BizTalk artifacts like receive handlers, send handlers and orchestrations are going to run within the scope of one or more BizTalk host instances. If any one of the host instance relevant to our BizTalk applications is not running, then the overall integration is going to fail. So it’s important to make sure all the relevant BizTalk host instances are healthy.

Monitoring Infrastructure

And finally the infrastructure monitoring is also key for the general well being of your BizTalk environment. It’s crucial to make sure your CPU is not hiking above 90% consistently, you memory is not always at the edge, etc. otherwise the system will get into unnecessary throttling states.

BizTalk360 Scenario based Monitoring

Until this point we were explaining about a potential integration scenario and list of things we need to monitor to ensure the successful functioning of the system. One important point to consider here is, this may be just one of the integration solution you deployed in your BizTalk environment, there may be various other similar BizTalk applications deployed. It’s important we set up monitoring and alerting that’s specific to this particular integration (Dynamic CRM to Salesforce) scenarios, hooking up all relevant components. If you monitor artifacts individually, you’ll not able to correlate them together for a single integration scenario. BizTalk360 is designed keeping this mind, let’s take a look at how you can efficiently monitor this scenario with BizTalk360

Introduction to BizTalk360 Alarms

Alarms in BizTalk360 is like a custom management pack, which allows you to combine various components that are relevant to an integration scenario into a single monitoring and notification unit. Example: In our case, we can create an alarm called “Dynamic CRM – Salesforce Integration – Monitoring/Notification” alarm, and configure various things that are relevant to this integration. As shown in the picture, for successful monitoring of this CRM integration scenario you need to keep an eye on various things like BizTalk applications artifacts (Orchestrations, Send ports, receive ports) health, external systems health like Rule engine, host instances, SQL jobs etc. BizTalk360 Scenario based Monitoring You also need to configure the notification channel (email, SMS etc.), so that BizTalk360 can send notification if something goes wrong. In our example we used a custom distribution list email alias “ACME CRM Integration distribution list“. As you can see the BizTalk360 alarm helps you to create a simple custom BizTalk monitoring and notification management pack, putting all the relevant parts together. As an integration consultant/developer you’ll have a very clear picture of what needs to be monitored for the scenario, you just need to configure it once (and occasional tweaking if required) and have peace of mind your system is under control. This is just one scenario and you created a corresponding monitoring and notification pack. You can follow the same procedure to create alarms for your various other scenarios like real time processing, batch processing, legacy processing, etc

Let’s put it in action using BizTalk360

Even though the picture and concepts looks bit cumbersome with too many components involved (that’s the nature of an integration solution), you can literally configure this setup in less than 10-15 minutes using BizTalk360. Please see the below video which explains how this can be done. Scenario based monitoring using BizTalk360 – Dynamic CRM – Salesforce Integration from BizTalk360.