Introducing BizTalk Server Data Monitoring in BizTalk360 (version 7.9)

Published on : Mar 3, 2015

Category : BizTalk360 Update



When demonstrating BizTalk360 to customers we would often be asked for “How can I monitor for…” and then they would go and describe something based on the data in their BizTalk environment. When we looked in more detail, it would  normally take us into running a query against one of the data sources in BizTalk and alert them when certain conditions are met. There are a lot of scenarios where this applies, it would range from a simple “How can I monitor for any suspended messages in my environment” as shown below Message Box Query Suspended Instances to a very complex scenarios where we want to find transmission failures of messages of a particular schema: Tracking Query Transmission Failure or even Monitoring based on ESB Faults of a specific severity: ESB Exceptions Severity Query These customers would then also tell us how frequently they wanted to run the queries based on their business requirements such as the frequency of daily validations (ex every 15 mins, 1 hour etc), end of business day or even monthly events such as month end processing. When thinking on how to implement this in BizTalk360 in a generic way that could benefit all the customers we went on the drawing board and what made more sense was to expand the Process Monitoring which was already delivering some type of tracking data query monitoring focusing on message processing on Receive Locations and Send Ports to start including all other query types.

What is possible in Data Monitoring?

We implemented the ability to automatically run queries against multiple data sources in BizTalk which include: Message Box Queries, Tracking Queries, BAM Queries, EDI Queries and ESB Queries. The way it works is by allowing you to define how do you want to monitor data in each of these sources by defining:
  • Basic Details – we need to associate the data monitoring with an Alarm (a logical container of monitoring rules, see more about it here) and provide a name for identification purposes
Data Monitoring Basic Details
  • Data Filter – allows you to define the query you want to monitor by selecting from the available queries and allows you to filter the result set to your needs
Data Monitoring Query Definition
  • Notification Condition – with these thresholds the result from the query will be evaluated and in case of any threshold violation you will be notified via email and the information will also be made available on the data monitoring dashboard; for Message Box Queries we allow you to include the service instance details in the notification email for easier understanding of the problems occurring.
Data Monitoring Thresholds
  • Monitoring Times – this is where you can specify how frequently you want to run this validation (Daily/Weekly/Monthly) and then how long and how frequently within the day (Business hours within the day and frequency within the day)
Data Monitoring Frequency
  • Date/time filters – this option allows you to choose if you want to run the query with a date time filter on one of the date time columns available for the query type which allows you to either look at the overall result of the query or to the time slot being tested
Data Monitoring Date Time Filter This is how we can describe a data monitoring rule, the monitoring service will then execute the queries periodically, validate it against error/warning thresholds and it will alert accordingly (email, sms etc) and also save the results into the Data Monitoring Dashboard (this is a calendar based dashboard where you can see a month and day view which will show a visual representation of the executions). In the month view of the calendar you will be able to see on which days there were data monitoring problems and by clicking on a particular day you will see on the day view of the calendar dashboard the data validations for that day (you can understand the time window of a validation)

Automatic Actions for Message Box Queries Monitoring

We also had a lot of customers asking for automatically taking actions based on the outcome of a data monitoring validation. This would help for example in scenarios where we want to monitor for the existence of any suspended messages that by the end of the business day are no longer relevant and should be terminated. So instead of waiting for a BizTalk Super User to go and manually run the query to find and terminate the messages you can now automate that in the Message Box Queries Monitoring configuration, here is how: Data Monitoring Set Action You now have the option to automate actions available for service instances (Resume, Suspend or Terminate) and you can choose to perform them on every execution of data monitoring, on Error Condition or Warning Condition. Any of these action performed will generate auditing information so that you can know when this was executed from the Governance And Auditing screen and at the same time will be populated into the Data Monitoring Dashboard.

Data Monitoring Dashboard

A vital part of Data Monitoring is how can you see the information executed by BizTalk360, I mentioned across this article multiple times which information we are making available, and so if you navigate to the data monitoring dashboard the first thing you need to look at is the calendar view which will help you understand what on which days you faced problems: Data Monitoring Dashboard Month View Then when you click on one of the days where we have problems you can visualize the day calendar view which will let you understand which data monitoring rules were executed and what was the result, we are colour codding both the type of query and the result of the validation. It comes with rich filter options, you can uncheck the check box and corresponding data monitoring results will be hidden (as shown below) Data Monitoring Dashboard Day View If you need to understand what happened for a particular execution we can click on one of the entries in the day view of the dashboard and below window will be displayed. Message Box Data Execution Results And here you can see the day and time for the validation, the warning and error thresholds, the query result count and which filters were applied (the query). We can also tell you the status of the validation (Success, Warning or Error) or if any action was performed. The “Task Result” tab will show all the actions performed (ex resumed/terminated instances and their details).

Are you ready? give it a try!

We believe this provides a big amount of value to you and that BizTalk360 monitoring story is now even more complete, please go ahead and try it in your environment and if you don’t have one then trial BizTalk360