Introduction:
In my current client, this is the situation: There are two separate teams involved in two different projects. One utilizing BizTalk Server 2006 building BPM solution and the other utilizing IBM WebSphere Message Broker building an even management system. The day has arrived, where we need to make these two systems talk to each other via MQ. Looking at very high level picture wearing an architecture hat this is what you would have drawn on a white board:
Easypeacy, BizTalk got an MQ adapter, so it can put messages into MQ and WMB will pickup the messages from there and both the system can talk to each other seamlessly. But it just turns out; it’s not as simple as that. In order for WMB to pickup the messages that come into MQ, the messages must be PUBLISHED rather than point-to-point PUT and while publishing certain TOPICS should be added. WMB in turn will do all its routing to the subscribers based on the topics that’s present in the message.
It’s always tricky if you don’t know the semantics of glossary terms used by different teams. After some research and with the help of the Utility tool RFHUTIL.exe provided by IBM to write messages into MQ, it turns out the difference between PUBLISH and PUT is just the addition of some extra MQ specific headers. The format of the header is called MQRFH2; one of the segments in the header will contain the PSC commands required for publishing. Relevant TOPICS as name value pair can be added as part of the PSC command.
All these things will make more sense if we see some pictures. I utilized RFHUTIL to place couple of messages into MQ. When you browse the messages in MQ using MQ explorer, it will look like this:
You can spot the difference between the messages, the PUBLSIHED message format is MQRFH2 and the message data now contains some additional RFH information in the beginning.
When you double click and open the message, the first one (regular PUT) will look like this
And the second one (PUBLISH) will look like the one displayed below with additional MQRFH2 properties containing PSC information.
Also, if you look at the data you can see the additional information at the beginning of the message as shown below:
At this stage at least you know, what you are working on, what’s your target goal is and now you need to concentrate on how you are going to manage it from BizTalk Server 2006.
Understanding BizTalk Adapters for MQ:
When it comes to BizTalk and MQ the clean way to integrate both of them is by using an adapter. BizTalk server 2006 comes with 2 different set of adapters to talk to MQ. The default MQSeries adapter which gets installed as part of the normal BizTalk installation and the second one MQSC adapter that comes as part of the “BizTalk Adapters for Host System” package. The major difference between the two is the first one can only be used on MQ hosted on windows environment, whereas the second one is used to talk to MQ hosted on windows/non-windows environment. The MQSC adapter is also called as client based adapter because its dependant on IBM MQ’s client API’s. There are two different flavours of MQSC client adapters, the one which supports transaction ? paid license required and the other without transaction ?free no license required (guarantees at-least once delivery, but cannot guarantee duplicate message delivery). Switching between the two types of client adapters is easy via BizTalk Send Port Configuration.
You can learn more about them here:
http://blogs.msdn.com/biztalkcpr/archive/2008/10/07/snap-shot-of-biztalk-2006-solutions-for-mqseries.aspx, and
http://msdn.microsoft.com/en-us/library/aa772116(BTS.10).aspx
A bit of research was required to settle for the right adapter, in our case it?s definitely a client based MQSC adapter since MQ was hosted on a non-windows environment.
Bonus Findings!!:
1. You can just install the WebSphere MQ client API’s by doing running the WebSphere MQ 7.0 custom setup and selecting Windows client tools as shown below:
2. Once you have settled for the MQSC client adapter, you need to figure out the supported MQ platforms. According to the official document for MQSC adapter (
http://download.microsoft.com/download/c/2/2/c22737c1-707e-42a3-ae45-5df40973a0a7/BizTalk%202006%20R2%20Datasheet.pdf) that comes as part of BizTalk Server 2006 (Host integration server 2006) the supported platforms are
5.3 w/ Fix Pack 10 or higher, and 6.0 w/ Fix Pack 1 or higher. But during the prototyping, I figured out the adapter happily talks to MQ 7.0 as long as the MQ 7.0 client is installed. BizTalk 2009 with Host Integration Server 2009 officially supports MQ 7.0.
3. If in case you are working on the MQSeries server based adapter, you will experience lot of issues with strange error codes. Majority of them will be related to transactions and MSDTC. The easiest way to fix it during development and focus on the business problem is to disable transaction on the send and receive port. The other proper option is to configure MSDTC ?Security Configuration correctly as described here
http://msdn.microsoft.com/en-us/library/aa544733.aspx
Setting up a development environment:
One of the biggest challenges in an integration project (like this!!) involving two different technology stacks is setting up the development environment. You will lose majority of the time sorting out the environment, rather than working on the real problem. In my case it was not any different and I settled for building a virtual machine with everything required. I installed the software in the following order.
1. Clean Windows XP SP3 Virtual machine (OS choice is yours)
2. Installed SQL Server 2005, and SP1 on top of it
3. Installed Visual Studio 2005 (including VC++ components), and SP1 on top of it
4. Installed BizTalk Server 2006 and configured it
5. Installed MQ 7.0 90 days trial version
6. Installed ?BizTalk Adapter for Host Systems?
7. Downloaded
http://www-01.ibm.com/support/docview.wss?uid=swg24000637 IH03.zip file containing RFHUTIL.
Make sure BizTalk is able to communicate to MQ:
Once the environment is setup as explained in previous step, the next thing to do is to make sure BizTalk can communicate to MQ without any trouble. In order to test it, create a BizTalk application with a FILE receive port and MQSC send port and hook them together with appropriate filter condition (hint:
BTS.ReceivePortName= ReceivePort1).
To setup MQSC send port, you need to configure 4 basic properties as shown in the below figure
Channel Name: This is the channel name of type server connection and its case sensitive, for testing purposes the easiest way to create a channel is just by enabling “Remote Administration” on the queue manager. Right-Click on the queue manager and select “Remote Administration”, on the pop-up window click the button “Create” next to “SYSTEM.ADMIN.SVRCONN channel” and click ok.
NOTE: I don’t think this is the safest or right way to create a server connection channel, but on a development environment it certainly works. You should contact your MQ administrator to setup a proper server connection channel.
Connection Name: It’s in the format “server name(port number)”. The default port when you create queue manager in MQ explorer is 1414.
You need to specify the queue manager name and queue name.
JMS MQRFH2 BizTalk Pipeline Component:
At this stage we have a working BizTalk application that’s able to send messages to MQ via the MQSC adapter. Now raises the million dollar question, how are we going to add the required format and headers to message as discussed in the beginning of the article to PUBLISH the message into MQ.
JMS MQRFH2 BizTalk Pipeline component, that comes as part of Microsoft ESB Guidance/Toolkit package comes to our rescue. It’s worth understanding the significance of this pipeline component before we proceed further.
As part of Microsoft ESB Guidance 1.0
http://msdn.microsoft.com/en-us/library/bb931189.aspx, which was released after BizTalk Server 2006 R2 (later became Microsoft ESB Toolkit 2.0
http://www.microsoft.com/downloads/details.aspx?FamilyID=BC86CF1E-EF29-4B19-95F7-388F64555090&displaylang=en after the release of BizTalk Server 2009), Microsoft released JMS MQRFH2 BizTalk encoder/decoder pipeline component to help customers utilizing Java Messaging Service (JMS) for their integration needs. At this point it’s worth understanding the basic structure of MQ message and how JMS integrates with MQ.
WebSphere MQ messages are composed of three components:
- The WebSphere MQ Message Descriptor (MQMD)
- One or more optional message headers (in our case MQRFH2 header)
- The message body.
The message descriptor contains standard message properties, applicable to all WebSphere MQ messages. The second component, optional message headers (example MQRFH2, CICS etc) are application specific. For example when MQ is integrated with JMS, certain information are mapped into the MQRFH2 headers as shown in the below diagram.
The JMS MQRFH2 encoder and decoder BizTalk pipeline component helps customers to extend their JMS integration with Microsoft BizTalk server as shown in the below figure.
Using the encoder component the correct JMS message with relevant MQRFH2 header information can be constructed and submitted into WebSphere MQ, on the other hand BizTalk can receive JMS messages from WebSphere MQ and decode the JMS specific properties present in the MQRFH2 headers and write it to the context properties of the message.
Installing JMS MQRFH2 component for BizTalk Server 2006:
JMS MQRFH2 component was bundled along with ESB Guidance/Toolkit packages which were released and targeted for Microsoft BizTalk Server 2006 R2 and 2009.
One of the constraints I forgot to mention in the beginning was, our environment is still BizTalk Server 2006 (not a big surprise in big enterprise where product migration doesn’t happen that often), which mean we are still on .NET Framework 2.0.
This constraint prevented me from installing the latest Microsoft ESB Toolkit 2.0; as soon as you start the installation it will halt straight away asking you to install .NET Framework 3.5. Also, the other important information, ESB Toolkit 2.0 is not an open source project anymore like its predecessor Microsoft ESB Guidance 1.0.
But still there is some good news; ESB Guidance 1.0 is still available for download with complete source code at
http://msdn.microsoft.com/en-us/library/bb931189.aspx and the JMS MQRFH2 is a normal pipeline component without any dependency on either BizTalk 2006 R2 or 2009 specific functionalities. That means, you can take the code from the package, compile it and deploy it in your BizTalk 2006 environment.
Once you have installed the downloaded msi file, navigate to “C:Program FilesMicrosoft ESB Guidance 1.0 – November 2007” and unzip the file ESBSource. We are interested in only 3 BizTalk projects highlighted in the below picture.
ESB.JMS.Schemas.Property:
This project contains all the relevant context properties associated with MQRFH2 header. Open the project in Visual Studio 2005, open the project properties and change the application name to “JMS.ESB” (any name you wish). Compile and deploy the project.
ESB.JMS.PipelineComponents:
This is the core component, and compiling this project requires some attention. This project is developed as a C++ project due to its dependencies with IBM C header files to construct the MQRFH2 headers. If you have installed VC++ on your system, you shouldn’t have any problem compiling it. As you probably know, any BizTalk Pipeline component assembly should be present in the GAC, but in order to put the assembly in GAC it needs to be signed with a strong name key. There are some constraints in signing the assembly generated by this project, which is documented here
http://msdn.microsoft.com/en-us/library/ee250074(BTS.10).aspx
Once the project is compiled successfully, the easy option to add a strong name key to the compiled assembly is by issuing the following command
SN ?R “C:Program FilesMicrosoft BizTalk Server 2006Pipeline Components Microsoft.Practices.ESB.JMS.PipelineComponents.dll” “C:Program FilesMicrosoft ESB Guidance 1.0 – November 2007ESBSourceKeys Microsoft.Practices.ESB.snk”
The above command will re-sign the assembly with the new key file. You can then put the assembly into the GAC by using gacutils.
ESB.JMS.Pipelines:
This project contains all the support BizTalk pipelines utilizing the encode and decode pipeline components created by EMS.JMS.PipelineComponents assembly. This project is found under the samplesJMSSource folder, open the project in Visual Studio 2005, open the project properties and change the application name to “JMS.ESB” (or whatever name you have selected while deploying ESB.JMS.Schemas.Property project). Compile and deploy the project.
The JMS MQRFH2 component logic:
The component first inspects the BizTalk context property MQMD_Format and if its equal to “MQHRF2 ” (important:with 2 trailing spaces, and its NOT MQ
RFH2 but MQ
HRF2 due to header structure), then it applies the logic and adds the MQRFH2 headers else its just pass the incoming message as it is.
The component builds a default MQRFH2 Header using the IBM Header definition file “cmqc.h”. Then the Message Context Properties are checked and for each of the MQRFH2_Properties:”MQRFH2_StrucLength” “MQRFH2_Encoding” “MQRFH2_CodedCharSetId? “MQRFH2_Flags” “MQRFH2_NameValueCCSID” “MQRFH2_Format” If the property is set, then the value is parsed to determine if it is the appropriate type, i.e. Integer or 6 Character String. If the value is a correct type, then the value is written to the MQRFH2 Header.
Once each of the fixed values is set, then the length of the “MQRFH2_NameValueData” variable is determined and written to the “MQRFH2.NameValueLength” value in the MQRFH2 structure. Finally the content of the “MQRFH2_NameValueData” is prepended to the outbound MQRFH2 structure.
This final structure is then pre-pended to the outbound message body.
The component also takes care of encoding rules during this process.
Steps to PUBLISH a message into WebSphere MQ?
As discussed in the beginning of the article, publishing a message into WebSphere MQ is nothing more than adding some additional MQRFH2 headers into the message. With the help of JMS MQRFH2 pipeline component the task is seamless. Here are the steps to successfully PUBLISH the message into WebSphere MQ
1. In your BizTalk Application add a reference to the BizTalk application “JMS.ESB” (refer to “Installing JMS MQRFH2 component for BizTalk Server 2006” topic), so that you get access to deployed schemas and pipelines.
2. Create a static one-way send port configured with MQSC adapter similar to the one discussed under the topic “Make sure BizTalk is able to communicate with MQ”.
3. On the send port change the pipeline to JMS.PassThroughSend
4. Inside your Orchestration set the following context properties on the outgoing message. Build and deploy the orchestration
MSG_OUT(MQSeries.MQMD_Format) = “MQHRF2 “;
MSG_OUT(Microsoft.Practices.ESB.JMS.Schemas.Property.MQRFH2_NameValueData) = “<psc><Command>Publish</Command><Topic>Test</Topic></psc>“;
NameValueData is where you specify the psc command with application specific topic information in a XML format.
NOTE: I assumed the readers are familiar with binding and running the BizTalk Application.
Run the application with a sample message and inspect the corresponding WebSphere queue using MQ explorer.
The queue will resemble to the one shown in below figure with all the relevant MQRFH2 properties.
Issues Encountered:
There were couple of issues which returned the error reason code 2142 and 2085. 2142 was mainly due to MQRC_HEADER_ERROR, By hard coding the CodedCharSetId value to 437 the issue was resolved. 2085 was mainly due to UNKNOWN OBJECT, which was resolved by hard coding the StructLength to 96 (based on the psc command length). Still some minor work required to remove these hard coded values, but it should not be a big show stopper.
Conclusion:
The article summarized how custom MQRFH2 headers can be added to the WebSphere MQ messages utilizing the JMS MQRFH2 pipeline component that comes as part of ESB Guidance 1.0 and ESB Toolkit 2.0. The article also explained how the component can be utilized in a BizTalk server 2006 environment by obtaining the source code. Even though the original component was written to transmit and receive JMS specific header in the MQ message, in this article we have seen how it can be utilized to PUBLISH messages into MQ.
The end result is, it would have been impossible to achieve it without the JMS MQRFH2 component that comes as part of ESB Guidance/Toolkit. This article is bit long, but I hope reading a document for couple of hours is better than spending 3 days finding a solution 🙂
Nandri!
Saravana
