Introduction

Imagine it is your birthday and you send an invitation letter to a friend. You kindly ask him to respond to your letter so you know if he will be there. When you receive this letter you know it came from your friend and you can start planning the party. Now imagine you getting married and instead of one friend you invite everyone you know plus everyone from your future spouse’s family and friends. When you receive hundreds of letters you will have to match up these responses to the letters you sent. This is known as correlation.

BizTalk Server 2004 Orchestrations

Orchestrations allow us to implement a company’s business processes. For example a company will have a business process to receive orders from customers, check the order against the current stock, wait for the customer to pay for the order and then ship the order to the customer. Because this company probably uses different applications for keeping track of the stock, ordering items from the suppliers and receiving payment from customers, these applications have to be tied together. Part of an orchestration to implement this is shown in figure 1:

Figure 1: A sample orchestration for order processing

In this orchestration we need to correlate the invoice message to the payment received message, and the stock request message to the stock status message.

BizTalk orchestration instances

A BizTalk Orchestration is a graphical description of a business process. When an actual order is received, BizTalk will start a new instance of an orchestration. Compare a design-time orchestration to a class, and a run-time orchestration instance to an instance of a class. In a normal business situation we will have plenty of these BizTalk orchestrations, some of which are waiting for a response from the stock application, while others are waiting for the customer’s payment of an order. This last example gets our particular interest as correlation comes into the picture: The application will receive a customer’s payment, and BizTalk has to match this payment to the orchestration instance that sent the request.

BizTalk Correlation

In previous versions of Microsoft BizTalk Server it was difficult to implement correlation, but with the latest version of BizTalk this has become a breeze. For a request that is sent in an orchestration instance we have to define in the orchestration which part of the message correlates the request. Compare this to sending a letter to someone and in that letter you give a unique reference number. The response to this letter should also specify this reference number, allowing us to easily match the request to the response. Another example is receiving an invoice; this invoice has an invoice ID, and when you pay the invoice you should supply this invoice ID so the company can easily match the payment to the invoice sent.
This is done in Microsoft BizTalk Server 2004 by defining a correlation type. A correlation type is simply a collection of one or more properties that uniquely identify a request and matching response.

Defining a Correlation Type

A correlation type is the definition of what a “message reference” looks like. In the sample provided with this article we have defined a simple message with an order ID and some other order details (see figure 2).

Figure 2: The sample’s message schema and property schema

When we have a message with fields to be used in an orchestration, we have to promote these fields. This is done with a special kind of schema, a property schema. For this sample we only need one field, the OrderID, to be promoted, so the property schema is very simple (see figure 2). Once we have a property schema for our messages we can define a correlation type. Correlation types are viewed and created in the Orchestration View (see figure 3).

Figure 3: The Orchestration View

Right-clicking the Correlation Types node and selecting New Correlation Type takes us to the Correlation Type Properties Dialog (see figure 4).

Figure 4: The Correlation Properties Dialog

This dialog presents us with a list of all available property schemas, many of which are predefined by BizTalk Server 2004; also our own property schemas will be in the list. We can now add each property to the correlation type. In the sample provided we have defined an U2UCorrelationType with only one field, the Correlation.OrderID field.
Now that we have a correlation type, we can use it in orchestrations by creating correlation sets.

Defining Correlation Sets

Typically, orchestrations send out several requests and receive several responses. Each of these responses might have to be correlated to some request, each with its own specific value. Therefore we create a correlation set per correlated request-response pair, what can be done in the Orchestration View, again by right-clicking on the Orchestration Set node and selecting “New Correlation Set”. Think of a correlation set as being a variable that you initialize with a request’s reference value, to match up with a response having the same reference value.

The Sample Orchestration

Let’s have a look at a simple example using the orchestration in figure 5.

Figure 5: The Sample Orchestration with Correlation

This orchestration receives an order and sends it as a request to some fictitious application represented by the SendOrderPort port shape. This send shape also initialises a correlation set with the message’s order id so that when this orchestration instance receives a response it will only accept a response with a matching order id from the request message, as defined by the correlation set. So for example when the send shape sends a message with OrderID = 1, it will only accept a response with OrderID equal to 1.
We configure this using the send shape’s properties (see figure 6).

Figure 6: The Send Shape Properties

The Send Correlated Message send-shape from figure 6 initializes the U2UCorrelationSet; this means that the correlation set will keep track of the order ID in the message sent through this shape.
Configuring the receive shape to follow a correlation set is easy, using the receive shape’s properties (see figure 7).

Figure 7: The Receive Shape Properties

The Receive Correlated Message receive-shape from figure 7 follows the U2UCorrelationSet. This means that it will only accept a message with matching OrderID as initialized by the send shape.

Using the sample

The sample that we present with this article (download it from our resource center www.u2u.info) is very simple yet allows you to see the effect of using correlation. The ReceiveOrderPort port shape uses a file receive location to listen for xml files in the OrderIn directory. If you download and install this sample, you can drop a file with an order in the OrderIn directory. A new orchestration instance will be started that sends the received order to the CorrOrderOut directory through the SendOrderPort. This correlation instance now waits for an order with the same order id on the ReceiveCorrIn port. Move the file from the CorrOrderOut directory to the CorrOrderIn directory where it is picked up by the ReceiveCorrIn port shape. The orchestration instance with matching order id will pick up this order and drop it in the OrderOut directory to complete the test.
To test the real effect of correlation we need to drop several order messages in the OrderIn directory, each with a different OrderID value. Use the U2UOrder1, U2UOrder2, etc… messages for this. Each message will trigger the creation of another orchestration instance. Since each instance is waiting for a message you will see several active instances using the Health & Activity Tracking application; using the Operations->Service Instances menu (see figure 8).

Figure 8: Orchestration instances waiting for a message

BTSSubscriptionViewer

Also have a look at the message box subscriptions that were added due to correlation with the BTSSubscriptionViewer application, located in the C:\Program Files\Microsoft BizTalk Server 2004\SDK\Utilities folder. This handy little application shows the active subscriptions in the message box. For each orchestration instance that is waiting for a message a subscription is added to this list. For example for our orchestration a subscription is added for the MessageType and OrderID (see figure 9):

Figure 9: The MessageBox subscriptions

When we drop a message from the CorrOrderOut directory in the CorrOrderIn directory, one of these orchestrations will have a matching correlation set (and subscription) and will complete the processing of the message. The other orchestration instances will still be waiting for their matching message. Move the other messages and you will see the other active orchestration instances complete their processing.

Figure 10: After processing one message

Dehydration and re-hydration

Business processes can take a long time to complete. A customer for example might wait several days or weeks before paying. This means that the orchestration will also be in a waiting state for a very long time, taking up resources on the server. BizTalk Server will save the orchestration instance until the response is received, freeing precious resources on the server. This process is known as dehydration. Dehydration means that one server can easily handle lots of orchestration instances, since idle orchestrations don’t take up resources. When a message is received that needs to be handled by an orchestration instance that has been dehydrated, BizTalk Server will reload the instance into memory to continue processing. This process is known as re-hydration.
BizTalk Server might even re-hydrate the orchestration instance on another server, and therefore effectively load-balancing orchestrations.