Introduction to BizTalk Server for those Familiar with SeeBeyond

Microsoft BizTalk Server 2004

Related Links • SeeBeyond WebSphere MQ to BizTalk Conversion

On This Page

	Executive Summary
	The Problem
	Solution Aspects
	Business Process Integration Styles
	User Interface
	Windows Server System
	Scalability and Reliability
	Total Cost of Ownership

Executive Summary

In order to properly understand an application like BizTalk Server, it is important to not only know the feature list but to understand how these features address typical problems in your industry. Regardless of your role within your organization, you will see the value that BizTalk Server can provide and be able to determine how BizTalk Server can benefit your organization.

To begin, we must start by describing BizTalk Server. BizTalk Server is the principal server application from Microsoft. It is geared to helping organizations reach their service-oriented goals. BizTalk Server can integrate with many applications and processes running on diverse platforms and automate interactions between these systems and people. In addition, BizTalk also provides the management, reporting, administration, and flexibility needed to be successful in such a major endeavor.

In order to describe the true value of BizTalk Server, we will walk through the various requirements or aspects of a business process and integration solution.

Top of page

The Problem

The necessity of discussing the integration of processes or applications is due to the complex nature of today’s businesses—large, medium, or small. It is increasingly rare to see a successful business operating with one application that offers all the functionality required. The exception to this rule, of course, is the large monolithic mainframe application or possibly one of the very large enterprise resource planning (ERP) solutions available today. If this is the case, then the integration between programs or modules could pose a very similar integration need to that of various applications. In fact, since these solutions are so large and the complexity of implementation is so high, companies often run various versions of the software, requiring custom integration even though they are all sourced from one provider.

More commonly we see various applications serving various functional needs of a dynamic organization. These solutions are often acquired as specialized solutions, selected as best-of-breed or acquired for divisional needs and have expanded to organization-wide usage. Most recently, we also see organizations that have grown significantly through acquisition and have the resulting mishmash of applications and platforms with which to cope. All these situations result in the need to get these diverse providers of functionality working together in some form.

Top of page

Solution Aspects

Every integration solution is unique. However, many aspects of the solution are common. For instance, all integrations begin by crossing boundaries and exchanging data with an application. That simple statement actually involves significant functionality that we discuss in the Connectivity section. Once data is involved we must ensure it is not only valid but formatted and translated correctly for the destination. Of course we will also want to be able to vary the processing based on the contents of the data. Next is the process to be implemented or the choreography that will lead us to a discussion of transactions across the before-mentioned platforms. Finally, once all these connectivity, choreography, and translations are running reliably, we can really extract the greatest value from the solution. We are going to monitor our business processes and obtain both real-time and analytical data that was never available before.

The Dimensions of Integration

The dimensions of integration help us describe a full integration solution. The dimensions are expressed in the following forms:

Over the course of this paper, I will refer to some familiar constructs and explain BizTalk Server by showing how it accomplishes these styles of integration. We will also see how Enterprise Service Bus (ESB) is really a subset of this functionality.

BizTalk Server Feature Overview

Figure 1: BizTalk Feature Set

The diagram in figure 1 highlights the feature areas of BizTalk Server. It is very important to note that everything shown above is available with any purchase of BizTalk Server. BizTalk is sold in three editions, each aimed at various scales of deployment, but all editions enable the identical feature set. Differences between the editions come in the form of the number of applications and trading partners and the ability to deploy a high-availability architecture with full redundancy. In an attempt to describe the BizTalk Server feature set, we will refer to the academic dimensions of integration. The dimensions of integration come in four groups as follows:

Style

The style dimension addresses the bases of the integration. Is this integration really data synchronization? Or is it higher on the application stack, representing "Straight Through Processing" or automation of a business process through to the full implementation of composite applications?

Domain

Domain addresses the boundaries to be crossed by the integration or the business process. The domain can be small—for example, within a business unit where platform technologies are likely to be common and network security may be provided by one identity provider. Intermediate domain size includes scenarios that cross divisional boundaries within an organization where platforms and applications often differ. Finally, large domains are interorganizational, where various platforms are encountered and security may be provided by multiple identity systems. Domains that cross trading partners often require the crossing of firewalls, the Internet, and platforms. Identity must be communicated in either a peer-to-peer model or by some trusted organization.

Mode

The mode of the integration addresses the timing of the transaction. Legacy systems often process data in batches, resulting in bulk numbers of documents transmitted as a group. The Electronic Data Interchange (EDI) trading partner systems are a strong example of batch mode. Newer systems are striving to become more timely, and we see the emergence of asynchronous messaging. For example, when an order is placed online, that order is communicated almost immediately to the fulfillment system of the e-commerce provider. This transaction is not instant but quick. The difference being this—it’s a framework that ensures agility in the business logic and inherent reliability, independent of the reliability of either system in the transaction. On the top of the mode stack is pure real time, a mode that requires a very tight integration between applications often at the expense of flexibility.

Endpoints

Endpoints address the connections, whether they are to people, systems, devices, or databases. It is at the endpoint that the communication occurs, and therefore integration systems must support significant flexibility here. Connections must abstract away platform, framework, and API and provide a common mechanism for connecting to any endpoint. Beyond the physical connection, several additional properties must be addressed including data, security, transactions, and so on.

In this section we will describe BizTalk Server and how it meets and exceeds the requirements outlined by the dimensions of integration. We will begin by referring to Figure 1.

Top of page

Business Process Integration Styles

Integration

Connectivity begins at the endpoints at the bottom of the diagram. BizTalk, through an adapter framework, can connect to any system, database, application, device, protocol, or trading partner. This incredible flexibility is provided by two mechanisms, the Single Sign-On service and the adapter framework.

Single Sign-On

The Single Sign-On (SSO) service is a result of the realization that virtually no company runs on a heterogeneous platform. Single Sign-On allows for the mapping of credentials on various applications or systems running on various platforms. For example, being integrated into the core messaging system of BizTalk Server allows a user running a UNIX system to be authenticated when a request is placed and passed through to an AS/400 system running DB2 and have data queried under the user’s credentials. This is quite powerful and allows all user management and restriction capabilities of end systems to be maintained.

Adapter Framework

The adapter framework provides the connectivity between BizTalk Server and the endpoint. The framework is actually a rich set of Application Program Interfaces (APIs) allowing for the creation of adapters. Adapters must only be created in special circumstances because hundreds of specialty adapters are available from Microsoft and the Microsoft partner community. These adapters are certified for quality and reliability. Furthermore, specialty partners are enabled to provide rich adapters with capabilities specific to their vertical market or their application. Additionally, generic adapters with rich added value are provided by partners who specialize in the creation of adapters. Adapters are available to connect to systems, for example ERP systems such as JD Edwards or SAP; protocols, such as HTTP, SOAP, and so forth; databases, such as Oracle, SQL Server, DB2, and so forth. The adapter framework is directly tied to the SSO or Single Sign-On capabilities and the pipelines where processing on the messages takes place.

Pipelines

Pipelines in BizTalk Server provide a serial processing capability where various components may be executed on a message in a serial and transactional way. Custom Pipeline components can be created, however, just like adapters. Microsoft and the partner community provide many specialized components. Pipeline components provide capabilities such as:

Provided out of the box are pipeline components that implement MIME/SMINE encoding and decoding, a parser for flat files as well as XML files, party resolution based on certificates or HTTP authentication, as well as support for the BizTalk Framework that enables reliable messaging independent of transport.

Pipelines also enable the execution of Map. The BizTalk Map is a component that allows document formats to be transformed. For example, a document produced by an ERP system in a format native to that ERP can be transformed into a document that can be consumed by a trading partner. Take, for example, patient and claim records produced by a patient management system transformed into a format consumable by an insurance company.

Pipelines exist between adapters and the message box and all processing of the adapter and the pipeline is performed as one service. This service is transactional.

Message Box

The BizTalk Message box is the core component of the messaging engine. The message box is made of a data persistence layer and a subscription service. The data persistence provides the asynchronous reliable messaging capabilities of BizTalk Server and the subscription engine provides the inherent flexibility. Send ports and Orchestrations or business processes can subscribe to messages in a seemingly infinite way, from where it came from, to who sent it, to any property in the message header or piece of data contained within the message. Any single message may match multiple subscriptions enabling the communication of source endpoints with any number of destination endpoints.

Integration Recap

From SSO through the adapter, the pipeline to the message box and through the same mechanism we have an infrastructure that allows connecting to endpoints, and allows endpoints to subscribe to other endpoints. This is quite a powerful concept. Imagine having your AS/400 capable of subscribing to events generated by your CRM system running on a totally different platform. Taken one step further, imagine your AS/400 seamlessly communicating a single message to both a CRM system running on UNIX and a manufacturing system running on Windows. This can be accomplished simply through the simultaneous abstraction of the communication layer and the message format and structure.

We have also addressed the capabilities required for the style data synchronization. Since we have adapters for various databases, we can connect and subscribe between them. It should be highlighted at this point that as we can do data synchronization with BizTalk Server, other tools such as SQL Server integration services are available as part of the Microsoft Server System which we will see shortly. We should always use the right tool for data synchronization.

Straight Through Processing

The next style to discuss is straight through processing. This is the style where a message enters the system and undergoes some sort of processing resulting in various actions occurring. In the previous section, we described how messages flow through adapters, pipelines, and the message box. In this section we will discuss what is known as the orchestration engine. Orchestration is a term synonymous with choreography and also referred to as the automation of a business process.

In order to automate a business process five aspects must be considered. These are:

Model

The model is often defined by the analyst as a diagram representing the processing steps of the business process. To address this, BizTalk Server exposes an interface to business analysts through Microsoft Office Visio, a tool with which many analysts are very familiar. This model can be transferred directly to the developer who consumes it with Visual Studio, the tool used for all development on the Microsoft platform. Within Visual Studio, the developer can integrate the process with the messaging engine by simply dragging and dropping, define transactions, message transformations, and so forth, essentially all that is necessary for the implementation of the business process.

Figure 2: Model Defined Using Visio

Figure 3: Model in the Developer Environment

Orchestration Runtime Engine

Once an orchestration is developed, it is compiled resulting in a .NET assembly that can be deployed on any BizTalk Server. The runtime engine of BizTalk will then host the business process, providing integration with the message box subscription engine, a transaction framework supporting both ATOMIC and long-running transaction support as well as correlation and dehydration capabilities.

Business transactions in the real world are often long running. For better scalability, BizTalk automatically manages the resource usage of orchestrations that are inactive, returning the resources to the process when needed. Essentially, whenever a process is waiting for input, the host will dehydrate the orchestration, freeing up resources on the system for other processes. This means that the entire state of the orchestration is saved to the database. In order to rehydrate or reactivate this orchestration when data is available, the subscription engine will automatically correlate incoming messages and associate them with the correct instance of a dehydrated orchestration. When this occurs, the orchestration is re-activated, the required data is supplied and orchestration continues. This process is very much like the multi-processing capabilities of an operating system that swaps process states of running programs in and out of active processing.

Management

Management of the business process integration platform is provided through the Microsoft Management Console (MMC). MMC is a console user interface that is used by all Microsoft servers, from Windows to SQL Server to BizTalk. MMC provides a common management tool for administrators regardless of the application. This enables management of the entire Windows Server System from one location.

The management tools observe messages as they pass through every component of BizTalk Server. All events are captured including the live state of running business processes. It is even possible to place a break-point in a business process and step through real-time while observing all state information. Within the console, business process state can be observed. Any errors are highlighted, and once the underlying issues are resolved (often applications being down or endpoints changing), the processes can be resumed with one click. For security-related purposes, not all users are permitted to perform such administrative functions, and there is therefore a rich role/permission infrastructure integrated with Active Directory protection for the system.

Microsoft Operations Manager, or MOM as it is most commonly known, is Microsoft’s operation platform that observes the vital statistics of the entire Windows Server System, from the operating system, to database, to application. All Windows servers support MOM with event observation packs known as MOM Packs. BizTalk is no exception, with its own MOM pack that monitors hundreds of vital statistics and communicates events back to the MOM console where operators can configure actions to be taken in the event of errors, warnings, or various status changes.

History and Tracking

As the management system gathers a real-time view of the status and health of the system, it is often desirable to also observe what took place, recently or in the past. BizTalk Server has a rich, non-intrusive tracking mechanism that stores event information as it is generated. Asynchronously and when the databases are idle, this data is moved into a tracking database structured for quick and dynamic reporting.

Figure 4: Health and Activity Tracking

Business Activity Monitoring

Beyond the ability to observe messages that pass through the system, an advanced solution requires the ability to analyze the content of the messages as they pass through the system. With BizTalk Server, the Business Activity Monitoring (BAM) component allows activities to be defined by a business user using Microsoft Excel, or directly on the process model in a graphical way using Microsoft Office Visio. The activity definition is then passed to the developer, who attaches them to the orchestrated business process through a drag-and-drop interface and deploys the activity completely independently from the process implementation. The result is that the activity can be changed by the analyst, enabling new information to be gathered as required without the need to change any implementation details. The isolation between the functionality and the monitoring extends to the collection of data, which is performed totally asynchronously with very little impact to the throughput capabilities of the solution.

Business Rules Engine

One item often neglected by the requirements of a straight-through processing system is flexibility. The ability for IT to align with business needs is a great advancement over the transactional messaging systems we saw a decade ago. However implementing a system today and having it remain valuable as your dynamic company evolves requires advanced flexibility. BizTalk Server delivers this in the form of the Business Rules Engine.

Say, for example, you are implementing an expense submission process. Certainly there will be some rules that somehow restrict the expenses that are acceptable or determine approval routing. With the Business Rules Engine, the Developer can simply build the orchestration by inserting a box that refers to the business rule required. In a separate activity, the business user who owns the expense review process can define the rule, test it in an automated way with multiple scenarios, and deploy it independently of the developer’s implementation.

The Business Rules Engine is a component often consumed within an orchestration that allows for the simple composition of rules based on messages, data or program components. With the combination of these sources virtually any rule can be composed and inserted into a business process by simply dragging and setting input/output properties. Interestingly, this is just half the solution. The Business Rules Engine also exposes an interfaced designed to allow the business users who determine the business rules to manage the business rules without developer or administrative intervention. Supported by an infrastructure that is rich in security enhancements, versioning, testing, and reporting, the Business Rules Engine is truly a powerful component of BizTalk Server. It sets itself apart from all other solutions aimed at integrating business processes.

Figure 5: Business Rule Composed and Implemented in Orchestration

Straight Through Processing: Recap

This section has described the functionality required to fully automate business processes. BizTalk Server uniquely allows us to implement a system that addresses the needs of businesses today. The inherent flexibility of the platform combined with the Business Rules Engine component allows us to implement a system that closely aligns IT systems with the always-evolving needs of today’s dynamic businesses. In the next section we will describe the next level of integration, known as the composed application.

Composed Application

The composed application takes straight through processing to the next level. Rather than pass through a document or message, the composed application exposes its capabilities to be called as part of another application. Essentially it exposes a service. Within the service, the composed application will consume the capabilities of various applications, systems, databases, or even trading partners.

Often the ability to compose an application is described as the ability to consume and expose services. BizTalk natively supports Simple Object Access Protocol (SOAP). SOAP-based communication is expected to be the most interoperable form of communication in the future. BizTalk, however, recognizes that the legacy systems and their proprietary communication protocols will be around for years to come. For this reason, BizTalk also supports adapters to connect to these heterogeneous endpoints. Through the mechanisms discussed earlier, the consumption of capabilities, from a service or exposure of a process, is independent of the communication mechanism. In this way, BizTalk is able to fully support its native SOAP messaging between services while equally supporting legacy protocols.

Top of page

User Interface

Much has been made in recent articles regarding the importance of a single user interface aimed at "running" integration solutions. BizTalk Server, due to its family heritage, has the very significant advantage of being able to leverage the best-of-breed interfaces of the entire Windows Server System. BizTalk provides the right single interface to the user based on the user’s role. In order to do this, BizTalk Server has defined four distinct roles that comprise all the interactions required of a business process and integration solution. These roles are:

Within each role are users with distinct abilities, habits and experience with certain types and style of applications. In this section, we will identify the interfaces that are available to each of these roles.

Visual Studio for the Developer

Visual Studio is the development tool for the entire development process for all development on the Microsoft platform. Visual Studio provides the ability to generate code, graphical design tools, data design and definition, integrated source code control, testing capability, team and project collaboration, and so on. Visual Studio is an infinitely rich product which has grown in its capabilities since its introduction as Visual Interdev over a decade ago. Since 2000, Microsoft has consolidated the development tools with all the capabilities of all their solutions into Visual Studio. This effort has allowed the concentration of investment in the Visual Studio platform, as well as provided a common interface that is intuitive to all developers.

Microsoft Management Console

As Visual Studio is the tool for developers, Microsoft Management Console (MMC) is the tool for administration of the Microsoft platform. Using this single tool, the administrator can manage the operating system, the database and any application running within the framework, in most cases even when Microsoft was not the author of the application. This capability is the result of a richly documented Application Programming Interface (API) that allows all third parties to expose management capabilities within the MMC. The value is of course a single central interface for the administration of the entire solution. For BizTalk Server, a management console "snap in" provides a rich management experience that allows administrators to determine the status of all deployed applications at a single glance. Should intervention be required, administrators can inspect the status of a group of messages or a single instance. The administrator can then resume the instance.

Microsoft Office for the Analyst and the Knowledge Worker

Business analysts and knowledge workers commonly use tools such as Microsoft Excel, Microsoft Visio, and Microsoft Word with collaboration provided by Windows SharePoint Products and Technologies. It is logical that the modeling tool exposed to analysts runs in Visio; the reports generated for knowledge workers are exported directly to Excel. This not only leverages the advanced capabilities of the Microsoft Office platform but also provides advanced capabilities, using tools with which end users are familiar and are already using, thus minimizing training and maximizing productivity.

Top of page

Windows Server System

To date BizTalk Server has been discussed on its own. However, it is important to note that BizTalk Server is one application within the Windows Server System. The Windows Server System is a true superplatform comprising the full suite of capabilities required by dynamic and agile businesses. Uniquely, the Windows Server System shares three architectural tenants that ensure a reliable, safe, and productive experience when using any of the applications that are components of the system.

These tenants are:

These tenants ensure that all best practices relating to security and privacy are applied rigorously to all components of the Windows Server System. Furthermore, a roadmap clearly specifies the future of the server system, minimizing an unexpected feature change that limits the value of a component for an organization. The tenants also ensure a common framework for operations that ensure all systems can be managed as one entity, with information available and presented to operators that is applicable across the entire system.

Components of the server system include data persistence from SQL Server, content management from Content Management Server, integration capabilities from BizTalk Server and Host Integration Server, and collaboration from Exchange, Project, SharePoint Products and Technologies, and Live Communication Server. Since all these servers share the best practices of engineering and expose a common user interface, the Windows Server System stands alone as one truly complete and integrated superplatform.

Figure 6: Windows Server System

Top of page

Scalability and Reliability

Integration solutions are keys to the capabilities of the business and must be reliable and performed in such a way as to meet the needs of the business. BizTalk Server, by design, enables the abstraction of the physical computer implementing the solution. BizTalk works through a combination of hosts which are an abstraction of a physical computer. Multiple hosts may execute on a single computer, and a single host can operate across computers. As seen in Figure 7, a business process is assigned to a host. The host can be executed across multiple computers, allowing increased processing capabilities, because two computers are sharing the load; and increased reliability, because any single point of failure can be removed from the implementation.

Figure 7: Hosts Operating Across Computers

Top of page

Total Cost of Ownership

Total cost of ownership must factor in all costs associated with the implementation of a solution, from the costs of people, licenses, maintenance, hardware, tools, project costs, and so on. The implementation of a solution based on BizTalk Server has some very significant advantages above and beyond the favorable license costs.

People

People are critical to the success of any IT project, and even more so when dealing with the complexities of integrating complex business processes. BizTalk Server and Microsoft foster a group of professionals dedicated to the implementation of integration solutions using BizTalk Servers. These professionals typically work with partners, a model that Microsoft has fostered for years. The result is a very broad base of skills available throughout the world. These professionals have access to a plethora of documentation, samples, and best practices, allowing them to prepare for the required proficiency exams. Additionally, Microsoft provides regular webcasts, presentations, and whitepapers that enable the community of professionals to grow and provide reliable and cost-effective solutions for customers.

License and Maintenance Costs

Licensing for BizTalk Server is simple and documented at the following URL:

http://www.microsoft.com/biztalk/howtobuy

With full-featured licenses starting under $1000 (US), Microsoft has redefined the cost of integration solutions.

Hardware

Microsoft BizTalk Server and the rest of the Windows Server System run on x86-based computers as well as the new x64 computers. Depending on performance needs, BizTalk Server can run on any system from a small desktop or notebook all the way to a multi-CPU system scaled out across multiple servers. Performance is directly related to system resources. However, demonstrations of BizTalk capabilities are regularly executed on laptop-grade systems.

Top of page