BizTalk Server 12th Birthday

Happy Birthday BizTalk Server 12/12/2012 Born on 12/12/2000

A Many more returns of the day for BizTalk Server ... The first version of the product "Microsoft BizTalk Server 2000" was released back in 12/12/2000

BizTalk Summit 2012 is happening in Redmond, Seattle on 10th and 11th of December 2012 .The main objective of the event is to show the roadmap, future direction of BizTalk and to highlight what’s coming in the upcoming BizTalk Server 2013 version and improvement that’s happening on the integration side in Windows Azure.

Some good news about BizTalk Server. In Scott Guthrie, Vice President, Microsoft developer division key note, he has mentioned that the "Microsoft is heavily investing on BizTalk Server".
But I did not get, What is the roadmap for BizTalk server beyond 2013

More about the summit please read here . 

But where Microsoft is investing ???

BizTalk 2010: Why WCF Custom adapter is better than WCF-NetTcp, WCF-NetMSMQ ?

Why WCF custom (In process or isolated) adapter is better than the other WCF adapters. When I discussed with a BizTalk team working for leading apparel client, they have used the custom adapter extensively and they haven’t used any other WCF adapters. I got some answer like

“For example one of the requirements is to fetch a column from a table which is not a value but a XML message instead. In this case using WCF-custom developer can type poll to have better control on the table column that are fetched.”

 Yes the above statement is true. But it has more reason, after some exploration and I found the following list of usage

• Implement and exploit extensibility points.
• Have full access to properties exposed by bindings/behaviors.
• Enable the use of the bamInterceptor endpoint behavior.
• Export/Import the binding configuration.
• Disable a receive location on failure.
• Run an http-based Receivel Location within an in-process host.
• Use bindings (e.g. wsDualHttpBinding) for which a WCF Adapter does not exist.

Not sure, Why we need other WCF adapters like WCF-NetTcp, WCF-NetMSMQ etc. Since WCF-Custom Adapter (Inprocess & Isolated) can do the job?

Only one use, I can think of , we can quickly configure other WCF adapters than the custom adapters and we don’t require to worry about any other behavior of the service. please comment if you know more reasons

EDI - Schema Validation error while developing

Today I was trying few edi schema, On trying to validate in VS 2005 for the BizTalk Project, I found a error
../X12_BatchSchema.xsd: error BEC2004: Object reference not set to an instance of an object.
..\X12_BatchSchema.xsd: error BEC2004: Validate Schema failed for file: .
..\X12_BatchSchema.xsd: error BEC2004: Validate Instance failed for schema X12_BatchSchema.xsd, file: .
Component invocation succeeded.

I tried to debug this issue, I found after lots of experiment. I removed my property schema, which I used for the schema. The error dissapeared and Validation succeeded. I am not sure why the error occured, Looks like the validation component not able to recognise the proper schema if there is property schema in the project.

Tip
So during development do your property promotion after validating you modified edi schema

The Canonical Data Model

We have used canonical Data Model(http://www.eaipatterns.com/CanonicalDataModel.html) pattern for my client, Which minimise the dependencies from any integration application, which may use data format.

I love the pattern of creating canonical schema since, you will have a single entity through out the Enterprise
Example, An Order, You will be having a single XML schema defined, which talks every thing about order w.r.t to the company. Billing System in the organisation is interested in knowing the customer information, financial information from order to execute its business process. Inventory system is interested in the know the item information and number of items from order, to check for availability and so on. All these details can be get from a single Order canonical schema
Designing the Canonical schema Since BizTalk and other EAI tools work well with XML. The schema is an XML schema. it contains a two parts Header and Body.
The Header will have all the information about the business document (eg order). We at EMI, call this as enterprise header.

1. Subject - Represent the action of the document or the represent the business document (Example Create.Order, Update.Order or Highvalue.Order
2. Version - Document version keeps changing when you add/modify/delete element or attribute in the XML Schema
3. Source system - Which system provides data for the canonic0
4. Unique ID - To identify the Message
5. Batch - you can add the batch information, if you are handling messages in batching

Well, What more, Add as much as information that might be interested w.r.t to your organisation. for example unique document number base on the source system. The Enterprise header should be same through out all your canonical schema. There will be one Enterprise Header through out the organisation

The Body part will contain the common business document (Say Order). Some Companies have a seperate department/team, which control what they look like. if you are lucky to have this department already in place, then your canonical schema body part is ready. or else you should define itSome good ways to identify the what should go into canoncal schema. Identify the information for you schema from source systems(Note : you may have more than one system provide the same business document). and cosolidate the structure. Easier one is the Master data. Identify the Master Data Management system in your enterprise. you can quickly create lot of canonical schema (Ex customer, employee, product etc)Follow International Standards, you can refer EDI Standards to create your body of your canonic. (Give meaning full name in your canonics)

Always populate the header only in the Middleware and remove the header before sending to target system. Since most of the target system expects their own format

So whats the Advantage
01. Transformation is Simple.
02. Help Developers the Business entities easily
03. Adding new subscriber will take lesser time to commission
04. Meaning full names in the Schema helps to align IT with the Business

Developing Integration solution using BizTalk server 2006 and Team Foundation server(TFS)

Microsoft has released a document, which talks about how to develop Integration solution using BizTalk server 2006 and Team Foundation server (TFS). Find it here

This document gave me wide range of ideas how we can exploit the capability of TFS to improve the efficiency of the team to deliver better results. 

The Document discuss about the 

Requirement gathering for Integration Application

Naming Convenstions

Planning the team

Setting up the TFS and how to work with it

Manage versioning of assemblies

Managing different enviroments (Development, test and Production)

Debugging Methodologies with BizTalk Artifacts and Custom components

Discussed about automated build using TFS

Deployment Stratergies

Peformance Testing with BizUnit and Load Gen

I was creating Automated performance testing to implement for my client, with tools like loadgen and BizUnit.

Tools used for Testing

LoadGen (BizTalk Server 2004 Load Generation Tool) - I am using the earlier version of LoadGen 2007 as the assemblies supplied for 2007 are not strong named.

BizUnit - To automate the test.

Nunit - Framework to run the test that created in BizUnit.

Performance Monitor (perfmon) - to analyse the log created in the testing.

LoadGen is a load generating tool. which can used for test the performance of integration Application

There were three components in loadgen
Load generators are responsible for transmitting messages via a particular transportCurrently it supports the following transport (In-Box)
File,HTTP,MQSeries
MSMQLarge,MSMQ,SOAP,
Web Services Enhancements (WSE),
Windows SharePoint Services (WSS),
Windows Communication Foundation (WCF) (LoadGen 2007 Version has)

Now a Question comes up can we extend to other transports ? the answer is YES you can

Message creators are an optional component that can be used when you need to
generate messages that contain unique data.

Throttle controllers ensure that messages are transmitted at a steady rate by
governing the load generators

you can find more details here

Performance Monitor(PerfMon) will be used as an anlysis tool . which can read the performance log file and give us a nice view on performance.

BizUnit is an framework to automate the test used to automate the test.
- Initialise the test like restarting the service, clearing the
message box in BizTalk.
- call the loadgen to create the load for the system.
- Invoke the performance monitor and log the details.
- Wait for all the message got processed and stop the test.
- Cleanup the system like stoping the monitor, remove any messages from the transport etc.

for more details on BizUnit check out in http://www.codeplex.com/bizunit

NUnit is a tool to invoke the BizUnit tests.
For more on NUnit http://www.nunit.org/index.php

I hope I have given a berief introduction, Let try out small scenario

Its a Simple File To File Scenario, where BizTalk receive file from a File location and Send it to another File location


Size of the file is 10 KB
Number of Message to send is 1000. I dont have any expected performance, Just want to know how my bizTalk Server Perform


Step 1 Construction of LoadGen Config File
We need to define
LoadGenerator - Here it is File transport, need to tell what is the lot size and total message
Message Creator - I dont need to invoke the component,as I am gonna replicate the same message
Throttle controller - I have used file monitor such that the thresholdrange is 40-50 Files.
Sample Configuration

<LoadGenFramework>

<CommonSection>

<LoadGenVersion>2</LoadGenVersion>

<OptimizeLimitFileSize>204800</OptimizeLimitFileSize>

<NumThreadsPerSection>5</NumThreadsPerSection>

<SleepInterval>200</SleepInterval>

<LotSizePerInterval>5</LotSizePerInterval>

<RetryInterval>10000</RetryInterval>

<StopMode Mode="Files">

<NumFiles>1000</NumFiles>

<TotalTime>3600</TotalTime>

</StopMode>

<Transport Name="FILE">

<Assembly>C:\Program Files\LoadGen\Bins\FileTransport.dll/FileTransport.FileTransport</Assembly>

</Transport>

<ThrottleController Mode="Custom">

<Monitor Name="File">

<Assembly>C:\Program Files\LoadGen\Bins\FileMonitor.dll/DropLocationFileMonitor.DropLocationFileMonitor</Assembly>

<ThresholdRange>40-50</ThresholdRange>

<SleepInterval>1000</SleepInterval>

<Parameters>C:\Picloc</Parameters>

</Monitor>

<ThrottleCondition>File</ThrottleCondition>

</ThrottleController>

</CommonSection>

<Section Name="FileSection">

<SrcFilePath>C:\TestData\test001.txt</SrcFilePath>

<DstLocation>

<Parameters>

<DstFilePath>C:\Picloc</DstFilePath>

</Parameters>

</DstLocation>

</Section>

</LoadGenFramework>

Step 2: Incorporating in BizUnit

Requirement is you need to run the test automatically, generate a log and analyse the result Whenever required.

As you now Performance Monitor(Perfmon) will help you to achieve this.

01. Restart the required host instance and messagebox

02. create the Location where you want to log the file

03. use PerfmonCountersStep to start the performance monitor and log the result the location mentioned

04. use LoadGenExecuteStep to run LoadGen tests

05 Stop the test when all the message has been sent. use PerfMonCounterMonitorStep to monitor the handler which send the message till it hit the target number of message

Step 04. Run the Test using Nunit

Step 05. Analyse the Result

My BizTalk Server performance on this scenario 
In an average it receive  4.993 Message/Sec and Send 4.993 Message/Sec

You have lot of other counters are there for you to select and analyse... for more on this toplic click here

BizTalk Server 2009 - The Future release

Microsoft has updated the Roadmap on BizTalk Server last week. There is no BizTalk 2006 R3 instead it is BizTalk Server 2009. It is expected to release in first half yearly of 2009.

Please find the RoadMap
http://www.microsoft.com/biztalk/en/us/roadmap.aspx

Exploring Activity Relationship in BizTalk BAM

I was exploring and evaluating activity relationship in BizTalk BAM to use the feature for my Client . I have taken a simple scenario, where my BizTalk application send Purchase order to a supplier and receive PO Progress and Invoice (There may be n number of other message related to the PO from the supplier with no patterns). Requirement is to find the status of the each Purchase Order and related activity. It should not affect the existing Client Integration Services

First question in my mind can we tackle the whole scenario as a single activity like PO Process or a muliple activity. But I have the following constraint

• l There is no pattern in receiving the message from supplier
• l BizTalk is not handling the business process and it is used only for integration
• l Orchestrating in a single activity , will consume lot of resource
   I need to keep the Orchestration instance live till the whole business process is over, It will directly impact my Disaster Recovery(DR) story
   I Should re-think about my capacity management to handle .

So I decided to handle the situation as multiple activity and by tracking the sending and receiving message on each activity (Not handling the business process)

I have created a messaging solution for sending and receiving the PO and Other documents. And track the each message. (The Integration Architecture is more of messaging solution)

Step 1

For tracking the PO and other activity using BAM, I have created the 3 Activity as given below

in the Excel

Activity Name Fields

POACTIVITY ------------------------- POID, PODATE...

INVOICEACTIVITY ------------------------ INVOICEID, POID, INVDATE...

PROGRESSACTIVITY ------------------------ PROGRESSID, POID, PROGRESSDATE...

The POID represent the Purchase Order ID, Which will uniquely represent the purchase Order

Step 2

Create a View will all the three activity (say InvoicePOView) using the excel and deploy to the BizTalk

You will be able to see the view in the BAM Portal as given below

Note: Activity Relationship can be done only within the same view.

Step 3 Tracking Profile – Implementing the Relation ship

For each activity Message Payload has been used for tracking

· POID has been used as the unique key to create Activity relation PO, PO Progress and Invoice

POActivity Tracking

Invoice Activity Tracking

Right Click on InvoiceActivity, Click on New Relationship and Rename as POActivity

Progress Activity Tracking

Right Click on InvoiceActivity, Click on New Relationship and Rename as POActivity

Deploy the TPE in BizTalk and test the tracking you can view the result as given below

Views and Results

The Activity Status Page for PO Activity Show as above

The PO (POID: 1000001 have 3 Invoice activities and 2 Progress Activity which are related the PO Activity)

On click on one of the InvoiceActivity we can find the details of the invoice

I have limited to show the capability of creating relationship using TPE Tool. You can also create relationship using custom code using addrelatedactivty method (refer http://msdn.microsoft.com/en-us/library/microsoft....)

Issue in Enlisting Send port in BizTalk 2006

Today I have wasted couple of hour on a small issue. I was working on one of the integration Application for my client
As part of the build I exported the port binding(xml file) through BizTalk Administration Console.
For Deployment we are using Nant deployment Framework
I need editing the port binding file. Since I have to put specify information based on the environment (Dev, Test, Stage and production) .
I normally edit the port binding file in notepad. Since my port binding file was too big. I edited the file in VS 2005
After Creating my Kit, I tried to deploy in Dev Server. My deployment failed and was giving following error:"Could not enlist send port '[Send Port Name]' Exception from HRESULT: 0xC00CE557(Microsoft.BizTalk.ExplorerOM) "
01. my binding file successfully imported, 02. when it tries to enlist the port it started throughing error
I tried directly in the Administration console manually. I got the Same error.
Since Portbinding file impoted successfully, assumed that there was issue in the port configuration When I checked the filter page of the port settings there was no filter listed yet there is a filter defined in the bindings file.
I have checked the environmental setting files and portbinding files everything seems ok. Then I realise when i tried to edit VS has poped a message as given below

I have clicked yes
VS tried to make the file looks good. By introducing (CR LF) where ever required. But it has affected the file when it was saved
When I exported the binding file it looks like this
<Filter><Group><Statement Property="BTS.MessageType" Operator="0" Value="Value1" /><Statement Property="EAI.SourceSystem" Operator="0" Value="LCSCanPop" /><Statement Property="EAI.Subject" Operator="0" Value="update" /></Group></Filter>

In VS it looks Like below
<Filter><Group><Statement Property="BTS.MessageType" Operator="0" Value="Value1" /><Statement Property="EAI.SourceSystem" Operator="0" Value="LCSCanPop" /><Statement Property="EAI.Subject" Operator="0" Value="update" /></Group></Filter>
Can you find any difference. The diffence (CR LF) near and .This is a very simple thing. But it has eaten my time.

BizTalk BAM - Data Insertion Mechanism

I have explored the way the BAM works in BizTalk , I would like to share the thoughts. Its very interesting as well.
Service Involved
BTSNTSVC.EXE
BizTalk Server Engine
BAM Event Bus Service

Database Involved

BizTalk Management
BizTalk Message Box
BAM Primary import

Storing Data through BizTalk

Role of BizTalk engine
The interceptors get invoked by the BizTalk engine, during execution of Orchestrations or messaging solution
Based on the Tracking profile the interceptor will determine what data to be captured.
At every persistence point the interceptor also send the data to the BizTalk Message box in a binary form, which will be stored in tracking table.

Role of Event Bus Service
The data is retrieved from the Tracking data table (BizTalk message box)
The binary form (which is a serialized .Net Objects), which are rehydrated to objects, Inturn it is converted to sql statements and store the data in the respective activity table in the BAM Primary Import database


Custom Data Insertion

The custom application must use Microsoft.BizTalk.BAM.EventObservation assembly to access BAM Methods of inserting data into a BAM Activity

There are two primary class for inserting data

• DirectEventStream
• BufferedEventStream
  
  DirectEventStream
  The data will be directly written to the BAM Activity Table in the BAM Primary Import database.
  
  BufferedEventStream
  This class work same as same as BizTalk. The class writes the binary blobs into an indirect database (BizTalk Messagebox). Then BAM Event bus will take care of writing the data to the activity table in BAM Primary Import database.
  
  
  What Happen when BAM Primary Import database Goes Offline
  
  I have conducted a simple test on BAM to know what happens
  
  I have created a simple BAM implementation on the sample BizTalk solution where it tracks the messageid and process date.
  
  BAM Definition and Tracking profile has been deployed in the respective database using BM.Exe and bttdeploy.exe tools
  
  Expected Result
  Number of message should be same as the number of record in the BAM Activity Table.
  
  I brought down the BAM Primary Import database to offline.
  Start the BizTalk services and sent 5 message through the BizTalk
  Once the BizTalk process is completed. I made the BAM database online
  I am not able find the data of the last 5 message for the first few min.
  After few min amazingly I am able to view all the 5 message data in the activity table
  
  
  Findings
  01. When BAM Primary Import database is offline the BAM Event Bus Service throws error as
  
  Event Type: Error
  Event Source: BAM EventBus Service
  Event Category: None
  Event ID: 25
  Date: 26/03/2008
  Time: 12:47:18
  User: N/A
  Computer: VLONP034T
  Description:
  Either another TDDS is processing the same data or there is an orphaned session in SQL server holding TDDS lock.A connection was successfully established with the server, but then an error occurred during the login process. (provider: Shared Memory Provider, error: 0 - No process is on the other end of the pipe.) SQLServer: VLONP034T, Database: BAMPrimaryImport.
  
  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
  02. The BAM Event Bus waits for the BAM Primary Import database to be online to insert the data.
  
  
  Based on the above test and Understanding I tried to answer some question
  
  What is the story with writing information/events to BAM when the BAM database is not available (offline, not visible, SQL upgrades etc)?
  
  The BAM Event Bus service will wait for the BAM Database to come online. Then it will send the data from messagebox to the respective activity table
  
  Especially with event streams other than the orchestration one, does the write simply
  fail?
  Its basically based on what class we use to send the data to BAM
  DirectEventStream – if we try to insert the data using the class the. Insertion will simply fail incase of non availability
  
  BufferedEventStream – This class has the ability to write only to the intermediate database. Inserting the data to the BAM database is the job of BAM Event Bus.
  
  
  Even a buffered stream needs to be flushed at some point. Is this true for BAM use both within and without BizTalk?
  
  If we use BAM using BizTalk message box as the intermediate storage, then BAM Event Bus service will take care of the situation.
  If we used direct event stream the insertion will fail simply.

BizTalk Message - concepts

- All Biztalk Messages are immutable.
- Message context properties are either written or promoted.
- Promoted properties have schemas, which has be shared by other schema
- Properties that are written are called Distinguished field which is used in orchestration
to read and write messages.
- Distinguished field are lighter than property field
- Distinguished field can be only used in the orchestration. instead property field can be used in all the artifacts
- Element is the only Property that can be promoted.
- All property schema information are stored in bts_documentspec database. - Promoted properties have maximum length of 255 character.
- They are instance specific and Exchange specific. Instance specific are those that pertain to particular message instance and
explicitly promoted during development where as exchange or system specific are automatically promoted by Biztalk server. exchange specific are determined with out looking in to actual content of the message
Instance Specific properties if not
specified are explicitly available for tracking. So once has to turn on the Security
Information flag to true.
ex: customerid is a instance specificRecieveportid is an example of system specific

whats new in BizTalk 2006

Installation and configuration
- Microsoft provides a single file that contains the latest version of all redistributable components required by BizTalk Server during installation.
- Simplified Configuration experience for first-time users using “Basic Configuration” and flexible configuration for advanced users using “Custom configuration”
- Upgrading from BizTalk Server 2004 to BizTalk Server 2006 is an in-place upgrade process. All earlier version will work unchanged in the new version

Coding
- Developers Zoom in and Zoom out the orchestration, allowing to view only the required portion of orchestration as required
- Flat File schema wizard, an very interactive wizard, which boost the productivity of the developers
- New functoids has been added like table looping functoids, mass copy functoids
Call rule shape can be placed only inside in Atomic scope. Now it can be placed without that. at the background the atomic scope is called automatically when you place the call rule shape

Run time improvement
- In BizTalk 2006 recoverable Interchange is one of the new feature introduced in the pipeline, In the batch processing; only messages that fail validation are suspended, when there is a batch processing and the messages can be resumed once the error is corrected. Wherein in BizTalk 2004 the whole batch will get suspended
- In BizTalk Server 2004, message resume is only enabled for the send side but not for the received side. In BizTalk Server 2006, nearly all receive-side messages can now be resumed through the Health and Activity tracking tool.
- In BizTalk Server 2006, functionality is introduced to allow orchestration and send ports to subscribe to failed messages. Failed message routing can be used for notifying users of failed messages or building rich error handling and message repair capabilities. (Failed message Routing)
- In BizTalk Server 2004, message resume is only enabled for the send side but not for the received side. In BizTalk Server 2006, nearly all receive-side messages can now be resumed through the Health and Activity Tracking tool.
- The send port can do In-order message delivery
- Large messages Transformation, Instead of doing it as in-process, the mapping will be done by the new large message transformation engine, which buffers message data to the file system, keeping the memory consumption flat
- Pipelines can be called within Orchestration, (I haven’t tried out. If anyone has came across this let me know)


Administration
- Introducing the Concept of BizTalk application, this is a logical container for all the artifacts for a given solution
- Packaging entire applications into .msi files are easier, just right click on the application and export as .msi file
- More powerful Administration console then earlier version
An administrator can view at a glance the health of currently running BizTalk applications.

Adapters
- BizTalk Server now supports receiving email messages and their attachments using version three of the Post Office Protocol (POP3) from any POP3 compliant mail server using POP3 Adapters
- The Windows SharePoint Services Adapter to provide built-in support for accessing and publishing documents stored in Windows SharePoint Services document libraries
- The MQSeries adapter now its shipped with BizTalk Server 2006, whereas earlier version adapters is available as a web download
- Other adapters SAP, PeopleSoft, JD Edwards, ODBC Adapter for Oracle, Siebel, TIBCO, IBM DB2

BAM Improvements
BAM portal
BAM has been enhanced in 2006 to include an out-of-the-box BAM portal that allows an information worker to easily examine and configure BAM information. Using the BAM portal, an information worker can select a particular instance of some business process to monitor and then choose a specific BAM view into the process to get a different perspective on the key performance indicators being monitored-sized image

BAM alerts
Through integration with SQL Server Notification Services, BizTalk Server 2006 allows BAM information to be delivered as notifications through e-mail or another communication channel to the business user, enabling them to make real-time business decisions based on this information.

BAM Web service
BAM in BizTalk Server 2006 builds on the existing Web services available in BizTalk Server 2004 and adds new Web service interfaces to expose the query of aggregate and instance data, creation of alerts, and retrieval of BAM configurations. The enhanced Web services interface can be utilized by custom applications to expose BAM functionalities within their user interface.