Friday, November 29, 2013

BizTalk Server:Using Global Variable in Mapping


Introduction

Mapping is one of the most important artifacts in BizTalk Server. It allows us to perform, in a simple and visual manner, transformations between XML messages.
When you create a map ,as a BizTalk developer, you should make sure that you build it with a good and optimal solution.
Mapping sometimes could be one of the bottlenecks of your BizTalk solution.
In this article, we will explain the benefits of using global variable in one of the real-life scenarios .

Problem

As BizTalk developer, you found out a problem in the current solution that when you upload a file with 10,000 records, it takes more that 15 minutes to process this file.
After investigation, you observed that the bottleneck is in mapping artifact. the map is shown in figure 1.
Figure 1. Bad solution map

In this map, it is calling the external assembly to pick up a value from database.
In the first look, you may think that it is calling the external assembly for 1 time but it will call it for many times.
If you take a look to xsl file, you will find out that the calling external assembly is inside the for-each loop as shown in figure 2.
<?xml version="1.0" encoding="UTF-16"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:var="http://schemas.microsoft.com/BizTalk/2003/var" exclude-result-prefixes="msxsl var s0 ScriptNS0" version="1.0" xmlns:ns0="http://Biztalk.Mapping.GlobalVariable.Destination" xmlns:s0="http://Biztalk.Mapping.GlobalVariable.Source" xmlns:ScriptNS0="http://schemas.microsoft.com/BizTalk/2003/ScriptNS0">
  <xsl:output omit-xml-declaration="yes" method="xml" version="1.0" />
  <xsl:template match="/">
    <xsl:apply-templates select="/s0:Source" />
  </xsl:template>
  <xsl:template match="/s0:Source">
    <ns0:Destiation>
      <xsl:for-each select="SourceRepeatedRecords">
        <DestinationRepeatedRecords>
          <xsl:variable name="var:v1" select="ScriptNS0:GetLookupValue(string(../Key/text()))" />
          <Value>
            <xsl:value-of select="$var:v1" />
          </Value>
          <FieldA>
            <xsl:value-of select="FieldA/text()" />
          </FieldA>
          <FieldB>
            <xsl:value-of select="FieldB/text()" />
          </FieldB>
        </DestinationRepeatedRecords>
      </xsl:for-each>
    </ns0:Destiation>
  </xsl:template>
</xsl:stylesheet>
Figure 2. Xsl file of bad solution map
If we run debug view to check how many times it call external assembly for the source that contains 3 records.
We will find out it will call it 3 times as shown in figure 3.
Figure 3. Debug View Event logs output for bad solution

 Now it is clear where is the problem. If we have 10,000 records in source schema , then it will hit a database table for 10,000 times.


Solution

We need to call the external assembly to pick up value from lockup table for 1 time instead of 10,000 times.
There are many solutions. One of these solutions is using a global variable.We can use a global variable to save a value within the first iteration and read the value from the global variable within the remain iterations.
as shown in figure 4.
Figure 4. Good Solution Map
In figure 4 , The functoid number 1 is the value mapping of the external assembly when iteration index is equal 1
In functoid number 7, we save value in a global variable and return it to value as shown in figure 5
Figure 5. Setting DB value in global variable
In functoid number 8 , we get the value of return the value of global variable which is in functoid number 6 when iteration is not equal 1.
functoid number 6 code is shown in figure 6
 
Figure 6. Get the value from global variable
If we test our solution, we will find out the external assembly was called only for 1 time instead of 3 times as shown in figure 7 
Figure 7. Debug View Event logs output for bad solution
So, without using a global variable if we have 10,000 records in the source schema it will hit database table for 10,000 time.On the other hand, when we use the global variable,it will hit database table for 1 time.

Sample Code

You can find the source code belonging to this article at MSDN Code Gallery:BizTalk Server:Using Global Variable in Mapping

Conclusion

In this article we illustrated how to use the global variable in BizTalk mapper and explained how global variable can improve the performance of BizTalk solution.


Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.
Posted by at No comments:
Labels: , , , , , , , , , ,

BizTalk Server Dynamic Schema Resolver Real-life Scenario


Introduction


As a BizTalk consultant you had an assignment that a service provider asked you to enhance the existing BizTalk solution.
You checked the current implementation and you found that the existing BizTalk solution as shown in Figure 1

Figure 1- Current BizTalk Solution Architecture

They are receiving different flat files from different customers and each customer has a different flat file format. All customers send these flat files in one receive path with a specific unique file name format ####YYYYMMDDHHMMSSNNN.txt
The first 4 digits are alphanumeric of customer code.
Each customer publishes a message to BizTalk Engine and other send ports subscribe these messages.
In this article, I will focus in dynamic resolver engine which is specifically in receive and send pipelines

Bad Solution

The implantation was any new customer added to the system, we need to do the followings steps:

Adding New Source Steps

  1. Add a new flat file schema matching the structure of a new customer as shown in Figure 2
    Figure 2- Sample Source Schema



    Figure 1- Current BizTalk Solution Architecture
  2. Add a new receive pipeline , add flat file disassembler and set Document Schema property to Flat file schema the created in step 1 as show in Figure 3
    Figure 3- Receive Pipeline
  3. Add a new receive location and set Receive Pipeline to the new receive pipeline that created in step 2 as shown in Figure 4
    Figure 4- Set Receive Pipeline to receive location
  4. Click on configure button and set File mask as shown Figure 5
    Figure 5- Set File Mask

In case there is a new Destination then you need to do the following steps

Adding New Destination Steps

  1. Add a new flat file schema matching the structure of a new destination as shown in Figure 6

    Figure 6- Sample Destination Schema


  2. Add a new send pipeline , add flat file assembler and set Document Schema property to Flat file schema the created in step 1 as show in Figure 7

    Figure 7 - Send Pipeline

  3. Add a new map between source and destination like sample as shown in Figure 8

    Figure 8 - Sample Map between source and destination

  4. Add a new Send Port in BizTalk Administration as show in Figure 9

    Figure 9- Send Pipeline in Send port


  5. Click configure and set file name format as show in Figure 10

    Figure 10- Setting File Name Format

  6. Select Outbound Maps and select the correct map as shown in Figure 11
    Figure 11- Setting Outbound Map
  7. Select Filters and Set BTS.ReceivePortName == Rcv_Messages which is the name of receive port as show in Figure 12

Figure 12- Setting Filter in send port

Problem

Now imagine that you want to add 100 customers as source then you need to add 100 of Receive Pipelines artifacts and you need to repeat steps of 2-4 of New Source Steps section 100 times. Another issue imagine that you want to add a new 100 destination then you need to create 100 of Send Pipelines artifacts and you need to repeat steps 2-7 of New Destination Steps section 100 times. 

Good Solution

We need to implement a dynamic schema resolver which is a mechanism of dynamically associate message types to inbound instances based on an identifier contained in each instance message.
We need to build a custom pipeline that takes a message type from configuration database like Business Rule Engine or using your custom configuration database table and set Document Schema to dis-assembler using the unique key that distinguishes your source message, you can use context properties of the message or using any unique data of the content of the message.
In our case the unique key is the customer code which is the first 4 digits of the file name
For send custom pipeline we have the xml message inside BizTalk engine which contains the schema type, we just need to set document schema into assembler at runtime
Figure 13 shown the Good solution architecture



Figure 13- Good Solution Architecture using Dynamic Resolver Mechanism 

We need to do the following implementation steps


Creating BRE Policy


  1. Create new schema with 2 element one for Source Code and one for message type(Namespace#RootNode) as shown in Figure 14 

    Figure 14- BRE Schema
  2. Create a new Policy SchemaResolverPolicy and add rule for each source as shown in Figure 15

Figure 15- Sample BRE Policy


Create Custom Pipeline Component


We need to create custom pipeline for receiving and sending pipeline as the following steps

Create Custom Receive Pipeline


  1. Add a new class library project as shown in Figure 16

    Figure 16- Add new class library to contain pipeline components


  2. Add a signature to the assembly, to allow assembly to be put into the GAC. To do this, right-click the new project and choose Properties. Then go to the Signing tab, as shown in Figure 17

    Figure 17- Adding a signature to the class library
  3. To create classes that BizTalk recognizes as pipeline components, you need to implement some specific interfaces. These interfaces are specified in the Microsoft.BizTalk.Pipeline namespace. You therefore need to reference the assembly that contains this namespace and these interfaces. Right-click your project or on the References node in your project and click Add Reference. Go to the Browse pane and browse your way to <InstallationFolder>\Microsoft.BizTalk.Pipeline.dll and add this as a reference
  4. To delegate call to Probe method which is implemented to investigate message to decide if message is recognizable or not. If message is recognizable then it returns “True” otherwise “False”. We need to add <InstallationFolder>\Microsoft.BizTalk.Pipeline.Components.dll.
  5. To call BRE api we need to reference to <installation drive>:\Program Files\Common Files\ Microsoft.RuleEngine.dll
  6. Delete the Class1.cs file that was automatically created for you and instead add a new class to your project by right-clicking the project and choosing Add, New Item. Choose Class as the type and provide a name for your class as show in Figure 18
    Figure 18-  Adding a new class to your class library.

  7. Mark your class as a public and decorate it with some attributes Guid telling the pipeline designer how the component can be used in our receiving custom pipeline we need to use: CategoryTypes.CATID_PipelineComponent This category does not affect which stages a component can be placed in, but merely serves as an indication for the pipeline designer that this is a pipeline component.
    CategoryTypes.CATID_DisassemblingParser Components in this category can be placed in the Disassemble stage of a receive pipeline and we assigning document schema in disassemble stage. To use the attribute you need to use namespace Microsoft.BizTalk.Component.Interop 
    as show in Figure 19
    using Microsoft.BizTalk.Component.Interop;
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    namespace TechNetWiki.SchemaResolver.PipelineComponent
    {
        [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
        [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
        public class SchemaResolverFlatFileDasmComp
        {
        }
    }
    Figure 19- Adding Pipeline attributes


  8. Decorate your class with GUID attribute by copy a new guid from TOOLS Create GUID As shown in Figure 20 and add namespace System.Runtime.InteropServices.
    Figure 20- Create new GUID

  9. All custom pipeline components must implement the following two interfaces:
    • IComponentUI, which lets you specify an icon for the component and a method for validating any properties the user can set values in
    • IBaseComponent, which basically lets you specify three properties that the pipeline designer uses to get a name, version, and description from your component to show in the designer As shown in Figure 21

    using Microsoft.BizTalk.Component.Interop;
    using System;
    using System.Collections;
    using System.Collections.Generic;
    using System.Linq;
    using System.Runtime.InteropServices;
    using System.Text;
    using System.Threading.Tasks;
    namespace TechNetWiki.SchemaResolver.PipelineComponent
    {
        [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
        [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
        [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
        public class SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI
        {
            #region IBaseComponent Members
            public string Description
            {
                get
                {
                    return "Flat file disassembler which resolves schemas using message context";
                }
            }
            public string Name
            {
                get
                {
                    return "Receive Pipeline Schema Resolver";
                }
            }
            public string Version
            {
                get
                {
                    return "1.0";
                }
            }
            #endregion
            #region IComponentUI Members
            public IntPtr Icon
            {
                get
                {
                    return System.IntPtr.Zero;
                }
            }
            public IEnumerator Validate(object obj)
            {
                return null;
            }
            #endregion
        }
    }
    Figure 21- Implement  IBaseComponent, IComponentUI interfaces

  10. Implement interface IDisassemblerComponent which exposes two methods needed for a disassembler as shown in Figure 22
    using Microsoft.BizTalk.Component;
    using Microsoft.BizTalk.Component.Interop;
    using Microsoft.BizTalk.Message.Interop;
    using System;
    using System.Collections;
    using System.Collections.Generic;
    using System.Linq;
    using System.Runtime.InteropServices;
    using System.Text;
    using System.Threading.Tasks;
    namespace TechNetWiki.SchemaResolver.PipelineComponent
    {
        [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
        [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
        [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
        public class SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent
        {
            #region IBaseComponent Members
            public string Description
            {
                get
                {
                    return "Flat file disassembler which resolves schemas using message context";
                }
            }
            public string Name
            {
                get
                {
                    return "Receive Pipeline Schema Resolver";
                }
            }
            public string Version
            {
                get
                {
                    return "1.0";
                }
            }
            #endregion
            #region IComponentUI Members
            public IntPtr Icon
            {
                get
                {
                    return System.IntPtr.Zero;
                }
            }
            public IEnumerator Validate(object obj)
            {
                return null;
            }
            #endregion
            private FFDasmComp disassembler = new FFDasmComp();
            #region IDisassemblerComponent Members
            public IBaseMessage GetNext(IPipelineContext pContext)
            {
                // Delegate call to Flat File disassembler
                return disassembler.GetNext(pContext);
            }
            public void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
            {
                // Delegate call to Flat File disassembler
                disassembler.Disassemble(pContext, pInMsg);
            }
            #endregion     
        }
    }
    Figure 22- Implement IDisassemblerComponent interface

  11. Implement interface IProbeMessage which will implement Probe method which called, and the component is then executed only if the Probe method returns a value of True and this will do the functionality of schema resolver  , We read ReceiveFileName context property from message context to read the first 4 digits of the file name to retrieve schema type from BRE  as show in Figure 23
using Microsoft.BizTalk.Component;
using Microsoft.BizTalk.Component.Interop;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.RuleEngine;
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
    public class SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent
    {
        private FFDasmComp disassembler = new FFDasmComp();
        private static Dictionary<string, string> cachedSources = new Dictionary<string, string>();
        #region IBaseComponent Members
        public string Description
        {
            get
            {
                return "Flat file disassembler which resolves schemas using message context";
            }
        }
        public string Name
        {
            get
            {
                return "Receive Pipeline Schema Resolver";
            }
        }
        public string Version
        {
            get
            {
                return "1.0";
            }
        }
        #endregion
        #region IComponentUI Members
        public IntPtr Icon
        {
            get
            {
                return System.IntPtr.Zero;
            }
        }
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
        #region IDisassemblerComponent Members
        public IBaseMessage GetNext(IPipelineContext pContext)
        {
            // Delegate call to Flat File disassembler
            return disassembler.GetNext(pContext);
        }
        public void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Delegate call to Flat File disassembler
            disassembler.Disassemble(pContext, pInMsg);
        }
        #endregion 
     
        #region IProbeMessage Members
        public bool Probe(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Check arguments
            if (null == pContext)
                throw new ArgumentNullException("pContext");
            if (null == pInMsg)
                throw new ArgumentNullException("pInMsg");
            // Check whether input message doesn't have a body part or it is set to null, fail probe in those cases
            if (null == pInMsg.BodyPart || null == pInMsg.BodyPart.GetOriginalDataStream())
                return false;
            Stream sourceStream = pInMsg.BodyPart.GetOriginalDataStream();
            string messageType = string.Empty;
            //Read ReceivedFileName from the message context
            string srcFileName = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
            srcFileName = Path.GetFileName(srcFileName);
            //Substring the first four digits to take source code to use to call BRE API
            string customerCode = srcFileName.Substring(0, 4);
            //create an instance of the XML object
            XmlDocument xmlDoc = new XmlDocument();
            xmlDoc.LoadXml(string.Format(@"<ns0:Root xmlns:ns0='http://TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE'>
                          <SrcCode>{0}</SrcCode>
                          <MessageType></MessageType>
                        </ns0:Root>", customerCode));
            //retreive source code in case in our cache dictionary
            if (cachedSources.ContainsKey(customerCode))
            {
                messageType = cachedSources[customerCode];
            }
            else
            {
                TypedXmlDocument typedXmlDocument = new TypedXmlDocument("TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE", xmlDoc);
                Microsoft.RuleEngine.Policy policy = new Microsoft.RuleEngine.Policy("SchemaResolverPolicy");
                policy.Execute(typedXmlDocument);
                XmlNode messageTypeNode = typedXmlDocument.Document.SelectSingleNode("//MessageType");
                messageType = messageTypeNode.InnerText;
                policy.Dispose();
                // Fail if message type is unknown
                if (string.IsNullOrEmpty(messageType))
                    return false;
                cachedSources.Add(customerCode, messageType);
            }
            // Get document spec from the message type
            IDocumentSpec documentSpec = pContext.GetDocumentSpecByType(messageType);
            // Write document spec type name to the message context so Flat File disassembler could access this property and
            // do message processing for a schema which has document spec type name we've discovered
            pInMsg.Context.Write("DocumentSpecName", "http://schemas.microsoft.com/BizTalk/2003/xmlnorm-properties", documentSpec.DocSpecStrongName);
            // Delegate call to Flat File disassembler
            return disassembler.Probe(pContext, pInMsg);
        }
        #endregion
    }
}
Figure 23- Implement IProbeMessage  interface

Create Custom Send Pipeline

  1. Add a new Item class to your class library project and rename it
  2. Do same steps of creating custom receive pipeline class but for sending custom pipeline we need to use CategoryTypes.CATID_AssemblingSerializer attribute and implement IAssemblerComponent inteface as show in Figure 24
using Microsoft.BizTalk.Component.Interop;
using System;
using System.Runtime.InteropServices;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.BizTalk.Component;
using System.Collections;
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]   
    [ComponentCategory(CategoryTypes.CATID_AssemblingSerializer)]
    [Guid("ACF451CE-E60D-42E4-8AB0-ED4CADD8B23E")]
    public class SchemaResolverFlatFileAsmComp : IBaseComponent, IComponentUI, IAssemblerComponent
    {
        System.Collections.Queue qOutputMsgs = new System.Collections.Queue();
        private FFAsmComp assembler = new FFAsmComp();
        #region IBaseComponent members      
        public string Description
        {
            get
            {
                return "Flat file assembler which resolves schemas using message context";
            }
        }
              
        public string Name
        {
            get
            {
                return "Send Pipeline Schema Resolver";
            }
        }
            
        public string Version
        {
            get
            {
                return "1.0";
            }
        }      
        #endregion
        #region IComponentUI Members
        public IntPtr Icon
        {
            get
            {
                return System.IntPtr.Zero;
            }
        }
        
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
        #region IAssemblerComponent Members
        public void AddDocument(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
        {
            qOutputMsgs.Enqueue(pInMsg);
        }
        public IBaseMessage Assemble(IPipelineContext pContext)
        {
            IBaseMessage msg = (IBaseMessage)qOutputMsgs.Dequeue();          
            assembler.AddDocument(pContext, msg);           
            return assembler.Assemble(pContext);
        }
        #endregion
    }
}
Figure 24-Custom Send Pipeline component code


Create Receive and Send Pipeline artifacts

  1. Build you Class library project and copy the dll file and paste it to this location <InstallationFolder>\Pipeline Components
  2. Add new receive pipeline to your BizTalk project
  3. Right Click on Toolbox then select choose items then choose as show in Figure 25

    Figure 25- Selecting Custom Pipeline components


  4. Drag Receive Pipeline Schema Resolver component from Toolbox and drop it to Disassemble part as shown in Figure 26

    Figure 26- Drag Drop Receive Pipeline Schema Resolver component

  5. Add new send pipeline to you BizTalk project
  6. Drag Send Pipeline Schema Resolver component from Toolbox and drop it to assemble part as shown in Figure 27

    Figure 27- Drag Drop Send Pipeline Schema Resolver component

  7. Deploy You Pipeline BizTalk project then open BizTalk Administration and create new Receive location in your BizTalk application, configure the location and select the receive pipeline of schema resolver as shown in Figure 28

    Figure 28-Select receive schema resolver pipeline

  8. Create new send port, select file adapter ,configure the location and select send pipeline as shown in Figure 29

    Figure 29-Select send schema resolver pipeline

  9. Select all required Outbound maps to map files from source to destination as show in Figure 30
    Figure 30-Select Outbound maps


  10. Filter your send port to subscribe from receive port as show in Figure 31
Figure 31-Filter send port


We can realized now that if we want to add any new source or any new destination, you just need to add schemas, maps and configure BRE or your configuration table.
This means that you reduced the number of artifacts in your solution pipelines and receive and send ports.
In the bad solution if you have 100 sources and 100 destinations then you will have 100 receive locations, 100 send ports and 100 pipelines.
However, in the good solution you will have 1 receive location, 1 send port and 2 pipelines and your configuration table.

One more thing, that we can do more enhancement to our solution by passing parameter generically to BRE instead of depending on the first 4 charters of file name,
we can use any property of the message context but for demonstration purposes I just use the file name.
More over, we can use canonical schema and mapping (inbound and outbound mapping) dynamically which I will write another article about this subject later on.

Sample Source Code

You can find the sample source code in the following link sample source code

Conclusion

This article demonstrated
  1. The real scenario of receiving and send Flat File from different sources with different formats and sending file to different destinations with different formats
  2. How to implement this scenario using the static biztalk solution
  3. How to implement the schema resolver mechanism using custom receiving and sending pipeline which made our solution more dynamic and easy to maintain
  4. How to build custom receive and send pipeline in details
  5. Benefits of using dynamic schema resolver mechanism
  6. Call BRE API from C# code

See Also

Read suggested related topics:

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.
Posted by at 1 comment:
Labels: , , , , , , , , , , , ,
Subscribe to: Posts (Atom)