Monday, December 30, 2013

Creating Custom Adapter Provider in ESB Toolkit SFTP As an Example

Introduction

ESB Toolkit "is a collection of tools and libraries that extend BizTalk Server capabilities of supporting a loosely coupled and dynamic messaging architecture. It functions as middleware that provides tools for rapid mediation between services and their consumers. Enabling maximum flexibility at run time, the BizTalk ESB Toolkit simplifies loosely coupled composition of service endpoints and management of service interactions." [MSDN]

ESB Toolkit Core Engine contains many components, one of the main component is Adapter Provider as shown in figure 1

Figure 1. ESB Toolkit Core Engine Component [MSDN]

"Adapter Providers are used in the BizTalk ESB Toolkit to set properties on outbound adapters. They provide a mapping between BizTalk ESB Toolkit configuration properties and Send Adapter context properties." [Flanders]

SFTP adapter is one of the new adapters added to the out of the box adapters of BizTalk Server 2013.

In this article I am going to explain how to create custom adapter provider using SFTP adapter as an example.

I supposed that you installed and configured BizTalk 2013 , ESB Toolkit 2.2 and prepared SFTP testing environment

Problem

There are a number of Adapter Providers included with the BizTalk ESB Toolkit 2.2 such as:

WCF-BasicHttp
WCF-Custom
WCF-WSHttp
FTP
File
SMTP
WebSphere MQ

However, how can I extend Adapter Provider Framework and use SFTP adapters?

Solution

The Adapter Provider framework provides a very easy way to extend it to include more BizTalk send adapters that are capable of dynamic configuration. These adapters are defined through the implementation of the IAdapterProvider interface within a .NET Framework assembly.

To create a custom SFTP adapter provider

"Step 1: Create an assembly that derives from the BaseAdapterProvider base class and contains a SetEndPoint method that sets the endpoint context properties of the message." [MSDN]

Add a class library project to you solution and give it a name like TechNetWiki.ESB.CommonAdapter.SFTPAdapter
Add a new class item to your class library project and name it AdapterProvider
Add references to the following assemblies
- %BTSINSTALLPATH%\Microsoft.BizTalk.Adapter.Sftp.dll
- %BTSINSTALLPATH%\\Microsoft.XLANGs.BaseTypes.dll
- %BTSINSTALLPATH%\\Microsoft.BizTalk.Pipeline.dll
- C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit\Bin\Microsoft.Practices.ESB.Adapter.dll
- C:\Windows\assembly\GAC_MSIL\Microsoft.BizTalk.GlobalPropertySchemas\3.0.1.0__31bf3856ad364e35\Microsoft.BizTalk.GlobalPropertySchemas.dll
The created class must implement BaseAdapterProvider add override AdapterName property and SetEndpoint method as show in the below code

01.namespace TechNetWiki.ESB.CommonAdapter.SFTPAdapter

02.{

03.    using Microsoft.Practices.ESB.Adapter;

04.    using Microsoft.XLANGs.BaseTypes;

05.    using System;

06.    using System.Collections.Generic;

07.    using System.Linq;

08.

09.    public class AdapterProvider : BaseAdapterProvider

10.    {

11.        public override string AdapterName

12.        {

13.            get

14.            {

15.                return "SFTP";

16.            }

17.        }

18.

19.        public override void SetEndpoint(Dictionary<string,string> resolverDictionary, XLANGMessage message)

20.        {

21.            if (resolverDictionary == null)

22.                throw new ArgumentNullException("resolverDictionary");

23.            if (message == null)

24.                throw new ArgumentNullException("message");

25.

26.            base.SetEndpoint(resolverDictionary, message);

27.

28.            try

29.            {

30.                string transportLocation = resolverDictionary["Resolver.TransportLocation"];

31.                string outboundTransportCLSID = resolverDictionary["Resolver.OutboundTransportCLSID"];

32.                string endpointConfig = resolverDictionary["Resolver.EndpointConfig"];

33.                string transportType = resolverDictionary["Resolver.TransportType"];

34.

35.                message.SetPropertyValue(typeof(BTS.OutboundTransportLocation), transportLocation);

36.                message.SetPropertyValue(typeof(BTS.OutboundTransportType), transportType);

37.                message.SetPropertyValue(typeof(BTS.OutboundTransportCLSID), outboundTransportCLSID);

38.

39.                if (!string.IsNullOrEmpty(endpointConfig))

40.                {

41.                    // parse delimited endpointconfig and set SFTP specific adapter properties

42.                    // endPointConfig data with this format "Key1=Value1;Key2=Value2;...."

43.                    var config = endpointConfig.Split(';').Select(part => part.Split('='))

44.                           .ToDictionary(split => split[0], split => split[1]);

45.                    // Set the context for the SFTP adapter

46.                    message.SetPropertyValue(typeof(SFTP.UserName), config["UserName"]);

47.                    message.SetPropertyValue(typeof(SFTP.Password), config["Password"]);

48.                    message.SetPropertyValue(typeof(SFTP.AccessAnyServerHostKey), config["AccessAnyServerHostKey"]);

49.                }

50.

51.            }

52.            catch (System.Exception ex)

53.            {

54.                throw;

55.            }

56.        }

57.    }

58.}
Right Click on project and select Properties -> Select Signing Tab-> check Sign the assembly -> Choose a strong name key file <New...> and name it like key.snk -> Build the project

"Step 2: Register the adapter provider by adding it to Esb.config configuration files using an <adapterProvider> element with a name for the adapter as the name attribute, the fully qualified name of the class as the type attribute, the moniker as the moniker attribute (multiple values should be separated by a comma), and optionally the assembly of the actual adapter as the adapterAssembly attribute." [MSDN]

We need to know the public key token of the created assembly to use it in esb.config file. Open Visual Studio Command Prompt -> Type sn -T <assemblyPath>
Open file C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit\esb.config
Add <adapterProvider name="SFTP" type="TechNetWiki.ESB.CommonAdapter.SFTPAdapter.AdapterProvider, TechNetWiki.ESB.CommonAdapter.SFTPAdapter, Version=1.0.0.0, Culture=neutral, PublicKeyToken=PublicKeyInStep1" moniker="SFTP" /> before </adapterProviders> tag and replace PublicKeyInStep1 with the generated public key token in step 1

"Step 3: Register the new assembly in the global assembly cache." [MSDN]

Open Visual Studio Command Prompt -> Type gacutil -i <assemblyPath>

Step 4: To let SFTP adapter be visible in Static Resolver Transport Name property

Create a new xml file and rename it SFTPPropertyManifest.xml in this path C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Microsoft.Practices.Services.Itinerary.DslPackage
Add the following script to your manifest file

<?xml version="1.0" encoding="utf-8" ?>

<adapterPropertyManifest adapterName="SFTP">

    <aliases>

        <alias name="globalPropertySchemas" value="Microsoft.BizTalk.GlobalPropertySchemas, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />

    </aliases>

    <properties>

        <property name="UserName" type="SFTP.UserName" description="The user name for the connection." encrypted="true" assembly="globalPropertySchemas" />

        <property name="Password" type="SFTP.Password" description="The password for the connection." encrypted="true" assembly="globalPropertySchemas" />

        <property name="AccessAnyServerHostKey" type="SFTP.AccessAnyServerHostKey" description="Determines if any SSH public host key fingerprint from the Server should be accepted." assembly="globalPropertySchemas" />

        <property name="EventArgs" type="System.EventArgs" assembly="mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />

    </properties>

</adapterPropertyManifest>
Close visual studio

Test the solution

Use these steps to test the new custom sftp adapter provider

Open Visual Studio 2012
Create a new Class Library project and name it such as ESB.Itinerary.Library
Right click on ESB.Itinerary.Library project -> Add-> New BizTalk ESB Itinerary Designer -> name it StaticSftp.itinerary
Right click on surface of Itinerary Designer and set Model Exporter to Database Itinerary Exporter and Require Encryption Certificate to false As shown in figure 2

Figure 2: Itinerary Properties
From the Toolbox, drag an On-Ramp model element to the design surface. In the OnRamp1 Properties window, configure the following properties
- Click the Name property, and then type OnRamp.
- In the Extender drop-down list, click On-Ramp Extender.
- In the BizTalk Application drop-down list, click Microsoft.Practices.ESB.
- In the Receive Port drop-down list, click OnRamp.Itinerary.
as shown in figure 3

Figure 3: OnRamp Properties
From the Toolbox, drag an Off-Ramp model element to the design surface, and then place it to the right of the On-Ramp model element. In the OffRamp1 Properties window, configure the following properties:
- Click the Name property, and then type OffRamp.
- In the Extender drop-down list, click Off-Ramp ESB Extender.
- In the BizTalk Application drop-down list, click Microsoft.Practices.ESB.
- In the Send Port drop-down list, click DynamicResolutionOneWay.
as shown in figure 4

Figure 4: OffRamp Properties
From the Toolbox, drag an Itinerary Service model element to the design surface, and then place it to the right of the On-Ramp model element. In the ItineraryService1 Properties window, configure the following properties:
- Click the Name property, and then type StaticSftpSendPort.
- In the Itinerary Service Extender drop-down list, click Off-Ramp Extender.
- In the Off-Ramp drop-down list, select Send Handlers.
as shown in figure 5

Figure 5: StaticSftpSendPort Properties
Right-click the Resolver collection of the StaticSftpSendPort model element, and then click Add new Resolver. In the Resolver1 Properties window, configure the following properties:
- Click the Name property, and then type StaticSftpResolver.
- In the Resolver Implementation drop-down list, click Static Resolver Extension.
- In the Transport Name drop-down list, click SFTP.
- In the Transport Location set sftp://127.0.0.1:22//C/Users/SFTPUser/MessageTesting/OUT/Static_%MessageID%.xml
- In the Endpoint Configuration set AccessAnyServerHostKey=true&UserName=sftpuser&password=sftpuserpassword
as shown in figure 6
Note: You cannot update UserName and Password because we set encrypted attribute for both to true. To work around it we set the value directly.

Figure 6: StaticSftpResolver Properties
In the Toolbox, click Connector. Drag a connection from the OnRamp model element to the StaticSftpSendPort model element.Add another connection from StaticSftpSendPort to the OffRamp as show in figure 7

Figure 7: StaticSftp Itinerary diagram Properties
Right click on the surface of the Itinerary designer then click deploy
Open BizTalk Administration Console in Microsoft.Practices.ESB application-> Receive Locations->New->One-way Receive Location..->Select OnRamp.Itinerary receive port->Name it Rcv_File -> Configure ->Set Receive Folder -> Receive Pipeline ItinerarySelectReceivePassthrough as shown in figure 8

Figure 8: Creating Receive Location
Click on ... of ItinerarySelectReceivePassthrough ->Set ItineraryFactKey Resolver.Itinerary -> set ResolveConnectionString ITINERARY:\\name=StaticSftp; as shown in figure 9

Figure 9: Configure ItinerarySelectReceivePassthrough pipeline
Test the solution by adding file to the receive location path and check the sftp folder that configured in step 8

Sample Code

All of this sample can be found and downloaded in Microsoft Code Gallery:

BizTalk Server 2013: Creating Custom SFTP Adapter Provider in ESB Toolkit

Conclusion

In this article I illustrated how to create a custom adapter provider using the new out of the box SFTP adapter which was added to BizTalk 2013 as an example You can do the same steps for other custom adapters.

Introduction

Mapping is one of the most important artifacts in BizTalk Server. It allows us to perform, in a simple and visual manner, transformations between XML messages.
When you create a map ,as a BizTalk developer, you should make sure that you build it with a good and optimal solution.
Mapping sometimes could be one of the bottlenecks of your BizTalk solution.
In this article, we will explain the benefits of using global variable in one of the real-life scenarios .

Problem

As BizTalk developer, you found out a problem in the current solution that when you upload a file with 10,000 records, it takes more that 15 minutes to process this file.
After investigation, you observed that the bottleneck is in mapping artifact. the map is shown in figure 1.

Figure 1. Bad solution map

In this map, it is calling the external assembly to pick up a value from database.
In the first look, you may think that it is calling the external assembly for 1 time but it will call it for many times.
If you take a look to xsl file, you will find out that the calling external assembly is inside the for-each loop as shown in figure 2.

<?xml version="1.0" encoding="UTF-16"?>

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:var="http://schemas.microsoft.com/BizTalk/2003/var" exclude-result-prefixes="msxsl var s0 ScriptNS0" version="1.0" xmlns:ns0="http://Biztalk.Mapping.GlobalVariable.Destination" xmlns:s0="http://Biztalk.Mapping.GlobalVariable.Source" xmlns:ScriptNS0="http://schemas.microsoft.com/BizTalk/2003/ScriptNS0">

  <xsl:output omit-xml-declaration="yes" method="xml" version="1.0" />

  <xsl:template match="/">

    <xsl:apply-templates select="/s0:Source" />

  </xsl:template>

  <xsl:template match="/s0:Source">

    <ns0:Destiation>

      <xsl:for-each select="SourceRepeatedRecords">

        <DestinationRepeatedRecords>

          <xsl:variable name="var:v1" select="ScriptNS0:GetLookupValue(string(../Key/text()))" />

          <Value>

            <xsl:value-of select="$var:v1" />

          </Value>

          <FieldA>

            <xsl:value-of select="FieldA/text()" />

          </FieldA>

          <FieldB>

            <xsl:value-of select="FieldB/text()" />

          </FieldB>

        </DestinationRepeatedRecords>

      </xsl:for-each>

    </ns0:Destiation>

  </xsl:template>

</xsl:stylesheet>

Figure 2. Xsl file of bad solution map

If we run debug view to check how many times it call external assembly for the source that contains 3 records.
We will find out it will call it 3 times as shown in figure 3.

Figure 3. Debug View Event logs output for bad solution

Now it is clear where is the problem. If we have 10,000 records in source schema , then it will hit a database table for 10,000 times.

Solution

We need to call the external assembly to pick up value from lockup table for 1 time instead of 10,000 times.
There are many solutions. One of these solutions is using a global variable.We can use a global variable to save a value within the first iteration and read the value from the global variable within the remain iterations.
as shown in figure 4.

Figure 4. Good Solution Map

In figure 4 , The functoid number 1 is the value mapping of the external assembly when iteration index is equal 1
In functoid number 7, we save value in a global variable and return it to value as shown in figure 5

Figure 5. Setting DB value in global variable

In functoid number 8 , we get the value of return the value of global variable which is in functoid number 6 when iteration is not equal 1.
functoid number 6 code is shown in figure 6

Figure 6. Get the value from global variable

If we test our solution, we will find out the external assembly was called only for 1 time instead of 3 times as shown in figure 7

Figure 7. Debug View Event logs output for bad solution

So, without using a global variable if we have 10,000 records in the source schema it will hit database table for 10,000 time.On the other hand, when we use the global variable,it will hit database table for 1 time.

Sample Code

You can find the source code belonging to this article at MSDN Code Gallery:BizTalk Server:Using Global Variable in Mapping

Conclusion

In this article we illustrated how to use the global variable in BizTalk mapper and explained how global variable can improve the performance of BizTalk solution.

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.

BizTalk Server Dynamic Schema Resolver Real-life Scenario

Introduction

As a BizTalk consultant you had an assignment that a service provider asked you to enhance the existing BizTalk solution.
You checked the current implementation and you found that the existing BizTalk solution as shown in Figure 1

Figure 1- Current BizTalk Solution Architecture

They are receiving different flat files from different customers and each customer has a different flat file format. All customers send these flat files in one receive path with a specific unique file name format ####YYYYMMDDHHMMSSNNN.txt

The first 4 digits are alphanumeric of customer code.

Each customer publishes a message to BizTalk Engine and other send ports subscribe these messages.

In this article, I will focus in dynamic resolver engine which is specifically in receive and send pipelines

Bad Solution

The implantation was any new customer added to the system, we need to do the followings steps:

Adding New Source Steps

Add a new flat file schema matching the structure of a new customer as shown in Figure 2

Figure 2- Sample Source Schema

Figure 1- Current BizTalk Solution Architecture
Add a new receive pipeline , add flat file disassembler and set Document Schema property to Flat file schema the created in step 1 as show in Figure 3

Figure 3- Receive Pipeline
Add a new receive location and set Receive Pipeline to the new receive pipeline that created in step 2 as shown in Figure 4

Figure 4- Set Receive Pipeline to receive location
Click on configure button and set File mask as shown Figure 5

Figure 5- Set File Mask

In case there is a new Destination then you need to do the following steps

Adding New Destination Steps

Add a new flat file schema matching the structure of a new destination as shown in Figure 6

Figure 6- Sample Destination Schema
Add a new send pipeline , add flat file assembler and set Document Schema property to Flat file schema the created in step 1 as show in Figure 7

Figure 7 - Send Pipeline
Add a new map between source and destination like sample as shown in Figure 8

Figure 8 - Sample Map between source and destination
Add a new Send Port in BizTalk Administration as show in Figure 9

Figure 9- Send Pipeline in Send port
Click configure and set file name format as show in Figure 10

Figure 10- Setting File Name Format
Select Outbound Maps and select the correct map as shown in Figure 11

Figure 11- Setting Outbound Map
Select Filters and Set BTS.ReceivePortName == Rcv_Messages which is the name of receive port as show in Figure 12

Figure 12- Setting Filter in send port

Problem

Now imagine that you want to add 100 customers as source then you need to add 100 of Receive Pipelines artifacts and you need to repeat steps of 2-4 of New Source Steps section 100 times. Another issue imagine that you want to add a new 100 destination then you need to create 100 of Send Pipelines artifacts and you need to repeat steps 2-7 of New Destination Steps section 100 times.

Good Solution

We need to implement a dynamic schema resolver which is a mechanism of dynamically associate message types to inbound instances based on an identifier contained in each instance message.
We need to build a custom pipeline that takes a message type from configuration database like Business Rule Engine or using your custom configuration database table and set Document Schema to dis-assembler using the unique key that distinguishes your source message, you can use context properties of the message or using any unique data of the content of the message.
In our case the unique key is the customer code which is the first 4 digits of the file name
For send custom pipeline we have the xml message inside BizTalk engine which contains the schema type, we just need to set document schema into assembler at runtime
Figure 13 shown the Good solution architecture

Figure 13- Good Solution Architecture using Dynamic Resolver Mechanism

We need to do the following implementation steps

Creating BRE Policy

Create new schema with 2 element one for Source Code and one for message type(Namespace#RootNode) as shown in Figure 14

Figure 14- BRE Schema
Create a new Policy SchemaResolverPolicy and add rule for each source as shown in Figure 15

Figure 15- Sample BRE Policy

Create Custom Pipeline Component

We need to create custom pipeline for receiving and sending pipeline as the following steps

Create Custom Receive Pipeline

Add a new class library project as shown in Figure 16

Figure 16- Add new class library to contain pipeline components
Add a signature to the assembly, to allow assembly to be put into the GAC. To do this, right-click the new project and choose Properties. Then go to the Signing tab, as shown in Figure 17

Figure 17- Adding a signature to the class library
To create classes that BizTalk recognizes as pipeline components, you need to implement some specific interfaces. These interfaces are specified in the Microsoft.BizTalk.Pipeline namespace. You therefore need to reference the assembly that contains this namespace and these interfaces. Right-click your project or on the References node in your project and click Add Reference. Go to the Browse pane and browse your way to <InstallationFolder>\Microsoft.BizTalk.Pipeline.dll and add this as a reference
To delegate call to Probe method which is implemented to investigate message to decide if message is recognizable or not. If message is recognizable then it returns “True” otherwise “False”. We need to add <InstallationFolder>\Microsoft.BizTalk.Pipeline.Components.dll.
To call BRE api we need to reference to <installation drive>:\Program Files\Common Files\ Microsoft.RuleEngine.dll
Delete the Class1.cs file that was automatically created for you and instead add a new class to your project by right-clicking the project and choosing Add, New Item. Choose Class as the type and provide a name for your class as show in Figure 18

Figure 18- Adding a new class to your class library.
Mark your class as a public and decorate it with some attributes Guid telling the pipeline designer how the component can be used in our receiving custom pipeline we need to use: CategoryTypes.CATID_PipelineComponent This category does not affect which stages a component can be placed in, but merely serves as an indication for the pipeline designer that this is a pipeline component.
CategoryTypes.CATID_DisassemblingParser Components in this category can be placed in the Disassemble stage of a receive pipeline and we assigning document schema in disassemble stage. To use the attribute you need to use namespace Microsoft.BizTalk.Component.Interop
as show in Figure 19

using Microsoft.BizTalk.Component.Interop;

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    public class SchemaResolverFlatFileDasmComp

    {

    }

}

Figure 19- Adding Pipeline attributes
Decorate your class with GUID attribute by copy a new guid from TOOLS Create GUID As shown in Figure 20 and add namespace System.Runtime.InteropServices.

Figure 20- Create new GUID
All custom pipeline components must implement the following two interfaces:
- IComponentUI, which lets you specify an icon for the component and a method for validating any properties the user can set values in
- IBaseComponent, which basically lets you specify three properties that the pipeline designer uses to get a name, version, and description from your component to show in the designer As shown in Figure 21
using Microsoft.BizTalk.Component.Interop;

using System;

using System.Collections;

using System.Collections.Generic;

using System.Linq;

using System.Runtime.InteropServices;

using System.Text;

using System.Threading.Tasks;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]

    public class SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI

    {

        #region IBaseComponent Members

        public string Description

        {

            get

            {

                return "Flat file disassembler which resolves schemas using message context";

            }

        }

        public string Name

        {

            get

            {

                return "Receive Pipeline Schema Resolver";

            }

        }

        public string Version

        {

            get

            {

                return "1.0";

            }

        }

        #endregion

        #region IComponentUI Members

        public IntPtr Icon

        {

            get

            {

                return System.IntPtr.Zero;

            }

        }

        public IEnumerator Validate(object obj)

        {

            return null;

        }

        #endregion

    }

}

Figure 21- Implement  IBaseComponent, IComponentUI interfaces
Implement interface IDisassemblerComponent which exposes two methods needed for a disassembler as shown in Figure 22

using Microsoft.BizTalk.Component;

using Microsoft.BizTalk.Component.Interop;

using Microsoft.BizTalk.Message.Interop;

using System;

using System.Collections;

using System.Collections.Generic;

using System.Linq;

using System.Runtime.InteropServices;

using System.Text;

using System.Threading.Tasks;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]

    public class SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent

    {

        #region IBaseComponent Members

        public string Description

        {

            get

            {

                return "Flat file disassembler which resolves schemas using message context";

            }

        }

        public string Name

        {

            get

            {

                return "Receive Pipeline Schema Resolver";

            }

        }

        public string Version

        {

            get

            {

                return "1.0";

            }

        }

        #endregion

        #region IComponentUI Members

        public IntPtr Icon

        {

            get

            {

                return System.IntPtr.Zero;

            }

        }

        public IEnumerator Validate(object obj)

        {

            return null;

        }

        #endregion

        private FFDasmComp disassembler = new FFDasmComp();

        #region IDisassemblerComponent Members

        public IBaseMessage GetNext(IPipelineContext pContext)

        {

            // Delegate call to Flat File disassembler

            return disassembler.GetNext(pContext);

        }

        public void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)

        {

            // Delegate call to Flat File disassembler

            disassembler.Disassemble(pContext, pInMsg);

        }

        #endregion

    }

}

Figure 22- Implement IDisassemblerComponent interface
Implement interface IProbeMessage which will implement Probe method which called, and the component is then executed only if the Probe method returns a value of True and this will do the functionality of schema resolver , We read ReceiveFileName context property from message context to read the first 4 digits of the file name to retrieve schema type from BRE as show in Figure 23

using Microsoft.BizTalk.Component;

using Microsoft.BizTalk.Component.Interop;

using Microsoft.BizTalk.Message.Interop;

using Microsoft.RuleEngine;

using System;

using System.Collections;

using System.Collections.Generic;

using System.IO;

using System.Linq;

using System.Runtime.InteropServices;

using System.Text;

using System.Threading.Tasks;

using System.Xml;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

[ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

[ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]

[Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]

public class


        SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent

{

private FFDasmComp disassembler = new FFDasmComp();

private static


        Dictionary<

string

> cachedSources =

new


        Dictionary<

string

string>();

#region IBaseComponent Members

public string


        Description

{

get

{

return "Flat file disassembler which resolves schemas using message context";

}

public string


        Name

{

get

{

return "Receive Pipeline Schema Resolver";

}

public string


        Version

{

get

{

return "1.0";

}

#endregion

#region IComponentUI Members

public IntPtr Icon

{

get

{

return System.IntPtr.Zero;

}

public IEnumerator Validate(object obj)

{

return null;

}

#endregion

#region IDisassemblerComponent Members

public IBaseMessage GetNext(IPipelineContext pContext)

{

// Delegate call to Flat File disassembler

return disassembler.GetNext(pContext);

}

public void


        Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)

{

// Delegate call to Flat File disassembler

disassembler.Disassemble(pContext, pInMsg);

}

#endregion

#region IProbeMessage Members

public bool


        Probe(IPipelineContext pContext, IBaseMessage pInMsg)

{

// Check arguments

if (null == pContext)

throw new


        ArgumentNullException(

"pContext");

if (null == pInMsg)

throw new


        ArgumentNullException(

"pInMsg");

// Check whether input message doesn't have a body part or it is set to null, fail probe in those cases

if (null == pInMsg.BodyPart || null == pInMsg.BodyPart.GetOriginalDataStream())

return false;

Stream sourceStream = pInMsg.BodyPart.GetOriginalDataStream();

string messageType = string.Empty;

//Read ReceivedFileName from the message context

string srcFileName = pInMsg.Context.Read("ReceivedFileName"

"http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();

srcFileName = Path.GetFileName(srcFileName);

//Substring the first four digits to take source code to use to call BRE API

string customerCode = srcFileName.Substring(0, 4);

//create an instance of the XML object

XmlDocument xmlDoc =

new


        XmlDocument();

xmlDoc.LoadXml(string

.Format(@"<ns0:Root
        xmlns:ns0=

'http://TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE'>

<SrcCode>{0}</SrcCode>

<MessageType></MessageType>

</ns0:Root>", customerCode));

//retreive source code in case in our cache dictionary

if (cachedSources.ContainsKey(customerCode))

{

messageType = cachedSources[customerCode];

}

else

{

TypedXmlDocument typedXmlDocument =

new


        TypedXmlDocument(

"TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE", xmlDoc);

Microsoft.RuleEngine.Policy policy =

new


        Microsoft.RuleEngine.Policy(

"SchemaResolverPolicy");

policy.Execute(typedXmlDocument);

XmlNode messageTypeNode = typedXmlDocument.Document.SelectSingleNode("//MessageType");

messageType = messageTypeNode.InnerText;

policy.Dispose();

// Fail if message type is unknown

if (string.IsNullOrEmpty(messageType))

return false;

cachedSources.Add(customerCode, messageType);

}

// Get document spec from the message type

IDocumentSpec documentSpec = pContext.GetDocumentSpecByType(messageType);

// Write document spec type name to the message context so Flat File disassembler could access this property and

// do message processing for a schema which has document spec type name we've discovered

pInMsg.Context.Write("DocumentSpecName"

"http://schemas.microsoft.com/BizTalk/2003/xmlnorm-properties", documentSpec.DocSpecStrongName);

// Delegate call to Flat File disassembler

return disassembler.Probe(pContext, pInMsg);

}

#endregion

}

Figure 23- Implement IProbeMessage interface

Create Custom Send Pipeline

Add a new Item class to your class library project and rename it
Do same steps of creating custom receive pipeline class but for sending custom pipeline we need to use CategoryTypes.CATID_AssemblingSerializer attribute and implement IAssemblerComponent inteface as show in Figure 24

using Microsoft.BizTalk.Component.Interop;

using System;

using System.Runtime.InteropServices;

using Microsoft.BizTalk.Message.Interop;

using Microsoft.BizTalk.Component;

using System.Collections;

namespace TechNetWiki.SchemaResolver.PipelineComponent

{

[ComponentCategory(CategoryTypes.CATID_PipelineComponent)]

[ComponentCategory(CategoryTypes.CATID_AssemblingSerializer)]

[Guid("ACF451CE-E60D-42E4-8AB0-ED4CADD8B23E")]

public class


        SchemaResolverFlatFileAsmComp : IBaseComponent, IComponentUI, IAssemblerComponent

{

System.Collections.Queue qOutputMsgs =

new


        System.Collections.Queue();

private FFAsmComp assembler = new FFAsmComp();

#region IBaseComponent members

public string


        Description

{

get

{

return "Flat file assembler which resolves schemas using message context";

}

public string


        Name

{

get

{

return "Send Pipeline Schema Resolver";

}

public string


        Version

{

get

{

return "1.0";

}

#endregion

#region IComponentUI Members

public IntPtr Icon

{

get

{

return System.IntPtr.Zero;

}

public IEnumerator Validate(object obj)

{

return null;

}

#endregion

#region IAssemblerComponent Members

public void


        AddDocument(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)

{

qOutputMsgs.Enqueue(pInMsg);

}

public IBaseMessage Assemble(IPipelineContext pContext)

{

IBaseMessage msg = (IBaseMessage)qOutputMsgs.Dequeue();

assembler.AddDocument(pContext, msg);

return assembler.Assemble(pContext);

}

#endregion

}

Figure 24-Custom Send Pipeline component code

Create Receive and Send Pipeline artifacts

Build you Class library project and copy the dll file and paste it to this location <InstallationFolder>\Pipeline Components
Add new receive pipeline to your BizTalk project
Right Click on Toolbox then select choose items then choose as show in Figure 25

Figure 25- Selecting Custom Pipeline components
Drag Receive Pipeline Schema Resolver component from Toolbox and drop it to Disassemble part as shown in Figure 26

Figure 26- Drag Drop Receive Pipeline Schema Resolver component
Add new send pipeline to you BizTalk project
Drag Send Pipeline Schema Resolver component from Toolbox and drop it to assemble part as shown in Figure 27

Figure 27- Drag Drop Send Pipeline Schema Resolver component
Deploy You Pipeline BizTalk project then open BizTalk Administration and create new Receive location in your BizTalk application, configure the location and select the receive pipeline of schema resolver as shown in Figure 28

Figure 28-Select receive schema resolver pipeline
Create new send port, select file adapter ,configure the location and select send pipeline as shown in Figure 29

Figure 29-Select send schema resolver pipeline
Select all required Outbound maps to map files from source to destination as show in Figure 30

Figure 30-Select Outbound maps
Filter your send port to subscribe from receive port as show in Figure 31

Figure 31-Filter send port

We can realized now that if we want to add any new source or any new destination, you just need to add schemas, maps and configure BRE or your configuration table.
This means that you reduced the number of artifacts in your solution pipelines and receive and send ports.
In the bad solution if you have 100 sources and 100 destinations then you will have 100 receive locations, 100 send ports and 100 pipelines.
However, in the good solution you will have 1 receive location, 1 send port and 2 pipelines and your configuration table.

One more thing, that we can do more enhancement to our solution by passing parameter generically to BRE instead of depending on the first 4 charters of file name,
we can use any property of the message context but for demonstration purposes I just use the file name.
More over, we can use canonical schema and mapping (inbound and outbound mapping) dynamically which I will write another article about this subject later on.

Sample Source Code

You can find the sample source code in the following link sample source code

Conclusion

This article demonstrated

The real scenario of receiving and send Flat File from different sources with different formats and sending file to different destinations with different formats
How to implement this scenario using the static biztalk solution
How to implement the schema resolver mechanism using custom receiving and sending pipeline which made our solution more dynamic and easy to maintain
How to build custom receive and send pipeline in details
Benefits of using dynamic schema resolver mechanism
Call BRE API from C# code

Suleiman Shakhtour Integration Tips

Pages

Monday, December 30, 2013

Creating Custom Adapter Provider in ESB Toolkit SFTP As an Example

Introduction

Problem

Solution

Test the solution

Sample Code

Conclusion

See Also

Friday, November 29, 2013

BizTalk Server:Using Global Variable in Mapping

Introduction

Problem

Solution

Sample Code

Conclusion

BizTalk Server Dynamic Schema Resolver Real-life Scenario

Introduction

Bad Solution

Adding New Source Steps

Adding New Destination Steps

Problem

Good Solution

Creating BRE Policy

Create Custom Pipeline Component

Create Custom Receive Pipeline

Create Custom Send Pipeline

Create Receive and Send Pipeline artifacts

Sample Source Code

Conclusion

See Also