Thursday, October 8, 2015

BizTalk Server: Receiving SAP IDOC Messages


"Intermediate documents (IDOCS) are standardized EDI-like documents supported by SAP for asynchronously communicating with both SAP and non-SAP systems. IDOCS are used to send and receive business documents like sales orders to or from a trading partner’s SAP system or external program."
The below is a sample file of IDOC message

In this post, I will explain how to receive and process IDOC message using BizTalk.
Note that I supposed that you installed sap client and sap-wcf adapter in your development machine.


You have a new request that in your integration middle-ware that using BizTalk server needs to integrates with SAP ERP system which will generate outbound messages for external system using IDOC messages over file system or ftp..etc. As a role of BizTalk developer you need to know how to parse this message and process it into your internal business process.


BizTalk Server provides a nice wcf-sap adapter to connect to SAP and generate a meta data (schema) of related IDOC messages.
First, when you check the structure of IDOC message, you will notice that it is a flat file content
which means you need to have a flat file schema as shown in previous sample message image.

IDOC messages are complicated messages, so you don't need to build schema manually because it will take a lot of time to build such kind of schema instead you can use wcf-sap adapter to generate schema for you.

There are some tricks and tips you need to consider to generate a proper IDOC schema which I will explain in the followings steps:

  1. Create a new BizTalk Project è  right click project è  select Add è  Add Generated Item
  2. In Add Generated Item Window select Consume Adapter Service è  Consume Adapter Service
  3. In Consume Adapter Service windows select sapBinding in Select a binding drop down list è  click configure button
  4. In Configure Adapter window security tab select User Name in Client credential type drop down list è  enter User name and password as provided by SAP admin
  5.  Select URI Properties tab then set Application Server Host è  Set System Number è   Set Client è   Set Connection type as provided by SAP team è  Click Ok button.  Note these are the minimum requirements you may need to fill other properties in case there are some special configuration in SAP server side
  6.  In Consume Adapter Service window click connect buttonè   select Service (Inbound operations) from select contract type drop down list è  Select IDOC node

      Note: you may receive the following error when you try to search for a message type without selcting IDOC node
  7. Let's say that you interested in Address Master message which is is ARDMAS when you try to search on ADRMAS* you will find this
  8. When you try to select message you will get the following error 
  9. Tip: Don't try to search for SAP IDOC message using search feature, search for message manually using your eyes instead, you are lucky because messages ordered alphabetically :)
  10. Let's Say SAP team informed us to use ADRMAS03.V3(731) which is the latest version  è select the message è    Select Receive then click Add button then click ok button 

  11. Now you can see the generated files which contain 4 scehmas and Binding files. 
  12. Note: IDocOperation.ADRMAS03.731.3.Receive is the main schema that you want to deal with , IDoc.ADRMAS03.731.3 is schema used by previous schema, IDocSharedTypes which is datatype schema imported by IDoc.ADRMAS03.731.3 schema. You can ignore IDocOperation.ADRMAS03.731.3.ReceiveResponse schema and Binding file  because we only receiving file over file system
  13.  When you check IDocOperation.ADRMAS03.731.3.Receive schema you will find out there is no a flat file tab and child delimiter  0xd 0xa which means it is expecting file that generated from Windows system which is Carriage Return Line feed (CR+LF), however in my case SAP in Linux os that generates file with 0x0A Line Feed (LF) delimiter inside the file content
  14. We need to change Schema Editor Extensions to Flat File and we need to replace child delimiter 0xd 0xa with  0x0A in IDocOperation.ADRMAS03.731.3.Receive.xsd and IDoc.ADRMAS03.731.3.xsd , open files using xml editor then replace. The question what if you receive IDOC messages from different SAP providers one using window and other using Linux then you need to use a customer pipeline to replace new line delimiter instead of updating schema manually.

  15. You can test your IDOC message by validating message using schema editor and setting Validate Instance Input type to Native and Input Instance Filename with path of your sample file 
  16. Now you need to create receive pipeline and add Flat file disassembler to Disassemble component and set Document Schema to IDocOperation.ADRMAS03.731.3.Receive schema data type 
  17. Now you are ready to receive idoc message from any File system, ftp, sftp ...etc , create related adapter and set your receive flat file pipeline then you can convert flat file file to idoc xml document


In this post I walked-throughed how to use wcf-sap adapter to generate IDOC schemas and I highlighted some tips and tricks for working IDOC message which I hope you will help you in any future works

Wednesday, August 5, 2015

BizTalk Server: Extending WCF Adapter


One of the good features of BizTalk server is the extensibility of WCF adapter. WCF (Windows Communication Foundation) enables developers to extend and updates its standard run time behavior with several extensibility points like behavior, binding, binding element, channels, etc.
In this post, I will explain the followings:
  • How to extend BizTalk WCF Adapter?
  • Illustrate how to overcome one of the limitations of WCF Adatpter


I faced an issue when I tried to integrate with a web service of one of our customers.
They sent us a web service url and this url is a long url (261 characters!!) when I tried to configure a BizTalk WCF adapter it gave me the following error as shown in figure 1:
"Error saving properties.(System.ArgumentException) Invalid address length; specified address length is 261 characters, limit is 256 characters."

Figure 1. Error Message

If you try to consume a web service using a console application you will not face any issue related to url length because there is not a limit for address property.However, BizTalk WCF adapter has a limit because BizTalk team defined End Point Address property of WCF Adapter as type of URI with length 256.


To work around this issue, WCF adapter provides the capabilities to be updated and extended.
What we need is to have custom endpoint behavior and to provide a dummy url to EndPoint Address URL and provide a real long address in an extended custom property

To prepare a custom endpoint behavior we need to followings the following steps:
  1. Create a class library project and we will call it as TechnetWiki.CustomEndpointBehavior as shown in figure 2

    Figure 2. Creating Class Library Project
  2. Now we need to create 2 classes one to extend abstract class BehaviorExtensionElement  to define custom property for our custom url and interface IEndpointBehavior to set a custom url to Endpoint url at run time.To use these abstract class and interface we need to add reference to System.ServiceModel dll as shown
  3. Create a new class and we will call it as CustomBehaviorExtensionElement that extends BehaviorExtensionElement
    using System;
    using System.Configuration;
    using System.ServiceModel.Configuration;
    namespace TechnetWiki.CustomEndpointBehavior
        public class CustomBehaviorExtensionElement : BehaviorExtensionElement
            private ConfigurationPropertyCollection properties;
            public Uri CustomAddress
                    return (Uri)base["CustomAddress"];
                    base["CustomAddress"] = value;
            public override Type BehaviorType
                    return typeof(CustomEndpointBehavior);
            protected override ConfigurationPropertyCollection Properties
                    if ( == null)
               = new ConfigurationPropertyCollection
                            new ConfigurationProperty("CustomAddress", typeof(Uri), null, null, null, ConfigurationPropertyOptions.IsRequired)
            protected override object CreateBehavior()
                return new CustomEndpointBehavior(this.CustomAddress);
  4. Create another class called CustomEndpointBehavior that extends IEndpointBehavior interface
    using System;
    using System.ServiceModel.Description;
    namespace TechnetWiki.CustomEndpointBehavior
        public class CustomEndpointBehavior : IEndpointBehavior
            public Uri CustomAddressUri { get; set; }
            public CustomEndpointBehavior(Uri customAddressUri)
                if (customAddressUri == null)
                    throw new ArgumentNullException("customAddressUri");
                this.CustomAddressUri = customAddressUri;
            public void AddBindingParameters(ServiceEndpoint serviceEndpoint, System.ServiceModel.Channels.BindingParameterCollection bindingParameters)
            public void ApplyClientBehavior(ServiceEndpoint serviceEndpoint, System.ServiceModel.Dispatcher.ClientRuntime behavior)
                serviceEndpoint.Address = new System.ServiceModel.EndpointAddress(this.CustomAddressUri);
            public void ApplyDispatchBehavior(ServiceEndpoint serviceEndpoint, System.ServiceModel.Dispatcher.EndpointDispatcher endpointDispatcher)
            public void Validate(ServiceEndpoint serviceEndpoint)
  5. Sign assembly by defining a strong key for the class library project as shown in figure 3

    Figure 3. Sign the assembly.
  6. Add assembly to GAC either by adding assembly to resources of BizTalk Application in BizTalk administration as shown in figure 4 or by run gacutil -i "{AssemblyPath}" command.<

    Figure 4. Adding Assembly to GAC
  7. To register behavior extensions to WCF Adapter, you need to update machine.config in path "C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config" by adding the following configuration inside behaviorExtensions tag <add name="customEndpointBehavior" type="TechnetWiki.CustomEndpointBehavior.CustomBehaviorExtensionElement, TechnetWiki.CustomEndpointBehavior, Version=, Culture=neutral, PublicKeyToken=415d904e2f9a9bcd" />
    Note: in case your WCF-Custom handler runs under 64 bit you need to update the machine.config in the following path  "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config"
  8. Now we are ready to use extension in WCF adapter, you need to create WCF-Custom send adapter and set any dummy url in Address (URI) as shown in figure 5

    Figure 5. Setting Dummy URL
  9. Select Behavior tab and right click EndPointBehavior then select customEndPointBehavior which is the extension the we created and define in machine.config as shown in figure 6

    Figure 6. Select Extention
  10. Set the real URL which has the long url as shown in figure 7 

    Figure 7. Custom URL Setting


BizTalk Server enables developers to extend WCF adapter to overcome any limitations in WCF capabilities. In this post we explained one of the scenarios that needed to extend WCF to overcome the limitation of WCF Adapter and we walk-throw in the steps needed to register wcf extension in WCF-Custom Adapter.

See Also

Another important place to find an extensive amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.

Monday, February 3, 2014

Dynamic Mapping Resolver


Mapping is one of the main activities in any integration solutions. We always use mapping to transform an input message to another message format and structure. It is very crucial to have a dynamic solution as much as we can to let our solution easy to update and deployment process smooth and not affecting the existing code.

Where do we use mapping?

  • We use mapping in inbound maps in receive port as shown in figure 1

Figure 1.
Inbound Maps

  • We can use mapping in outbound maps in send port  as shown in figure 2

Figure 2. 
Outbound Maps
  • We can use mapping in orchestration as shown in figure 3

Figure 3. 
Mapping in Orchestration


In orchestration we can see a big problem that if there is any a new map that we need to use in orchestration then we have to change orchestration which means to recompile and a redeploy our solution and that will affect the existing BizTalk solution process. On the other hand, in inbound and outbound maps if you imagine that we have dozens of maps, it will be headache for BizTalk administrator and developer to make sure that all maps are configured correctly and sometimes we need an operator user to update a BizTalk application configuration from BRE or a web portal because he is not convenient with BizTalk administration console and what if you have some cases that you want to map depending on context or content of that message in inbound and outbound maps without using orchestration.


We can have a solution by applying a dynamic mapping resolver.

What is dynamic map resolver?

It is a mechanism of dynamically(at run time) associate map name from existing repository like business rule engine or custom configuration database table to transformer based on an identifier contained in each source instance message.

How to implement dynamic map resolver?

We need to call transformation code from .NET code in custom pipeline component. I used ILSpy to reflect the code of TransformStream method from Microsoft.Practices.ESB.Itinerary.Services assembly. This method takes as parameters a Stream containing the message to transform, a String containing the fully qualified name of a map deployed within BizTalk, a Boolean for validating map source with message type passed and reference to message type. The method returns a Stream containing the transformed document.

I created a map helper that will be called from custom pipeline component to transform the stream of source to the expected destination stream.
01.using Microsoft.XLANGs.BaseTypes;
02.using Microsoft.XLANGs.RuntimeTypes;
03.using System;
04.using System.Globalization;
05.using System.IO;
06.using System.Xml.XPath;
08.namespace TechNetWiki.DynamicMappingResolver.Library
10.    public class MapHelper
11.    {
12.        public static Stream TransformStream(Stream stream, string mapName, bool validate, ref string messageType)
13.        {
14.            Type mapType = Type.GetType(mapName);
15.            if (null == mapType)
16.                throw new Exception(string.Format("Map {0} is not exist.", mapName));
18.            TransformMetaData metaData = TransformMetaData.For(mapType);
20.            SchemaMetadata schema = metaData.SourceSchemas[0];
21.            String sourceMap = schema.SchemaName;
23.            SchemaMetadata targetSchema = metaData.TargetSchemas[0];
25.            if (validate)
26.            {
27.                if (string.Compare(messageType, sourceMap, false, CultureInfo.CurrentCulture) != 0)
28.                    throw new Exception(string.Format("Unable to execute map,'{0}'."+
29.                        "The source document type,'{1}'does not match the maps's "+
30.                        "target source document type,'{2}'", mapName,
31.                           messageType,
32.                           sourceMap));
33.            }
34.            messageType = targetSchema.SchemaName;
36.            XPathDocument doc = new XPathDocument(stream);
37.            ITransform transform = metaData.Transform;
38.            Stream output = new MemoryStream();
39.            transform.Transform(doc, metaData.ArgumentList, output, null);
40.            output.Flush();
41.            output.Seek(0, SeekOrigin.Begin);
42.            return output;
43.        }      
44.    }

Note that you have to add <BizTalkInstallationFolder>\Microsoft.XLANGs.BaseTypes.dll and <BizTalkInstallationFolder>\Microsoft.XLANGs.RuntimeTypes.dll
to your class library project to build map helper code.

Then we need to create a custom pipeline component by creating class library project that has a reference to the created project that including MapHelper code and add reference to these dlls <BizTalkInstallationFolder>\Microsoft.BizTalk.Pipeline.dll<BizTalkInstallationFolder>\Microsoft.XLANGs.BaseTypes.dll.

We need to use any configuration repository to retrieve map information at run time like Business Rule Engine (BRE ) or custom configuration table in database . In this article, I am using Business Rule Engine to retrieve map information.

BRE implementation

I will retrieve mapping map name depending on the MessageType which represents the source message type.However, you can use any context information to retrieve map name or any content information of the message. I created schema for this purpose containing MessageType and MapStrongName
as shown in figure 4.

Figure 4.
BRE Schema

 Then I created BRE rules using Business Rule Composer as shown in figure 5.

Figure 5.
Sample BRE Rule

Note: that the message type is formed as Namespace#SchemaRoot and map strong name  is formed as MapName, assembly strong name.
You can use vocabulary to ease life of operator user too.

Dynamic map resolver in Inbound Maps

We need to create a custom pipeline component and adding this component to pre-assemble part in the receive pipeline
so first we need to implement IComponent which contains Execute where we need to call TransformStream method as shown in the following code

01.public IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
02.        {                     
03.            string messageType = null;
04.            string messageSourceType = pInMsg.Context.Read("MessageType", "").ToString();
05.            string mapName = GetMapStrongName(messageSourceType);           
06.            System.IO.Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();           
07.            pInMsg.BodyPart.Data = MapHelper.TransformStream(originalStream, mapName, false, ref messageType);
08.            return pInMsg;
09.        }

We need to redecorate custom pipeline class with CategoryTypes.CATID_Any to be allowed add it to Pre-assemble component in receive pipeline and validate component in send pipeline .

The complete code of custom pipeline as following

001.using Microsoft.BizTalk.Component.Interop;
002.using Microsoft.BizTalk.Message.Interop;
003.using Microsoft.RuleEngine;
004.using System;
005.using System.Collections;
006.using System.Collections.Generic;
007.using System.Runtime.InteropServices;
008.using System.Xml;
009.using TechNetWiki.DynamicMappingResolver.Library;
011.namespace VeriCash.BizTalk.PipelineComponents
013.    [ComponentCategory(CategoryTypes.CATID_Any)]   
014.    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
015.    [Guid("5153F77B-F6E3-4F6A-AB96-670D324C8F7A")]
016.    public class DynamicMapResolverComp : IBaseComponent, IComponent, IComponentUI
017.    {
018.        /// <summary>
019.        /// The cached sources to cache values of sources schema message types
020.        /// </summary>
021.        private static Dictionary<string, string> cachedSources = new Dictionary<string, string>();
023.        #region IBaseComponent Members
025.        public string Description
026.        {
027.            get
028.            {
029.                return "Dynamic mapping resolver";
030.            }
031.        }
033.        public string Name
034.        {
035.            get
036.            {
037.                return "Dynamic Mapping Resolver";
038.            }
039.        }
041.        public string Version
042.        {
043.            get
044.            {
045.                return "1.0";
046.            }
047.        }
049.        #endregion
051.        #region IComponent Members
053.        public IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
054.        {                     
055.            string messageType = null;
056.            string messageSourceType = pInMsg.Context.Read("MessageType", "").ToString();
057.            string mapName = GetMapStrongName(messageSourceType);           
058.            System.IO.Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();           
059.            pInMsg.BodyPart.Data = MapHelper.TransformStream(originalStream, mapName, false, ref messageType);
060.            return pInMsg;
061.        }    
063.        #endregion
065.        #region IComponentUI Members
066.        public IntPtr Icon
067.        {
068.            get
069.            {
071.                return System.IntPtr.Zero;
072.            }
074.        }
075.        public IEnumerator Validate(object obj)
076.        {
077.            return null;
078.        }
079.        #endregion
081.        #region Helper
083.        private string GetMapStrongName(string messageType)
084.        {
085.            string mapStrongName = string.Empty;
086.            ////create an instance of the XML object
087.            XmlDocument xmlDoc = new XmlDocument();
088.            xmlDoc.LoadXml(string.Format(@"<ns0:BRERoot xmlns:ns0='http://TechNetWiki.DynamicMappingResolver.Schemas.BRESchema'>
089.                              <MessageType>{0}</MessageType>
090.                              <MapStrongName></MapStrongName>
091.                            </ns0:BRERoot>", messageType));
092.            if (cachedSources.ContainsKey(messageType))
093.            {
094.                mapStrongName = cachedSources[messageType];
095.            }
096.            else
097.            {
098.                ////Prepare object to call BRE API
099.                TypedXmlDocument typedXmlDocument = new TypedXmlDocument("TechNetWiki.DynamicMappingResolver.Schemas.BRESchema", xmlDoc);
100.                Microsoft.RuleEngine.Policy policy = new Microsoft.RuleEngine.Policy("DynamicMappingResolverPolicy");
101.                policy.Execute(typedXmlDocument);
102.                XmlNode messageTypeNode = typedXmlDocument.Document.SelectSingleNode("//MapStrongName");
103.                mapStrongName = messageTypeNode.InnerText;
104.                policy.Dispose();
105.                //// Fail if message type is unknown
106.                if (string.IsNullOrEmpty(mapStrongName))
107.                {
108.                    throw new Exception(string.Format("Map for message type {0} is not exist", messageType));
109.                }
111.                cachedSources.Add(messageType, mapStrongName);
112.            }
113.            return mapStrongName;
114.        }
116.        #endregion
117.    }

  • Create a new BizTalk project and add assembly to <BizTalkInstallationFolder>\Pipeline Components  
  • Add new item and choose  Receive Pipeline
  • Right click to toolbox and choose dynamic mapping resolver as shown in figure 6 

Figure 6.
Adding Dynamic Mapping Resolver in ToolBox
  • Drag XML dis-assembler to disassemble part and drag Dynamic Mapping Resolver to Validate part as shown in figure 7.

Figure 7.
Final Receive Pipeline

Dynamic map resolver in Outbound Maps

  • Add new item and select send pipeline. 
  • Drag Dynamic Mapping Resolver to Pre-Assemble part and Xml-Assembler to Assemble part 
    as shown in
    figure 8.

Figure 8.
Final Send Pipeline
  • Deploy BizTalk project
Note: that I am using xml instance for testing sample source code, if you want to use flat file then you can use Flat file dissembler and assembler.

Dynamic map resolver in Orchestration

  • You can have a sample orchestration as shown in figure 9

Figure 9.
Sample orchestration

We can use dynamic transformation by using  XLANG which exposes a transform method that can be called from within a Message Assignment shape inside of a Construct Message shape. This is the same method that is called when a Transform shape is used.

2.transform(msgDestination) = MapType(msgSource);

Message / Variable Identifier Type Comments
Message msgBRE TechNetWiki.DynamicMappingResolver.Schemas.BRESchema The BRE instance message
Message msgSource System.Xml.XmlDocument The incoming instance message
Message msgDestination System.Xml.XmlDocument The outgoing instance message
Variablut MsgType System.Type Type of map

Sample Code

You can find the source code belonging to this article on MSDN Code Gallery: BizTalk Server: Dynamic Mapping Resolver


In this article I explained and answered the following questions:
  • What is dynamic mapping resolver?
  • What are the problems that we can solve by using dynamic mapping resolver?
  • How to implement dynamic mapping resolver in receive/send pipeline and orchestration?

See Also

Read suggested related topics:
Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.