Summary: Learn about the improvements to the XML tools BizTalk Editor and BizTalk Mapper in Microsoft BizTalk Server 2002. (27 printed pages)
Using Mixed Content in BizTalk Editor
Creating Native Instances
Testing with Delimiters
Extending the Dictionary
Mapping Grid Pages
Using Mixed Content in BizTalk Mapper
Testing with XML and Native Instances
Testing with Document Delimiters
Generating XSLT Output
Using the Looping Functoid
Using the Logical Existence Functoid
Using the Value Functoids
XML Tools: Tips and Tricks
Using Subfield Delimiters with Flat Files
Using Functoids to Perform Aggregation
Extracting Data from a Text Document
Declaring Global Variables
Using Element Groups
Using Custom Namespaces
Microsoft® BizTalk™ Server 2002 provides the XML tools BizTalk Editor and BizTalk Mapper, which contain many improvements over their Microsoft BizTalk Server 2000 versions. This article describes some of those improvements and also provides information about less commonly known functionality available since BizTalk Server 2000. The improvements to BizTalk Editor that are discussed include consuming mixed content, creating native instances, using delimiters to validate document instances, and performing self-routing. The improvements to BizTalk Mapper that are discussed include creating multiple grid pages, testing native instances directly in a map, and using functoids in new ways. The information in this document is intended for developers who have worked with BizTalk Editor and BizTalk Mapper in BizTalk Server 2000.
BizTalk Editor is an XML developer tool included in Microsoft BizTalk Server 2002 that enables you to create, edit, and manage XML-Data Reduced (XDR) specifications. You use these specifications to uniformly structure your data so that you can set up e-commerce business processes with your trading partners in BizTalk Server.
BizTalk Editor displays the hierarchical structure of documents on the left side of its user interface. On the right side, it displays tabs that present the property settings for the nodes in the document hierarchy. Regardless of the document format (flat file, EDI, XML, and so on), the representation of fields and records is always in Extensible Markup Language (XML).
The following topics discuss some of the new functionality included in BizTalk Editor:
- Using Mixed Content in BizTalk Editor shows how to create a specification from text with embedded elements.
- Creating Native Instances discusses the process of creating non-XML-based instance files within BizTalk Editor.
- Testing with Delimiters discusses how delimiters function within BizTalk Editor.
- Exporting XSD describes how to export a specification to interoperate with other systems that are currently using XML schema definition language (XSD).
- Extending the Dictionary discusses adding customized properties to create a self-routing document in BizTalk Editor.
The term "mixed content" refers to elements within a schema that contain both text and subelement content; a mixed-content element can also carry attributes. Mixed content pertains only to XML. It does not apply to flat files or EDI (X12 or EDIFACT). The ability to use mixed content is a new capability in BizTalk Server 2002, and is meant primarily for organizations that use XDR schemas natively. You can use mixed content only with records in BizTalk Editor.
The following example is a sample used to create a mixed-content schema:
<NewsArticle>A <Genre>Technology</Genre> article appeared on <Source>MSNBC</Source> during August 2001. The article is titled <Title>Growing up with the PC</Title> and was written by <ByLine>Bob Sullivan</ByLine>. <BodyText>Lucky for me, Dad was always bringing home the latest…</BodyText></NewsArticle>
In this sample, tags are embedded within the article text. If you tried to create a schema for this sample in BizTalk Server 2000, an error would be returned because BizTalk Server 2000 does not process mixed content.
However, in BizTalk Server 2002, the mixed-content schema for the previous sample would look like the following schema fragment:
<ElementType name="NewsArticle" content="mixed" model="closed"> <element type="Genre" maxOccurs="1" minOccurs="0"/> <element type="Source" maxOccurs="1" minOccurs="0"/> <element type="Title" maxOccurs="1" minOccurs="0"/> <element type="ByLine" maxOccurs="1" minOccurs="0"/> <element type="BodyText" maxOccurs="1" minOccurs="0"/> </ElementType>
To create this schema in BizTalk Server 2002, the content attribute of an ElementType definition in the schema needs to be set. To do this in BizTalk Editor, set the Content property to Mixed on the Declaration tab. The following illustration shows the Content property set to Mixed for the NewsArticle root node.
Figure 1. Content property set to Mixed (click image to see larger picture)
When the Content property is set to Mixed in BizTalk Editor, the resulting output is represented as:
<ElementType name="NewsArticle" content="mixed" model="closed">
The Content property can have one of four possible values. These are:
- Element Only The element can contain only child elements.
- Empty The element can contain attributes and cannot have any subelements under it.
- Text Only The element can contain only textual content and cannot have any subelements under it. This is the only value available for fields in BizTalk Editor.
- Mixed The element can have both textual and subelement content.
BizTalk Server 2002 enables you to create native instances of data and use them to test schemas in BizTalk Editor. Native data is any non-XML data, such as electronic data interchange (EDI) or flat file. This is a significant addition to BizTalk Server 2000, which enabled you to create only an XML instance to provide a sample file for use with a schema. For example, in BizTalk Server 2000, if you used native data, you had to use a separate tool to create the instance file to test all specifications not based on XML.
BizTalk Server 2002 includes the Create Native Instance option on the Tools menu. When you click this option, BizTalk Editor looks at the Structure setting on the Parse tab pertaining to the root node. It uses this setting, which indicates the type of data, to create an instance document conforming to the schema. This is done by directly invoking the serializer (a run-time component for non-XML generation) to generate the native output.
In addition to enabling you to create a file, BizTalk Editor shows the native instance directly on the Output tab. The Output tab, shown in the following illustration, displays the instance data in the format represented by the schema.
Figure 2. The Output tab (click image to see larger picture)
Note If the specification being created represents an XML document, then the Create Native Instance option is unavailable.
With flat files, you simply use the Create Native Instance option. In contrast, when your specification is based on an EDI structure, you use the Create Native Instance option, but you also need to set delimiter values. BizTalk Editor displays the Document Delimiters dialog box (shown in the following illustration), which you use to set the delimiter values you want or accept the default values for the native instance. You can find this Document Delimiters box by clicking Options on the Tools menu. You can set delimiters only once or have BizTalk Editor prompt you for specific delimiters during each instance test.
Figure 3. The Document Delimiters dialog box
Note The ability to create native instances by using the Create Native Instance functionality is for creating and testing specifications. Use BizTalk Editor in a development/test environment to create test instances, but continue to use the BizTalk Messaging Manager run-time infrastructure to test envelopes.
In BizTalk Server 2000, you could set delimiter properties to specify which characters were used to separate data within an envelope and the documents of an interchange. Now, in BizTalk Server 2002, you can also use delimiters to validate document instances.
When you work with delimited flat files, there are two main issues to keep in mind:
- If you are creating a schema for a positional flat file, set up the root element as a positional record type. If you are creating a schema for a delimited flat file or for a flat file that is both positional and delimited, set up the root element as a delimited record type.
- The delimiter character specified for the parent delimited record cannot appear in the data being parsed by the positional record portion of the specification. If the delimiter appears in the data being parsed by the child positional record, you must use an escape for that data to prevent it from being interpreted as a delimiter. This means that either a different delimiter needs to be used, or there needs to be a preprocessing step.
When you set up a document instance, four options can be set for specifying delimiters. These options apply to both the parser and the serializer. These options are set for the root node on the Reference tab and are available only when the Standard property is set to CUSTOM. The following illustration shows an example of a document instance with the default delimiters set on the Reference tab.
Figure 4. A document instance with the default delimiters set on the Reference tab (click image to see larger picture)
The default delimiters are the following:
- Default Record Delimiter This is the delimiter used to notify BizTalk Server of the end of the record. This can be any of the listed values or you can enter another value manually. In the previous illustration, the record delimiter is an asterisk (*).
- Default Field Delimiter This is the delimiter used to separate the data fields within the record. In the previous illustration, the field delimiter is a comma (,).
- Default Subfield Delimiter This is the delimiter used to separate subcomponents within a field. In the previous illustration, the subfield delimiter is a colon (:).
Important The subfield delimiter is for use with EDI document types only.
- Default Escape Character This is the character that indicates to BizTalk Server that the next character should not be evaluated as a delimiter. In the previous illustration, the escape character is a question mark (?).
After the delimiters and their associated values for the root level of a document have been set, a delimiter for the child nodes directly below the current node must be selected. You do this by choosing one of the following options for the Delimiter Type property on the Parse tab:
- Default Record Delimiter
- Default Field Delimiter
- Default Subfield Delimiter
If Character is selected, the Delimiter Value property is enabled and you can enter the appropriate delimiter value. If the Default Record Delimiter, Default Field Delimiter, or Default Subfield Delimiter option is selected, BizTalk Server uses the same value that is entered for the root node on the Reference tab. For example, if you select Default Field Delimiter for the child node on the Parse tab, this directs BizTalk Server to use the value entered for the Default Field Delimiter property for the root node on the Reference tab.
If you are presented with the Document Delimiters dialog box, it is because the Prompt for delimiters during instance test check box is selected. If you change the delimiters during instance testing, those delimiters are used only to test the current instance. The delimiters specified on the Document Delimiters tab remain the default delimiters.
BizTalk Server 2002 has the ability to export specifications created in BizTalk Editor into XSD schemas. This capability existed with BizTalk Server 2000, in the form of an Extensible Stylesheet Language (XSL) style sheet located in the Program Files\Microsoft BizTalk Server\SDK\Messaging Samples\XSDConverter directory. Now you invoke the same XSD style sheet on the Tools menu in BizTalk Editor. This update provides better capability to interoperate with other systems that use XSD, and enables the creation of standards-based XSD representations of the specifications created in BizTalk Editor.
Creating the XSD representation is a two-step process. First, you create the XDR representation of the schema as a specification in BizTalk Editor. Then, BizTalk Editor feeds the XDR representation into the style sheet for conversion into the XSD representation. Remember that BizTalk Editor is still using XDR natively, so you need to create a specification that can transform an XSD schema. Trading partners can send instances of XML that are derived from an XSD schema. Those instances will be compatible with BizTalk Server if the XSD schema on the trading partner side of the transaction/communication is a functional match of the XDR equivalent on the BizTalk Server side.
For example, if you use XDR in your BizTalk Server configuration, but your trading partner uses XSD, you can use BizTalk Editor with the XDR schema to create an XSD-equivalent specification. Your trading partner can then use its XSD schema to send you XML document instances, which BizTalk Server can successfully process because the XSD data and your specification are functionally equivalent.
Unlike BizTalk Server 2000, BizTalk Server 2002 enables you to do self-routing by extending the properties on the Dictionary tab and adding customized properties to create a self-routing document in BizTalk Editor. A self-routing document contains all the necessary routing information, such as source and destination organization identifiers and a document definition, in the routing tags or within the document. You can also assign values to the predefined properties and to any custom properties you create.
The predefined properties on the Dictionary tab are:
- Document Container Node This node is used in an envelope, and specifies the node that contains the document.
- Document Name This node is used to set the path to the field that contains the document instance itself.
- Source Type and Source Value These nodes are used in combination and represent values for the source organization. The Source Type is the qualifier (such as Organization Name or Telephone) and the Source Value is the value assigned to the type. As such, if the qualifier is set to Telephone, the Source Value would equate to the value that was set by using the new organization functionality on the Identifier tab in BizTalk Messaging Manager.
- Destination Type and Destination Value These nodes are the same as those for the Source Type and Source Value, except that they represent the values for the destination organization.
When performing self-routing in BizTalk Server, there are two ways to specify the data for BizTalk Server to select:
- You can assign fields in the document to the predefined properties on the Dictionary tab. This enables you to add the values that are necessary to match to the required elements presented by default on the Dictionary tab.
- You can use BizTalk Server to create custom properties and assign values to them on the Dictionary tab. If there are values that already exist in the document, they can be used to provide BizTalk Server with the necessary values to perform self-routing.
To create a custom property, you create a specification in BizTalk Editor, designate the field from which the value for the custom property will be extracted, and then create the custom property. BizTalk Server uses an XPath query to locate and extract the corresponding value from within the document instance and put it into the Node Path column for the new custom property on the Dictionary tab.
After you create all the specifications and custom dictionary properties your self-routing process requires, the next step is to create a document definition in BizTalk Messaging Manager. Then, you enter the names and values of the custom properties you created in BizTalk Editor, so that the BizTalk routing engine can match the values in the document with the new document definition. At this point, BizTalk Server is set up to route the document instance based on the values already contained in the document itself.
The following illustration shows the Dictionary tab for the TRAN-ID field. This field contains two custom properties, TransactionID and ErrorCode, located in the Property column, and their assigned values, located in the Node Path column. In this example, the results of the transaction will contain either error information or data.
Figure 5. The Dictionary tab for the TRAN-ID field (click image to see larger picture)
The BizTalk routing engine uses the values extracted from the information provided on the Dictionary tab in BizTalk Editor. These values are then used to search for the document definition that matches these values. After this is accomplished, the channel in which the document definition is specified is selected. The document instance is then submitted to the channel for processing.
BizTalk Mapper is an XML developer tool included in Microsoft BizTalk Server 2002 that enables you to create XSL Transformations (XSLT) maps to be used to transform the structure of an incoming document instance to the structure of an outgoing document instance, with no coding. You use these maps to process and translate data into formats that can be shared within your own organization and with trading partners.
Just as BizTalk Editor represents fields and records identically, regardless of the document format, so too does BizTalk Mapper. The representation of specifications in BizTalk Mapper is format independent. BizTalk Mapper performs the same task in the same manner no matter whether the transformation is from or to XML, EDI, or flat-file formats.
The following topics discuss some of the new functionality included in BizTalk Mapper:
- Mapping Grid Pages shows how the mapping grid can be split into logical pages for easier viewing.
- Using Mixed Content in BizTalk Mapper describes how to use mixed content in BizTalk Mapper.
- Testing with XML and Native Instances discusses how testing can be performed with native instances.
- Testing with Document Delimiters describes the new functionality available to set the delimiters for the source and destination schemas.
- Generating XSLT Output describes how to include or omit the XML declaration in an output document.
- Using the Looping Functoid describes this functoid and its relationship to source and destination documents.
- Using the Logical Existence Functoid describes this new functoid in the logical functoids group.
- Using the Value Functoids describes the functionality of the Value Mapping and Value Mapping (Flattening) functoids.
In BizTalk Server 2000, there was a large mapping grid on which to place your functoids, but only a small viewing area with which to view complex maps. You could choose the Grid Preview option from the View menu to reposition your view, but, in some cases, the complexity of a map that contained thousands of links resulted in a density of links and functoids that made the map hard to read.
In BizTalk Server 2002, you have the ability to create multiple grid pages, up to 20 total. Also, you can move functoids to a new page if the original page becomes too densely populated. These grid pages work in much the same way as the extra sheets that can be added to a workbook in Microsoft® Excel. You can also name these grid pages to help keep track of which page contains which parts of the map. The name for each page can be up to 32 characters in length. When the map is compiled, the compiler collapses all the pages, treating the functoids as if they were all on one page.
These new grid pages enable you to associate the layers with logical or physical parts of the map. For example, you could create grid pages for Header, Detail, and Summary sections of a map.
As stated previously, mixed content refers to elements within a schema that contain both text and subelement content; the elements can also carry attributes. Mixed content pertains only to XML. It does not apply to flat files or EDI (X12 or EDIFACT). The ability to use mixed content is meant primarily for organizations that use XDR schemas natively.
With BizTalk Server 2002, you can use specifications imported from BizTalk Editor that contain mixed content. You can decide how much of the subelement content is represented when you compile maps in BizTalk Mapper, which enables you to control how the compiler navigates through the Destination Specification tree. To compile all the elements and subelements, you use the new Copy text and sub-content value option. When you choose to compile all the mixed content, the concatenated values of the node and of all the child elements are copied in the incoming document instance. Changing the compiled content changes the outcome of the results when you test the map.
Figure 6. Copy text and sub-content value option (click image to see larger picture)
For example, if you took the following specifications and imported them into BizTalk Mapper, you would get a map with mixed content in the source specification and no mixed content in the destination specification.
Figure 7. MixedContentRecord (click image to see larger picture)
Figure 8. EmptyContentRecord (click image to see larger picture)
In the following illustration, the map was tested with the Instance XML to XML option with the Copy text value compiler option enabled. As you can see, the resulting output (on the Output tab) for Field2 of the destination node contains only data from the MixedContentRecord record.
Figure 9. Instance XML to XML option with the Copy text value compiler option enabled (click image to see larger picture)
In the following illustration, the map was tested with the Instance XML to XML option with the Copy text and sub-content value compiler option enabled. As you can see, the resulting output (on the Output tab) for Field2 of the destination node contains data from both the MixedContentRecord record and the eleField2 field, which is both the element and subelement content.
Figure 10. Instance XML to XML option with the Copy text and sub-content value compiler option enabled (click image to see larger picture)
Just as BizTalk Editor can create native instances, BizTalk Mapper can now create native output from the map test process. BizTalk Mapper also supports both native instances (non-XML data) and XML as input to test the created style sheet.
BizTalk Mapper has extended the Test Map option on the Tools menu. This menu now provides the following testing features:
- Generated XML to XML This feature contains the same functionality as the Test Map option in BizTalk Server 2000. This feature invokes the instance generator function to create a contrived instance of the source document. This document instance is then submitted to the style sheet created by BizTalk Mapper and the output is represented in XML format on the Output tab of BizTalk Mapper.
- Generated XML to Native The contrived instance of the source document gets created and submitted to the style sheet. At this point, the document gets serialized to the native format and displayed on the Output tab. If the destination schema is EDI, then this process also prompts for the delimiters to use for the serialization process.
- Instance XML to XML This feature prompts you to select the instance document to be tested in the map. The resulting XML is displayed on the Output tab.
- Instance XML to Native This feature prompts you to select the instance document and then uses that instance to generate a document that is serialized to the native format specified by the destination document and displayed on the Output tab.
- Native Instance to XML This feature prompts you to select the instance document to be tested in the map. Unlike Instance XML to XML, BizTalk Mapper looks for a native document instance for this feature. The native instance is assumed to be the in same format as the source schema. The output of this feature produces the XML format and displays it on the Output tab.
- Native Instance to Native This feature prompts you to select the instance document to be tested in the map. Unlike Instance XML to XML, BizTalk Mapper looks for a native document instance for this feature. The native instance is assumed to be the same format as the source schema. The output of this feature produces the native format and displays it on the Output tab.
Note For EDI native instances, the Native Instance to XML and Native Instance to Native features are capable of automatically removing the envelopes for a wrapped EDI interchange.
In addition to creating native instances, BizTalk Mapper can also perform validations, according to the specifications, during map tests. New in BizTalk Server 2002 are two check boxes in the BizTalk Mapper Options dialog box. These features are:
- Validate content before map test The map will validate the instance against the source schema before submitting it to the style sheet.
- Validate content after map test The map will validate the instance against the destination schema after the style sheet has created the schema.
Note For both of these validation features, if a schema used in the map was not natively created by BizTalk Editor (such as an imported XDR specification) and you receive warnings regarding namespaces or element groups, you should not use these features because they might cause the validation to fail.
By default, when BizTalk Mapper validates incoming and outgoing instances, it validates the source schema, applies the appropriate style sheet, and then validates the schema again. You can turn off this validation process by clearing one or both of the previously listed check boxes. For example, if you are trying to debug a map failure and you know validation will fail, clearing both Validate content before map test and Validate content after map test will enable you to see the results of the map, which will return errors that you can use to troubleshoot the problem. After you fix the problem, you should select these options again and retry the validation to verify your fixes.
Note When BizTalk Mapper performs a test that uses a native instance, it automatically takes the schema for that native instance and submits that as the envelope. The ability to create native instances is for creating and testing schemas for business documents. Use BizTalk Mapper in a development/test environment to create test instances, but continue to use the BizTalk Messaging Manager run-time infrastructure to test envelopes.
If you want envelope processing outside of this functionality, create a port and channel, identify the inbound/outbound document specifications and the envelopes, and then perform the required testing through this method. The standard functionality of BizTalk Mapper will provide the functionality required for the body of the document, while the map test setup will enable you not only to test the body again, but also to test the envelope and the entire mapping process.
In BizTalk Server 2002, BizTalk Mapper enables you to specify document delimiters when you test maps. The Document Delimiters tab provides the ability to set the delimiters for both the source schema and the destination schema.
After you have chosen the document type, set the required delimiters for the following:
- Record delimiter
- Field delimiter
- Subfield delimiter
- Escape character
At the bottom of this tab is the Prompt for delimiters during instance test check box. This option indicates whether or not BizTalk Mapper should display the Document Delimiters dialog box every time you validate a document instance.
If you are prompted with the Document Delimiters dialog box during the instance test, because the Prompt for delimiters during instance test check box is selected, the delimiters specified are used only for the current instance. The delimiters chosen on the Document Delimiters tab remain the default delimiters.
Microsoft BizTalk Server 2002 uses the data that you provide in a map to generate Extensible Stylesheet Language Transformations (XSLT) between source and destination specifications. Then, the XSLT output is used by Microsoft XML Parser (MSXML) version 3.0 at run time to transform the schema on the server. When you create a map, the XSLT can be viewed in BizTalk Mapper. This functionality has been provided to help non-BizTalk Server trading partner recipients whose systems require this declaration to process XML.
You can find this functionality on the XSLT output tab, which is new in BizTalk Server 2002. It presents only one option, Omit XML declaration. If this is set to No, an XML declaration similar to
<?xml version="1.0" encoding="UTF-16" ?> will be appended to the top of the output document. If this is set to Yes, the declaration will be omitted.
The Looping functoid is a functoid you use with the XSLT compiler. The purpose of the Looping functoid is to explicitly tell the XSLT compiler what the looping relationship is between the source and destination specifications. The compiler usually infers this behavior based on other mappings, but, in some cases, due to the ambiguous structure of a map, inference is not possible.
Consider the following illustration. The VendorPartNumber and BuyerPartNumber fields from the Source Specification tree are linked to the Looping functoid. This, in turn, is connected to the MyPartNumbers record in the Destination Specification tree. In addition, each of the Source Specification tree fields is directly linked to the Destination Specification tree fields.
Figure 11. The Looping functoid (click image to see larger picture)
The link between VendorPartNumber and Qualifier and the link between BuyerPartNumber and Qualifier are set up so that the compiler directive is set to copy the name. The link between VendorPartNumber and PartNumber, and BuyerPartNumber and PartNumber, are set up so that the compiler directive is set to copy the value. Notice also that, for example, VendorPartNumber is mapped to three places in the destination specification, only one of which is MyPartNumbers, which indicates the following:
- One link is to the Looping functoid.
- The second link is to copy the actual value in the PartNumber field.
- The third link is for populating the Qualifier field, which contains the name of the part number (VendorPartNumber). (Node-name mapping is an option that you can specify on the link.)
After setting up the direct links, there are also the links to the Looping functoid. This functoid tells the transformation engine to create multiple output records—one for the Vendor information and the other for the Buyer information.
The following incoming document instance contains both Vendor information and Buyer information:
<Specific><PartNumbers VendorPartNumber="123456vn" BuyerPartNumber="987654bp" /></Specific>
If this is submitted to the map, it produces the following output:
<Generic> <MyPartNumbers Number="1" Qualifier="VendorPartNumber" PartNumber="123456vn" /> <MyPartNumbers Number="2" Qualifier="BuyerPartNumber" PartNumber="987654bp" /> </Generic>
Without the Looping functoid, the XSLT compiler would incorrectly infer that there was a MyPartNumbers record in the destination specification for every VendorPartNumber and BuyerPartNumber field in the source specification. A warning would appear on the Warnings tab, stating the following:
Warning The destination node "Qualifier" has multiple inputs but none of its ancestors is connected to a Looping functoid.
The same warning would also be received for the PartNumber record. The output of the transformation would be the following:
<Generic> <MyPartNumbers Number="1" Qualifier="BuyerPartNumber" PartNumber="987654bp" /> </Generic>
In this case, the transformation did not know to perform the loop and only the first record was created. If this output occurs and two records were expected, then this is a case where the Looping functoid needs to be used. By using the Looping functoid, the compiler can correctly infer to have a MyPartNumbers record for each field inside the VendorPartNumber and BuyerPartNumber fields.
Logically filtered looping
In BizTalk Server 2000, the Looping functoid behavior was absolute, which means it looped explicitly from Node A to Node B. Now, BizTalk Mapper enables you to terminate a link from the Looping functoid and a logical functoid, such as the Equal functoid, in the same place. This means that instead of a one-to-one correlation between Node A and Node B, now there are only as many Node Bs as there are Node As or child nodes that meet the values in the corresponding logical functoid. For this use of functoids to work, both the Looping functoid and the logical functoid must terminate at the same node. This is referred to as logically filtered looping.
For example, if you had a purchase order (PO) that you wanted to loop only if certain conditions or values existed, you could use functoids to create a logically filtered looping scenario, as shown in the following illustration.
Figure 12. Logically filtered looping (click image to see larger picture)
In this example, the POHeader record is linked to the Looping functoid and the Purpose field is linked to the Equal functoid. Both the POHeader record and the Purpose field terminate at the BEG record, creating a linking where Node B is looped only if POHeader or Purpose is equal to BEG.
The Logical Existence functoid is the new logical functoid added to BizTalk Server 2002. The logical functoids, including the Logical Existence functoid, are especially useful when you need to transform XML from one format to another using the Value Mapping and the Value Mapping (Flattening) functoids (which are discussed in the following topic). The Logical Existence functoid returns "true" if the input record or field exists in the source specification. This functoid requires exactly one input parameter.
This section discusses the two BizTalk Server value functoids: Value Mapping and Value Mapping (Flattening).
The Value Mapping functoid enables you to transform a hierarchical XML document. This functoid transforms an attribute from a field into an attribute of a record.
Consider this example. The following document instance has both a Name and Value attribute on the Field element:
<Root> <Record> <Field Name="Sales" Value="1000"/> <Field Name="ThirtyDayMovingAvg" Value="2843"/> <Field Name="SixtyDayMovingAvg" Value="2198"/> </Record> <Record> <Field Name="Sales" Value="1876"/> <Field Name="ThirtyDayMovingAvg" Value="1734"/> <Field Name="SixtyDayMovingAvg" Value="1098"/> </Record> <Record> <Field Name="Sales" Value="1262"/> <Field Name="ThirtyDayMovingAvg" Value="3198"/> <Field Name="SixtyDayMovingAvg" Value="2835"/> </Record> </Root>
If you want to combine all of the attributes associated with the Field element and make them attributes on the Record element, the Value Mapping functoid can accomplish this.
After the value mapping has taken place, the XML output looks like the following:
<Root> <Record Sales="1000" /> <Record ThirtyDayMovingAvg="2843" /> <Record SixtyDayMovingAvg="2198" /> <Record Sales="1876" /> <Record ThirtyDayMovingAvg="1734" /> <Record SixtyDayMovingAvg="1098" /> <Record Sales="1262" /> <Record ThirtyDayMovingAvg="3198" /> <Record SixtyDayMovingAvg="2835" /> </Root>
The following illustration shows how this is accomplished in BizTalk Editor by using a pair of functoids, the Equal functoid and the Value Mapping functoid, for each of the records in the destination document. The records consist of the Sales, ThirtyDayMovingAvg, and SixtyDayMovingAvg fields.
Figure 13. The Equal functoid and the Value Mapping functoid (click image to see larger picture)
The following illustration shows the Functoid Properties dialog box for the Equal functoid.
Figure 14. The Functoid Properties dialog box for the Equal functoid
There needs to be an Equal functoid for each pair, with the only difference between them being the value of the constant. The Functoid Properties dialog box in the previous illustration shows the constant for the Sales field. There is also an Equal functoid for the ThirtyDayMovingAvg and SixtyDayMovingAvg fields.
In the first functoid pair, the Equal functoid and the Value Mapping functoid work together so that the value of the Sales field is passed from each of the records in the source specification to the Sales attribute in the schema representing the transformed format. This also occurs for each of the other fields.
The Value Mapping functoid returns the value of the second parameter defined in the Functoid Properties dialog box only if the first value is "true." Therefore, it is very important to ensure that the link from the Equal functoid is first in the list. The Equal functoid linked to the Value Mapping functoid returns "true" only when the value of the field is equal to the value of the constant. Therefore, for the first element, the Equal functoid mapped to the first attribute will return "true" only when the value is the Sales field. The other Equal functoids, likewise, will return "true" only when the value is the same as the defined constants.
Value Mapping (Flattening)
As its name implies, the Value Mapping (Flattening) functoid enables you to transform a hierarchical XML document into a flattened XML document. A flattened document is a document in which all the source hierarchies are flattened to the parent of the destination node. In other words, if a record repeats three times in the Source Specification tree, only one occurrence will be created in the Destination Specification tree.
Note The Value Mapping (Flattening) functoid demonstrates functionality similar to the Looping functoid in that they are both used with loops; however, the Looping functoid is used to explicitly dictate the occurrence of loops, while the Value Mapping (Flattening) functoid is used to explicitly suppress looping.
For example, in a Looping functoid (non-flattening) scenario, if there are three fields in the source specification, there will usually be three fields in the destination specification. In a Value Mapping (Flattening) functoid scenario, if there are three fields in the source specification, there will be only one field in the destination specification.
Consider the following sample hierarchical document instance:
<Root> <Record> <Field Name="Sales" Value="1000"/> <Field Name="ThirtyDayMovingAvg" Value="2843"/> <Field Name="SixtyDayMovingAvg" Value="2198"/> </Record> <Record> <Field Name="Sales" Value="1876"/> <Field Name="ThirtyDayMovingAvg" Value="1734"/> <Field Name="SixtyDayMovingAvg" Value="1098"/> </Record> <Record> <Field Name="Sales" Value="1262"/> <Field Name="ThirtyDayMovingAvg" Value="3198"/> <Field Name="SixtyDayMovingAvg" Value="2835"/> </Record> </Root>
The output should appear in the following format:
<Root> <Record Sales="1000" ThirtyDayMovingAvg="2843" SixtyDayMovingAvg="2198" /> <Record Sales="1876" ThirtyDayMovingAvg="1734" SixtyDayMovingAvg="1098" /> <Record Sales="1262" ThirtyDayMovingAvg="3198" SixtyDayMovingAvg="2835" /> </Root>
Then, there needs to be a mechanism to transform the name and value pairs from each of the fields in the hierarchical format into the attributes of the record in the flattened format. This is what the Value Mapping (Flattening) functoid does in the map shown in the following illustration.
Figure 15. The Value Mapping (Flattening) functoid (click image to see larger picture)
It is important to maintain a one-to-one correspondence between the number of records in the instance document and the records in the flattened schema. This is accomplished by using a pair of functoids, the Equal functoid and the Value Mapping (Flattening) functoid, for each of the elements in the destination document. In this example, there are three.
The following illustration shows the Functoid Properties dialog box for the Equal functoid.
Figure 16. The Functoid Properties dialog box for the Equal functoid
There needs to be an Equal functoid for each pair with the only difference between them being the value of the constant. The previous Functoid Properties dialog box shows the constant for the ThirtyDayMovingAvg field. There is also an Equal functoid for the Sales and SixtyDayMovingAvg fields.
In the first functoid pair, the Equal functoid and the Value Mapping (Flattening) functoid work together so that the value of the Sales field is passed from each of the records in the source specification to the Sales attribute in the schema representing the flattened format. This also occurs for each of the other fields.
Just like the Value Mapping functoid, the Value Mapping (Flattening) functoid returns the value of the second parameter defined in the Functoid Properties dialog box only if the first value is "true." The Equal functoid linked to the Value Mapping (Flattening) functoid returns "true" only when the value of the field is equal to the value of the constant. So, the Equal functoid mapped to the first attribute will return "true" only when the value is the Sales field. The other Equal functoids, likewise, will return "true" only when the value is the same as the defined constants.
It is important to consider that if there is more than one field with an attribute that matches the second input parameter of one of the Equal functoids, only the last matching record is mapped to the output document. For example, if there were three fields in the first record that had name attributes of Sales, only the last value would be mapped.
The following topics provide additional information for working with BizTalk Mapper and BizTalk Editor:
- Using Subfield Delimiters with Flat Files describes how to use subfield delimiters with flat files.
- Using Functoids to Perform Aggregation describes how to use cumulative functoids to calculate totals and perform some basic statistical functions.
- Extracting Data from a Text Document describes how to submit a text document as a file.
- Declaring Global Variables describes how to declare variables outside of functions, so that all functoids can use the same variables in their processing.
- Using Element Groups discusses how to create a relationship between a set of elements, without necessarily applying that relationship to all peers under a common parent. You can then apply ordering constraints to the element group.
- Using Custom Namespaces describes how to validate document instances at run time with custom namespaces.
In the Testing with Document Delimiters topic earlier in this article, the functionality of the subfield delimiter was discussed. This section discusses how to use subfield delimiters to do additional processing within BizTalk Mapper. While EDI does support the concept of subfield delimiters natively, you can achieve a similar result with flat files.
As an example, assume the following is the format for the address information sent by a trading partner (a flat file with tabs as field delimiters and a carriage return for the record delimiter):
638 Elm Street[tab] Edmonds[tab] WA[tab] 98020-5554[carriage return]
Furthermore, in this example the internal format separates the ZIP Code and the ZIP Code Suffix into separate elements. You could pull the flat file into a word-processing program and search and replace the hyphen (-) with a tab character, for example,
98020[tab]5554. This manual process would need to be done every time you received the data, and introduces the possibility of human error, as well as the inadvertent possibility of omitting this step altogether. Alternatively, you could write a BizTalk Server preprocessor component to automate this process. However, this still introduces an additional step of configuration.
Instead, you could submit the raw data into BizTalk Mapper and separate the fields using functoids. The map and the source and destination document specifications would be created as shown in the following illustration.
Figure 17. Subfield delimiters (click image to see larger picture)
To use this process, you need to:
- Create a specification from the source file in BizTalk Editor to represent the fields in the flat file.
- Create a specification from the destination file in BizTalk Editor to represent your internal representation in an XML format.
- Use the functoids in BizTalk Mapper to separate the postal code information.
First, you map the data that is needed for only the Zip field. The first functoid to be placed on the map is the String Find functoid. This functoid returns the location of a string or character within a string. This performs the equivalent of the Microsoft® Visual Basic® InStr function. The inputs for this first functoid are the PostalCode field itself and a constant defined to be a hyphen (-).
Then, plug the return value into the Subtraction functoid (found on the Mathematical tab). This functoid will take the output from the String Find functoid and a constant defined as 1. The output will be the position of the hyphen in the string minus 1 (in this example, 6-1).
To finish the path, you need to use the String Left functoid. This performs the equivalent of the Visual Basic Left function. Another link from the PostalCode field needs to be linked as the first parameter into the functoid. The functoid will then need the output from the Subtraction functoid as its second parameter. The output will be the left portion of the PostalCode string, which will be the entire five-digit ZIP Code. This output is directly linked to the Zip field on the destination document. After the Zip field has been parsed, create the link for the ZIP Code Suffix.
Because you already have the String Find functoid, which is returning the location of the hyphen on the map, you can link directly to it. Place an Addition functoid (found on the Mathematical tab) on the map. Link the output from the String Find functoid to the Addition functoid, as its first parameter. Create a constant and set its value to 1. The output will be position of the hyphen in the string plus 1 (in this example, 6+1, which puts you at the starting character of the suffix).
To finish processing this path, you need to use the String Extract functoid. This functoid enables you to provide the start and end of the substring to extract. Because this functoid is expecting three parameters, the first will be the string (PostalCode), the second is the start location provided by the Addition functoid, and the third is the end location, which is provided by a constant. In this example, the constant has been set to a high enough number to ensure that we get the remainder of the characters in the string. The results of this functoid are linked to the Suffix field on the destination document.
An alternative way to map the suffix is to replace the Addition functoid with a Subtraction functoid. Link the output from the String Find functoid to the first input parameter of the Subtraction functoid. Then, create a constant as the second parameter and set its value to 2 in the Functoid Properties dialog box.
The value is set at 2 because this value will be subtracted from the value given by the String Find functoid. Next, add a String Right functoid. This functoid takes the string, linked directly from the PostalCode field, and also a starting point, linked from the Subtraction functoid. The functoid will begin at this starting point and take all of the characters from the rightmost end of the string. So, the value passed in from the String Find functoid is 6 and the Subtraction functoid takes that and subtracts the constant value, 2. This gives you the exact number of characters counting from the right end of the string.
When the map is tested, the data output will appear as follows:
<DocumentX> <Address StreetAddress="638 Elm Street" City="Edmonds" State="WA"> <PostalCode Zip="98020" Suffix="5554" /> </Address> </DocumentX>
Either way you decide to map the Suffix field, you will end up with the correct transformation without needing to perform any manual steps. This will enable you to perform the whole document transformation process faster and with less potential for errors.
Another use of functoids is for aggregation. By using the functoids on the Cumulative tab, you can calculate totals, as well as perform some basic statistical functions.
The map in the following illustration shows the transformation from an XML document specification that represents daily statistics values to an XML document specification that represents the same daily statistics with an added summary statistics record. The addition of the summary statistics record is done by using nearly every functoid on the Cumulative tab.
Figure 18. Aggregation map (click image to see larger picture)
To create the summary statistics record, the Cumulative Average functoid is used to create the AvgHiTemp field. This same functoid is also used for the AvgLoTemp field. The Cumulative Average functoid takes exactly one input parameter and works by iterating over all instances of the source specification. This functoid takes the values of each of the source fields and adds them together. It then takes that number and divides it by the total number of values iterated.
To create the MinDailyRainfall field, the Cumulative Minimum functoid is used. This functoid takes only one input parameter and is linked from the Rainfall field. The functoid iterates over all the values and outputs the lowest value.
Using just the opposite method, the Cumulative Maximum functoid is used to create the MaxDailyRainfall field. Again, this functoid takes only one input parameter and is also linked from the Rainfall field. This functoid iterates over all the values and outputs the highest value.
To create the TotalRainfall field, the Cumulative Summary functoid is used. This functoid takes only input and is also linked to the Rainfall field. This functoid outputs the sum of all of the values the functoid iterated over.
The last element of the SummaryStats record is the MidPeriodHiTemp field. This element will show the high temperature for the fifteenth day of the month. This functionality is provided by the Index functoid. The Index functoid provides the ability to specify from which record to extract a value. This functoid takes two parameters. The first is the element itself and the second specifies from which record to extract the data. Because the middle of the month value is required, the second parameter will be a constant set to the value of 15. This sets the Index functoid to return only one value—the value for the fifteenth day of the month.
There is only one more functoid on this map. The Scripting functoid is used to transform the SeasonCode field value and place the new value into the Season field.
When this map is used, it will directly map the daily values over to the destination. It will also use the functoids to iterate over each of the one-to-many values in the source document specification and create a single record for the SummaryStats records in the destination document specification.
Sometimes you might want to submit a text document as a file to extract the individual data elements that are important in the document. The following BizTalk Server 2002 Help topic is an example of a schema:
$00:0000090489: $02: BizTalk Server 2002 Help $95:BizTalk Mapper $89:#C103# #C107# #C335# $145:BizTalk Server 2002 ~ Mapping Data $20:JANUARY 3, 2002 $60:USING BIZTALK MAPPER $120:$T Microsoft BizTalk Mapper is a translation design tool that enables you to create a correspondence between the records and fields in two different specification formats. BizTalk Mapper uses links and functoids to accomplish this translation. Functoids perform operations that range from simple calculations to elaborate script functionality. $TYou can use BizTalk Mapper to graphically represent the structural transformation relationship between source-specification data elements and destination-specification data elements. This cross-reference and data-manipulation functionality creates a map that provides a set of instructions that defines the relationship between two different specification formats. The specification formats are defined by using BizTalk Editor.
In this example, the dollar sign ($) is used as a record delimiter and the colon (:) as a field delimiter. In creating a schema for a document instance, BizTalk Editor provides the functionality to define the delimiters no matter where they might appear in the document. In this case, the delimiters appear at the beginning of the record, so the value of the Field Order property is set to Prefix.
After the Prefix value is set, the record separation behavior is defined. The next task is to differentiate and define the records themselves. The numbers that appear in front of each paragraph are the source tag IDs. These can be used as Source Tag Identifier values. The value of the Source Tag Identifier property is set to these paragraph markings for each of the records in BizTalk Editor.
After all of these header-type records have been set up, the Identifier property of the BodyParagraph record is set to T. There are three notable differences with regard to this record:
- This record is set up as a positional record.
- The Maximum Occurrences property is set to an asterisk (*). This will allow BizTalk Editor to iterate over all of the body text paragraphs and create the correct XML hierarchy.
- Because this record is set up as positional, instead of delimited as all of the other records are, BizTalk Editor needs to know where in the positional stream to find the Source Tag Identifier value. The Source Tag Position value is set on the Parse tab. In this example, it appears at the beginning of the record, but it could appear anywhere within the record.
The following illustration shows how the final setup looks in BizTalk Editor.
Figure 19. Final setup in BizTalk Editor (click image to see larger picture)
The following illustration shows how to map this data after the schema has been created.
Figure 20. How to map data after the schema has been created (click image to see larger picture)
Depending on the format that is required, this text file can be transformed into another text file or into an XML representation.
In this example, the destination is set to be a flat file. However, the text file is in a different layout. There are many fields in the source document that can be directly linked to fields in the destination document. There are four functoids that provide advanced functionality. The Scripting functoid takes the Date field as input and outputs the year portion of the date. The output is then used as input on two Concatenate functoids, and the third Concatenate functoid is used as a constant value.
The output of the first Concatenate functoid is linked to the Copyright field. This functoid has two constants as input, as well as the output of the Scripting functoid. The two constants are set to Copyright and Microsoft. The value that is output to the Copyright field is Copyright 2002 Microsoft. The constants are set up to reflect the correct copyright holder of the topic.
The output of the second Concatenate functoid is linked to the ReportNo field. This Concatenate functoid has one constant as input and the output of the Scripting functoid. The constant is set to BizTalk. The value that is output to the ReportNo field is BizTalk Server 2002.
The last Concatenate functoid is set up to be a constant value in the Language field. This can be accomplished in different ways, for example, you can use a Concatenate functoid or a Scripting functoid. Another way that this can be accomplished is to select the Language field on the destination document and click the Values tab. Then, enter the constant value in the Destination constant value box.
One common mistake when using functoids and BizTalk Mapper is to create local variables to hold values (such as date parts) while, in another functoid, more local variables are created with slightly different names to hold the same values. By declaring the variables outside of the function, all functoids can use the same variables in their processing. This is one of the less familiar capabilities of functoids—the ability to declare global variables.
The map in the following illustration shows how this is accomplished. In this example, there are two functoids, Scripting and Looping, with the majority of the functionality in the Scripting functoid.
Figure 21. Declaring global variables (click image to see larger picture)
The following code is from the Scripting functoid. Notice that there is no input into the functoid:
Dim MyCounter MyCounter=0 Function MyFunction0( ) MyCounter=MyCounter+1 MyFunction0 = MyCounter End Function
By putting the variable declaration outside of the function, the variable is now declared globally. This variable is available to all functoids in the map.
The other functoid on this map is the Looping functoid. This functoid is linked from both fields in the source specification to the record in the destination specification. It creates multiple output records by iterating over each input record.
The direct links to the Qualifier field are set to copy the name of the element (that is, VendorPartNumber or BuyerPartNumber) instead of the default setting that is the value.
The output of this map is:
When there is more than one record, the first value will be incremented, and either VendorPartNumber or BuyPartNumber will appear in the second field with the value in the last field.
Global variables can be set up in any of the functoids where you create your own functions.
An element group is a set of elements that are related without that relationship being applicable to all children under a common parent (that is, a tagless association). The benefit to using element groups is that you can place ordering constraints on sets of associated elements that provide more flexibility in designing your schemas and document type definitions (DTDs).
For example, if an element A had child elements B, C, D, E, and F, you can specify in your XDR schema that C, D, and E are an element group whose constituents must appear in that order (called "sequence" order). Note that this does not affect sibling nodes B and F.
The BizTalk Server 2002 runtime can support the use of element groups today (given that MSXML DOM supports it), but the XML tools cannot directly support this at design time. However, the following steps offer alternatives that will enable you to use element groups in your schemas:
Important If you are starting with a DTD instead of an XDR schema, you cannot use the first option in step 1 to use element groups.
- BizTalk Mapper now supports opening an XDR schema that did not come from BizTalk Editor, provided it does not contain instance namespaces. To open the non-BizTalk Editor schema, bring the XDR schema with the groups in it into BizTalk Mapper as the source or target, propagate the other side of the map, and then perform step 2.
Alternatively, you can view BizTalk Editor as simply a stepping stone on the way to BizTalk Mapper. For example, assume that element groups are the only unsupported feature of XDR used by a particular schema. To successfully process data against the schema, you import the schema that contains the element groups into BizTalk Editor, at which time the <group> constructs are removed.
In the previous sequence order example, this means element A is still defined with child nodes B, C, D, E, and F. However, the relationship between C, D, and E is lost. The schema is then saved and used as either a source or target document in BizTalk Mapper. Because the group concept is relevant only at validation time (addressed in the following step), BizTalk Mapper needs only to access all the nodes of a particular document. In other words, passing this schema through BizTalk Editor, and then using it in BizTalk Mapper, enables you to define a valid map.
- The schema that contains the element groups can be used directly in BizTalk Messaging Manager when configuring point-to-point document flows. The XDR schema with element groups, used as-is, can represent either the source or target document in a BizTalk Server channel. The result of doing this is that the runtime will use the schema to validate instances. Thus schemas with element groups can be used to both define a map (described in the previous step) and process instances at run time.
In BizTalk Server 2002, you can use BizTalk Editor to declare custom namespaces. Usually, these namespaces are used to associate the annotations of a single schema. However, you can use custom namespaces instead to enable run-time validation of document instances against (possibly) multiple schemas representing the different custom namespaces.
BizTalk Editor assumes that all the nodes in a schema, and therefore an associated instance at run time, belong to the same namespace. As a result, you can enable run-time validation of your document instances by treating any element that might contain elements or attributes from other namespaces as "open" content, meaning that the content is not validated.