Using XML Schemas in BizTalk Server 2004

Overview

Schemas are required for most BizTalk Server applications because they are used to process and validate message information. This document covers the basic concepts of XML schemas and explains how they are used in Microsoft® BizTalk® Server 2004.

This document consists of five parts:

Introduction to XML and Schemas. A conceptual overview of XML and schemas.
Processing and Validating Data. A tutorial on how BizTalk Server uses schemas to process data by promoting properties and using the mapper, including an explanation of how to validate schemas and data.
Creating Schemas. A discussion of how to use schema editor extensions with BizTalk Editor. Covers details of encoding for use with external editors.
Flat File Extensions for Schemas. A conceptual overview of positional and delimited flat files. This is followed by an extensive walkthrough that demonstrates all the steps necessary to use BizTalk Server to process a positional flat file.
Using Processing Instructions with Microsoft InfoPath®. A brief discussion of how to add information to a schema so that InfoPath can read files generated by BizTalk Server.

Links to further information about XML and schemas are included at the end of this document.

Part 1: Introduction to XML and Schemas

BizTalk schemas are documents that define the structure of the XML (eXtended Markup Language) data in BizTalk messages, and their purpose is to create templates for processing and validating XML messages. To create BizTalk schemas, you need a basic understanding of XML. The first part of this document outlines the basic concepts of XML and then discusses how schemas are used as template definitions for XML messages.

What Is XML?

Before the creation of XML, markup languages such as HTML were primarily used to define how text is displayed on a computer screen or printed page. XML can do much more than any previous markup language. Soon after XML was created, people realized that instead of just marking up text, they could use XML as a way to structure and identify information in new ways. Previous database formats only contained raw data, but XML revolutionized the database world by making it possible to embed a description with each portion of the data. Having the description accompanying the data made it possible to reorganize that information into new categories more easily.

XML provides a way to structure data in a flexible and efficient manner. XML quickly became popular because it enabled people to create documents that included both data and the definition of that data.

XML structures and identifies information by using markup codes to enclose fields of data in a hierarchical format. XML uses a handful of standardized symbols to set off information. A typical inventory file written in XML might look like this:

<Chair>
  <Name>Straight Back</Name>
  <Number>040754</Number>
  <Price>79.99</Price>
</Chair>

In this example, the information about a particular chair is contained in XML markup codes that define the name, number, and price. The words "Chair", "Name", "Number", and "Price" are surrounded by the standard XML code characters "<", ">", and "/". These standard codes are used to mark up the data so that a database program can read and write the information efficiently.

Each set of enclosing codes, and the word inside, is called an element. These elements surround and define the corresponding data. In the previous example, the cost of the chair, 79.99, is surrounded by the XML elements of "<Price>" and "</Price>". This combination of XML elements and enclosed data is the basic format for all XML documents, and provides an easy-to-read and easy-to-program way to store the description of the data together with the data itself.

What Are Schemas?

XML not only structures and identifies information with standardized markup codes, but also has the ability to use schemas. A schema is an XML document that works like a dictionary and is used as a reference by other XML documents. The schema code defines the spelling of XML elements and the type of data enclosed by those elements. Using schemas provides an easy way for a program to process XML documents and ensures that the structure and type of information is correct.

In the previous chair inventory example, an XML schema document would need to be created in order to define the spelling and data type of each element. For example, it would define the spelling of "Chair," "Name," "Number," and "Price." The schema would not only include the exact spelling of these words, but would also define the requirements of the data in each element. In the case of the price, the schema would require that the price be a number, and not a text string.

A typical XML schema for the chair inventory example would look like this:

<schema xmlns="http://www.w3.org/2001/XMLSchema">
  <element name="Chair" type="chairType">
    <complexType name="chairType">
      <sequence>
       <element name="Name" type="text">
       <element name="Number" type="text">
       <element name="Price" type="decimal">
      </sequence>
    </complexType>
  </element>
</schema>

This chair inventory schema creates code that corresponds to each element of the XML document that the schema references. It defines the spelling and content of each element. For example, in the chair inventory example, the price is defined with this line:

<Price>79.99</Price>

The chair inventory schema uses the following line to make sure that the element called "Price" is spelled correctly and that the data type of the information is "decimal".

<element name="Price" type="decimal">

If an XML document had the value of "Free" for the price instead of a decimal number, an error would be generated when the document was processed with the schema.

This is the basic principle of how schemas are structured. "Part 3: Creating Schemas" provides more detail about how to build schemas.

Part 2: Processing and Validating Data

BizTalk Server uses XML schemas to perform two major functions:

Processing data by copying it and performing actions on the contents of the data, by promoting properties and using the mapper.
Testing data to ensure that it is valid and accurate.

Using Schemas to Process Data

BizTalk schemas are most commonly used to alter data in a message and route it to a new destination. BizTalk Server can take data from an incoming message and copy it to a new message, or use the data to make a decision about further action. All of this data processing requires the use of XML schemas in order to read and write the data accurately.

When BizTalk Server receives messages from one computer system, it uses schemas to convert the message into the format required for a different computer system. A source schema tests the data in the incoming message and uses a destination schema to transform it to the appropriate data in the outgoing message. For example, you may have messages coming in from a furniture store requesting a chair from the warehouse. The format of the incoming message from the store may look like this:

<Chair>
  <ID>040754</ID>
  <Price>59.52</Price>
  <Color>Red</Color>
</Chair>

However, if the computer at the warehouse uses a different message format, it may be expecting a message that looks like this:

<Chair>
  <ID>040754</ID>
  <Hue>Red</Hue>
</Chair>

If you send the message from the store directly to the warehouse, the warehouse computer won't be able to read the message correctly, because the data fields won't match. The data fields won't match because the data in the incoming message has three sections (ID, Price, Color), but the outgoing message has only two (ID, Hue).

You can use BizTalk Server to solve this type of problem by defining a transformation relationship between a field in one message and a field in another. You define this relationship by using BizTalk Mapper, a visual tool that you use to drag and drop the relationship criteria into place.

After you have created the relationship by using the editor and the mapper, BizTalk Server provides many different ways to copy, transform, and route the data by using pipelines and orchestrations.

Using the Mapper

BizTalk Mapper uses a visual drag-and-drop user interface to define transformational relationships between fields in one message and fields in another message. After you have promoted the properties you wish to use, you need to create a map file that outlines the relationships.

To create a map file using the BizTalk Mapper

In Solution Explorer, right-click the solution name, point to Add, and then click Add New Item.
In the Add New Item dialog box, click the Map Files folder.
In the Name box, type a unique name for the map file, and then click Open.

The following figure shows the Add New Item dialog box.

Figure 1 Add New Item dialog box

You can display the contents of the new map file by double-clicking the map file name in Solution Explorer. The following figure shows a blank map.

Figure 2 Blank map file

Click Open Source Schema and Open Destination Schema to add schemas to the map. After you have inserted the schemas, you can drag the mouse pointer from the properties in one schema to the properties in another. When you release the mouse, a line appears showing the link between the two properties. The following figure shows a sample map with the relationships drawn in.

Figure 3 Sample map file

Using XML schemas and relationship mapping enables BizTalk pipelines and orchestrations to receive, transform, and send messages from one computer system to another.

Note
When you build a BizTalk solution and do not add a map, BizTalk Server may create a default map for you and define default relationships between incoming and outgoing message fields. If you are not aware of this, you may get unexpected results. For this reason, it is recommended that for most solutions, you use BizTalk Mapper to create a map and define the relationships.

Promoting Properties in Schemas

A schema can have many elements, but your application may only require that you use a few of them for your data processing. To save computer resources, BizTalk Server doesn't automatically read each schema element. If you want BizTalk Server to read data from a specific element, you must identify that element by using BizTalk Editor to promote its properties.

Use the following steps to promote a property that you wish to identify.

To promote a property

Open the desired schema in BizTalk Editor.
Select the schema field node you want to promote.
Right-click the field node and then click Promote.
Click Show Promotions.
In the Promote Properties dialog box, click Add to add the field to the promoted properties list.

The following figure shows the Promote Properties dialog box with a field named PetType.

Figure 4 Promote Properties dialog box

When you promote a property, you add code to the schema. In the preceding example, when you promote the PetType field, the following code will be added to the root node of the schema:

<b:properties>
  <b:property distinguished="true" xpath="/*[local-name()='Pets' and namespace-uri()='http://ffdelrec.ffdelrec']
/*[local-name()='StudentPets' and namespace-uri()='']/*[local-name()='PetType' and namespace-uri()='']" /> 
</b:properties>

After a property is promoted, BizTalk Server has the information it needs to process the property with the mapper and to create a transformation relationship.

For more information about BizTalk Mapper and about promoting properties, see BizTalk Server 2004 Help.

Validating Schemas and Data

When you create a schema, the structure of the schema may not be correct. BizTalk Server provides a way to test the accuracy of the schema to make sure that it is internally consistent and valid.

To validate a schema for accuracy, add the schema to the BizTalk project and right-click it in Solution Explorer. One of the options will be to validate the schema. Select this option and BizTalk Server will run tests on the schema to make sure that it is constructed properly. If there are any errors, they will be displayed in the output window.

To validate data in an XML message, you can use schemas embedded in a custom pipeline. See BizTalk Server 2004 Help for more information about advanced types of data validation.

Part 3: Creating Schemas

The best way to create schemas is to use BizTalk Editor.

Using BizTalk Editor

BizTalk Editor makes it easy to create schemas because instead of typing commands and hoping you don't make a typing error, you can use menus to add elements to your schema. As each element is added, BizTalk Server makes sure that it is valid, preventing syntax errors.

To create a schema using BizTalk Editor

In Solution Explorer, right-click the solution name, point to Add, and then click Add New Item.
In the left pane of the Add New Item dialog box, click Schema Files, and then click the Schema object in the right pane.
In the Name box, type a unique name for the schema, and then click Open.

The following figure shows the Add New Item dialog box for a new schema.

Figure 5 Add New Item dialog box

After you have added your new schema, you can view it in BizTalk Editor by double-clicking the schema name in Solution Explorer. The following figure shows a newly created schema.

Figure 6 Newly created schema

When you create a schema in BizTalk Server, the editor displays the following lines of code:

<?xml version="1.0" encoding="utf-16" ?> 
<xs:schema xmlns="http://chair.ChairSchema" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" 
targetNamespace="http://chair.ChairSchema" 
xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="Root">
    <xs:complexType /> 
  </xs:element>
</xs:schema>

This default schema has four standard parts. They are:

Declaration
Schema node
Root node
Root type

Schema Declaration

The first part of the default schema is the declaration:

<?xml version="1.0" encoding="utf-16" ?>

Every schema has this type of declaration as the first line. The standard XML codes "<?" and "?>"indicate that this is an XML schema. The version number (1.0) is defined and so is the encoding. BizTalk Server prefers UTF-16 encoding. See "Using External Schemas" later in this document for more information about encoding.

Schema Node

The second part of the schema consists of these two lines:

<xs:schema xmlns="http://chair.ChairSchema" 
xmlns:b="http://schemas.microsoft.com/BizTalk/2003" 
targetNamespace="http://chair.ChairSchema" 
xmlns:xs="http://www.w3.org/2001/XMLSchema">

</xs:schema>

The first line, which takes up four lines on the page, defines the basic schema information. All BizTalk Server-generated schema parts are prefixed with "xs:", which stands for XML schema. Note that this is all one logical line that begins with "<xs:schema" and ends with ">". This part of the schema definition defines the basic namespaces of the schema. It is called the schema node because it encloses all other nodes.

Namespaces are used to make sure that components in your schema will not be confused with components of the same name in another schema. For example, you may have two schemas that both have a field named "customer". If you don't put them in different namespaces, BizTalk Server won't know which customer you are referring to.

The following table shows the namespaces that BizTalk Server generates.

Table 1 BizTalk Server namespaces

Namespace type	Element name	Definition
Default namespace	xmlns	This is called the default namespace and uses a name that is generated from the solution and schema names. Even though this is prefixed with http:// it doesn't need to point to an actual Web location. This is the namespace that will be used if no other namespaces are defined.
BizTalk namespace	xmlns:b	This points to the definition of an additional schema namespace that BizTalk Server uses for specific purposes. You can use components that share this common BizTalk namespace.
Target namespace	targetNamespace	This sets the target namespace, which is the namespace that will be used in your document. The target namespace overrides the default namespace, but by making the two namespaces the same, you avoid any potential namespace overlap.
Standard schema namespace	Xmlns:xs	This is the namespace that defines standard XML namespace elements. If you don't use this as the default namespace, you must put "xs:" before each standard element. This makes sure that all non-standard elements are defined by your schema.

The second line of the schema node:

</xs:schema>

is the closing tag. All the other nodes of the schema are enclosed by these two tags. As with all XML code, you must make sure that you close all opening tags. Additional properties of the schema node are defined in the BizTalk Server Programmer's Reference.

Schema Root Node

The third part of the schema is called the root node. BizTalk Server creates a default root node with the following two lines:

<xs:element name="Root">

</xs:element>

The root node is simply the top node of the XML schema tree. Additional properties of the root node are defined in the BizTalk Server Programmer's Reference.

Schema Root Type

The final part of the schema added by BizTalk Server is:

<xs:complexType />

This line declares the data type of the root as complexType. The definition of what complexType will consist of will be defined later when you add more nodes using the BizTalk editor..

After you have finished creating a schema, you can edit or add new items to any node by right-clicking the node name and selecting options from the menu. See BizTalk Server 2004 Help for more information about using BizTalk Editor.

Using External Schemas

BizTalk Server allows you to import externally created schemas. If you use a schema created outside BizTalk Server, the schema data must be stored in a text file in the proper format. The format formula for editing and storing text file characters is called encoding. If you edit your schemas in Microsoft Notepad, the encoding will be compatible with BizTalk Server. However, if you use another text editor, you must make sure that your editor saves text data using the following encoding criteria:

The file must be saved in UTF-16 format. UTF-16 provides the capability of storing text characters for many different written languages. For example, you can store not only English or French in UTF-16, but you can also store Chinese, Russian, or Hebrew characters.
In addition, the file must be saved with "little-endian" encoding. This refers to the order of the bytes that make up each 16-bit character.
Finally, the file header must have the byte-order mark (BOM).

If you are creating XML data files, BizTalk Server converts the encodings to UTF-8. Note that because both UTF-8 and UTF-16 are commonly known as Unicode, you must make sure that you know which of the two encodings you are using.

Note
If you attempt to use a UTF-8 schema that follows all other requirements, BizTalk Server will convert it to UTF-16.

If you would like to know more about the UTF and BOM standards, the Unicode Web site has a useful FAQ at http://go.microsoft.com/fwlink/?LinkId=41784.

Part 4: Flat File Extensions for Schemas

Even though XML is quickly becoming a popular format for storing message data, many computer systems still use a "flat file" format that corresponds to standard database file storage formats. Flat files have structures that were designed many years ago, more for the convenience of computers than for humans.

XML uses a hierarchical structure for storing information, but flat files store their information in a continuous string of bytes. If you look at an XML file, you can easily view the tree-like structure of the data.

Understanding Flat Files

There are two basic types of flat file structures: positional and delimited.

About Positional Flat Files

The positional structure is based on early computer punch cards, where the meaning of the data was based on the columns in which the information was stored. For example, a name would be in the first 20 columns and the address in the next 30 columns. Each group of columns corresponded to a field and the set of columns corresponded to a single record. Positional storage was very efficient for electromechanical reading and writing, and worked well enough for slow computer systems of the past.

A typical positional record might look like this:

0123456789012345678901234567890123456789012345678901234567890123456789
FirstName           LastName            Street Address

The first 20 columns are allocated for the first name, the next 20 for the last name, and the remaining 30 columns for the street address. Note that the repeating numbers above the data are not part of the file, but are shown here to help visualize contents of the file in terms of position and length.

About Delimited Flat Files

As computers got faster, a second type of filing system evolved to save storage space, because positional storage can be very wasteful of space. For example, if you have allocated 20 characters for a first name and the name stored is "Laura", you are wasting 15 characters of space. By using a delimited storage, instead of allocating a fixed number of characters for each field, a special character is used to separate fields from each other. The special character, called a delimiter, must not be used inside any field. A typical delimiter might be a comma.

A typical delimited file might look like this, with a comma being used as a delimiter:

0123456789012345678901234567890123456789012345678901234567890123456789
FirstName,LastName,StreetAddress

The delimited file takes up 32 bytes, whereas the positional equivalent took up 70 bytes. Note that the repeating numbers above the data are not part of the file, but are shown here to help visualize contents of the file in terms of position and length.

Delimited files take a bit more processing, but the processing difference is pretty much irrelevant with higher-speed computers. The only danger with using delimited files is that you must make sure that the data in the fields does not contain a delimiter. For example, a street address that contained a comma (for example, "24th St., NW") would be interpreted as two fields, not one.

Developing a Flat File Solution

BizTalk Server can receive flat files and transform them into XML messages for processing. Conversely, BizTalk Server can take an internal XML message and convert it to a flat file that can be sent as an outgoing message. However, to transform flat files, you must use specific schema annotations known as flat file extensions. The following sections show you how to develop a flat file solution. The solution uses the BizTalk Server flat file extensions to receive a positional flat file and transform it into an XML file.

There are many different ways to develop a BizTalk solution, but this document will use the following four-step process:

Set up a solution. You must create necessary folders on your hard disk, build a container assembly to hold the code, and create a project in Microsoft Visual Studio® using the BizTalk Framework. You also need to create an instance of the flat file you will use to test the solution.
Create a schema. You must create a schema that defines the transformations that your messages will undergo as they enter and leave BizTalk Server.
Write the code. Usually your solution will require some code to transform or route the incoming and outgoing messages. You can create this code by using the graphical editors in BizTalk Server and you can add C# code to specific objects. You will then compile the code into the container assembly you created in step 1, and, if there are no compilation errors, you will deploy the assembly.
Configure and test the solution. The final step is to configure your send and receive ports and locations and then test your solution.

Example: Receiving a Positional Flat File

The following example uses the four-step process described earlier to develop a BizTalk solution that receives a positional flat file message and uses a flat file extension to transform the data into an XML message. The name of the solution is FFPosRec.

Before you begin, be sure that you have BizTalk Server 2004 set up on a single computer. It is suggested that you run other solutions, such as the "Hello, World" solution in the BizTalk Server SDK, to ensure that your BizTalk Server installation works properly.

Step 1: Set Up the FFPosRec Solution

You must set up your BizTalk solution every time you create a new solution. Each time, you need to create new folders, build a container assembly, and create a new project. For this example, the name of the solution is FFPosRec which stands for Flat File Positional Receive.

Creating the FFPosRec Folders

First, set up a series of folders. Begin by setting up one folder that will contain all the other folders. For this example, call it FFPosRec. Then, inside this folder, create three folders, called In, Out, and Original.

The In folder is the receive location for your solution. When the solution is created, you will drop a flat file in this location for processing.

The Out folder is the send location for your solution. After your file is received and transformed, a new XML file will be created in this location.

The Original folder is not required, but is suggested as a safety procedure. Because BizTalk Server consumes any flat file you drop into the In folder, you need to make sure that you save an original so you can test it again. Put the original flat file you create in this folder and copy it to the root of the FFPosRec folder.

Your folder structure should look like this:

C:\FFPosRec

C:\FFPosRec\In

C:\FFPosRec\Out

C:\FFPosRec\Original

After you have created the folders, make sure that the user of the computer has permission to use the folders. The instance of BizTalk Server that you are running must have read and write permission for all the folders.

Creating the FFPosRec Assembly

You must create a .NET strong-named key assembly to contain the code for your solution. Creating a strong-named key assembly guarantees that your solution will be unique and will not collide with the namespaces of other .NET assemblies. A strong name consists of the assembly name, version, culture (if available), public key, and digital signature. For more information about strong-named key assemblies, see the Microsoft .NET Framework Developer's Guide.

When you install Visual Studio .NET 2003, a command prompt shortcut is created in the Visual Studio .NET Tools folder that you can select from the Start menu. Type the following in the command prompt window:

sn –k "C:\FFPosRec\FFPosRec.snk"

This runs the strong-named key file generator program with the "k" switch, which generates the new file with the name you provide. Be sure to type the quotes to include the entire path information, and to specify the file extension as .snk.

Creating the FFPosRec BizTalk Project

You are now ready to create a new BizTalk project.

To create a new BizTalk project

Start Microsoft Visual Studio .NET 2003.
On the File menu, point to New, and then click Project.

In the New Project dialog box, in the Project Types area, select BizTalk Projects, and in the Templates area, select Empty BizTalk Server Project.

Note
Make sure you only click on the project once, so you don't accidentally create a file with the default project name. One of the most common errors in creating BizTalk solutions is to accidentally choose default names for projects and parts of projects. BizTalk Server keeps track of every name you select, and if you have two ports with a default name, BizTalk Server won't know which port you want.

In the Empty BizTalk Server Project dialog box, in the Project Name box, type FFPosRec (or anything else you want as long as it is unique), and fill in the path to point to the location of the FFPosRec folder you created earlier.

Your project is now created and you are almost finished with the first step.

Connecting the FFPosRec Project to the Assembly

You must tell BizTalk Server where your strong-named assembly is stored on your computer.

To connect the project to the assembly

In Solution Explorer, right-click the project name, and then click Properties.
In the Property page, expand Common Properties, and then click Assembly.
In the right pane, scroll down to the Strong Name section, click the field to the right of Assembly Key File, and then click the ellipses.
In the Assembly Key File page, browse to find the .snk file that you created earlier, select it, and then click Open.
The Assembly Key File page closes and the path to the strong name file appears in the field.
Click OK to close the Property page.

You have just created the basic framework necessary to hold your solution. The next step is to create your schema.

Step 2: Create the FFPosRec Schema

The schema defines what data comes in and what data goes out. This information must be completely defined before any further processing can take place. In most cases, it makes sense to create the schema before you create the rest of the solution. But because one person may design the schema and another may do the actual programming, these tasks may be done in the opposite order.

FFPosRec Schema Requirements

When you work with flat files, you need to know the exact requirements for your data. In other words, you must know the length of your record, how many fields it contains, and the length of each field.

This example uses one record with two fields. The record stores the name of a student at a school and the type of pet that the student owns. The following table shows a list of the data requirements for each field in the record.

Table 2 Data requirements

Field number	Field name	Field offset	Field length
1	StudentName	0	20
2	PetType	0	20

Note that the field offset is measured from the end of the previous field, or, for the first field, from the beginning of the record. Offset and length are measured in bytes or characters.

A typical file that was created to this specification might look like this:

0123456789012345678901234567890123456789012345678901234567890123456789
Harry               Owl

Note that the repeating numbers above the data are not part of the file, but are shown here to help visualize the contents of the file in terms of position and length.

When you create a schema for this example, follow these steps:

Create the schema and add the Flat File Extension to the schema.
Add a positional child record to define the structure for the record.
Add child fields to each record and define the offset and length for each field.

Adding the FFPosRec Schema

Use the procedure outlined earlier in "Part 3: Creating Schemas" to create your new schema. The schema that BizTalk Server generates for this example should have the following text:

<?xml version="1.0" encoding="utf-16" ?> 
- <xs:schema xmlns="http://ffposrec.FFPosRec" 
xmlns:b="http://schemas.microsoft.com/BizTalk/2003" 
targetNamespace="http://ffposrec.FFPosRec" 
xmlns:xs="http://www.w3.org/2001/XMLSchema">
- <xs:element name="Root">
     <xs:complexType /> 
  </xs:element>
  </xs:schema>

The only differences between the FFPosRec schema and the chair inventory schema are in the second and fourth lines, where the namespace is created by combining the solution and schema names: ffposrec.FFPosRec.

Adding the FFPosRec Flat File Extension

Now you must add an extension to the schema to define how to convert the flat file data to XML so that BizTalk Server can process it. Schemas were designed to process XML documents, but by using the Flat File Extension, you can convert flat files into BizTalk Server XML files.

To add the Flat File Extension

In BizTalk Editor, select the schema node in the left pane.
The properties for this node will be displayed on the right.
Scroll down to the Schema Editor Extensions property and then click the input box on the right.
Select the Flat File Extension check box and then click OK.

After you click OK, you will notice that BizTalk Server has added several lines to your schema. BizTalk Server has added two sets of annotations to the schema. Annotations are a standard technology used to extend schemas.

The first newly added annotation code section in the schema is:

<xs:annotation>
- <xs:appinfo>
    <b:schemaInfo count_positions_by_byte="false" standard="Flat File" root_reference="Root" /> 
    <schemaEditorExtension:schemaInfo namespaceAlias="b" 
extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" 
standardName="Flat File" 
xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003
/SchemaEditorExtensions" /> 
  </xs:appinfo>
</xs:annotation>

Note that this first annotation refers to the "b" namespace that was previously defined in the schema node. This namespace contains elements that are unique to BizTalk Server and will define additional namespaces and properties for flat files.

The second annotation is:

<xs:annotation>
- <xs:appinfo>
    <b:recordInfo structure="positional" 
preserve_delimiter_for_empty_data="true" 
suppress_trailing_delimiters="false" sequence_number="1" /> 
  </xs:appinfo>
</xs:annotation>

This also uses the "b" namespace and provides some default information about the record structure for flat files. For more information about Flat File Extension properties, see the BizTalk Server Programmer's Reference.

Naming the FFPosRec Root

This step is not required, but it is good practice to give every schema a meaningful root name.

To name the root node

In BizTalk Editor, right-click the root node in the left pane, and then click Rename.
For the root name, type Pets, and then press ENTER.
Click the root name (now Pets), scroll down in the property pane, and change the Structure property to Positional (the default is Delimited).
This makes sure that all further nodes will be positional.

Adding the FFPosRec Child Record

You must add at least one record for a flat file schema, because a flat file is a collection of one or more records.

To add a child record

Right-click the root node (Pets), point to Insert Schema Node, and then click Child Record.
Rename the record to something meaningful such as "StudentPets".
If you scroll down in the property pane, you will notice that the Structure property is Positional.

When you view the modified schema in the schema editor, you will notice that adding the new record opened up the <xs:complexType/> element and inserted a sequence under the root node with the following code:

<xs:sequence>
- <xs:annotation>
-   <xs:appinfo>
      <b:groupInfo sequence_number="0" /> 
    </xs:appinfo>
  </xs:annotation>
-   <xs:element name="StudentPets">
-     <xs:annotation>
-       <xs:appinfo>
          <b:recordInfo sequence_number="1" structure="positional" 
preserve_delimiter_for_empty_data="true" 
suppress_trailing_delimiters="false" /> 
        </xs:appinfo>
      </xs:annotation>
    <xs:complexType /> 
  </xs:element>
</xs:sequence>

This allows other records to be inserted later and identifies this record as the first one in the sequence. Then it creates an element called "StudentPets" and defines the record as delimited.

Adding the FFPosRec Child Fields

You have now added a record that can contain positional fields. The next step is to add child elements for each field.

To add the child fields

For the first field, right-click the StudentPets node, point to Insert Schema Node, and then click Child Field Element.
Rename the field to something meaningful such as "StudentName".
Scroll down in the property pane and change the Positional Length property to 20 and the Positional Offset property to 0. (Remember that the offset is measured from the beginning of the record for the first field.) You have now generated the following code for this field:
```
<xs:element name="StudentName" type="xs:string">
- <xs:annotation>
-   <xs:appinfo>
      <b:fieldInfo sequence_number="1" justification="left" 
pos_length="20" pos_offset="0" /> 
    </xs:appinfo>
  </xs:annotation>
</xs:element>
```

This defines a field of 20 characters that will contain the name of the student. The field code will look like this:

<xs:element name="PetType" type="xs:string">
- <xs:annotation>
-   <xs:appinfo>
      <b:fieldInfo sequence_number="2" justification="left" 
pos_length="20" pos_offset="0" /> 
    </xs:appinfo>
  </xs:annotation>
</xs:element>

Repeat the preceding procedure (taking care to select the record name, not the field name) to create a field named "PetType" for the type of pet. This field will also have an offset of 0 and a length of 20. (Note that in this case, the offset is from the end of the previous field.) The two fields together make a total of 40 characters.

You have now created a schema for your flat file receive solution.

For more information about additional record and field properties, see the BizTalk Server Programmer's Reference.

Step 3: Write the FFPosRec Code

Most BizTalk solutions require using the BizTalk Server graphical editing tools to write the code and sometimes adding C# code for specific objects. Because the FFPosRec solution does not involve complex routing or decision-making, Orchestration Designer was not used to create this solution. However, because this solution involves converting the document format from flat file to XML, you must create a custom BizTalk Server pipeline to create the code necessary to process incoming flat file messages.

BizTalk Server pipelines process messages in stages. The FFPosRec solution uses a custom pipeline to create a stage that disassembles the message. Disassembling converts the flat-file message into the native XML format that BizTalk Server uses.

Adding the FFPosRec Pipeline

To add a pipeline to the FFPosRec solution

In Solution Explorer, right-click your solution name, point to Add, and then click Add New Item.
In the Add New Item dialog box, click Pipeline Files in the left pane and then click the Receive Pipeline object in the right pane. Make sure not to double-click, so you can give the pipeline a unique name.
For this example, name the new pipeline FFPosRec.btp.
In Solution Explorer, select the new pipeline you have created and make sure you can see Pipeline Designer. It should look something like the following figure.
Figure 7 Pipeline Designer

You may have to move some of the windows in Visual Studio to see both parts of the designer. When you view the pipeline in Pipeline Designer, you will see on the left a list of components and on the right the visual representation of the pipeline stages. The four stages for a receive pipeline are: decode, disassemble, validate, and resolve party (not shown). A send pipeline has other stages.
Select the Flat file disassembler component in the left pane (the BizTalk Pipeline Components portion of the toolbox) and drag it to the box that says Drop Here! under the Disassemble portion of Pipeline Designer.
When you release the button, the box should read Flat file disassembler.

Connecting the FFPosRec Flat File Schema

You have created a custom receive pipeline with a disassembler stage. Now you must tell BizTalk Server which schema you want to use for the disassembly. You will use the schema you created in step 2, the one with the Flat File Extension.

To connect the pipeline component to the schema

Select the box that you dragged the component to, which should now be labeled Flat File disassembler.
On the lower right, you should see a property page labeled Flat file disassembler, which contains a property called Document schema with a value of (None).
Click in the value box for that property and select the name of the schema you created. You may have to scroll down past several other schemas that come installed with BizTalk Server.

You have now connected your pipeline to your schema, and your custom pipeline is ready to be added to your solution.

Building the FFPosRec Assembly

You have created a schema with flat-file extension and you also have created a custom pipeline with a flat-file disassembly stage. The pipeline component is connected to the proper schema. Now you must combine the schema and the pipeline component into one package that BizTalk Server can use to implement the solution. The schema and pipeline information will be added to the .NET assembly you created in step 1.

To build the solution

Save your project.
In Solution Explorer, right-click your solution name, and then click Build Solution or Rebuild Solution.

A few moments later, the results of your build will appear in the Output window of Visual Studio. If the build had no errors, the results should read:

---------------------- Done ----------------------

    Build: 1 succeeded, 0 failed, 0 skipped

The most common error in this part of the process is not filling in the offset and length of the positional fields. Another common error is not giving all components a unique name. A third error may occur if you need to revise your schema. Because of caching issues, before you can successfully rebuild, you may need to delete your custom pipeline and make sure that the revised pipeline is included when you rebuild. Any time you alter a pipeline, you may need to rebuild your assembly.

Deploying the FFPosRec Assembly

You must perform one more step before you can test your solution. Even though you have created an assembly containing your code, you must deploy it. The assembly is sitting on your computer, but to share your assembly, it must reside in the global assembly cache so that it is accessible to the .NET Framework. BizTalk Server only uses the assembly version that you have deployed, so you must redeploy your assembly every time you rebuild it.

To deploy the assembly

Save your solution.
In Solution Explorer, right-click your solution name, and then click Deploy.

A few moments later, the results of your deployment will appear in the Output window of Visual Studio. If the deployment had no errors, the results should read:

---------------------- Done ----------------------

    Build: 1 succeeded, 0 failed, 0 skipped
    Deploy: 1 succeeded, 0 failed, 0 skipped

Note that BizTalk Server first builds your assembly one last time and then deploys it.

To make sure that your assembly is properly deployed, you can look at the deployment of all BizTalk Server assemblies on your computer by opening BizTalk Explorer in Visual Studio.

To view BizTalk Server assemblies

In Visual Studio, on the View menu, click BizTalk Explorer.
In BizTalk Explorer, right-click the Configuration database, which will have a name similar to:
```
<YOURMACHINENAME>.BizTalkMgmtDb.dbo
```
Click Refresh to make sure that you are always looking at the latest view of the BizTalk Server components that are installed on your computer.
Expand the tree under the database to see the objects listed under Assemblies. Your new assembly should be listed there with the version number.

The most common error in deployment is that you may already have an assembly deployed with the same characteristics. This will happen if you try to deploy the same assembly after you have already deployed it. If you try to redeploy without undeploying, you'll get an error saying that the assembly is already deployed.

To undeploy an assembly

In Visual Studio, on the View menu, click BizTalk Explorer.
In BizTalk Explorer, expand Assemblies, right-click the assembly and then click Undeploy.
Right-click the Configuration database name (as in the earlier procedure) and then click Refresh to make sure that your assembly is really undeployed.
Occasionally you may need to start and stop BizTalk Server itself to undeploy a solution completely.

The following sequence will be useful when you are going through an iterative build cycle with BizTalk Server:

Build your solution.
Deploy your solution.
Configure and test.
If there is a problem, you must:
1. Undeploy the assembly.
2. Fix the problem.
3. Compensate for changes (for example, a, new schema requires a new pipeline).
4. Rebuild your solution.
5. Redeploy your solution.
6. Configure and test
7. Repeat steps a, b, c, d, e, and f as often as needed.

Step 4: Configure the FFPosRec Solution

You have now created, built, and deployed a BizTalk schema and a custom pipeline. Your solution for translating flat files to XML is almost finished. All that is left to do is hook up the send and receive ports, and when finished with that, test your solution. This involves following a set of steps in the correct order.

Creating the FFPosRes Receive Port

The first thing you must do to configure your solution is to create a receive port. This will tell BizTalk Server how you want to receive a flat file, but not where.

To create a BizTalk Server receive port

In Visual Studio, on the View menu, click BizTalk Explorer.
In BizTalk Explorer, right-click Receive Ports, and then click Add Receive Port.
In the Create New Receive Port dialog box, select One-Way Port and then click OK.
You do not need to fill in any properties in the next dialog box, except that you should give your receive port a unique name. Call your receive port FFPosRec_RP, where RP stands for receive port.
It is a good idea to name your ports and locations with a systematic but unique naming system. Any names can be used, but your names should be unique and meaningful.
When you have named your receive port, click OK to close the dialog box.

Creating the FFPosRes Receive Location

In the last section you created a receive port that told BizTalk Server how you want to receive your file, but not where. You must now specify a location where the file will be dropped so that BizTalk Server can receive it.

To create a BizTalk Server receive location

In Visual Studio, on the View menu, click BizTalk Explorer.
In BizTalk Explorer, find the receive port you just created and click the Explorer node to the right of it to view the receive locations for that port. Right-click the Receive Locations folder and then click Add Receive Locations.
In the Receive Location Properties dialog box, follow these steps:
1. In the Name box, type FFPosRec_RL. (The "RL" addition is not required, but will help you to keep this separate from the receive port and the send port.)
2. For Transport Type, select FILE.
3. For Address (URI), click inside the Address (URI) box to display another dialog box.
4. In the Receive Folder box, type the path to the folder you want to receive the file in. To match the folder set you created in step 1, type C:\FFPosRec\In\
5. In the File Mask box, type *.txt
  This is because you will be dropping a text file (.txt extension) into the folder, and not a file with the .xml extension.
6. Click OK to close that dialog box and return to the Receive Location dialog box.
For Receive Handler, select BizTalkServerApplication.
For Receive Pipeline, select the name of the custom receive pipeline you created.
Check over this dialog box carefully. There are two common errors that occur at this step, not counting the error that would occur if you don't have a unique name for the receive location. The first common error is to keep the default file mask of *.xml. You need to use *.txt for flat files. The second common error happens if you undeploy your assembly but do not rebuild the pipeline again: the receive location property for Receive Pipeline will not have your custom pipeline on the pipeline list.
After you have filled in all the properties and made sure everything is correct, click OK to close the dialog box.

You have now finished creating the receive ports and locations.

Creating the FFPosRes Send Port

You must configure the send port so that BizTalk Server can send out the XML file that it created after disassembling the flat file. BizTalk Server uses the XML format as the default for sending and receiving files, so this step will be much easier than configuring the receive port. Also, unlike for the receive port, you do not need to go through an extra step to specify the send location.

To create a BizTalk Server send port

In Visual Studio, on the View menu, click BizTalk Explorer.
Iin BizTalk Explorer, right-click Send Ports and then click Add Send Port
In the Create New Send Port dialog box, select One-Way Port and then click OK.
In the Send Port Properties dialog box, open the Transport folder and select Primary.
For Transport Type, select FILE.
For Address (URI), click the box to display another dialog box.
In the Send Folder box, type the path to the folder you want to send the file from. To match the folder set you created in step 1, type C:\FFPosRec\Out\
The default file name is:
This will name your message with the message ID generated for it and give it an .xml extension. This will make sure that each time you create a new file, it will have a new name.
Do not close the dialog box at this point! Instead, click the Send option in the left pane. Then change the value of the Send Pipeline property by clicking on the box to choose the DefaultPipelines.PassThroughTransmit pipeline. This will make sure that BizTalk Server sends out the default values for an XML message.
Click the Filters and Maps node on the left. Click Filters to display the Filter Expression dialog box. Then enter the expression described in the next steps.
On the left side of the expression, click the drop-down list and scroll until you find:
```
BTS.ReceivePortName
```
This may require carefully clicking and scrolling. After you have found BTS.ReceivePortName, click it to select it in the Filter Expression editor.
Next, in the same editor, click the box to the right of the box you just filled in, type the name of your receive port, and then press ENTER.
Your expression should now look like this:
```
BTS.ReceivePortName == FFPosRec_RP Add
```
You can ignore the "Add" at the end of the line, but you must have the rest of the expression filled in so that the send port knows the receive location where the file came from. Because we are not using the Mapper, Functoids, or Orchestration, this step is necessary.
You are now finished with this dialog box, and can click OK to close it. You might want to check it carefully for errors. Possible errors include not giving the send port a unique name, not adding the passthrough pipeline, and not setting up your filter properly.

Enabling the FFPosRes Receive Location

To enable a BizTalk Server receive location

In Visual Studio, on the View menu, click BizTalk Explorer.
In BizTalk Explorer, refresh the object list and find your way to the receive location you created. Right-click it and then click Enable.

Your receive location is now ready to receive.

Starting the FFPosRes Send Port

To start a BizTalk Server send port

In Visual Studio, on the View menu, click BizTalk Explorer.
In BizTalk Explorer, refresh the object list and find your way to the send port you created. Right-click it and then click Enlist.
Right-click the send port again and this time click Start.
Right-click it one more time and make sure that the menu has the Unenlist and Stop options, which tell you that the send port is enlisted and started.

Testing Your Solution

Drop a flat file into the In folder. After a moment an XML file should appear in the Out folder. If a new XML file does not appear, use the BizTalk Administration console to see what may have gone wrong. Usually the Event Viewer will give you a clue whether the problem is in the send port, receive ports, pipeline, or schema.

Using Delimited Flat Files

Delimited flat files are similar to positional flat files except that they use delimiters between each field. A typical delimited flat file is the common CSV (Comma Separated Version) file that Microsoft Excel generates if you choose the CSV output format. Follow the same procedure that was shown in the FFPosRec solution, but choose delimited values and specify the delimiter.

The following figure shows an Excel file with one row and two columns, representing output from a database of student pets. The first column represents a student name and the second represents a pet type.

Figure 8 Excel data file

You must choose the Excel Save As option and select the CSV option, to save your file in CSV (comma delimited) format. If you view the file in a text editor, it will look like this:

Harry,Owl

You can process this with BizTalk Server by using the same procedure provided in the FFPosRec solution for positional flat files, but you must set the appropriate properties for comma-delimited files.

The following schema can be used to process a CSV flat file that follows the data saved from the Excel file above:

<?xml version="1.0" encoding="utf-16" ?>
<xs:schema xmlns="http://ffdelrec.ffdelrec"   xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://ffdelrec.ffdelrec" xmlns:xs="http://www.w3.org/2001/XMLSchema">
   <xs:annotation>
      <xs:appinfo>
         <b:schemaInfo count_positions_by_byte="false"
           standard="Flat File" root_reference="Pets"
           schema_type="document" />
         <schemaEditorExtension:schemaInfo namespaceAlias="b"
         extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension"
         standardName="Flat File"
         xmlns:schemaEditorExtension=
         "http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
      </xs:appinfo>
   </xs:annotation>
   <xs:element name="Pets">
      <xs:annotation>
         <xs:appinfo>
            <b:recordInfo structure="delimited" 
            preserve_delimiter_for_empty_data="true"
            suppress_trailing_delimiters="false" sequence_number="1" />
         </xs:appinfo>
      </xs:annotation>
      <xs:complexType>
         <xs:sequence>
            <xs:annotation>
               <xs:appinfo>
                  <b:groupInfo sequence_number="0" />
               </xs:appinfo>
            </xs:annotation>
            <xs:element name="StudentPets">
               <xs:annotation>
                  <xs:appinfo>
                     <b:recordInfo sequence_number="1"
                       structure="delimited"
                       preserve_delimiter_for_empty_data="true"
                       suppress_trailing_delimiters="false"
                       child_delimiter_type="char"
                       child_delimiter="," />
                  </xs:appinfo>
               </xs:annotation>
               <xs:complexType>
                  <xs:sequence>
                     <xs:annotation>
                        <xs:appinfo>
                           <b:groupInfo sequence_number="0" />
                        </xs:appinfo>
                     </xs:annotation>
                     <xs:element name="StudentName" type="xs:string">
                        <xs:annotation>
                           <xs:appinfo>
                              <b:fieldInfo sequence_number="1"
                              justification="left" />
                           </xs:appinfo>
                        </xs:annotation>
                     </xs:element>
                     <xs:element name="PetType" type="xs:string">
                        <xs:annotation>
                           <xs:appinfo>
                              <b:fieldInfo sequence_number="2"
                              justification="left" />
                           </xs:appinfo>
                        </xs:annotation>
                     </xs:element>
                  </xs:sequence>
               </xs:complexType>
            </xs:element>
         </xs:sequence>
      </xs:complexType>
   </xs:element>
</xs:schema>

This schema can be used to generate the following XML file:

<?xml version="1.0" encoding="utf-8" ?> 
<Pets xmlns="http://ffdelrec.ffdelrec">
  <StudentPets>
    <StudentName>Harry</StudentName> 
    <PetType>Owl</PetType> 
  </StudentPets>
</Pets>

Part 5: Using Processing Instructions with InfoPath

You can use Microsoft InfoPath in conjunction with BizTalk Server to display XML messages sent from BizTalk Server. If you are using InfoPath to display XML data generated by BizTalk Server, you must use the Processing instruction property of the message context for InfoPath to display the data correctly.

When an XML file is created or edited using InfoPath, InfoPath creates a processing instruction at the beginning of that XML file, which indicates that the document should be edited with InfoPath. A processing instruction is part of the XML standard and does not interfere with the schema on which the XML file may be based. If you do not include the processing instruction, another XML editor may be the default XML editor and InfoPath will not be able to read or edit your document.

To determine the processing instruction data, you should create a form in InfoPath using the schema you will be using in BizTalk Server. Then publish the form and save it. The processing instruction will be part of the form.

The following links discuss processing instruction information in greater detail.

http://go.microsoft.com/fwlink/?LinkId=41810

http://go.microsoft.com/fwlink/?LinkId=41815

For more information about processing instructions, see the InfoPath documentation.

Going Further with XML and Schemas

The following books provide useful information about XML and schemas:

Stanek, William R. XML Pocket Consultant. Redmond, WA: Microsoft Press, 2002.
Bradley, Neil. The XML Schema Companion. Addison-Wesley, 2004.
Daum, Berthold. Modeling Business Objects with XML Schema. Morgan Kaufmann, 2003.
Harold, Elliotte Rusty, and W. Scott Means. XML in a Nutshell, 3rd Edition. O'Reilly, 2004.
Fitzgerald, Michael. XML Hacks. O'Reilly, 2004.
Lenz, Evan, Mary McRae, and Simon St. Laurent. Office 2003 XML. O'Reilly, 2004.

The following Web sites provide useful information about XML and schemas:

W3C Schema Specification: http://go.microsoft.com/fwlink/?LinkId=41989
Microsoft Office 2003 Reference Schemas: http://go.microsoft.com/fwlink/?LinkId=41991
Introduction to XML Schemas: http://go.microsoft.com/fwlink/?LinkId=41992
XML Schema Tutorial: http://go.microsoft.com/fwlink/?LinkId=41993
XML in 10 points: http://go.microsoft.com/fwlink/?LinkId=41994
Namespaces in XML: http://go.microsoft.com/fwlink/?LinkId=41995
Schema Primer: http://go.microsoft.com/fwlink/?LinkId=41996
Schema Best Practices: http://go.microsoft.com/fwlink/?LinkId=41997
XML History and Introduction: http://go.microsoft.com/fwlink/?LinkId=41998
MSDN XML Developer Center: http://go.microsoft.com/fwlink/?LinkId=41999