Was this page helpful?
Your feedback about this content is important. Let us know what you think.
Additional feedback?
1500 characters remaining
Export (0) Print
Expand All

Run-time Methods of a Data Flow Component

 

At run time, the data flow task examines the sequence of components, prepares an execution plan, and manages a pool of worker threads that execute the work plan. The task loads rows of data from sources, processes them through transformations, then saves them to destinations.

During the execution of a data flow component, a subset of the methods in the T:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent base class is called. The methods, and the sequence in which they are called, are always the same, with the exception of the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PrimeOutput(System.Int32,System.Int32[],Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer[]) and M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer) methods. These two methods are called based on the existence and configuration of a component's T:Microsoft.SqlServer.Dts.Pipeline.Wrapper.IDTSInput100 and T:Microsoft.SqlServer.Dts.Pipeline.Wrapper.IDTSOutput100 objects.

The following list shows the methods in the order in which they are called during component execution. Note that M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PrimeOutput(System.Int32,System.Int32[],Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer[]), when called, is always called before M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer).

At run time, data flow components perform the following tasks:

  • Source components add rows.

  • Transformation components with synchronous outputs receive rows provided by source components.

  • Transformation components with asynchronous outputs receive rows and add rows.

  • Destination components receive rows and then load them into a destination.

During execution, the data flow task allocates T:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer objects that contain all the columns defined in the output column collections of a sequence of components. For example, if each of four components in a data flow sequence adds one output column to its output column collection,the buffer that is provided to each component contains four columns, one for each output column per component. Because of this behavior, a component sometimes receives buffers that contain columns it does not use.

Since the buffers received by your component may contain columns that the component will not use, you must locate the columns that you want to use in your component's input and output column collections in the buffer provided to the component by the data flow task. You do this by using the M:Microsoft.SqlServer.Dts.Pipeline.Wrapper.IDTSBufferManager100.FindColumnByLineageID(System.Int32,System.Int32) method of the P:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.BufferManager property. For performance reasons, this task is ordinarily performed during the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PreExecute method, rather than in M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PrimeOutput(System.Int32,System.Int32[],Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer[]) or M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer).

M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PreExecute is called before the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PrimeOutput(System.Int32,System.Int32[],Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer[]) and M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer) methods, and is the first opportunity for a component to perform this work after the P:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.BufferManager becomes available to the component. During this method, the component should locate its columns in the buffers and store this information internally so the columns can be used in either the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PrimeOutput(System.Int32,System.Int32[],Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer[]) or M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer) methods.

The following code example demonstrates how a transformation component with a synchronous output locates its input columns in the buffer during M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PreExecute.

private int []bufferColumnIndex;
public override void PreExecute()
{
    IDTSInput100 input = ComponentMetaData.InputCollection[0];
    bufferColumnIndex = new int[input.InputColumnCollection.Count];

    for( int x=0; x < input.InputColumnCollection.Count; x++)
    {
        IDTSInputColumn100 column = input.InputColumnCollection[x];
        bufferColumnIndex[x] = BufferManager.FindColumnByLineageID( input.Buffer, column.LineageID);
    }
}

Components supply rows to downstream components by adding rows to T:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer objects. The data flow task provides an array of output buffers - one for each T:Microsoft.SqlServer.Dts.Pipeline.Wrapper.IDTSOutput100 object that is connected to a downstream component - as a parameter to the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PrimeOutput(System.Int32,System.Int32[],Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer[]) method. Source components and transformation components with asynchronous outputs add rows to the buffers and call the M:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.SetEndOfRowset method when they are finished adding rows. The data flow task manages the output buffers that it supplies to components and, as a buffer becomes full, automatically moves the rows in the buffer to the next component. The M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PrimeOutput(System.Int32,System.Int32[],Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer[]) method is called one time per component, unlike the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer) method, which is called repeatedly.

The following code example demonstrates how a component adds rows to its output buffers during the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.PrimeOutput(System.Int32,System.Int32[],Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer[]) method, then calls the M:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.SetEndOfRowset method.

public override void PrimeOutput( int outputs, int []outputIDs,PipelineBuffer []buffers)
{
    for( int x=0; x < outputs; x++ )
    {
        IDTSOutput100 output = ComponentMetaData.OutputCollection.GetObjectByID( outputIDs[x]);
        PipelineBuffer buffer = buffers[x];

        // TODO: Add rows to the output buffer.
    }
    foreach( PipelineBuffer buffer in buffers )
    {
        /// Notify the data flow task that no more rows are coming.
        buffer.SetEndOfRowset();
    }
}

For more information about developing components that add rows to output buffers, see Developing a Custom Source Component and Developing a Custom Transformation Component with Asynchronous Outputs.

Components receive rows from upstream components in T:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer objects. The data flow task provides a T:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer object that contains the rows added to the data flow by upstream components as a parameter to the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer) method. This input buffer can be used to examine and modify the rows and columns in the buffer, but cannot be used to add or remove rows. The M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer) method is called repeatedly until there are no more available buffers. The last time it is called, the P:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.EndOfRowset property is true. You can iterate over the collection of rows in the buffer by using the M:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.NextRow method, which advances the buffer to the next row. This method returns false when the buffer is on the last row in the collection. You do not have to check the P:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.EndOfRowset property unless you have to perform an additional action after the last rows of data have been processed.

The following text shows the correct pattern for using the M:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.NextRow method and the P:Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer.EndOfRowset property:

while (buffer.NextRow())

{

// Do something with each row.

}

if (buffer.EndOfRowset)

{

// Optionally, do something after all rows have been processed.

}

The following code example demonstrates how a component processes the rows in input buffers during the M:Microsoft.SqlServer.Dts.Pipeline.PipelineComponent.ProcessInput(System.Int32,Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer) method.

public override void ProcessInput( int inputID, PipelineBuffer buffer )
{
    {
        IDTSInput100 input = ComponentMetaData.InputCollection.GetObjectByID(inputID);
        while( buffer.NextRow())
        {
            // TODO: Examine the columns in the current row.
        }
}

For more information about developing components that receive rows in input buffers, see Developing a Custom Destination Component and Developing a Custom Transformation Component with Synchronous Outputs.

Integration Services icon (small) Stay Up to Date with Integration Services

For the latest downloads, articles, samples, and videos from Microsoft, as well as selected solutions from the community, visit the Integration Services page on MSDN:


For automatic notification of these updates, subscribe to the RSS feeds available on the page.

Show:
© 2015 Microsoft