This documentation is archived and is not being maintained.

Extending the ATL Framework Part 2: Persistence and Marshalling for STL Collections

Visual Studio .NET 2003

Martin Lapierre

February 2002

Summary: This second part of the ATL Extension Series examines the core of ATL support for STL collections, and discusses how to build implementation templates for STL collections, persistence, and marshalling. This article assumes you are familiar with COM, ATL, STL collections, persistence, and marshalling. The last part of the series will discuss creation of a smart, persistent, list component (with proxy characteristics), and also outline how to create an MBV persistent-map component. (15 printed pages)

Download Sample from "Extending the ATL Framework Part 1."


Exploring the Framework
Adding STL Support to ATL
Reusable Design
ATL Extension Code Updates


In this series of articles we explore the ATL Framework design and use the ATL Framework as a model to create extended support for persistence and marshalling. In this second of three articles, we examine the core of ATL support for STL collections, and build implementation templates for STL collections, persistence, and marshalling.

This article assumes you are familiar with COM, ATL, STL collections, persistence, and marshalling.

Exploring the Framework

COM uses a commonly accepted standard for collection handling in the form of the _NewEnum, Item, and Count properties, as well as the IEnumXXXX interfaces. The ATL Framework supports this standard by offering various class templates. Additionally, the ATL Framework simplifies the creation of COM collections by implementing the COM standard for STL-based collections.

The secret lies in the IEnumOnSTLImpl and ICollectionOnSTLImpl templates. IEnumOnSTLImpl implements an IEnumXXXX interface over STL iterators:

template <class Base, const IID* piid, class T, class Copy, 
class CollType>
class ATL_NO_VTABLE IEnumOnSTLImpl : public Base
  // ... code skipped ...

and ICollectionOnSTLImpl manages an STL-based collection's default properties:

template <class T, class CollType, class ItemType, class CopyItem, 
class EnumType>
class ICollectionOnSTLImpl : public T
  // ... code skipped ...

For more information on IEnumOnSTLImpl and ICollectionOnSTLImpl, see "References" at the end of this article. We will now examine template arguments.

The parameters of interest in IEnumOnSTLImpl include:

  • T
  • Copy
  • CollType

T, Copy, and CollType are matched respectively in ICollectionOnSTLImpl:

  • ItemType
  • CopyItem
  • CollType

The following table shows the relations and meaning of the parameters.

IEnumOnSTLImpl ICollectionOnSTLImpl Description
CollType CollType Type of STL collection.
T ItemType Type of element(s) exposed by the enumeration interface and the Item method.
Copy CopyItem Class policy that manages initialization, copy, and destruction of the elements.

The class policy handles the manipulation of the collection's elements, since some may require special initialization, destruction, and copy procedures. ATL offers a generic class policy, the _Copy class template, and specialized policies for VARIANTs, interface pointers, and other types.

This code example shows the _Copy<VARIANT> policy:

class _Copy<VARIANT>
   static HRESULT copy(VARIANT* p1, VARIANT* p2) {return VariantCopy(p1, 
   static void init(VARIANT* p) {p->vt = VT_EMPTY;}
   static void destroy(VARIANT* p) {VariantClear(p);}

The following code is a fragment from CComEnumImpl::Init that illustrates how the policies can play a part in the lifecycle of an element.

// From CComEnumImpl::Init
for (T* i=begin; i != end; i++)
  HRESULT hr = Copy::copy(m_iter, i);
  if (FAILED(hr))
    T* p = m_begin;
    while (p < m_iter)
    delete [] m_begin;
    m_begin = m_end = m_iter = NULL;
    return hr;

An interesting area of the framework design is the flexibility provided by the templates — they open up a wide range of possibilities for the type of STL collections we can use, as well as for the type of elements managed by the collection.

We will now apply these principles to create seamless persistence and marshalling for STL collections.

Adding STL Support to ATL

Part 1 shows that persistence and marshalling work in a very similar way. However, delegating marshalling to the persistence interfaces as presented in Part 1 presents some limitations:

  • First, a collection could expose neither IPersistStream nor IpersistStreamInit, but still provide custom marshalling. In this case, AtlIMarshalOnStream_XXXX global functions would not be useful.
  • Second, in some situations we would like to use standard marshalling for elements that do not support MBV.

These issues will be addressed in our design.

The current ATL implementation of IPersistStreamInitImpl includes flaws:

  • It does not use delegation for GetClassID, GetSizeMax, and InitNew. Consequently, the associated redefinition-mechanism will be different except for the Load and Save methods.
  • Including the option to supply a different CLSID (for perhaps a component that supports emulation) in the same way we redefine the other methods, is ideal.
  • The common functionality of the GetClassID, IsDirty, and InitNew methods is not regrouped in an abstract class that can be used by custom implementations of IPersistStreamInit (such as the one we are about to do).
  • The GetSizeMax method does nothing in Microsoft® Visual Studio® version 6.0, and returns the size of the property map in Microsoft Visual Studio .NET, which does not answer our needs. We will make sure to offer a more generic code structure for it in our solution.

Before starting our creation of persistence and marshalling for STL collections, we must consider the following:

  • Provide the necessary size for the write and the read/write operations.
  • Each mechanism must manipulate the STL collection and its elements, whether to learn their required size, to put them on, or to get them off the stream. For this to work, we'll devise class policies in the same way ATL does for handling copying.

We must also review the marshalling strategy. We could again be tempted to simply delegate calls to a persistence interface, but if we want to have more flexibility in our MBV design, we'll have to enable the marshalling of components that do not support persistence. This is important, because a collection can contain different types of elements including:

  • long
  • BSTR
  • Interface pointer

When a collection holds interface pointers, we may want it to act as a smart proxy — that is, marshal the elements by value when possible, but keep remote references for components unable to comply to MBV. For that reason, it is necessary to have the marshalling information available when handling the collection's elements. Persistence and marshalling methods are very similar. Consequently, the best approach, for providing availability for the marshalling information throughout the framework, is to make the persistence operation oblivious to it.

With all these considerations in mind, we hold the fundamentals to the creation of the basic class policy for persistence and marshalling, the _Persist class template:

// Marshalling information structure
typedef struct
  DWORD dwDestContext;
  void __RPC_FAR *pvDestContext;
  DWORD mshlflags;
} _MarshalInfo;

// _Persist<>
// The basic persistence policy class.
template <class T>
class _Persist
  static HRESULT Load(T *pT, LPSTREAM pStm) 
    return pStm->Read(pT, sizeof(T), NULL);
  static HRESULT Save(T *pT, LPSTREAM pStm, _MarshalInfo *pMshlInfo = 
    return pStm->Write(pT, sizeof(T), NULL);
  static HRESULT GetSizeMax(T *pT, ULARGE_INTEGER *pcbSize, _MarshalInfo 
*pMshlInfo = NULL)   
    pcbSize->QuadPart = sizeof(T); 
    return S_OK;
  static void init(T*) {}
  static void destroy(T*) {}
}; // class _Persist

_Persist uses basic type information to handle the persistence operations. _Persist also exposes the init and the destroy methods, which are going to play the same role as in the _Copy class policy. _Persist only manages simple types. For types such as VARIANTs and interface pointers, we need specialized policies.

To manage interface pointers, we define the _PersistInterface class template. _PersistInterface handles any type of interface pointer.

// _PersistInterface<>
// The Interface persistence policy class.
template <class T = IUnknown, const IID* pIID = &IID_IUnknown>
class _PersistInterface
  typedef enum
  } _PersistMethod;

  static HRESULT Load(T **pT, LPSTREAM pStm);
  static HRESULT Save(T **pT, LPSTREAM pStm, _MarshalInfo *pMshlInfo = 
  static HRESULT GetSizeMax(T **pT, ULARGE_INTEGER *pcbSize, _MarshalInfo 
*pMshlInfo = NULL);
  static void init(T **pT) {*pT = NULL;}
  static void destroy(T **pT) {if (*pT) (*pT)->Release();}
}; // class _PersistInterface

We have established the improved behaviors for interface marshalling. To achieve our goals, we continue to rely on the presence of IPersistStreamInit or IPersistStream to marshal the component by value, but will use the standard marshalling calls (which will invoke standard, handler, or custom marshalling) if the persistent interfaces are not found.

When using MBV, a component writes a CLSID on the stream to indicate to the unmarshalling process what component to use to read the data back from the stream. To delegate calls to the persistence interfaces follow this procedure:

  1. Write the CLSID obtained by GetClassID.
  2. Call the Save method and recreate the appropriate component while unmarshalling to finally call its Load method.

If we are to combine both persistence marshalling and COM marshalling, we have to be cautious, because COM marshalling does not write only a CLSID to the stream. COM marshalling also writes a more complex structure called an object reference (OBJREF). It is essential to know what to expect when reading the data back from the stream. For that reason, we define the _PersistMethod enumeration, which contains a series of values used to indicate the strategy adopted by the marshalling process. That value will be written to the stream prior to any other operation.

This is how we define Save for the _PersistInterface class policy:

template <class T, const IID* pIID>
inline HRESULT _PersistInterface<T, pIID>::Save(T **pT, LPSTREAM pStm, 
_MarshalInfo *pMshlInfo) 

  _PersistMethod ePersistMethod(NONE);
  if ( *pT == NULL ) // Nothing to persist.
    return pStm->Write(&ePersistMethod, sizeof(_PersistMethod), NULL);

  // Persistence method priority:
  // 1) IPersistStreamInit
  // 2) IPersistStream
  // 3) Standard marshalling

  CComPtr<IPersistStream> ccpIPS;
  CComQIPtr<IPersistStreamInit> ccpIPSI(*pT);
  if ( ccpIPSI ) // Look for IPersistStreamInit.
    ePersistMethod = PERSIST_STREAM_INIT;
    ccpIPS = reinterpret_cast<IPersistStream*>(ccpIPSI.p); // Same 
  else // Look for IPersistStream.
    CComQIPtr<IPersistStream> ccpIPS2(*pT);
    if ( ccpIPS2 )
      ePersistMethod = PERSIST_STREAM;
      ccpIPS.Attach( ccpIPS2.Detach() );
  // Commit IPersistStream / IPersistStreamInit to stream.
  if ( ccpIPS )
    TestHR(pStm->Write(&ePersistMethod, sizeof(_PersistMethod), NULL))
    return OleSaveToStream(ccpIPS, pStm);

  // Standard marshalling
  if ( pMshlInfo )
    ePersistMethod = STANDARD_MARSHALLING;
    TestHR(pStm->Write(&ePersistMethod, sizeof(_PersistMethod), NULL))
    return CoMarshalInterface(pStm, *pIID, *pT, pMshlInfo->dwDestContext,
                    pMshlInfo->pvDestContext, pMshlInfo->mshlflags);

  // No operation possible.
  return STG_E_CANTSAVE;

The GetSizeMax method works like the Save method, collecting size information rather that writing to the stream. If we were to simply persist an interface pointer on the stream rather than marshal it, we would pass NULL as the value of the marshalling information structure, thus avoiding marshalling altogether.

Reading the information back from the stream is quite straightforward:

template <class T, const IID* pIID>
inline HRESULT _PersistInterface<T, pIID>::Load(T **pT, LPSTREAM pStm)
  ATLASSERT(pT != NULL && *pT == NULL);

  _PersistMethod ePersistMethod(NONE);

  // First read the persisting method.
  TestHR(pStm->Read(&ePersistMethod, sizeof(_PersistMethod), NULL))

  switch (ePersistMethod)
    // No Interface.
    case NONE:
      *pT = NULL;

    // IPersistStreamInit
        // Can't use the V-Table trick here.
        CLSID clsid;
        CComPtr<IPersistStreamInit> ccpIPSI;
        TestHR(ReadClassStm(pStm, &clsid))
        TestHR(ccpIPSI->QueryInterface(*pIID, (void**)pT))

    // IPersistStream
      TestHR(OleLoadFromStream(pStm, *pIID, (void**)pT))

    // Standard marshalling
      TestHR(CoUnmarshalInterface(pStm, *pIID, (void**)pT))

     // Unsupported method.
      return E_FAIL;
  return S_OK;

A class policy for VARIANTs will perform the same operation. In fact, the ATL CComVariant class already supports writing to and reading from a stream, but at a lesser level than what we are doing in this exercise. The _Persist<VARIANT> class policy delegates the handling of interface pointers to _PersistInterface and manages a VARIANT much like the ATL class. The following illustration shows the byte stream layout of a VARIANT containing an IDispatch pointer of a component exposing the IPersistStreamInit interface.

Figure 1

_Persist<BSTR> is also available for BSTR persistence.

Now that we can handle all elements, lets address the collection itself. Since Part 1 describes many of the concepts behind the design of the persistence mechanisms, the balance of this article will briefly discuss the persistent and marshalling class templates, and focus mainly on the core global functions.

To correct the flaws in IPersistStreamInitImpl, create an abstract class for persistence in the same way IMarshalBaseImpl is implemented in Part 1. This process creates IPersistStreamInitBaseImpl, which provides basic code for IsDirty, as well as redefinition support for GetClassID and InitNew.

The implementation class for persistence, IPersistStreamInitOnSTLImpl, inherits IpersistStreamInitBaseImpl, and supplies Load, Save and GetSizeMax. To support any type of collection, make it a class template whose arguments are similar to those of ICollectionOnSTLImpl and IEnumOnSTLImpl:

IPersistStreamInitOnSTLImpl Description
CollType Type of STL collection.
ItemType Type of elements to be persisted.
PersistItem Class policy that manages the persistence operations for the collection's elements.

Finally, IPersistStreamInitOnSTLImpl delegates the task(s) of handling the collection persistence to global functions. The following code shows how this works for the Save method:

template <class T, class CollType, class ItemType, class PersistItem = 
_Persist<ItemType> >
class ATL_NO_VTABLE IPersistStreamInitOnSTLImpl : public 
// ... code skipped ...
  STDMETHOD(Save)(LPSTREAM pStm, BOOL fClearDirty)
    ATLTRACE2(atlTraceCOM, 0, _T("IPersistStreamInitOnSTLImpl::Save\n"));
    T* pT = static_cast<T*>(this);
    return pT->IPersistStreamInitOnSTL_Save(pStm, fClearDirty, 
  HRESULT IPersistStreamInitOnSTL_Save(LPSTREAM pStm, BOOL fClearDirty,
CollType *pColl)
    T* pT = static_cast<T*>(this);
    HRESULT hr = AtlPersistOnSTL_Save<CollType, ItemType, 
PersistItem>(pStm, fClearDirty, pColl);
    if (SUCCEEDED(hr))
      pT->m_bRequiresSave = FALSE;
    return hr;
}; // class IPersistStreamInitOnSTLImpl

AtlPersistOnSTL_Save features the same template parameters as IPersistStreamInitOnSTLImpl and this flexibility makes it reusable for general collection persistence. Writing the collection to the stream is simple. First, save the library version to know how to read the collection back if the library evolves. Then write the number of elements found in the collection, and apply the class policy to save each element:

// Save the version number.
DWORD dwVersion = _ATLEXT_VER;
TestHR(pStm->Write(&dwVersion, sizeof(DWORD), NULL))

// Save the number of item in the collection.
CollType::size_type size(pColl->size());
TestHR(pStm->Write(&size, sizeof(CollType::size_type), NULL))

// Save the collection.
CollType::reverse_iterator it;
for (it = pColl->rbegin(); it != pColl->rend(); it++)
  TestHR(PersistItem::Save(&*it, pStm, pMshlInfo))

To load the collection, read the version number and the number of items, then use a temporary element to read the data from the stream. Then build back the collection. The class policy is used again to initialize the element, read it from the stream, and finally release it:

// Load the version number.
DWORD dwVersion(-1);
TestHR(pStm->Read(&dwVersion, sizeof(DWORD), NULL))
if (dwVersion > _ATLEXT_VER)
  return E_FAIL; // Unsupported version.

// Load the number of item in the collection
CollType::size_type size(0);
TestHR(pStm->Read(&size, sizeof(CollType::size_type), NULL))

// Load the collection
ItemType item; // Temporary item
for (CollType::size_type i(0); i < size; i++)
  hr = PersistItem::Load(&item, pStm);
  if (FAILED(hr))
     // Clear the data properly.
    for (it = pColl->begin(); it != pColl->end(); it++)
    return hr;
  it = pColl->insert(pColl->begin(), item);

Adding IMarshal support is the last step. Since we want a better mechanism than IMarshalOnStreamImpl, a specialized class template, called IMarshalOnSTLImpl, is used to marshal collections. IMarshalOnSTLImpl offers interception points for method overwrites and delegates the actual marshalling to global functions. These global functions handle the marshalling context appearing in Part 1, and use standard marshalling when MBV is not required. However, IMarshalOnSTLImpl delegates marshalling to the persistence global functions by passing them the marshalling context, which in turn is passed back down to the collection's elements persistence-operation. As an example, examine how AtlIMarshalOnSTL_GetMarshalSizeMax delegates to AtlPersistOnSTL_GetSizeMax:

if (MSH_ISMBV(dwDestContext, mshlopt))
  // Use marshalling by value/smart proxy.
  ULARGE_INTEGER cbSize = {0};
  _MarshalInfo mshlinfo = {dwDestContext, pvDestContext, mshlflags};
  TestHR((AtlPersistOnSTL_GetSizeMax<CollType, ItemType, 
MarshalItem>(&cbSize, pColl, &mshlinfo)))
  *pSize = cbSize.LowPart;
  return S_OK;

The resulting byte stream layout of a COM-based collection, marshalled by value over the wire, would have this structure:

Figure 2

The OBJREF would then contain the CLSID of the collection component as provided by IMarshalBaseImpl::GetUnmarshalClass. The unmarshalling process will use that CLSID to create the proper component and read the rest of the stream with IMarshalOnSTLImpl::UnmarshalInterface.

Reusable Design

At this point in the exercise, a powerful framework is at our disposal. A basic COM collection will simply use ICollectionOnSTLImpl, IEnumOnSTLImpl, and the new IPersistStreamInitOnSTLImpl and IMarshalOnSTLImpl class templates to implement the COM commonly-accepted standards for collections, persistence, and optimized marshalling.

The framework design provides basic abstract classes and method redefinition for custom implementations.

Whether a part of COM or not a part of COM, the global functions are solid tools for supporting persistence and marshalling for any STL collection. Additionally, the specialized class policies can also be used independently for custom persistence and marshalling of VARIANTs, BSTRs or interface pointers — giving us MBV and smart-proxy functionality in the blink of an eye.

ATL Extension Code Updates

For the latest code update, see ATLExtension Update.


Part 2 of the ATL Extension Series has exposed the fundamentals of enhanced STL support and also provided a reusable framework based on ATL. The last part of the ATL Extension Series will build on Part 1 and Part 2 by creating a smart, persistent, list component (with proxy characteristics).

About the Author

Martin Lapierre is a freelance writer and consultant hosted in Montreal, Quebec, Canada. He holds a BS in Computer Science from the University of Montreal and has been designing desktop, n-tiers, and Web applications on the Windows platform since 1995.