We recommend using Visual Studio 2017
This documentation is archived and is not being maintained.


Finds the next token in a target string

CStringT Tokenize(
   PCXSTR pszTokens,
   int& iStart
) const;


A string containing token delimiters. The order of these delimiters is not important.


The zero-based index to begin the search.

A CStringT object containing the current token value.

The Tokenize function finds the next token in the target string. The set of characters in pszTokens specifies possible delimiters of the token to be found. On each call to Tokenize the function starts at iStart, skips leading delimiters, and returns a CStringT object containing the current token, which is the string of characters up to the next delimiter character. The value of iStart is updated to be the position following the ending delimiter character, or -1 if the end of the string was reached. More tokens can be broken out of the remainder of the target string by a series of calls to Tokenize, using iStart to keep track of where in the string the next token is to be read. When there are no more tokens the function will return an empty string and iStart will be set to -1.

Unlike the CRT tokenize functions like strtok_s, _strtok_s_l, wcstok_s, _wcstok_s_l, _mbstok_s, _mbstok_s_l, Tokenize does not modify the target string.

// typedef CStringT<TCHAR, StrTraitATL<TCHAR, ChTraitsCRT<TCHAR>>> CAtlString;
CAtlString str(_T("%First Second#Third"));
CAtlString resToken;
int curPos = 0;

resToken= str.Tokenize(_T("% #"),curPos);
while (resToken != _T(""))
   _tprintf_s(_T("Resulting token: %s\n"), resToken);
   resToken = str.Tokenize(_T("% #"), curPos);

The output from this example is as follows:

Resulting Token: First

Resulting Token: Second

Resulting Token: Third

Header: cstringt.h