CStringT::Tokenize (ATL/MFC)

CStringT::Tokenize

Collapse All

Language Filter

This page is specific to

Microsoft Visual Studio 2005/.NET Framework 2.0

Other versions are also available for the following:

Shared Visual C++ Classes Reference

CStringT::Tokenize

Finds the next token in a target string

CStringT Tokenize(
   PCXSTR pszTokens,
   int& iStart
) const;

Parameters

pszTokens: A string containing token delimiters. The order of these delimiters is not important.

iStart: The zero-based index to begin the search.

Return Value

A CStringT object containing the current token value.

Remarks

The Tokenize function finds the next token in the target string. The set of characters in pszTokens specifies possible delimiters of the token to be found. On each call to Tokenize the function starts at iStart, skips leading delimiters, and returns a CStringT object containing the current token, which is the string of characters up to the next delimiter character. The value of iStart is updated to be the position following the ending delimiter character, or -1 if the end of the string was reached. More tokens can be broken out of the remainder of the target string by a series of calls to Tokenize, using iStart to keep track of where in the string the next token is to be read. When there are no more tokens the function will return an empty string and iStart will be set to -1.

Unlike the CRT tokenize functions like strtok_s, _strtok_s_l, wcstok_s, _wcstok_s_l, _mbstok_s, _mbstok_s_l, Tokenize does not modify the target string.

Example

//typedef CStringT< TCHAR, StrTraitATL< TCHAR > > CAtlString;
CAtlString str( "%First Second#Third" );
CAtlString resToken;
int curPos= 0;

resToken= str.Tokenize("% #",curPos);
while (resToken != "")
{
   printf("Resulting token: %s\n", resToken);
   resToken= str.Tokenize("% #",curPos);
};

Output

Resulting Token: First
Resulting Token: Second
Resulting Token: Third

Reference

CStringT Class

Other Resources

CStringT Members

Tags

: Add a tag

Community Content

Add new content

Annotations

Example wrong?

Todd Aspeotis ... SamJ__ | Edit | Show History

Quote:

Shouldn't the example check while (curPos > 0) instead of resToken != "" ? What happens if you have a CSV file that has

value,value,,

value,value,value

You will never parse the last 3 values.

Consecutive delimeters will be collapsed, so in your example it would correctly return 5 values e.g.:

Code:

CAtlString Input = "value1,value2,,\nvalue3,value4,value5\n";
CAtlString tok;
int pos = 0;
while( (tok=input.Tokenize(", \n", pos)) != "" )
    printf("tok=%s\n", tok );

Output:

tok=value1
tok=value2
tok=value3
tok=value4
tok=value5