<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta http-equiv="Content-Language" content="en-us"> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="GENERATOR" content="Microsoft FrontPage 6.0"> <meta name="ProgId" content="FrontPage.Editor.Document"> <title>Introduction</title> </head> <body bgcolor="#FFFFFF"> <p><img src="../../boost.png" alt="C++ Boost" width="277" height= "86"><br></p> <h1 align="center">Introduction</h1> <p align="left">The boost Tokenizer package provides a flexible and easy to use way to break of a string or other character sequence into a series of tokens. Below is a simple example that will break up a phrase into words.</p> <div align="left"> <pre> // simple_example_1.cpp #include<iostream> #include<boost/tokenizer.hpp> #include<string> int main(){ using namespace std; using namespace boost; string s = "This is, a test"; tokenizer<> tok(s); for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){ cout << *beg << "\n"; } } </pre> </div> <p align="left">You can choose how the string gets broken up. You do this by specifying the TokenizerFunction. If you do not specify anything, the default TokenizerFunction is char_delimiters_separator<char> which defaults to breaking up a string based on space and punctuation. Here is an example of using another TokenizerFunction called escaped_list_separator. This TokenizerFunction parses a superset of comma separated value (csv) lines. The format looks like this</p> <p align="left">Field 1,"putting quotes around fields, allows commas",Field 3</p> <p align="left">Below is an example that will break the previous line into its 3 fields</p> <div align="left"> <pre> // simple_example_2.cpp #include<iostream> #include<boost/tokenizer.hpp> #include<string> int main(){ using namespace std; using namespace boost; string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3"; tokenizer<escaped_list_separator<char> > tok(s); for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){ cout << *beg << "\n"; } } </pre> </div> <p align="left">Finally, for some TokenizerFunctions you have to pass in something into the constructor in order to do anything interesting. An example is offset_separator. This class breaks a string into tokens based on offsets for example</p> <p align="left">12252001 when parsed using offsets of 2,2,4 becomes 12 25 2001. Below is an example to parse this.</p> <div align="left"> <pre> // simple_example_3.cpp #include<iostream> #include<boost/tokenizer.hpp> #include<string> int main(){ using namespace std; using namespace boost; string s = "12252001"; int offsets[] = {2,2,4}; offset_separator f(offsets, offsets+3); tokenizer<offset_separator> tok(s,f); for(tokenizer<offset_separator>::iterator beg=tok.begin(); beg!=tok.end();++beg){ cout << *beg << "\n"; } } </pre> </div> <p align="left"> </p> <hr> <p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src= "../../doc/images/valid-html401.png" alt="Valid HTML 4.01 Transitional" height="31" width="88"></a></p> <p>Revised <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25 December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p> <p><i>Copyright © 2001 John R. Bandela</i></p> <p><i>Distributed under the Boost Software License, Version 1.0. (See accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or copy at <a href= "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p> </body> </html>