Reading tokens from file (complicated)

Question

I have a basic tokenization structure/algorithm in place. It's pretty complicated, and I hope I can clarify it simply enough to enlighten you about the "flaw" in my design.

class ParserState

// bool functions return false if getline() or stream extraction '>>' fails static bool nextLine(); // reads and tokenizes next line from file and puts it in m_buffer static bool nextToken(); // gets next token from m_buffer, via fetchToken(), and puts it in m_token static bool fetchToken( std::string &token ); // procures next token from file/buffer static size_t m_lineNumber; static std::ifstream m_fstream; static std::string m_buffer; static std::string m_token;

The reason for this setup is being able to report the line number if a syntax error occurs. Depending on the phase/state of the parser, differend things happen in my program, and subclasses of this ParserState use m_token and nextToken to continue. fetchToken calls nextLine if m_buffer is empty, and puts the next token in its argument:

istringstream stream; do // read new line until valid token can be extracted { Debug(5) << "m_buffer contains: " << m_buffer << "\n"; stream.str( m_buffer ); if( stream >> token ) { Debug(5) << "Token extracted: " << token << "\n"; m_token = token; return true; // return when token found } stream.clear(); } while( nextLine() ); // if no tokens can be extracted from the whole file, return false return false;

The problem is that the token removed from m_buffer isn't removed, and the same token gets read with every call to nextToken(). The problem is that m_buffer can be modified, thus the call to istringstream::str in the loop. But this is the cause of my issue, and as far as I can see, it can't be worked around, hence my question: How can I let stream >> token remove something from the string pointed to internally by the stringstream? Perhaps I need to not use a stringstream, but something more elementary in this situation (like find first whitespace and cut the first token from the string)?

Thanks a billion!

PS: any suggestions altering my function/class structure are ok as long as they allow line numbers to be kept track of (so no full file read into m_buffer and a class member istringstream, which is what I had before I wanted line number error reporting).

@Oli: all subclasses need to access the same data (the subclasses parse different parts/words of the file, so if one state reads a token, the next state needs to read the next token, not some state-specific thing, hence static=the same for all subclasses) — rubenvb
– rubenvb, Commented Nov 7, 2010 at 20:59

André Caron · Accepted Answer · 2010-11-07 20:55:30Z

1

Why not simply make m_buffer an std::istringstream instead of a std::string? You would remove a temporary variable as well as get the desired effect. Whenever you change m_buffer in statements such as

m_buffer = ...

write this instead:

m_buffer.str(...);

answered Nov 7, 2010 at 20:55

André Caron

45.5k12 gold badges72 silver badges125 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

rubenvb Over a year ago

Then I would have to change my getline( m_fstream, m_buffer ) by something else right? Because that's really the only reason I chose a string. I'll try this and let you know how it goes. Thanks

André Caron Over a year ago

Yes, you would have to use a string temporary there or use std::istreambuf_iterator and std::ostreambuf_iterator, with a variant of std::copy() that takes an optional delimiter.

rubenvb Over a year ago

Well, I converted the whole thing to use a stringstream m_buffer instead of a string and I use stream operators to insert and extract tokens. Works great, thanks for showing me the part I missed :)

vitaut · Accepted Answer · 2010-11-07 21:03:27Z

To avoid reading the same token multiple times I think you have to get the position in stream using tellg and then restore it using seekg (these methods are described here). However using std::istringstream looks as an overkill for me here. I would rather work with m_buffer directly.

That's my problem with stringstream, it's yet another buffer, and a string can do everything I need. tellg will also allow me to point to the erronous token in the error message, which is a nice side effect. I'll look into this!

Ira Baxter · Accepted Answer · 2010-11-08 00:39:09Z

The usual scheme for handling line-number reporting is to read lines one at time, as you have, incrementing a the line count, and then as your tokenizer starts to build a token, it takes a snapshot of the line number and stores it into the token data structure (typically containing the line number, token type, and token value if any).

This decouples line-reading from token building without losing the line number. It also means you can have lots of tokens, they can all have line numbers (including different ones), a token can start on one line and and finish on another, etc.

Collectives™ on Stack Overflow

Reading tokens from file (complicated)

3 Answers 3

3 Comments

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

Comments

Related