0

I am curious about how bison's string literal tokens are stored.

Consider these definitions in myparser.y:

%token FOR %token TO "->"

If I understand the way the system works, the first line will result in the string "FOR" being stored in yytname, and a system-generated number, perhaps 257 in this case, being stored in yytoknum. Is that correct?

What happens in the second line? I assume "TO" will end up in yytname and 258 in yytoknum, but where is the string literal "->" stored? I cannot find a vector for it.

My goal is to be able to iterate over these literals in my flex scanner, but I am not sure how to do so.

Improve this question
7
  • You can look at the generated C code for the parser. Commented May 22 at 17:02
  • It doesn't store them anywhere. The scanner recognizes the character sequence and returns the token number to Bison. The character sequence is not passed to Bison. You may be able to get at it via yytext in the lowest-level production it appears in, but it depends on the lookahead conditions and other things. Why do you think you need this? Commented May 23 at 1:23
  • "My goal is to be able to iterate over these literals in my flex scanner": well, it shouldn't be. That will produce an O(N^2) scanner. You need to have a look at how manual scanners are actually written. This isn't it. Commented May 23 at 3:48
  • The token names exist only for the programmer. They vanish during compilation and internally Bison uses numbers instead. gnu.org/software/bison/manual/html_node/Token-Decl.html Commented May 23 at 6:18
  • If you really want token names at runtime, you're best bet is to check how the syntax error messages look up token names. (This code will be present only if you set the error messages to be sufficiently verbose.) Commented May 23 at 6:21
Load 7 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.