precedence in antl4 OR rules

Question

Suppose I have a simple grammar which recognizes lower-case words. Among the words I have some reserved words that I need to address differently.

grammar test; prog: Reserved | Identifier; Reserved: 'reserved'; Identifier: [a-z]+;

Given grammar above, is it promised that in case of "reserved" input always the Reserved token will be produced by lexer?

Yes. The lexer is not guided by the parser. Antlr lexers follow basic rules: 1) The lexer rule that matches the longest input string is selected. 2) If two or more lexer rules match the same string, the first is selected. 3) String literals in parser rules define new lexer rules before all explicit lexer rules internally if there isn't an explicit lexer rule that declares the string literal. In your grammar, Reserved occurs before Identifier so Reserved always matches. Always print out the token types of the tokens when degugging your grammar. — kaby76
– kaby76, Commented May 13 at 9:34

Bart Kiers · Accepted Answer · 2025-05-13 08:11:36Z

Yes, ANTLR's lexer operates using the following rules:

try to create a token with the most amount of characters
if there are 2 or more rules that match the same characters, let the one defined first "win"

Because the input "reserved" can be matched by both the Reserved and Identifier rule, the one defined first (Reserved) gets precedence.

It sometimes happens that keywords kan also be used as identifiers. This is often done by introducing a identifier parser rule that matches an Identifier token or some reserved keywords:

identifier : Reserved | Identifier; Reserved : 'reserved'; Identifier : [a-z]+;

and then use identifier in other parser rules instead of directly using Identifier.

EDIT

After rereading your question: yes, parser rule alternatives are tried from left to right (top to bottom). In the rule p: p : a | b | c;, first a is tried, then b and lastly c.

Note that in your example prog: Reserved | Identifier;, there is no ambiguity since the input "reserved" will never become an Identifier token (the first part of my answer).

Collectives™ on Stack Overflow

precedence in antl4 OR rules

1 Answer 1

EDIT

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

EDIT

Comments

Related