Questions tagged [lexer]
a lexer is a program performing lexical analysis: it converts a sequence of characters into a sequence of tokens.
49 questions
2 votes
5 answers
2k views
Rewrite or Transpiler - How to move away from a proprietary SAAS solution
We are using a Software as a Service platform that allows to create custom code which integrates in the platform and all its features (dialogues for common objects like Account, Customer, Address, and ...
- 181
0 votes
2 answers
676 views
How Should Lexers Be Stateful?
Aside from modes, Antlr grammars can use "actions" which have to be written in the target language, sometimes seen used to conditionally push and pop from the mode stack. If I were to make a ...
0 votes
1 answer
758 views
How is it possible to store the AST nodes location in the source code?
I created a simple parser in Rust and defined the AST like this: enum Expression { Number(i32), BinaryOperator(Box<Expression>, Operator, Box<Expression>), Identifier(String), }...
- 111
0 votes
2 answers
405 views
Do lexers have to go word by word or can they go line by line
So I'm trying to write a interpreter with a lexer. Currently, it adds a token line by line and does some more processing later on. But when I look at sources online, they all seem to go word by word ...
0 votes
0 answers
488 views
Parsing custom if statement input in PHP
I'm working on a feature where users can get data based on the if statement they write. The if statement looks something like the excel's conditionals. Basic syntax: IF ( lhs == rhs, ifTrue, ifFalse)...
- 151
0 votes
1 answer
419 views
Differences between enumeration-based and hierarchical token typing
When writing a lexer/parser, why/when would an advised developer chose to define the tokens' types through an enumeration field/type hierarchy? The closest question I've found here so far was Lexing: ...
user276191
-2 votes
1 answer
214 views
Multiple variable declaration, multiple variable assignment, context-sensitive 'in' statement
Lately I've been playing with writing my own programming language, following the excellent Crafting Interpreters book but I've hit something of a snag. I'd like to extend the parser to accept ...
- 135
3 votes
3 answers
590 views
How to create simulator for distributed algorithms written in simple language
I started development of simulator for simulation of distributed algorithms in language C. My work consist of creating simple language for algorithm description and simulator which takes the described ...
- 133