public interface LexerToken
LexerToken
interface encapsulates a single token
retrieved by a call to Lexer.lex()
. This simply
provides a convenient way for the lexer to return multiple pieces
of information (token class, start offset, end offset) from the
Lexer to its clients. It is defined as an interface to allow language-specific Lexer implementations to add additional information pieces to the token object.
The LexerToken
is allocated using a call to
Lexer.createLexerToken()
, and is passed into the
Lexer.lex()
to be filled in.
Lexer
Modifier and Type | Method and Description |
---|---|
int |
getEndOffset()
Fetches the end offset of the token (offset relative to the original
ReadTextBuffer being lexed. |
int |
getStartOffset()
Fetches the start offset of the token (offset relative to the
original
ReadTextBuffer being lexed. |
int |
getToken()
Fetches the (language-specific) token represented by this
LexerToken . |
int getToken()
LexerToken
. You must refer to the language-specific
Lexer that was responsible for filling this
LexerToken
instance to interpret the token.int getStartOffset()
ReadTextBuffer
being lexed. Note that the
start offset is inclusive (compare to end offset which is
exclusive), to match the behavior in the String
class.int getEndOffset()
ReadTextBuffer
being lexed. Note that the end offset is
exclusive (similar to end offsets in the String class.)