public abstract class FastLexer extends AbstractLexer implements LexerToken
Lexer
AbstractLexer.DefaultLexerToken
Modifier and Type | Field and Description |
---|---|
protected boolean |
skipComments
Whether to skip comments or not.
|
currentPos, textBuffer
TK_EOF, TK_NOT_FOUND
Constructor and Description |
---|
FastLexer() |
Modifier and Type | Method and Description |
---|---|
void |
backup()
Unlexes the last found token.
|
abstract int |
getEndOffset()
Fetches the end offset of the token (offset relative to the original
ReadTextBuffer being lexed. |
abstract int |
getStartOffset()
Fetches the start offset of the token (offset relative to the
original
ReadTextBuffer being lexed. |
abstract int |
getToken()
Fetches the (language-specific) token represented by this
LexerToken . |
protected boolean |
isComment(int tk)
Used by the ArrayedLexer.
|
abstract int |
lex()
Calls lex( LexerToken ) but does not call fillLexerToken.
|
abstract int |
lex(LexerToken lexedToken)
Scans the text buffer at the current position and returns the
token that was found.
|
void |
setSkipComments(boolean skipComments)
Sets whether the
FastLexer should generate tokens
for Source comments. |
createLexerToken, getTextBuffer, setPosition, setTextBuffer
public void setSkipComments(boolean skipComments)
FastLexer
should generate tokens
for Source comments.skipComments
- true to ignore comments in token generationprotected boolean isComment(int tk)
public abstract int getToken()
LexerToken
LexerToken
. You must refer to the language-specific
Lexer that was responsible for filling this
LexerToken
instance to interpret the token.getToken
in interface LexerToken
public abstract int getStartOffset()
LexerToken
ReadTextBuffer
being lexed. Note that the
start offset is inclusive (compare to end offset which is
exclusive), to match the behavior in the String
class.getStartOffset
in interface LexerToken
public abstract int getEndOffset()
LexerToken
ReadTextBuffer
being lexed. Note that the end offset is
exclusive (similar to end offsets in the String class.)getEndOffset
in interface LexerToken
public abstract int lex()
public abstract int lex(LexerToken lexedToken)
AbstractLexer
lexedToken
instance passed in to the
call.lex
in interface Lexer
lex
in class AbstractLexer
lexedToken
- the instance passed in where token info is storedlexedToken.getToken()
(for convenience)public void backup()
lex()
will return the last token and offset information found.backup
in interface Lexer
backup
in class AbstractLexer