public interface Lexer
Lexer
interface describes the set of calls that
need to be implemented by a Lexer for any given language. It is
intended for providing lexical information only, and does not do
any syntactical or semantic translation.
The underlying document model passed into any Lexer
implementation must implement the ReadTextBuffer
interface
to insulate the Lexer from the actual model. Please refer
to the ReadTextBuffer
interface for more information
on why this interface is being used.
Token information is returned via the LexerToken structure. This
is for convenience so that multiple pieces of (possibly
language-specific) information may be returned to clients in the
same call as the lex()
operation. Clients need to
call the createLexerToken()
of a given Lexer
implementation to create the LexerToken
object
corresponding to that given language and Lexer
.
LexerToken
,
ReadTextBuffer
Modifier and Type | Field and Description |
---|---|
static int |
TK_EOF
Constant indicating that there are no more tokens remaining
because the Lexer reached the end of the file.
|
static int |
TK_NOT_FOUND
Constant indicating a bad token value.
|
Modifier and Type | Method and Description |
---|---|
void |
backup()
Unlexes the last found token.
|
LexerToken |
createLexerToken()
Creates a new empty
LexerToken for a specific
Lexer implementation to be passed in to the
lex() call to be filled in. |
ReadTextBuffer |
getTextBuffer()
Gets the text buffer that is being used by this Lexer.
|
int |
lex(LexerToken lexedToken)
Scans the text buffer at the current position and returns the
token that was found.
|
void |
setPosition(int offset)
Sets the current lex (read) position to the given offset in the
buffer.
|
void |
setTextBuffer(ReadTextBuffer buffer)
Sets the text buffer that is to be used by this Lexer.
|
static final int TK_EOF
static final int TK_NOT_FOUND
KeywordTable
when a keyword lookup is not found.void setTextBuffer(ReadTextBuffer buffer)
buffer
- the text buffer to useReadTextBuffer getTextBuffer()
void setPosition(int offset)
offset
- the offset for the next lex()
operation.LexerToken createLexerToken()
LexerToken
for a specific
Lexer
implementation to be passed in to the
lex()
call to be filled in.int lex(LexerToken lexedToken)
lexedToken
instance passed in to the
call.lexedToken
- the instance passed in where token info is storedlexedToken.getToken()
(for convenience)void backup()
lex()
will return the last token and offset information found.