oracle.javatools.parser

Interface Lexer

All Known Implementing Classes:

AbstractLexer, ArrayedLexer, CSSAtRuleLexer, CSSCounterStyleRuleLexer, CSSDeclarationsLexer, CSSFontFaceRuleLexer, CSSGenericAtRuleLexer, CSSImportRuleLexer, CSSLexer, CSSMediaRuleLexer, CSSNamespaceRuleLexer, CSSPageRuleLexer, CSSPseudoRuleSetLexer, CSSRegionRuleLexer, CSSRuleSetLexer, DefinitionLexer, DiffLexer, FastLexer, GenericLexer, HTMLLexer, JavadocLexer, JavaLexer, PlsqlLexer, PlsqlLexer, PropertiesLexer, SqlLexer, SqlPlsqlLexer, TagLexer

public interface Lexer

The Lexer interface describes the set of calls that need to be implemented by a Lexer for any given language. It is intended for providing lexical information only, and does not do any syntactical or semantic translation.

The underlying document model passed into any Lexer implementation must implement the ReadTextBuffer interface to insulate the Lexer from the actual model. Please refer to the ReadTextBuffer interface for more information on why this interface is being used.

Token information is returned via the LexerToken structure. This is for convenience so that multiple pieces of (possibly language-specific) information may be returned to clients in the same call as the lex() operation. Clients need to call the createLexerToken() of a given Lexer implementation to create the LexerToken object corresponding to that given language and Lexer.

See Also:

LexerToken, ReadTextBuffer

Field Summary

Fields

Modifier and Type	Field and Description
static int	TK_EOF Constant indicating that there are no more tokens remaining because the Lexer reached the end of the file.
static int	TK_NOT_FOUND Constant indicating a bad token value.

Method Summary

Methods

Modifier and Type	Method and Description
void	backup() Unlexes the last found token.
LexerToken	createLexerToken() Creates a new empty LexerToken for a specific Lexer implementation to be passed in to the lex() call to be filled in.
ReadTextBuffer	getTextBuffer() Gets the text buffer that is being used by this Lexer.
int	lex(LexerToken lexedToken) Scans the text buffer at the current position and returns the token that was found.
void	setPosition(int offset) Sets the current lex (read) position to the given offset in the buffer.
void	setTextBuffer(ReadTextBuffer buffer) Sets the text buffer that is to be used by this Lexer.

Field Detail

TK_EOF

static final int TK_EOF

Constant indicating that there are no more tokens remaining because the Lexer reached the end of the file.

See Also:

Constant Field Values

TK_NOT_FOUND

static final int TK_NOT_FOUND

Constant indicating a bad token value. This is returned by the KeywordTable when a keyword lookup is not found.

See Also:

Constant Field Values

Method Detail

setTextBuffer

void setTextBuffer(ReadTextBuffer buffer)

Sets the text buffer that is to be used by this Lexer.

Parameters:

buffer - the text buffer to use

getTextBuffer

ReadTextBuffer getTextBuffer()

Gets the text buffer that is being used by this Lexer.

Returns:

the text buffer being used

setPosition

void setPosition(int offset)

Sets the current lex (read) position to the given offset in the buffer. It is the clients responsibility to ensure that this offset corresponds to the start of a token, otherwise unexpected (and incorrect) results may occur.

Parameters:

offset - the offset for the next lex() operation.

createLexerToken

LexerToken createLexerToken()

Creates a new empty LexerToken for a specific Lexer implementation to be passed in to the lex() call to be filled in.

Returns:

a newly created LexerToken

lex

int lex(LexerToken lexedToken)

Scans the text buffer at the current position and returns the token that was found. The token and offset information is also stored in the lexedToken instance passed in to the call.

Parameters:

lexedToken - the instance passed in where token info is stored

Returns:

the token that was found, same as calling lexedToken.getToken() (for convenience)

backup

void backup()

Unlexes the last found token. The next call to lex() will return the last token and offset information found.