Skip navigation links

oracle.javatools.parser.generic
Class GenericLexer

java.lang.Object
oracle.javatools.parser.AbstractLexer
oracle.javatools.parser.generic.GenericLexer

All Implemented Interfaces:

GenericTokens, Lexer

public class GenericLexer

extends AbstractLexer

implements GenericTokens

The GenericLexer is a concrete implementation of the Lexer interface that can be used to lex Language implementations. This allows you to quickly prototype a language without having to write an optimized language-specific Lexer implementation.

Nested Class Summary

Nested classes/interfaces inherited from class oracle.javatools.parser.AbstractLexer

AbstractLexer.DefaultLexerToken

Field Summary

protected int	endOffset The ending offset of the last token found.
protected KeywordTable	keywordTable The keyword table of the language.
protected Language	language The language that this lexer was created for.
protected int	lastToken The last token found from lex() operation.
protected boolean	skipComments Whether to skip comments or not.
protected int	startOffset The starting offset of the last token found.
protected boolean	useLastToken Whether to use the last token that was found.

Fields inherited from class oracle.javatools.parser.AbstractLexer

currentPos, textBuffer

Fields inherited from interface oracle.javatools.parser.generic.GenericTokens

TK_CLOSE_BRACE, TK_FLOAT, TK_IDENTIFIER, TK_INTEGER, TK_KEYWORD, TK_MULTI_COMMENT, TK_OPEN_BRACE, TK_OPERATOR, TK_QUOTES, TK_SINGLE_COMMENT

Fields inherited from interface oracle.javatools.parser.Lexer

TK_EOF, TK_NOT_FOUND

Constructor Summary

GenericLexer(Language language)
Constructs a GenericLexer for the specified language with a starting position of 0.

Method Summary

void	backup() Unlexes the last found token.
protected int	fillLexerToken(LexerToken lexedToken) Utility routine to fill in the LexerToken structure with the current token information we have.
int	lex(LexerToken lexedToken) Scans the text buffer at the current position and returns the token that was found.
void	setPosition(int offset) Sets the current lex (read) position to the given offset in the buffer.
void	setSkipComments(boolean skipComments) Sets whether the GenericLexer should generate tokens for comments.
static java.lang.String	tokenToString(int token) Utility routine to map the token to a string representation of the token (for debug printing.)
static java.lang.String	tokenToText(int token) Utility routine to map the token to the original text (if retrievable) of the token (for debug printing.)

Methods inherited from class oracle.javatools.parser.AbstractLexer

createLexerToken, getTextBuffer, setTextBuffer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

lastToken

protected int lastToken

The last token found from lex() operation.

startOffset

protected int startOffset

The starting offset of the last token found.

endOffset

protected int endOffset

The ending offset of the last token found.

useLastToken

protected boolean useLastToken

Whether to use the last token that was found.

skipComments

protected boolean skipComments

Whether to skip comments or not.

keywordTable

protected KeywordTable keywordTable

The keyword table of the language.

language

protected Language language

The language that this lexer was created for.

Constructor Detail

GenericLexer

public GenericLexer(Language language)

Constructs a GenericLexer for the specified language with a starting position of 0. Clients must call setTextBuffer() to initialize the text buffer used for the Lexer. To start lexing from an offset other than 0, call setPosition().

Method Detail

setSkipComments

public void setSkipComments(boolean skipComments)

Sets whether the GenericLexer should generate tokens for comments.

Parameters:

skipComments - true to ignore comments in token generation

lex

public int lex(LexerToken lexedToken)

Scans the text buffer at the current position and returns the token that was found. The token and offset information is also stored in the lexedToken instance passed in to the call.

Specified by:

lex in interface Lexer

Specified by:

lex in class AbstractLexer

Parameters:

lexedToken - the instance passed in where token info is stored

Returns:

the token that was found, same as calling lexedToken.getToken() (for convenience)

backup

public void backup()

Unlexes the last found token. The next call to lex() will return the last token and offset information found.

Specified by:

backup in interface Lexer

Specified by:

backup in class AbstractLexer

setPosition

public void setPosition(int offset)

Sets the current lex (read) position to the given offset in the buffer. It is the clients responsibility to ensure that this offset corresponds to the start of a token, otherwise unexpected (and incorrect) results may occur.

Specified by:

setPosition in interface Lexer

Overrides:

setPosition in class AbstractLexer

Parameters:

offset - the offset for the next lex() operation.

fillLexerToken

protected int fillLexerToken(LexerToken lexedToken)

Utility routine to fill in the LexerToken structure with the current token information we have. Returns the current token.

Parameters:

lexedToken - the client-specified LexerToken

Returns:

the last token found

tokenToString

public static java.lang.String tokenToString(int token)

Utility routine to map the token to a string representation of the token (for debug printing.)

Parameters:

token - the token to map

Returns:

a printable representation of the token

tokenToText

public static java.lang.String tokenToText(int token)

Utility routine to map the token to the original text (if retrievable) of the token (for debug printing.)

Parameters:

token - the token to map

Returns:

a printable representation of the token

Skip navigation links