oracle.javatools.parser.java.v2.scanner

Class TokenArray

java.lang.Object

oracle.javatools.parser.java.v2.scanner.TokenArray

public final class TokenArray
extends java.lang.Object

A precalculated array of tokens. Pay the cost of lexing the first time in order to calculate the array of tokens. All subsequent lexing is a matter of array accesses.

Field Summary

Fields

Modifier and Type	Field and Description
int	tokenCount The number of tokens.
int[]	tokenEnds The (dense) array of token end offsets.
int[]	tokenStarts The (dense) array of token start offsets.
short[]	tokenValues The (dense) array of token values.

Constructor Summary

Constructors

Constructor and Description
TokenArray(FastLexer lexer)

Method Summary

Modifier and Type	Method and Description
void	adjust(int offset) Adjusts every offset in the token array by the given parameter.
int	findTokenIndexAtOffset(int offset) Get the index of the token that encloses an offset.
int	offset2index(int query, boolean biasAfter) Translates a TextBuffer offset into a TokenArray index with the given bias.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tokenCount

public final int tokenCount

The number of tokens.

tokenValues

public final short[] tokenValues

The (dense) array of token values.

tokenStarts

public final int[] tokenStarts

The (dense) array of token start offsets.

tokenEnds

public final int[] tokenEnds

The (dense) array of token end offsets.

Constructor Detail

TokenArray

public TokenArray(FastLexer lexer)

Method Detail

adjust

public void adjust(int offset)

Adjusts every offset in the token array by the given parameter.

findTokenIndexAtOffset

public int findTokenIndexAtOffset(int offset)

Get the index of the token that encloses an offset.

Parameters:

offset - The offset to be searched for.

Returns:

If the offset is inside a token, return the index of that token. If the offset is in between tokens, return -(insertion point) - 1, where the insertion point is the index of the token after the offset.

offset2index

public int offset2index(int query,
                        boolean biasAfter)

Translates a TextBuffer offset into a TokenArray index with the given bias. The bias is relevant only if there's a collision (that is, if the query lies between two tokens that are not separated by whitespace).

Assumes that all tokens are non-empty (except for the special EOF token). If there are empty tokens (besides the special EOF token), then no guarantee is made regarding the return value.

Parameters:

query - The offset to be searched for.

biasAfter - True means use an after-bias. Otherwise, use a before-bias. After-bias means choose the next token. Before-bias means choose the previous.

Returns:

If the offset lies within a token's boundaries (inclusive), this returns the index of that token. If the offset lies on the boundary between two tokens, then the input bias determines which one is returned.

Note that whereas EOF is a proper token (the index of EOF is tokenCount-1), start-of-file is not. So, for clarity, if the offset is 0 and the first token is at 0, then 0 is returned, regardless of bias.

If the query is in whitespace, then supposing the query lies between tokens i-1 and i, this will return -(i+1). (This matches the behavior of the Arrays.binarySearch() methods).

To do the math for you, that means that: If the query is before the first token (assuming the first token is not at start-of-file), -1 is returned. If the query is between the last non-EOF token and EOF itself (that is, the file's trailing whitespace), then -tokenCount is returned.