Skip navigation links

oracle.javatools.parser.java.v2.scanner
Class TokenArray

java.lang.Object
oracle.javatools.parser.java.v2.scanner.TokenArray

public final class TokenArray

extends java.lang.Object

A precalculated array of tokens. Pay the cost of lexing the first time in order to calculate the array of tokens. All subsequent lexing is a matter of array accesses.

Field Summary

int	tokenCount The number of tokens.
int[]	tokenEnds The (dense) array of token end offsets.
int[]	tokenStarts The (dense) array of token start offsets.
short[]	tokenValues The (dense) array of token values.

Constructor Summary

TokenArray(FastLexer lexer)

Method Summary

void	adjust(int offset) Adjusts every offset in the token array by the given parameter.
int	offset2index(int query, boolean biasAfter) Translates a TextBuffer offset into a TokenArray index with the given bias.
int	search(int query)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tokenCount

public final int tokenCount

The number of tokens.

tokenValues

public final short[] tokenValues

The (dense) array of token values.

tokenStarts

public final int[] tokenStarts

The (dense) array of token start offsets.

tokenEnds

public final int[] tokenEnds

The (dense) array of token end offsets.

Constructor Detail

TokenArray

public TokenArray(FastLexer lexer)

Method Detail

adjust

public void adjust(int offset)

Adjusts every offset in the token array by the given parameter.

search

public int search(int query)

Parameters:

query - The start offset to be searched for.

Returns:

If the query finds a matching start offset, this returns the index of the token. If the query is in whitespace, the index of the next token is returned. If the query is before 0, then 0 is returned. If the query is past EOF, then the index of EOF is returned. Else, the negative of the previous start offset's index is returned. If the offset is in the middle of the an existing token or at the end of an existing token (but not at the start of the next), that means the index of said token is returned.

Throws:

java.lang.ArrayIndexOutOfBoundsException - Only if Arrays.binarySearch is wrong or if my array data got corrupted.

offset2index

public int offset2index(int query,
                        boolean biasAfter)

Translates a TextBuffer offset into a TokenArray index with the given bias. The bias is relevant only if there's a collision (that is, if the query lies between two tokens that are not separated by whitespace).

Assumes that all tokens are non-empty (except for the special EOF token). If there are empty tokens (besides the special EOF token), then no guarantee is made regarding the return value.

Parameters:

query - The offset to be searched for.

biasAfter - True means use an after-bias. Otherwise, use a before-bias. After-bias means choose the next token. Before-bias means choose the previous.

Returns:

If the offset lies within a token's boundaries (inclusive), this returns the index of that token. If the offset lies on the boundary between two tokens, then the input bias determines which one is returned.

Note that whereas EOF is a proper token (the index of EOF is tokenCount-1), start-of-file is not. So, for clarity, if the offset is 0 and the first token is at 0, then 0 is returned, regardless of bias.

If the query is in whitespace, then supposing the query lies between tokens i-1 and i, this will return -(i+1). (This matches the behavior of the Arrays.binarySearch() methods).

To do the math for you, that means that: If the query is before the first token (assuming the first token is not at start-of-file), -1 is returned. If the query is between the last non-EOF token and EOF itself (that is, the file's trailing whitespace), then -tokenCount is returned.

Skip navigation links