public final class TokenArray
extends java.lang.Object
Modifier and Type | Field and Description |
---|---|
int |
tokenCount
The number of tokens.
|
int[] |
tokenEnds
The (dense) array of token end offsets.
|
int[] |
tokenStarts
The (dense) array of token start offsets.
|
short[] |
tokenValues
The (dense) array of token values.
|
Constructor and Description |
---|
TokenArray(FastLexer lexer) |
Modifier and Type | Method and Description |
---|---|
void |
adjust(int offset)
Adjusts every offset in the token array by the given parameter.
|
int |
findTokenIndexAtOffset(int offset)
Get the index of the token that encloses an offset.
|
int |
offset2index(int query,
boolean biasAfter)
Translates a TextBuffer offset into a TokenArray index with the
given bias.
|
public final int tokenCount
public final short[] tokenValues
public final int[] tokenStarts
public final int[] tokenEnds
public TokenArray(FastLexer lexer)
public void adjust(int offset)
public int findTokenIndexAtOffset(int offset)
offset
- The offset to be searched for.public int offset2index(int query, boolean biasAfter)
query
- The offset to be searched for.biasAfter
- True means use an after-bias. Otherwise,
use a before-bias. After-bias means choose the
next token. Before-bias means choose the previous.