public final class Tokenizer
extends java.lang.Object
tokens
that enable simple lexical analysis.
If a subclass of Token is required rather than the default Token
a token factory can be registered which will create the tokens to be added
to the list.Modifier and Type | Class and Description |
---|---|
static class |
Tokenizer.Config
Configuration object that drives the tokenization
|
static interface |
Tokenizer.TokenFactory
Token factory which will create subclass of Token if required
|
Constructor and Description |
---|
Tokenizer(Tokenizer.Config config) |
Modifier and Type | Method and Description |
---|---|
Token |
getEndMarker() |
Token |
getFirst() |
Token |
getStartMarker() |
public Tokenizer(Tokenizer.Config config)