public static final class Tokenizer.Config
extends java.lang.Object
Constructor and Description |
---|
Config() |
Modifier and Type | Method and Description |
---|---|
void |
addTokens(Token.Type type,
java.util.Collection<java.lang.String> list)
Register extra Strings against a given Token type.
|
void |
setFactory(Tokenizer.TokenFactory factory)
Register a Token factory, enabling subclasses of Token to be created by
the Tokenizer.
|
void |
setMaxTokens(java.lang.Integer maxTokens)
Set a maximum number of tokens to create.
|
void |
setMultiLineCommentEnd(java.lang.String mlcEnd)
Override the default end of multiline comment characters ( */ )
|
void |
setMultiLineCommentStart(java.lang.String mlcStart)
Override the default start of multiline comment characters ( /* )
|
void |
setReader(java.io.Reader reader)
A Reader providing the source to be tokenized.
|
void |
setSingleLineCommentStart(java.lang.String slcStart)
Override the default start of single line comment characters ( -- )
|
void |
setSource(java.lang.String source)
Set the source to be tokenized.
|
public void setSource(java.lang.String source)
setSource(java.lang.String)
or setReader(java.io.Reader)
should be called. If neither are set, no Tokens are
created. If both, the source set in setSource(java.lang.String)
takes precedence.source
- The string to be tokenizedpublic void setReader(java.io.Reader reader)
setSource(java.lang.String)
or setReader(java.io.Reader)
should be called. If neither are set, no Tokens are
created. If both, the source set in setSource(java.lang.String)
takes precedence.reader
- The Reader providing the string to be tokenizedpublic void setFactory(Tokenizer.TokenFactory factory)
factory
- The token factory.public void setSingleLineCommentStart(java.lang.String slcStart)
slcStart
- the character sequence to indicate the start of a single
line commentpublic void setMultiLineCommentStart(java.lang.String mlcStart)
mlcStart
- the character sequence to indicate the start of a
multiline commentpublic void setMultiLineCommentEnd(java.lang.String mlcEnd)
mlcEnd
- the character sequence to indicate the end of a
multiline commentpublic void setMaxTokens(java.lang.Integer maxTokens)
maxTokens
- The maximum number of tokens to create.public void addTokens(Token.Type type, java.util.Collection<java.lang.String> list)