Tokenizer (Oracle Fusion Middleware Java API Reference for Oracle Extension SDK)

java.lang.Object
- oracle.javatools.parser.css.Tokenizer

```
public class Tokenizer
extends java.lang.Object
```
This class represents a CSS Tokenizer - an object which decodes CSS lexical units.

Field Summary

Fields
Modifier and Type	Field and Description
`protected boolean`	`_currentCharIsEscaped` True if the current char is escaped
`protected int`	`blankCharacters` The characters to skip to create the string which represents the current token.
`protected char[]`	`buffer` The recording buffer.
`protected int`	`column` The current column.
`protected boolean`	`completeToken` like: "this is string" url("myurl") For: url("myurl") if completeToken == true: start = 0, end = 12 if completeToken == false: start = 5, end = 10
`protected int`	`current` The current char.
`protected int`	`currentEscapedCharacters` Number of escaped characters in the last token.
`protected int`	`end` The end offset of the last lexical unit.
`protected static int`	`ERROR`
`protected java.lang.String`	`errorMessage`
`protected int`	`escapedCharacters` Total number of escaped characters.
`protected int`	`line` The current line.
`protected int`	`position` The current position in the buffer.
`protected char[]`	`readBuffer` The reading buffer.
`protected int`	`readCount` The current read buffer count.
`protected java.io.Reader`	`reader` The reader.
`protected int`	`readPosition` The current position in the read buffer.
`protected boolean`	`replaceEscapedCharacters` Whether escaped characters should be replaced.
`protected boolean`	`returnErrorToken` Whether to return an error tokwn insted of throwing a ParserException.
`protected int`	`start` The start offset of the last lexical unit.
`protected int`	`type` The type of the current lexical unit.

Constructor Summary

Constructors
Constructor and Description

Tokenizer(java.io.Reader r)
Creates a new Tokenizer object.

Tokenizer(java.lang.String s)
Creates a new Tokenizer object.

Constructors
Constructor and Description
`Tokenizer(java.io.Reader r)` Creates a new Tokenizer object.
`Tokenizer(java.lang.String s)` Creates a new Tokenizer object.

Method Summary

Methods
Modifier and Type	Method and Description
`void`	`clearBuffer()` Clears the buffer.
`void`	`completeToken(boolean complete)` Whether to return complete tokens.
`int`	`currentChar()`
`protected int`	`dotNumber()` Scans the decimal part of a number.
`protected int`	`endGap()` Returns the end gap of the current lexical unit.
`protected void`	`escape()`
`protected void`	`escape(boolean moveToNextChar)` Scans an escape sequence, if one.
`char[]`	`getBuffer()` Returns the buffer used to store the chars.
`int`	`getColumn()` Returns the current column.
`protected int`	`getCurrentChar()`
`int`	`getEnd()` Returns the end offset of the last lexical unit.
`java.lang.String`	`getErrorMessage()` In case the tokenizer is setup to return error tokens instead of throwing exceptions, this method will return the message describing the last error token.
`int`	`getLine()` Returns the current line.
`int`	`getStart()` Returns the start offset of the last lexical unit.
`java.lang.String`	`getStringValue()` Returns the string representation of the current lexical unit.
`java.lang.String`	`getStringValue(int endOffset)`
`int`	`getType()` The current lexical unit type like defined in LexicalUnits.
`protected boolean`	`isCurrentCharEscaped()`
`protected static boolean`	`isEqualIgnoreCase(int i, char c)` Compares the given int with the given character, ignoring case.
`boolean`	`isNextEOF()` Returns true of the next char is EOF.
`void`	`moveForwardOneChar()`
`int`	`next()` Returns the next token.
`protected int`	`nextChar()` Sets the value of the current char to the next character or -1 if the end of stream has been reached.
`protected void`	`nextToken()` Returns the next token.
`protected int`	`number()` Scans a number.
`protected int`	`numberUnit(boolean integer)` Scans the unit of a number.
`void`	`replaceEscaped(boolean replace)`
`protected void`	`replaceEscapeSequance(int escStart, int escEnd)` current is positioned after first character
`void`	`reset()` Puts Tokenizer in the same state as it was after creation.
`boolean`	`returnCompleteToken()`
`protected boolean`	`returnErrorToken()`
`void`	`returnErrorToken(boolean useToken)`
`void`	`scanAtRule()` Scans a @rule value.
`int`	`scanTillOneOf(char char1, char char2)` Skips chars still one of char1 or char2 is found.
`protected void`	`setCurrentChar(int current)`
`protected void`	`setCurrentChar(int current, boolean escaped)`
`protected void`	`setCurrentCharEscaped(boolean currentCharIsEscaped)`
`void`	`setPosition(int newPosition, int newLine, int newColumn)` Puts Tokenizer in the same state as it was after scanning up to new position.
`void`	`setStartStringToCurrent()`
`protected int`	`string1()` Scans a single quoted string.
`protected int`	`string2()` Scans a double quoted string.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - reader
```
protected java.io.Reader reader
```
    The reader.
  - readBuffer
```
protected char[] readBuffer
```
    The reading buffer.
  - readPosition
```
protected int readPosition
```
    The current position in the read buffer.
  - readCount
```
protected int readCount
```
    The current read buffer count.
  - buffer
```
protected char[] buffer
```
    The recording buffer.
  - replaceEscapedCharacters
```
protected boolean replaceEscapedCharacters
```
    Whether escaped characters should be replaced.
  - completeToken
```
protected boolean completeToken
```
    like: "this is string" url("myurl") For: url("myurl") if completeToken == true: start = 0, end = 12 if completeToken == false: start = 5, end = 10
  - escapedCharacters
```
protected int escapedCharacters
```
    Total number of escaped characters.
  - currentEscapedCharacters
```
protected int currentEscapedCharacters
```
    Number of escaped characters in the last token.
  - returnErrorToken
```
protected boolean returnErrorToken
```
    Whether to return an error tokwn insted of throwing a ParserException.
  - ERROR
```
protected static final int ERROR
```
    See Also:
    Constant Field Values
  - line
```
protected int line
```
    The current line.
  - column
```
protected int column
```
    The current column.
  - current
```
protected int current
```
    The current char.
  - _currentCharIsEscaped
```
protected boolean _currentCharIsEscaped
```
    True if the current char is escaped
  - position
```
protected int position
```
    The current position in the buffer.
  - type
```
protected int type
```
    The type of the current lexical unit.
  - start
```
protected int start
```
    The start offset of the last lexical unit.
  - end
```
protected int end
```
    The end offset of the last lexical unit.
  - blankCharacters
```
protected int blankCharacters
```
    The characters to skip to create the string which represents the current token.
  - errorMessage
```
protected java.lang.String errorMessage
```
- Constructor Detail
  - Tokenizer
```
public Tokenizer(java.io.Reader r)
          throws ParseException
```
    Creates a new Tokenizer object.
    
    Parameters:
    r - The reader to scan.
    
    Throws:
    
    ParseException
  - Tokenizer
```
public Tokenizer(java.lang.String s)
          throws ParseException
```
    Creates a new Tokenizer object.
    
    Parameters:
    s - The reader to scan.
    
    Throws:
    
    ParseException
- Method Detail
  - returnErrorToken
```
public void returnErrorToken(boolean useToken)
```
  - replaceEscaped
```
public void replaceEscaped(boolean replace)
```
  - completeToken
```
public void completeToken(boolean complete)
```
    Whether to return complete tokens.
  - isNextEOF
```
public final boolean isNextEOF()
```
    Returns true of the next char is EOF.
    
    Returns:
  - reset
```
public void reset()
```
    Puts Tokenizer in the same state as it was after creation.
  - setPosition
```
public void setPosition(int newPosition,
               int newLine,
               int newColumn)
```
    Puts Tokenizer in the same state as it was after scanning up to new position. This call is allowed only when Tokenizer is setup to return complete tokens, and not to replace escape sequances. It is callers responibility to insure that new position correspond to the start of next token. Also, caller needs to set reader's offset to new position.
    
    Throws:
    
    java.lang.IllegalStateException
  - getBuffer
```
public char[] getBuffer()
```
    Returns the buffer used to store the chars.
  - getStart
```
public int getStart()
```
    Returns the start offset of the last lexical unit.
  - getEnd
```
public int getEnd()
```
    Returns the end offset of the last lexical unit.
  - clearBuffer
```
public void clearBuffer()
```
    Clears the buffer.
  - getStringValue
```
public java.lang.String getStringValue(int endOffset)
```
  - getStringValue
```
public java.lang.String getStringValue()
```
    Returns the string representation of the current lexical unit.
  - next
```
public int next()
         throws ParseException
```
    Returns the next token.
    
    Throws:
    
    ParseException
  - currentChar
```
public int currentChar()
```
  - replaceEscapeSequance
```
protected void replaceEscapeSequance(int escStart,
                         int escEnd)
```
    current is positioned after first character
    
    Parameters:
    escStart - offset of first character after '\'
  - isEqualIgnoreCase
```
protected static boolean isEqualIgnoreCase(int i,
                        char c)
```
    Compares the given int with the given character, ignoring case.
  - nextChar
```
protected int nextChar()
                throws java.io.IOException
```
    Sets the value of the current char to the next character or -1 if the end of stream has been reached.
    
    Throws:
    
    java.io.IOException
  - returnCompleteToken
```
public boolean returnCompleteToken()
```
  - returnErrorToken
```
protected boolean returnErrorToken()
```
  - getLine
```
public int getLine()
```
    Returns the current line.
  - getColumn
```
public int getColumn()
```
    Returns the current column.
  - getType
```
public int getType()
```
    The current lexical unit type like defined in LexicalUnits.
  - getErrorMessage
```
public java.lang.String getErrorMessage()
```
    In case the tokenizer is setup to return error tokens instead of throwing exceptions, this method will return the message describing the last error token.
  - scanAtRule
```
public void scanAtRule()
                throws ParseException
```
    Scans a @rule value. This method assumes that the current lexical unit is a at keyword. It is convinience method for going over unknown at-rules without looking in them.
    
    Throws:
    
    ParseException
  - scanTillOneOf
```
public int scanTillOneOf(char char1,
                char char2)
                  throws ParseException
```
    Skips chars still one of char1 or char2 is found.
    
    Throws:
    
    ParseException
  - moveForwardOneChar
```
public void moveForwardOneChar()
                        throws ParseException
```
    Throws:
    
    ParseException
  - setStartStringToCurrent
```
public void setStartStringToCurrent()
```
  - endGap
```
protected int endGap()
```
    Returns the end gap of the current lexical unit.
  - setCurrentChar
```
protected final void setCurrentChar(int current,
                  boolean escaped)
```
  - setCurrentChar
```
protected final void setCurrentChar(int current)
```
  - isCurrentCharEscaped
```
protected final boolean isCurrentCharEscaped()
```
  - setCurrentCharEscaped
```
protected final void setCurrentCharEscaped(boolean currentCharIsEscaped)
```
  - getCurrentChar
```
protected int getCurrentChar()
```
  - nextToken
```
protected void nextToken()
                  throws ParseException
```
    Returns the next token.
    
    Throws:
    
    ParseException
  - string1
```
protected int string1()
               throws java.io.IOException
```
    Scans a single quoted string.
    
    Returns:
    lexical unit type
    
    Throws:
    
    java.io.IOException
  - string2
```
protected int string2()
               throws java.io.IOException
```
    Scans a double quoted string.
    
    Returns:
    lexical unit type
    
    Throws:
    
    java.io.IOException
  - number
```
protected int number()
              throws java.io.IOException
```
    Scans a number.
    
    Throws:
    
    java.io.IOException
  - dotNumber
```
protected int dotNumber()
                 throws java.io.IOException
```
    Scans the decimal part of a number.
    
    Throws:
    
    java.io.IOException
  - numberUnit
```
protected int numberUnit(boolean integer)
                  throws java.io.IOException
```
    Scans the unit of a number.
    
    Throws:
    
    java.io.IOException
  - escape
```
protected final void escape()
                     throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - escape
```
protected void escape(boolean moveToNextChar)
               throws java.io.IOException
```
    Scans an escape sequence, if one.
    
    Parameters:
    moveToNextChar - If true, the method is allowed to move to the next char after processing the escaped char
    
    Throws:
    
    java.io.IOException

Class Tokenizer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

reader

readBuffer

readPosition

readCount

buffer

replaceEscapedCharacters

completeToken

escapedCharacters

currentEscapedCharacters

returnErrorToken

ERROR

line

column

current

_currentCharIsEscaped

position

type

start

end

blankCharacters

errorMessage

Constructor Detail

Tokenizer

Tokenizer

Method Detail

returnErrorToken

replaceEscaped

completeToken

isNextEOF

reset

setPosition

getBuffer

getStart

getEnd

clearBuffer

getStringValue

getStringValue

next

currentChar

replaceEscapeSequance

isEqualIgnoreCase

nextChar

returnCompleteToken

returnErrorToken

getLine

getColumn

getType

getErrorMessage

scanAtRule

scanTillOneOf

moveForwardOneChar

setStartStringToCurrent

endGap

setCurrentChar

setCurrentChar

isCurrentCharEscaped

setCurrentCharEscaped

getCurrentChar

nextToken

string1

string2

number

dotNumber

numberUnit

escape

escape