Skip navigation links

oracle.javatools.parser.css
Class Tokenizer

java.lang.Object
oracle.javatools.parser.css.Tokenizer

public class Tokenizer

extends java.lang.Object

This class represents a CSS Tokenizer - an object which decodes CSS lexical units.

Field Summary

protected int	blankCharacters The characters to skip to create the string which represents the current token.
protected char[]	buffer The recording buffer.
protected int	column The current column.
protected boolean	completeToken like: "this is string" url("myurl") For: url("myurl") if completeToken == true: start = 0, end = 12 if completeToken == false: start = 5, end = 10
protected int	current The current char.
protected int	currentEscapedCharacters Number of escaped characters in the last token.
protected int	end The end offset of the last lexical unit.
protected java.lang.String	errorMessage
protected int	escapedCharacters Total number of escaped characters.
protected int	line The current line.
protected int	position The current position in the buffer.
protected char[]	readBuffer The reading buffer.
protected int	readCount The current read buffer count.
protected java.io.Reader	reader The reader.
protected int	readPosition The current position in the read buffer.
protected boolean	replaceEscapedCharacters Whether escaped characters should be replaced.
protected boolean	returnErrorToken Whether to return an error tokwn insted of throwing a ParserException.
protected int	start The start offset of the last lexical unit.
protected int	type The type of the current lexical unit.

Constructor Summary

Tokenizer(java.io.Reader r)
Creates a new Tokenizer object.

Tokenizer(java.lang.String s)
Creates a new Tokenizer object.

Method Summary

void	clearBuffer() Clears the buffer.
void	completeToken(boolean complete) Whether to return complete tokens.
int	currentChar()
protected int	dotNumber() Scans the decimal part of a number.
protected int	endGap() Returns the end gap of the current lexical unit.
protected void	escape() Scans an escape sequence, if one.
char[]	getBuffer() Returns the buffer used to store the chars.
int	getColumn() Returns the current column.
int	getEnd() Returns the end offset of the last lexical unit.
java.lang.String	getErrorMessage() In case the tokenizer is setup to return error tokens instead of throwing exceptions, this method will return the message describing the last error token.
int	getLine() Returns the current line.
int	getStart() Returns the start offset of the last lexical unit.
java.lang.String	getStringValue() Returns the string representation of the current lexical unit.
int	getType() The current lexical unit type like defined in LexicalUnits.
protected static boolean	isEqualIgnoreCase(int i, char c) Compares the given int with the given character, ignoring case.
int	next() Returns the next token.
protected int	nextChar() Sets the value of the current char to the next character or -1 if the end of stream has been reached.
protected void	nextToken() Returns the next token.
protected int	number() Scans a number.
protected int	numberUnit(boolean integer) Scans the unit of a number.
void	replaceEscaped(boolean replace)
protected void	replaceEscapeSequance(int escStart, int escEnd) current is positioned after first character
void	reset() Puts Tokenizer in the same state as it was after creation.
void	returnErrorToken(boolean useToken)
void	scanAtRule() Scans a @rule value.
void	setPosition(int newPosition, int newLine, int newColumn) Puts Tokenizer in the same state as it was after scanning up to new position.
protected int	string1() Scans a single quoted string.
protected int	string2() Scans a double quoted string.
protected int	uriString1() Scans a single quoted string for URI.
protected int	uriString2() Scans a double quoted string for URI.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

reader

protected java.io.Reader reader

The reader.

line

protected int line

The current line.

column

protected int column

The current column.

current

protected int current

The current char.

readBuffer

protected char[] readBuffer

The reading buffer.

readPosition

protected int readPosition

The current position in the read buffer.

readCount

protected int readCount

The current read buffer count.

buffer

protected char[] buffer

The recording buffer.

position

protected int position

The current position in the buffer.

type

protected int type

The type of the current lexical unit.

start

protected int start

The start offset of the last lexical unit.

end

protected int end

The end offset of the last lexical unit.

blankCharacters

protected int blankCharacters

The characters to skip to create the string which represents the current token.

replaceEscapedCharacters

protected boolean replaceEscapedCharacters

Whether escaped characters should be replaced.

completeToken

protected boolean completeToken

like: "this is string" url("myurl") For: url("myurl") if completeToken == true: start = 0, end = 12 if completeToken == false: start = 5, end = 10

escapedCharacters

protected int escapedCharacters

Total number of escaped characters.

currentEscapedCharacters

protected int currentEscapedCharacters

Number of escaped characters in the last token.

returnErrorToken

protected boolean returnErrorToken

Whether to return an error tokwn insted of throwing a ParserException.

errorMessage

protected java.lang.String errorMessage

Constructor Detail

Tokenizer

public Tokenizer(java.io.Reader r)
          throws ParseException

Creates a new Tokenizer object.

Parameters:

r - The reader to scan.

Throws:

ParseException

Tokenizer

public Tokenizer(java.lang.String s)
          throws ParseException

Creates a new Tokenizer object.

Parameters:

s - The reader to scan.

Throws:

ParseException

Method Detail

returnErrorToken

public void returnErrorToken(boolean useToken)

replaceEscaped

public void replaceEscaped(boolean replace)

completeToken

public void completeToken(boolean complete)

Whether to return complete tokens.

reset

public void reset()

Puts Tokenizer in the same state as it was after creation.

setPosition

public void setPosition(int newPosition,
                        int newLine,
                        int newColumn)

Puts Tokenizer in the same state as it was after scanning up to new position. This call is allowed only when Tokenizer is setup to return complete tokens, and not to replace escape sequances. It is callers responibility to insure that new position correspond to the start of next token. Also, caller needs to set reader's offset to new position.

Throws:

java.lang.IllegalStateException

getLine

public int getLine()

Returns the current line.

getColumn

public int getColumn()

Returns the current column.

getBuffer

public char[] getBuffer()

Returns the buffer used to store the chars.

getStart

public int getStart()

Returns the start offset of the last lexical unit.

getEnd

public int getEnd()

Returns the end offset of the last lexical unit.

clearBuffer

public void clearBuffer()

Clears the buffer.

getType

public int getType()

The current lexical unit type like defined in LexicalUnits.

getStringValue

public java.lang.String getStringValue()

Returns the string representation of the current lexical unit.

getErrorMessage

public java.lang.String getErrorMessage()

In case the tokenizer is setup to return error tokens instead of throwing exceptions, this method will return the message describing the last error token.

scanAtRule

public void scanAtRule()
                throws ParseException

Scans a @rule value. This method assumes that the current lexical unit is a at keyword. It is convinience method for going over unknown at-rules without looking in them.

Throws:

ParseException

next

public int next()
         throws ParseException

Returns the next token.

Throws:

ParseException

currentChar

public int currentChar()

endGap

protected int endGap()

Returns the end gap of the current lexical unit.

nextToken

protected void nextToken()
                  throws ParseException

Returns the next token.

Throws:

ParseException

string1

protected int string1()
               throws java.io.IOException

Scans a single quoted string.

Returns:

lexical unit type

Throws:

java.io.IOException

string2

protected int string2()
               throws java.io.IOException

Scans a double quoted string.

Returns:

lexical unit type

Throws:

java.io.IOException

uriString1

protected int uriString1()
                  throws java.io.IOException

Scans a single quoted string for URI.

Returns:

lexical unit type

Throws:

java.io.IOException

uriString2

protected int uriString2()
                  throws java.io.IOException

Scans a double quoted string for URI.

Returns:

lexical unit type

Throws:

java.io.IOException

number

protected int number()
              throws java.io.IOException

Scans a number.

Throws:

java.io.IOException

dotNumber

protected int dotNumber()
                 throws java.io.IOException

Scans the decimal part of a number.

Throws:

java.io.IOException

numberUnit

protected int numberUnit(boolean integer)
                  throws java.io.IOException

Scans the unit of a number.

Throws:

java.io.IOException

escape

protected void escape()
               throws java.io.IOException

Scans an escape sequence, if one.

Throws:

java.io.IOException

replaceEscapeSequance

protected void replaceEscapeSequance(int escStart,
                                     int escEnd)

current is positioned after first character

Parameters:

escStart - offset of first character after '\'

isEqualIgnoreCase

protected static boolean isEqualIgnoreCase(int i,
                                           char c)

Compares the given int with the given character, ignoring case.

nextChar

protected int nextChar()
                throws java.io.IOException

Sets the value of the current char to the next character or -1 if the end of stream has been reached.

Throws:

java.io.IOException

Skip navigation links