Skip navigation links

oracle.search.sdk.crawler
Interface QueueService

public interface QueueService

QueueService is an interface used by a crawler plugin to submit/retrieve URL-related data.

Method Summary

int	enqueue(DocumentMetadata docNode) enqueue a document to the queue.
int	enqueue(String displayUrl, String accessUrl, int depth) enqueue a URL to the queue.
void	enqueueExistingURLs() put all previously crawled URLs into the queue.
void	enqueueExistingURLs(int statusCode) enqueue URLs that have the specified status code
void	enqueueNotIndexedURLs() put all previously crawled URLs that are not indexed into the queue.
DocumentMetadata	getNextItem() get the next URL from the queue.
int	getQueueSize() get the size of the queue
boolean	isEnqueued(String displayURL) check whether the given URL is in the queue or not
boolean	isQueueEmpty() check whether the queue is empty
void	purgeQueue() clean up the queue
void	purgeQueue(int statusCode) remove documents from the queue with the specified status code
void	removeUrlNotInQueue() remove any previously crawled URL that is not in the queue
void	stopQueueService() Stopping the queue service operation; any pending getNextItem request will be replied with null result

Method Detail

enqueue

public int enqueue(DocumentMetadata docNode)

enqueue a document to the queue. Use this method when there are attribute data or property to go along with the URL. Any of the metadata is optional except the display URL.

• If the display URL does not exist in the data source, it is created and put into the queue
• If the display URL is already in the queue then this is a no operation
• If the display URL is not in the queue but exist in the current data source then it is put back into the queue. Be careful about the possibility of a URL been put into the queue, processed, and been put into the queue again.

Parameters:

docNode - the document metadata

Returns:

0 if the enqueue is successful. Return 1 if this document is already exist in the queue.

enqueue

public int enqueue(String displayUrl,
                   String accessUrl,
                   int depth)

enqueue a URL to the queue. Use this method when there is only URL string to be enqueued. Access URL can be null. URL.

• If the display URL does not exist in the data source, it is created and put into the queue
• If the display URL is already in the queue then this is a no operation
• If the display URL is not in the queue but exist in the current data source then it is put back into the queue. Be careful about the possibility of a URL been put into the queue, processed, and got put into the queue again.

Parameters:

displayUrl - the display URL

accessUrl - the access URL, can be null

depth - depth of the document

Returns:

0 if the enqueue is successful. Return 1 if this document is already exist in the queue.

enqueueExistingURLs

public void enqueueExistingURLs()

put all previously crawled URLs into the queue.

enqueueExistingURLs

public void enqueueExistingURLs(int statusCode)

enqueue URLs that have the specified status code

Parameters:

statusCode - the document status code

enqueueNotIndexedURLs

public void enqueueNotIndexedURLs()

put all previously crawled URLs that are not indexed into the queue.

isQueueEmpty

public boolean isQueueEmpty()

check whether the queue is empty

Returns:

true if the queue is empty, false if not

isEnqueued

public boolean isEnqueued(String displayURL)

check whether the given URL is in the queue or not

Returns:

true if the URL exist in the queue, false if not

getNextItem

public DocumentMetadata getNextItem()

get the next URL from the queue. Call is blocked if the queue is empty. Null is returned if there is no possibility of filling the queue by other thread.

Returns:

the document metadata

getQueueSize

public int getQueueSize()

get the size of the queue

Returns:

the queue size

purgeQueue

public void purgeQueue()

clean up the queue

purgeQueue

public void purgeQueue(int statusCode)

remove documents from the queue with the specified status code

Parameters:

statusCode - the document status code

stopQueueService

public void stopQueueService()

Stopping the queue service operation; any pending getNextItem request will be replied with null result

removeUrlNotInQueue

public void removeUrlNotInQueue()

remove any previously crawled URL that is not in the queue

Skip navigation links