public class LoaderImpl extends GenericService
The loader works primarily from an IndexingOutputConfig
, which
specifies the repository, the repository items of interest, their
properties, and any customizing classes to use.
The basic algorythm for generating the documents is a recursive
breadth-first traversal of the hierarchy of repository items defined in
the XML definition file. As the loader traverses down the hierarchy it
pushes parent repository items onto the Context
object. In each
case, it uses PropertyAccessor.getMetaPropertyValue(atg.repository.search.indexing.Context, atg.repository.RepositoryItem, java.lang.String, atg.repository.search.indexing.specifier.PropertyTypeEnum)
to retrieve the child item property.
For a bulk/full load (see BulkLoaderImpl
), the loader performs a
query for top-level repository items, then iterates through any sub-items
specified, until it reaches a document-level item.
For an incremental load (see IncrementalLoaderImpl
), the loader starts
from an item, looks from parents, building Context stack as it traverses
up the hierarchy. Once it has a complete set of parents, it pops off any
sub-document level items, then then generates a document for the
document-level item.
In either case (bulk or incremental), once a document-level repository item is encountered, the loader begins collecting property values. It first collects any property values specified for parents (or other not document branches of the item hierarchy), and then continues the breadth-first traversal, collecting property values as it goes.
The loader then begins generating the XML document, once all the property
values are collected. It uses any specified PropertyFormatter
s and
PropertyValuesFilter
s to convert the collected property values
into string values.
Once the document has been created, the Loader then uses the DocumentSubmitter
to send the output to the indexer for indexing.
Typically the submitter will be an instance of the atg.repository.search.indexing.submitter.BpiDocumentSubmitter
, but
other submitters may be used for debugging.
The order of customizer class invocation is:
VariantProducer
(once per document, will be called again
if returns true)
PropertyAccessor
(once per item property,
once per text and meta property)
PropertyFormatter
(invoked on results of PropertyAccessor property
values). Maybe called before or after a given filter. Invoked
after property values have all been collected.
PropertyValuesFilter
(invoked after all property values are
gather). May be called before or after a give formatter. Invoked
after property values have all been collected.
Created: March 30 2005
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
CLASS_VERSION
Class version string
|
protected static java.lang.String |
DELETE_DOCUMENT
For performance monitor
|
protected static java.lang.String |
GET_GENERATOR_PROPERTY
For performance monitor
|
protected static java.lang.String |
GET_META_PROPERTY
For performance monitor
|
protected static java.lang.String |
GET_TEXT_PROPERTY
For performance monitor
|
protected static java.lang.String |
MY_RESOURCE_NAME
Resource name used for messages.
|
protected static java.lang.String |
PROCESS_PROPERTIES
For performance monitor
|
protected static java.lang.String |
REGENERATE_FOR_CHANGE
For performance monitor
|
protected static java.util.ResourceBundle |
sResourceBundle |
protected static java.lang.String |
START_QUERY
For performance monitor
|
SERVICE_INFO_KEY
DEFAULT_LOG_TRACE_STATUS
DEFAULT_LOG_DEBUG_STATUS, DEFAULT_LOG_ERROR_STATUS, DEFAULT_LOG_INFO_STATUS, DEFAULT_LOG_WARNING_STATUS
Constructor and Description |
---|
LoaderImpl() |
Modifier and Type | Method and Description |
---|---|
void |
addActiveContext(Context pContext)
Add a Context to the list of active contexts.
|
void |
adjustIncrementalQueue(IndexingOutputConfig pIndexingOutputConfig,
atg.search.index.IndexInfo pIndexInfo)
Pass-through method to adjust incremental queues.
|
protected void |
afterSessionStart(Context pContext)
This method is invoked immediately after a DocumentSubmitter session
is started.
|
protected void |
beforeSessionEnd(Context pContext)
This method is invoked immediately before a DocumentSubmitter session
is closed.
|
boolean |
cancelActiveContext(java.lang.String pConfigurationPath)
Cancel the active context with the specified name.
|
protected int |
claimGeneration(java.lang.String pConfigPath,
java.lang.String pContentId,
boolean pIncremental,
boolean pUseConfigurationClaims,
IndexingOutputConfig pIOC)
Claim a generation for updating.
|
protected javax.servlet.Servlet |
createAdminServlet()
Create the admininistration servlet.
|
protected Context |
createContext()
Create a new context.
|
protected atg.repository.search.indexing.OutputDocumentContent |
createEmptyOutputDocumentContent()
Create a new, empty, OutputDocumentContent object.
|
java.util.Set<atg.repository.MembershipContext> |
createMembershipContextSet(atg.search.multisite.SiteIndexInfo pSiteIndexInfo)
Create and return the MembershipContext set from the SiteIndexInfo
|
protected void |
deleteDocumentItem(RepositoryItem pItem,
Context pContext)
Remove the specified repository item pItem from the index.
|
protected boolean |
deleteOrExcludeItem(Context pContext,
atg.repository.search.indexing.specifier.OutputItemSpecifier pSpecifier,
RepositoryItem pItem) |
void |
doStartService()
This is called after a Service has been created, placed into the
naming hierarchy, and initialized with its configured property
values.
|
void |
doStopService()
When doStopService is called, exit out of our update activity thread, if any.
|
atg.repository.search.indexing.ContextStatus[] |
getActiveContextStatuses()
Get an array of ContextStatuses...
|
protected RepositoryItem[] |
getArrayFromMultiRepoItems(java.lang.Object pItems)
Return an array of RepositoryItems representing the repository
items the come from the property pItem.
|
ConfigStatePersister |
getConfigStatePersister()
Get the configuration state persister.
|
int |
getDocumentsPerTransaction()
Get the number of documents per transaction.
|
atg.repository.search.indexing.IncrementalItemQueue |
getIncrementalItemQueue()
Get the incremental item queue component, used for tracking
changes to repository items for the registered configurations.
|
int |
getLoggingInfoStatusCount()
Get how often to print out count/progress status information.
|
javax.transaction.TransactionManager |
getTransactionManager()
Get the transaction manager from the configStatePersister.
|
long |
getUpdateActivityTimeMillis()
Get the number of milliseconds to wait between updates
to lastActivity time.
|
boolean |
getUpdateLastActivityTimeInSeparateThread()
Whether to update last activity time in a separate thread.
|
protected void |
initContext(Context pContext,
IndexingOutputConfig pConfig,
int pClaimedGeneration,
atg.search.index.IndexInfo pIndexInfo)
Initialize the context, using the specified parameter
values.
|
protected boolean |
isConnectionRelated(java.lang.Throwable pEx)
Return true if the exception is, or was caused by, a
java.net.ConnectException.
|
boolean |
isEchoDocumentsToStdout()
Whether to echo documents to stdout.
|
boolean |
isItemLoggingDebug()
Whether to log matching attempts for document generation.
|
boolean |
isPrettyPrint()
Get whether the XHTML should be pretty printed (that is, indented).
|
boolean |
isPropertyLoggingDebug()
Whether to turn on logging debug for invidual property values.
|
static boolean |
isSubTypeOfItemDesc(RepositoryItemDescriptor pItemDesc,
RepositoryItemDescriptor pSuper)
Returns whether pItemDesc is a subtype (or equal to) pSuper.
|
boolean |
isTransactionPerDocument()
Whether to wrap all repository calls within a document
in a transaction.
|
atg.repository.search.indexing.LoaderImpl.UpdateActivityInfo |
nextQueuedUpdateActivity()
Return the next queued item for an update activity.
|
protected void |
outputAndSubmitDocument(Context pContext)
Create the XHTML file for the Context's OutputDocumentContent
and submit it using the DocumentSubmitterSession.
|
void |
popMembershipContexts(atg.search.index.IndexInfo pIndexInfo)
Deprecated.
Use two arg version, instead.
|
void |
popMembershipContexts(IndexingOutputConfig pIoc,
atg.search.index.IndexInfo pIndexInfo)
If this is a multisite indexing operation, pop site membership
context from into the MembershipContext.
|
void |
processItem(Context pContext,
RepositoryItem pItem,
atg.repository.search.indexing.specifier.OutputItemSpecifier pSpecifier)
Process the specified item.
|
protected void |
processMetaProperty(Context pContext,
RepositoryItem pItem,
atg.repository.search.indexing.specifier.OutputProperty pOutputProperty,
PropertyAccessor pPropAccessor,
java.lang.Object pPropertyValue)
Process a single meta property
|
protected void |
processParentProperties(Context pContext)
Process any of the properties of containing items on the
the context's stack.
|
protected void |
processProperties(Context pContext,
RepositoryItem pItem,
atg.repository.search.indexing.specifier.OutputItemSpecifier pSpecifier,
boolean pIsMeta)
Process any properties: that is collect any property values.
|
protected void |
processProperties(Context pContext,
RepositoryItem pItem,
atg.repository.search.indexing.specifier.OutputItemSpecifier pSpecifier,
atg.repository.search.indexing.specifier.OutputProperty[] pProperties,
boolean pIsMeta,
java.util.Map<atg.repository.search.indexing.specifier.OutputProperty,java.lang.Object> pGeneratorMap)
Process any properties: that is collect any property values.
|
protected void |
processTextProperty(Context pContext,
RepositoryItem pItem,
atg.repository.search.indexing.specifier.OutputProperty pOutputProperty,
PropertyAccessor pPropAccessor,
java.lang.Object pPropertyValue)
Process a single text property
|
boolean |
pushMembershipContexts(atg.search.index.IndexInfo pIndexInfo)
Deprecated.
Use two argument version, instead.
|
boolean |
pushMembershipContexts(IndexingOutputConfig pIoc,
atg.search.index.IndexInfo pIndexInfo)
If this is a multisite indexing operation, push site membership
context values into the MembershipContext.
|
boolean |
pushMembershipContextsIfUsedByRepository(IndexingOutputConfig pIoc,
atg.search.index.IndexInfo pIndexInfo)
If the repository used by the specified IOC uses the membership
context, push the context.
|
void |
queueUpdateActivity(atg.repository.search.indexing.LoaderImpl.UpdateActivityInfo pUpdateActivityInfo)
Queue an last activity time update change request.
|
protected void |
releaseContext(Context pContext)
Release any resources claimed by the specified context.
|
protected void |
releaseGeneration(java.lang.String pConfigPath,
java.lang.String pContentId,
int pGeneration,
boolean pCompletedSuccessfully,
boolean pIncremental,
boolean pUseConfigurationClaims)
Release the specified generation.
|
void |
removeActiveContext(Context pContext)
Remove a Context from the list of active contexts.
|
protected boolean |
repositoryUsesMembershipContext(IndexingOutputConfig pIoc)
Return true if the repository used by the IOC has a context
membership property.
|
protected void |
resetContextAfterDocument(Context pContext)
Reset the Context after a generating a document.
|
void |
setConfigStatePersister(ConfigStatePersister pConfigStatePersister)
Set the configuration state persister.
|
void |
setDocumentsPerTransaction(int pDocumentsPerTransaction)
Set the number of documents per transaction.
|
void |
setEchoDocumentsToStdout(boolean pEchoDocumentsToStdout)
Whether to echo documents to stdout.
|
void |
setIncrementalItemQueue(atg.repository.search.indexing.IncrementalItemQueue pIncrementalItemQueue)
Set the incremental item queue component, used for tracking
changes to repository items for the registered configurations.
|
void |
setItemLoggingDebug(boolean pItemLoggingDebug)
Whether to log matching attempts for document generation.
|
void |
setLoggingInfoStatusCount(int pLoggingInfoStatusCount)
Set how often to print out count/progress status information.
|
protected void |
setNextIncrementalGeneration(java.lang.String pConfigPath,
java.lang.String pContentId,
int pGeneration)
Release the specified generation.
|
void |
setPrettyPrint(boolean pPrettyPrint)
Set whether the XHTML should be pretty printed (that is, indented).
|
void |
setPropertyLoggingDebug(boolean pPropertyLoggingDebug)
Whether to turn on logging debug for invidual property values.
|
void |
setTransactionPerDocument(boolean pTransactionPerDocument)
Whether to wrap all repository calls within a document
in a transaction.
|
void |
setUpdateActivityTimeMillis(long pUpdateActivityTimeMillis)
Set the number of milliseconds to wait between updates
to lastActivity time.
|
void |
setUpdateLastActivityTimeInSeparateThread(boolean pUpdateLastActivityTimeInSeparateThread)
Whether to update last activity time in a separate thread.
|
protected java.lang.Object[] |
splitPropertyValue(atg.repository.search.indexing.specifier.OutputProperty pOutputProperty,
java.lang.Object pValue)
Split the property value into multiple values, if appropriate.
|
void |
updateActivity(Context pContext)
Update our last activity time.
|
addLogListener, getAbsoluteName, getAdminServlet, getAdminServletOutputStreamEncoding, getLoggingForVlogging, getLogListenerCount, getLogListeners, getName, getNameContext, getNucleus, getRoot, getServiceConfiguration, getServiceInfo, isAdminServletUseServletOutputStream, isLoggingDebug, isLoggingError, isLoggingInfo, isLoggingTrace, isLoggingWarning, isRunning, logDebug, logDebug, logDebug, logError, logError, logError, logInfo, logInfo, logInfo, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, nameContextElementBound, nameContextElementUnbound, removeLogListener, reResolveThis, resolveName, resolveName, resolveName, resolveName, sendLogEvent, setAdminServletOutputStreamEncoding, setAdminServletUseServletOutputStream, setLoggingDebug, setLoggingError, setLoggingInfo, setLoggingTrace, setLoggingWarning, setNucleus, setServiceInfo, startService, stopService
vlogDebug, vlogDebug, vlogDebug, vlogDebug, vlogDebugTrace, vlogError, vlogError, vlogError, vlogError, vlogInfo, vlogInfo, vlogInfo, vlogInfo, vlogTrace, vlogTrace, vlogTrace, vlogTrace, vlogWarning, vlogWarning, vlogWarning, vlogWarning
public static java.lang.String CLASS_VERSION
protected static final java.lang.String MY_RESOURCE_NAME
protected static final java.lang.String START_QUERY
protected static final java.lang.String PROCESS_PROPERTIES
protected static final java.lang.String GET_META_PROPERTY
protected static final java.lang.String GET_TEXT_PROPERTY
protected static final java.lang.String DELETE_DOCUMENT
protected static final java.lang.String GET_GENERATOR_PROPERTY
protected static final java.lang.String REGENERATE_FOR_CHANGE
protected static java.util.ResourceBundle sResourceBundle
public void setIncrementalItemQueue(atg.repository.search.indexing.IncrementalItemQueue pIncrementalItemQueue)
public atg.repository.search.indexing.IncrementalItemQueue getIncrementalItemQueue()
public void setConfigStatePersister(ConfigStatePersister pConfigStatePersister)
public ConfigStatePersister getConfigStatePersister()
public void setUpdateActivityTimeMillis(long pUpdateActivityTimeMillis)
public long getUpdateActivityTimeMillis()
public void setEchoDocumentsToStdout(boolean pEchoDocumentsToStdout)
public boolean isEchoDocumentsToStdout()
public void setPropertyLoggingDebug(boolean pPropertyLoggingDebug)
public boolean isPropertyLoggingDebug()
public void setItemLoggingDebug(boolean pItemLoggingDebug)
public boolean isItemLoggingDebug()
public void setTransactionPerDocument(boolean pTransactionPerDocument)
public boolean isTransactionPerDocument()
public void setLoggingInfoStatusCount(int pLoggingInfoStatusCount)
public int getLoggingInfoStatusCount()
public void setPrettyPrint(boolean pPrettyPrint)
public boolean isPrettyPrint()
public void setDocumentsPerTransaction(int pDocumentsPerTransaction)
public int getDocumentsPerTransaction()
public void setUpdateLastActivityTimeInSeparateThread(boolean pUpdateLastActivityTimeInSeparateThread)
public boolean getUpdateLastActivityTimeInSeparateThread()
protected boolean deleteOrExcludeItem(Context pContext, atg.repository.search.indexing.specifier.OutputItemSpecifier pSpecifier, RepositoryItem pItem) throws IndexingException
IndexingException
public void processItem(Context pContext, RepositoryItem pItem, atg.repository.search.indexing.specifier.OutputItemSpecifier pSpecifier) throws IndexingException
pContext
- the indexing context object.pRepositoryItem
- the repository item that corresponds to this item.pSpecifier
- the specifier for this item.IndexingException
public static boolean isSubTypeOfItemDesc(RepositoryItemDescriptor pItemDesc, RepositoryItemDescriptor pSuper)
pItemDesc
- The (presumably child) item descriporpSuper
- The item descriptor to test for parent-hood.public void updateActivity(Context pContext) throws IndexingException
IndexingException
protected RepositoryItem[] getArrayFromMultiRepoItems(java.lang.Object pItems)
pItems
- the object from which to extract repository
itemsprotected boolean isConnectionRelated(java.lang.Throwable pEx)
protected void outputAndSubmitDocument(Context pContext) throws IndexingException
IndexingException
protected void resetContextAfterDocument(Context pContext)
protected atg.repository.search.indexing.OutputDocumentContent createEmptyOutputDocumentContent()
protected void processParentProperties(Context pContext) throws IndexingException
pContext
- the context, currently set to the document
level.IndexingException
protected void processProperties(Context pContext, RepositoryItem pItem, atg.repository.search.indexing.specifier.OutputItemSpecifier pSpecifier, boolean pIsMeta)
pItem
. Gets the static set of
properties of pSpecifier
, and possibly additional
property values from the PropertyAccessor, if instance of
AdditionalPropertiesPropertyAccessor.pContext
- the current contextpItem
- the current item whose property values to collectpSpecifier
- the specifier corresponding to pItem
pIsMeta
- whether these properties are meta propertiesprotected void processMetaProperty(Context pContext, RepositoryItem pItem, atg.repository.search.indexing.specifier.OutputProperty pOutputProperty, PropertyAccessor pPropAccessor, java.lang.Object pPropertyValue)
protected void processTextProperty(Context pContext, RepositoryItem pItem, atg.repository.search.indexing.specifier.OutputProperty pOutputProperty, PropertyAccessor pPropAccessor, java.lang.Object pPropertyValue)
protected void processProperties(Context pContext, RepositoryItem pItem, atg.repository.search.indexing.specifier.OutputItemSpecifier pSpecifier, atg.repository.search.indexing.specifier.OutputProperty[] pProperties, boolean pIsMeta, java.util.Map<atg.repository.search.indexing.specifier.OutputProperty,java.lang.Object> pGeneratorMap)
pItem
.pContext
- the current contextpItem
- the current item whose property values to collectpSpecifier
- the specifier corresponding to pItem
pProperties
- the array of output properties to collectpIsMeta
- whether these properties are meta propertiesprotected java.lang.Object[] splitPropertyValue(atg.repository.search.indexing.specifier.OutputProperty pOutputProperty, java.lang.Object pValue)
pOutputProperty
- the output property the corresponds
to this value.pValue
- the property valueprotected Context createContext()
protected void initContext(Context pContext, IndexingOutputConfig pConfig, int pClaimedGeneration, atg.search.index.IndexInfo pIndexInfo)
pContext
- the context to initializepSession
- the document submitter sessionpConfig
- the indexing output configpClaimedGeneration
- the claimed generation for this
update.protected void releaseContext(Context pContext)
pContext
- The context to release.protected int claimGeneration(java.lang.String pConfigPath, java.lang.String pContentId, boolean pIncremental, boolean pUseConfigurationClaims, IndexingOutputConfig pIOC) throws IndexingException
ConfigStatePersister
to claim a given generation
for updating.pConfigPath
- the configuration path for for
indexing output config whose index shoudl be udpated.pContentId
- The content ID that uniquely identifies the
configuration.pIncremental
- whether we are claiming this
generation for an incremental update.pUseConfigurationClaims
- If true, use configuration claims
and enforce exclusive access to a configuration + content ID by
throwing exceptions if calims cannot be obtaind.IndexingException
protected void releaseGeneration(java.lang.String pConfigPath, java.lang.String pContentId, int pGeneration, boolean pCompletedSuccessfully, boolean pIncremental, boolean pUseConfigurationClaims) throws IndexingException
ConfigStatePersister
to release the specified generation.pConfigPath
- The configuration path of the generationpGeneration
- the numeric id of the generation, previously
returned by claimedGeneration.pCompletedSuccessfully
- whether the generation
completed successfully.pIncremental
- whether the generation is an
incremental generation.pUseConfigurationClaims
- If true, use configuration claims
and enforce exclusive access to a configuration + content ID by
throwing exceptions if calims cannot be obtaind.IndexingException
protected void setNextIncrementalGeneration(java.lang.String pConfigPath, java.lang.String pContentId, int pGeneration) throws IndexingException
ConfigStatePersister
to release the specified generation.pConfigPath
- The configuration path of the generationpGeneration
- the numeric id of the generation, previously
returned by claimedGeneration.pCompletedSuccessfully
- whether the generation
completed successfully.pIncremental
- whether the generation is an
incremental generation.IndexingException
public void addActiveContext(Context pContext)
pContext
- the context to add.public void removeActiveContext(Context pContext)
pContext
- the context to remove.public boolean cancelActiveContext(java.lang.String pConfigurationPath)
public atg.repository.search.indexing.ContextStatus[] getActiveContextStatuses()
protected javax.servlet.Servlet createAdminServlet()
createAdminServlet
in class GenericService
public javax.transaction.TransactionManager getTransactionManager()
protected void deleteDocumentItem(RepositoryItem pItem, Context pContext) throws IndexingException
IndexingException
protected void afterSessionStart(Context pContext) throws IndexingException
pContext
- the Context for the current indexing operation. The
DocumentSubmitterSession
is available as a property, as is
the IndexingOutputConfig
and other state information.IndexingException
protected void beforeSessionEnd(Context pContext) throws IndexingException
pContext
- the Context for the current indexing operation. The
DocumentSubmitterSession
is available as a property, as is
the IndexingOutputConfig
and other state information.IndexingException
public void queueUpdateActivity(atg.repository.search.indexing.LoaderImpl.UpdateActivityInfo pUpdateActivityInfo)
public atg.repository.search.indexing.LoaderImpl.UpdateActivityInfo nextQueuedUpdateActivity()
public void doStartService() throws ServiceException
GenericService
doStartService
in class GenericService
ServiceException
- if the Service had a problem starting uppublic void doStopService() throws ServiceException
doStopService
in class GenericService
ServiceException
- if the Service had a problem shutting downpublic void adjustIncrementalQueue(IndexingOutputConfig pIndexingOutputConfig, atg.search.index.IndexInfo pIndexInfo) throws IndexingException
IndexingException
public java.util.Set<atg.repository.MembershipContext> createMembershipContextSet(atg.search.multisite.SiteIndexInfo pSiteIndexInfo)
protected boolean repositoryUsesMembershipContext(IndexingOutputConfig pIoc)
@Deprecated public boolean pushMembershipContexts(atg.search.index.IndexInfo pIndexInfo)
pIndexInfo
- the object containing the context informationpublic boolean pushMembershipContexts(IndexingOutputConfig pIoc, atg.search.index.IndexInfo pIndexInfo)
pIndexInfo
- the object containing the context informationpublic boolean pushMembershipContextsIfUsedByRepository(IndexingOutputConfig pIoc, atg.search.index.IndexInfo pIndexInfo)
pIoc
- the indexing output config from which to obtain the
repository in questionpIndexInfo
- the object containing context/site information.@Deprecated public void popMembershipContexts(atg.search.index.IndexInfo pIndexInfo)
pIndexInfo
- the object containing the context informationpublic void popMembershipContexts(IndexingOutputConfig pIoc, atg.search.index.IndexInfo pIndexInfo)
pIoc
- the indexing output config.pIndexInfo
- the object containing the context information