com.endeca.portal.data.processing

Class DataProcessingUtils

java.lang.Object

com.endeca.portal.data.processing.DataProcessingUtils

public class DataProcessingUtils
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataProcessingUtils.LockAcquired This class is to store the lock acquiring states: - allLocksAcquired: all the locks have been acquired - primaryLockAcquired: the lock to the primary dataset has been acquired - joinDataSetLocksAquired: the locks to the join datasets in the existing transforms have been acquired
static class	DataProcessingUtils.TranformLockAcquired

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CONTEXTROOT
static java.lang.String	ERROR_LOG
static java.lang.String	HOST
static java.lang.String	PORT

Constructor Summary

Constructors

Constructor and Description
DataProcessingUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static DataProcessingUtils.LockAcquired	acquireTransformLocks(javax.portlet.PortletRequest request, long projectId, java.lang.String dataSetKey, boolean onlyConsiderUncommittedTransforms)
static DataProcessingUtils.LockAcquired	acquireTransformLocks(javax.portlet.PortletRequest request, long projectId, java.lang.String dataSetKey, boolean onlyConsiderUncommittedTransforms, java.util.List<Transform> newTransformsToAdd) Acquires tranform locks, including the locks for the join datasets if any.
static DataProcessingUtils.LockAcquired	acquireTransformLocks(javax.portlet.PortletRequest request, long projectId, java.lang.String dataSetKey, boolean onlyConsiderUncommittedTransforms, Transform newTransformToAdd)
static java.util.Map<java.lang.String,java.lang.String>	buildNotificationProperties(long groupId, long userId, java.lang.String dataSetDisplayName)
static java.util.Map<java.lang.String,java.lang.String>	buildNotificationProperties(long groupId, long userId, java.lang.String projectSpecificCollectionName, com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier nativeCollectionId, java.lang.String dataSetDisplayName)
static java.util.Map<java.lang.String,java.lang.String>	buildNotificationProperties(long groupId, long userId, java.lang.String projectSpecificCollectionName, com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier nativeCollectionId, java.lang.String dataSetDisplayName, java.lang.String projectDisplayName)
static java.util.Map<java.lang.String,java.lang.String>	buildNotificationProperties(long groupId, long userId, java.lang.String projectSpecificCollectionName, com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier nativeCollectionId, java.lang.String dataSetDisplayName, java.lang.String projectDisplayName, java.lang.String appWizardUuid)
static java.util.Map<java.lang.String,java.lang.String>	buildNotificationProperties(long groupId, long userId, java.lang.String projectSpecificCollectionName, com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier nativeCollectionId, java.lang.String dataSetDisplayName, java.lang.String projectDisplayName, java.lang.String appWizardUuid, java.lang.Long transformIndex)
static java.lang.String	convertToCollectionIdString(ApiMdexCollectionIdentifier cid) Converts collection identifier object to its string representation
static com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier	convertToCommonCollectionId(ApiMdexCollectionIdentifier cid) Converts collection identifier object from WorkflowManager API to edpCommon's API.
static CopyCollectionStatus	getCollectionCopyStatus(long projectId, java.lang.String projectSpecificSourceCollectionName) Ascertains whether a collection copy exists or if a job is currently running to create such a copy for a given source collection.
static java.lang.String	getDataDomainName(java.lang.String responsePath)
static boolean	getDataSetInProgressFlag(long projectId, java.lang.String projectSpecificCollectionName)
static java.lang.String	getEndecaServerPath()
static boolean	getEndecaServerSSLEnabled()
static java.lang.String	getErrorMessage(java.lang.String logPath)
static java.util.Set<java.lang.String>	getJoinDataSets(long projectId, java.lang.String dataSetKey, boolean onlyConsiderUncommittedTransforms)
static java.util.List<UserPersistence>	getPersistedTransformLock(java.lang.String dataSetKey, long projectId) Gets the transform lock for the current project data set from the UserPersistence store.
static void	handleEDPFailure(java.lang.String ecid, java.lang.Throwable exception)
static void	handleEDPFailure(Workflow event, java.lang.Throwable exception)
static void	invokeApplyToDataSetWorkflow(long projectId, java.lang.String projectName, long userId, java.lang.String projectSpecificCollectionName, com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier targetCollectionId, java.lang.String dataSetDisplayName, Workflow event, long toIndex) Invokes DP's ApplyTransformToDataSet workflow, which applies a transform script to a previously copied project-private collection.
static void	invokeCurateDataSetWorkflow(CollectionBaseView dataSetToCurate, long projectId, java.lang.String projectName, long userId, java.util.Locale locale, RequestIds requestIds, Workflow.WorkflowNameEnum workflowName) Invokes DP's CreateCuratedDataSet, which reloads a Hive source in full (not sampled) into a new collection, while optionally applying a transform.
static void	invokeDatasetRefreshWorkflow(ApiDatasetRefreshConfig refreshConfig, java.lang.String persistenceKey, Workflow event) Invokes DP's DatasetRefresh workflow, which replaces a public data set with a new one provisioned from a Hive source table.
static void	invokeProvisionOrRefreshWorkflow(ApiEdpWorkflowConfigBase workflowConfig, java.lang.String persistenceKey, Workflow event) Invokes either the ProvisionDataSetFromHive or DatasetRefresh EDP workflow, depending on which type of workflowConfig is specified.
static void	invokeProvisionWorkflow(ApiProvisionDataSetFromHiveConfig provisionConfig, java.lang.String persistenceKey, Workflow event) Invokes DP's ProvisionDataSetFromHive workflow, which creates a new public data set from a Hive source table.
static void	markInvalidAttributesInQueryState(DataSource ds, java.lang.String dataSetKey) This method examines each QueryFunction in the current QueryState that matches the given data set key, and marks their invalid flags based on the following criteria: (1) If the corresponding attribute cannot be found in the current data set, mark invalid as true (2) If the corresponding attribute's type does not match the QueryFunction's attribute type, mark invalid as true (3) Otherwise, mark invalid as false If any QueryFunction's invalidity is altered, the QueryState is updated to reflect that.
static void	replaceReloadedDataset(javax.portlet.PortletRequest request, CollectionBaseView baseView) For a data set that has been reloaded, switches over to use the replacement collection.
static void	rollbackAppliedTransforms(javax.portlet.PortletRequest request, java.lang.String dataSetKey)
static void	setDataSetInProgressFlag(long projectId, java.lang.String projectSpecificCollectionName, boolean onlyConsiderUncommittedTransforms, boolean inProgress)
static void	setTransformsAppliedStatus(long groupId, java.lang.String dataSetKey, long toIndex, boolean apply)
static java.util.Map<java.lang.String,java.lang.String>	splitPath(java.lang.String path)
static void	updateTransformAttributeMetadata(long projectId, java.lang.String dataSet, java.util.Map<java.lang.String,java.lang.Object> metadatas)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HOST

public static final java.lang.String HOST

See Also:

Constant Field Values

PORT

public static final java.lang.String PORT

See Also:

Constant Field Values

CONTEXTROOT

public static final java.lang.String CONTEXTROOT

See Also:

Constant Field Values

ERROR_LOG

public static final java.lang.String ERROR_LOG

See Also:

Constant Field Values

Constructor Detail

DataProcessingUtils

public DataProcessingUtils()

Method Detail

getEndecaServerPath

public static java.lang.String getEndecaServerPath()
                                            throws javax.portlet.PortletException

Throws:

javax.portlet.PortletException

getEndecaServerSSLEnabled

public static boolean getEndecaServerSSLEnabled()
                                         throws javax.portlet.PortletException

Throws:

javax.portlet.PortletException

splitPath

public static java.util.Map<java.lang.String,java.lang.String> splitPath(java.lang.String path)

getDataDomainName

public static java.lang.String getDataDomainName(java.lang.String responsePath)
                                          throws javax.portlet.PortletException

Throws:

javax.portlet.PortletException

getErrorMessage

public static java.lang.String getErrorMessage(java.lang.String logPath)
                                        throws javax.portlet.PortletException

Throws:

javax.portlet.PortletException

setTransformsAppliedStatus

public static void setTransformsAppliedStatus(long groupId,
                              java.lang.String dataSetKey,
                              long toIndex,
                              boolean apply)
                                       throws TransformException

Throws:

TransformException

markInvalidAttributesInQueryState

public static void markInvalidAttributesInQueryState(DataSource ds,
                                     java.lang.String dataSetKey)
                                              throws KeyNotFoundException,
                                                     DataSourceException

This method examines each QueryFunction in the current QueryState that matches the given data set key, and marks their invalid flags based on the following criteria: (1) If the corresponding attribute cannot be found in the current data set, mark invalid as true (2) If the corresponding attribute's type does not match the QueryFunction's attribute type, mark invalid as true (3) Otherwise, mark invalid as false If any QueryFunction's invalidity is altered, the QueryState is updated to reflect that. This method leaves room for future updates to the logic to determine if a QueryFunction is invalid.

Parameters:

ds -

dataSetkey -

Throws:

KeyNotFoundException

DataSourceException

invokeCurateDataSetWorkflow

public static void invokeCurateDataSetWorkflow(CollectionBaseView dataSetToCurate,
                               long projectId,
                               java.lang.String projectName,
                               long userId,
                               java.util.Locale locale,
                               RequestIds requestIds,
                               Workflow.WorkflowNameEnum workflowName)
                                        throws javax.portlet.PortletException

Invokes DP's CreateCuratedDataSet, which reloads a Hive source in full (not sampled) into a new collection, while optionally applying a transform.

Parameters:

dataSetToCurate - the data set to curate

projectId - numeric identifier of the Studio project within which this transform will run

projectName - name of this Studio project used in EDP logging

userId -

locale -

requestIds -

workflowName -

Throws:

java.lang.Exception - if something goes wrong either setting up or invoking the workflow

javax.portlet.PortletException

invokeApplyToDataSetWorkflow

public static void invokeApplyToDataSetWorkflow(long projectId,
                                java.lang.String projectName,
                                long userId,
                                java.lang.String projectSpecificCollectionName,
                                com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier targetCollectionId,
                                java.lang.String dataSetDisplayName,
                                Workflow event,
                                long toIndex)
                                         throws java.lang.Exception

Invokes DP's ApplyTransformToDataSet workflow, which applies a transform script to a previously copied project-private collection.

Parameters:

projectId - numeric identifier of the Studio project within which this transform will run

projectName - name of this Studio project used in EDP logging

projectSpecificCollectionName - collection name used by the project to bind to the data set being transformed

targetCollectionId - the "inventoried"/"native" collection id of a previously copied collection being transformed

dataSetDisplayName -

event -

toIndex - indicates a subset of the transform script to apply

Throws:

java.lang.Exception - if something goes wrong either setting up or invoking the workflow

invokeProvisionWorkflow

public static void invokeProvisionWorkflow(ApiProvisionDataSetFromHiveConfig provisionConfig,
                           java.lang.String persistenceKey,
                           Workflow event)

Invokes DP's ProvisionDataSetFromHive workflow, which creates a new public data set from a Hive source table.

Parameters:

provisionConfig -

persistenceKey - key for persisting instance-specific progress information about this provisioning attempt

event - Workflow event in context of this provisioning attempt

invokeDatasetRefreshWorkflow

public static void invokeDatasetRefreshWorkflow(ApiDatasetRefreshConfig refreshConfig,
                                java.lang.String persistenceKey,
                                Workflow event)

Invokes DP's DatasetRefresh workflow, which replaces a public data set with a new one provisioned from a Hive source table.

Parameters:

refreshConfig -

persistenceKey - key for persisting instance-specific progress information about this refresh attempt

event - Workflow event in context of this refresh attempt

invokeProvisionOrRefreshWorkflow

public static void invokeProvisionOrRefreshWorkflow(ApiEdpWorkflowConfigBase workflowConfig,
                                    java.lang.String persistenceKey,
                                    Workflow event)

Invokes either the ProvisionDataSetFromHive or DatasetRefresh EDP workflow, depending on which type of workflowConfig is specified.

Parameters:

workflowConfig -

persistenceKey - key for persisting instance-specific progress information about this workflow

event - Workflow event in context of this workflow

Throws:

java.lang.IllegalArgumentException - if workflowConfig is neither an ApiProvisionDataSetFromHiveConfig nor an ApiDatasetRefreshConfig

getCollectionCopyStatus

public static CopyCollectionStatus getCollectionCopyStatus(long projectId,
                                           java.lang.String projectSpecificSourceCollectionName)

Ascertains whether a collection copy exists or if a job is currently running to create such a copy for a given source collection.

Parameters:

projectId - a project ID indicating the Studio project (Group)

projectSpecificSourceCollectionName - project-specific (suffixed) collection name used by the project

Returns:

CollectionCopyState containing information about the collection copy and associated job

rollbackAppliedTransforms

public static void rollbackAppliedTransforms(javax.portlet.PortletRequest request,
                             java.lang.String dataSetKey)
                                      throws javax.portlet.PortletException,
                                             TransformException

Throws:

javax.portlet.PortletException

TransformException

replaceReloadedDataset

public static void replaceReloadedDataset(javax.portlet.PortletRequest request,
                          CollectionBaseView baseView)
                                   throws DataSourceException,
                                          KeyNotFoundException

For a data set that has been reloaded, switches over to use the replacement collection. If this replacement is successful, a database entry is created as a signal to affected projects. Note this method is intentionally marked synchronized (on static class). This is to prevent two concurrent reload procedures from running at the same time and potentially stepping on each other. TODO: consider implementing a String-based lock and lock on replacementCollectionName instead.

Parameters:

request -

baseView -

Throws:

DataSourceException

KeyNotFoundException

getPersistedTransformLock

public static java.util.List<UserPersistence> getPersistedTransformLock(java.lang.String dataSetKey,
                                                        long projectId)
                                                                 throws javax.portlet.PortletException

Gets the transform lock for the current project data set from the UserPersistence store.

Parameters:

dataSetKey -

projectId -

Returns:

Throws:

javax.portlet.PortletException

buildNotificationProperties

public static java.util.Map<java.lang.String,java.lang.String> buildNotificationProperties(long groupId,
                                                                           long userId,
                                                                           java.lang.String dataSetDisplayName)

buildNotificationProperties

public static java.util.Map<java.lang.String,java.lang.String> buildNotificationProperties(long groupId,
                                                                           long userId,
                                                                           java.lang.String projectSpecificCollectionName,
                                                                           com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier nativeCollectionId,
                                                                           java.lang.String dataSetDisplayName)

buildNotificationProperties

public static java.util.Map<java.lang.String,java.lang.String> buildNotificationProperties(long groupId,
                                                                           long userId,
                                                                           java.lang.String projectSpecificCollectionName,
                                                                           com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier nativeCollectionId,
                                                                           java.lang.String dataSetDisplayName,
                                                                           java.lang.String projectDisplayName)

buildNotificationProperties

public static java.util.Map<java.lang.String,java.lang.String> buildNotificationProperties(long groupId,
                                                                           long userId,
                                                                           java.lang.String projectSpecificCollectionName,
                                                                           com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier nativeCollectionId,
                                                                           java.lang.String dataSetDisplayName,
                                                                           java.lang.String projectDisplayName,
                                                                           java.lang.String appWizardUuid)

buildNotificationProperties

public static java.util.Map<java.lang.String,java.lang.String> buildNotificationProperties(long groupId,
                                                                           long userId,
                                                                           java.lang.String projectSpecificCollectionName,
                                                                           com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier nativeCollectionId,
                                                                           java.lang.String dataSetDisplayName,
                                                                           java.lang.String projectDisplayName,
                                                                           java.lang.String appWizardUuid,
                                                                           java.lang.Long transformIndex)

getDataSetInProgressFlag

public static boolean getDataSetInProgressFlag(long projectId,
                               java.lang.String projectSpecificCollectionName)

setDataSetInProgressFlag

public static void setDataSetInProgressFlag(long projectId,
                            java.lang.String projectSpecificCollectionName,
                            boolean onlyConsiderUncommittedTransforms,
                            boolean inProgress)

getJoinDataSets

public static java.util.Set<java.lang.String> getJoinDataSets(long projectId,
                                              java.lang.String dataSetKey,
                                              boolean onlyConsiderUncommittedTransforms)

acquireTransformLocks

public static DataProcessingUtils.LockAcquired acquireTransformLocks(javax.portlet.PortletRequest request,
                                                     long projectId,
                                                     java.lang.String dataSetKey,
                                                     boolean onlyConsiderUncommittedTransforms,
                                                     java.util.List<Transform> newTransformsToAdd)

Acquires tranform locks, including the locks for the join datasets if any.

Parameters:

request -

projectId -

dataSetKey -

onlyConsiderUncommittedTransforms - true: only locks the join datasets in the uncommitted transforms, besides the primary dataset and the join datasets in the newTransformsToAdd list; false: locks the join datasets in all the transforms (including the committed transforms), besides the primary dataset and the join datasets in the newTransformsToAdd list.

newTransformsToAdd -

Returns:

a LockAquired object

acquireTransformLocks

public static DataProcessingUtils.LockAcquired acquireTransformLocks(javax.portlet.PortletRequest request,
                                                     long projectId,
                                                     java.lang.String dataSetKey,
                                                     boolean onlyConsiderUncommittedTransforms,
                                                     Transform newTransformToAdd)

acquireTransformLocks

public static DataProcessingUtils.LockAcquired acquireTransformLocks(javax.portlet.PortletRequest request,
                                                     long projectId,
                                                     java.lang.String dataSetKey,
                                                     boolean onlyConsiderUncommittedTransforms)

updateTransformAttributeMetadata

public static void updateTransformAttributeMetadata(long projectId,
                                    java.lang.String dataSet,
                                    java.util.Map<java.lang.String,java.lang.Object> metadatas)

convertToCollectionIdString

public static java.lang.String convertToCollectionIdString(ApiMdexCollectionIdentifier cid)

Converts collection identifier object to its string representation

Parameters:

cid - a collection identifier object used by Workflow Manager API

Returns:

a String of the form .

convertToCommonCollectionId

public static com.oracle.endeca.pdi.concepts.mdex.MdexCollectionIdentifier convertToCommonCollectionId(ApiMdexCollectionIdentifier cid)

Converts collection identifier object from WorkflowManager API to edpCommon's API.

Parameters:

cid - a collection identifier object used by Workflow Manager API

Returns:

an MdexCollectionIdentifier representing the same

handleEDPFailure

public static void handleEDPFailure(java.lang.String ecid,
                    java.lang.Throwable exception)

handleEDPFailure

public static void handleEDPFailure(Workflow event,
                    java.lang.Throwable exception)