com.endeca.portal.data.processing

Class DataProcessingUtils

java.lang.Object

com.endeca.portal.data.processing.DataProcessingUtils

public class DataProcessingUtils
extends java.lang.Object

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CONTEXTROOT
static java.lang.String	ERROR_LOG
static java.lang.String	HOST
static java.lang.String	PORT

Constructor Summary

Constructors

Constructor and Description
DataProcessingUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static boolean	acquireTransformLock(javax.portlet.PortletRequest request, long projectId, java.lang.String dataSetKey) Acquires a transform lock, when possible.
static java.util.List<java.lang.String>	getAllOtherCollectionNames(java.lang.String collectionName) Returns the list of collection names, excluding the specified one, found on the specified collection.
static CopyCollectionStatus	getCollectionCopyStatus(long projectId, java.lang.String projectSpecificSourceCollectionName) Ascertains whether a collection copy exists or if a job is currently running to create such a copy for a given source collection.
static java.lang.String	getDataDomainName(java.lang.String responsePath)
static java.lang.String	getEndecaServerPath()
static boolean	getEndecaServerSSLEnabled()
static java.lang.String	getErrorMessage(java.lang.String logPath)
static java.util.List<UserPersistence>	getPersistedTransformLock(java.lang.String dataSetKey, long projectId) Gets the transform lock for the current project data set from the UserPersistence store.
static void	invokeApplyToDataSetWorkflow(long projectId, java.lang.String projectName, java.lang.String projectSpecificCollectionName, java.lang.String targetCollectionName, ApplyToSampleProgressInfo progressInfo, RequestIds requestIds) This method kicks off a thread to call DP's invokeApplyTransformToDataSet() API that applies a transform script to a previously copied project-private collection.
static void	invokeCurateDataSetWorkflow(CollectionBaseView dataSetToCurate, long projectId, java.lang.String projectName, long userId, java.util.Locale locale, RequestIds requestIds) This method kicks off a thread to call DP's invokeCreateCuratedDataSet() API that reloads a Hive source in full (not sampled) into a new collection, while optionally applying a transform.
static void	markInvalidAttributesInQueryState(DataSource ds, java.lang.String dataSetKey) This method examines each QueryFunction in the current QueryState that matches the given data set key, and marks their invalid flags based on the following criteria: (1) If the corresponding attribute cannot be found in the current data set, mark invalid as true (2) If the corresponding attribute's type does not match the QueryFunction's attribute type, mark invalid as true (3) Otherwise, mark invalid as false If any QueryFunction's invalidity is altered, the QueryState is updated to reflect that.
static void	replaceReloadedDataset(javax.portlet.PortletRequest request, CollectionBaseView baseView) For a data set that has been reloaded, switches over to use the replacement collection.
static void	rollbackAppliedTransforms(javax.portlet.PortletRequest request, java.lang.String dataSetKey)
static void	setTransformsAppliedStatus(long groupId, java.lang.String dataSetKey, boolean apply)
static java.util.Map<java.lang.String,java.lang.String>	splitPath(java.lang.String path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HOST

public static final java.lang.String HOST

See Also:

Constant Field Values

PORT

public static final java.lang.String PORT

See Also:

Constant Field Values

CONTEXTROOT

public static final java.lang.String CONTEXTROOT

See Also:

Constant Field Values

ERROR_LOG

public static final java.lang.String ERROR_LOG

See Also:

Constant Field Values

Constructor Detail

DataProcessingUtils

public DataProcessingUtils()

Method Detail

getEndecaServerPath

public static java.lang.String getEndecaServerPath()
                                            throws javax.portlet.PortletException

Throws:

javax.portlet.PortletException

getEndecaServerSSLEnabled

public static boolean getEndecaServerSSLEnabled()
                                         throws javax.portlet.PortletException

Throws:

javax.portlet.PortletException

splitPath

public static java.util.Map<java.lang.String,java.lang.String> splitPath(java.lang.String path)

getDataDomainName

public static java.lang.String getDataDomainName(java.lang.String responsePath)
                                          throws javax.portlet.PortletException

Throws:

javax.portlet.PortletException

getErrorMessage

public static java.lang.String getErrorMessage(java.lang.String logPath)
                                        throws javax.portlet.PortletException

Throws:

javax.portlet.PortletException

setTransformsAppliedStatus

public static void setTransformsAppliedStatus(long groupId,
                              java.lang.String dataSetKey,
                              boolean apply)
                                       throws TransformException

Throws:

TransformException

markInvalidAttributesInQueryState

public static void markInvalidAttributesInQueryState(DataSource ds,
                                     java.lang.String dataSetKey)
                                              throws KeyNotFoundException,
                                                     DataSourceException

This method examines each QueryFunction in the current QueryState that matches the given data set key, and marks their invalid flags based on the following criteria: (1) If the corresponding attribute cannot be found in the current data set, mark invalid as true (2) If the corresponding attribute's type does not match the QueryFunction's attribute type, mark invalid as true (3) Otherwise, mark invalid as false If any QueryFunction's invalidity is altered, the QueryState is updated to reflect that. This method leaves room for future updates to the logic to determine if a QueryFunction is invalid.

Parameters:

ds -

dataSetkey -

Throws:

KeyNotFoundException

DataSourceException

invokeCurateDataSetWorkflow

public static void invokeCurateDataSetWorkflow(CollectionBaseView dataSetToCurate,
                               long projectId,
                               java.lang.String projectName,
                               long userId,
                               java.util.Locale locale,
                               RequestIds requestIds)
                                        throws javax.portlet.PortletException

This method kicks off a thread to call DP's invokeCreateCuratedDataSet() API that reloads a Hive source in full (not sampled) into a new collection, while optionally applying a transform.

Parameters:

projectId - numeric identifier of the Studio project within which this transform will run

projectName - name of this Studio project used in EDP logging

projectSpecificCollectionName - collection name used by the project to bind to the data set being transformed

targetCollectionName - the "inventoried"/"native" name of a previously copied collection being transformed

progressInfo - persisted object that tracks workflow state

Throws:

java.lang.Exception - if something goes wrong either setting up or invoking the workflow

javax.portlet.PortletException

invokeApplyToDataSetWorkflow

public static void invokeApplyToDataSetWorkflow(long projectId,
                                java.lang.String projectName,
                                java.lang.String projectSpecificCollectionName,
                                java.lang.String targetCollectionName,
                                ApplyToSampleProgressInfo progressInfo,
                                RequestIds requestIds)
                                         throws java.lang.Exception

This method kicks off a thread to call DP's invokeApplyTransformToDataSet() API that applies a transform script to a previously copied project-private collection.

Parameters:

projectId - numeric identifier of the Studio project within which this transform will run

projectName - name of this Studio project used in EDP logging

projectSpecificCollectionName - collection name used by the project to bind to the data set being transformed

targetCollectionName - the "inventoried"/"native" name of a previously copied collection being transformed

progressInfo - persisted object that tracks workflow state

Throws:

java.lang.Exception - if something goes wrong either setting up or invoking the workflow

getCollectionCopyStatus

public static CopyCollectionStatus getCollectionCopyStatus(long projectId,
                                           java.lang.String projectSpecificSourceCollectionName)

Ascertains whether a collection copy exists or if a job is currently running to create such a copy for a given source collection.

Parameters:

projectId - a project ID indicating the Studio project (Group)

projectSpecificSourceCollectionName - project-specific (suffixed) collection name used by the project

Returns:

CollectionCopyState containing information about the collection copy and associated job

rollbackAppliedTransforms

public static void rollbackAppliedTransforms(javax.portlet.PortletRequest request,
                             java.lang.String dataSetKey)
                                      throws javax.portlet.PortletException,
                                             TransformException

Throws:

javax.portlet.PortletException

TransformException

replaceReloadedDataset

public static void replaceReloadedDataset(javax.portlet.PortletRequest request,
                          CollectionBaseView baseView)
                                   throws DataSourceException,
                                          KeyNotFoundException

For a data set that has been reloaded, switches over to use the replacement collection. If this replacement is successful, a database entry is created as a signal to affected projects. Note this method is intentionally marked synchronized (on static class). This is to prevent two concurrent reload procedures from running at the same time and potentially stepping on each other. TODO: consider implementing a String-based lock and lock on replacementCollectionName instead.

Parameters:

request -

baseView -

Throws:

DataSourceException

KeyNotFoundException

getAllOtherCollectionNames

public static java.util.List<java.lang.String> getAllOtherCollectionNames(java.lang.String collectionName)
                                                                   throws DataSourceException,
                                                                          KeyNotFoundException

Returns the list of collection names, excluding the specified one, found on the specified collection.

Throws:

KeyNotFoundException - if the collection by the specified name is not found

DataSourceException

acquireTransformLock

public static boolean acquireTransformLock(javax.portlet.PortletRequest request,
                           long projectId,
                           java.lang.String dataSetKey)
                                    throws javax.portlet.PortletException

Acquires a transform lock, when possible. When the lock acquisition is successful, this method returns true, notifying the caller that it may continue. Otherwise false is returned, notifying the caller that it should return a failure message. 1. Check for existing locks 1a. if there is a lock more than 30 minutes old, remove it 1b. if we own the oldest lock update it, return true, if not return false, if no locks, continue 2. Attempt to get lock 2a. put in db our intent 2b. get list of intents 2c. if ours is the oldest return true, if not or tied return false

Parameters:

request -

projectId -

dataSetKey -

Returns:

Throws:

javax.portlet.PortletException

getPersistedTransformLock

public static java.util.List<UserPersistence> getPersistedTransformLock(java.lang.String dataSetKey,
                                                        long projectId)
                                                                 throws javax.portlet.PortletException

Gets the transform lock for the current project data set from the UserPersistence store.

Parameters:

dataSetKey -

projectId -

Returns:

Throws:

javax.portlet.PortletException