Processing Staged Data

Certain types of data that you import into Enterprise Learning Management through EIPs goes into intermediary staging tables before it is loaded into the system. A control value assigned to each row of incoming data identifies whether the data is from a full sync or an incremental sync EIP.

After an EIP imports data into the staging tables, you run a validation process to ensure that there are no data errors. Valid data can then be quickly loaded without the risk of errors. You can delete data in the staging tables as often as needed. The system uses a truncate command on supported platforms, and uses a delete command otherwise.

Staging tables are used for data that undergoes frequent changes or for which there is a high volume in Enterprise Learning Management. Steps for running the validation and load processes depend on whether you are importing data from a full sync EIP message or an incremental sync EIP message:

Full Sync Processing

The typical sequence for processing data from a full sync EIP is as follows:

Run the delete process to delete any existing full sync data from the staging tables.
Run the full sync EIP data publish message from the external application, such as PeopleSoft HR.
Run the full sync EIP data subscribe message from Enterprise Learning Management to import the data into the staging tables.
Ensure the full sync message has completed processing in PeopleSoft HCM and Enterprise Learning Management, and that no errors exist. If errors exist, resolve them before you continue.
Use the Process FullSync Data component to run the validation process.
View the log file for errors and ensure the data is corrected.

If you are satisfied, continue to the next step. Otherwise, correct the errors and return to step one.

Note: Make the corrections in the system of record. For example, if you are importing person data and information is missing, you may need to contact the HR administrator.
Run the upload process to load the validated data into the application tables.
Run the delete process to remove the full sync data from the staging tables.

Note: Rerunning a full sync EIP that uses a staging table does not delete data that resides only in Enterprise Learning Management, such as learner IDs, job codes, and learner department assignments.

Incremental Sync Processing

When incremental sync EIP messages import data into the staging tables in Enterprise Learning Management, a sequence log table captures the messages in sequence. When you initiate the validation process, data is validated and loaded in the order in which it was received.

When you process asynchronous data, the system automatically validates all incremental sync data in the staging table and loads the valid data. After the process is complete, you can check the message log files for errors. The system provides an option to automatically delete incremental sync data in the staging tables after the upload.

To ensure consistency between the data imported into Enterprise Learning Management and data in the source database, schedule the validation and load process for asynchronous data to run at a regular interval, such as every hour or every day. A minimum frequency of daily is recommended.

Pages Used to Process Staged Data

Page Name	Definition Name	Navigation	Usage
Process FullSync Data	LM_RNCTL_FULL_LOAD	select Set Up ELM, then select Process FullSync Data, then select Process FullSync Data	Run the Process FullSync Data (LM_LD_STGDAT ) Application Engine process to validate full sync data in the staging table, load validated data into Enterprise Learning Management, and delete data from the staging tables.
Process Asynchronous Data	LM_RNCTL_ASYNC	select Set Up ELM, then select Process Asynchronous Data, then select Process Asynchronous Data	Run the Process Asynchronous Data (LM_LD_ASYNC) Application Engine process to validate asynchronous data in the staging tables, load validated data into Enterprise Learning Management, and delete data from the staging tables.

Prerequisite for Processing Staged Data

Run the full sync EIP for the data you want to import before running any processes for staged data.

Process FullSync Data Page

Use the Process FullSync Data page (LM_RNCTL_FULL_LOAD) to run the Process FullSync Data (LM_LD_STGDAT ) Application Engine process to validate full sync data in the staging table, load validated data into Enterprise Learning Management, and delete data from the staging tables.

Navigation

select Set Up ELM, then select Process FullSync Data, then select Process FullSync Data

Image: Process FullSync Data page

This example illustrates the fields and controls on the Process FullSync Data page.

You can validate data, load validated data into Enterprise Learning Management, and delete data in the staging tables in separate processes or at the same time. The process runs in this order, for all selected check boxes: validate, upload, and delete.

Validate Staging Tables

Select to validate the imported data. The process looks for data with missing required values and missing rows. For example, a person record with no name record or an employment record with no job record. The system stamps all valid data with a status of valid, and reports all errors in the process scheduler log.

Upload Data to ELM

Select to move validated data from the staging tables to the Enterprise Learning Management application tables.

Note: The system loads all valid data, including data that is future effective-dated. However, learners who are future effective-dated do not appear in the Internal Learners search page until their effective date is current.

Process

Select the type of data to validate, load, or delete. Your selection here may cause other data entry fields to display on this page.

SetID From and SetID To

These fields appear only when you select Organization Data in the Process field. Enter the lowest and highest SetID values that you want to process.

Employee ID From and Employee ID To

These fields appear only when you select Person Data in the Process field. Enter the lowest and highest employee ID values that you want to process.

Create unique employee ID ranges and run these ranges in separate processes, being mindful of the human resource database's collation sequence (for example, how it sorts data). The process is designed to run multithreaded, utilizing temporary tables to reduce contention. The groups of IDs must be unique have no overlap.

Note: Consider having a technical person use SQL queries to verify that the ID ranges are unique and encompass all employee IDs. To identify your system's employee ID range, use the following SQL statements: SELECT MIN(LM_HR_EMPLID), MAX(LM_HR_EMPLID) FROM PS_LM_STG_PRS —for person load, and SELECT MIN(LM_HR_EMPLID), MAX(LM_HR_EMPLID) FROM PS_LM_STG_PRS_ATT —for workforce load.

Delete Staging Tables

Select to delete the full sync data from the staging tables. This option uses a truncate command on supported platforms, and uses a delete command otherwise.

Note: For organizations with large database volumes of approximately 50,000 employees or more, and no truncate support, have a database administrator drop and recreate the staging tables. Dropping and recreating the tables is more efficient than using the delete command.

Process Asynchronous Data Page

Use the Process Asynchronous Data page (LM_RNCTL_ASYNC) to run the Process Asynchronous Data (LM_LD_ASYNC) Application Engine process to validate asynchronous data in the staging tables, load validated data into Enterprise Learning Management, and delete data from the staging tables.

Navigation

select Set Up ELM, then select Process Asynchronous Data, then select Process Asynchronous Data

Image: Process Asynchronous Data page

This example illustrates the fields and controls on the Process Asynchronous Data page.

When you process asynchronous data, the system automatically validates all data in the staging table and loads the validated data. To have the system delete the incremental sync data in the staging table after it loads the data, select the check box labeled Automatically delete completed process.