5 Run jobs and check history

This section contains the following topics:

• View and edit your subscriptions to job status notifications
  
• Run transformations and view run history
  
• Run validation check batches and view run history
  
• Reprocess discrepancies that failed to be sent to InForm
  
• Check if data files loaded
  
• FAQs
  

View and edit your subscriptions to job status notifications

You can subscribe to receive email notifications about the completion status of jobs.

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. In the Studies title bar, click the Notify icon.
3. Select the checkbox for each type of job and completion status that you want to be notified about:

   • Job types: File loads, InForm loads, Transformations, Validation Checks

   • Completion statuses: Success, Warning, Failure

     Note:

     Your subscriptions for the current study appear as selected. An administrator can also subscribe or unsubscribe you from notifications.

   To stop receiving a particular type of notification, deselect its checkbox.

4. Click OK.

To subscribe or unsubscribe to notifications for other studies, select a different study and repeat the steps in this procedure.

Run transformations and view run history

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. Click the Transformations tab to open tables on the Target Model and Run History.
3. To view all jobs, click the View Full Job History icon.

   To view only recent jobs again, click the View Recent Jobs Only icon.

• View transformation job history
  
• Run a transformation
  
• Cancel a transformation job
  

View transformation job history

As described in Run transformations and view run history, you access transformation through the Transformations tab on the Home page. Transformations are displayed by the name of their target clinical data model.

• To view table transformations, click a transformation's node.

• To view run history and pending jobs, select a transformation in the upper pane.

• To view log files, click the icon in the column for the type of job:

  • Log: The most recent manually submitted job.

  • Triggered Job Log: The most recent triggered job.

  • Install Job Log: The most recent installation of the transformation.

Run a transformation

To run or schedule a transformation:

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. Click the Transformations tab to open tables on the Target Model and Run History.
3. Select a transformation and click the Submit Job icon.

   Tip:

   The Submit Job icon is not active if the selected transformation has not been installed. Check the Install Status column.

   If the installation status of the transformation is Warning, you may still be able to run the batch. Check the installation log file.

4. Enter values:

   • Submission Mode: Select one:

     • Full mode includes data deletion. Use Full mode only if you are confident that you are reloading all current data.

     • Incremental mode is faster and does not include data deletion.

     If you are submitting a transformation for a single table and the table is defined with Unit of Work processing, select:

     • Full UOW includes data deletion. Use Full UOW only if you are confident that you are reloading all current data for each subject or subject visit that has any data included.

     • Incremental UOW is faster and does not include data deletion.

       Tip:

       Set up regular Incremental loads at frequent intervals and do Full loads at longer intervals.

   • Force Execution: Select if you want to run the job even though the source data currency, parameter values, and the version number of the program(s) have not changed since the last run. The system uses Full mode regardless of the Submission Mode setting. Full mode includes data deletion.

     If not selected and all the conditions are the same as the last run, the system does not execute the job and returns a status of Success.

   • Submission Type:

     • Immediate Run the job once, as soon as possible.

     • Scheduled Set up a regular schedule.

     • Deferred Run the job once, at a future time.

   • Trigger Downstream Transformations and Validation Checks: Select to make this job trigger validation checks on the target model and transformations from the target model to all others that come after it, in sequence. This can happen only if the source models are set up to trigger downstream processes.

Click the Refresh icon for an update to the Job Status.

Cancel a transformation job

As described in Run a transformation, you access transformation through the Transformations tab on the Home page. Select a transformation in the Run History pane and click:

• Cancel Job to cancel the currently running or next pending job.

• Cancel All Jobs to cancel the currently running or next pending job and all future jobs in the schedule for the selected transformation. Available only for scheduled jobs.

  Note:

  If there is an immediate or deferred job for the same transformation, or another schedule for the same transformation, those jobs are not affected.

• Cancel Triggered Job to cancel the currently running or pending job. Available only for jobs that are set up to be triggered by another job.

Run validation check batches and view run history

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. Click the Validation Checks tab to open tables with details on the validation check batches for a model and run history.
3. From the Model field, select the clinical data model from the drop-down list.
4. Select a validation check batch from the Validation Check Batches for Model <model name>.

   The system displays information about its validation checks in the middle pane and information about its run history in the lower pane.

5. To view all jobs, click the View Full Job History icon.

   To view only recent jobs again, click the View Recent Jobs Only icon.

   To view log files, click the icon in the column for the type of job:

   • Log (Run History pane) The most recent manually submitted job.

   • Triggered Job Log (Run History pane) The most recent triggered job.

   • Install Job Log (Validation Check Batch pane) The most recent installation of the validation check batch.

   Note:

   A validation check can be disabled so that it is not included in the batch execution. To find out if a check was included in the run, check the log file.

• Run a validation check batch
  
• Cancel a validation check batch job
  

Run a validation check batch

You must run validation checks as a batch.

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. Click the Validation Checks tab to open tables on the validation check batches for a model and run history.
3. Select a validation check batch from the Validation Check Batches for Model <model name> table and click the Submit Job icon.

   Tip:

   The Submit Job icon does not appear if the validation check batch is not installed. Check the Installed Status.

   If the installation status of a validation check batch is Warning, you may still be able to run the batch. Check the installation log file. If the status is Warning because one of the source tables is not used in the current study, the batch runs without input from that table.

4. Enter values:

   • Submission Mode: Select one:

     • Full mode processes all records.

     • Incremental mode processes only new and changed records.

   • Force Execution: Select to run the job even though the source data currency, parameter values, and the version number of the program(s) have not changed since the last run. The system uses Full mode regardless of the Submission Mode setting. Full mode includes data deletion.

     If not selected and all the conditions are the same as the last run, the system does not execute the job and returns a status of Success.

   • Submission Type: Select:

     • Immediate to run the job once, as soon as possible.

     • Scheduled to set up a regular schedule.

     • Deferred to run the job once, at a future time.

   • Trigger Downstream Transformations and Validation Checks: Select this check box if you want the system to detect all transformations and validation checks set up for this data model and all others that come after it, and submit them sequentially.

     Note:

     This option appears only if the validation check batch is set up to allow it.

   • Click the Refresh icon at any time for an update.

   • To check the log file, click the icon in the Log column.

Cancel a validation check batch job

As described in Run a validation check batch, you access validation check batches through the Validation Checks tab on the Home page. Select a validation check batch in the Run History table and click:

• Cancel Job to cancel the currently running or next pending job.

• Cancel All Jobs to cancel the currently running or next pending job and all future jobs in the schedule for the selected validation check batch (available only for scheduled jobs).

  Note:

  Cancelling a job does not affect an immediate or deferred job for the same batch or another schedule for the same batch.

• Cancel Triggered Job to cancel the currently running or pending job (available for jobs triggered by another job).

Reprocess discrepancies that failed to be sent to InForm

Validation checks with an Initial Action of Send to InForm immediately send discrepancies they create to InForm as queries. To check the status of this operation:

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. Click the Validation Checks tab to open tables on the validation check batches for a model and run history.
3. Click the Failed to Send Discrepancies icon in the upper-right corner of the tab.

   The system displays counts for the selected study and lifecycle stage:

   • Processing for System shows the number of discrepancies currently being processed.

   • Failed Processing for System shows the number of discrepancies that should have been sent to InForm but weren't.

4. If any discrepancies fail to send, click the Reprocess Discrepancies icon to send them again. Click the Refresh icon to see progress.

Check if data files loaded

The section contains the following topics:

• View data files not processed
  
• View data load history
  
• Review data load errors (optional)
  

View data files not processed

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. Click the Files Not Processed tab to open a table with uploaded files that did not load into the selected study.

   A file may not load into a study for any of these reasons:

   • It is misnamed or uses the different capitalization compared to the file specification defined for the clinical data model.

   • There is a mistake in the File Specification regular expression defined for the model.

   • There is a matching File Specification, but the end date passed.

   • There is a matching File Specification, but data loading is suspended.

   • There is a matching File Specification, but the model is not installed.

3. View unprocessed files.

   Columns include:

   • File Name: Name of the file.

   • Path: Path to the file location.

   • Status: The possible statuses are:

     • DETECTED: The file has been detected in the watched folder but has not yet been submitted.

     • MISSING: The file was detected but deleted before the scheduled deletion or archive date.

     • DELETED: The file was deleted by File Watcher as scheduled.

     • ARCHIVED: The file was archived by File Watcher as scheduled.

   • File Type: The type of file.

   • File Modified: The modification date of the file on the file system.

   • Detection Date: The date and time the file was detected, using the date and time in the DMW database.

   • Archive Date displays the scheduled archive date before the file is archived and the actual archive date afterward.

     Note:

     Archiving is not supported in this release.

   • Deletion Date displays the scheduled deletion date before the file is deleted and the actual deletion date afterward.

   • Date Missing: If the file is overwritten or removed from the file system before it is archived or deleted, then the Date Missing is store here.

   • File Error: Information about the problem.

View data load history

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. Click the Data Loads tab to open a table with details on files that loaded into the selected study.
3. To view all jobs, click the View Full Job History icon.

   To view only recent jobs again, click the View Recent Jobs Only icon.

4. You can filter by entering a value in the blank field above any column. If blank fields are not displayed, click the Query By Example icon.

   Columns include:

   • Data Model: The clinical data model into which data was or will be loaded.

   • Type:

     • FILEWATCHER for data loaded from a file.

     • INFORM for data loaded from InForm.

       Note:

       All pending jobs for InForm data loads are for the Production lifecycle stage. It isn't possible to schedule InForm data loads for the Development or Quality Control lifecycle stage.

   • File Name/InForm Load Process:

     • For files, the full path of the file loaded.

     • For InForm, the type of data load: INFORM_DEFINITION, INFORM_METADATA, INFORM_MANUAL_DATA or INFORM_SCHEDULED_DATA.

   • Last Job shows the date of the last completed job.
   • Last Load shows the date of the last load time.

   • Last Load Results shows the job status: Success, Warnings, or Failure for completed jobs.

   • View Output opens a file with the output of the data load history.
   • View Error File opens an error file on the data load.
   • View Comprehensive Error Report opens a detailed error report on the data load.
   • User Name shows the name of user who loaded the data.
   • Next Run Date shows the date of the next data load run.
   • Frequency shows how often data loads into the study.
   • Units shows the size of the data load.

   • Columns with icons: Click the icon to open the error, log, or output file.

   Note:

   For details on viewing errors in reports, see Review data load errors (optional).

• Statuses for uncompleted jobs
  

Statuses for uncompleted jobs

As described in View data load history, you can view data load history through the Data Loads tab on the Home page. For incomplete jobs, the system displays the job's current status. Click the Refresh icon to update the table.

• Pending: The job has not yet started running.

• Started: The job has begun pre-processing.

• Executing: The Program has connected to the database and is running.

• Finalizing: The job has begun post-processing.

• Aborted: The job has been manually stopped while underway.

• On Hold: The job is waiting for the quiesce process to complete for the clinical data model work area.

• Expired: The system removed the job from the queue after the timeout interval passed.

• Duplicate: The job is a duplicate of another job; the currency of the source data, parameter values, and executable instance version are the same. The system does not rerun the job unless the person submitting the job chooses to force reexecution.

Review data load errors (optional)

To review data load errors:

1. Click Home in the navigation bar and select a study from the Studies panel.

   Note:

   For details on selecting a study, see Select a study.
2. Click the Data Loads tab to open a table with details on files that loaded into the selected study.

3. Open a file in one of the following columns:

   • Last Load Results: Contains the results of the most recent data load. This file is produced whether the data load is successful or failed.

   • View Output:: Produced during a data load (the SQL*Loader log). This file is produced whether the data load is successful or failed.

   • View Error File: Created when a text file for a data load contains more fields than there are columns in the table you're loading the data into. This file is produced for failed data loads only.

     Tip:

     This column is typically empty.

   • View Comprehensive Error Report: The error log includes a list of the specific data record and field from the file that caused the error. This file is produced for failed data loads only.

FAQs

• Why can't I see a job that ran a while ago?
  
• Why am I seeing a Diagnostics Icon?
  

Why can't I see a job that ran a while ago?

By default you can see only recent jobs. To see a job that happened longer ago, click the View Full Job History icon.

To change back, click the View Recent Jobs Only icon.

This applies to the Home page Data Loads tab and the Run History pane of the Transformations and Validation Checks tab.

Why am I seeing a Diagnostics Icon?

The icon indicates that your administrator is running diagnostics for troubleshooting purposes. Performance may be slower than usual.