7 Load data and run jobs

This section describes how to:

• Load data from a file
  
• Load data from InForm
  
• Run transformations and view run history
  
• Run validation check batches and view run history
  
• Resend discrepancies that failed to be sent to InForm
  
• What if...
  
• FAQs
  

Load data from a file

The study File Watcher checks a file system location for data files to load.

• Upload data files
  
• Suspend or resume data file loading
  
• Verify that a data file loaded
  
• View data load history
  
• View data files not processed
  
• Load data and discrepancies from a non-InForm clinical data system
  

Upload data files

To load data files:

• Place them on the server in the location your administrator specified for the study, in the correct subdirectory. There is a subdirectory for each lifecycle stage or for each lifecycle stage/file type combination.

• The file name must exactly match the file specification defined for the clinical data model; see Specify a file name convention for each lifecycle stage.

  If you are loading data from a clinical data capture system other than InForm, see Load data and discrepancies from a non-InForm clinical data system.

Suspend or resume data file loading

1. After you select a study, click the Study Configuration icon from the navigation bar. Then click the Clinical Data Model tab.
2. Select the clinical data model.
3. In the Watcher Configuration tab, select a file specification.
4. Click the appropriate icon, either Suspend or Resume.

Verify that a data file loaded

1. Go to the Home page, Data Loads tab to see successful data loads.

2. If a data load you are expecting is not there:

   1. Click the Study Configuration icon from the navigation bar.

   2. Select the clinical data model.

   3. Select the Detected Files tab. It lists all the uploaded data files that File Watcher has detected for a clinical data model.

3. If a file failed to load, you can select it and:

   • Click the Delete File icon to start over with a new or renamed file. The most likely problem is that the file name does not match the File Specification for the model. See Specify a file name convention for each lifecycle stage.

   • Click the Load Data icon to try again with the same file, for example after fixing the data load parameters.

For each detected file the system displays:

• File Name and File Specification Name for comparison.

• Status The possible statuses are:

  • DETECTED The file is in the watched folder but has not been submitted. The scheduled submission time is in the Data Load Date column.

  • SUBMITTED The file has been submitted.

    Tip:

    This status does not change when the load completes. Select the Data Loads tab on the Home page to see if the load was successful.

  • MISSING The file was deleted before or after it was submitted, before the scheduled deletion date.

  • DELETED The file was deleted as scheduled.

  • ARCHIVED The file was archived as scheduled.

• File Modified The modification date of the file on the file system.

• Data Load Date The scheduled data load date before the data has been loaded, with an icon to indicate "scheduled," and the actual data load date afterward.

• Detection Date The date and time the file was detected, using the date and time in the DMW database.

• Archive Date The scheduled archive date before the file is archived and the actual archive date afterward. (Not supported in this release.)

• Error Information about a problem, if any.

• Deletion Date The scheduled deletion date before the file is deleted and the actual deletion date afterward.

• Date Missing If a file is overwritten or removed from the file system before it is archived or deleted, then the Date Missing is stored here.

View data load history

1. Go to the Home page, Data Loads tab.
2. To view all jobs, click the View Full Job History icon.

   To view only recent jobs again, click the View Recent Jobs Only icon.

3. You can filter by entering a value in the blank field above any column. If blank fields are not displayed, click the Query By Example icon.

• Statuses for uncompleted jobs
  

Statuses for uncompleted jobs

For incomplete jobs, the system displays the job's current status. Click the Refresh icon to update.

• Pending: The job has not yet started running.

• Started: The job has begun pre-processing.

• Executing: The Program has connected to the database and is running.

• Finalizing: The job has begun post-processing.

• Aborted: The job has been manually stopped while underway.

• On Hold: The job is waiting for the quiesce process to complete for the clinical data model work area.

• Expired: The system removed the job from the queue after the timeout interval passed.

• Duplicate: The job is a duplicate of another job; the currency of the source data, parameter values, and executable instance version are the same. The system does not rerun the job unless the person submitting the job chooses to force reexecution.

View data files not processed

1. Go to the Home page and select a study.
2. Select the Files Not Processed tab.

   The system displays files that were uploaded but could not be loaded into the selected study. A file may not be loadable because:

   • It is misnamed or in nonmatching case, compared to the file specification defined for the clinical data model.

   • There is a mistake in the File Specification regular expression defined for the model.

   • There is a matching File Specification, but its end date has passed.

   • There is a matching File Specification, but its data loading is suspended.

   • There is a matching File Specification, but the model is not installed.

3. View unprocessed files.

   Columns include:

   • Status: The possible statuses are:

     • DETECTED: The file has been detected in the watched folder but has not yet been submitted.

     • MISSING: The file was detected but deleted before the scheduled deletion or archive date.

     • DELETED: The file was deleted by File Watcher as scheduled.

     • ARCHIVED: The file was archived by File Watcher as scheduled.

   • File Modified: The modification date of the file on the file system.

   • Detection Date: The date and time the file was detected, using the date and time in the DMW database.

   • Archive Date displays the scheduled archive date before the file is archived and the actual archive date afterward.

     Note:

     Archiving is not supported in this release.

   • Deletion Date displays the scheduled deletion date before the file is deleted and the actual deletion date afterward.

   • Date Missing: If the file is overwritten or removed from the file system before it is archived or deleted, then the Date Missing is store here.

   • File Error: Information about the problem.

Load data and discrepancies from a non-InForm clinical data system

Clinical data systems other than InForm must load data and discrepancies into Oracle DMW in data files. Each load must include an additional file. The file must:

• Be named exactly as specified in the DISC_LOAD_TBL column of the DME_EXTERNAL_MODEL_ATTRIBS table in your database.

• The file must be formatted the same way as data files (SAS files or for text: fixed or delimited format, and if delimited, either using enclosing characters or not).

• Be in the same .zip file as the data and discrepancy files for text files or the same XPort, CPort, or SAS datasets.

• Include values in three columns for each record as shown in the following table. These values are written to an internal table in the input clinical data model.

Table 7-1 Columns in the Generated Clinical Data Model Forms Table

Order	Column Name	Datatype	Length	Description
1	TABLE_NAME	VARCHAR2	30	The Oracle name of the table in the external system that contains the record.
2	KEY_VALUES	VARCHAR2	4000	The unique key values for the record, separated by tildes (~), without quotation marks, and in column order. DMW checks these values against the surrogate key value it generates for each record for use in data lineage tracing.
3	PARTIAL_URL	VARCHAR2	4000	The ID or other string required to uniquely identify the record in the external system.

For more information, see Using the Generic Connector to Integrate DMW with a Clinical Data System, My Oracle Support article 2172786.1, or the documentation provided by the integration vendor.

Load data from InForm

• Load InForm data immediately in Development or Quality Control
  
• Schedule InForm data loads in Production
  
• Suspend and resume InForm data loading
  

Load InForm data immediately in Development or Quality Control

In the InForm Configuration tab, click the Load Data to DMW Development or Load Data to DMW Quality Control icon to immediately load the latest data from InForm into the DMW Development lifecycle stage.

You cannot schedule data loads in Development or QC.

See InForm data isn't loading? for more information (if necessary).

Schedule InForm data loads in Production

1. After you select a study, click the Study Configuration icon from the navigation bar. Then click the Clinical Data Model tab.
2. Select Production from the Lifecycle drop-down at the top of the page.
3. Click the Suspend icon.
4. Install the model if it has not already been installed in Production.
5. Select Production under Study Lifecycle in the table below.
6. Select the Schedule Production Data Load checkbox.

   Note:

   If you do not check this box, it is possible to enter Fetch Frequency and click Save, but the system does not actually schedule the load.

7. Specify a Fetch Frequency: the number of minutes between automatic data loads.

   Note:

   A data load will not start until the previous one completes.

8. Save.
9. Click the Resume icon. The first scheduled data load starts immediately.

See InForm data isn't loading? for more information (if necessary).

Suspend and resume InForm data loading

You must suspend data loading to load metadata, install the model, or change the remote location, account, or web service.

1. Click the Study Configuration icon at the top of any page and then click the Clinical Data Models tab.
2. Select the lifecycle from the Lifecycle drop-down at the top of the page.
3. Select the clinical data model.
4. In the InForm Configuration tab, click the Suspend icon to stop any scheduled data loads and prevent manually loading data.

   The system replaces the Suspend icon with the Resume icon.

5. Click the Resume icon to allow manual and scheduled data loading.

See InForm data isn't loading? for more information (if necessary).

Run transformations and view run history

1. Go to the Home page, Transformations tab.
2. To view all jobs, click the View Full Job History icon.

   To view only recent jobs again, click the View Recent Jobs Only icon.

3. See:

• View transformation job history
  
• Run a transformation
  
• Cancel a pending tranformation job
  

View transformation job history

Transformations are displayed by the name of their target clinical data model.

• To view table transformations, click a transformation's node.

• To view run history and pending jobs, select a transformation in the upper pane.

• To view log files, click the icon in the column for the type of job:

  • Log: The most recent manually submitted job.

  • Triggered Job Log: The most recent triggered job.

  • Install Job Log: The most recent installation of the transformation.

Run a transformation

To run or schedule a transformation:

1. Go to the Home page, Transformations tab.
2. Select a transformation and click the Submit Job icon.

   Tip:

   The Submit Job icon does not appear if the selected transformation has not been installed. Check the Install Status column.

   If the installation status of the transformation is Warning, you may still be able to run the batch. Check the installation log file.

3. Enter values:

   • Submission Mode: Select one:

     • Full mode includes data deletion. Use Full mode only if you are confident that you are reloading all current data.

     • Incremental is faster and does not include data deletion.

     If you are submitting a transformation for a single table and the table is defined with Unit of Work processing, select:

     • Full UOW includes data deletion. Use Full UOW only if you are confident that you are reloading all current data for each subject or subject visit that has any data included.

     • Incremental UOW is faster and does not include data deletion.

       Tip:

       Set up regular Incremental loads at frequent intervals and do Full loads at longer intervals.

   • Force Execution: Select if you want to run the job even though the source data currency, parameter values, and the version number of the program(s) have not changed since the last run. The system uses Full mode regardless of the Submission Mode setting. Full mode includes data deletion.

     If not selected and all the conditions are the same as the last run, the system does not execute the job and returns a status of Success.

   • Submission Type:

     • Immediate Run the job once, as soon as possible.

     • Scheduled Set up a regular schedule.

     • Deferred Run the job once, at a future time.

   • Trigger Downstream Transformations and Validation Checks: Select to make this job trigger validation checks on the target model and transformations from the target model to all others that come after it, in sequence. This can happen only if the source models are set up to trigger downstream processes.

Click the Refresh icon for an update to the Job Status.

Cancel a pending tranformation job

Select a transformation in the Run History pane and click:

• Cancel Job to cancel the currently running or next pending job.

• Cancel All Jobs to cancel the currently running or next pending job and all future jobs in the schedule for the selected transformation. Available only for scheduled jobs.

  Note:

  If there is an immediate or deferred job for the same transformation, or another schedule for the same transformation, those jobs are not affected.

• Cancel Triggered Job to cancel the currently running or pending job. Available only for jobs that are set up to be triggered by another job.

Run validation check batches and view run history

1. Go to the Home page, Validation Checks tab.
2. From the Model drop-down list, select the clinical data model.
3. Select a validation check batch in the upper pane.

   The system displays information about its validation checks in the middle pane and information about its run history in the lower pane.

4. To view all jobs, click the View Full Job History icon.

   To view only recent jobs again, click the View Recent Jobs Only icon.

   To view log files, click the icon in the column for the type of job:

   • Log (Run History pane) The most recent manually submitted job.

   • Triggered Job Log (Run History pane) The most recent triggered job.

   • Install Job Log (Validation Check Batch pane) The most recent installation of the validation check batch.

   Note:

   A validation check can be disabled so that it is not included in the batch execution. To find out if a check was included in the run, check the log file.

   See:

• Run a validation check batch
  
• Cancel a validation check batch job
  

Run a validation check batch

You must run validation checks as a batch.

1. Go to the Home page, Validation Checks tab.
2. Select a validation check batch and click the Submit Job icon.

   Tip:

   The Submit Job icon does not appear if the validation check batch is not installed. Check the Installed Status.

   If the installation status of a validation check batch is Warning, you may still be able to run the batch. Check the installation log file. If the status is Warning because one of the source tables is not used in the current study, the batch runs without input from that table.

3. Enter values:

   • Submission Mode: Select one:

     • Full includes data deletion. Use Full mode only if you are confident that you are reloading all current data.

     • Incremental is faster and does not include data deletion.

   • Force Execution: Select to run the job even though the source data currency, parameter values, and the version number of the program(s) have not changed since the last run. The system uses Full mode regardless of the Submission Mode setting. Full mode includes data deletion.

     If not selected and all the conditions are the same as the last run, the system does not execute the job and returns a status of Success.

   • Submission Type: Select:

     • Immediate to run the job once, as soon as possible.

     • Scheduled to set up a regular schedule.

     • Deferred to run the job once, at a future time.

   • Trigger Downstream Transformations and Validation Checks: Select this checkbox if you want the system to detect all transformations and validation checks set up for this data model and all others that come after it, and submit them sequentially.

     Note:

     This option appears only if the validation check batch is set up to allow it.

   • Click the Refresh icon at any time for an update.

   • To check the log file, click the icon in the Log column.

Cancel a validation check batch job

Select a validation check batch in the Run History pane and click:

• Cancel Job to cancel the currently running or next pending job.

• Cancel All Jobs to cancel the currently running or next pending job and all future jobs in the schedule for the selected validation check batch. Available only for scheduled jobs.

  Note:

  If there is an immediate or deferred job for the same batch, or another schedule for the same batch, those jobs are not affected.

• Cancel Triggered Job to cancel the currently running or pending job. Available only for jobs that are set up to be triggered by another job.

Resend discrepancies that failed to be sent to InForm

Validation checks with an Initial Action of Send to InForm immediately send discrepancies they create to InForm as queries. To check the status of this operation:

1. Go to the Home page, Validation Checks tab.
2. Click the Failed to Send Discrepancies icon in the upper right corner.

   The system displays counts for the selected study and lifecycle stage:

   • Processing for System shows the number of discrepancies currently being processed.

   • Failed Processing for System shows the number of discrepancies that should have been sent to InForm but weren't.

3. If any discrepancies failed to be sent, click the Reprocess Discrepancies icon to send them again. Click the Refresh icon to see progress.

What if...

• InForm data isn't loading?

• DMW is not sending discrepancies to InForm?

• InForm data isn't loading?
  
• DMW is not sending discrepancies to InForm?
  

InForm data isn't loading?

To load data:

• The most current version of the model must be installed.

• In Production, data loading must be resumed, not suspended.

• The validation status of the model must be equal to or greater than the model's lifecycle stage; that is, a QC lifecycle model must have a validation status of QC or Production, and a Production lifecycle model must have a validation status of Production.

DMW is not sending discrepancies to InForm?

DMW sends discrepancies to InForm only if the lifecycle stages match. InForm's UAT lifecycle matches the Quality Control lifecycle in DMW.

If the lifecycle stages match and discrepancies still are not sent, see Resend discrepancies that failed to be sent to InForm.

FAQs

• Should I use Full or Incremental processing?

• Can I load data from any InForm lifecycle into DMW?

• Should I use Full or Incremental processing?
  
• Can I load data from any InForm lifecycle into DMW?
  

Should I use Full or Incremental processing?

See Data processing types and modes.

Can I load data from any InForm lifecycle into DMW?

You can load data from any InForm lifecycle into any DMW lifecycle except production, which requires that you assert that you are loading data from an InForm production lifecycle database.

Which InForm data is loaded depends on the setting of the Remote Study Account field. Its lifecycle stage is indicated in the InForm Lifecycle field.