Apply Step

The apply step is the step where records in the target environment are added or updated. Like the comparison step, the apply step is actually multiple steps to optimally handle high volume and dependencies between records as smoothly as possible.

Note: Refer to Running Batch Jobs for more information about streamlining the various steps in the process.

Before explaining the apply steps in detail, the following points highlight the type of data that may be included in a given data set.

Records that have no foreign keys and therefore no dependencies on other records. Examples: Message, Display Profile.
Records that have foreign keys that may already be in the target. Examples: Algorithms for existing algorithm types, To Do Roles for existing To Do Types.
Records that have foreign keys that are new records but also part of the migration. CMA detected the relationship at export time and grouped the objects in the same transaction. Example: Script-based Algorithm Type where the script is also in the migration.
Records that have foreign keys that are new records but also part of the migration. CMA did not detect the relationship. This may occur if the referenced foreign key is in a XML or parameter column and the migration plan did not include an instruction to explicitly define the relationship. Example, a Zone that references a visibility script.
Records that have circular references where both records are new and are part of the migration. CMA detected the relationship at export time and grouped the objects in the same transaction. Example: plug-in Script for a BO plug-in spot. The script references the BO and the BO references an algorithm for the script’s algorithm type. Another example is when the same record is maintained by multiple maintenance objects and therefore exists in multiple migration objects.

To handle high volume data, the first step in the apply process is to perform the apply logic at the migration object level via a multi-threaded batch job. This should result in all records in categories 1 and 2 above being applied successfully.

For records in categories 3 and 4 above, if a record with a foreign key is added or updated before its related record, the validation will fail. However, if the related record is added first and then the record referring to it is added, validation will pass. The tool handles these dependencies as follows:

Dependency between master and transaction entities is typically hierarchical and in most cases straight-forward. The tool leverages that knowledge to orchestrate the processing of objects in an optimal way that follows their dependency order as much as possible. Note that relationship between entities could be complex and this approach does not eliminate all processing order related errors but rather significantly reduce them.
Dependency between configuration entities is more complex and inter-twined and therefore migration objects are not ordered, i.e. the multi-threaded batch process may not process records in the desired order.
To overcome the potential issue of processing order related errors, the Apply step has special functionality, described in detail below.

For records in category 5 above, the circular reference will mean that the apply process at the object level will not successfully add or update these records. The apply process at the transaction level will cover these records. This is described in detail below.

Apply Objects

Once the Data Set is in the state of Apply Objects, the Migration Object Monitor - Apply process (F1-MGOAP) runs to attempt to apply the objects.

Note:

When using separate batch processes for business data the Migration Object Monitor (Business)- Apply process (F1-MGOAB) works in the same way to apply business migration objects.

The background process in conjunction with the Apply algorithm have special functionality to ensure records in categories 3 and 4 (above) successfully apply during this step:

The Migration Object Monitor - Apply process is a special one that continually re-selects records in the Approved state until there are no more eligible records to process.
When an error is received in the Apply Object algorithm, the algorithm increments an "iteration count" on the migration object record. If the iteration count does not exceed a maximum count (noted in the algorithm), the object remains in the Approved state and is eligible to be picked up for processing again. If the iteration count exceeds the maximum defined in the algorithm, the record transitions to the Error Applying state.

Note: When submitting this Apply batch job, be sure to set the number of threads to a number that does not exceed the number of threads supported by the thread pool worker. Doing this will cause the ‘excess’ threads to wait for the supported number of threads to finish.

The following diagram is the portion of the migration object lifecycle that pertains to the Apply step.

At the completion of the Apply monitor process, typically the objects will be in the Applied state or the Error Applying state. The records in the Error Applying state are in that state for one of two reasons.

They are in category 5 described above where the records have a circular reference with another record. For this scenario, the Apply Transactions step described below should successfully apply the records.
There is some other error that is unrelated to the records in the current migration. In this case, manual intervention may be required. Refer to the Resolving Errors section below for more information.

As shown in the diagram, the Apply Objects algorithm may also detect a reason that the object cannot be applied. This may occur if the object in the target environment has been updated since the comparison step, making the SQL captured at that point no longer applicable. If this occurs, after the current migration is fully applied, the original file may imported again, and new comparisons can be generated and applied.

Apply Transactions

Ideally, after the Apply Objects step, all the objects are Applied or are in Error Applying due to the "circular reference" situation. The typical next step is to turn over responsibility to the transactions. The migration transactions can then attempt to apply their objects in bulk.

In order to ensure that multiple background processes are not trying to select migration objects to run the Apply step, the Transactions are only eligible to attempt to "apply my objects" if the Data Set is in the Apply Transactions state.

A monitor algorithm (executed by the data set monitor batch process) on the Apply Objects state checks to see if all migration objects are no longer Approved or the count of records in Error Applying does not exceed a configured limit. If so, it automatically transitions the record to the Apply Transactions state.

If the number of objects in Error Applying exceeds a configured limit, the monitor algorithm does not automatically transition the record. In that case, a user must determine if the large number of errors can be resolved or manually transition to Apply Transactions (despite the large number of errors). The Resolving Errors section below describes alternative steps that the user may take if there are errors.

Once the Data Set is in the state of Apply Transactions, the Migration Transaction Monitor - Apply process (F1-MGTAP) runs. It attempts to apply the transaction's objects. If no migration objects are in error, the migration transaction simply transitions to Applied. If any of the migration objects are in Error Applying, the background process and the Apply algorithm have special functionality to try to overcome dependencies in migrated objects:

The Apply algorithm selects all migration objects in error and performs all their SQL, then validates all the records. If there are objects in the transaction with circular references, they should pass validation at this point.
Because there may still be some dependencies across transactions, similar error handling described in the Apply Objects step occurs here. When an error is received in the Apply Transaction's Object algorithm for any of the objects in the transaction, the algorithm increments an "iteration count" on the migration transaction record. If the iteration count does not exceed a maximum count (noted in the algorithm), the transaction remains in the Ready to Apply state and is eligible to be picked up for processing again. If the iteration count exceeds the maximum, the record transitions to the Error Applying state. Note that if any objects in the transaction are in error, none of the objects are applied. They all remain in error.
The Migration Transaction Monitor - Apply process is a special one that continually re-selects records in the Ready to Apply state until there are no more eligible records to process.

Note: When submitting this Apply batch job, be sure to set the number of threads to a number that does not exceed the number of threads supported by the thread pool worker. Doing this will cause the 'excess' threads to wait for the supported number of threads to finish, erasing the benefit of the iteration processing.

The following diagram is the portion of the migration transaction lifecycle that pertains to the Apply step illustrating the points above.

If at the end of the transaction level Apply process there are transactions in error (and therefore there are still objects in error), a user must review the errors and determine how to fix them. Refer to the Resolving Errors section below for more information.

Resolving Errors

As mentioned in the previous sections, errors may be received after the Apply Objects process runs. If the number of records in error is below a certain limit (and the data set monitor batch job is submitted to execute the monitor algorithms) the system will automatically transition the data set to the Apply Transactions. If the monitor batch job is not run or if the number of objects in error exceeds a certain limit, a user must make the decision after viewing the errors in the Objects in Error zone on the Migration Data Set Import portal.

If the errors appear to be dependency related, the user can decide to let the "transactions apply their objects" and transition the data set to Apply Transactions, described above.
If the errors appear to be related to an outside issue that can be manually resolved, the user may choose to fix the issue and redo the Apply Objects step.
The user may also decide to reject one or more objects to remove them from the migration.

After the Apply Transactions step, if there are still errors, a user must review the records and determine how to proceed. Errors are visible in the Transactions in Error zone on the Migration Data Set Import portal.

The user may decide to reject one or more objects to remove them from the migration.
The user may manually resolve an issue external to the migration and then decide to do one of the following:
- Redo the Apply Objects step. This is recommended if there are a large number of Objects still in error and not a large number of dependencies expected. The benefits of running the Apply Objects multi-threaded will ensure that the process runs efficiently.
- Redo the Apply Transactions step.

Because the objects and transactions are in Error Applying, in order to "retry" the Apply step after manually fixing an error, the system needs to move the records back to the state that allows them to be picked up by the appropriate Apply process. For migration objects, records need to be moved back to Approved. For migration transactions, records need to be moved back to Ready to Apply. The following points describe the Retry logic for migration objects.

If a user decides to Retry Objects (using an action button on the Migration Data Set Import page), the data set transitions to the Retry Objects state. At this point the Migration Object monitor must be run.
The monitor on the Error Applying state for the objects detects that the data set is in the state of Retry Objects and that triggers the transition back to Approved.
The next step is to transition the data set from Retry Objects to Apply Objects. This may be done manually or by running the Migration Data Set Import monitor process.
Now the objects are eligible to be picked up by the object level Apply process.

Analogous logic exists for the migration transactions.

If a user decides to Retry Transactions (using an action button on the Migration Data Set Import page), the data set transitions to the Retry Transactions state. At this point the Migration Transaction monitor must be run.
The monitor on the Error Applying state for the transactions detects that the data set is in the state of Retry Transactions and that triggers the transition back to Ready to Apply.
The next step is to transition the data set from Retry Transactions to Apply Transactions. This may be done manually or by running the Migration Data Set Import monitor process.
Now the transactions are eligible to be picked up by the transaction level Apply process.

The retry logic may also occur when transitioning between the Apply Objects and Apply Transactions depending on whether or not there are errors. The following scenario highlights this point.

After the Apply Objectsstep there are objects in Error Applying. The data set transitions to Apply Transactions and the Apply step is done at the transaction level.
After the Apply Transactions step there are transactions in Error Applying.
User chooses to try to apply objects again (by clicking Retry Objects). The steps outlined above for retrying objects are followed at this point.
After the apply objects, user may choose to retry objects again (after fixing errors if applicable).
At some point the user will transition toApply Transactions again. If there are transactions in Error Applying, the system will automatically transition the data set to Retry Transactions and the steps outlined above for retry transactions are followed.

Finalize Apply Step

Once all the migration objects for a migration transaction are in a final state (Applied, Rejected or Cannot Apply), the migration transaction transitions to the Applied state. Once all the migration transactions are in the Applied state, the Migration Data Set record transitions to the Completed and the import is complete.

Note: To review the full lifecycle for each record, refer to the Business Object - Summary tab in the application metadata for the base business objects Migration Data Set Import (F1-MigrObjectImport), Migration Transaction (F1-MigrTransactionImport) and Migration Object (F1-MigrObjectImport).