D Appendix: Best Practices

This section outlines the key recommendations while performing data conversion activities.

Data Conversion Tool

To gain better performance and seamless conversion using the data conversion tool, the following should be considered,

1. Utilize the offline data validator tool to verify input files containing transformed data. The tool helps to identify errors related to data type mismatch, size mismatch and check constraint violations. The process of validation and corrections at the source should be repeated until input files are error free.

2. Mock runs are essential in establishing the data quality and detailed migration process. It is recommended to perform at least one mock run for initial data, two or more mock runs for full data volume and at least one cutover rehearsal in preparation for the cut-over migration.

3. Golden Gate replication should be turned off during data conversion as keeping the replication enabled would severely degrade the import performance.

4. It is also suggested, before the cut-over migration, to carry out full batch cycle runs on the converted data in the system for minimum of a week to ensure that daily batch run along with week-end processing is occurring without any issues.

5. You may consider following for better performance during business validation phase:

   • For large volume entities like items, start with lower volume files during the first mock. There are often a large number of errors in the input files at the initial stages and running a lower record set will ease error identification and correction.

   • In a new instance, the DB optimizer may settle for better execution plan after 2-3 runs of execution, so continue loading the files with smaller volumes for few mock runs even if performance is not as expected.

   • Delete and collect the database stats at regular intervals while performing conversion for large volume data, such as items and item/locations. Use GATHER_DATABASE_STATS and DELETE TABLE STATS tasks in the System Administration Screen.

   • Through initial mock runs, load only new or previously failed records, excluding the records that are already processed. In this way, the system will not consume time validating and logging such records that you know to be successful.

   • Before reloading the files, ensure that you have corrected your transformation logic based on errors that you incurred in earlier mock runs so that you do not continue to encounter the same errors.

   • The migration tool is available in all non-prod environments. However, it is advisable to perform conversion in only one environment and lift-shift the converted data, if the data is required in other environments as well. This will avoid unnecessary steps of configuring additional environments for the higher volumes expected during conversion that may not be needed otherwise for that environment.

   • It is highly recommended to truncate staging and error tables for an entity using the PURGE_TABLE task in the System Administration Screen at regular intervals. This will improve the execution timings for the subsequent runs. As needed, take backup of these tables before purging for future reference.

6. While loading and reloading of data, use TRUNCATE instead of DELETE for database tables, as deleting a table with larger volume data deteriorates the performance significantly.

7. Environment configuration to attain optimal performance with the data conversion tool will be reliant on expected data volume for conversion and outspread of data based on system configurations. For instance, based on the business requirements you may need to convert expenses and assessments for open purchase orders.

   The conversion environment size which includes number of CPUs, RAM, temp tablespace etc. plays a key role in determining the overall conversion performance. The thread and chunk sizes for supporting parallel processing of data should be varied to check what runs best for your data and environment. You can configure thread and chunk size through Entity Sequence List screen. For more details see the "Appendix: Entity Sequence List" section above.

   Below are the recommended configurations for thread and chunk sizes on the basis on available machine size for a few big tables converted by the data conversion tool.

   Functional Area	Available CPU	Volume per load	Thread Size	Chunk Size	Parallel Files
   Item	8 16 32 64	50K -100K 50K -100K 100K -300K 100K -300K	1 1 1 1	5000 5000 10000 10000	- - - -
   ITEM_LOC RANGING - if using location level ranging	8 16 32 64	50K 100K 200K 200K	8 15 20 20	50 50 5050	1 1 1 1
   ITEM_LOC - for retail updates	8 16 32 64	1M 1M 4M 4M	8 15 20 20	10000 10000 10000 10000	2 2 2 2
   ITEM_LOC_SOH - for stock on hand updates	8 16 32 64	1M 1M 2M 2M	8 15 20 20	20000 20000 50000 50000	1 1 1 1
   ITEM_RANGING - for location level ranging	8 16 32 64	1-5M 1-5M 1-5M 1-5M	30 30 30 30	1000 1000 2000 2000	- - - -
   PRICE_HIST	8 16 32 64	1M 1M 4M 4M	8 15 20 20	10000 10000 10000 10000	2 2 2 2
   VAT_ITEM	8 16 32 64	1M 1M 1M 1M	15 15 15 15	10000 10000 10000 10000	>1 >1 >1 >1
   ITEM_HTS	8 16 32 64	1M 1M 1M 1M	15 15 15 15	10000 10000 10000 10000	>1 >1 >1 >1
   ITEM_IMAGE	8 16 32 64	1M 1M 1M 1M	15 15 15 15	10000 10000 10000 10000	>1 >1 >1 >1
   ITEM_EXPENSES	8 16 32 64	1M 1M 1M 1M	15 15 15 15	10000 10000 10000 10000	>1 >1 >1 >1
   ITEM_SEASONS	8 16 32 64	1M 1M 1M 1M	15 15 15 15	10000 10000 10000 10000	>1 >1 >1 >1
   ITEM_UDA	8 16 32 64	1M 1M 1M 1M	15 15 15 15	10000 10000 10000 10000	>1 >1 >1 >1
   INSERT_RIZP	8 16 32 64	1M 1M 1M 1M	15 15 15 15	10000 10000 10000 10000	>1 >1 >1 >1
   ITEM_LOC_TRAITS	8 16 32 64	1M 1M 1M 1M	15 15 15 15	10000 10000 10000 10000	>1 >1 >1 >1
   CREATE_PURCHASE_ORDER	8 16 32 64	2000 2000 2000 2000	15 15 20 20	50 50 100 100	2 2 2 2
   RPM_CLEARANCE	8 16 32 64	50K 100K 200K 200K	8 15 20 20	1000 1000 1000 1000	1 1 1 1

Note:

For the functional area not mentioned in the above table, it is recommended to start with default configurations in the tool and, as needed, update the thread count and chunk size based on the data volume and available CPUs.

Item Conversion

For item conversion, generally higher volume data is converted and one of the key considerations during data conversion is performance. Following needs to be taken into account for item conversion,

• Break the input files at a dept/class/subclass level and not load the whole set.

• Ensure expense profiles are correctly configured to default item expenses during the item conversion. Incorrect configuration of expense profiles can result in performance issues.

• It is highly recommended to disable MFQUEUE publishing triggers during data conversion and leverage other integration options for seeding data to the downstream systems. See the section "MFQUEUE Triggers" in the appendix for the list of MFQUEUE triggers.

• If you have configured Merchandising to run with Global Tax as the default tax type, then it is advisable to complete foundation data setup before loading tax rules, as loading rules earlier may slow down the item conversion.

• If item expense and HTS information are to be loaded, it is recommended to convert only non-zero expenses or assessments in the input files for estimated value (that is, EST_EXP_VALUE and EST_ASSESS_VALUE columns). Although these values can be recalculated during data conversion, it is not advisable, as it will adversely impact the load performance. If the system is required to recalculate these column values, DC_SYSTEM_OPTIONS.RECALCULATE_EXPENSES should be set to 'Y'.

• It is highly advisable to convert items using the DC_ITEM entity which is built with improved performance capabilities. The ITEM and ITEM_DIRECT_LOAD entities which were previously used for item conversion will be discontinued in future release. There is no difference in the file structures for all these entities. The DC_ITEM entity supports certain key item-related entities be converted after items, to enhance performance of the item load by limiting the amount of data being converted in a single load. Below entities support bulk processing of data independent of item data:

  • UDAs

  • Expenses

  • VAT

  • Images

  • HTS

  • Seasons/Phases

  • Zone-level Pricing

• To enhance overall item conversion performance, the DC_ITEM entity does not support cascading of data from parent to child items. Each item level should be loaded separately with the complete set of data applicable for that level. For the same reason, inheritance and defaulting for Price History, Item/Supplier/Country/Dimension, Item/Expenses, and Pack Item Breakout tables are removed from this entity. If data conversion is needed for them, then these should be loaded in separate files.

• When using the ITEM entity, for better performance, consider limiting your file size; for instance, loading 20,000 items per file might be an acceptable file size for a 32-CPU system.

• It is highly recommended to truncate staging and error tables using the PURGE_TABLE task in the System Administration Screen after converted data is verified for each conversion cycle. This will improve the execution timings for the subsequent runs.

• For the RPM_ITEM_ZONE_PRICE conversion, it is advisable to execute the INSERT_RIZP entity, which is built with bulk validations and processing capabilities. The file structure for the INSERT_RIZP entity is the same as the RPM_ITEM_ZONE_PRICE entity. The latter would be deprecated in a future release.

Item Location Conversion

The following should be considered to achieve optimum performance during item location conversion:

• Item location conversion is a threaded process. The thread and chunk size could be varied to check what suits best for your data and environment. See the recommended configurations for thread and chunk sizes above for initial configuration.

• Only one item/location file can be converted at a time. This is to prevent database resource contention among multiple files.

• Ensure that the fields associated with consignment set up including calculation basis, purchase type, and purchase rate, match to that of item/supplier/country unless you want to explicitly range them differently at a location level. This controls the number records going into ITEM_SUPP_COUNTRY_LOC table. Wrong input data will unnecessarily write too many records into ITEM_SUPP_COUNTRY_LOC table deteriorating performance.

• If you haven't already disabled triggers that capture data for integration to downstream systems on ITEM_LOC table, this should be done as part of item/location conversion. See the section "MFQUEUE Triggers" in the appendix for the list of triggers to disable.

• Execute GATHER_DATABASE_STATS and DELETE TABLE STAT tasks in the System Administration Screen which gathers and deletes database statistics respectively. It is recommended to gather statistics multiple times a day in between conversion runs. Also, it is advisable to delete statistics for every 5-10 million item/locations processed through item location ranging.

• For item/location ranging, it is highly recommended to execute the ITEM_RANGING entity which is built with bulk validations and processing capabilities. The file structure for the ITEM_RANGING entity is different from the item ranging entity, ITEM_LOC_RANGING. The latter would be deprecated in a future release. Key features of the ITEM_RANGING entity are:

  • Pricing at item/location can be provided in the input file. If the value is not provided, it will be defaulted based on the zone level pricing.

  • Item can be ranged only at store and warehouse level in the organizational hierarchy.

  • If you are using parent/child item hierarchy, the input data will need to be added for both the parent and the child. Parent ranging will not be defaulted to child items using this process.

  • Item/location traits are loaded separately after item ranging is completed. This will limit the data being converted in a single process.

• If you choose to run ITEM_LOC_RANGING entity, the following points should be noted:

  • If an item is to be ranged to all stores, then consider ranging it at a higher hierarchy level (that is, chain or region).

  • Consider adding only transaction-level items to the files, which will auto-range the parent items.

Pricing Conversion

It is highly recommended for the future retail seeding and active clearances conversion you use the RPM_CLEARANCE entity and the DC_RPM_SEED_FUTURE_RETAIL tasks, respectively, in the data conversion tool. The future retail seeding and active clearances conversion can also be done by scheduling the Data Conversion Seed Future Retail batch process^Foot 1 and the Clearance Conversion processes^Foot 2, respectively, using the Process Orchestration and Monitoring (POM) solution. However, the process of scheduling the pricing conversion from POM would be deprecated in a future release. For the clearance conversion, there is no difference in the input file format for the POM job and the data conversion entity except for the file delimiter. The POM job uses | (pipe) as the field separator whereas the conversion file format uses , (comma) as the delimiter.

Purchase Orders Conversion

The following should be taken into consideration while converting purchase orders using the tool:

• It is highly recommended you close all POs that can be closed in the legacy system prior to conversion. Loading a large volume of old orders may hinder the performance of the data conversion.

• The relevant item/location records required to support the purchase orders conversion should be loaded through the item/location ranging process. If not already ranged, item/location records are created during the purchase order conversion, increasing the processing time.

---

Footnote Legend

Footnote 1:

This can be scheduled in the Process Orchestration and Monitoring (POM) solution. The job that correlates with this is RPM_SFR_DATA_CONVERSION_JOB, the process is RPM_SFR_DATA_CONVERSION_PROCESS_ADHOC.

Footnote 2:

This can be scheduled in the Process Orchestration and Monitoring (POM) solution. The job that correlates with this is RPM_CLR_DATA_CONVERSION_JOB, the process is RPM_CLR_DATA_CONVERSION_PROCESS_ADHOC.