| Oracle® Retail Xstore Suite 17.0/Merchandising 16.0.1 Implementation Guide Release 17.0 E90914-06 |
|
 Previous |
 Next |
| Oracle® Retail Xstore Suite 17.0/Merchandising 16.0.1 Implementation Guide Release 17.0 E90914-06 |
|
Previous |
Next |
This chapter covers the data flow from RMS and RPM to Xcenter/Xstore.
RMS is the source of foundation data. RMS foundation data sent to Xcenter/Xstore is limited to the following:
Merchandise hierarchy
Organizational hierarchy
Store (including addresses)
Diff IDs
Diff groups
Items
VAT tax rules and item associations
Related items
RPM is the source of pricing data. RPM pricing data sent to Xcenter/Xstore is limited to the following:
Price changes
Promotions
Clearance prices
Figure 2-1 illustrates the data flow from the Merchandising applications to Xcenter/Xstore.
The following steps describe the flow in Figure 2-1:
Manual process to set up some infrequently changing foundation data in both Xstore Office and RMS, for example, currency codes.
RMS produces foundation data consumed by Xcenter. Xcenter loads this foundation data to, among other things, facilitate cross location transactions. Xcenter loads foundation data to the appropriate lead registers at individual stores.
RMS produces store-specific foundation data consumed by Xcenter (3a). Xcenter does not load this data into its own repository, but instead distributes these files to the appropriate store lead registers (3b).
RPM produces store-specific pricing data consumed by Xcenter (4a). Xcenter does not load this data into its own repository, but instead distributes these files to the appropriate store lead registers (4b).
Lead register distributes store specific foundation data to all other Xstore instances in the store. This step occurs when the store is closed, or in a 24x7 configuration, when the retail period ends.
The technical implementation of the foundation/price data from Merchandising to Xcenter/Xstore consists of three main components:
RMS writes data to .dat files using a series of .ksh extract scripts. These scripts support both kill/fill (full) and delta processing. Many of these scripts also support creating files that apply either to all stores (for Xcenter, to facilitate cross store processing) or store-specific files. For an example, see Figure 2-5.
There are some entity specific variations, but RMS uses a general pattern for foundation data bulk export as shown in Figure 2-2.
The following steps describe the data bulk export flow shown in Figure 2-2:
A business user using the RMS application UI (1A) or an API/Batch Process (1B) performs an insert/update/delete on a System of Record (SOR) table.
Trigger on SOR entity table fires on insert/update/delete (2A). Trigger writes new/changed/deleted information to the outbound staging table (2B).
In delta mode, the program reads the bulk export staging table to get recently created, modified, and deleted records and writes them to a file. Records are marked as exported.
In full mode, the program reads all current records from the SOR table and writes them to a file. Note that recently deleted records are not part of the data set.
export_stg_purge.ksh drops aged partitions from the export outbound staging tables.
|
Note: If bulk extract programs are not run for some time, it is possible that delta records will be purged without having been exported. It is important to run these jobs daily. |
For more detailed information, see the following documents:
Retail Reference Architecture available on My Oracle Support
Oracle Retail Merchandising System Operations Guide, Volume 1 - Batch Overviews and Designs
The following batch jobs are used for the integration:
export_merchhier.ksh
export_orghier.ksh
export_stores.ksh
export_diffs.ksh
export_diffgrp.ksh
export_itemmaster.ksh
export_itemloc.ksh
export_itemvat.ksh
export_relitem.ksh
export_vat.ksh
export_stg_purge.ksh
For more information, see the Oracle Retail Merchandising System Operations Guide, Volume 1 - Batch Overviews and Designs.
RPM writes data to .dat files using a set of base processes. These base processes send pricing data to both Xcenter/Xstore and Oracle Retail Store Inventory Management (SIM). This ensures consistent price information across the Xstore applications. For an example, see Figure 2-5.
The following steps describe the flow in Figure 2-3:
A Pricing Analyst creates and approves price events (regular price changes, clearances, and promotion details) in the RPM UI (1A). Price events are created and approved in a bulk fashion using the Price Event Injector batch process (1B).
With data created on the SOR tables in RPM, the conflict checking process stages data on the outbound (payload) tables (2).
Data on the outbound staging tables is read by the flat file extraction batch processes to create delimited flat files. The associated outbound staging data is flagged as having been extracted as part of these batch processes (3).
purgePayloadsBatch.sh purges data from the staged outbound tables that has already been extracted. This purge batch is run in conjunction with the extract batch processes (4).
For more detailed information, see the following documents:
Retail Reference Architecture available on My Oracle Support
Oracle Retail Price Management Operations Guide
The following process describes the flow of the Merchandising data file import:
Xstore Office plays a central role in the Merchandising data importing. It periodically polls the configured auto-deploy directory on the file system. The interval is configurable, and is 15 minutes by default. For information on how to configure these settings, see the Oracle Retail Xstore Office User Guide.
Data files (.dat) generated by the RMS/RPM extract programs are delivered to Xstore Office's auto-deploy directory in the form of a zip-format archive file using the file extension .momzip. It is the System Integrator's responsibility to create the archive, and deliver it using a preferred file transfer protocol.
Once the archive is detected by Xstore Office, it regroups its content into deployments based on their targeted locations. For data files targeted to corporate, it invokes DataLoader immediately to import them into the Xcenter data source. For data files targeted to a store, it creates a deployment of these files to the store, and uploads them to an Apache Server for the store to download when updates are applied. For a traditional store, this happens at store close, and for a 24x7 store, this happens when the retail period changes. For details on the set of Merchandising files targeted to corporate or stores, see "Merchandising File Consumption by Location".
Once a store is closed or when the retail period changes in a 24x7 configuration, Xenvironment of the lead register pulls down the files from the Apache server and runs DataLoader to import all the files deployed into the store primary database.
DataLoader is the Xstore component responsible for translating RPM and RMS flat files into database data that can be used by Xstore. It can consume .mnt files from third-party sources, or the foundation data batch .dat files produced by RMS and RPM.
The DataLoader interacts with Xstore Office, Xcenter, Xenvironment, and Xstore Point of Service to provide a complete automated solution for the propagation of foundation data changes to the centralized and store-level databases used in an enterprise Xstore deployment. Xstore data not supplied by RPM and RMS can also be loaded by the DataLoader using its native .mnt format.
The DataLoader is designed to adapt flat files of data into relational data that Xstore can use. These flat files are referred to generically as data files within the DataLoader. Each field in a data file is delimited by a vertical bar (|) character. The DataLoader is configured to detect file types so it can process a data file's lines in distinct units of work appropriate for the type of file. For most file types, a unit of work corresponds to a single line of flat file data; however, RPM promotion files are an example of a file type where a unit of work can consist of multiple lines of flat file data.
If a failure occurs during DataLoader processing of a data file, all SQL statements associated with the unit of work are rolled back and the error is logged. Processing continues with the next unit of work in the data file.
For more detailed information, see the following documents:
Retail Reference Architecture
Oracle Retail Xstore Point of Service Host Interface Guide available on My Oracle Support.
Both documents are available on My Oracle Support.
DataLoader is configured at the corporate and store level. It is responsible for detecting and sorting incoming data files, and iterating through them to convert each file record into persistable objects (DAO or SQLQuery) and writing them into the database. All its major components are spring loaded from dataloader-bean.xml, and are therefore highly customizable.
Figure 2-4 shows the flow of the DataLoader.
The following sections describe each major execution phase.
File Detection
DataLoader is configured with a list of detectors to identify known file types that can be processed. Unknown file types are skipped and not processed. A Merchandising file detector is configured to identify all types of Merchandising data files and their meta data.
The detection is based solely on file names. Regular expressions are configured to perform pattern matching in file name to identify Merchandising file types and its meta data including target store ID, fill type, timestamp, and line count:
Merchandising file type detection
A file name is matched against regular expressions configured to detect its Merchandising type. If no match is found, the file is not a Merchandising file type. The keys in the bean configuration are the Merchandising file types that Xstore/Xcenter care about.
Target store detection
Target store ID is used by DataLoader as well as Xstore Office to determine the deployment target of a Merchandising file.
A file name is matched against regular expressions configured to detect its target store ID. Not all Merchandising file types have a target store ID configured:
If a store ID is not detected, the file is deployed to all stores and imported into Xcenter.
If a store ID is detected and is corp, the file is imported into Xcenter only.
If a store ID is detected and is not corp, it is deployed to the store. With the exception of Item Header batch, it is also imported into Xcenter.
Timestamp detection
Timestamp is used by DataLoader to sort the files. For more details, see "File Sorting". A file name is matched against regular expressions configured to detect its timestamp.
Line count detection
Line count is used by DataLoader to validate a file. If the number of lines in the file (excluding FHEAD and FTAIL) does not match the line count, a warning is logged. A file name is matched against regular expressions configured to detect its line count. Only RMS extracts support line count in their file names.
File Sorting
There are some data dependencies when importing RMS files into Xstore, such as related item detail that needs to be imported after the related item header. When DataLoader is called to import multiple Merchandising files in the same deployment, it applies sorting to the files before importing them.
A detector is configured to have a sorting strategy, which is used to sort all the files the detector detects. A Merchandising file sorting strategy bean is configured for the Merchandising file detector to perform sorting for all Merchandising files based on their file types. Files of the same Merchandising file type are sorted based on their timestamps. Out of the box the following sorting order is specified:
Org Hierarchy
Store
Store Address
Merchandise Hierarchy
VAT
Diff Group Head
Diff Group Detail
Diff
Item Head
Item Loc
VAT Item
Related Item Head
Related Item Detail
Regular Price Change
Clearance Price Change
Promotion Price Change
File Loading Dependency
Although the sorting strategy configuration lists all Merchandising file types, not all file types have file loading dependencies. The actual dependencies are shown in the following table:
| File Type | Depends on |
|---|---|
| VAT Item | Item Loc |
| Store Address | Store |
| Related Item Detail | Related Item Head, Item Loc |
| Item Loc | Item Head |
| Item Head | Diff, Diff Group Detail, Diff Group Head, Merchandise Hierarchy |
| Diff Group Detail | Diff Group Head |
| Diff | Diff Group Detail, Diff Group Head |
File Iteration and Transformation
DataLoader processes each file in the sorted order. It invokes a file iterator to process each file. A file iterator implements Java Iterator interface. During each iteration, it transforms flat file records into a list of IPersistable (DAO or SQLQuery) objects, and returns them.
A Merchandising file iterator is configured for each Merchandising file type. It processes lines between unit dividers as a data unit that should be transformed together:
A single line iterator that expects each line in the file, other than FHEAD or FTAIL, is a data unit that gets transformed during each iteration. One and only one line in a unit is expected. An exception is raised if that is not the case.
A multi-line iterator expects multiple lines to form a data unit that gets transformed together during each iteration. A unit may contain one or more lines. Out of the box, only promotion price change is configured to have a multi-line iterator.
Unit dividers are lines that end a unit. They are configured as unit definitions for each Merchandising file type.
A Merchandising transformer is called to convert a unit of data from a flat file to a list of IPersistable (DAO or SQLQuery) objects. A transformer is configured for each Merchandising file type.
All Merchandising transformers implement the IMOMDataTransformer interface, which defines two APIs:
The transform API is invoked by the iterator in each iteration. It does all the transformation to turn a unit of flat file data to a list of IPersistable objects to create, update, or delete foundation data records in database.
The purgeData API is invoked once for a file by the iterator. It is only called if the file is for a full reload. It returns a list of IPersistable objects to remove all existing records sourced from Merchandising.
Persisting into the Database
DataLoader saves IPersistable objects to database in batches. A batch contains a list of AtomicPersistables objects. The maximum number of AtomicPersistables objects in a batch is configurable. An AtomicPersistables is a container of a group of IPersistable objects that must all be persisted or rolled back together as a unit. All IPersistable objects returned in one iteration are grouped into one AtomicPersistable object.
DataLoader first attempts to persist and commit all IPersistable objects from all AtomicPersistables objects in a batch together. If it fails, it tries to persist and commit IPersistable objects from one AutomiPersistables at a time. The number of succeeded and failed records are written to summary.ini files.
The files produced by the RPM and RMS extract programs containing data loaded into the Xcenter and Xstore databases comprise four data sets. A data file's targeted location is specified in its file name:
Data loaded into Xcenter
Data loaded into Xcenter and all stores
Data loaded into one store
Data Loaded into all stores
Figure 2-5 illustrates the type of RMS and RPM files loaded at each location, using a two store chain example.
