Chapter 4   Running the Import Utility


After you create CSF and CSS files, you are ready to specify the information that the Import utility needs to create the load file to load in the Public Master catalog (PMC). The Import utility uses the make command to create the load file. The make command inputs and the output targets are shown as dependency trees in and Figure 4-1 and Figure 4-2. Figure 4-1 shows the dependency tree to create a seller ontology when there is not an exiting PMC ontology. Figure 4-2 shows the dependency tree to update an existing PMC ontology.

Figure 4-1    Dependency Tree for Creating a Seller Ontology with no Existing PMC Ontology

Figure 4-2    Dependency Tree for Updating an Existing PMC Ontology

This chapter describes how to set up and run the Import utility to create the load file and load it in the PMC. It contains the following sections.

• Setting Up Your Environment

• Preparing To Run the Import Utility

• An Example of Running the Import Utility

• Loading the Load File in the PMC

Setting Up Your Environment

---

The Import utility uses the make command. To be sure that the make command runs properly, check the following items.

• Be sure that the make command is in your command path. This command is in the /usr/ccs/bin directory.

• Source either the mm.kshrc or mm.cshrc files, depending on whether you run the Import utility from a Korn shell or C-shell. These files are in the iPlanet Market Maker installation directory.

  If you get a word too long error message when you source the mm.cshrc file, you need to create a symbolic link from your root directory to the iPlanet Market Maker installation directory (usually iMM). To create this link, go to your root directory and type the command ln -s <install_dir> <link_name>, where <install_dir> is the path to your iPlanet Market Maker installation directory and <link_name> is the name of the symbolic link. After you create this link, replace all instances of the iPlanet Market Maker installation directory name in your mm.cshrc file with the directory name defined by the symbolic link.

Preparing To Run the Import Utility

---

Before you can create the load file, you need to specify the input file names. To specify this information, you edit a file named makefile in a text editor. To create the makefile file, you must start with the filenames.mk file in the following directory.

<imm_install_dir>/catalog/import/filenames.mk

Copy the filenames.mk file to your working directory. Then rename filenames.mk to makefile. This section describes the information you need to provide in the makefile file. This file contains all the information the Import utility needs to run properly, so make sure that you enter this information completely and correctly. Figure 4-3 shows a sample makefile file.

The "#" character in a makefile file is a comment. The Import utility ignores all the lines that begin with this character. Be sure to remove the "#" character when you specify an entry in the makefile file.

The makefile file contains the following sections.

• Output targets

• Required source files

• Optional files

• Ultimate targets

• Ontology Mapping

• Optional variables

• Error reports

• Command arguments

• Internal targets

• Directory locations

Specifying Output Targets

This section of the makefile contains the output targets that the Import utility uses to create the load file. You do not need to edit this section of the makefile file. You specify the output targets on the Import utility command line. See Table 4-1.

Table 4-1    Output Targets

Target	Description
ALL	Creates or updates the following files: Ontology definition (default is omd.xml) Document type definition (default is mm.dtd) PMC (default is mm.xml)
CATALOG	Creates or updates the PMC (default is mm.xml).
ONTOLOGY_MAP	Creates or updates the ontology definition file (default is omd.xml).
DTD	Creates or updates the document type definition file (default is mm.dtd).
CHECK	Checks for both name and path clash problems. The default report files names are name_clashes.xml and path_clashes.xml, respectively. These reports tell you what to do if your catalog has name or path clashes.
CLEAN	Creates or updates the name and path clash reports.
CLOBBER	Removes all files except the source input files, column structure specification, ontology map definition, and the name and path files. You use this output target to remove the intermediate files from a previous run of the Import utility.

Specifying the Required Source File Names

Table 4-2 shows the required source input file names you provide in the makefile file.

Table 4-2    Required Source Input File Names

Required Input	Default File Name or Path	Description
PWD=		For Windows compatibility only. Not required for UNIX systems.
INPUT=input.csf	input.csf	Specifies the name of your CSF file. You can use a relative path name.
CSS=$(PWD)/css.xml	css.xml	Specifies the name of your CSS file. You must use an absolute path name.
IMPORT_ROOT=$(IMM_HOME)/catalog/import	$(IMM_HOME) /catalog /import	Specifies the base location of the iPlanet Market Maker Import utility files. The default path is specified when you install the iPlanet Market Maker software. You must use an absolute path.

Specifying the Optional File Names

Table 4-3 shows the optional source input file names you can provide in the makefile file.

Table 4-3    Optional Input File Names

Option	Description
ENCODING=	Encoding of the input CSF file. For information about encoding rules, see the following URL. http://java.sun.com/products/jdk/1.2/docs/guide/internat/encoding.doc.html
NAMES=	Name of the file you specify to resolve name clashes. For an example of this file, see "Resolving Name Clashes" in Chapter 3 "Representing Catalog Information."
PATHS=	Name of the file you specify to resolve path clashes. For an example of this file, see "Resolving Path Clashes" in Chapter 3 "Representing Catalog Information."
UNITS=	Name of the XML file you use to perform attribute unit normalization. For more information about unit normalization, see "Renaming Attribute Units" in Chapter 2 "Catalog Concepts."
NAME-MAP=	Name of the XML file you use to perform attribute name normalization. For details, see "Attribute Name Normalization" in Chapter 2 "Catalog Concepts".
ATTRIBUTE-TYPES=	Name of the XML file you use to specify attribute types. For details, see "Specifying Attribute Types" in Chapter 2 "Catalog Concepts."

Specifying the Ultimate Targets

The ultimate targets are the result of running the Import utility. The CLEAN option does not remove these targets. You can use the default targets to run the Import utility, or you can change them. See Table 4-4.

Table 4-4    Ultimate Target File Names

Target	Default File Name	Description
DTD=	mm.dtd	Name of the DTD file for the load file. You must specify an absolute path. The default is $(PWD)/mm.dtd.
OUTPUT=	mm.xml	Name of the load file to load into the iPlanet Market Maker catalog database to create or update the PMC. The default file name is mm.xml.

Specifying the Ontology Mapping File

The Import utility can generate the ontology mapping file automatically, but you might need to modify this file depending on the ontologies of your seller catalog and load file. The default file name is omd.xml. For details about ontology mapping, see "Defining the Ontology Mapping" in Chapter 3 "Representing Catalog Information."

Specifying Optional Variables

The optional variables define the characteristics of your CSF file. You can change the defaults.

Table 4-5    Optional Variable Names

Variable	Description
SEPARATOR=	Name of the character that separates the text fields in the CSF. The default is the tilde ("~") character.
SKIP=	Specifies the number of header lines to skip (ignore) in the CSF file. The default is 0.

Specifying Error Reports

The error reports tell you if your seller catalog has name or path clash problems. You can change the default report file names.

Table 4-6    Error Report File Names

Report	Default File Name	Description
NAME_CLASH_REPORT=	name_clashes.xml	Specifies the file name of the name clash report that the Import utility generates when you use the CHECK option. For an example of this file, see "Resolving Name Clashes" in Chapter 3 "Representing Catalog Information."
PATH_CLASH_REPORT=	path_clashes.xml	Specifies the file name of the path clash report that the Import utility generates when you use the CHECK option. For an example of this file, see "Resolving Path Clashes" in Chapter 3 "Representing Catalog Information."

Specifying Command Arguments

The command argument specify whether or not to suppress the output messages from the programs that the Import utility uses. See Table 4-8.

Table 4-7    Command Arguments

Target	Description
# QUIET=Q	Uncomment this argument if you want to suppress the output messages from the XSLT process that the Import utility uses. The default is not to suppress the output messages.
# .SILENT	Uncomment this argument if you want to suppress the output messages from the make command that runs the Import utility uses. The default is not to suppress the output messages.

Specifying Internal Targets

The internal targets are the internal files that the Import utility uses. You can change the default names. See Table 4-8.

Table 4-8    Internal Targets

Target	Description
VENDOR=	Specifies the seller catalog file name that the Import utility maps to the PMC. The default file name is vendor.xml.
CM=	Specifies the catalog map file name. The default file name is cm.xsl.

Specifying Directory Locations

The Import utility accesses these directories for XSLT and XML processing. You should not change these directory paths unless you are creating a customized environment to run the Import utility.

Table 4-9    Directory Locations

Directory	Location
XSLT_DIR=	Specifies the path of the XSLT directory. The default path is $(IMPORT_ROOT)/xsl.
XML_DIR=	Specifies the path of the XML directory. The default path is $(IMPORT_ROOT)/xml.
DTD_DIR	Specifies the path to the directory that contains the standard Import utility .dtd files. The default path is $(IMPORT_ROOT)/dtd.

Figure 4-3 shows a sample makefile file.

Figure 4-3    Sample makefile File

# Targets

# ALL - this will update the ontology map, the DTD and the catalog

# from the input.

# CATALOG - this will use the current ontology map to make a new

# catalog. The ontology map will not be updated.

# ONTOLOGY_MAP - this will update the ontology map from the input.

# DTD - this will update the DTD from the current ontology map. The

# ontology map will not be updated

# CHECK - this will check for both name clashes and validity problems

# document

# CLEAN - a phony target that cleans up report files

# CLOBBER - a phony target that will remove everything but the source files.

#################################################################

#

#####

# SOURCE FILES

#####

#

### REQUIRED

# PWD is the name of the directory that this makefile is located in

#PWD =

# INPUT is the name of the CSF file - it can be a relative name

#INPUT=input.csf

# CSS is the name of the Column Structure Specification file. Must be

# an absolute path name

#CSS=$(PWD)/css.xml

#

# IMPORT_ROOT - the base location of the transform system. Must be an

# absolute file name.

#IMPORT_ROOT=$(IMM_HOME)/catalog/import

#include $(IMPORT_ROOT)/defaults.mk

### OPTIONAL

# ENCODING is the name of the character encoding of the input

# file. It defaults to whatever the platform default is. This must be

# an encoding understood by Java 1.2.

# See

# http://java.sun.com/products/jdk/1.2/docs/guide/internat/encoding.doc.html

# for a complete list

#ENCODING=

#

# NAMES is the name of the file containing a list of name

# aliases. It must be an absolute path. This file must be valid wrt

# $(DTD_DIR)/names.dtd. Optional. Default is no file.

#NAMES=

#

# PATHS is the name of the file which will translate a path to

# a category name. This must be an absolute path. This file must be

# valid wrt $(DTD)/paths.dtd. Optional. Default is no file

#PATHS=

#

# UNITS is the name of the file containing vendor-specific unit

# aliases. This must be an absolute path. This file must be valid wrt

# $(DTD)/units.dtd. Optional. Default is no file.

#UNITS=

#

# NAME-MAP is the name of the xml file containing a mapping between

# supplier's attribute names and those to be used by the market

# maker. This file must be valid wrt

# $(DTD)/attribute-renaming.dtd. This must be an absolute

# path. Optional. Default is no file

#NAME-MAP=

#

# ATTRIBUTE-TYPES is the name of the file containing a mapping between

# attribute names and attribute types. This must be an absolute

# path. This file must be valid wrt

# $(DTD)/attribute-types.dtd. Optional. Default is no file

#ATTRIBUTE-TYPES =$(PWD)/attr-type.xml

#

#####

# ULTIMATE TARGETS - these are never removed by the CLEAN operation

# DTD - the name of the dtd file. Optional. Absolute path. Defaults to

# $(PWD)/mm.dtd

#DTD=

#

# OUTPUT is the name of the market maker catalog

#file. Optional. Defaults to mm.xml

#OUTPUT=

#

#####

# Intermediate (generated) files

#####

# OMD is the name of the Ontology Mapping Description. This might be

# modified by hand to support a particular mapping. Optional. Defaults

# to omd.xml

#OMD=

#

#####

# OPTIONAL variables

#####

#

# SEPARATOR - this is the character that separates fields in the

# $(INPUT) file - defaults to '~'

#SEPARATOR=

#

# SKIP - this is the number of header lines to be skipped in the

# $(INPUT) file - defaults to 0

#SKIP=

#

#

#####

# Error reports

#####

# Error reports can be generated from the "CHECK" target.

#

# NAME_CLASH_REPORT is the name of the file which will contain the name check

# output. Optional. Defaults to "name_clashes.xml"

# NAME_CLASH_REPORT=

#

# PATH_CLASH_REPORT is the name of the file which will contain the path

# problem report. Optional. Defaults to "path_clashes.xml"

# PATH_CLASH_REPORT=

#

#####

# Command arguments

#####

# Uncomment the following if you don't want any "noise" from the XSLT

# processor. Defaults to no value (i.e. noisy)

# QUIET=-Q

#

# Uncomment the following if you don't want any "noise" from

# make. Defaults to not defined (i.e. noisy)

# .SILENT :

#

#####

# INTERNAL TARGETS - these files are used internally and you can

# change them as you like

#####

# VENDOR is the name of the vendor vertical file. Optional. Defaults

# to "vendor.xml"

#VENDOR=

# CM is the name of the catalog map file. Optional. Defaults to cm.xsl

#CM=

#

#####

# DIRECTORY LOCATIONS

#

# These variables define which directories various utilities are

# located in. Only change them if you know what you're doing

#####

#

# XSLT_DIR - the location of the XSLT directory. Optional. Defaults to

# $(IMPORT_ROOT)/xsl

# XSLT_DIR=

#

# XML_DIR - the location of the XML directory. Optional. Defaults to

# $(IMPORT_ROOT)/xml

# XML_DIR=

#

# DTD_DIR - the location of the directories holding the "standard"

# dtds. Optional. Defaults to $(IMPORT_ROOT)/dtd

# DTD_DIR=

#

#

include $(IMPORT_ROOT)/makefile

An Example of Running the Import Utility

---

This section provides an example using some sample files you can copy to your working directory and use to run the Import utility. This example uses the CSF and CSS files shown in Figure 4-4 and Figure 4-5.

To run the Import utility with the sample files, do the following steps.

1. Make sure that your environment is set up correctly.

2. Create a working directory to contain your sample files.

3. Copy the sample input.csf CSF file shown in Figure 4-4 to your working directory.

   Make sure that your CSF file has separate lines for each of the two items. Also make sure that you do not have a blank line at the end of your CSF file.

Figure 4-4    input.csf Example

Hard Drives~SCSI~1234452~case~15000.00~$~9.1 Gig HD~MassFastSCSI~add

Hard Drives~EIDE~1234472~case~15000.00~JPY~9.1 Gig HD~MassFastSCSI~add

4. Copy the sample css.xml CSS file shown in Figure 4-5 to your working directory.

   Make sure that you specify the correct path to the ColumnStructureSpecification.dtd file in the header of your CSS file. Replace <imm_install_dir> with your iPlanet Market Maker installation directory.

Figure 4-5    css.xml Example

<?xml version="1.0" ?>

<!DOCTYPE column-specifications SYSTEM "<imm_install_dir>/catalog/import/dtd/ColumnStructureSpecification.dtd">

<column-specifications>

<column-specification number="1" >

<category level="1" />

</column-specification>

<column-specification number="2" >

<category level="2" />

</column-specification>

<column-specification number="3" >

<value key="yes"><name><fixed>SKU</fixed></name></value>

</column-specification>

<column-specification number="4" >

<value><name><fixed>UOM</fixed></name></value>

</column-specification>

<column-specification number="5" >

<value>

<name><fixed>Price</fixed></name>

<unit><column-ref column-number="6"/></unit>

</value>

</column-specification>

<column-specification number="7" >

<value>

<name><fixed>Description</fixed></name>

</value>

</column-specification>

<column-specification number="8" >

<value>

<name><fixed>Product Name</fixed></name>

</value>

</column-specification>

<column-specification number="9" >

<action />

</column-specification>

</column-specifications>

5. To map the category name "Product Name" in the seller catalog to "Name" in the load file, use the attr-renaming.xml file shown in Figure 4-6.

   Make sure that you specify the correct path to the attribute-renaming.dtd file in the header of your file. Replace <imm_install_dir> with your iPlanet Market Maker installation directory.

Figure 4-6    attr-renaming.xml Example

<?xml version="1.0" ?>

<!DOCTYPE attributes SYSTEM "<imm_install_dir>/catalog/import/dtd/attribute-renaming.dtd">

<attributes>

<attribute>

<s_name>Product Name</s_name>

<m_name>Name</m_name>

</attribute>

</attributes>

6. To map the attribute unit of "$" in the seller catalog to "USD" in the load file, uses the units.xml file shown in Figure 4-7.

   Make sure that you specify the correct path to the units.dtd file in the header of your file. Replace <imm_install_dir> with your iPlanet Market Maker installation directory.

Figure 4-7    units.xml Example

<?xml version="1.0" ?>

<!DOCTYPE units SYSTEM "<imm_install_dir>/catalog/import/dtd/units.dtd">

<unit>

<name>$</name>

<alias>USD</alias>

</unit>

7. Copy the filenames.mk file from the <imm_install_dir>/catalog/import directory to your working directory.

8. Rename filenames.mk in your working directory to makefile.

9. Edit the makefile file in your working directory and specify the input file names.

   Be sure to remove the "#" character when you specify an entry in the makefile file. See Figure 4-8. Note that Figure 4-8 omits the parts of the makefile that you do not need to edit for this Import utility run.

Figure 4-8    Edited makefile File

...

INPUT=input.csf

# CSS is the name of the Column Structure Specification file. Must be

# an absolute path name

CSS=$(PWD)/css.xml

#

...

# NAME-MAP is the name of the xml file containing a mapping between

# supplier's attribute names and those to be used by the market

# maker. This file must be valid wrt

NAME-MAP=$(PWD)/attr-renaming.xml

#

...

#

# UNITS is the name of the file containing vendor-specific unit

# aliases. This must be an absolute path. This file must be valid wrt

# $(DTD)/units.dtd. Optional. Default is no file.

UNITS=$(PWD)/units.xml

#

...

10. To create the load file from a UNIX command line, type

    make ALL

    If successful, this command creates the mm.xml file shown in Figure 4-9.

Figure 4-9    mm.xml File Created by the Import utility

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE vortex-data-load SYSTEM "file:////<imm_install_dir>/catalog/labs/mm.dtd">

<vortex-data-load version="1.0">

<Hard_Drives name="Hard Drives" vortex-type="category">

<SCSI name="SCSI" vortex-type="category">

<Item vortex-type="item" action="add">

<Attribute vortex-type="attribute" value="1234452" name="SKU" key="yes"/>

<Attribute vortex-type="attribute" value="case" name="UOM" key="no"/>

<Attribute vortex-type="attribute" value="15000.00" dataType="currency" name="Price" key="no"/>

<Attribute vortex-type="attribute" value="9.1 Gig HD" name="Description" key="no"/>

<Attribute vortex-type="attribute" value="MassFastSCSI" name="Name" key="no"/ unit="USD"/>

</Item>

</SCSI>

<EIDE name="EIDE" vortex-type="category">

<Item vortex-type="item" action="add">

<Attribute vortex-type="attribute" value="1234472" name="SKU" key="yes"/>

<Attribute vortex-type="attribute" value="case" name="UOM" key="no"/>

<Attribute vortex-type="attribute" value="15000.00" dataType="currency" name="Price" key="no"/>

<Attribute vortex-type="attribute" value="9.1 Gig HD" name="Description" key="no"/>

<Attribute vortex-type="attribute" value="MassFastSCSI" name="Name" key="no"/ unit="JPY"/>

</Item>

</EIDE>

</Hard_Drives>

</vortex-data-load>

11. To load the mm.xml file in the iPlanet Market Maker catalog database, type the following information on a UNIX command line. Substitute your user name, password, and company name for the entries within the brackets ("<>").

    java com.iplanet.ecommerce.vortex.catalog.catimport.Loader -USER <your_user_name> -PASSWORD <your_password> -COMPANYID <your_company_name> -IN mm.xml -ontology

    For information about your load run, see the <-COMPANYID>.error and <-COMPANYID>.log files.

    For more information about how to load the load file in the PMC, see the following "Loading the Load File in the PMC" section.

Loading the Load File in the PMC

---

After you have created your mm.xml file, you can load it into the iPlanet Market Maker catalog database. Note that you must register your user and company name in an iPlanet Market Maker marketplace to load catalog information.

To load an mm.xml file into the catalog database, use the following command.

java com.iplanet.ecommerce.vortex.catalog.catimport.Loader [inputs]

The inputs to this command line are described in the following table. The required inputs to the loader are in bold in the left column in Table 4-10.

Table 4-10    Inputs to the Loader Command Line

LOADER INPUT	DEFINITION
-USER	The user name.
-PASSWORD	The user password.
-COMPANYID	Company login name or ID of the user running the loader. If you do not specify the -VENDORID option, the loader adds items under this company name or ID.
-VENDORID	Optional vendor identification, typically a company name.
-LOCALE	Specifies an optional locale other than the default. If you specify a locale, use the form <language code>-<country code>, where the language and country code comply to the ISO 639 standard. The locale for French, for example, is fr-FR.
-IN XML_file_name	Name of the load file. The default file name is mm.xml.
-LOG log_file_name	Log file name. The default file name is the <-VENDORID> name with a .log extension.
-ERROR error_file_name	Error file name. The default file name is the <-VENDORID> name with an .error extension.
-ITEM item_number	Optional item number to begin processing. The default item number is 0.
-ontology	Creates a new ontology. This option is only for marketmaker administrators and should be used very carefully. This option allows the loader to make ontology changes by adding new categories in the PMC. You should turn this option on only when you want to create a new ontology, such as when you create a catalog database for the first time. It is turned off by default.
-THREADS	Specifies the number of threads. The default is 1. In some cases increasing the number of threads can improve performance.
-MAXITEMBATCHSIZE	Specifies the number of items in a category to process in a transaction. The default is the number of items in a category as specified in the Public Master catalog XML file.
-D vortex_debugLevel	Specifies a Java property that can have the following values. 0 - No debugging output. 3 - Exceptions in output. 7 - Sql statements in output.
stats file_name	Outputs statistics. For iPlanet Market Maker internal use.