Importing Flex Assets with the BulkLoader Utility

33 Importing Flex Assets with the BulkLoader Utility

You need the BulkLoader utility to import flex assets when you’re setting up your WebCenter Sites system.

Topics:

33.1 About the BulkLoader Utility

With the BulkLoader utility, you can quickly extract large amounts of flex asset data in a user-defined way from your own data sources. This utility can import that data into the WebCenter Sites database on any of your systems (development, management, testing, or delivery).

The extraction mechanism is abstracted away using a Java interface that customers can implement. BulkLoader invokes methods on this interface to extract input data from your data sources. For backward functional and data compatibility, WebCenter Sites also includes an implementation of this Java interface so that BulkLoader will still be able to extract data from an external JDBC-compliant data source.

The following topics provide information about the BulkLoader utility:

33.1.1 Understanding BulkLoader Features

Features in BulkLoader include the following:

Support for a user-defined extraction mechanism, using a Java API. Users can provide a custom implementation of this extraction interface or use the built-in support for extracting from a JDBC data source.
Support for inserts, voids, and updates of flex asset and group data.
Support for incremental inserts, voids and updates.
Performance improvements for higher throughput, using concurrent multi-threaded import operations while data extraction is in progress.
Support for chunk (slice) processing of input data.
Support for importing asset data that belongs to multiple flex families.
Backward functional and data compatibility. Supports importing asset data from an external JDBC source.

33.1.2 How BulkLoader Works

The BulkLoader has been redesigned for higher performance, throughput, and scalability. Instead of reading all input data and then generating output SQL files, BulkLoader reads input data in chunks. As soon as each chunk is read, it is handed over to an import thread while the main BulkLoader thread goes back to read the next chunk. The import thread uses a direct JDBC connection to the WebCenter Sites database. In this way, reading and importing are done in parallel, thereby achieving higher throughput. For scalability, the number of BulkLoader import threads can be increased using the database computer's hardware's additional CPUs and an I/O configuration that supports higher concurrency.

The BulkLoader utility requires a configuration file containing parameters that specify the number of processing threads, the name of the Java class that implements the data extraction interface, commit frequency, the starting unique ID to be used as the asset ID, and more.

The following figures show a client-specific implementation and the built-in ready-to-use implementation supplied by WebCenter Sites.

Figure 33-1 Client-specific implementation of BulkLoader

Description of "Figure 33-1 Client-specific implementation of BulkLoader"

Figure 33-2 Built-in OOTB (out-of-the-box) implementation of BulkLoader

Description of "Figure 33-2 Built-in OOTB (out-of-the-box) implementation of BulkLoader"

33.1.3 About Using the BulkLoader Utility

There are two ways to use the BulkLoader depending on how you supply input data to import into the WebCenter Sites database.

To use BulkLoader to import input data from an external JDBC data source, you provide input data in a flat table or view.
To provide your own way of supplying input data to the BulkLoader, you use a Java object that implements the extraction interface, IDataExtract.

Note:

For reference, sample BulkLoader code is provided on the WebCenter Sites installation medium, in the Samples folder. The same folder contains the readme.txt file that describes the sample files.

33.1.4 Importing Flex Assets from Flat Tables

This section describes the general procedure that you use to import flex assets with BulkLoader, followed by subsequent sections that describe each step in detail. Using this model, import new flex assets and parents and void assets that were previously imported. This model also supports changing and deleting attribute values for existing assets.

33.1.4.1 The Basic Steps

To import flex assets with the BulkLoader utility:

Use XMLPost to import the structural assets into the WebCenter Sites database on the management system. The structural flex assets are as follows: attribute editors, flex attributes, flex parent definitions, flex definitions, and flex parent assets.
Write a view or a stored procedure that gives you a view of the source database that you want to import into the WebCenter Sites database as a flat table. This flat table is your source table.
In the same source database, create a mapping table with two columns: one column that lists the names of the columns in the source file and the other column that lists the names that are used for those attributes in the WebCenter Sites database.
Code a configuration file that identifies the source table and the mapping table.
Put the configuration file on a system from which you have access to both the WebCenter Sites database on the management system, and to your source database.
Stop the application server on the management system.
Run the BulkLoader utility. BulkLoader will import the flex asset data and gives the feedback in a table named bulk_feedback, that has been created at the input data source.
Restart the application server on the management system.
Use the BulkApprover utility to approve all of the assets that were loaded.

Note:

Because the BulkLoader utility is designed for speed, it does not check for the existence of the attributes or flex parent definitions or flex definitions. You must import all of the structural asset types before you run the BulkLoader utility.

33.1.4.2 Driver Requirements

The BulkLoader requires JDBC (or Java DataBase Connectivity) drivers, which are not provided by Oracle Corporation. You must obtain JDBC drivers for both the source database and the destination database, that is, the WebCenter Sites database. For an ODBC-compliant source database, use a JDBC-ODBC bridge, which is included as part of the Java SDK.

33.1.4.3 Requirement for DB2

Run the usejdbc2.bat file on the client computer before you can use BulkLoader. Run the batch file once, and then run BulkLoader as usual.

33.1.5 When to Use XMLPost to Import Structural Assets

Use XMLPost and the RemoteContentPost posting element to import the structural assets into the WebCenter Sites database on the management system. Import assets of the following types:

attribute editors
flex attributes
flex parent definitions
flex definitions
flex parent assets

See About Importing the Structural Asset Types in the Flex Model.

33.1.6 Creating the Input Table (Data Source)

You must create input flat tables (data sources) for holding all new asset data and for holding update data. These are flat tables/views in which each row corresponds to a single flex asset item and each column corresponds to a flex attribute asset for the BulkLoader utility.

There is no requirement regarding the names of columns in the data source, but you must supply a separate mapping table, described in Creating the Mapping Table.

This section includes the following topics:

33.1.6.1 Inserts

The name of the data source table is specified by the inputTable parameter in the configuration file.

The source table must also include the names of the following four columns, which you specify in the configuration file with the following properties:

inputTableTemplateColumn: The name of the column in the source table that holds the names of the flex definitions.
inputTableNameColumn: The name of the column in the source table that holds the names of the flex assets. The name of this column cannot exceed 64 characters.
inputTableDescriptionColumn: The name of the column in the source table that holds the description of the flex assets.
inputTableGroupsColumn: The name of the column in the source table that holds the parent names. Each value in this column can include multiple flex parent names, separated by the multivalueDelimeter character, which is defined in the configuration file.

Note that you can optionally specify the name of the column that serves as a unique identifier for each input item, using the following parameter in the configuration file: inputTableUniqueIdColumn. If there is no value assigned for this parameter, BulkLoader will generate a unique identifier for each input item and store it in a mapping table (bulkloader_ids) in the WebCenter Sites database.

The following is an example of a source table (input table):

Figure 33-3 Sample Source Input Table

Description of "Figure 33-3 Sample Source Input Table"

Based on the column names in this source table, the source table properties in the corresponding configuration file would be set as follows:

inputTableTemplateColumn=SNSTEMPLATE
inputTableNameColumn=SNSNAME
inputTableDescriptionColumn=SNSDESCRIPTION
inputTableGroupsColumn=SNSGROUPS

33.1.6.2 Updates

To update attribute data for existing assets, to add new parents or delete existing parents for existing assets, you use the update parameter.

Use the inputTableForUpdates parameter in the configuration file to specify the name of the data source table. The source table must also include the names of the following three columns, which you specify in the configuration file with the following properties:

inputTableForUpdatesUniqueIdColumn: The name of the column in the source table that uniquely identifies the flex asset or parent in the WebCenter Sites database.
inputTableForUpdatesDeleteGroupsColumn: The name of the column in the source table that specifies a list of parents to be deleted for the current flex asset.
inputTableForUpdatesAddGroupsColumn: The name of the column in the source table that specifies a list of parents to be added for the current flex asset.

BulkLoader interprets column values as follows when applying updates to the attributes:

A null value in a specific attribute column indicates that the attribute for the current flex asset should be deleted. For example, a null value in the deletegroups column indicates that no parents have to be deleted. A null value in the addgroups column indicates that no parents have to be added.
A non-null value indicates that the existing attribute value should be replaced with the given value. For example, a non-null value in the deletegroups column specifies a list of parents to be deleted. A non-null value for addgroups denotes the addition of new parents to a given flex asset.

33.1.7 Creating the Mapping Table

You must also create a mapping table for the BulkLoader utility, and it must have the following two columns:

A column that holds the names of the flex attribute columns in your flat data source.
A column that holds their corresponding names in the WebCenter Sites database.

The mapping table provides a one-to-one correspondence between these two columns. For example, your source table might have a column of vendor names with an automatically generated name like A96714328445 that maps to a product attribute asset named, simply, VENDOR_ID.

You include the following configuration file properties for the mapping table:

inputAttributeMapTable: The name of the mapping table file.
inputAttributeMapTableKeyCol: The name of the column in the mapping table that lists the attribute names in the source table.
inputAttributeMapTableValCol: The name of the column that lists the corresponding attribute asset names in the WebCenter Sites database.

The following is an example of a mapping table:

Figure 33-4 Sample Mapping Table

Description of "Figure 33-4 Sample Mapping Table"

Based on the column names in this source table, the source table properties in the corresponding configuration file would be set as follows:

inputAttributeMapTable=93ATTR_MAP
inputAttributeMapTableKeyCol=SOURCENAME
inputAttributeMapTableValCol=ATTRIBUTENAME

33.1.8 Creating the BulkLoader Configuration File

You configure the BulkLoader utility by creating a configuration file for it that has the properties described in this section. You can name the file anything you want.

You set the properties in the file according to the following syntax:

property=value

Note:

All property names and values in the configuration file are case-sensitive.

This section includes the following topics:

33.1.8.1 BulkLoader Configuration File Properties

This table describes properties in a BulkLoader configuration file:

Table 33-1 BulkLoader Configuration File Properties

Property Name	Required/ Optional	Comments
`maxThreads`	Required	The maximum number of concurrent processing threads. This can be the number of database connections to the WebCenter Sites database server. Use as many threads as the number of CPUs on the database host. For a single CPU database host, set it to 2. Example: `4`
`dataSliceSize`	Required	Number of items retrieved in one read request; this number will also be processed by a single processing thread. Example: `2000`
`dataExtractionImplClass`	Required	User-specific Implementation class for data extraction API. Needs a constructor with (`String configFilename`) signature. The one mentioned here is a reference implementation class for backward compatibility. Data in flat tables. Default value (ready-to-use): `com.openmarket.gatorbulk.objects.DataExtractImpl`
`initId`	Required	Starting WebCenter Sites ID used the very first time BulkLoader operates; subsequently will use the value from `idSyncFile`. Example: `800000000000`
`idSyncFile`	Required	Next available WebCenter Sites ID is saved in this file; updated during a BulkLoader session. Example: `C:\FutureTense\BulkLoaderId.txt`
`idPoolSize`	Required	Each time BulkLoader needs to generate WebCenter Sites IDs, it collects this many IDs and caches in memory. A good estimate is (number of assets * average number of attributes *2). Example: `1000`
`commitFrequency`	Required	Number of flex asset groups to be part of a database transaction. Example: `100`
`outputJdbcDriver`	Required	The name of the JDBC driver class to access the WebCenter Sites database. The value here reflects the Oracle 9.0 driver. Example: `oracle.jdbc.driver.OracleDriver`
`outputJdbcURL`	Required	The JDBC URL. The following example value is a typical type2 oracle JDBC driver URL: `Jdbc:oracle:oci8:@foo`
`outputJdbcUsername`	Required	WebCenter Sites database user name.
`outputJdbcPassword`	Required	WebCenter Sites database user password.
`inputTable`	Required	Name of the flat, input table from which new asset data is inserted.
`inputAttributeMapTable`	Required	Name of the mapping table that lists the source table columns and the corresponding attribute names.
`inputAttributeMapTableKeyCol`	Required	The name of the column in the mapping table that lists the source table column names. For example: `inputAttributeMapTableKeyCol=SOURCENAME`
`inputAttributeMapTableValCol`	Required	The name of the column in the mapping table that lists the corresponding attribute names. For example: `inputAttributeMapTableValCol=ATTRIBUTENAME`
`inputTableDescriptionColumn`	Required	The name of the column in the source table that contains the descriptions of the flex assets. For example: `inputTableDescriptionColumn=SNSDESCRIPTION`
`inputTableGroupsColumn`	Required	Name of the column in the source table that contains the names of parents. Each value can include several parents, separated by the `multivalueDelimeter` character, which is defined in the configuration file. For example: `inputTableGroupsColumn=SNSGROUP`
`inputTableNameColumn`	Required	The name of the column in the source table that contains the name of the product (or advanced article or advanced image) for each row. For example: `inputTableNameColumn=SNSNAME`
`inputTableTemplateColumn`	Required	The name of the column in the source table that contains the flex definitions. For example: `inputTableTemplateColumn=SNSTEMPLATE`
`createdby`	Required	The user name that you want to be entered in the `createdby` field for your flex assets. For example: `createdby=editor`
`multivalueDelimeter`	Required	The delimiter that separates multiple attribute values. The default character is the semicolon (;). For example: `multivalueDelimiter=;`
`siteName`	Required	The name of the site. All products will behave as if they were created under this site. For example: `siteName=AA Illumination`
`status`	Required	The status code for all imported flex assets. You should set this to `PL` for imported. For example: `status=PL`
`tableProducts`	Required	Name of the flex asset type as defined in the WebCenter Sites database.
`inputTableUniqueIdColumn`	Required	Name of the column in the source table that serves as a unique identifier when importing a new flex asset. This will be used for any subsequent updates and void operations. To allow BulkLoader to generate unique identifiers, leave this value blank.
`targetName`	Required	Name of the publish target, as defined in WebCenter Sites.
`renderTemplate`	Optional	Name of the template used for rendering flex assets (deprecated).
`inputFeedbackTable`	Required	Name of the table that BulkLoader creates and uses for recording the processing feedback for every input item that was processed. Note that this table is created in the input data source.
`inputTableForUpdates`	Optional	Needed only if an update action is specified when running the BulkLoader utility. Otherwise, this can be an empty value. This is the name of the source table that contains attributes and parents that need updates.
`inputTableForUpdatesUniqueIdColumn`	Optional	Needed only if an update action is specified when running the BulkLoader utility. This is the name of the column in the source table that specifies a unique identifier for the flex asset.
`inputTableForUpdatesDeleteGroupsColumn`	Optional	Needed only if update action is specified and you have one or more flex assets that need one or more parents to be deleted. This is the name of the column in the source table that specifies the list of parents to be deleted.
`inputTableForUpdatesAddGroupsColumn`	Optional	Needed only if an update action is specified and you have one or more flex assets that need one or more parents to be added. This is the name of the column in the source table that specifies a list of parents to be added.
`inputLimitRows`	Optional	Needed only for testing. Limits the number of input items processed for each action (insert, void, or update).
`updatedby`	Optional	The user name that you want to be entered in the `createdby` field for your flex assets. For example: `updateby=editor`
`updatedstatus`	Optional	The status code for all updated flex assets. This must be set to `ED`. For example: `updatestatus=ED`

33.1.8.2 Setting the initID Parameter

The initID parameter is the seed value that the BulkLoader starts at and increments from when creating a unique asset ID for each asset. You must choose a seed value number that allows the BulkLoader to create a contiguous block of ID numbers that cannot cause ID conflicts with existing (or future) asset ID numbers that are generated by WebCenter Sites.

Currently, WebCenter Sites starts at 1 trillion for the asset IDs that it creates. To be sure that you won't have conflicts, select a number low enough that when the BulkLoader utility is done, the highest ID number is under 900,000,000,000.

The BulkLoader creates one asset for each row/column value in the data source table. Each output table row requires its own unique asset ID.

Use these guidelines to determine the approximate number of asset IDs that are created by the BulkLoader utility:

Five rows for each flex asset, plus
Two rows per attribute for each flex asset

For example, if your data source table contains the following:

10,000 product assets
20 attributes per product (as determined by the product definition)
10 inherited attributes per product (as determined by the product parent definitions)

Then you have to allow for the following number of IDs:

(5 x 10,000) + (2 x 30 x 10,000) = 50,000 + 600,000 = 650,000 asset IDs

If your initID value is 800,000,000,000, then the BulkLoader creates ID numbers ranging from 800,000,000,000 to approximately 800,000,650,000.

33.1.8.3 Example Configuration File

The following is an example of a BulkLoader configuration file:

# New BulkLoader configuration for backward compatibility
#
# input datasource configuration
inputJdbcDriver=sun.jdbc.odbc.JdbcOdbcDriver
inputJdbcURL=jdbc:odbc:access-db-conn
inputJdbcUsername=
inputJdbcPassword=
#
# Source tables
#
inputTable=PRD_FLAT_50000
inputAttributeMapTable=PRD_FLAT_ATTRIBUTE_MAP
inputAttributeMapTableKeyCol=SOURCENAME
inputAttributeMapTableValCol=ATTRIBUTENAME
#
# input column names
#
inputTableTemplateColumn=CCTemplate
inputTableNameColumn=CCName
inputTableDescriptionColumn=CCDescription
inputTableGroupsColumn=CCGroups
#
# WebCenter Sites database
#
#  This database is always used for looking up Attributes, #  Product Types and Product Group Types. #  Data is imported into this database.
#
outputJdbcDriver=oracle.jdbc.driver.OracleDriver
outputJdbcURL=jdbc:oracle:oci8:@foo
outputJdbcUsername=csuser
outputJdbcPassword=csuser
#
# Data-specific settings
#
siteName=AA Illumination
targetName=Mirror Publish to burst37
initId=800000000000
createdby=user_designer
status=PL
renderTemplate=CLighting Detail
MAX_ATTRIBUTES=100
multivalueDelimiter=;
commitFrequency=50
#
# The following denotes the flex asset type that we are importing.
tableProducts=Products
#
# Additional information needed for BulkLoader
maxThreads=2
# dataSliceSize 0 means read all input data in one slice.
dataSliceSize=500
dataExtractionImplClass=com.openmarket.gatorbulk.objects.DataExtractImpl
idSyncFile=C:\\FutureTense50\\bulk_uniqueid.dat
idPoolSize=50000
# For inserts
inputTableUniqueIdColumn=
inputFeedbackTable=bulk_feedback
# For updates
inputTableForUpdates=prod_flat_2_upd
inputTableForUpdatesUniqueIdColumn=input_id
inputTableForUpdatesDeleteGroupsColumn=CCGroups
inputTableForUpdatesAddGroupsColumn=
inputLimitRows=1000
############################################################

33.1.9 Running the BulkLoader Utility

Before you begin, be sure that you have the appropriate JDBC drivers for both your source database and your target WebCenter Sites database.

To run the BulkLoader utility:

In the Admin interface, disable Lucene search engine indexing. See Disabling the Lucene Search Engine in Administering Oracle WebCenter Sites.
Put the configuration file on a system from which you have access to both the WebCenter Sites database on the management system, and to your source database.
Stop the application server on the management system.
To get Bulkloader up and running, set the following directories to classpath: <ORACLE_HOME>\wcsites\webcentersites\sites-home\lib\* and <ORACLE_HOME>\oracle_common\modules\clients\*.
If WebCenter Sites is using the SQL Server database, include in classpath: Oracle_Home\oracle_common\modules\*.

Enter the following command, all on a single line, with paths that are appropriate for your installation:

For UNIX

java -ms16m -mx256m -cp <ORACLE_HOME>/wcsites/webcentersites/sites-home/lib/*:<ORACLE_HOME>/oracle_common/modules/clients/* com.openmarket.gatorbulk.objects.BulkLoader config=bulkloader.ini action=<insert|void|update> validate=<yes|no>

For Windows

java -ms16m -mx256m -cp <ORACLE_HOME>\wcsites\webcentersites\sites-home\lib\*:<ORACLE_HOME>\oracle_common\modules\clients\* com.openmarket.gatorbulk.objects.BulkLoader config=bulkloader.ini action=<insert|void|update> validate=<yes|no>

Note that the action parameter specifies what BulkLoader needs to do: insert, void, or update. Setting the validate parameter to yes makes BulkLoader do extra validations during updates and voids. You may also have to increase the memory for the JVM, depending on the size of your input data.

Examine the screen output to be sure that the BulkLoader utility was able to connect to the appropriate database.

33.1.10 Enabling Access to Imported Assets in the Contributor Interface

To access assets imported with the BulkLoader utility from the Oracle WebCenter Sites: Contributor interface, complete the following steps:

Re-enable the Lucene search engine indexing. See Setting Up Search Indices in Administering Oracle WebCenter Sites.
Add the assets you imported with the BulkLoader utility to the search index. See Adding Asset Types to the Search Index in Administering Oracle WebCenter Sites.
Verify the assets are accessible from the Contributor interface, by conducting a search, in the Contributor interface, for the assets you imported with the BulkLoader utility. For instructions about conducting searches in the Contributor interface, see Finding and Organizing Assets in Using Oracle WebCenter Sites.

33.1.11 Reviewing Feedback Information

After the BulkLoader utility completes an operation, review the feedback information in the bulk_feedback table that is located in your input data source. That table contains information about all the input items that BulkLoader processed.

After reviewing that information, take any corrective actions that might be necessary. If you modify any of your input data, run BulkLoader again to verify that the errors were corrected.

33.1.12 Approving and Publishing the Assets to the Delivery System

Use the BulkApprover utility to approve the assets that you just loaded. See Using BulkApprover.

33.2 Importing Flex Assets Using a Custom Extraction Mechanism

Sometimes users need alternative mechanisms to provide input asset data to BulkLoader. In such cases, the data may have to be gathered from multiple types of sources, such as XML documents, files, and legacy databases. To accomplish that, users can implement their own mechanism to provide data to BulkLoader, using the Java interface com.openmarket.bulkloader.interfaces.IDataExtract, which is provided with WebCenter Sites.

A user can implement a Java object supporting IDataExtract and specify the Java object in the BulkLoader configuration file. BulkLoader will then invoke methods on this interface to initialize a read request, to repetitively read chunks of input data and then signal the end of the read request. This interface also has a method that provides import feedback from the BulkLoader utility, which can be used by the input provider to know the status of import and know any errors that may occur during import.

There are three Java interfaces that can help users with custom implementations of IDataExtract:

IDataExtract: Required for any custom extraction.
IPopulateDataSlice: Provides data to the BulkLoader utility. A container object supporting this interface is created by BulkLoader and passed into the client.
IFeedback: Provides the status of each input item that has been processed by the BulkLoader. A feedback object that is created and populated by BulkLoader import thread is passed into the client.

See these topics:

33.2.1 IDataExtract Interface

This interface is required for any custom extraction.

The following is a sample code that implements this interface:

com.openmarket.gatorbulk.interfaces.IDataExtract

package com.openmarket.gatorbulk.interfaces;
import java.util.Iterator;

/**
 * To be implemented by input data provider. 
 * Interface for extracting data from an input source 
 * for BulkLoader.
 * BulkLoader loads an object supporting this interface and invokes 
 * the GetNextInputDataSet() method on this interface repeatedly to 
 * fetch data in batches.
 */

public interface IDataExtract {

  public  final int HAS_DATA    = 100;
  public  final int NO_DATA = 101;

  public  final int SUCCESS = 0;
  public  final int ERROR = -1;

  public  final int INSERT_ASSETS = 1000;
  public  final int VOID_ASSETS = 1010;
  public  final int UPDATE_ASSETS = 1020;
  public  final int NONE_ASSETS = 1030;

  /**
   * Begin requesting input data; tells the client to
   * start the database query, get a cursor, etc.
   * @param requestType
   *    IDataExtract.INSERT_ASSETS,
   *    IDataExtract.VOID_ASSETS,
   *    IDataExtract.UPDATE_ASSETS
   * @param sliceOrNot  true/false
   * true - if data will be requested in batches 
   * false - data will be requested all in one attempt
   * @param sliceSize  >0 number of rows to be 
   * retrieved in one data set
   * @return none
   * @exception java.lang.Exception
   */

  public void InitRequestInputData(int requestType, 
  boolean sliceOrNot, int sliceSize) throws Exception ;

  /**
   * Get a set/slice of input data records.
   * @param dataSlice object to be populated using the
   * methods from IPopulateDataSlice 
   * @return IDataExtract.HAS_DATA when dataSlice has some data,
   *    IDataExtract.NO_DATA when there is no data,
   *    IDataExtract.ERROR when there is an error
   * @exception java.lang.Exception    
   */

  public int  GetNextInputDataSet(IPopulateDataSlice dataSlice)
  throws Exception;

  /**
   * Signal the end of extracting data for given request type
   * @param requestType
   *    IDataExtract.INSERT_ASSETS,
   *    IDataExtract.VOID_ASSETS,
   *    IDataExtract.UPDATE_ASSETS
   * @return none
   * @exception java.lang.Exception
   */

  public void EndRequestInputData(int requestType) 
    throws Exception;

  /**
   * Update the client as to what happened to input data
   * processing. Note that this method would be called by multiple
   * threads, with each thread passing its own IFeedback
   * handle. The implementor of this method should write
   * thread-safe code.
   * @param requestType
   *    IDataExtract.InsertAsset,
   *    IDataExtract.VoidAsset,
   *    IDataExtract.UpdateAsset
   * @param processingStatus - An object containing processing
   * status for all items in one dataset. The implementor of this
   * interface should invoke the IFeedback interface
   * methods on processingStatus to get status for individual
   * rows. This method will be invoked by multiple BulkLoader
   * threads, so make sure this method is implemented in a
   * thread-safe way.
   * @return none
   * @exception java.lang.Exception    
   */

  void UpdateStatus(int requestType, IFeedback  processingStatus) 
    throws Exception;
}

Implementation Notes for IDataExtract

The Java object implementing IDataExtract needs to have a constructor with a string parameter. BulkLoader will pass the name of its configuration file to the constructor when instantiating this object.

The method UpdateStatus(..) is invoked by multiple BulkLoader threads, so the implementation of this method should be thread-safe.

This table lists and describes the configuration parameters for the BulkLoader utility when using custom data extraction method:

Table 33-2 Configuration Parameters for BulkLoader

Property Name	Required/ Optional	Comments
`maxThreads`	Required	The maximum number of concurrent processing threads. This can be the number of database connections to the WebCenter Sites database server. Use as many threads as the number of CPUs on the database host. For a single CPU database host, set it to 2. Example: `4`
`dataSliceSize`	Required	Number of items retrieved in one read request; this number will also be processed by a single processing thread. Example: `2000`
`dataExtractionImplClass`	Required	User-specific Implementation class for data extraction API. Needs a constructor with (String configFilename) signature. The one mentioned here is a reference implementation class for backward compatibility. Data in flat tables. Default value (ready-to-use): `com.openmarket.gatorbulk.objects.DataExtractImpl`
`initId`	Required	Starting WebCenter Sites ID used the very first time BulkLoader operates; subsequently will use the value from `idSyncFile`. Example: `800000000000`
`idSyncFile`	Required	Next available WebCenter Sites ID is saved in this file; updated during a BulkLoader session. Example: `C:\FutureTense\BulkLoaderId.txt`
`idPoolSize`	Required	Each time BulkLoader needs to generate WebCenter Sites IDs, it collects this many IDs and caches in memory. A good estimate is (number of assets * average number of attributes *2). Example: `1000`
`commitFrequency`	Required	Required. Specifies when "`COMMIT`" statements will be inserted into the generated SQL file. A value of 0 means that "`COMMIT`" statements will be inserted every 50 lines (the default); any positive integer specifies the number of lines between each "`COMMIT`" statement. For example: `commitFrequency=5` (A `COMMIT` statement will be inserted for every 5 lines of SQL code.)
`outputJdbcDriver`	Required	The name of the JDBC driver class to access the WebCenter Sites database. The value here reflects the Oracle 9.0 driver. Example: `oracle.jdbc.driver.OracleDriver`
`outputJdbcURL`	Required	The JDBC URL. The following example value is a typical type2 oracle JDBC driver URL: `Jdbc:oracle:oci8:@foo`
`outputJdbcUsername`	Required	WebCenter Sites database user name.
`outputJdbcPassword`	Required	WebCenter Sites database user password.

33.2.2 IPopulateDataSlice

The following is the sample code that implements this interface:

com.openmarket.gatorbulk.interfaces.IPopulateDataSlice

package com.openmarket.gatorbulk.interfaces;

import java.sql.Timestamp;

/**
 * To be implemented by Oracle Corporation
 * Interface to populate a dataSlice by the client.
 * BulkLoader creates an object implementing this interface and then 
 * hands it over to the client, which uses this interface's methods 
 * to populate that object with input data records.
 */

public interface IPopulateDataSlice {

/**
 * Creates a new input data object to hold all the data for a
 * flex asset and makes it the current object. This method is
 * invoked repetitively to populate this object with flex asset
 * input data. Each invocation is to be followed by Set..()
 * methods and AddAttribute..() methods to supply data for one
 * flex asset.
 */

public void AddNewRow();

/**
 * Specify a unique identifier for flex asset input data
 * @param id  user-specific unique identifier
 * @exception java.lang. Exception thrown if any unique-id
 * validation is enabled.
 */

public voidSetAssetUniqueId(String id);

/**
 * Specify the name of the site with which the current flex
 * asset is created or to be created under.
 * @param sitename  name of the site
 */

public void SetSiteName(String sitename);

/**
 * Set the asset type for the flex asset.
 * @param flexAssetType asset type as defined in WebCenter Sites system
 */

public void SetFlexAssetType(String flexAssetType);

/**
 * Specify the name of the parent for the current flex asset. 
 * Use this method repeatedly to add a list of parent names.
 * @param groupName  name of a parent that the current asset 
 * inherits some of its attributes from.
 */

public void AddParentGroup(String groupName);

/**
 * Specify the name of the parent to be deleted for the current 
 * flex asset.
 * Use this method repeatedly to add a list of parent names.
 * @param groupName - name of a parent that the current asset  
 * inherited some of its attributes from.
 */

public void AddParentGroupForDelete(String groupName);

/**
 * Specify definition asset name for the current flex asset.
 * @param definitionAssetName name of the flex definition asset
 */

public void SetDefinitionAssetName(String definitionAssetName);

/**
 * Specify name of the flex asset.
 * @param name - name of the flex asset.Should be unique in 
 * a flex asset family
 */

public voidSetAssetName(String name);

/**
 * Specify description for the flex asset
 * @param description  description
 */

public void SetAssetDescripiton(String description);

/**
 * Specify WebCenter Sites username with which this flex asset is being
 * processed
 * @param username WebCenter Sites username
 */

public void SetCreatedByUserName(String userName);

/**
 * Set WebCenter Sites status code for this asset
 * @param status
 */

public void SetAssetStatus(String status);

/**
 * Set template name
 * @param template  WebCenter Sites template name
 */

public void SetRenderTemplateName(String template);

/**
 * Specify startMenu for workflow participation
 * @param startMenuName  start menu name for this flex asset
  */

public void SetStartMenuName(String startMenuName);

/**
 * WebCenter Sites
 * Specify publish approval target name
 * @param targetName  approval target name
 */

public void SetApprovalTargetName(String targetName);

/**
 * Add a name/value pair to specify a WebCenter Sites attribute 
 * of type 'text' for the current input object.
 * Call this method more than once, if this is a 
 * multi-valued attribute.
 * @param attrName  attribute name as defined in the WebCenter Sites
 * database for the flex asset being processed
 * @param value java.lang.String
 */

public void AddAttributeValueString(String attrName, String value);

/**
 * Add a name/value pair to specify a WebCenter Sites attribute
 * of type 'date' for the current input object.
 * Call this method more than once, if this is a 
 * multi-valued attribute.
 * @param attrName  attribute name as defined in the WebCenter Sites
 * database for the flex asset being processed
 * @param value  java.sql.Timestamp
 */

public void AddAttributeValueDate(String attrName, Timestamp value);

/**
 * Add a name/value pair to specify an attribute for the current 
 * input object.
 * Call this method more than once, if this is a multi-valued *attribute
 * @param attrName  attribute name as defined in WebCenter Sites database 
 * for the flex asset being processed
 * @param value  java.lang.Double 
 */

public void AddAttributeValueDouble(String attrName, Double value);

/**
 * Add a name/value pair to specify a WebCenter Sites attribute 
 * of type 'money' for the current input object
 * Call this method more than once if this is a 
 * multi-valued attribute
 * @param attrName  attribute name as defined in WebCenter Sites database
 * for the flex asset being processed
 * @param value  java.lang.Float
 */

public void AddAttributeValueFloat(String attrName, Float value);

/**
 * Add a name/value pair to specify a WebCenter Sites attribute
 * of type 'int' for the current input object.
 * Call this method more than once, if this is a 
 * multi-valued attribute.
 * @param attrName  attribute name as defined in WebCenter Sites
 * database for the flex asset being processed
 * @param value  java.lang.Integer 
 */

public void AddAttributeValueInteger(String attrName, 
  Integer value);

/**
 * Add a name/value pair to specify any WebCenter Sites attribute for the
 * current input object.
 * Use the datatype-specific methods above instead of this
 * method, as this one is for
 * supporting any other new types in future.
 * Call this method more than once, if this is a 
 * multi-valued attribute
 * @param attrName  attribute name as defined in the WebCenter Sites
 * database for the flex asset being processed.
 * @param value  java.lang.Object 
 */

public void AddAttributeValueObject(String attrName, 
  Object value);
}

33.2.3 IFeedback Interface

The following is the sample code that implements this interface:

com.openmarket.gatorbulk.interfaces.IFeedback

package com.openmarket.gatorbulk.interfaces;

import java.util.Iterator;

/**
 * To be implemented by Oracle Corporation
 * Interface for the BulkLoader client to get the status of
 * processing request to insert/void/update flex assets.
 */

public interface IFeedback {
  public final int ERROR=-1;
  public final int SUCCESS=0;
  public final int NOT_PROCESSED=1;

/**
 * Get a list of keys from input data slice that has
 * been processed
 * @return java.util.Iterator
 */

public Iterator GetInputDataKeyValList();

/**
 * Get WebCenter Sites asset ID for given input identifier
 * @param inputDataKeyVal key value of the unique identifier 
 * in the input data record
 * @return Get the associated asset ID from the WebCenter Sites system.
 * null if missing.
 */

public String GetWebCenter SitesAssetId(String inputDataKeyVal);

/**
 * Get the processing status for the input data record
 * identified by a key
 * @param inputDataKeyVal key value of the unique identifier
 * column in the input data record
 * @return ERROR - processed but failed, SUCCESS - processed
 * successfully, NOT_PROCESSED - unknown item or not part of 
 * the processing dataset.
 */

public int GetStatus(String inputDataKeyVal);

/**
 * Get the associated error message for a given key, 
 * unique identifier in input data
 * @param inputDataKeyVal  unique identifier for input data
 * @return error message, if GetStatus() returned ERROR 
 * or NOT_PROCESSED
 */

public String GetErrorDescription(String inputDataKeyVal);

Note:

When you implement a custom extraction method, you use the same previously described procedures to run BulkLoader. See About Using the BulkLoader Utility.

33.3 Approving Flex Assets with the BulkApprover Utility

BulkApprover quickly and easily approves large numbers of flex assets that you have loaded into the system via BulkLoader.

BulkApprover can perform the following tasks:

Notify the approval system of all updates and deletions that were made during a previous BulkLoader session.
Approve all newly loaded flex assets for one or more publishing targets.
Mark all newly loaded flex assets as published for a given mirror destination, without actually publishing the assets. For example, to bypass a long mirror publishing session, copy selected assets from the content management database to a mirror destination on the delivery system and have BulkApprover mark the assets as published to the mirror destination.

Note:

Only users with the xceladmin role can run BulkApprover.

This section includes the following topics:

33.3.1 Configuring BulkApprover

Before running BulkApprover for the first time, you must create a configuration file for the utility. You can create a separate BulkApprover.ini file on a system that can access WebCenter Sites database, or you can append the BulkApprover configuration information to one of BulkLoader's .ini files.

This table lists required and optional configuration parameters:

Table 33-3 BulkApprover Configuration Parameters

Parameter Description

Parameter	Description
`bulkApprovalURL` (Required)	The URL on the host server that has the data imported with BulkLoader. The correct value is as follows: `http://<myServer>/cs/ContentServer?pagename=OpenMarket/Xcelerate/Actions/BulkApproval` where `<myServer>` is the name of the host server.
`adminUserName` (Required)	The WebCenter Sites username of a user with the `xceladmin` role.
`adminUserPassword` (Required)	The password for `adminUserName`.
`approvalTargetList` (Required)	A list of destinations that the assets are to be approved for. For destination names, see the Publish option on the Admin tab, or the name column of the `pubtarget` table. Separate each destination with the delimiter that you specify in the `multiValueDelimiter` parameter. The syntax is: `name1<multiValueDelimiter>name2<multiValueDelimiter>name3`
`multiValueDelimiter (Required)`	A delimiter that you select. Use this delimiter to separate the approval targets that you specify in the `appovalTargetList` parameter.
`assetIdSqlFilter` (Optional)	A statement that can be appended to a SQL `WHERE` clause to filter asset IDs. For example: `asset_id%20=0` or `asset_id%20!=0`
`debug` (Optional)	Turn BulkApprover debugging on and off. A value of `true` turns debugging on. Leave this parameter blank for no debugging. Debug messages are written to the file specified in the `output_file` parameter of the command line.
`assetschunksize` (Optional)	Specifies the number of assets that are approved in a single transaction. For example, setting this property to `20` means that assets will be approved in groups of `20`. Setting this property helps prevent session timeouts. Default value: `25`
`outputJdbcDriver` (Required)	The name of the JDBC driver class to access the WebCenter Sites database. Example: `oracle.jdbc.driver.OracleDriver`
`outputJdbcURL` (Required)	The JDBC URL. The following example value is a typical type 2 oracle JDBC driver URL: `Jdbc:oracle:oci8:@foo`
`outputJdbcUsername` (Required)	WebCenter Sites database user name.
`outputJdbcPassword` (Required)	WebCenter Sites database user password.

bulkApprovalURL

(Required)

The URL on the host server that has the data imported with BulkLoader.

The correct value is as follows:

http://<myServer>/cs/ContentServer?pagename=OpenMarket/Xcelerate/Actions/BulkApproval

where <myServer> is the name of the host server.

adminUserName

(Required)

The WebCenter Sites username of a user with the xceladmin role.

adminUserPassword

(Required)

The password for adminUserName.

approvalTargetList

(Required)

A list of destinations that the assets are to be approved for. For destination names, see the Publish option on the Admin tab, or the name column of the pubtarget table. Separate each destination with the delimiter that you specify in the multiValueDelimiter parameter. The syntax is:

name1<multiValueDelimiter>name2<multiValueDelimiter>name3

multiValueDelimiter (Required)

A delimiter that you select. Use this delimiter to separate the approval targets that you specify in the appovalTargetList parameter.

assetIdSqlFilter

(Optional)

A statement that can be appended to a SQL WHERE clause to filter asset IDs.

For example:

asset_id%20=0

asset_id%20!=0

debug

(Optional)

Turn BulkApprover debugging on and off.

A value of true turns debugging on. Leave this parameter blank for no debugging.

Debug messages are written to the file specified in the output_file parameter of the command line.

assetschunksize

(Optional)

Specifies the number of assets that are approved in a single transaction. For example, setting this property to 20 means that assets will be approved in groups of 20.

Setting this property helps prevent session timeouts.

Default value: 25

outputJdbcDriver

(Required)

The name of the JDBC driver class to access the WebCenter Sites database.

Example: oracle.jdbc.driver.OracleDriver

outputJdbcURL

(Required)

The JDBC URL. The following example value is a typical type 2 oracle JDBC driver URL:

Jdbc:oracle:oci8:@foo

outputJdbcUsername

(Required)

WebCenter Sites database user name.

outputJdbcPassword

(Required)

WebCenter Sites database user password.

Sample BulkApprover.ini File

The following sample shows the proper syntax of the BulkApprover configuration parameters:

bulkApprovalURL=http://MyServer/cs/ContentServer?pagename=OpenMarket/Xcelerate/Actions/BulkApproval
multiValueDelimiter=;;;;;;
assetIdSqlFilter=
assetsChunkSize=3
debug=true
outputJdbcDriver=oracle.jdbc.driver.OracleDriver
outputJdbcURL=jdbc:oracle:thin:@19zln:1521:MyServer
#outputJdbcUsername=izod10
outputJdbcUsername=ftuser3
outputJdbcPassword=ftuser3

33.3.2 Using BulkApprover

After you have configured the BulkApprover utility, use it to approve assets that were imported into the database via the BulkLoader utility.

BulkApprover runs from the command line. Parameters are described in the following table and included in the example that follows.

Table 33-4 BulkApprover Parameters

Command-Line Parameter Description

Command-Line Parameter	Description
`config`	The name of the file that stores your BulkApprover configuration information; for example, `BulkApprover.ini`.
`action`	The action or actions that you want BulkApprover to perform. You must set this parameter to `notify` or `approve`. Add other actions, as necessary. To have BulkApprover perform multiple actions, name the actions in a comma-separated list. Valid values: `notify`: Notifies the approval system about all updates and voids processed during a previous BulkLoader session. `approve`: Instructs BulkApprover to approve all of the assets that it processes for the given publishing destination(s). `mark_publish`: Marks all of the assets that it processes as published to a given mirror destination, without actually publishing the assets. Specify the publishing targets in the `approvalTargetList` parameter in the BulkApprover configuration file. Do not include this parameter if the assets should not be marked published.
`output_file`	The name of the log file that contains all output from the server; for example, `bulkapprover.txt`.

config

The name of the file that stores your BulkApprover configuration information; for example, BulkApprover.ini.

action

The action or actions that you want BulkApprover to perform. You must set this parameter to notify or approve. Add other actions, as necessary. To have BulkApprover perform multiple actions, name the actions in a comma-separated list.

Valid values:

notify: Notifies the approval system about all updates and voids processed during a previous BulkLoader session.
approve: Instructs BulkApprover to approve all of the assets that it processes for the given publishing destination(s).
mark_publish: Marks all of the assets that it processes as published to a given mirror destination, without actually publishing the assets. Specify the publishing targets in the approvalTargetList parameter in the BulkApprover configuration file. Do not include this parameter if the assets should not be marked published.

output_file

The name of the log file that contains all output from the server; for example, bulkapprover.txt.

To run BulkApprover, set paths as shown in the following example.

This example uses Windows syntax. For UNIX-based systems including Linux and Solaris, the forward-slash (/) and colon (:) should be used as separators:

java -ms16m -mx64m -cp <ORACLE_HOME>\wcsites\webcentersites\sites-home\lib\*;<ORACLE_HOME>\oracle_common\modules\clients\*;<path to jtds-1.2.2.jar>\jtds-1.2.2.jar;<path to servlet-api.jar>\servlet-api.jar; com.openmarket.gatorbulk.objects.BulkApprover config=bulkapprover.ini action=<notify|approve|mark_publish> output_file=<out.txt>  target_list=Dynamic;;;;;;testdest username=<adminUserName(fwadmin)> password=<adminUserPassword(xceladmin)>