9Running EIM
Running EIM
This chapter covers how to run an EIM process and check the results. This chapter is organized into the following sections:
Preparing to Run an EIM Process
You can run an EIM process (import, export, delete, or merge) once you have:
Identified the data for EIM processing
Prepared the related EIM tables
Modified the EIM configuration file accordingly
You can start an EIM process by running a server task for the Enterprise Integration Manager component. You can run the server task using either the GUI or the command-line interface. For more information on running server tasks, see Siebel System Administration Guide.
Running an EIM Process
On each pass, EIM processes one EIM table and performs a particular action on all rows in that table for that batch. Most passes affect only the EIM table’s temporary columns; for example, resolving foreign keys. There are two methods for running an EIM process:
Running an EIM Process Using the Graphical User Interface
The most common method for starting an EIM server task is to use the graphical user interface (GUI). When performing this procedure, be aware that passes in Step 8 in EIM Import Process (update), Step 9 in EIM Import Process (insert), and Step 10 in EIM Import Process (primary keys) affect the base tables. All steps are performed for all columns used in the import process.
updatestats.sql script (located in dbserver_home\db2) each time before running EIM, or performance issues may be encountered when loading the dictionary. For more information, see
Siebel Performance Tuning Guide.
To run an EIM process using the GUI
Navigate to the Administration-Server Management screen, and then Jobs view.
In the Jobs list, click New.
The component job status field changes to Creating.
In the Component/Job field, click the Select button.
The Component/Jobs pick applet appears.
From the Find drop-down list, select Name, and perform the following query: Enterprise Integration Mgr, then click OK.
If you want to use a component job template based on EIM for your component request, you must first define the component job template. For information on defining component job templates, see Siebel System Administration Guide.
In the Job Detail view, enter data in other appropriate fields as described in the table that follows.
Field Description Scheduled Start
The scheduled start date and time of the component job.
Expiration
The date at which the component job is no longer valid.
Requested Server
Set if you want to target a server component on a specific Siebel Server.
Request Key
Set if you want to target a component or repeating component job to a specific instance of the server component identified by the request key. In all other situations, keep this field blank.
Delete Interval
Set with Delete Unit field, this field determines the length of time before the component job is deleted. If not updated, this field defaults to 1.
Delete Unit
Set with Delete Interval field, this field determines the length of time before the component job is deleted. If not updated, this field defaults to Weeks.
Retry on Error
Check this box to retry the component job in case of error.
Sleep Time
This field is available when the Retry on Error check box is true and determines the amount of time before the component job is retried.
Number of Retries
This field is available when the Retry on Error check box is true and determines the number of times the component job is retried.
Click the menu button, and then click Save Record.
In the Job Parameters list, add or change any component job parameters for the EIM process:
Click the New button.
In the Name field, click the Select button.
The Job Parameters dialog box appears. The parameters that appear in the Job Parameters dialog box vary depending on the server component you selected in Step 3.
Select a parameter in the Component Parameters dialog box, and modify its value.
- Click the menu button and then click Save Record.
In the Jobs list, click the Submit Job button.
The Status field changes from Creating to Queued.
Caution: EIM is a multistep process. Once the EIM process is running, do not stop or pause the task. Otherwise, some steps may not roll back correctly.
Running an EIM Process Using the Command-Line Interface
You can also start the EIM server task through the command-line interface. For example, if you are using a UNIX operating system or if you have experienced the EIM server task being QUEUED when the job was submitted by the GUI, use the command-line interface to run an EIM process.
To run an EIM process using the command-line interface
Start the srvrmgr program in the command-line interface.
For information on srvrmgr program, see Siebel System Administration Guide.
Execute a start task command or a run task command on the Enterprise Integration Mgr component. Be sure to specify the configuration file with the config parameter.
Note: You cannot use the Uniform Naming Convention (UNC) in the Server Manager command-line interface when specifying the configuration file.If you do not specify a configuration file, the default.ifb configuration file will be used. If you put the .IFB file you want to use in a directory other than the default directory (<SiebSrvr\Admin> folder), you will need to specify the path to the .IFB file when you start the EIM component.
The following example shows how to use the run task command to start an import process:
run task for component eim with config=import.ifb
For more information on the start task command and the run task command, see Siebel System Administration Guide.
Caution: EIM is a multistep process. When the EIM process is running, do not interrupt the task. Otherwise, some steps may not roll back correctly.The following example shows how to use the run task command to start an import process that uses a different LOV language than the default setting of the EIM LOV language parameter:
run task for component eim with config=import.ifb, LovLang=ESN
Viewing the EIM Log File
In the Task Info Log view, you can view information about the results of an EIM process by drilling down in an EIM server task that has completed. This information is also provided in the EIM log file within the siebel server\log directory. The log consists of three general sections:
Startup messages. This section pertains to dictionary loading, parameter loading, and .IFB file parsing.
Run-time messages. This section shows the begin and end times for each process.
Row-count summary of each process. This section shows the number of rows updated in each table.
If error flags, SQL trace flags, or trace flags were activated for the EIM process, the EIM log file will also contain the results of each flag. For more information on trace flags and error flags, see Using Trace Flags, SQL Trace Flags, and Error Flags.
Further information on the EIM log file is provided as follows:
To view EIM log file information in the Task Info Log
Navigate to Administration - Server Management screen, then the Tasks view.
In the Tasks list, select the task for the EIM process.
Click the Log tab.
The log for the selected task is displayed in the Log list.
siebel server\log directory.
Using Trace Flags, SQL Trace Flags, and Error Flags
You can activate trace flags and error flags to log transactions. This topic covers the following types of flags:
Error flags. See Error Flags.
SQL Trace flags. See SQL Trace Flags.
Trace flags. See Trace Flags.
Recommended settings for error flags, SQL trace flags, and trace flags include the following:
To display errors and unused foreign keys. Start with the following setting combination. The setting Trace Flag=1 provides a summary (after each batch) of the elapsed time in EIM steps 10 and 11.
Setting |
Value |
|---|---|
Error Flag |
1 |
SQL Trace Flag |
1 |
Trace Flag |
1 |
To determine SQL performance. The following setting combination produces a log file with SQL statements including the elapsed time for each statement.
Setting |
Value |
|---|---|
Error Flag |
1 |
SQL Trace Flag |
8 |
Trace Flag |
3 |
To determine optimal batch size and monitor performance in a particular step. The following setting combination produces a log file showing the elapsed time for each EIM step.
Setting |
Value |
|---|---|
Error Flag |
0 |
SQL Trace Flag |
0 |
Trace Flag |
1 |
Error Flags
To activate error flags, you must complete Step 7 in Running an EIM Process Using the Graphical User Interface when running an EIM process. Setting the Error Flags parameter to 1 produces a detailed explanation of rows that were not successfully processed.
There are a variety of reasons why rows might not be processed. The following sample shows an excerpt from an EIM Error Flag 1 trace. The log begins with a header that describes an export failure that occurred during Step 2, Pass 101.
2001-04-04 03:47:59 4/4/01 3:47 Warning: No rows in S_ORG_EXT matched by expressions for export. 2001-04-04 03:47:59 Process [Export Old Accounts] had all rows fail 2001-04-04 03:47:59 on EIM_ACCOUNT for ] 2001 in step 2, pass 101: 2001-04-04 03:47:59 No base table rows matched expressions. (severity 5) 2001-04-04 03:47:59 Base table: 2001-04-04 03:47:59 S_ORG_EXT (Account) 2001-04-04 03:47:59 The match expressions specified for exporting rows through this interface table 2001-04-04 03:47:59 did not match any of the rows currently in the target base table. 2001-04-04 03:47:59 Since there were no matches for the given match expressions, processing for 2001-04-04 03:47:59 this interface table was discontinued. However, processing of other interface 2001-04-04 03:47:59 tables will continue. 2001-04-04 03:47:59 Recorded 1 group of failures.
SQL Trace Flags
To activate SQL trace flags, you must complete Step 7 in Running an EIM Process Using the Graphical User Interface when running an EIM process.
Setting the SQL Trace Flags parameter to 8 creates a log of all SQL statements that make up the EIM task. The lower values for SQL Debug Flags (1, 2, and 4) are used for logging at the ODBC level.
Trace Flags
Trace flags contain logs of various EIM operations. To activate trace flags, you must complete Step 7 in Running an EIM Process Using the Graphical User Interface when running an EIM process. Also set event logging for the EIM component, as described in Setting Event Logging from the Graphical User Interface.
Trace flags are bit-based. Available trace flags include 1, 2, 4, 8, and 32. To activate multiple trace flags, set the Trace Flags parameter to the sum of individual trace flag numbers. For example, to log trace flags 2 and 4, set the Trace Flags parameter to 6.
Setting Event Logging from the Graphical User Interface
You can set event logging for the EIM component using the Administration - Server Configuration views in the Siebel client.
To set event logging for the EIM component from the GUI
Navigate to the Administration - Server Configuration screen, Servers, Components, and then the Events view.
In the Components list, select Enterprise Integration Manager as the component.
Click the Events tab to view all the configurable event types for the selected component.
The log level is set to a default value of 1.
Perform a query and enter the specified log level for each of the following event types:
Event Type Log Level Value EIM SQL
4
SQL Summary
4
Task Configuration
4
EIM Trace
3
It is not necessary to restart the Siebel Server to apply the event type log level changes. The changed settings are active in the next EIM task executed.
For more information on event logging administration, see Siebel System Monitoring and Diagnostics Guide.
Setting Event Logging from the Command-Line Interface
You can also set event logging for the EIM component from the Server Manager command line.
To set event logging for the EIM component from the command line
Use the following commands:
change evtloglvl SQLSummary=4 for component eim change evtloglvl EIMSQL=4 for component eim change evtloglvl TaskConfig=4 for component eim change evtloglvl EIMTrace=3 for component eim
Other necessary commands for activating tracing levels are the following:
When running the EIM task. Specify the following parameters:
Srvrmgr> run task for component eim with config=<configfile.ifb>, TraceFLags=1, ErrorFlags=1, SQLFlags=8
To view existing event log levels for the EIM component. Use the following command:
Srvrmgr> list evtloglvl for component eim
Trace Flag Settings
This topic provides Trace Flag setting information as follows:
Trace Flag 1
Setting the Trace Flags parameter to 1 creates a step-oriented log of the task. This can be used to determine the amount of time EIM spends on each step of the EIM task, or for each EIM table processed. The following sample shows an EIM Trace Flag 1 output:
Initializing
Loading configuration file imacct.ifb 0s
Opening server database ora_dev 6s
Loading Siebel dictionary 15s
Initializing 21s
Import Accounts 14
Importing EIM_ACCOUNT
Step 1: initializing IF Table 0s
Step 4: resolving foreign keys S_ORG_EXT 0s
Step 5: locating existing row S_ORG_EXT 0s
Step 7: finding new foreign keys 4s
Step 9: inserting new rows S_ORG_EXT 2s
Importing EIM_ACCOUNT 15s
Updating primaries
Step 10: updating primary keys S_ORG_EXT 3s
Updating primaries 3s
Import Accounts 14 18s
Trace Flag 2
Setting the Trace Flags parameter to 2 creates a file log that traces all substitutions of user parameters. The following example shows an EIM Trace Flag 2 output:
[TRC01] Parameter Set << AFTER RESOLUTION >> [TRC01] UserParams = IFTABLE=EIM_ACCOUNT [TRC01] [0] $IFTABLE = EIM_ACCOUNT [TRC01] [1] $CURRENT_USER = wgong [TRC01] [2] $CURRENT_DATETIME = 4/6/01 13:17 [TRC01] [Siebel Integration Manager] [TRC01] log transactions = false [TRC01] $COLUMN_VALUE = 'EIM ins_acct Test%' [TRC01] [ins_acct_shell] [TRC01] TYPE = SHELL [TRC01] INCLUDE = del_acct [TRC01] INCLUDE = ins_acct [TRC01] [del_acct] [TRC01] SESSIONSQL = DELETE FROM DEV50.EIM_ACCOUNT WHERE IF_ROW_BATCH_NUM=21 [TRC01] TYPE = DELETE [TRC01] BATCH = 20 [TRC01] TABLE = EIM_ACCOUNT [TRC01] $COLUMN_NAME = NAME [TRC01] DELETE MATCHES = EIM_ACCOUNT,(NAME LIKE 'EIM ins_acct Test%') [TRC01] [ins_acct] [TRC01] SESSIONSQL = INSERT INTO DEV50.EIM_ACCOUNT (IF_ROW_STAT, ROW_ID, IF_ROW_BATCH_NUM, ACCNT_NAME, ACCNT_LOC) SELECT 'X', ROW_ID, 21, 'EIM ins_acct Test ' || ROW_ID, 'Loc' FROM DEV50.S_SYS_PREF [TRC01] TYPE = IMPORT [TRC01] BATCH = 21 [TRC01] TABLE = EIM_ACCOUNT
Trace Flag 4
Setting the Trace Flags parameter to 4 creates a file log that traces all user-key overrides. The following example shows an EIM Flag 4 output for a user key override to the EIM_ACCOUNT table:
[TRC02] ------------------------------------------------------ [TRC02] ***** IF TABLE <EIM_ACCOUNT> uses USER_KEY_COL ***** [TRC02] Action: No Move & Insert [TRC02] overriding UK Index (S_TERR_ITEM_U1) at position (0) [TRC02] ##### Destination TABLE (S_TERR_ITEM) index vector: [S_TERR_ITEM_U1] [TRC02] --- Column (T_TERITE_OUID) index vector: [S_TERR_ITEM_U1] [TRC02] --- Column (T_TERITE_TERID) index vector: [S_TERR_ITEM_U1] [TRC02] ------------------------------------------------------
Trace Flag 8
Setting the Trace Flags parameter to 8 creates a file log that traces all Interface Mapping warnings. The following example shows an EIM Flag 8 output for an Interface Mapping warning between the EIM_ACCOUNT and S_TERR_ITEM tables:
[TRC03] -------------------------------------------------------- [TRC03] IF table EIM_ACCOUNT destination S_TERR_ITEM [TRC03] IF column EIM_ACCOUNT.T_TERITE_TERID: [TRC03] imports to: S_TERR_ITEM.TERR_ID [TRC03] exports from: S_TERR_ITEM.TERR_ID [TRC03] Column NAME of join isn't in table! [TRC03] Missing join to user key NAME [TRC03] --------------------------------------------------------
Trace Flag 32
Setting the Trace Flags parameter to 32 creates a file log that traces all file attachment status. The trace file contains four labels, three of which are used to trace file attachment processes as described in the following table.
| Label | Description |
|---|---|
Attachment Imported |
Indicates whether the file attachment was encoded, compressed, and copied to the Siebel file server with the new name. |
Attachment (Old) Deleted |
This label applies only to updates and indicates whether an existing file was replaced and deleted. |
Attachment Not Found |
Indicates that the file attachment cannot be found in the input directory. |
The following sample shows an EIM Flag 32 output for an opportunity file attachment:
[TRC32] Attachment Imported: E:\V50\output\openpost.doc -> \\BALTO\SIEBFILE\ORADEV50\S_OPTY_ATT_10+413+1_10-41R-0.saf [TRC32] Attachment (Old) Deleted: \\BALTO\SIEBFILE\ORADEV50\S_OPTY_ATT_10+413+1_10- 40Y-0.saf [TRC32] Attachment Not Found: E:\V50\output\openpost.doc [TRC32] Attachment Identical: E:\V50\output\openpost.doc IDENTICAL TO \\BALTO\SIEBFILE\ORADEV50\S_OPTY_ATT_10+413+1_10-41R-0.saf
Optimizing EIM Performance
There are several ways you can improve EIM run-time performance. The best practices suggested in this section optimize EIM performance. For additional information on improving the performance of EIM, see Siebel Performance Tuning Guide.
Table Optimization for EIM
This section discusses ways that you can optimize tables for EIM processing.
Configuration Parameters
Limit base tables and columns to be processed. Four EIM parameters can help improve performance by limiting the affected tables and columns:
ONLY BASE TABLES
IGNORE BASE TABLES
ONLY BASE COLUMNS
IGNORE BASE COLUMNS
The ONLY BASE COLUMNS parameter is critical for the performance of an EIM process updating a few columns in many rows.
For other suggestions involving parameter settings, see Parameter Settings Optimization for EIM.
Indexes
Verify that all indexes exist for the tables involved. In most implementations, the tables and corresponding indexes in the following list tend to be the most heavily used and should be separated across devices. In general, the following indexes should be on different physical devices from the tables on which they are created.
S_ACCNT_POSTN
S_OPTY
S_ADDR_ORG
S_OPTY_POSTN
S_CONTACT
S_POSTN_CON
S_DOCK_TXN_LOG
S_PARTY_RPT_RE
S_SRV_REQ
S_EVT_ACT
S_OPTY
S_ORG_EXT
For organizations that plan to use EIM extensively, you should put your key EIM tables (based on your unique business requirements) on different devices from the Siebel base tables, because all tables are accessed simultaneously during EIM operations.
You can speed up deletes and merges involving S_ORG_EXT by adding an index to one or more columns. For more information, see Siebel Performance Tuning Guide.
Maintenance of EIM Tables
Perform regular table maintenance on EIM tables. Frequent insert or delete operations on EIM tables can cause fragmentation in the table. Ask your database administrator to detect and correct fragmentation in the EIM tables.
Always delete batches from EIM tables upon completion. Leaving old batches in the EIM table wastes space and can adversely affect performance. For other suggestions on working with batches, see Limiting the Number of Records and Rows for Merge Processes.
Batch Processing Optimization for EIM
This section suggests ways in which you can optimize EIM batch processing. Try using different batch sizes. Large batch sizes are often not efficient. For import and delete processes that use the DELETE EXACT parameter, use approximately 20,000 rows in a single batch.
Limiting the Number of Records and Rows for Merge Processes
You can improve performance by limiting the number of records in a batch. For information, see Siebel Performance Tuning Guide.
Using Batch Ranges
Try using batch ranges (BATCH = x–y). This allows you to run with smaller batch sizes and avoid the startup overhead on each batch. The maximum number of batches that you can run in an EIM process is 1,000.
For IBM DB2, load a few batches of data into the EIM table and run EIM for just one of these batches. This primes the statistics in the DB2 catalogs. Afterward, do not update statistics on the EIM tables, and run EIM with the parameter UPDATE STATISTICS = FALSE in the .IFB file. This helps achieve consistent performance results when running EIM. See Parameter Settings Optimization for EIM for other suggestions about parameters.
Run-Time Optimization for EIM
This section describes the ways you can optimize EIM performance at run time.
Parallel Processing
Run independent EIM jobs in parallel. Two or more EIM processes can be started simultaneously by using the Siebel Server Manager.
A special setup is not required to run EIM processes in parallel. For parallel processing, the following conditions must be met:
No duplicate unique keys between runs for inserts.
No duplicate updates or deletes between runs.
No lock escalations on either EIM tables or target tables can be tolerated. Set LOCKLIST and MAXLOCKS as high as necessary to prevent this.
Note: If you run EIM jobs in parallel on the same base tables, you might encounter unique constraint errors if you have the same values for the unique index fields in batches being processed by two different EIM jobs.
For more information on parallel processing, see Siebel Performance Tuning Guide.
Transaction Logging
Consider disabling the Enable Transaction Logging system preference in the Administration - Siebel Remote screen during the EIM run. Switching off transaction logging improves performance; however, this benefit must be balanced with the need for mobile users to reextract afterward. To disable transaction logging, complete Step 2 in Importing an Initial Batch of Legacy Data.
Parameter Settings Optimization for EIM
This section discusses ways that you can optimize EIM performance through parameter settings.
USING SYNONYMS Parameter for Optimizing EIM
Ignore account synonyms. Set the USING SYNONYMS parameter to FALSE in the .IFB file to indicate that account synonyms can be ignored during processing. This logical operator indicates to EIM that account synonyms do not require processing during import, thus reducing the amount of processing. Do not set the USING SYNONYMS parameter to FALSE if you plan to use multiple addresses for accounts. Otherwise, EIM will not attach addresses to the appropriate accounts. You can use EIM_ACCOUNT to import accounts with multiple addresses and then specify the primary address for an account by setting ACC_PR_ADDR to Y.
Trace Flag Settings for Optimizing EIM
Generate a task log to identify slow-running steps and queries by using Trace Flags. To use Trace Flags, set Error Flags=1, Trace Flags=1, and SQL Trace Flags=8. Rerun the batch and use the resulting task log to determine which steps and queries are running especially slowly. For additional information on trace flag settings, see Trace Flag Settings.
Database Server Optimization for EIM
The overall performance of EIM is largely dependent on the overall performance of the database server. To achieve optimal database server performance, it is critical that the tables and indexes in the database be arranged across available disk devices in a manner that evenly distributes the processing load.
The mechanism for distributing database objects varies by RDBMS, depending on the manner in which storage space is allocated. Most databases have the ability to assign a given object to be created on a specific disk.
A redundant array of independent disks (or RAID) can provide large amounts of I/O throughput and capacity, while appearing to the operating system and RDBMS as a single large disk (or multiple disks, as desired, for manageability).
The use of RAID can greatly simplify the database layout process by providing an abstraction layer over the physical disks while achieving high performance. Regardless of the RDBMS you implement and your chosen disk arrangement, be sure that you properly distribute the following types of database objects:
Database log or archive files.
Temporary workspace used by the database.
By following these suggestions, you should be able to improve the performance of the database server.