Skip Headers
Oracle® Fusion Middleware Administrator's Guide for Oracle Internet Directory
11g Release 1 (11.1.1)

Part Number E10029-06
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Master Index
Master Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

15 Performing Bulk Operations

This chapter consists of the following sections:

15.1 Introduction to Performing Bulk Operations

For processing large quantities of data, bulk operations are typically more efficient than standard LDAP operations You can perform bulk operations only by using the command-line bulk tools.

Before you begin using the Oracle Identity Management command-line tools, you must configure your environment. This involves setting the appropriate environment variables.

The syntax and examples provided in this guide require that you have the following environment variables set:

Notes:

  • The bulk tools do not support attribute uniqueness.

  • If your schemas were created during installation of a version prior to 11g Release 1 (11.1.1.6.0), you must add datafiles to the OLTS_CT_STORE and OLTS_ATTRSTORE tablespaces if you intend to add more than a million entries to Oracle Internet Directory. Perform this step prior to the bulkload or ldapadd operation. For details, see the section "Creating Datafiles and Adding Datafiles to a Tablespace" in Oracle Database Administrator's Guide.

In addition to this chapter, bulk tools are also discussed in the following sections of this book:

See Also:

The chapter on Oracle Internet Directory administration tools in Oracle Fusion Middleware Reference for Oracle Identity Management.

Notes:

  • Stop all instances of Oracle Internet Directory before using bulkload. Before using any of the other bulk tools, either stop all instances of Oracle Internet Directory or disable the entry cache.

  • Do not start Oracle Internet Directory while a bulk tool is executing.

Tip:

The syntax descriptions in this chapter show true or false arguments in the following format:

check="TRUE" 

You can type true or false in uppercase or lowercase, you can specify just the first letter, and you can omit the double quotes. That is, the following specifications are equivalent:

check=t
check=true
check="true"
check=T
check=TRUE
check="TRUE" 

15.2 Changing Server Mode

Some bulk tool operations and replication setup procedures require you to switch an Oracle Internet Directory instance from read/write to read-only mode or from read-only to read/write mode. You can do so by using either Oracle Enterprise Manager Fusion Middleware Control or ldapmodify.

While the server is in read-only mode, only the administrator cn=orcladmin can write to the directory. As a result, you can only make changes to the directory by using utilities that enable you to connect as cn=orcladmin. These include:

You cannot make changes while the server is in read-only mode by using WLST or Oracle Enterprise Manager Fusion Middleware Control because they connect to the Oracle Internet Directory server as the user cn=emd admin, cn=oracle internet directory.

15.2.1 Setting the Server Mode by Using Fusion Middleware Control

To set the server mode to read-only from Fusion Middleware Control:

  1. Select Administration, then Server Properties from the Oracle Internet Directory menu.

  2. Select General.

  3. Choose Read-only for Server Mode.

  4. Click Apply.

To set the server mode to read/write mode from Fusion Middleware Control, use the same procedure but choose Read/Write in Step 3.

15.2.2 Setting the Server Mode by Using ldapmodify

To set the server mode to Read-Only by using ldapmodify, run the following command:

ldapmodify -D "cn=orcladmin" -q -h host_name \
            -p port -f change_mode.ldif

where the file change_mode.ldif has the following content:

dn: cn=componentname,cn=osdldapd,cn=subconfigsubentry
changetype: modify
replace: orclservermode
orclservermode: r

To set the server mode to Read/Write by using ldapmodify, use the same command but change the last line in the LDIF file to:

orclservermode: rw

15.3 Loading Data Into the Schema by Using bulkload

The bulk loader, bulkload, is a bulk management tool. It takes input data in LDIF or SQL*Loader format and loads the data directly into Oracle Internet Directory's schema in the metadata repository. It has three main phases: check, generate and load.

In the check phase, bulkload parses and verifies the LDIF input data for the schema.

In the generate phase, bulkload generates intermediate files in SQL*Loader format.

In the load phase, bulkload can use either of two methods: bulk mode loading or incremental mode loading.

Bulk mode loading is faster than incremental mode loading.

The bulk loader also supports the following features:

At the beginning of the generate phase, the server's orclServerMode, in the instance-specific configuration entry, is changed from read/write to read-modify. At the end of the generate phase, it is left in the read-modify state so that you cannot add entries to Oracle Internet Directory between the generate and load phases. This is necessary to maintain internal sequence numbers. You are expeced to run the load phase immediately after the generate phase. At the end of the load phase, the servers' orclServerMode is set back to read/write. Using bulkload with the recover option also sets orclServerMode back to read/write.

At the start of the load operation, bulkload determines the current configured value of orclRIenabled, then disables referential integrity. At the end of load phase, bulkload returns orclRIenabled to its original value. If is any referential integrity violations occurred, however, referential integrity is disabled, and you see the message:

There is a violation of Referential Integrity and hence it is Disabled now. Run the OIDDIAG tool with diagnostic option to collect the Entries which have dangling DN attribute values and Fix the violation

Fix the violation and then set orclRIenabled to the desired value.

The bulkload tool generates the following output files in the ORACLE_INSTANCE/diagnostics/logs/OID/tools directory:

The bulkload tool generates the following output files in the ORACLE_INSTANCE/OID/load directory:

Notes:

  • If a directory server instance is participating in a replication agreement, do not use the bulkload tool to add data into the node. Instead, use ldapadd.

  • Before using bulkload, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.

  • Running the bulkload load operation sets the server mode to read/write. If you require a different mode, reset it after performing the load operation.

  • If the applicable password policy has the pwdmustchange attribute set to 1, then for every new entry loaded by bulkload, the pwdreset attribute is set to 1 by default. See Chapter 29, "Managing Password Policies" for more information.

  • If you do not use the bulkload utility to populate the directory, then you must run the oidstats.sql tool to avoid significant search performance degradation.

See Also:

The oidstats.sql command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity Management for a description and syntax for the oidstats.sql tool

The bulkload tool has the following syntax:

bulkload [connect=connect_string]
{[check="TRUE"|"FALSE" [file=ldif_file]] [generate="TRUE"|"FALSE"
[append="TRUE"|"FALSE"] [restore="TRUE"|"FALSE"] [thread=num_of_threads]
file=ldif_file] [load="TRUE"|"FALSE" [append="TRUE"|"FALSE"] 
[threads=num_of_threads]] [index="TRUE"|"FALSE"] [missing="TRUE"|"FALSE"]
[recover="TRUE"|"FALSE"]} [encode=character_set] [debug="TRUE"|"FALSE"]
[verbose="TRUE"|"FALSE"]

Some of the parameter combinations are valid while others are invalid.

You must set the environment variable ORACLE_INSTANCE. Specify the fully qualified path to the Oracle Instance where the intermediate file is generated.

You must specify at least one of the following actions when you invoke bulkload: check, generate, load, append, recover, or index.

If check is TRUE, bulkload performs a schema check.

If generate is TRUE, bulkload generates intermediate files.

When using the check or generate action, you must specify the path name to the LDIF data file.

If load is TRUE, bulkload loads intermediate files.

When append is TRUE, bulkload can perform its actions while the server is up and running.

Use the restore flag only when the LDIF file contains operational attributes, such as orclguid or creatorsname. Avoid having operational attributes in the LDIF file when the restore flag is not specified or is set to FALSE.

Do not specify recover with any other option.

The option combination check index verifies the existing indexes.

15.3.1 Importing an LDIF File by Using bulkload

To import an LDIF file, you use the bulkload utility. This section discusses the tasks to process an LDIF file through bulkload.

See Also:

The bulkload command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity Management.

This section contains these topics:

Task 1: Stopping Oracle Internet Directory Processes

Stop all Oracle Internet Directory server instances by using either Fusion Middleware Control or the command line.

Task 2: Backing Up the Oracle Database Server

Before you import the file, back up the Oracle database server as a safety precaution.

See Also:

Oracle Database Backup and Recovery User's Guide in the Oracle Database Documentation Library

Task 3: Finding Out the Oracle Internet Directory Password

To use bulkload, you must provide the Oracle Internet Directory ODS schema password.

See Also:

The oidpasswd command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity Management

Task 4: Checking Input for Schema and Data Consistency Violations and Generating the Input Files for SQL*Loader

On UNIX, the bulkload tool usually resides in $ORACLE_HOME/ldap/bin. On Microsoft Windows, this tool usually resides in ORACLE_HOME\ldap\bin.

Check the input file and generate files for the SQL*Loader by typing:

bulkload connect="connect_string" \ 
   check="TRUE" generate="TRUE" file="full_path_to_ldif-file_name"

When you specify both the check and generate options, the entries are checked for schema compliance.

All check-related errors are reported as command line output. All schema violations are reported in ORACLE_INSTANCE/diagnostics/logs/OID/tools/bulkload.log. All bad entries are logged in ORACLE_INSTANCE/OID/load/badentry.ldif.

If there are duplicate entries, their DNs are logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/duplicateDN.log. This is just for information purpose. The bulkload tool does not generate duplicate data for duplicate entries. It ignores duplicate entries.

Use a text editor to fix all bad entries, then re-run bulkload with the check and generate options. Repeat until there are no errors, or until the remaining errors are acceptable to you. For example, you might be willing to load a small number of entries with ldapadd.

The bulkload tool generates intermediate *.ctl and *.dat files in the ORACLE_INSTANCE/OID/load directory. Even when errors occur, bulkload generates the intermediate files for those entries that had no check errors.

When bulkload completes successfully or with acceptable errors, you can use the intermediate files with the SQL*Loader in load mode. Do not modify these files.

Note:

Always use the check and generate options together if you plan to ignore check-related errors. If you use the generate option without the check option, none of the validation checks are performed. In that case, the intermediate files will contain erroneous entries. Loading such files can lead to data inconsistency and index creation failures.

Task 5: Loading the Input Files

After you have generated the input files, run bulkload with the load option. During this step, bulkload loads the *.dat files, which are in Oracle SQL*Loader specific format, into the database, creates attribute indexes, and generates database statistics. The syntax is:

bulkload connect="connect_string" load="TRUE"

The tool will indicate any errors on the screen. All loading errors are reported in the ORACLE_INSTANCE/diagnostics/logs/OID/tools directory. They reside in bulkload.log and in the SQL*Loader-generated files *.bad and bsl_*.log.If load fails, the database might be in an inconsistent state. Restore the database to its state prior to the bulkload operation, either by using bulkload with the recover option or by restoring Oracle Internet Directory directory from a backup taken before you invoked bulkload. Then repeat the command:

bulkload connect="connect_string" load="TRUE"

If you encounter an error during the indexing phase, you can use:

bulkload coonect="con_str" index=true

to re-create all indexes.

If you encounter an error during database statistics generation, you can use the oidstats.sql command to generate statistics.

See Also:

The "oidstats.sql" command reference in Oracle Fusion Middleware Reference for Oracle Identity Management

15.3.2 Loading Data in Incremental or Append Mode By Using bulkload

If you must add entries to an Oracle Internet Directory server that already contains data, and the server must be up and running at the same time, then you must use the incremental or append mode. This mode is usually faster than other methods of adding entries to the directory. However, you must ensure that the Oracle Internet Directory LDAP instances are in read-modify mode so that bulkload can append data.

You invoke bulkload in incremental or append mode with command lines similar to these:

bulkload connect="conn_str" \
   check="TRUE" generate="TRUE" append="TRUE" file="LDIF_file" 
bulkload connect="conn_str" \
    load="TRUE" append="TRUE" 

15.3.3 Performing Index Verification By Using bulkload

The bulkload operation can either update indexes or create indexes. Sometimes, however, bulkload does not update or create the indexes properly. This is typically due to issues like improper sizing. If this happens, you can use bulkload to verify and re-create all the indexes.

Use the following syntax to invoke bulkload for verification of indexes:

bulkload connect="conn_str" \
    check="TRUE" index="TRUE"

15.3.4 Re-Creating Indexes By Using bulkload

To re-create indexes, use the following syntax:

bulkload connect="conn_str" index="TRUE"

15.3.5 Recovering Data After a Load Failure By Using bulkload

The load phase of bulkload can fail because of issues like improper disk sizing. After such a failure, the directory data might be inconsistent. You can use the recover option to return the directory data to its pre-bulkload state. The syntax is:

bulkload connect="conn_str" recover="TRUE"

15.4 Modifying Attributes of a Large Number of Entries By Using bulkmodify

The bulkmodify tool is useful for modifying the attributes of a large number of entries in an existing directory. It can perform add and replace operations on attribute values. It can operate on a naming context. Using filters, it can also operate selectively on a few entries under a specified naming context.

The bulkmodify tool does not allow add or replace operations on the following attributes:

It does not allow replace operation on the attribute objectclass.

It does not allow add for single-valued attributes.

Output from bulkmodify is logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/bulkmodify.log.

See Also:

The bulkmodify command-line tool reference inOracle Fusion Middleware Reference for Oracle Identity Management

bulkmodify has the following syntax.

bulkmodify connect=connect_string basedn=Base_DN
{[add="TRUE"|"FALSE"]|[replace="TRUE"|"FALSE"]} attribute=attribute_name
value=attribute_value [filter=filter_string] [size=transaction_size]
[threads=num_of_threads] [debug="TRUE"|"FALSE"] [encode=character_set]
[verbose="TRUE"|"FALSE"]

The number of threads should be from one to six times the number of processors.

Select either the add or the replace option. By default both are set to FALSE.

Note:

Before using bulkmodify, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.

15.4.1 Adding a Description for All Entries Under a Specified Naming Context

This example adds descriptions for all the entries under "c=us".

bulkmodify connect="connect_str" baseDN="c=us" add="TRUE" \
    attribute="description" value="US citizen" filter="objectclass=*"

15.4.2 Adding an Attribute for Entries Under a Specified Naming Context Matching a Filter

This example adds telephonenumber for all the entries under "c=us" that have the manager Anne Smith.

bulkmodify connect="connect_str" baseDN="c=us" add="TRUE" \
     attribute="telephoneNumber" \
     value="408-123-4567" filter="manager=cn=Anne Smith"

15.4.3 Replacing an Attribute for All Entries Under a Specified Naming Context

This example replaces pwdreset for all the entries under "c=us".

bulkmodify connect="connect_str"  baseDN="c=us" replace="TRUE" \
    attribute="pwdreset" value="1" filter="objectclass=*" 

15.5 Deleting Entries by Using bulkdelete

bulkdelete is useful for deleting a large number of entries in an existing directory. bulkdelete can delete entries specified under a naming context. By default, it deletes the entries completely. It removes all traces of an entry from the database. If you use the option cleandb FALSE, bulkdelete turns all entries into tombstone entries instead of deleting them completely.

Bulkdelete output is logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/bulkdelete.log.

See Also:

The bulkdelete command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity Management

The bulkdelete tool has the following syntax.

bulkdelete connect=connect_string {[basedn=Base_DN] | [file=file_name]}
[cleandb="TRUE"|"FALSE"] [size=transaction_size] [encode=character_set]
[debug="TRUE"|"FALSE"] [threads=num_of_threads] [verbose="TRUE"|"FALSE"]

Select either the basedn or the file option. If cleandb is TRUE, bulkdelete removes entries completely from the database. By default cleandb is set to TRUE. The number of threads should be from one to six times the number of CPUs.

Notes:

  • Before using bulkdelete, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.

  • If the number of entries to be deleted in the specified naming context is large, you must tune the Oracle Database as specified in the Oracle Internet Directory chapter of Oracle Fusion Middleware Performance and Tuning Guide.

15.5.1 Deleting All Entries Under a Specified Naming Context by Using bulkdelete

This example deletes all the entries under "c=us".

bulkdelete connect="connect_str" baseDN="c=us" cleandb="TRUE"  

15.5.2 Deleting Entries Under Naming Contexts and Making them Tombstone Entries

This example deletes all the entries under "c=us" and leaves them as tombstone entries.

bulkdelete connect="connect_str" baseDN="c=us" cleandb=FALSE

This example deletes all the entries under given base DNs specified in file and leaves them as tombstone entries.

bulkdelete connect="connect_str" file="file" cleandb=FALSE 

15.6 Dumping Data from Oracle Internet Directory to a File by Using ldifwrite

The ldifwrite tool is used to dump of data from an Oracle Internet Directory store to a file. Having the data in a file facilitates loading the data into another node for replication or backup storage. As it writes to the output file, the ldifwrite tool performs a subtree search, including all entries below the specified DN, and the DN itself. It dumps data in LDIF format. It can also dump entries under a specified replication agreement DN.

The ldifwrite tool can dump entries located by using specified filters.Output from ldifwrite is logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/ldifwrite.log.

See Also:

The ldifwrite command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity Management

The ldifwrite tool has the following syntax.

ldifwrite connect=connect_string basedn=Base_DN ldiffile=LDIF_Filename
[filter=LDAP_Filter] [threads=num_of_threads] [debug="TRUE"|"FALSE"]
[encode=character_set] [verbose="TRUE"|"FALSE"]

Use the basedn option to specify the base DN or replication agreement DN.

The number of threads should be from one to six times the number of CPUs.

Note:

Before using ldifwrite, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.

15.6.1 Dumping Part of a Specified Naming Context to an LDIF File

This example uses the following naming context objects defined in partial replication:

dn: cn=includednamingcontext000001, cn=replication namecontext,
 orclagreementid=000001, orclreplicaid=node replica identifier,
 cn=replication configuration
orclincludednamingcontexts: c=us
orclexcludednamingcontexts: ou=Americas, c=us
orclexcludedattributes: userpassword
objectclass: top
objectclass: orclreplnamectxconfig

In this example, all entries under "c=us" are backed up except "ou=Americas,c=us". The userpassword attribute is also excluded. The command is:

ldifwrite connect="conn_str"  \
  baseDN="cn=includednamingcontext000001, cn=replication namecontext, \  
  orclagreementid=000001,orclreplicaid=node replica identifier,\ 
  cn=replication configuration" ldiffile="ldif_file_name" 

15.6.2 Dumping Entries Under a Specified Naming Context to an LDIF File

This example writes all the entries that satisfy LDAP search filter criteria under "ou=Europe, o=imc, c=us" into the output.ldif file.

ldifwrite connect="connect_str" baseDN="ou=Europe, o=imc, c=us"\
   filter="uid=abc" ldiffile="output.ldif" 

15.7 Creating and Dropping Indexes from Existing Attributes by Using catalog

As of Oracle Internet Directory 11g Release 1 (11.1.1.6.0), a new autocatalog feature is enabled by default in fresh installs. You can also enable it if you have upgraded from a previous release. When this feature is enabled, Oracle Internet Directory automatically invokes the catalog command to index attributes when you search for them. If the autocatalog feature is not enabled, and you want to use previously uncataloged attributes in search filters, you must add them to the catalog entry, as in previous releases.

You can now use ldapmodify to create and drop indexes. See Section 21.3.7, "Indexing an Attribute by Using ldapmodify." The ldapmodify command invokes catalog to perform the operation. You can still use catalog for this purpose.

The catalog tool is useful for creating indexes for or dropping indexes from existing attributes. The catalog tool makes an attribute searchable. Output from catalog is logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/catalog.log.

catalog has the following syntax.

catalog connect=connect_string {[add="TRUE"|"FALSE"]|[delete="TRUE"|"FALSE"]}
{[attribute=attribute_name]|[file=file_name]} [logging="TRUE"|"FALSE"]
[threads=num_of_threads] [debug="TRUE"|"FALSE"] [iot="TRUE"|"FALSE"] [verbose="TRUE"|"FALSE"]

Select either the add or the delete option. By default both are set to FALSE.

The number of threads should be from one to six times the number of CPUs.

If logging is TRUE, catalog generates a redo log.

You can specify only one attribute argument on the command line at a time. To add or delete more than one attribute in a single command invocation, use the file option and specify a list of attributes in the file. Use one line for each attribute, for example:

description
sn
title

Notes:

  • As of Oracle Internet Directory 11g Release 1 (11.1.1.6.0), you can use the LDAP tool ldapmodify to create and drop indexes from attributes. See Section 21.3.7, "Indexing an Attribute by Using ldapmodify." The ldapmodify tool actually invokes catalog, and you can still use catalog for this purpose.

  • Before using catalog, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.

  • The catalog command cannot index more than 1000 attributes at a time. If more than 1000 attributes are present in the file, the tool throws an error. If you need to index more than 1000 attributes, use multiple files.

  • As of Oracle Internet Directory 11g Release 1 (11.1.1.7.0), you can specify the IOT option to improve performance. For more information, see the "Oracle Internet Directory Data Management Tools" chapter in the Oracle Fusion Middleware Reference for Oracle Identity Management.

15.7.1 Changing a Searchable Attribute into a Non-searchable Attribute

This example drops an index from the attribute title.

catalog connect="connect_str" delete="TRUE" attribute="title" 

Note:

Unless you are absolutely sure that the indexes were not created by the base schema that was installed with Oracle Internet Directory, be careful not to use the catalog delete=T option to remove indexes from attributes. Removing indexes from base schema attributes can adversely impact the operation of Oracle Internet Directory.