This chapter describes how to perform bulk operations for Oracle Internet Directory to manage object classes, attributes, attribute aliases, and attribute indexes. It describes how to change the server mode and to use tools such as bulkload, bulkmodify, bulkdelete, the Catalog management tool, and LDAP command-line utilities to perform bulk operations.
This chapter includes the following sections:
Section 15.3, "Loading Data Into the Schema by Using bulkload"
Section 15.4, "Modifying Attributes of a Large Number of Entries By Using bulkmodify"
Section 15.6, "Dumping Data from Oracle Internet Directory to a File by Using ldifwrite"
Section 15.7, "Creating and Dropping Indexes from Existing Attributes by Using catalog"
For processing large quantities of data, bulk operations are typically more efficient than standard LDAP operations You can perform bulk operations only by using the command-line bulk tools.
Before you begin using the Oracle Identity Management command-line tools, you must configure your environment. This involves setting the appropriate environment variables.
The syntax and examples provided in this guide require that you have the following environment variables set:
ORACLE_HOME - The location of non-writable files in your Oracle Identity Management installation.
ORACLE_INSTANCE - The location of writable files in your Oracle Identity Management installation.
NLS_LANG (APPROPRIATE_LANGUAGE.AL32UTF8) - The default language set at installation is AMERICAN_AMERICA.
WLS_HOME - The location where the WebLogic Server is installed. This environment variable is required for Oracle Directory Integration Platform commands but not Oracle Internet Directory commands.
PATH - The following directory locations should be added to your PATH:
ORACLE_HOME/bin
ORACLE_HOME/ldap/bin
ORACLE_HOME/ldap/admin
Notes:
The bulk tools do not support attribute uniqueness.
If your schemas were created during installation of a version prior to 11g Release 1 (11.1.1.6.0), you must add datafiles to the OLTS_CT_STORE and OLTS_ATTRSTORE tablespaces if you intend to add more than a million entries to Oracle Internet Directory. Perform this step prior to the bulkload or ldapadd operation. For details, see Oracle Database Administrator's Guide.
In addition to this chapter, bulk tools are also discussed in the following sections of this book:
Section 21.3.9, "Indexing an Attribute by Using the Catalog Management Tool"
Section 37.2, "Migrating Data from LDAP-Compliant Directories"
See Also:
The chapter on Oracle Internet Directory administration tools in Oracle Fusion Middleware Reference for Oracle Identity Management.Notes:
Stop all instances of Oracle Internet Directory before using bulkload. Before using any of the other bulk tools, either stop all instances of Oracle Internet Directory or disable the entry cache.
Do not start Oracle Internet Directory while a bulk tool is executing.
Tip:
The syntax descriptions in this chapter show true or false arguments in the following format:check="TRUE"
You can type true or false in uppercase or lowercase, you can specify just the first letter, and you can omit the double quotes. That is, the following specifications are equivalent:
check=t check=true check="true" check=T check=TRUE check="TRUE"
Some bulk tool operations and replication setup procedures require you to switch an Oracle Internet Directory instance from read/write to read-only mode or from read-only to read/write mode. You can do so by using either Oracle Enterprise Manager Fusion Middleware Control or ldapmodify.
While the server is in read-only mode, only the administrator cn=orcladmin can write to the directory. As a result, you can only make changes to the directory by using utilities that enable you to connect as cn=orcladmin. These include:
Oracle Directory Services Manager
Command line tools
You cannot make changes while the server is in read-only mode by using WLST or Oracle Enterprise Manager Fusion Middleware Control because they connect to the Oracle Internet Directory server as the user cn=emd admin, cn=oracle internet directory.
To set the server mode to read-only from Fusion Middleware Control:
Select Administration, then Server Properties from the Oracle Internet Directory menu.
Select General.
Choose Read-only for Server Mode.
Click Apply.
To set the server mode to read/write mode from Fusion Middleware Control, use the same procedure but choose Read/Write in Step 3.
To set the server mode to Read-Only by using ldapmodify, run the following command:
ldapmodify -D "cn=orcladmin" -q -h host_name \ -p port -f change_mode.ldif
where the file change_mode.ldif has the following content:
dn: cn=componentname,cn=osdldapd,cn=subconfigsubentry
changetype: modify
replace: orclservermode
orclservermode: r
To set the server mode to Read/Write by using ldapmodify, use the same command but change the last line in the LDIF file to:
orclservermode: rw
The bulk loader, bulkload, is a bulk management tool. It takes input data in LDIF or SQL*Loader format and loads the data directly into Oracle Internet Directory's schema in the metadata repository. It has three main phases: check, generate and load.
In the check phase, bulkload parses and verifies the LDIF input data for the schema.
In the generate phase, bulkload generates intermediate files in SQL*Loader format.
In the load phase, bulkload can use either of two methods: bulk mode loading or incremental mode loading.
When using bulk mode loading, bulkload loads generated intermediate files into the database. As it does so, it drops old indexes and generates new ones.
When using incremental mode loading, bulkload loads intermediate files to tables in the database using insert mode. While it loads the data, bulkload updates the indexes.
Bulk mode loading is faster than incremental mode loading.
The bulk loader also supports the following features:
It enables you to specify the number of threads in order to achieve parallelism during the generate and load phases.
It has an encode option that enables you to use data in other languages.
It has a restore option that enables you to retain the operational attributes specified in the LDIF file.
It has an index option for index recreation and a missing option for creation of missing indexes.
It has a recover option that is useful for recovering from bulkload failures.
When appending the data to existing directories, bulkload supports both bulk mode and incremental mode loading.
The append option enables you to load data while the LDAP server is up and running.
At the beginning of the generate phase, the server's orclServerMode, in the instance-specific configuration entry, is changed from read/write to read-modify. At the end of the generate phase, it is left in the read-modify state so that you cannot add entries to Oracle Internet Directory between the generate and load phases. This is necessary to maintain internal sequence numbers. You are expeced to run the load phase immediately after the generate phase. At the end of the load phase, the servers' orclServerMode is set back to read/write. Using bulkload with the recover option also sets orclServerMode back to read/write.
At the start of the load operation, bulkload determines the current configured value of orclRIenabled, then disables referential integrity. At the end of load phase, bulkload returns orclRIenabled to its original value. If is any referential integrity violations occurred, however, referential integrity is disabled, and you see the message:
There is a violation of Referential Integrity and hence it is Disabled now. Run the OIDDIAG tool with diagnostic option to collect the Entries which have dangling DN attribute values and Fix the violation
Fix the violation and then set orclRIenabled to the desired value.
The bulkload tool generates the following output files in the ORACLE_INSTANCE/diagnostics/logs/OID/tools directory:
An output log, bulkload.log
A list of duplicate DNs, duplicateDN.log
Intermediate files, *.ctl and *.dat
The bulkload tool generates the following output files in the ORACLE_INSTANCE/OID/load directory:
A list of bad LDIF entries, badentry.ldif
A list of all dynamic group entries that can be added using ldapadd, dynGrp.ldif
Intermediate log files generated by the SQL*Loader, bsl_*.log
Notes:
If a directory server instance is participating in a replication agreement, do not use the bulkload tool to add data into the node. Instead, use ldapadd.
Before using bulkload, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.
Running the bulkload load operation sets the server mode to read/write. If you require a different mode, reset it after performing the load operation.
If the applicable password policy has the pwdmustchange attribute set to 1, then for every new entry loaded by bulkload, the pwdreset attribute is set to 1 by default. See Chapter 29, "Managing Password Policies" for more information.
If you do not use the bulkload utility to populate the directory, then you must run the oidstats.sql tool to avoid significant search performance degradation.
See Also:
Theoidstats.sql command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity Management for a description and syntax for the oidstats.sql toolThe bulkload tool has the following syntax:
bulkload [connect=connect_string] {[check="TRUE"|"FALSE" [file=ldif_file]] [generate="TRUE"|"FALSE" [append="TRUE"|"FALSE"] [restore="TRUE"|"FALSE"] [thread=num_of_threads] file=ldif_file] [load="TRUE"|"FALSE" [append="TRUE"|"FALSE"] [threads=num_of_threads]] [index="TRUE"|"FALSE"] [missing="TRUE"|"FALSE"] [recover="TRUE"|"FALSE"]} [encode=character_set] [debug="TRUE"|"FALSE"] [verbose="TRUE"|"FALSE"]
Some of the parameter combinations are valid while others are invalid.
You must set the environment variable ORACLE_INSTANCE. Specify the fully qualified path to the Oracle Instance where the intermediate file is generated.
You must specify at least one of the following actions when you invoke bulkload: check, generate, load, append, recover, or index.
If check is TRUE, bulkload performs a schema check.
If generate is TRUE, bulkload generates intermediate files.
When using the check or generate action, you must specify the path name to the LDIF data file.
If load is TRUE, bulkload loads intermediate files.
When append is TRUE, bulkload can perform its actions while the server is up and running.
Use the restore flag only when the LDIF file contains operational attributes, such as orclguid or creatorsname. Avoid having operational attributes in the LDIF file when the restore flag is not specified or is set to FALSE.
Do not specify recover with any other option.
The option combination check index verifies the existing indexes.
To import an LDIF file, you use the bulkload utility. This section discusses the tasks to process an LDIF file through bulkload.
See Also:
Thebulkload command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity Management.This section contains these topics:
Task 1: Stopping Oracle Internet Directory Processes
Stop all Oracle Internet Directory server instances by using either Fusion Middleware Control or the command line.
Task 2: Backing Up the Oracle Database Server
Before you import the file, back up the Oracle database server as a safety precaution.
See Also:
Oracle Database Backup and Recovery User's Guide in the Oracle Database Documentation LibraryTask 3: Finding Out the Oracle Internet Directory Password
To use bulkload, you must provide the Oracle Internet Directory ODS schema password.
See Also:
Theoidpasswd command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity ManagementTask 4: Checking Input for Schema and Data Consistency Violations and Generating the Input Files for SQL*Loader
On UNIX, the bulkload tool usually resides in $ORACLE_HOME/ldap/bin. On Microsoft Windows, this tool usually resides in ORACLE_HOME\ldap\bin.
Check the input file and generate files for the SQL*Loader by typing:
bulkload connect="connect_string" \ check="TRUE" generate="TRUE" file="full_path_to_ldif-file_name"
When you specify both the check and generate options, the entries are checked for schema compliance.
All check-related errors are reported as command line output. All schema violations are reported in ORACLE_INSTANCE/diagnostics/logs/OID/tools/bulkload.log. All bad entries are logged in ORACLE_INSTANCE/OID/load/badentry.ldif.
If there are duplicate entries, their DNs are logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/duplicateDN.log. This is just for information purpose. The bulkload tool does not generate duplicate data for duplicate entries. It ignores duplicate entries.
Use a text editor to fix all bad entries, then re-run bulkload with the check and generate options. Repeat until there are no errors, or until the remaining errors are acceptable to you. For example, you might be willing to load a small number of entries with ldapadd.
The bulkload tool generates intermediate *.ctl and *.dat files in the ORACLE_INSTANCE/OID/load directory. Even when errors occur, bulkload generates the intermediate files for those entries that had no check errors.
When bulkload completes successfully or with acceptable errors, you can use the intermediate files with the SQL*Loader in load mode. Do not modify these files.
Note:
Always use thecheck and generate options together if you plan to ignore check-related errors. If you use the generate option without the check option, none of the validation checks are performed. In that case, the intermediate files will contain erroneous entries. Loading such files can lead to data inconsistency and index creation failures.Task 5: Loading the Input Files
After you have generated the input files, run bulkload with the load option. During this step, bulkload loads the *.dat files, which are in Oracle SQL*Loader specific format, into the database, creates attribute indexes, and generates database statistics. The syntax is:
bulkload connect="connect_string" load="TRUE"
The tool will indicate any errors on the screen. All loading errors are reported in the ORACLE_INSTANCE/diagnostics/logs/OID/tools directory. They reside in bulkload.log and in the SQL*Loader-generated files *.bad and bsl_*.log.If load fails, the database might be in an inconsistent state. Restore the database to its state prior to the bulkload operation, either by using bulkload with the recover option or by restoring Oracle Internet Directory directory from a backup taken before you invoked bulkload. Then repeat the command:
bulkload connect="connect_string" load="TRUE"
If you encounter an error during the indexing phase, you can use:
bulkload coonect="con_str" index=true
to re-create all indexes.
If you encounter an error during database statistics generation, you can use the oidstats.sql command to generate statistics.
See Also:
The "oidstats.sql" command reference in Oracle Fusion Middleware Reference for Oracle Identity ManagementIf you must add entries to an Oracle Internet Directory server that already contains data, and the server must be up and running at the same time, then you must use the incremental or append mode. This mode is usually faster than other methods of adding entries to the directory. However, you must ensure that the Oracle Internet Directory LDAP instances are in read-modify mode so that bulkload can append data.
You invoke bulkload in incremental or append mode with command lines similar to these:
bulkload connect="conn_str" \
check="TRUE" generate="TRUE" append="TRUE" file="LDIF_file"
bulkload connect="conn_str" \
load="TRUE" append="TRUE"
The bulkload operation can either update indexes or create indexes. Sometimes, however, bulkload does not update or create the indexes properly. This is typically due to issues like improper sizing. If this happens, you can use bulkload to verify and re-create all the indexes.
Use the following syntax to invoke bulkload for verification of indexes:
bulkload connect="conn_str" \
check="TRUE" index="TRUE"
To re-create indexes, use the following syntax:
bulkload connect="conn_str" index="TRUE"
The load phase of bulkload can fail because of issues like improper disk sizing. After such a failure, the directory data might be inconsistent. You can use the recover option to return the directory data to its pre-bulkload state. The syntax is:
bulkload connect="conn_str" recover="TRUE"
The bulkmodify tool is useful for modifying the attributes of a large number of entries in an existing directory. It can perform add and replace operations on attribute values. It can operate on a naming context. Using filters, it can also operate selectively on a few entries under a specified naming context.
The bulkmodify tool does not allow add or replace operations on the following attributes:
dn (use ldapmoddn instead)
cn (use ldapmodify instead)
userpassword (use ldapmodify instead)
orclpassword (use ldapmodify instead)
orclentrylevelaci (use ldapmodify instead)
orclaci (use ldapmodify instead)
orclcertificatehash
orclcertificatematch
any binary attribute
any operational attribute
It does not allow replace operation on the attribute objectclass.
It does not allow add for single-valued attributes.
Output from bulkmodify is logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/bulkmodify.log.
See Also:
Thebulkmodify command-line tool reference inOracle Fusion Middleware Reference for Oracle Identity Managementbulkmodify has the following syntax.
bulkmodify connect=connect_string basedn=Base_DN {[add="TRUE"|"FALSE"]|[replace="TRUE"|"FALSE"]} attribute=attribute_name value=attribute_value [filter=filter_string] [size=transaction_size] [threads=num_of_threads] [debug="TRUE"|"FALSE"] [encode=character_set] [verbose="TRUE"|"FALSE"]
The number of threads should be from one to six times the number of processors.
Select either the add or the replace option. By default both are set to FALSE.
Note:
Before usingbulkmodify, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.This example adds descriptions for all the entries under "c=us".
bulkmodify connect="connect_str" basedn="c=us" add="TRUE" \
attribute="description" value="US citizen" filter="objectclass=*"
This example adds telephonenumber for all the entries under "c=us" that have the manager Anne Smith.
bulkmodify connect="connect_str" basedn="c=us" add="TRUE" \
attribute="telephoneNumber" \
value="408-123-4567" filter="manager=cn=Anne Smith"
bulkdelete is useful for deleting a large number of entries in an existing directory. bulkdelete can delete entries specified under a naming context. By default, it deletes the entries completely. It removes all traces of an entry from the database. If you use the option cleandb FALSE, bulkdelete turns all entries into tombstone entries instead of deleting them completely.
Bulkdelete output is logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/bulkdelete.log.
See Also:
Thebulkdelete command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity ManagementThe bulkdelete tool has the following syntax.
bulkdelete connect=connect_string {[basedn=Base_DN] | [file=file_name]} [cleandb="TRUE"|"FALSE"] [size=transaction_size] [encode=character_set] [debug="TRUE"|"FALSE"] [threads=num_of_threads] [verbose="TRUE"|"FALSE"]
Select either the basedn or the file option. If cleandb is TRUE, bulkdelete removes entries completely from the database. By default cleandb is set to TRUE. The number of threads should be from one to six times the number of CPUs.
Notes:
Before using bulkdelete, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.
If the number of entries to be deleted in the specified naming context is large, you must tune the Oracle Database as specified in the Oracle Internet Directory chapter of Oracle Fusion Middleware Performance and Tuning Guide.
This example deletes all the entries under "c=us".
bulkdelete connect="connect_str" basedn="c=us" cleandb="TRUE"
This example deletes all the entries under "c=us" and leaves them as tombstone entries.
bulkdelete connect="connect_str" basedn="c=us" cleandb=FALSE
This example deletes all the entries under given base DNs specified in file and leaves them as tombstone entries.
bulkdelete connect="connect_str" file="file" cleandb=FALSE
The ldifwrite tool is used to dump of data from an Oracle Internet Directory store to a file. Having the data in a file facilitates loading the data into another node for replication or backup storage. As it writes to the output file, the ldifwrite tool performs a subtree search, including all entries below the specified DN, and the DN itself. It dumps data in LDIF format. It can also dump entries under a specified replication agreement DN.
The ldifwrite tool can dump entries located by using specified filters.Output from ldifwrite is logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/ldifwrite.log.
See Also:
Theldifwrite command-line tool reference in Oracle Fusion Middleware Reference for Oracle Identity ManagementThe ldifwrite tool has the following syntax.
ldifwrite connect=connect_string basedn=Base_DN ldiffile=LDIF_Filename [filter=LDAP_Filter] [threads=num_of_threads] [debug="TRUE"|"FALSE"] [encode=character_set] [verbose="TRUE"|"FALSE"]
Use the basedn option to specify the base DN or replication agreement DN.
The number of threads should be from one to six times the number of CPUs.
Note:
Before usingldifwrite, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.This example uses the following naming context objects defined in partial replication:
dn: cn=includednamingcontext000001, cn=replication namecontext, orclagreementid=000001, orclreplicaid=node replica identifier, cn=replication configuration orclincludednamingcontexts: c=us orclexcludednamingcontexts: ou=Americas, c=us orclexcludedattributes: userpassword objectclass: top objectclass: orclreplnamectxconfig
In this example, all entries under "c=us" are backed up except "ou=Americas,c=us". The userpassword attribute is also excluded. The command is:
ldifwrite connect="conn_str" \ basedn="cn=includednamingcontext000001, cn=replication namecontext, \ orclagreementid=000001,orclreplicaid=node replica identifier,\ cn=replication configuration" ldiffile="ldif_file_name"
This example writes all the entries that satisfy LDAP search filter criteria under "ou=Europe, o=imc, c=us" into the output.ldif file.
ldifwrite connect="connect_str" basedn="ou=Europe, o=imc, c=us"\ filter="uid=abc" ldiffile="output.ldif"
As of Oracle Internet Directory 11g Release 1 (11.1.1.6.0), a new autocatalog feature is enabled by default in fresh installs. You can also enable it if you have upgraded from a previous release. When this feature is enabled, Oracle Internet Directory automatically invokes the catalog command to index attributes when you search for them. If the autocatalog feature is not enabled, and you want to use previously uncataloged attributes in search filters, you must add them to the catalog entry, as in previous releases.
You can now use ldapmodify to create and drop indexes. See Section 21.3.7, "Indexing an Attribute by Using ldapmodify." The ldapmodify command invokes catalog to perform the operation. You can still use catalog for this purpose.
The catalog tool is useful for creating indexes for or dropping indexes from existing attributes. The catalog tool makes an attribute searchable. Output from catalog is logged in ORACLE_INSTANCE/diagnostics/logs/OID/tools/catalog.log.
catalog has the following syntax.
catalog connect=connect_string {[add="TRUE"|"FALSE"]|[delete="TRUE"|"FALSE"]} {[attribute=attribute_name]|[file=file_name]} [logging="TRUE"|"FALSE"] [threads=num_of_threads] [debug="TRUE"|"FALSE"] [iot="TRUE"|"FALSE"] [verbose="TRUE"|"FALSE"]
Select either the add or the delete option. By default both are set to FALSE.
The number of threads should be from one to six times the number of CPUs.
If logging is TRUE, catalog generates a redo log.
You can specify only one attribute argument on the command line at a time. To add or delete more than one attribute in a single command invocation, use the file option and specify a list of attributes in the file. Use one line for each attribute, for example:
description sn title
Notes:
As of Oracle Internet Directory 11g Release 1 (11.1.1.6.0), you can use the LDAP tool ldapmodify to create and drop indexes from attributes. See Section 21.3.7, "Indexing an Attribute by Using ldapmodify." The ldapmodify tool actually invokes catalog, and you can still use catalog for this purpose.
Before using catalog, ensure that the environment variable ORACLE_INSTANCE has been set to the full path name of the Oracle instance.
The catalog command cannot index more than 1000 attributes at a time. If more than 1000 attributes are present in the file, the tool throws an error. If you need to index more than 1000 attributes, use multiple files.
As of Oracle Internet Directory 11g Release 1 (11.1.1.7.0), you can specify the IOT option to improve performance. For more information, see the "Oracle Internet Directory Data Management Tools" chapter in the Oracle Fusion Middleware Reference for Oracle Identity Management.
This example drops an index from the attribute title.
catalog connect="connect_str" delete="TRUE" attribute="title"
Note:
Unless you are absolutely sure that the indexes were not created by the base schema that was installed with Oracle Internet Directory, be careful not to use thecatalog delete=T option to remove indexes from attributes. Removing indexes from base schema attributes can adversely impact the operation of Oracle Internet Directory.This example adds an index to the attribute title.
catalog connect="connect_str" add="TRUE" attribute="title"