Bulk Operations Command-Line Tools Syntax

This section contains these topics:

• bulkdelete Syntax

• bulkload Syntax

• bulkmodify Syntax

• ldifwrite Syntax

  Note: To run shell script tools on the Windows operating system, you need one of the following UNIX emulation utilities: Cygwin 1.3.2.2-1 or later. Visit: http://sources.redhat.com MKS Toolkit 6.1. Visit: http://www.datafocus.com/

  Note: All bulk tools require you to enter the correct password in order to access the ods database.

bulkdelete Syntax

The bulkdelete command-line tool enables you to delete a subtree efficiently. It can be used when both an Oracle directory server and Oracle directory replication servers are in operation. It uses a SQL interface to benefit performance. For this release, the bulkdelete tool runs on only one node at a time.

Note: Make sure that when bulkmodify is invoked, server-side entry cache is disabled.

This tool does not support filter-based deletion. That is, it deletes an entire subtree below the root of the subtree. If the base DN is a user-added DN, rather than a DN created as part of the installation of the directory, it is included in the delete. You must restrict LDAP activity against the subtree during deletion.

The bulkdelete tool uses this syntax:

bulkdelete.sh -connect connect_string -base "base_dn" -size number_of_entries 
-encode "character_set"

Table A-16  Arguments for bulkdelete

Mandatory Argument	Description
-connect connect_string	Specifies the connect string to connect to the directory database. This argument is mandatory. See Also: Oracle9i Net Services Administrator's Guide in the Oracle Database Documentation Library
-base "base_dn"	Specifies the base DN of the subtree to be deleted. This argument is mandatory.
-size number_of_entries	Specifies the number of entries to be committed as a part of one transaction.
-encode "character_set"	Specifies native character set encoding. See Also: Chapter G, "Globalization Support in the Directory".

bulkload Syntax

The bulkload command-line tool is useful for loading large number of entries to a directory server. It uses Oracle SQL*Loader to load directory entries. The bulkload tool expects the input file to be in LDIF.

See Also: "LDAP Data Interchange Format (LDIF) Syntax" Oracle Application Server 10g Upgrading to 10g (9.0.4) for any special instructions about upgrading orclguids when bulkloading an LDIF file from an older version of Oracle Internet Directory

The bulkload tool performs its operation in following phases:

1. Check

   In the check phase, all entries of LDIF files are verified for valid LDAP schema and duplicate entries. If there are any errors reported by bulkloader, then the user needs to rectify the error and retry bulkload.

2. Generate

   In the generate phase, the LDIF input is converted into intermediate files that can be used by SQL*Loader to load the data into the Oracle Internet Directory directory store.

3. Load

   The Intermediate files generated in generate phase are loaded into the Oracle9i Database Server which is the Oracle Internet Directory directory store. Bulkloader supports two types of loading of data:

   • Incremental Mode Loading

     In incremental mode, you can append data to existing directory data. It should be used when you want to append a small amount of data. It is faster than other "add" methods, but slower than bulk mode loading. In this mode, Bulkloader does not drop and rebuild catalog indexes. Instead, it uses SQL*Loader in insert mode to add data to the database and update indexes through inserts.

     Here, "small amount" is a relative number, it really depends upon existing data in directory as well amount of data to be loaded and the hardware capabilities to handle the load.

     To invoke incremental mode, user need to specify -append along with other options.

     It is important to note that when using bulkload in incremental mode, one must put the directory server in read-modify mode. During read-modify mode, search and modify operations are allowed but add, delete, and modifyDN operations are restricted.

     See Also: "Task 2: Configure Structural Access Items" for instructions on using Oracle Directory Manager to set access rights

   • Bulk Mode Loading

     In bulk mode, you must be able to add or append large number of entries to a directory. By default, Bulkloader runs in bulk mode. Bulk mode is faster than incremental mode.

     In bulk mode, all Oracle Internet Directory server instances should be stopped. In this mode, Bulkloader drops existing indexes and recreates them after loading of data. For data loading, it uses SQL*Loader direct-path mode.

     ti

     See Also: "Stopping an Oracle Directory Server Instance"

4. Index Creation

   After the load is complete, the indexes are recreated if the load was done in "bulk" mode. Also, the Bulkloader tool provides an option just to recreate all indexes. This is useful in case if previous index creation was unsuccessful for some reason.

5. Directory Data Recovery

   A failure in the 'load' phase of bulkload operation can leave directory data in inconsistent state. Bulkloader can revert back to original state that existed prior to the invocation of bulkload. Use the -recover option to recover directory data in case of Bulkload failure.

Usage Scenarios for the bulkload Tool

The bulkload tool can be used in single node as well as multiple node environments.

Single Node Environment

Loading in 'bulk' mode

The typical usage scenario is to load directory data after Oracle Internet Directory installation. One would want to 'check' the LDIF file for schema errors, 'generate' the intermediate files and 'load' the data into the Oracle Internet Directory directory store. The 'parallel' option is normally faster since the load and index creation happens in parallel. The invocation of bulkload will be something like:

bulkload.sh -connect <conn_str> -check -generate -load -parallel <LDIF>

One can break up the above operation one into separate 'check, 'generate' and 'load' invocations. The 'check' can also be avoided if the LDIF data is from another Oracle Internet Directory directory node.

Loading in 'incremental' or 'append' Mode

If one needs to add directory entries to an Oracle Internet Directory directory store that already has some user LDIF data, then the 'incremental' or 'append' mode is the way to go. This mode is normally faster than other methods of adding entries to the directory. However, it must be ensured that the Oracle Internet Directory LDAP instances are in read-modify mode before bulkload begins to append data. The typical invocation of bulkload will be something like:

bulkload.sh -connect <conn_str> -check -generate -load -append <LDIF>

Index recreation

The bulkload operation will take care of either updating the indexes or creating the indexes. However, due to issues like improper sizing, it may so happen that the indexes may not be updated or created properly. Bulkloader provides an option to recreate all the indexes. The invocation of bulkload will be:

bulkload.sh -connect <conn_str> -index

Data recovery on errors

Due to issues like improper disk sizing, the 'load' phase of bulkload may fail. If this happens, there are chances that the directory data is inconsistent and hence, bulkloader provides an option to recover the directory data to the state that existed prior to the invocation of bulkload:

bulkload.sh -connect <conn_str> -recover

Multi-Node Environment

Bulk Mode Loading

One must specify the connect strings of all the Oracle Internet Directory nodes involved. The bulkload will be invoked something like:

bulkload.sh -connect "<conn_str1> <conn_str2> <conn_str3>" -check -generate 
-load -parallel <LDIF>

The bulkloader will handle this as given below:

1. Bulkloader prompts the user to put all Oracle Internet Directory LDAP servers on all the nodes in 'read-modify' mode.

2. Bulkloader will prompt the user to bring down the Oracle Internet Directory servers on the node corresponding to <conn_str1>

3. The 'check' and 'generate' will be performed on the node corresponding to <cons_str1>

4. The 'load' will be performed on the node corresponding to <conns_str1>

5. Bulkloader will prompt the user to bring up the Oracle Internet Directory servers on the node corresponding to <conn_str1>

6. Steps 2, 4 and 5 will be repeated for all the nodes.

7. Now, all the Oracle Internet Directory LDAP servers can be changed to read-write mode.

Incremental Mode Loading

Here, the approach is the same as in the bulk mode loading except that the Oracle Internet Directory LDAP servers need not be shutdown, rather all of them need to be in 'read-only' mode.

Limitations of Bulkloader in Oracle Internet Directory 10g (9.0.4)

In multi-node environments, it is the directory administrator's responsibility to make sure that all nodes have same schema before running bulkloader.

If the user sees bad entries logged in badentry.ldif but does not rectify the entries, then data can be inconsistent.

The 'check' mode of bulkloader does not check and report the lack of parent/child relationships between entries.

The 'incremental' or 'append' mode is only for adding new entries not new attribute values to existing entries.

In multi-node environments, the first connect string specified must refer to the local node.

See Also: "LDAP Data Interchange Format (LDIF) Syntax".

Syntax for the bulkload Tool

The bulkload tool uses this syntax:

bulkload.sh -connect connect_string <[-check] [-generate] [-restore] 
[-numThread] [-parallel] [-encode] [-append] [-load] | [-index] | [-recover] 
absolute_path_to_LDIF_data_file

Table A-17 lists and describes the arguments.

Table A-17  Arguments for bulkload.sh

Argument	Description	Mandatory?
-connect	Specifies the net service name defined in the tnsnames.ora file. For loading data in single node, specify its connect-string--for example orcl. For loading data in multiple nodes, specify connect-strings of all nodes--for example, orcl1 orcl2 orcl3 See Also: Oracle9i Net Services Administrator's Guide in the Oracle9i Database Server Documentation Library	Yes
-check	Checks LDAP schema for inconsistencies and for existence of duplicate DNs in the file	No
-generate	Creates intermediate files suitable for loading into Oracle Internet Directory using SQL*Loader	No
-restore	Assumes operational attributes, such as orclguid, creatorsname, and createtimestamp, are already present in the specified LDIF file. When used with -generate, bulkload.sh avoids creating duplicate operational attribute values in the output SQL*Loader files. When used with -check, bulkload.sh suppresses errors associated with finding pre-existing operational attribute values in LDIF files.	No
-numThread n	Specifies the number of threads to be created. - numThread is useful only in -generate mode. The default value is number of CPUs on machine + 1	No
-parallel	Specifies that the loading should be done in parallel. Useful with -load option	No
-encode	Specifies native character set encoding See Also: Chapter G, "Globalization Support in the Directory"	No
-append	Specifies append incremental mode (Default is bulkmode append)	No
-load	Loads files resulting from generate phase into specified database	No
-index	Recreates indexes on all catalog tables	No
-recover	In case of bulkload.sh failure, recovers directory with original data	No
-file_name	Absolute path of ldif file	No

The LDIF data file path must be fully specified for check or generate operations.

While calling bulkload at least one of -check, -generate, -load, -recover or -index actions must be specified.

There are certain combinations of options that should be called together for effective bulkloading.

• The -restore flag should only be used when ldif file contains operational attributes such as orclguid, creatorsname, and so forth.

• The path name to the LDIF data file should be fully specified, and the data file must be specified for the -check or -generate actions.

• -numThread is useful only if given with -generate option.

• -parallel should be called with -load only.

• -recover or -index should not be specified with any other option.

  See Also: "Directory Replication" for further information and resources for bulk loading multiple nodes in a replicated environment

Bulk Loading Multiple Nodes in a Replicated Environment

After generating a file with the generate option, you can use the load option to load multiple computers with the identical SQL*Loader file. Do this only when creating a new replica node.

See Also: "Oracle Directory Replication Administration"

When you load the same data into multiple nodes in a replicated network, ensure that the orclGUID parameter (global IDs) is consistent across all the nodes. You can accomplish this by generating the bulkload data file once only (using the -generate option), and then using the same data file to load the other nodes (using the -load option).

bulkmodify Syntax

The bulkmodify command-line tool enables you to modify a large number of existing entries in an efficient way. The bulkmodify tool supports the following:

• Subtree based modification

• A single attribute filter. For example, the filter could be objectclass=*, objectclass=oneclass, or telephonenumber=*.

• Attribute value addition and replacement. It modifies all matched entries in bulk.

The bulkmodify tool performs schema checking on the specified attribute name and value pair during initialization. All entries that meet the following criteria are modified:

• They are under the specified subtree.

• They meet the single filter condition.

• They contain the attribute to be modified as either mandatory or optional.

The Oracle directory server and Oracle directory replication server may be running concurrently while bulk modification is in progress, but the bulk modification does not affect the replication server. You must perform bulk modification against all replicas.

Note: LDIF file based modification is not supported by bulkmodify. This type of modification requires per-entry-based schema checking, and therefore the performance gain over the existing ldapmodify tool is insignificant. Make sure that when bulkmodify is invoked, server side entry cache is disabled.

You must restrict user access to the subtree during bulk modification. If necessary, ACI restriction can be applied to the subtree being updated by bulkmodify.

You cannot use bulkmodify to add a value to single-valued attributes that already contain one value. If a second value is added, you must alter the directory schema to make that attribute multi-valued.

The bulkmodify tool uses this syntax:

bulkmodify -c connect_string -b "base_dn" {-a|-r} attr_name -v att_value [-f 
filter] [-s size]

Table A-18  Arguments for bulkmodify

Argument	Description
-c connect_string	Specifies the connect string for the directory database. This argument is mandatory. See Also: Oracle9i Net Services Administrator's Guide in the Oracle Database Documentation Library
-b "base_dn"	Specifies the base DN of the subtree to be modified. This argument is mandatory.
-a attr_name	Specifies the attribute name for addition. This argument is mandatory.
-r attr_name	Specifies the attribute name for replacement. This argument is mandatory.
-v attr_value	Specifies the attribute value for either addition or replacement. This argument is mandatory.
-f filter	Specifies the filter to be used
-s number_of_entries	Specifies the number of entries to be committed as a part of one transaction. If not specified, default is 100.
-E "character_set"	Specifies native character set encoding. See Chapter G, "Globalization Support in the Directory".

The filter specified with the -f option must contain a single attribute.

If a filter is not specified, the default filter objectclass=* is assumed.

There can be only one attribute name specified in the -a or -r option in each execution.

There can be only one value specified in the -v option in each execution. For example, the following bulkmodify command adds the telephone number 408-123-4567 to the entries of all employees who have Anne Smith as their manager:

bulkmodify -c my_database -b "c=US" -a telephoneNumber -v "408-123-4567" -f 
"manager=Anne Smith"

To assure that the modified entries are read, after completing the bulkmodify procedure, restart the Oracle Internet Directory server.

ldifwrite Syntax

The ldifwrite command-line tool enables you to convert to LDIF all or part of the information residing in an Oracle Internet Directory. This makes that information available for loading into a new node in a replicated directory or into another node for backup storage.

Note: The ldifwrite tool output does not include operational data of the directory itself--for example, cn=subschemasubentry, cn=catalogs, and cn=changelog entries. To export these entries into LDIF format, use ldapsearch with the -L flag.

The ldifwrite tool performs a subtree search, including all entries below the specified DN, including the DN itself.

The ldifwrite tool uses this syntax:

ldifwrite -c connect_string -b "base_DN" -f file_name 

Table A-19  Arguments for ldifwrite

Mandatory Argument	Description
-c connect_string	Specifies the net service name for the directory that is the source of the data, as defined in the tnsnames.ora file. This argument is mandatory. See Also: Oracle9i Net Services Administrator's Guide in the Oracle Database Documentation Library
-b "base_dn"	Specifies the base of the subtree to be written out in LDIF format. This argument is mandatory. If the base DN is the replication agreement entry, then you can back up part of the naming context based on the LDAP naming context configuration. In this case, the syntax is: ldifwrite -c connect string -b "replication agreement DN" -f file name See Also: "Rules for Partial Replication Filtering"
-f file_name	Specifies the name of the LDIF file to be created. This argument is mandatory.
-E "character_set"	Specifies native character set encoding. See Also: "Using Globalization Support with ldifwrite"

Example 1: Converting All Entries Under a Specified Naming Context to an LDIF File

This example writes all the entries under ou=Europe, o=imc, c=us into the output1.ldi file.

ldifwrite -c nldap -b "ou=Europe, o=imc, c=us" -f output1.ldi

All the arguments are mandatory.

The LDIF file and the intermediate file are always written to the current directory.

The ldifwrite tool includes the operational attributes of each entry in the directory, including createtimestamp, creatorsname, and orclguid.

When prompted for the Oracle Internet Directory password, enter the password of the underlying ODS user. The default password is ods.

Example 2: Converting Part of a Specified Naming Context to an LDIF File

This example uses the following naming context objects defined in partial replication:

dn: cn=includednamingcontext000001,

cn=replication namecontext,

orclagreementid=000001,

orclreplicaid=node replica identifier,

cn=replication configuration
orclincludednamingcontexts: c=us
orclexcludednamingcontexts: ou=Americas, c=us
orclexcludedattributes: userpassword
objectclass: top
objectclass: orclreplnamectxconfig

In this example, all entries under c=us are backed up except ou=Americas,c=us. The userpassword attribute is also excluded. The command is

ldifwrite -c connect string -b "cn=includednamingcontext000001,cn=replication 
namecontext,orclagreementid=000001,orclreplicaid=node replica 
identifier,cn=replication configuration" -f file name