| Sun ONE Meta-Directory 5.1.1 Deployment Guide |
Chapter 2
Planning the Meta-Directory SystemThe Meta-Directory solution involves planning, installing, and configuring the Sun ONE Directory Server and Sun ONE Meta-Directory systems so they can produce the Meta View you have designed. In addition to these software components, you might also need to consider the installation and configuration of any additional services that your external directory data sources might need to operate.
Successful integration of Meta-Directory in the enterprise occurs in three phases: Design, Deployment, and Maintenance:
- In the design phase, you analyze your existing system and establish the architecture and requirements of the Meta-Directory system you will deploy. In this phase, you must take a careful look at your existing directory data sources and plan how to incorporate these into a single Meta View.
- In the deployment phase, you devise a systematic approach to the configuration of components and the implementation of servers and services in the enterprise. This phase deals with how to set up a Meta-Directory system, the different servers and services you need to install, and the topology of the hardware that you will be employing.
- In the maintenance phase, you provide a plan for maintaining the Meta-directory system once it has been deployed. This third and last phase deals with what needs to be done to keep your Meta-Directory system properly tuned and maintained.
This chapter contains the following sections:
Designing Your Meta-Directory SystemIdeally, all of your directory data would be contained in one integrated Sun ONE Directory Server system, and the system would be spread evenly through the entire enterprise. We know, however, that such an ideal system is rarely in place. More often than not, many different types of directory systems are implemented across a large corporate network. Although the ideal of a single LDAP directory system is often out of reach, you can use Meta-Directory to join the heterogeneous types of directory systems running across your organization, and thus create a single LDAP directory of the information.
This section addresses the different tasks that you need to perform to design a robust Meta-Directory system. Included in this section are the following subsections:
Beginning the System Design
When reviewing the systems that are implemented across your organization, it’s likely that you will see Sun ONE Directory Server already being used as part of the directory system mix. When you add Meta-Directory to the directory architecture, you will be adding at least another instance of Sun ONE Directory Server to the mixture of directory systems already in place. When you begin to address Meta-Directory system design, it’s a good idea to separate the Directory Server instances that host organizational data and the Directory Server instances that host any of the Meta-Directory views. Its also advised to store configuration data c=netscaperoot in a separate directory server.
Before you can begin to design your Meta-Directory system, you must clearly define and design the Directory Server system(s) you will use to host your organization’s user directories. To help with this task, the Sun ONE Directory Server Deployment Guide details the planning steps that you should follow to successfully deploy Directory Server in your organization.
Although the entire Sun ONE Directory Server Deployment Guide is devoted to designing and implementing your user directory system, most of the principles and concepts described in that guide are also valid for designing and deploying a Meta-Directory system (Meta-Directory is built on Sun ONE Directory Server). This guide assumes that you understand the concepts presented in the Sun ONE Directory Server Deployment Guide and that you are familiar with configuring and running Directory Server, as explained in the Sun ONE Directory Server Administrator’s Guide.
The design of Meta-Directory begins with a site survey to gather information about the existing data sources and how you want to incorporate them into a single LDAP directory (the Meta View). The next steps are to determine the data flow (selecting the attributes you want to use from each database) and plan the physical design of the system.
When designing the system, keep in mind the impact the new system will have on existing directory structures.
Installing Directory Server to Support Meta-Directory
Meta-Directory stores its configuration data in a Directory Server directory information tree (DIT). Like other Sun ONE servers and components, Meta-Directory stores its configuration under the suffix o=netscaperoot.
When you install both Sun ONE Directory Server and Sun ONE Meta-Directory, the following components store their configuration data under o=netscaperoot:
To optimize system performance, it is best to use a dedicated Directory Server instance for the Meta-Directory configuration database and a separate dedicated Directory Server instance for the Meta View (which contains the user data that you will be synchronizing with Meta-Directory). Often, the Directory Server instance that hosts the Meta-Directory configuration is called the “config server.”
In addition to using separate Directory Server instances, it is also recommended that you use a dedicated disk partition to store the directory information tree that hosts the Meta View. In production scenarios where you plan to synchronize large amounts of user data in the Meta View, using dedicated systems for the individual Directory Server instances can greatly increase system performance. For more information on setting up your Meta-Directory components and services, refer to "System Topology".
Directory Server Design and Deployment
Detailed information describing the design of LDAP directory services abounds. In addition to the Sun ONE Directory Server documentation, refer to the following references for more information on deploying Directory Server in your organization:
Performing a Site Survey
The first stage of planning your Meta-Directory system is to perform a site survey.
Determining Data Sources and Authoritative Attributes
In the site survey, you will, among other things, determine the following:
The first stage of planning your Meta-Directory system design is to determine the different data sources that you wish to link with Meta-Directory. While this might at first seem obvious, it is a good idea to poll the different organizations in your enterprise to see if they have implemented any directories or databases that contain information that you want to join with the Meta View.
Once you have determined each of the data sources you want to join in the Meta View, you will need to compile a list of the attributes or fields contained in each directory or database. From this list, you can begin the process of defining which attributes or fields you will use from each of the directories or databases in your organization.
Authoritative attributes are the attributes that you will use to populate your Meta View. For example, suppose you have two external data sources that flow attributes to the Meta View: a Human Resources database and an Information Systems (IS) directory DIT. Consider that both data sources contain information on the company’s employees. You will need to decide which attributes (fields) will be the authoritative source for each of the attributes contained in the Meta View. It is likely that both data sources will contain the first and last names of the employees. However, it’s likely that the Human Resources database will contain the most up-to-date and correct names. On the other hand, it’s likely that the IS directory will contain the correct network IDs for each of the users.
Determining the authoritative attributes in your Meta-Directory system is a bit like going through the normalization process when you design a relational database. You must make sure to keep track of the authoritative attributes as you flow them from different data sources and join them in the Meta View. Be especially attentive when you design how modified data flows back out to the various external data repositories. When designing your system, anticipate events that could affect system performance and stability. Minimize problems that could occur if a database goes down and keep enough flexibility in your plan to allow for the changes that could occur when a new data source comes on line. When planning, be sure that modifications made to one data source will not adversely affect the data contained in another.
In the example shown in Table 2-1, several data sources provide the authoritative attribute data.
Determining Data Flow
In addition to determining the data sources and attributes you will use to populate the connector and Meta Views, you will also need to define the flow of the attributes into and out of the different Meta-Directory views. To help you determine this, you should collect all of the following information during the Meta-Directory planning phase:
- The data sources you want to join into your Meta View.
- The user attributes (or database fields) contained in each of the external data repositories.
- The user attributes that need to flow to and from the unified directory (the Meta View). These are often the authoritative attributes from each data source.
- The frequency that attributes need to flow to and from the Meta View.
The following questions should be answered during your site survey:
- The mapping of user attributes from the external data source to the Connector View, and vice-versa. Mapping specifies which attributes from the external data sources flow to the specific attributes in the Connector Views, and vice-versa.
- The direction of attribute flow. Attributes may flow from the external data source to the Connector View, from the Meta View to the Connector View, or both.
- The data ownership. Data ownership defines who owns a data object: the Meta View or the external data source that populates a Connector View. Data objects can be deleted only by the “owner” of the object. While you might be able to temporarily delete an object from a Meta-Directory view own the object, the object will reappear when the data is refreshed.
Data Flow Worksheet
Once you have collected the data needed to determine the data flow, you should create a data flow worksheet. The worksheet will list all the data sources you plan to include in your Meta-Directory system, the attributes in each data source, the attributes that you will flow from each data source and their name mappings, any constructed attributes that you plan to incorporate in your join rules, and so forth.
Determining the Data Design for Your Views
You should determine the design of the directory information trees (DITs) that host your Meta-Directory views during the planning phase. Whether you use one or more Directory Server instances, best performance will be achieved if you dedicate a directory suffix (namespace) for each view that you create. This means that the Meta View will have its own directory suffix, as well as the individual Connector Views that you instantiate in your Meta-Directory system.
If you use individual Directory Server instances for your meta and Connector Views, this recommendation is moot. However, if you are instantiating different views on a single instance of Directory Server, then you will want to pay close attention to how you set up the DIT.
System performance is not influenced much by whether you create a relatively flat tree structure or a more vertical tree. However, a flat tree structure is easier to manipulate as you administer the different branches in your directory tree. In addition, should the need arise, troubleshooting the system will be a bit easier if you have a flat tree structure rather than a vertical tree structure.
For example, consider the following flat tree structures for hosting a Meta View and three Connector Views:
o=metaview o=notesCV4
o=oracleCV1 o=ndsCV5
o=ldapCV2 o=excCV6
o=adCV3Compare this flat tree structure to a tree structure that has more levels, as the one shown below:
ou=metaview, ou=metadir, o=siroe
ou=oracleCV1, ou=connectorviews, ou=metadir, o=siroe
ou=ldapCV2, ou=connectorviews, ou=metadir, o=siroe
ou=adCV3, ou=connectorviews, ou=metadir, o=siroe
ou=notesCV4, ou=connectorviews, ou=metadir, o=siroe
ou=ndsCV5, ou=connectorviews, ou=metadir, o=siroe
ou=excCV6, ou=connectorviews, ou=metadir, o=siroe
When you determining the structure of the directory trees that host your different Meta-Directory views, need to determine the base dn for your Meta View as well as the base dn for each of the Connector Views in your system.
Setting Up the Meta-Directory DIT
You set up the namespaces for your Meta-Directory views through the Directory Server console that hosts the views.
In Directory Server 5.x, do the following:
- From the Directory Server console, select the Configuration tab.
- In the navigation tree in the left pane, right-click the Data icon and choose Create New Root Suffix.
- Enter the dn for the root entries for the directory tree. In the dn specified, make sure there are no leading spaces or spaces between commas and attribute types.
Planning the LDAP Schema Used in Your Views
In the system you design, Directory Server and Meta-Directory will likely exist to support one or more directory-enabled applications. These applications have requirements concerning the data contained in the directory, its format, and how the data is interpreted.
It is always best to use an existing LDAP schema to represent the data you are hosting in your Meta-Directory views. Using an existing schema will allow your data to be more portable to other applications if the need arises. If you do need to customize an existing schema, be sure to consider carefully any custom attributes you will add. Once you move forward with a custom schema, it is difficult to undo your modifications and return to a standard set of LDAP attributes.
Planning System Security
Many Meta-Directory design decisions have security implications. Just as with all applications that deal with a company’s proprietary data, you want to ensure that the Meta-Directory system you deploy provides a secure environment for the data it manipulates. System security for Meta-Directory includes (but is not limited to) physical access to the systems used by Meta-Directory, network access to those systems, binding under commonly used accounts (for example, cn=directory manager), non-Meta-Directory client access to the directory server instances, and so on.
You can use the user authentication and access control mechanisms that Directory Server provides to protect the data that you host in the different Meta-Directory views. Using Directory Server services, you should be able to meet the security and privacy requirements of your system administrators, as well as the users of the applications that access the Meta-Directory views.
Between instances of Directory Server, you can use LDAPS for secure communication. However, it’s important to note that Meta-Directory components do not use LDAPS to communicate between the components and views.
Allocating the Necessary Resources
Allocating resources includes obtaining the hardware and software necessary to support Meta-Directory and the other services you will use with your Meta-Directory system (this might include messaging systems, relational databases, network monitoring tools, and any special programs or utilities). Choosing hardware and software that satisfies design requirements for scaling and performance is crucial to successful deployment. Special attention to fast disk subsystems is key to overall system performance as often Sun ONE Meta-Directory systems suffer from slow overloaded disks.
In addition to hardware and software, consider the human resources that you might need to get your system up and running, and the resources that you will need to maintain the system that you design.
Deploying Your System and Creating a Maintenance Plan
At this point, the design phase is over and it is time that you begin to deploy the system you have planned. The remaining sections in this chapter cover deployment and what you need to plan for once your system is up and running.
As part of the Planning phase, you should create a maintenance plan. While some of the items to consider are discussed on "Maintaining Your Meta-Directory System", you should make sure to properly plan for the growth and expansion of your Meta-Directory system.
Deploying Your Meta-Directory SystemDeploying your Meta-Directory system consists of implementing the system you mapped out in the design phase, as described in the preceding section,"Designing Your Meta-Directory System."
Effective Meta-Directory deployment centers around the use of system resources (your system hardware) to set up the instances of Sun ONE Directory Server needed to host the different Meta-Directory components. When sizing system needs, you must also consider the services provided by any external data repositories and the resources they use.
A careful, systematic approach to your Meta-Directory deployment is strongly advised. By deploying your system incrementally, you can make sure each discrete portion of the system works before moving on to the next part of the deployment phase. For example, with indirect connectors, first prove flow rules work before installing the Join Engine and defining Joins.
The deployment phase of Meta-Directory consists of properly sizing your host systems and effectively locating them on the network. Once your systems are up and running, tuning issues, system maintenance, and scaling the system for growth become the system administrator’s biggest concern.
Data Entry Size and RAM Considerations
The size of the data objects that you will be flowing through your Meta-Directory system plays an important part in the overall design of your system setup. For example, you might be flowing entries that are relatively small (for example, 1024 Kb per entry) or you may be flowing larger entries that contain binaries, such as photos or digital signatures. Entry size plays an important role in the resources that the join engine uses to synchronize entries between views. Throughput of the system depends on having the available system resources to meet the demands of your setup.
In general, you can calculate the size of an entry by exporting it to LDIF and counting the number of characters it takes to represent the entry in text form. Add two characters for each line in the entry to account for the carriage return and line feed.
If your data contains binary data, you can use a different method to calculate the size of entries. Binary data is base64 encoded, meaning that binary data is stored in the Directory Server database as text characters. To calculate the size of an entry, export the entry to LDIF, then check the size of the LDIF file. On a UNIX system, you can use the following command to report the size of an LDIF file:
ls -l LDIF_file
On Windows NT systems, use the DIR command to view file size.
If you are estimating the size of entries for sizing information, you will likely want to average the size of several different entries to get a better idea of your database needs.
RAM Sizing
Not surprisingly, join engine performance is directly related to system resources: the more system memory (RAM), the faster the join engine will be able to synchronize entries between the Meta-Directory views. While you can supplement system RAM with virtual memory (swap space), you must keep in mind that overloading virtual memory will impact the performance of other system processes, such as other applications running on the system.
This section offers some simple examples to help you judge the amount of system RAM you will need in the units you deploy in your system. The database entries used in the examples below are relatively small, around 1024 Kb per entry. While this is a normal size for most data contained in a DIT of object class inetorgPerson, the size of the entries synchronized in the system you are designing might be vastly different, especially if you are flowing binary data.
When sizing the RAM requirements for your system units, consider the following data points observed from system testing:
- When synchronizing 300,000 entries, the join engine used roughly 400 Mb of memory.
Note that in this scenario, Directory Server was actually hosting 600,000 entries: 300,000 in the Meta View and 300,000 in the Connector View. Here, both the Meta View and the Connector View were hosted by a single Directory Server instance located on a single machine, however, the join engine was hosted on a different machine.
- When synchronizing 500,000 entries, the join engine used roughly 665 Mb of memory. In this scenario, there were 1 million data entries hosted by Directory Server, with each entry being 1024 Kb. Both the Meta View and the Connector View were hosted by a single Directory Server instance located on a single machine, however, the join engine was hosted on a different machine.
When you start the join engine, the memory consumption will increase rapidly until it reaches the load where it begins to level off.
The minimum hardware requirement for Meta-Directory is 256 Mb of RAM (or 512 Mb if you are hosting Directory Server on the same machine). However, keep in mind that a larger production system will require more RAM, and perhaps more system units, than the established minimum requirements. Also, it is recommended that you hose the Meta-Directory components (the join engine and any components you have installed) on a different machine that the ones that host your Directory Server instances.
Table 2-2 gives additional examples on the memory used to synchronize entries between a Connector View and a Meta View. In this table, the average entry size is 1024 Kb, or 1 mega-byte of data per entry, per view.
There is a delicate balance that needs to be achieved when you have large amounts of data. For details on the adjustments that you can make to customize your Meta-Directory setup, see Chapter 4, "Meta-Directory Performance Tuning." You should also refer to the Directory Server Deployment guide for further details about hardware requirements.
Directory Sizing for Windows NT
Calculating the number of Windows NT Domain entries that will be synchronized is useful for determining the RAM and disk space required for the directory service (for optimal performance, the Directory Server needs to hold the entire directory data in RAM).
In this case, entry size is not the only consideration. It is equally important to determine how often the data will be changing in each of the supported Windows NT Domains, how many systems the changed data will have to flow to, and the complexity of the data transformation (for example, attribute mapping, DN mapping, and the filter and rule sets that need to be established).
System Topology
The Meta-Directory system topology is the physical layout and locations on the network of the machines that you use to host all the servers and services in your Meta-Directory system. When designing the system topology, you are concerned with the physical implementation of the Meta-Directory components and where you place them on the network. In designing the system, you want to strike a balance between resource use and growth potential.
When you perform a site survey during the Meta-Directory planning phase, you will determine (among other things) the network location of the applications that will provide user information to your Connector Views and, ultimately, the Meta View. Into this network layout, you will add the Meta-Directory components and services (the join engine, the Meta View, connectors, and the associated Connector Views; plus the servers and services that support these components).
When deploying Meta-Directory, you must plan the topology of the system by looking at the types and amounts of data you plan to flow through the Meta-Directory connectors and join engine. Best performance is achieved when you design the topology to ensure that the set of hardware hosting the Meta-Directory components is properly load balanced. Network latency can also have an impact on synchronization times. It is possible for a smaller number of system units to outperform a larger set of system units if the smaller system is sized correctly.
Operating System Configuration
The configurations of the operating systems involved in your deployment is an important factor to consider when designing the Meta-Directory topology. When designing the system, you must take into account the physical memory (RAM) of the systems used, operating system settings (such as the amount of swap space used), and hard disk partitioning and configuration.
The slapd Processes
In Meta-Directory, individual processes are used to manage the main Meta-Directory components:
Depending on your needs, you can host some of these processes on separate machines.
Directory Server Instances
When you are synchronizing large databases, it is best to create separate Sun ONE Directory Server instances for the Connector Views, as well as using a different Directory Server instance for the Meta View. Because of this, it is likely that your Meta-Directory system will contain a number of different instances of Sun ONE Directory Server.
Hosting Connector Views on separate Directory Server instances allows you to balance the load of processing across different servers and machines. When designing the layout of the Directory Sever instances, distinguish between Connector Views that are populated by indirect connectors and the Connector Views that are directly updated by the join engine through the LDAP direct connector. Making this distinction is important when you tune the different Directory Server instances incorporated in your Meta-Directory system—Directory Server instances that host indirect Connector Views should be tuned for write performance while those that host “external” data should be tuned for read performance.
In determining the number and location of the Directory Server instances, keep in mind the Meta-Directory configuration is normally stored in the same Directory Server instance and DIT as the configuration for all other Sun ONE servers used in your Meta-Directory setup.
When creating different Directory Server instances to be used in a single Meta-Directory system, be sure to give the different instances different names as you create them.
Data Servers
Within the Meta-Directory console, each Directory Server instance is configured through the Data Server tab. To view the currently configured Directory Server instances:
Refer to Sun ONE Meta-Directory Administration Guide for instructions on how to add a new Directory Server instance to the Meta-Directory configuration.
Planning the Layout of Your System
There are three basic scenarios for deploying the Sun One Meta-Directory system:
- The ideal, separate Meta-Directory views (either a Meta View or a Connector Views) on dedicated Directory Server instances ideally on separate machines (or Solaris domains).
- The compromise, separate the Meta View from the other Meta-Directory views and connectors. The Meta View and its host system will typically require different performance characteristics than the other meta components. Combine the Connector Views and configuration directory on the same machine in separate Directory instances.
- The consolidated view, is where all Connector Views, Meta View and configuration all stored in one Directory Server instance using separate namespaces.
In a large deployment, it is likely you will employ a combination of these setups.
Locating Meta-directory views on separate machines can increase the performance of the services, but you must balance any gains with the loss incurred through the added network latency. Through system analysis and comparisons with projected loads, you must decide on the setup that is needed for your particular implementation. However, using separate systems for each Directory Server instance on separate hosts is the best choice. If your system resources are limited, consider hosting at least the Meta View on a dedicated system.
Whether more than one Meta-Directory view should be hosted on a single machine is ultimately dependent on the machine’s processing capacity, the memory allocated to each of the system units, availability of disk space, the load on disk controllers and file systems, and the volume of data flowing between Connector Views and the Meta View. System profiling and piloting swill provide the data required to judge the performance of your particular Meta-Directory set up.
Requirements for Hosting the Meta View
The Meta View presents the final “image” of the collected data to end users. Most likely, client applications will retrieve the information stored in the Meta View. Because of this, the Meta View normally has a higher performance requirement than does the typical Connector View. While search indexes must be built for the Meta View to speed up the different types of search operations performed by the applications accessing the Meta View, the same level of searches are not generally used in Connector Views that are dedicated to the Sun ONE Meta-Directory system.
See "Determining Which Indexes to Create" for a list of the different indexes you should create for the different views in your deployment.
Requirements for Hosting Connector Views
The task of any Connector View is to host an LDAP copy of the data contained in an external data source. Connector views should not be used for dual purpose tasks. The primary operations performed on the Connector View are more heavily weighted on add, delete, and modify; searching through Connector Views is not a priority as it is with the Meta View. With a lighter demand for searches, there is a lesser need for indexes in the Connector View. This directly affects your design for system performance: the fewer indexes that need to be updated in the Connector View directory, the better the performance will be.
Reasons for Choosing a Multiple-Instance Configuration
Hosting the Meta View and Connector Views on different Directory Server instances will result in better performance and diminish the likelihood that the failure of one component will greatly affect the performance of another.
A multiple Directory Server configuration makes it easier to track log file data for a component. This also makes it easier to maximize the overall Meta-Directory performance since you will be able to optimize components individually. For more information about Directory Server optimization for Meta-Directory, refer to Chapter 4, "Meta-Directory Performance Tuning."
Using Different Directory Server Instances for the Configuration Database and the Meta View
It is often desirable to use one Directory Server instances for the Meta-Directory configuration database and a different one for the Meta View. By default, Meta-Directory expects these two DITs to exists on the same Directory Server instance. To use separate Directory Server instances for these databases, do the following:
- Install two Sun ONE Directory Server instances: one for the Meta View and one for the configuration database. These instances can be on the same machine or they can be on different machines. Be sure to give the different instances different names.
- Install Sun ONE Meta-Directory. Make sure you reference the Directory Server instance that you want to host the configuration database during the Meta-Directory installation.
- Start the Sun ONE Console from the Directory Server instance that will host the configuration database. Notice that the Sun ONE Console shows several Server Groups.
- Create an instance of the join engine:
- Select the Server Group that represents Meta-Directory.
- Right click the Server Group, then choose Create Instance of the Join Engine.
- When supplying information for the join engine, type in the information for the Directory Server instance that you want to host the join engine and the Meta View (this Directory Server instance will not be in the drop down list of available Data Servers until you specify it when creating an instance of a Meta-Directory component).
Once you have configured several Directory Server instances, you can view them in the Data Servers tab from the Meta-Directory console.
Server Groups
Sun ONE Console uses “server groups” to represents the different sets of Sun ONE servers and services installed in your Meta-Directory system.
When you install different instances of Administration Server, each one is represented as a server group within Sun ONE Console. In your Meta-Directory system, you will have at least one server group displayed in your Sun ONE Console. However, it is possible to have several, depending on your Meta-Directory system setup.
For example, suppose you are using Sun ONE Directory Server 5.x to host your Meta-Directory views. In this case, you will first install Sun ONE Directory Server then Sun ONE Meta-Directory. Sun One Meta-Directory is supported by Sun ONE Administration Server 5.2. Sun ONE Directory Server is managed by Sun ONE AdministrationServer 5.0.
Suppose you then use Netscape Directory Server 4.15 to host a direct Connector View. Netscape Directory Server 4.15 is supported by Netscape Administration Server 4.2.
In this scenario, there are three versions of Administration Server. When you open Sun ONE Console to administer your Meta-Directory system, you will see three server groups, one for each Administration Server.
The figure below shows three server groups. In this setup, the configuration database and the Meta View are hosted by different Directory Server instances, with each instance supported by its own Administration Server. A third server group was created for the Meta-Directory components because a third Administration Server was installed when Meta-Directory was installed. Here, the join engine uses the Directory Server instance in Server Group (2) to host the Meta View.
Figure 2-1 Multiple Server Groups in Sun ONE Console
Different versions of Administration Server must be installed into different file paths—you cannot install Sun ONE software into the same file path if the components use different versions of Administration Server.
Non-Participating Connector Views
In Meta-Directory, it is possible to populate a Connector View, but you may choose to not synchronize the data in that view with the Meta View. For example, you might wish to flow entries from an Active Directory data repository to those contained in an LDAP Connector View for the sake of having the entries available in an LDAP directory. In this configuration, you would not configure the Connector View to be a Participating View of the Meta View, since you would not be flowing entries from this view to the Meta View.
If you find a need to populate a Connector View, but do not wish to make it a Participating View of the Meta View, you should use a Directory Server instance other than the one that hosts the Meta View as the host for the non-participating Connector View.
The reason for this recommendation is that the join engine processes all the change log entries of the Connector Views that participate in the Meta View. If you have a non-participating Connector View sharing a Directory Server instance with the Meta View, the join engine will waste time processing the change log entries generated by the non-Participating View when there is no need to do so.
Locating the Log Files
During the initial phases of deployment, you will likely have verbose logging turned on to help you ensure that things are running smoothly. In these stages of deployment, Meta-Directory can generate copious log files. Setting up your system so that it writes log files to different file systems on separate partitions, can improve performance when system logging is tuned to a high level.
It’s important to note that in addition to increased controller traffic on a partition, increasing the logging level of your Meta-Directory components can result in a performance hit on your synchronization times. A higher level of logging will generate more disk write operations. These extra disk write operations use processor cycles that could otherwise be spent on synchronization. Once your system is running smoothly, you might see a gain in performance when you reduce the level of system logging.
Changelog
Sun ONE Directory Server version 5.x implements two different change log mechanisms. One is used for multi-master replication and the other tracks updates made to the Directory Server database.
- Enabling the Directory Server retro change log (cn=changelog) allows Meta-Directory to participate in change notifications with other components. If it is not enabled, Meta-Directory will not function. If the Directory Server retro change log is not enabled, Meta-Directory will display a dialog that prompts you to enable the change log when you create an instance of a Meta-Directory component.
- RetroChangelog (cn=changelog) Directory Server plug-in must be installed on all Directory Server instances in the Meta-Directory System. This includes the configuration directory too. If the changelog on this configuration instance is not enabled then the Join Engine does not start properly.
Locating the retro change log on dedicated disk partitions can improve overall performance.
Multi-Master Replication
Meta-Directory supports Sun ONE Directory Server 5.x in an MMR configuration. However, failover via MMR is not supported.That is, the Meta-View can reside on one and only one master. The join-engine is configured to point to this master. Then via MMR the master meta-view data can be replicated to another master and, if required, 1-n consumers. When MMR is configured, two kinds of change log must be utilized in order to allow Meta-Directory and MMR to function correctly:
The join engine in Meta-Directory depends on the architecture of the change log facility that was implemented in Netscape Directory Server 4.1x. This is known as the retro change log plug-in within Sun ONE Directory Server 5.x. See the Sun ONE Meta-Directory Administration Guide for details of how to configure this.
To synchronize two master directory servers, the new change log in Sun ONE Directory Server 5.x architecture is also utilized. See the Directory Server Administration Guide on how to configure this.
Cascading Meta Views
The idea of cascading Meta View comes into use, when there is a need to synchronize data from various Sun ONE Meta-Directory installations into one single repository. The diagram below illustrates a typical setup with three Meta-Directory installations.Two subordinate or delegate Meta installations connected to one main Corporate Meta installation.
Each entry in a Meta-Directory installation is owned by a particular view. This ownership information is stored in the attribute mdsEntityOwner. Any modifications made to an entry in any of the views flows to all the views. However, only the owner view has the right to delete an entry. If an entry is deleted from any other view, it should get added back. A non-owner, however, can modify an entry. In a cascading Meta-View setup, such as the one shown above in the figure, deletion of entries even by owner views was not happening in the previous release.
This was reported in the problem (#4636032) "Entries failed to delete from cascade Meta View setting". This problem has now been fixed. Now entries can be deleted from the owner views, across all the Meta-Directory installations. If an entry is deleted by a non-owner, it gets added back.
Steps for setting up Cascading Meta views
This set up consists of 2 Delegation Installations and 1 Corporate Installation. The idea is to synchronize the data from various views to Corporate Meta View. All these 3 installations have separate Sun ONE Directory Server installations.
- Install Sun ONE Meta-Directory on the first machine (refer this installation as delegation 1 installation) and create the Join Engine instance and give the view name as Del1MV.
Figure 2-2 Del1MV
- Create a Connector view 1 (refer this as Del1CV1 or UTC CV1, use any UTC connector or Oracle connector) and populate data into Connector View.
Figure 2-3 Del1CV1
- Create a Connector view 2 (refer this as Del1CV2 or UTC CV2, use any UTC connector or Oracle connector) and populate data into Connector View.
Figure 2-4 Del1CV2
- Add Del1CV1 & Del1CV2 as JoinEngine's Participating Views and enable the Participating Views. Assume contents of Del1MV is present under suffix o=Del1MV.
Figure 2-5 Del1 Predicating View
- Do another Delegation installation of Meta-Directory on the second machine (refer this as Delegation 2 installation) and create the Join Engine instance and give the view name as Del2MV.
Figure 2-6 Del2MV
- Create a Connector view 3 (refer this as Del2CV1 or UTC CV3, use any connector and populate data into Connector View.
Figure 2-7 Del2CV1
- Create a Connector view 3 (refer this as Del2CV2 or UTC CV4, use any connector and populate data into Connector View.
Figure 2-8 Del2CV2
- Add Del2CV1 & Del2CV2 as Join Engine's Participating Views and enable the Participating Views. Assume contents of Del2MV is present under suffix o=Del2MV.
Figure 2-9 Del2 Participating View
- Now, do Corporate Installation of Sun ONE Meta-Directory on the third machine (refer this as Corporate installation) and create the Join Engine instance and give the view name as CorpMV.
Figure 2-10 CorpMV
- Create a Corporate Connector view1 (refer this as CorpCV1) and populate data into Connector View. To create the Connector View, you have two options:
Either (a) Create a LDAP data server from Sun ONE Meta-Directory - Data Servers, and use this URL while creating the CorpCV1.
host name -- host name of the system where delegation mds installation(Del1MV) is running.
port -- port number that stores contents of o=Del1MV.
Bind dn and password for that ids instance. (Assume that data server created as a result of this operation as baba.india.sun.com:389)
or (b) Create a new connector (refer this cv as CorpCV1) on corporate mds installation which actually points to delegation 1 installation's MV (For example, give baba.india.sun.com:389 as data server url. enter o=Del1MV as base view dn).
Figure 2-11 CorpCV1
Either (a) Create a LDAP data server from Sun ONE Meta-Directory - Data Servers, and use this URL while creating the CorpCV2
host name -- host name of the system where delegation mds installation(Del2MV) is running.
port -- port number that stores contents of o=Del2MV.
Bind dn and password for that ids instance. (Assume that data server created as a result of this operation as baba.india.sun.com:389) or (b) Create a new connector (refer this cv as CorpCV2 ) on corporate Sun ONE Meta-Directory installation which actually points to delegation 2 installation's MV (For example, give baba.india.sun.com:389 as data server url. enter o=Del2MV as base view dn).
Figure 2-12 CorpCV2
With this configuration, data flows properly between Del1MV, Del1CV1, Del1CV2, Del2MV, Del2CV1, Del2CV2, CorpMV, CorpCV1 and CorpCV2.
Recommendations
- It is recommended that the view IDs of all the Connector-Views be unique across all the Meta-Directory installations.
- The cascaded view functionality requires that the capabilities to be on in both sides; i.e. the Connector View as well as the Meta View. This is required for the add backs to happen when an entry, owned by a view outside this Meta-Directory installation, is deleted.
Limitations
- Deletion of entries owned by delegation Meta-Views or Corporate Connector-Views results in inconsistent behavior. However, by default, the entries are owned by the data source Connector Views and as such this limitation should not normally affect use.
- All the View IDs of the Meta-Views of all the Meta-Directory installation in the cascaded setup must be unique.
Bringing the System On Line
At this point you have gone through the process of planning and physically deploying your Meta-Directory system. You have finished buying system resources (for the time being, at least) and have installed all the software needed to bring the system on line. To open the Sun ONE Console so you can begin instantiating and configuring the various components in your Meta-Directory system, you will need to start the following servers and services:
At this point, you can use Sun ONE Console to configure your Meta-Directory system, as described in the Sun ONE Meta-Directory Administration Guide. However, before you begin the configuration process, consider piloting and tuning your system before going into production.
Piloting
The best way to determine how Meta-Directory will perform in your particular scenario is to run a pilot test of the proposed deployment and measure the system performance. By piloting the system you have designed, you will gain the information needed to properly gauge the production system you will need to put in place. In a pilot deployment, you set up a smaller scale version of the Meta-Directory system you have designed.
During the pilot run, you not only test system topology configurations, but you can also test the different sets of rules that you will use to synchronize data from the disparate data sources. Running and tuning the rules you set up is a key ingredient to achieving the best possible system performance in your production system.
During the pilot run, test one connector at a time before attempting to merge additional connectors into the mix. The purpose is to test your Meta-Directory design, including the data flow, the application of join rules and DN mapping rules, the attribute flow and filtering of entries, and the functionality of constructed attributes. Ideally, your pilot run will give you the confidence to move the system into the complete system that you will bring on line.
Tuning
By monitoring network status and join engine performance, you can look for bottlenecks and other inefficiencies in system performance. Are the system links to external directories working consistently? Are the links between Directory Server instances that host Connector Views operating optimally?
For more information about tuning Directory Server performance for Meta-Directory, refer to Chapter 4, "Meta-Directory Performance Tuning."
Going Production
When it is time to finally go into production, the best course of action is to begin by synchronizing your main source of data, first into a Connector View, then into the Meta View. From there, flow other external data sources into Connector Views, one at a time. Once you have populated a second Connector View, go ahead and link the entries with the ones contained in the Meta View.
Check your logs to ensure things are flowing correctly by looking for failed join rules. A join rule failure will result in an entry being orphaned; entries are missed when a join rule fails, when a join rule returns more than a single object, or when a join rule encounters bad data. Use the Query Fix-It Tool to search for any unlinked entries and fix them whenever needed. Also, you can search the logs files with using the following commands:
grep -i fail *
grep -i error *If you move one view at a time, running the system and checking the logs after each incremental step, you can more easily identify the location of performance bottlenecks. More importantly, you can address and fix any potential problems as you bring up the system.
Maintaining Your Meta-Directory SystemDuring the system design phase, a maintenance plan should be created for maintaining the system once it has been deployed. Meta-Directory system maintenance includes (but is not limited to) the following items:
Performing Data Backups
It should almost go without saying: Perform regular backups of Meta-Directory data. At a minimum, plans should be in place to back up the Meta View on a regular basis. In a more elaborate system, you should also consider backing up any Connector Views that you have populated.
In Meta-Directory, all the views are hosted by Directory Server (except when you use Oracle to host a Direct Connector View). The Directory Server provides inherent replication capabilities that can be used to fulfill the task of data backup. Refer to Directory Server documentation for details on this feature.
Monitoring the System
How will you assure reliability and availability in your Meta-Directory system? You should design into the maintenance plan the structure needed to insure that system resources are available and the throughput of the system can be maintained at the needed level. Your enterprise may have an existing structure in place that monitors system resources such as the network traffic and individual system loads. If possible, use this existing infrastructure to monitor your Meta-Directory deployment. In other cases, you may need to set up an individual monitoring system as part of the Meta-Directory maintenance plan.
Monitoring your system resources also encompasses the task of trimming old log files to free up disk space. Provide a schedule for monitoring the log file directories and a procedure for deleting the old, outdated files.
Network security might also be a concern. The maintenance plan should have an eye toward future needs and include a section on how the system security will be kept at required levels.
Planning for System Expansion
In the course of monitoring system loads, you might find that faster performance is needed. Plans should be in place to add more resources in case you need to scale the system.
When does CPU utilization reach the point where additional processing power is required? Should one consider installing additional/faster performing disks to offset performance degradation? When does network bandwidth utilization reach the point where additional bandwidth is required? Through the analysis of the system load and performance, you should be able to gauge the resources that are necessary to support the needs of your system.
While it is the job of the system administrator to make sure the system is available, it is the job of the system planners to ensure that the proper resources are on line to fulfill the needs of the system they design. For more information on the requirements of Directory Server, refer to the Sun ONE Directory Server Deployment Guide.
