4 Using a Source Control Management System for Repository Development

This chapter describes how to integrate the Oracle BI Administration Tool with third-party source control management systems for Oracle BI repository development.

The Administration Tool achieves this integration through the ability to save repository metadata as a set of XML documents in MDS XML format rather than as a single binary repository file (RPD). Using this integration, you can configure the Administration Tool to work with your own source control management system and save your repository output as MDS XML.

This chapter contains the following topics:

• About Using a Source Control Management System with the Administration Tool

• Setting Up Your System for Repository Development Under Source Control Management

• Using Source Control Management in Day to Day Repository Development

• Using Source Control Management with MUD

About Using a Source Control Management System with the Administration Tool

You can choose to integrate the Administration Tool with a third-party source control management system, such as Subversion or Rational ClearCase, during your repository development process. This feature is centered around the following integration points:

• Converting your binary RPD file to a set of MDS XML documents. Rather than using a single large binary repository file, you can save your repository in MDS XML format. In this format, each repository object, such as a connection pool, physical table, or business model, is represented in its own XML file. The set of XML files that make up your repository can then be managed in your source control management system.

• Setting up a Source Control Management (SCM) configuration file. You can use the SCM Configuration Editor in the Administration Tool to specify commands specific to your SCM system, such as Add File, Delete, and Check Out, as well as environment variables required by your SCM system.

• Designating your repository as under source control. The first time you open your MDS XML repository in the Administration Tool, you are prompted to specify whether the repository is a standalone MDS XML repository, or whether it is under source control. Choose Use Source Control to enable SCM integration for this repository in the Administration Tool.

About MDS XML

MDS XML format is typically used for repositories under source control. MDS XML is not the same XML format that was used in previous releases to represent the Oracle BI repository in XML format. The previous Oracle BI Server XML schema, based on the xudml1.xsd XML schema file in ORACLE_HOME/bifoundation/server/bin, represents the Oracle BI repository in one large XML file. MDS XML, in contrast, represents the Oracle BI repository across a set of XML files, rather than in a single file.

For example, each repository connection pool is stored in its own file, with an XML representation like the following:

<?xml version="1.0" encoding="UTF-8" ?>
<ConnectionPool mdsid="m80ca62c5-0bd5-0000-714b-e31d00000000"
name="SampleApp_Lite_Xml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.oracle.com/obis/repository/"
password="94F9321C85340FC48E4D9093AA941FF28844074B88D5AA6364E4815DEED7F9B
8792EF452219C2155DB68F61EE1555B4FA886F77E060E2E17F45AD8D18CAB2E4D3EFA15B75E
30D8B4BFA8C7B2D70552BD" timeout="4294967295" maxConnDiff="10" maxConn="10"
dataSource="VALUEOF(BI_EE_HOME)/sample/SampleAppFiles/Data" type="Default"
reqQualifedTableName="false" isSharedLogin="false"
isConcurrentQueriesInConnection="false" isCloseAfterEveryRequest="true"
xmlRefreshInterval="2147483647" outputType="xml" ignoreFirstLine="false"
bulkInsertBufferSize="0" transactionBoundary="0" xmlaUseSession="false"
multiThreaded="false" supportParams="false" isSiebelJDBSecured="false"
databaseRef="/oracle/bi/server/base/Database/Sample App Lite Data_80ca62c4
-0bcf-0000-714b-e31d00000000.xml#m80ca62c4-0bcf-0000-714b-e31d00000000"
>
<Description>
<![CDATA[ 
SampleAppLite connection pool to XML datasource. This connection pool points the 
database to the location where physical XML files are stored. The location uses 
the value of an RPD variable : BI_EE_HOME.
This variable needs to be correctly set in order for the server to connect to the 
files.
]]>
</Description>
</ConnectionPool>

The SampleAppLite repository will generate MDS XML files in a structure like the following:

Note that there is not a one-to-one relationship between repository objects in the Administration Tool and the set of files produced as XML output. For example, physical columns appear as independent objects in the Administration Tool, but in MDS XML they are considered part of the Physical Table object.

See "About the Oracle BI Server MDS XML API" in Oracle Fusion Middleware XML Schema Reference for Oracle Business Intelligence Enterprise Edition for full information about the MDS XML schema representation of repository objects.

Setting Up Your System for Repository Development Under Source Control Management

This section explains how to set up an SCM configuration file with commands specific to your SCM system, as well as how to generate an MDS XML repository and check it into your SCM system.

This section contains the following topics:

• Creating an SCM Configuration File

• Creating an MDS XML Repository and Checking In Files to the SCM System

Creating an SCM Configuration File

To integrate the Administration Tool with your source control management system, you must create an XML configuration file based on your specific SCM system. The configuration file contains the SCM system commands for adding, deleting, checking out, and renaming files. The Administration Tool will issue these commands to the SCM system when repository objects are created or updated, resulting in corresponding new or changed MDS XML files.

Note:

The Administration Tool does not commit the changes to the SCM system. The repository developer must always check the files into the SCM system directly. This way, the repository developer can view any conflicts or make merge decisions in the SCM environment rather than the Administration Tool environment.

To create a configuration file for your SCM system:

1. Open the Administration Tool and select Tools, then select Options.

2. Select the Source Control tab.

3. Click New to create a new configuration file. The Specify new configuration file window is displayed.

   Note: This procedure assumes that you do not have an open MDS XML repository in the Administration Tool. If you create or edit an SCM configuration file while an MDS XML repository is open, you must ensure that Use Source Control is selected to enable the New or Edit buttons.

4. Provide a file name and click Save. The file must have the XML file extension.

   The default location for SCM configuration files is ORACLE_INSTANCE/config/OracleBIServerComponent/coreapplication_obisn. Although templates are also available in this location, do not select a template file during this step. Instead, you can load a template in the next step.

   Note:

   The SCM configuration template files are called scm-conf-ade.template.xml and scm-conf-svn.template.xml. In addition to being available in the ORACLE_INSTANCE location indicated, they are also available on the Oracle Technology Network (OTN) at:

   http://www.oracle.com/technetwork/middleware/bi-foundation/downloads/obieescmconfigfiles-1568980.zip

5. To load a template configuration file, click Load in the SCM Configuration Editor. Then, select a template file (for example, scm-conf-svn.template) and click Open.

   Unless it is your intention to modify the configuration file template itself, ensure that Edit in Configuration Editor is not selected. If you select this option, the file name displayed in the Configuration File field in the SCM Configuration Editor changes from the file name you provided in the proceeding step to the template file name, and changes are saved by default to the template file.

6. In the SCM Configuration Editor, provide an optional description, then enter or edit commands for your system in the Commands subtab. For longer commands, click the ellipsis button to enter commands in the Command Editor window.

   Use the ${file}, ${filelist}, ${from}, and ${to} tokens to define the commands. You can also use the List File option in conjunction with the ${filelist} command to set the behavior. The tokens can be used as follows:

   • ${file} specifies that a command must be run sequentially, one file at a time. ${file} is required for the Add Folder and Add File commands.

   • The behavior of ${filelist} varies, depending on whether List File is selected:

     • ${filelist} without List File selected causes the Administration Tool to group as many files as possible for the given command (such as Pre-Delete, Delete, or Check-out), staying under the 32k character limit for launching a process. Execution is repeated until all files have been processed.

     • ${filelist} with List File selected instructs the Administration Tool to create a temporary file that holds the file list. The file list format is one file name for each line, which is a typical format among SCM vendors. List File is valid for Pre-Delete, Delete or Check-out commands. It results in much faster operations and should always be selected for SCM systems that support it.

     You can use either ${file} or ${filelist} for Pre-Delete, Delete, and Checkout. List File only works in conjunction with ${filelist}.

   • ${from} and ${to} are used to specify the original file name and new file name in Rename commands.

     Not all SCM systems support file rename operations natively. If this is the case, leave the Rename field blank rather than attempting to construct a rename operation by concatenating different commands. The Administration Tool will do this for you with greater efficiency.

   Note:

   Some SCM systems do not include commands for working with folders. If this is the case, leave Add Folder blank. The Administration Tool always creates folders for you when needed.

   Even if your SCM system does include folder management commands, the Administration Tool does not remove folders. You must remove folders directly in the SCM system if necessary.

   Figure 4-1 shows the Commands tab of the SCM Configuration Editor.

   Figure 4-1 Commands Tab of SCM Configuration Editor

   
   

7. Select the Environment Variables subtab, and then specify environment variables required by your SCM system.

   You can paste environment variables directly from your operating system variable list, paste environment variables from the clipboard, or manually add environment variables. Table 4-1 describes the options available for managing environment variables in the Environment Variables subtab.

   Table 4-1 Options for Managing Environment Variables in the SCM Configuration Editor

   Option	Description
   Paste environment variables	Enables you to paste environment variables directly from your operating system. Click this option to open a window where you can enter filter criteria, then click OK. Enter * in the filter window to paste all environment variables.
   Paste from clipboard	Enables you to paste text from the clipboard. To use this option, copy text in the following format: variable_name1=variable_value1 variable_name2=variable_value2 Each environment variable must be on its own line. Note: The standard Windows command set provides output in this format for environment variables.
   Add	Adds a row to the table so that you can manually enter environment variables. Provide the variable name in the Variable column, and its definition in the Value column. Leading and trailing white space is trimmed. You can use %VAR% in the variable definition to reference a previously defined variable.
   Delete	Deletes the given row in the environment variables table.

   
   

   Note that you should not store security-sensitive environment variables in the configuration file. If security-sensitive variables are required by your SCM system, to avoid the security risk, you can launch the Administration Tool from a DOS window with any security-sensitive variables already set.

8. Click Test in the Environment Variables subtab to open the Test SCM Configuration window. Then, enter a command and click Execute to test a particular command. If the environment is correct, the correct output should appear after executing the command.

9. Select the Post-save comment subtab to enter text that will appear after changes are saved in the Administration Tool. This capability is a way to remind developers to check files directly into their SCM system after saving. For example, a post-save comment might be:

   Files have been synchronized to source control. Remember to check in changes after testing.
   

10. Click OK to save the configuration file, or click Save As to save a copy if you loaded and modified a template configuration file.

Creating an MDS XML Repository and Checking In Files to the SCM System

To integrate with an SCM system, you must convert your Oracle BI repository to MDS XML format. Use one of the following options to create an MDS XML repository and check it into your source control system:

• Saving an Existing Repository File in MDS XML Format

• Creating a New Repository in MDS XML Format

• Linking to Source Control Files to Convert Your Repository (Small Repositories Only)

Saving an Existing Repository File in MDS XML Format

If you have an existing repository file, follow these steps to convert it to MDS XML.

To save an existing repository file in MDS XML format:

1. Open your existing repository file (RPD) in the Administration Tool in offline mode.

2. Select File, then select Save As, then select MDS XML Documents.

3. Select a root location for your MDS XML repository files, and then click OK.

4. Perform the necessary steps in your source control management system to add and check in the files.

   Use the specialized commands for bulk file import, available for most SCM systems. These commands are optimized to deliver entire trees of files to source control in a very efficient way. For example, in Subversion, use the following command:

   svn import module_name -m "Initial import"
   

The steps described in this section are the recommended method for initial import.

Note:

You can also use the biserverxmlgen utility with the -M and -D options to generate MDS XML from an existing RPD. See "Generating MDS XML from an Existing RPD Using a Command-Line Utility" in Oracle Fusion Middleware XML Schema Reference for Oracle Business Intelligence Enterprise Edition for more information.

Creating a New Repository in MDS XML Format

Use the following steps to create a new repository in MDS XML format.

To create a new repository in MDS XML format:

1. Open the Administration Tool and select File, then select New Repository to open the Create New Repository Wizard.

2. Select the MDS XML Documents option in the wizard. Complete the other wizard steps.

3. Perform the necessary steps in your source control management system to add and check in the files. For large repositories, use the specialized commands for bulk file import for your SCM system.

Note:

Do not create a new MDS XML-format repository, add objects, and then select Link to Source Control. This method will not work, and no SCM commands will be generated.

Linking to Source Control Files to Convert Your Repository (Small Repositories Only)

For very small repositories, you can use the Link to Source Control files method to convert a binary RPD file to DMS XML format.

To link to source control files to convert your repository:

1. Ensure that you have an SCM configuration file defined. See "Creating an SCM Configuration File" for more information.

2. Create an empty root folder for the MDS XML repository.

3. Open your existing RPD file in the Administration Tool in offline mode.

4. Select File, then select Source Control, then select Link to Source Control Files.

5. Select the root folder you created, and the appropriate SCM configuration file.

   Note: If you need to change the configuration file later, you can go Tools > Options > Source Control and click Edit to change the configuration file.

6. Click Save. An MDS XML repository is created, and the necessary add file operations are performed in your source control system.

7. Commit the changes in your SCM system.

Note:

Using the Link to Source Control Files method to initially import your repository is only recommended for very small repositories. This method is too slow for large repositories (tens of thousands of files) because the Administration Tool imports the files one at a time using the standard "add file" command, rather than using specialized commands for bulk file import.

Note also that the repeated invocation of the "add file" command might increase the chances of transient errors. If these occur, you might need to restart the process a few times before all files are successfully imported to source control.

Using Source Control Management in Day to Day Repository Development

This section describes typical scenarios that occur during day to day repository development.

This section contains the following topics:

• Updating, Saving, and Checking In Changes for Repositories Under Source Control

• Handling Errors

• Testing Repositories Under Source Control

• Viewing the Source Control Log

Updating, Saving, and Checking In Changes for Repositories Under Source Control

After your MDS XML repository is set up under source control, follow these steps to update, save, and check in changes to your repository.

To update, save, and check in changes to your repository:

1. Ensure that you have a local copy of your working MDS XML repository files that are under source control by issuing the appropriate commands in your SCM system. For example, for Subversion, you can issue the command svn info as shown in the following example text:

   C:\myProj\repos>svn info
   Path: .
   Working Copy Root Path: C:\myProj\repos
   URL: file:///C:/SVN/myProj/trunk/sample1
   Repository Root: file:///C:/SVN/myProj
   Repository UUID: 6b995c92-3ec0-fa4b-9d58-c98e54f41792
   Revision: 3
   Node Kind: directory
   Schedule: normal
   Last Changed Author: joe_user
   Last Changed Rev: 2
   Last Changed Date: 2011-11-19 15:20:42 -0600 (Sat, 19 Nov 2011)
   

2. Open your MDS XML repository in the Administration Tool in offline mode. To do this, select File, then select Open, then select MDS XML.

3. Select the root folder location for your MDS XML files and click OK.

4. If this is the first time you have opened this MDS XML repository in the Administration Tool, you are prompted to specify whether this repository is a standalone MDS XML repository, or whether it is under source control. Select Use Source Control and click OK.

   This choice is saved for this repository. To view the status of this repository at any time, select Tools, then select Options, then select the Source Control tab.

5. After you make changes to your repository, select File, then select Save, or click Save on the toolbar. The Administration Tool displays a list of changes. For example:

   
   

6. Click Yes. The Administration Tool runs the necessary commands in the SCM system.

   After you accept the changes, you cannot cancel. Canceling in the middle would leave an inconsistent repository. You must wait for the SCM commands to be executed.

   Note also that when the Administration Tool issues the SCM commands, they may be rearranged into the most optimal order.

7. Check in the changes directly in your SCM system.

Handling Errors

Sometimes, errors might occur when the Administration Tool delivers changes to the SCM system, such as an expired label or network problem. If errors occur, perform the following steps.

To handle errors:

1. In the Administration Tool, select File, then select Save As to save the repository to a temporary location in RPD format or MDS XML format. Close the Administration Tool.

   Note:

   Saving to a binary RPD is the simplest option for transient problems like network errors, where you just need to try again later. Saving as MDS XML is required when some sort of work is required to fix the problem, such as merging conflicting changes.

2. Take action to resolve the issue. For example, refresh an expired label or test and view a failed network connection.

   In the case of an expired label, you also need to merge the contents of the refreshed label with the temporary saved MDS XML repository. Use a third-party merge tool to do this.

   For detailed information about the MDS XML representation of repository objects so that you can successfully make merge decisions, see Oracle Fusion Middleware XML Schema Reference for Oracle Business Intelligence Enterprise Edition.

3. Open the saved repository in the Administration Tool.

4. Select File, then select Source Control, then select Link to Source Control.

5. Click Save to save changes from the saved repository into the MDS XML files under source control.

Steps 4 and 5 of this procedure cause the Administration Tool to keep memory objects loaded from the saved RPD file or MDS XML files, but to then consider them to belong to the source control MDS XML repository instead. When you click Save, the Administration Tool saves the memory objects to the source control repository.

Testing Repositories Under Source Control

During the course of repository development, you will need to perform testing in online mode to validate your repository. You can only load an Oracle BI repository in RPD format into the Oracle BI Server to make it available for queries. Because of this, you must save your development MDS XML repository in RPD format from time to time when you want to perform online testing. To do this, open your MDS XML repository in offline mode and select Save As, then select Repository.

See "Making the Repository Available for Queries" for more information about uploading repositories.

Viewing the Source Control Log

The Source Control Log window shows the commands that the Administration Tool issues to your SCM system. It also shows any post-save text you specified in the Post-save comment tab of the SCM Configuration Editor.

By default, the Source Control Log window appears when SCM commands are being executed. Alternatively, you can select File, then select Source Control, then select View Logs to see the Source Control Log window.

Figure 4-2 shows the Source Control Log window.

Figure 4-2 Source Control Log Window

Description of "Figure 4-2 Source Control Log Window"

You can choose the following options for this dialog:

• Close when commands finish: Causes the log window to close automatically when commands are complete, unless errors occur.

• Only show dialog when errors occur: Hides the window during SCM command execution unless errors occur. By default, the Source Control Log appears automatically when SCM commands are being executed unless this option is selected.

The text displayed in the Source Control Log is persistent until you close the repository. This means that all SCM command output is available for view, regardless of whether the dialog is open during individual operations.

The Source Control Log does have a 32K character limit. When the window buffer becomes full, then the oldest commands are removed from the Source Control Log display to make room for the latest command output. To see the full output, go to the Administration Tool log at:

ORACLE_INSTANCE/diagnostics/logs/OracleBIServerComponent/coreapplication_obisn/
user_name_NQSAdminTool.log

Note:

While SCM commands are being executed, the Close button is disabled until the SCM commands have finished or have stopped with an error (unless Only show dialog when errors occur has been selected).

Using Source Control Management with MUD

You can use source control management with your multiuser development environment. This section provides a general outline of how you might use both processes together.

For example, if you have an existing repository under multiuser development and you want to begin using source control management, you might follow the steps described in the following subsections:

• Putting the MUD Master Repository and MUD Log File Under Source Control

• Checking In New Versions of the MUD Master and MUD Log File to Source Control

Putting the MUD Master Repository and MUD Log File Under Source Control

Use the following steps to put the MUD master repository and MUD log file under source control.

To put the MUD master repository and MUD log file under source control:

1. Convert your master MUD RPD to a set of MDS XML files on the file system. See "Saving an Existing Repository File in MDS XML Format" for information on how to do this.

2. Use the mhlconverter command-line utility to convert your MUD log file (*.mhl) to an XML file. To do this, follow these steps:

   1. Run bi-init to launch a command prompt that is properly initialized. See "Running bi-init to Launch a Shell Window Initialized to Your Oracle Instance" for more information.

   2. At the command prompt, type mhlconverter with the input MHL file name and path, and the output XML file name and path. For example:

      mhlconverter -I C:\MUD\mud_repository.mhl -O C:\MUD\mud_repository.xml
      

3. Check the MDS XML files and XML-format MUD log file into your SCM system.

Checking In New Versions of the MUD Master and MUD Log File to Source Control

After creating and checking in the initial version of the master MUD repository, you will need to check in updated versions of the MUD master repository on an ongoing basis. This section describes two different ways to perform this task.

To check in new versions of the MUD master and MUD log file to source control:

• Manually Checking In the Updated MUD Master Repository and Log File

• Using a Script to Check In the Updated MUD Master Repository and Log File

Manually Checking In the Updated MUD Master Repository and Log File

Follow these steps to manually check in changes to the master RPD and log file that have occurred as part of the multiuser development process.

To manually check in the updated MUD master repository and log file:

1. Open the latest copy of the master RPD in the Administration Tool.

2. Create or select the appropriate SCM configuration file. See "Creating an SCM Configuration File" for more information.

3. Select File, then select Source Control, then select Link to Source Control. Select the directory that contains the MDS XML version of the master MUD repository.

   Using Link to Source Control is not recommended for large repositories and might cause time-outs. Consider using the automated check-in method described in "Using a Script to Check In the Updated MUD Master Repository and Log File" if you have a large repository.

4. Click Save to save changes from the master MUD repository into the MDS XML files under source control. The Administration Tool determines which files to add, check out, modify, and delete and issues the commands to your SCM system.

5. Close the Administration Tool.

6. Follow these steps to update the MUD log file:

   1. In your SCM system, check out the XML-format MUD log file.

   2. Use the mhlconverter utility to overwrite the XML-format MUD log file with the latest changes from the .mhl version.

   3. Check in the latest XML-format MUD log file to your SCM system.

7. Check all changes into your SCM system.

It is recommended that you perform the steps in this section regularly to avoid having too many changes in a single transaction.

Using a Script to Check In the Updated MUD Master Repository and Log File

As an alternative to manually checking in changes, you can create a script to perform the check-in tasks and then schedule it to run at regular intervals.

To use a script to check in the updated MUD master repository and log file:

1. Identify the latest copy of the master RPD that you want to check into your SCM system.

2. Identify the last version of the master RPD that was checked into the SCM system. You can review the latest XML-format MUD log file under source control to determine this version.

   Note:

   If you do not have the last checked-in version of the master repository in RPD format, you can use the biserverxmlexec utility with the -D option to read the latest MDS XML files checked into source control and re-create an RPD version.

3. Use the comparerpd utility with the -M option to compare the latest copy of the master RPD (the modified version) with the version that was last checked in (the original version). An MDS XML format diff is generated. See "Comparing Repositories Using comparerpd" for more information.

4. Create a script that does the following:

   1. Reads the MDS XML diff directory to identify which files are present.

   2. Issues commands in source control to check out the identified files or add new files.

   3. Copies the latest version of the files from the MDS XML diff directory to the source control directory.

   4. Reads the oracle\bi\server\base\DeletedFiles.txt file inside the MDS XML diff directory to determine which files to delete.

   5. Issues commands in source control to delete the appropriate files.

   6. Checks out the MDS XML-format MUD log file, runs the mhlconverter utility to convert the latest MHL-format log file to XML format, overwrites the existing MDS XML-format MUD log file with the new one, and checks it in.

   7. Performs all necessary check-in steps in the SCM system.