4 Using a Source Control Management System for Repository Development

You can integrate the Administration Tool with third-party source control management systems for Oracle BI repository development.

The Administration Tool achieves this integration through the ability to save repository metadata as a set of XML documents in MDS XML format rather than as a single binary repository file (RPD). Using this integration, you can configure the Administration Tool to work with your own source control management system and save your repository output as MDS XML.

This chapter contains the following topics:

About Using a Source Control Management System with the Administration Tool

You can integrate Oracle Administration Tool with a third-party source control management systems such as Subversion, Rational ClearCase, or Git during your repository development process.

In the Administration Tool:

Convert your binary RPD file to a set of MDS XML documents.

You can save your repository in MDS XML format rather than using a single large binary repository file. In the MDS XML format, each repository object such as a connection pool, physical table, or business model is represented in its own XML file. You can manage the set of XML files that make up your repository in your source control management system.
Set up a Source Control Management (SCM) configuration file.

Use the SCM Configuration Editor in the Administration Tool to specify commands specific to your SCM system such as add file, delete, and check out and specify environment variables required by your SCM system.
Designate your repository as under source control.

The first time you open your MDS XML repository in the Administration Tool, you are prompted to specify whether the repository is a standalone MDS XML repository, or whether it is under source control. Choose Use Source Control to enable SCM integration for this repository in the Administration Tool.

About MDS XML

MDS XML format is used for repositories under source control.

MDS XML represents the Oracle BI Repository across a set of XML files rather than in a single file.

Each repository connection pool is stored in its own file, with an XML representation like the following:

<?xml version="1.0" encoding="UTF-8" ?>
<ConnectionPool mdsid="m80ca62c5-0bd5-0000-714b-e31d00000000"
name="SampleApp_Lite_Xml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.oracle.com/obis/repository/"
password="94F9321C85340FC48E4D9093AA941FF28844074B88D5AA6364E4815DEED7F9B
8792EF452219C2155DB68F61EE1555B4FA886F77E060E2E17F45AD8D18CAB2E4D3EFA15B75E
30D8B4BFA8C7B2D70552BD" timeout="4294967295" maxConnDiff="10" maxConn="10"
dataSource="VALUEOF(BI_EE_HOME)/sample/SampleAppFiles/Data" type="Default"
reqQualifedTableName="false" isSharedLogin="false"
isConcurrentQueriesInConnection="false" isCloseAfterEveryRequest="true"
xmlRefreshInterval="2147483647" outputType="xml" ignoreFirstLine="false"
bulkInsertBufferSize="0" transactionBoundary="0" xmlaUseSession="false"
multiThreaded="false" supportParams="false" isSiebelJDBSecured="false"
databaseRef="/oracle/bi/server/base/Database/Sample App Lite Data_80ca62c4
-0bcf-0000-714b-e31d00000000.xml#m80ca62c4-0bcf-0000-714b-e31d00000000"
>
<Description>
<![CDATA[ 
SampleAppLite connection pool to XML datasource. This connection pool points the 
database to the location where physical XML files are stored. The location uses 
the value of an RPD variable : BI_EE_HOME.
This variable needs to be correctly set in order for the server to connect to the 
files.
]]>
</Description>
</ConnectionPool>

The SampleAppLite repository generates MDS XML files in a structure like the following:

Description of GUID-612FBCF6-0F1C-4BA5-847E-6D4E63DDB0F1-default.gif follows

Description of the illustration GUID-612FBCF6-0F1C-4BA5-847E-6D4E63DDB0F1-default.gif

Note:

There is no one-to-one relationship between repository objects in the Administration Tool and the set of files produced as XML output. For example, physical columns appear as independent objects in the Administration Tool, but in MDS XML they are considered part of the Physical Table object.

See “About the Oracle BI Server MDS XML API” in XML Schema Reference for Oracle Business Intelligence Enterprise Edition for full information about the MDS XML schema representation of repository objects.

Setting Up Your System for Repository Development Under Source Control Management

To set up your system for repository development under source control management, you must set up an SCM configuration file with commands specific to your SCM system, and generate an MDS XML repository and check it into your SCM system.

This section contains the following topics:

Creating an SCM Configuration File

To integrate the Oracle BI Administration Tool with your source control management system (SCM), you must create an XML configuration file based on your specific SCM system.

The configuration file contains the SCM system commands for adding, deleting, checking out, and renaming files. The Administration Tool issues these commands to the SCM system when repository objects are created or updated, resulting in corresponding new or changed MDS XML files.

Note:

The Oracle BI Administration Tool does not commit the changes to the SCM system. The repository developer must always check the files into the SCM system directly. The separate check-in in the SCM system facilitates viewing any conflicts or implementing merge decisions in the SCM environment rather than the Administration Tool environment.

If you create or edit an SCM configuration file while an MDS XML repository is open, you must ensure that Use Source Control is selected to enable the New or Edit buttons.

The default location for SCM configuration files is ORACLE_INSTANCE/config/OracleBIServerComponent/coreapplication_obisn. Although templates are also available in this location, do not select a template file during this step. Instead, you can load a template in the next step.

The SCM configuration template files are called scm-conf-ade.template.xml and scm-conf-svn.template.xml. In addition to being available in the ORACLE_INSTANCE location indicated, they are also available on the Oracle Technology Network (OTN) at:

http://www.oracle.com/technetwork/middleware/bi-foundation/downloads/obieescmconfigfiles-1568980.zip

Unless it is your intention to modify the configuration file template itself, ensure that Edit in Configuration Editor is not selected. If you select this option, the file name displayed in the Configuration File field in the SCM Configuration Editor changes from the file name you provided in the proceeding step to the template file name, and changes are saved by default to the template file.

You should not store security-sensitive environment variables in the configuration file. If security-sensitive variables are required by your SCM system, to avoid the security risk, you can launch the Administration Tool from Windows Command Prompt with any security-sensitive variables already set.

Open the Administration Tool and select Tools, then select Options.
Select the Source Control tab.
Click New to create a new configuration file. The Specify new configuration file window is displayed.
Provide a file name using the XML file extension, and click Save.
Click Load in the SCM Configuration Editor. Then,
Select a template file, and click Open.
In the SCM Configuration Editor, provide an optional description, then enter or edit commands for your system in the Commands tab.
For longer commands, click the ellipsis button to enter commands in the Command Editor window.

Use the ${file}, ${filelist}, ${from}, and ${to} tokens to define the commands. You can also use the List File option in conjunction with the ${filelist} command to set the behavior. The tokens can be used as follows:
- ${file} specifies that a command must be run sequentially, one file at a time. ${file} is required for the Add Folder and Add File commands.
- The behavior of ${filelist} varies, depending on whether List File is selected:
  - ${filelist} without List File selected causes the Administration Tool to group as many files as possible for the given command such as Pre-Delete, Delete, or Check-out, staying under the 32k character limit for launching a process. Execution is repeated until all files have been processed.
  - ${filelist} with List File selected instructs the Administration Tool to create a temporary file that holds the file list. The file list format is one file name for each line, which is a typical format among SCM vendors. List File is valid for Pre-Delete, Delete or Check-out commands. It results in much faster operations and should always be selected for SCM systems that support it.
  You can use${file} or ${filelist} for Pre-Delete, Delete, and Checkout. List File only works in conjunction with ${filelist}.
- ${from} and ${to} are used to specify the original file name and new file name in Rename commands.
  
  Not all SCM systems support file rename operations natively. If this is the case, leave the Rename field blank rather than attempting to construct a rename operation by concatenating different commands. The Administration Tool performs the rename operation.
Note:

Some SCM systems do not include commands for working with folders. If this is the case, leave Add Folder blank. The Administration Tool always creates folders for you when needed.

Even if your SCM system does include folder management commands, the Administration Tool does not remove folders. You must remove folders directly in the SCM system if necessary.
Select the Environment Variables tab, and then specify environment variables required by your SCM system.
Click Test in the Environment Variables tab to open the Test SCM Configuration window. Then, enter a command and click Execute to test a particular command. If the environment is correct, the correct output should appear after executing the command.
Select the Post-save comment tab to enter text that appears after changes are saved in the Administration Tool. You can use the Post-save comment to remind developers to check-in there changes.
Click OK to save the configuration file, or click Save As to save a copy if you loaded and modified a template configuration file.

Creating an MDS XML Repository and Checking In Files to the SCM System

To integrate with an SCM system, you must convert your Oracle BI repository to MDS XML format.

Use one of the following options to create an MDS XML repository and check it into your source control system:

Saving an Existing Repository File in MDS XML Format

If you have an existing repository file, use these steps for initial import to convert it to MDS XML.

Open your existing repository file (RPD) in the Administration Tool in offline mode.
Select File, then select Save As, then select MDS XML Documents.
Select a root location for your MDS XML repository files, and then click OK.
Perform the steps in your source control management system to add and check in the files.

Use the specialized commands for bulk file import, available for most SCM systems. These commands are optimized to deliver entire trees of files to source control in a very efficient way. For example, in Subversion, use the following command:

svn import module_name -m "Initial import"

Note:

You can also use the biserverxmlgen utility with the -M and -D options to generate MDS XML from an existing RPD. See “Generating MDS XML from an Existing RPD Using a Command-Line Utility” in the XML Schema Reference for Oracle Business Intelligence Enterprise Edition.

Creating a New Repository in MDS XML Format

Use these steps to create a new repository in MDS XML format.

Open the Administration Tool and select File, then select New Repository to open the Create New Repository Wizard.
Select the MDS XML Documents option in the wizard. Complete the other wizard steps.
Perform the necessary steps in your source control management system to add and check in the files. For large repositories, use the specialized commands for bulk file import for your SCM system.

Note:

Do not create a new MDS XML-format repository, add objects, and then select Link to Source Control. This method does not work, and SCM commands are not generated.

Linking to Source Control Files to Convert Your Repository (Small Repositories Only)

For very small repositories, you can use the Link to Source Control files method to convert a binary RPD file to MDS XML format.

See Creating an SCM Configuration File.

Ensure that you have an SCM configuration file defined.
Create an empty root folder for the MDS XML repository.
Open your existing RPD file in the Administration Tool in offline mode.
Select File, then select Source Control, then select Link to Source Control Files.
Select the root folder you created, and the appropriate SCM configuration file.

Note:

If you need to change the configuration file later, select Tools, select Options, select Source Control, and then click Edit to change the configuration file.
Click Save. An MDS XML repository is created, and the necessary add file operations are performed in your source control system.
Commit the changes in your SCM system.

Note:

Using the Link to Source Control Files method to initially import your repository is only recommended for very small repositories. This method is too slow for large repositories, tens of thousands of files, because the Administration Tool imports the files one at a time using the standard add file command, rather than using specialized commands for bulk file import.

The repeated invocation of the add file command might increase the chances of transient errors. If these occur, you might need to restart the process a few times before all files are successfully imported to source control.

Using Source Control Management in Day to Day Repository Development

These topics describes typical scenarios that occur during day to day repository development.

This section contains the following topics:

Updating, Saving, and Checking In Changes for Repositories Under Source Control

After your MDS XML repository is set up under source control, follow these steps to update, save, and check in changes to your repository.

Ensure that you have a local copy of your working MDS XML repository files that are under source control by issuing the appropriate commands in your SCM system. For example, for Subversion, you can issue the command svn info as shown in the following example text:

C:\myProj\repos>svn info
Path: .
Working Copy Root Path: C:\myProj\repos
URL: file:///C:/SVN/myProj/trunk/sample1
Repository Root: file:///C:/SVN/myProj
Repository UUID: 6b995c92-3ec0-fa4b-9d58-c98e54f41792
Revision: 3
Node Kind: directory
Schedule: normal
Last Changed Author: joe_user
Last Changed Rev: 2
Last Changed Date: 2011-11-19 15:20:42 -0600 (Sat, 19 Nov 2011)

In the Administration Tool , from the File menu, select Open, then select MDS XML.
Open the file in offline mode.
Select the root folder location for your MDS XML files and click OK.
If this is the first time you have opened this MDS XML repository in the Administration Tool, you are prompted to specify whether this repository is a standalone MDS XML repository, or whether it is under source control. Select Use Source Control and click OK.

This choice is saved for this repository. To view the status of this repository at any time, select Tools, then select Options, then select the Source Control tab.
After you make changes to your repository, select File, then select Save, or click Save on the toolbar. The Administration Tool displays a list of changes.
Click Yes to run the commands in the SCM system.

After accepting the changes, you cannot cancel. Canceling the changes creates an inconsistent repository. You must continue the SCM commands execution.

When the Administration Tool issues the SCM commands, the commands rearranged into the most optimal order.
Check in the changes directly in your SCM system.

Handling Errors

Learn how to handle errors in the SCM system.

Sometimes errors, such as an expired label or network problem, occur when the Administration Tool delivers changes to the SCM system.

In the Administration Tool, select File, then select Save As to save the repository to a temporary location in RPD format or MDS XML format. Close the Administration Tool.

Note:

Saving to a binary RPD is the simplest option for transient problems like network errors, where you just need to try again later. Saving as MDS XML is required when some sort of work is required to fix the problem, such as merging conflicting changes.
Take action to resolve the issue. For example, refresh an expired label or test and view a failed network connection.

In the case of an expired label, you also need to merge the contents of the refreshed label with the temporary saved MDS XML repository. Use a third-party merge tool to do this.

For detailed information about the MDS XML representation of repository objects so that you can successfully make merge decisions, see XML Schema Reference for Oracle Business Intelligence Enterprise Edition.
Open the saved repository in the Administration Tool.
Select File, then select Source Control, then select Link to Source Control.
Click Save to save changes from the saved repository into the MDS XML files under source control.

Steps 4 and 5 of this procedure cause the Administration Tool to keep memory objects loaded from the saved RPD file or MDS XML files, but to then consider them to belong to the source control MDS XML repository instead. When you click Save, the Administration Tool saves the memory objects to the source control repository.

Testing Repositories Under Source Control

During the course of repository development, you need to perform testing in online mode to validate your repository.

You can only load an Oracle BI repository in RPD format into the Oracle BI Server to make it available for queries. Because of this, you must save your development MDS XML repository in RPD format from time to time when you want to perform online testing.

See Making the Repository Available for Queries.

In the Oracle BI Administration Tool, open your MDS XML repository in offline mode, select Save As, then select Repository to make the repository available for queries.

Viewing the Source Control Log

The Source Control Log window shows the commands that the Oracle Administration Tool issues to your SCM system.

It also shows any post-save text you specified in the Post-save comment tab of the SCM Configuration Editor.

By default, the Source Control Log window appears when SCM commands are being executed. Alternatively, you can select File, then select Source Control, then select View Logs to see the Source Control Log window.

You can choose the following options for this dialog:

Close when commands finish: Causes the log window to close automatically when commands are complete, unless errors occur.
Only show dialog when errors occur: Hides the window during SCM command execution unless errors occur. By default, the Source Control Log appears automatically when SCM commands are being executed unless this option is selected.

The text displayed in the Source Control Log is persistent until you close the repository. This means that all SCM command output is available for view, regardless of whether the dialog is open during individual operations.

The Source Control Log does have a 32K character limit. When the window buffer becomes full, then the oldest commands are removed from the Source Control Log display to make room for the latest command output. To see the full output, go to the Administration Tool log at:

ORACLE_INSTANCE/diagnostics/logs/OracleBIServerComponent/coreapplication_obisn/user_name_NQSAdminTool.log

Note:

While SCM commands are being executed, the Close button is disabled until the SCM commands have finished or have stopped with an error, unless Only show dialog when errors occur has been selected.

Using Source Control Management with MUD

You can use source control management with your multiuser development environment.

For example, if you have an existing repository under multiuser development and you want to begin using source control management, you might follow the steps described in the following subsections:

Putting the MUD Master Repository and MUD Log File Under Source Control

Use this procedure to put the MUD master repository and MUD log file under source control.

Run the mhlconverter command-line utility to convert your MUD log file (*.mhl) to an XML file.

See Saving an Existing Repository File in MDS XML Format to convert your master MUD RPD to a set of MDS XML files on the file system.

At the command prompt, type mhlconverter with the input MHL file name and path, and the output XML file name and path. For example:

mhlconverter -I C:\MUD\mud_repository.mhl -O C:\MUD\mud_repository.xml
Check the MDS XML files and XML-format MUD log file into your SCM system.

Checking In New Versions of the MUD Master and MUD Log File to Source Control

After creating and checking in the initial version of the master MUD repository, you need to check in updated versions of the MUD master repository on an ongoing basis.

This section describes two different ways to perform this task.

Manually Checking In the Updated MUD Master Repository and Log File

Use these steps to manually check in changes to the master RPD and log file that have occurred as part of the multiuser development process.

Consider using the automated check-in method described in Using a Script to Check In the Updated MUD Master Repository and Log File if you have a large repository. See Creating an SCM Configuration File.

Open the latest copy of the master RPD in the Administration Tool.
Create or select the appropriate SCM configuration file.
Select File, then select Source Control, then select Link to Source Control. Select the directory that contains the MDS XML version of the master MUD repository.

Using Link to Source Control is not recommended for large repositories and might cause time-outs.
Click Save to save changes from the master MUD repository into the MDS XML files under source control. The Administration Tool determines which files to add, check out, modify, and delete and issues the commands to your SCM system.
Close the Administration Tool.
Follow these steps to update the MUD log file:
1. In your SCM system, check out the XML-format MUD log file.
2. Use the mhlconverter utility to overwrite the XML-format MUD log file with the latest changes from the .mhl version.
3. Check in the latest XML-format MUD log file to your SCM system.
Check all changes into your SCM system.

It is recommended that you perform the steps in this section regularly to avoid having too many changes in a single transaction.

Using a Script to Check In the Updated MUD Master Repository and Log File

As an alternative to manually checking in changes, you can create a script to perform the check-in tasks and then schedule it to run at regular intervals.

See Comparing Repositories Using comparerpd.

Identify the latest copy of the master RPD that you want to check into your SCM system.
Identify the last version of the master RPD that was checked into the SCM system. You can review the latest XML-format MUD log file under source control to determine this version.

Note:

If you do not have the last checked-in version of the master repository in RPD format, you can use the biserverxmlexec utility with the -D option to read the latest MDS XML files checked into source control and re-create an RPD version.
Use the comparerpd utility with the -M option to compare the latest copy of the master RPD, the modified version, with the version that was last checked in, the original version. An MDS XML format diff is generated.
Create a script that does the following:
1. Reads the MDS XML diff directory to identify which files are present.
2. Issues commands in source control to check out the identified files or add new files.
3. Copies the latest version of the files from the MDS XML diff directory to the source control directory.
4. Reads the oracle\bi\server\base\DeletedFiles.txt file inside the MDS XML diff directory to determine which files to delete.
5. Issues commands in source control to delete the appropriate files.
6. Checks out the MDS XML-format MUD log file, runs the mhlconverter utility to convert the latest MHL-format log file to XML format, overwrites the existing MDS XML-format MUD log file with the new one, and checks it in.
7. Performs all necessary check-in steps in the SCM system.