5 Configuring and Using Automated Harvesting

The tool used to populate Oracle API Catalog is called the Harvester. The Harvester reads metadata from Oracle products and deployed services. This includes Oracle SOA Suite and Oracle Service Bus.

The Harvester automatically creates assets and populates asset metadata based on the information in the harvested service.

The files are harvested as they are deployed to the runtime environment. The Harvester can be used from the command line, and within Ant and the Weblogic Scripting Tool (WLST).

This chapter contains the following sections:

5.1 Getting Started with Harvester

This section describes how to get started with Harvester and its use in various high level use cases.

This section contains the following topics:

5.1.1 Prerequisites

Before using the Harvester, you must perform the following prerequisites:

  • If you wish to harvest from Oracle Service Bus, install the harvester to the same FMW_HOME to which Oracle Service Bus is installed or to a shared file system accessible to the Oracle Service Bus installation. See Section 5.2.2.2, "Selecting the Artifacts to Harvest for the Command Line" for more information.

    Note:

    If you wish to harvest projects from Oracle Service Bus, you must use the osb-harvest.bat or osb-harvest.sh utilities instead of the usual harvest.bat or harvest.sh Harvester utilities. This requires Oracle Service Bus 12c libraries to be installed. See Chapter 5, "Harvesting Web Services from Oracle Service Bus" for more information about the OSB Harvester.
  • When harvesting assets from deployed applications such as Oracle SOA Suite or Oracle Service Bus, then the application must be deployed on WebLogic 11g or higher.

  • Harvester requires Java SDK version 6 or higher.

5.1.2 Harvester Functionality

You can use the Harvester to publish services exposed by the artifacts and products listed in Table 5-1.

5.1.3 Artifacts/Products Version Matrix

Table 5-1 describes the version matrix of the artifacts/products that is supported with Harvester.

Table 5-1 Artifacts/Products Version Matrix

Artifact/Product Version

Deployed WSDL

1.1

Deployed WADL

1.1.5

Oracle SOA Suite

11g and 12c (12.1.3 and 12.2.1)

Oracle Service Bus

11g and 12c (12.1.3 and 12.2.1)

WebLogic Web Services

11g and 12c (12.1.3 and 12.2.1)


Note:

The harvester does not support harvesting end-to-end JSON or Native REST services in Oracle SOA Suite and Service Bus, introduced in the 12.2.1 release. Harvesting of other artifacts from the 12.2.1 releases of these products is supported.

5.2 Configuring the Harvester

This section describes how you can configure Harvester.

This section contains the following topics:

5.2.1 Obtaining the Harvester

The Harvesters are automatically unzipped into the following directories when installing Oracle API Catalog:

  • <FMW_HOME>/oer/tools/harvester

  • <FMW_HOME>/oer/tools/osbharvester

The Harvesters are also available in 12.1.3.0.0-OER-Harvester.zip, which is bundled with the Oracle API Catalog 12c installation, at the following location:

<FMW_HOME>/oer/modules/tools/solutions/12.1.3.0.0-OER-Harvester.zip

The OSB Harvester is also available in 12.1.3.0.0-OSB12c-Harvester.zip, which is bundled with the Oracle API Catalog 12c installation, at the following location:

<FMW_HOME>/oer/modules/tools/solutions/12.1.3.0.0-OSB12c-Harvester.zip

This manual refers to <FMW_HOME>/oer/tools/harvester or <FMW_HOME>/oer/tools/osbharvester as the <Harvester Home> directory, depending on which harvester you are using.

Table 5-2 describes the relationship matrix for the product and the corresponding version.

Table 5-2 Product and Version Matrix for Harvester

Product and Version Harvest IDE Search, Browse, and Consume

Service Bus 11g and 12c (12.1.3 and 12.2.1)

OER 12c

Yes

SOA Suite 11g and 12c (12.1.3 and 12.2.1)

OER 12c

Yes

WSDL

OER 12c

Varies

WADL

OER 12c

Varies


Note:

The harvester does not support harvesting end-to-end JSON or Native REST services in Oracle SOA Suite and Service Bus, introduced in the 12.2.1 release. Harvesting of other artifacts from the 12.2.1 releases of these products is supported.

5.2.2 Configuring the Harvester for the Command Line

You can configure the Harvester from the command-line. Organizations can easily bootstrap their API Catalog installation using the Harvester from the command-line.

This section describes the tasks you need to perform to configure the Harvester for the command-line:

5.2.2.1 Setting Repository Connection Information for the Command Line

Open the XML file HarvesterSettings.xml located at <Harvester Home> and modify the following XML to point the Harvester to an Oracle API Catalog instance with the correct credentials:

<repository>
  <uri>http://localhost:7101/oer</uri>
  <credentials>
    <user>smith</user>
    <password>*****</password>
     //To ensure security, the password must be encrypted.
     //The password encryption tool (encrypt.bat/encrypt.sh) allows you to encrypt
     // the passwords that are stored in the Harvester configuration
     // (HarvesterSettings.xml) file.
  </credentials>
  <timeout>30000</timeout>
</repository>

Note:

It is recommended that you run the Harvester as a user with the admin role.

5.2.2.2 Selecting the Artifacts to Harvest for the Command Line

The Harvester can be run from the command line using the harvest.bat or harvest.sh utilities.

Note:

You must use the OSB Harvester if you are harvesting projects from Oracle Service Bus. The OSB Harvester is installed into the <FMW_HOME>/oer/tools/osbharvester directory and can be run using the osb-harvest.bat or osb-harvest.sh utilities. The OSB Harvester must be installed into the same FMW_HOME to which OSB 12c is installed.

See Section 5.2.5.3, "Harvesting Web Services from Oracle Service Bus" for more information about using the OSB Harvester.

The harvest.bat and harvest.sh utilities can be run from the server to which Oracle API Catalog is installed. If you want to run the harvester from a different client, obtain the 12.1.3.0.0-OER-Harvester.zip file from the <FMW_HOME>/oer/modules/tools/solutions directory and extract the zip to any client that has a valid JAVA_HOME environment variable set, as discussed in Table 5-3.

The osb-harvest.bat and osb-harvest.sh utilities can be run from the server to which Oracle API Catalog is installed. If you want to run the harvester from a different client:

  • Install Oracle Service Bus into a <FMW_HOME> directory on the client machine.

  • Ensure that the JAVA_HOME environment variable is set on the client machine, as shown in Table 5-3.

  • Obtain the 12.1.3.0.0-OSB12c-Harvester.zip file from the <FMW_HOME>/oer/modules/tools/solutions directory on the machine to which Oracle API Catalog is installed. Extract the file to the <FMW_HOME>/oer/tools directory on the target machine. The full path to the harvester after extracting the files should be <FMW_HOME>/oer/tools/osbharvester. The OSB Harvester must be installed into the same FMW_HOME to which OSB 12c is installed.

Before running harvest.bat, harvest.sh, osb-harvest.bat, or osb-harvest.sh, ensure that the environment variables mentioned in Table 5-3 are set. In Windows, from a command window, you can type "set X" to see the value of the variable X, and "set X=abc" to set the value of X to "abc".

Table 5-3 Command Line Script

Environment Variable Description

JAVA_HOME

Ensure that the JAVA_HOME environment variable points to an installed java runtime (JRE) or SDK. For Oracle Service Bus introspection, this must be Java version 6 or higher.

JAVA_OPTS

Optionally, set your JAVA_OPTS parameter to add any additional java parameters that are necessary. For example, if you need to use an HTTP proxy server, set the value to -Dhttp.proxyHost=www-proxy.yourhost.com -Dhttp.nonProxyHosts= "*.yourhost.com |localhost"

See Also: http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html


Table 5-4 shows the options that can be specified using the Harvester command line utility:

Table 5-4 Command Line Options for the Harvester

Harvester Options Description

-settings<file>

Specifies the configuration settings XML file.

-url <URL>

Specifies the URL of Oracle API Catalog.

-user <OER/OAC User Name>

Specifies the user name of the Oracle API Catalog user.

-asset_version

Version to be applied to all assets created in the model

-file <filename or URL>

Specifies the file or directory to be harvested. This can be a filename or URL to the file.

-file_type <type>

Specifies the file type of the file to be harvested. If not specified, then the type is derived from the file extension. This must correspond to one of the filetypes in the config/plugins folder. By default, the following are supported: .bpel, .mfl, .policy, .wsdl, .xsd, .xquery, .xslt.

-remote_url <URL>

Specifies the running server from which to harvest the remote project, instead of from a file.

-remote_username <username>

Specifies the username to connect to the remote server.

-remote_server_type <type>

Specifies the type of remote server. Remote server could include either of the following: SOASuite, SOASuite11g, OSB, WLS.

Note: Use SOASuite11g when running against a SOA Suite 11g server. Use SOASuite when running against a SOA Suite 12c server.

-remote_project <type>

Specifies the name of remote project to harvest, instead of a file. If omitted, all of the projects on the server are harvested. In the case of Oracle SOA Suite, this should be the name of the composite plus revision, for example, MyComposite_rev1. In the case of WLS, this should be the Application Name, as seen in the WebLogic Administration console and Enterprise Manager.

-soa_partition <type>

Specifies the name of partition for Oracle SOA Suite and by default, it uses the soa "default" partition.

-version

Specifies the print version information.

-preview

Specifies if the harvest can be run on a preview mode. If set to true, then runs harvester in preview mode. A detailed information about successes and failures are logged, and no changes are committed to Oracle API Catalog.

If set to false or no value is specified, then runs harvester in production mode. A detailed information about successes and failures are logged, and the changes are committed to Oracle API Catalog.

-simple_model

Creates single-asset models when set to true. This requires the API asset type. This is set to true by default for Oracle API Catalog.

-help

Displays the online help for the Harvester command line utility.


An example of a remote harvest is as follows:

harvest.bat -remote_url mysoasuiteserver:8001 -remote_username weblogic -remote_server_type SOASuite -remote_project MyComposite_rev1.0 -soa_partition department1

Figure 5-1 shows the command line utility options and online help displayed by the harvest.bat -help command.

Figure 5-1 Harvest Command Line Utility Options

Description of Figure 5-1 follows
Description of ''Figure 5-1 Harvest Command Line Utility Options''

Command line options are required only when harvesting from an Oracle SOA Suite 11g server; the -remote_server_type SOASuite11g option must be included if this is the case. See Section 5.2.5.1, "Harvesting from Oracle SOA Suite Server" for more information.

If the options are omitted, then the Harvester uses the information in the HarvesterSettings.xml file in the <Harvester Home> directory, where harvest.bat resides. If options are specified on the command line, then these override the settings in HarvesterSettings.xml.

To ensure security, passwords are not passed via the command line and they must be encrypted. The only way to set passwords for Oracle API Catalog and remote servers is by storing them in the Harvester configuration (HarvesterSettings.xml) file. The password encryption tool (encrypt.bat/encrypt.sh) allows you to encrypt the passwords that are stored in the file. For more information about password encryption, see Chapter 8, "Password Encryption".

To point to the artifacts to be harvested using the HarvesterSettings.xml file in the <Harvester Home>, modify the following XML:

<query>
  <fileQuery>
     <rootDir>C:\samples</rootDir>
     <files>BPEL</files>
  </fileQuery>
</query>

or

<query>
  <fileQuery>
     <files>http://remote/server/my_generated_wsdl</files>
     <fileType>.wsdl</fileType>
  </fileQuery>
</query>

5.2.3 Invoking the Harvester Using the Repository.Submit Ant Task

You can invoke harvester as an Ant Task to ensure that all deployment information is stored in Oracle API Catalog at deployment time. You can use the repository.submit Ant task provided with the Harvester to harvest and import metadata into Oracle API Catalog. This task can be defined in the harvest-tasks.xml file, which is located in the <FMW_HOME>/oer/tools/harvester directory.

This section contains the following topics:

5.2.3.1 Specifying Parameters for the repository.submit Ant Task

Table 5-5 shows parameters that can be specified for the repository.submit Ant task in the harvest-tasks.xml file:

Table 5-5 Parameters for the repository.submit Ant Task

Attribute Description Required

repositoryURL

Repository instance to connect to.

Yes, unless specified by a property.

repositoryUsername

Username to log into Oracle API Catalog.

Yes, unless specified by a property.

repositoryPassword

Password to log into Oracle API Catalog.

To ensure security, the password must be encrypted.

The Oracle API Catalog Web console has a tool to encrypt passwords:

http://<host>:<port>/<domain>/diag/encryptstrings.jsp

Yes, unless specified by a property.

timeout

Number of seconds before calls to Oracle API Catalog will time out.

No. Defaults to 300 (5 minutes).

failOnError

Fails the entire build script if the Oracle API Catalog operation results in an error.

No. Defaults to "true."

errorProperty

Name of the Ant property to set if the repository operation results in an error. This is only useful if failOnError is set to false. If there is an error, the specified property is set to true. Otherwise, the specified property will remain unset.

No.

debug

Display debug information from the Oracle API Catalog task regardless of the Ant -debug setting. Debug information will also be displayed if you pass the -debug parameter to the Ant runtime.

No. Defaults to "true."

description

Harvester Description to associate with each asset created in Oracle API Catalog. This is visible in the Harvester Properties.

No.

namespace

Namespace with which to prefix each asset created in Oracle API Catalog.

No.

version

Harvester Version to associate with each asset created in Oracle API Catalog. This is visible in the "Harvester Properties."

No.

beaHome

The BEA_HOME directory of an installed Oracle Service Bus server.

Yes, when harvesting Oracle Service Bus project jars.

settingsFile

Location of settings XML file. Must conform to BPEL_Harvester_Settings.xsd. This file configures what Harvester classes are mapped to which file types, and how entity and relationships are mapped to types in Oracle API Catalog.Optionally, you can specify this attribute or the settingsURL attribute, but not both.

No. Defaults to settings XML that is bundled with the Ant task.

settingsURL

Location of settings XML file. Must conform to BPEL_Harvester_Settings.xsd. This file configures what Harvester classes are mapped to which file types, and how entity and relationships are mapped to types in Oracle API Catalog.Optionally, you can specify this attribute or the settingsFile attribute, but not both.

No. Defaults to settings XML that is bundled with the Ant task.

preview

This option sets the Harvester to Preview mode. The Harvester displays all assets to be created or modified, however, no transaction is committed. To enable, set value to "true".

No. Defaults to false.


5.2.3.2 Specifying Parameters as Nested Elements

FileSet

FileSets are used to select sets of files to harvest. One or more fileSets must be specified.The Harvester examines all the files selected by the fileSet, including files in .zip format (including .zip, .jar., and .ear files, for example).In the Ant repository.submit task shown in Example 5-1, the Harvester examines all the files and directories under the fileSet directory (the /tmp/components directory) and imports them into Oracle API Catalog.

URI

URI identifies a file to introspect. One or more uris can be specified.This can be an "http:" URL that points to a remote file, or a "file:" URL that points to a local file, or any other URL supported by java.This can point to a single file, or a file in zip format that includes zips, jars, ears, etc. This could include exported Oracle Service Bus project jars. Oracle Service Bus projects can be exported from Oracle Service Bus Workshop, using the Export|Oracle Service Bus|Configuration Jar command.The artifactStore attribute is the name of an Oracle API Catalog artifact store to look in. The artifact store must be created beforehand in the Oracle API Catalog Asset Editor. If specified, then the URI is resolved relative to the artifact store URL. When specifying a URI relative to an Artifact Store, the URI must resolve to a file such as a .wsdl or .zip file. The URIs that point to directories are not supported. The fileType attribute indicates the file type of the file to be harvested. If not specified, then the type is derived from the file extension. This must correspond to one of the file types in the config/plugins folder. By default, the following are supported: .bpel, .mfl, .policy, .wsdl, .xsd, .xquery, .xslt.

Example 5-1 Specifying the Files to Harvest with the Ant repository.submit Task

<repository.submit repositoryurl="http://server.example.com:8080/oer"
                   repositoryusername="myuser"
                   registrypassword="*****"
     //To ensure security, the password must be encrypted.
     //The password encryption tool (encrypt.bat/encrypt.sh) allows you to encrypt
     // the passwords that are stored in the Harvester configuration
     // (HarvesterSettings.xml) file.
                   settingsFile="../MyCustomSettings.xml">
  <fileset dir="/tmp/components/">
    <include name="**/*"/>
  </fileset>
</repository.submit>

RemoteProject

RemoteProject specifies that the Harvester should read a project from a remote server rather than from a file. The uri attribute indicates the running server from which to harvest the remote project. The username and password attributes indicate login information for the remote server.

The serverType attribute indicates the remote server type, which could be either Oracle SOA Suite (SOASuite or SOASuite11g), Oracle Service Bus (OSB) or WebLogic Server (WLS). The projectName element indicates the name of remote project to harvest, instead of a file. If omitted, all of the projects on the server are harvested. In the case of Oracle SOA Suite, this should be the name of the composite plus revision, for example, MyComposite_rev1. In the case of WLS, this should be the Application Name, as seen in the WebLogic Administration console and Enterprise Manager.

Example 5-2 Specifying Parameters as Nested Elements - RemoteProject

<repository.submit repositoryurl="http://server.example.com:8080/oac"
                   repositoryusername="myuser"
                   registrypassword="*****"
     //To ensure security, the password must be encrypted.
     //The password encryption tool (encrypt.bat/encrypt.sh) allows you to encrypt
     // the passwords that are stored in the Harvester configuration
     // (HarvesterSettings.xml) file.
                   settingsFile="../MyCustomSettings.xml">
  <remoteProjects uri="http://mywlsserver:7001" username="admin" password="*****"
 serverType="SOASuite" soaPartition="${partiton}">
     //To ensure security, the password must be encrypted.
     //The password encryption tool (encrypt.bat/encrypt.sh) allows you to encrypt
     // the passwords that are stored in the Harvester configuration
     // (HarvesterSettings.xml) file.
     <projectName>MySOAComposite1_rev1.0</projectName>
     <projectName>MySOAComposite3_rev2.0</projectName>
  </remoteProjects>
</repository.submit>

Producing Projects

You can specify multiple projects tagged as producing projects for all assets that are created from the harvest. The projects must already exist in OER.

Within the <repository.submit> element, include:

  • <producingProject name="MyProjectName"/>

  • <producingProject name="MyOtherProjectName"/>

Producing projects may also be set within the HarvesterSettings.xml file, in the same way.

5.2.3.3 Running the Harvester from Ant

To import the Harvester Ant tasks, include a line, as follows, in your Ant XML:

<taskdef file=${harvester.dir}/harvest-tasks.xml/>

where harvester.dir is the <FMW_HOME>/oer/tools/harvester directory.

When running from the command line, ensure that the Harvester libraries are available to Ant's classpath. Harvester includes a script called runant that sets up the ant classpath correctly. This script must be used to launch ant when using the harvester ant tasks. For example:

runant -f mybuild.xml

(where mybuild.xml is your ant build XML).

Any arguments passed to runant are passed along to ant. Before running harvest.bat or harvest.sh, ensure that the required environment variables are set. Refer to Table 5-6 for the list of environment variables that you need to set.

Table 5-6 Command Line Script

Environment Variable Description

JAVA_HOME

Ensure that the JAVA_HOME environment variable points to an installed java runtime (JRE) or SDK. For Oracle Service Bus introspection, this must be Java version 6 or higher.

ANT_HOME

Ensure that the ANT_HOME environment variable points to an installation of Apache Ant, version 1.6.2 or higher. For Oracle Service Bus introspection, ensure that the Ant version is 1.6.5 or higher.

BEA_HOME

Ensure that the BEA_HOME environment variable points to the installation directory containing Oracle Service Bus server, if you plan to harvest projects from Oracle Service Bus. For example, C:\bea.

JAVA_OPTS

Optionally, set your JAVA_OPTS parameter to add any additional java parameters that are necessary. For example, if you need to use an HTTP proxy server, set the value to -Dhttp.proxyHost=www-proxy.yourhost.com -Dhttp.nonProxyHosts= "*.yourhost.com |localhost"

See Also: http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html


5.2.3.4 Using the Third-Party Tasks

The runant script uses ant's old launcher: org.apache.tools.ant.Main. The new launcher uses a URIClassLoader interface, which interferes with the usage of custom URIStreamHandlers of the Oracle Service Bus.

The old ant launcher does not support automatic discovery of custom ant tasks in the ant/lib directory. When defining custom tasks with <taskdef>, you must specify a classpath attribute.

For example, the following is an example of how the <taskdef> should NOT be defined:

<taskdef id="ant-contrib" resource="net/sf/antcontrib/antcontrib.properties"/>

The following is an example of how the <taskdef> should be defined:

<taskdef id="ant-contrib" resource="net/sf/antcontrib/antcontrib.properties" classpath="${ant.home}\lib\ant-contrib.jar"/>

5.2.4 Invoking Harvester from WLST

WLST (WebLogic Scripting Tool) is a command-line scripting interface that system administrators can use to manage WebLogic Server instances. WLST supports Oracle Service Bus 11g and 12c and SOA Suite 11g and 12c. For more information about WLST, see Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.You can invoke harvester from WLST to ensure that all deployment information is stored in Oracle API Catalog at deployment time. Perform the following steps to invoke harvester from WLST:

  1. Copy the oer.py file from the harvester installation folder to the <BEA_HOME>\wlserver\common\wlst\lib directory.

  2. In the command window, enter the following commands:

    • Initialize the WLST classpath: run <BEA_HOME>\wlserver\server\bin\setWLSEnv.cmd

    • Initialize the Harvester classpath: cd to the <harvester> folder and run <harvester>\setenv.bat

    • Invoke WLST: java weblogic.WLST<scriptname>

  3. To invoke WLST from ant, perform the following steps:

    • In the command prompt, change to the <harvester> directory by using the cd command.

    • Enter runant.bat -f<antscriptname>, where the ant script invokes the weblogic.WLST java class using ant's <java> task.

    • The <java> task should set fork = true.

    • The <java> task should include the environment's CLASSPATH (runant.bat will set CLASSPATH to include the harvester classes)

    • The <java> task should also include on the classpath any libraries referenced by the WLST script.

    The sample ant script is as follows:

    <property environment="env"/> 
    <target name="harvest"> 
    <java classname="weblogic.WLST" fork="true">
    <arg line="${domain.import.script}"/>
    <arg line="${import.config.file}"/>
    <classpath refid="extra.class.path"/> <!--extra jars used by WLST script, e.g.
     OSB jars-->
    <classpath path="${env.CLASSPATH}"/>
    </java>
    </target>
    

    The oer.harvest() method takes one argument: a dictionary of command line parameters. These are the same parameters that are available from the harvester command line.

    For more information about using the harvester from command line, see Section 5.2.2.2, "Selecting the Artifacts to Harvest for the Command Line".

    A sample usage is as follows:

    argMap = {}
    argMap['-harvester_home'] = '.'
    argMap['-bea_home'] = 'c:/bea'
    argMap['-remote_url'] = 'http://mywlsserver:7001'
    argMap['-remote_username'] = 'admin'
    argMap['-remote_password'] = '*****'
    //To ensure security, the password must be encrypted.
    //The password encryption tool (encrypt.bat/encrypt.sh) allows you to encrypt
    // the passwords that are stored in the Harvester configuration
    // (HarvesterSettings.xml) file.
    argMap['-remote_server_type] = 'SOASuite'
    argMap['-remote_project] = 'MySOAComposite_rev1.0'
    argMap['-soa_partition'] = 'department1'
    oer.harvest(argMap)
    

    The -harvester_home argument must point to the location of the harvester installation. The -bea_home argument is required for Oracle Service Bus harvesting.

5.2.5 Runtime Harvesting Details

As assets move from development to testing, staging, and production environments, you may want to harvest these assets into Oracle API Catalog, so that Oracle API Catalog has the most up-to-date endpoints. Oracle API Catalog overwrites the existing endpoints of existing assets in Oracle API Catalog. If the WSDL is modified as the asset moves through its lifecycle, the updated WSDL is also associated with the existing asset in Oracle API Catalog.

This section describes the runtime harvesting details for different servers. This section contains the following topics:

5.2.5.1 Harvesting from Oracle SOA Suite Server

When an Oracle SOA Suite project deployed to a running server is introspected, any services exposed by the composite results in the creation of API Assets in Oracle API Catalog.

You need the following WLS security roles when harvesting from Oracle SOA Suite server:

  • Admin

  • Operator

  • Monitor

The -remote_url parameter should point to the port of the soa-infra managed server. The default value of this in Oracle SOA Suite is 8001.

The soa-infra managed server must be up and running. Harvesting connects to the MDS database as part of the remote harvesting. The MDS database must be running and accessible from the machine where harvesting is taking place.

The SSL/HTTP protocol is supported. Harvester connects to the server MBeans via the t3 protocol.

The harvester automatically saves the value of -remote_url in the Deployment URI property, in the assets harvested from the runtime servers.

An example for harvesting from Oracle SOA Suite server is as follows:

harvest.bat -remote_url mysoasuiteserver:8001 -remote_username weblogic -remote_server_type SOASuite -remote_project MyComposite_rev1.0 -soa_partition default

An example for harvesting from Oracle SOA Suite 11g server is as follows:

harvest.bat -remote_url mysoasuiteserver:8001 -remote_username weblogic -remote_server_type SOASuite11g -remote_project MyComposite_rev1.0 -soa_partition default

To ensure security, passwords are not passed via the command line and they must be encrypted. The only way to set passwords for Oracle API Catalog and remote servers is by storing them in the Harvester configuration (HarvesterSettings.xml) file. The password encryption tool (encrypt.bat/encrypt.sh) allows you to encrypt the passwords that are stored in the file. For more information about password encryption, see Chapter 8, "Password Encryption".

The Oracle SOA Suite server has also implemented partitions to further categorize composites during runtime. This enables you to select the partition to use, along with the composite name, in the harvester. The parameter name in the HarvesterSettings.xml file is called soaPartition. A sample HarvesterSettings.xml file is as shown below:

<remoteQuery>
       <serverType>SOASuite</serverType>
       <projectName>MyComposite_rev1.0</projectName>
       <uri>http://remotehost:8001/</uri>
       <credentials>
         <user>weblogic</user>
         <password>password</password>
       </credentials>
       <soaPartition>partition_name</soaPartition>
</remoteQuery>

Note:

For Oracle SOA Suite 11g, you must set the <servertype> element to SOASuite11g. Additionally, you must include the -remote_server_type SOASuite11g command line argument when harvesting from Oracle SOA Suite 11g server.

Note:

Partitions should be used only with SOA servers of version 11g R1 PS2 or later. You should comment the soaPartition element, if you are using a version earlier to PS2.

By default, the Harvester harvests using the partition_name option set to "default". If it is not specified, then partition "default" comes out-of-the-box by SOA deployments.

Note:

You can deploy the same composite under different partitions in SOA Suite.

If this occurs and you harvest more than one of them, only one API asset gets created in OAC and will continually be overwritten by the latest harvest.

Harvesting SOA Suite Composites in a Cluster

The OHS server is deployed as a part of the WebTier label into the Middleware Home and is associated with the domains configured within this Middleware Home, as well.

There is a number of URI's that would normally be assigned to the OHS server's mod_wl_2x.so HTTP server module, some of which can be seen below (taken from the OHS mod_wl_ohs.conf file and modified in order to make it generic for external consumption):

LoadModule weblogic_module

"/opt/Oracle/Middleware/wlserver_10.3/server/plugin/linux/x86_64/mod_wl_22.so"
<IfModule weblogic_module>
    ConnectTimeoutSecs 10
    ConnectRetrySecs 2
    Debug ALL
    WLLogFile /tmp/weblogic.log
    DebugConfigInfo ON
    WLSocketTimeoutSecs 2
    WLIOTimeoutSecs 300
    Idempotent ON
    FileCaching ON
    KeepAliveSecs 20
    KeepAliveEnabled ON
    DynamicServerList ON
    WLProxySSL OFF
</IfModule>
<Location /soa-infra>
    SetHandler weblogic-handler
    WebLogicCluster slc01ese.example.com:23049,wls02.example.com:15980
</Location>
<Location  ~ "/bea_wls_internal/iiop/Client*">
    SetHandler weblogic-handler
    WebLogicCluster wls01.example.com:23049,wls02.example.com:15980
</Location>
<Location  ~ "/bea_wls_internal/HTTPClnt*">
    SetHandler weblogic-handler
    WebLogicCluster wls01.example.com:23049,wls02.example.com:15980
</Location>
<Location /console>
    SetHandler weblogic-handler
    WebLogicHost wls01.example.com
    WebLogicPort 19617
</Location>
<Location /em>
    SetHandler weblogic-handler
    WebLogicHost wls01.example.com
    WebLogicPort 19617
</Location>
<Location /bea_wls_internal>
    SetHandler weblogic-handler
    WebLogicHost wls01.example.com
    WebLogicPort 19617
</Location>
<Location /jndi>
    SetHandler weblogic-handler
    WebLogicHost wls01.example.com
    WebLogicPort 23049
</Location>
<Location /HTTPClnt>
    SetHandler weblogic-handler
    WebLogicHost wls01.example.com
    WebLogicPort 23049
</Location>

These special URI patterns allow for JMS and JMX requests to be routed through the OHS WebLogic module.

Other than any changes that you had applied to code, these are the only customizations that had been applied to the SOA clustered installation.

5.2.5.2 Harvesting Web Services from WebLogic Server

An Oracle Service Bus project can be submitted to Oracle API Catalog from the command line. The project must be exported from Oracle Service Bus Workshop, using the command Export|Oracle Service Bus: Configuration jar.

The harvester will handle Proxy Services and WSDL artifacts. The resulting asset will be an API asset type.

You do not need any WebLogic Server security roles when harvesting from WebLogic Server.

The -remote_url parameter should point to the port of the WLS admin server. The default value of this in Weblogic is 7001. The WLS admin server must be up and running. The SSL/HTTP protocol is supported. Harvester connects to the server MBeans via the t3 protocol.

The harvester automatically saves the value of -remote_url in the Deployment URI property, in the assets harvested from the runtime servers.

5.2.5.3 Harvesting Web Services from Oracle Service Bus

You need the Admin WLS security role when harvesting Web Services from Oracle Service Bus.

You must use the OSB Harvester to harvest web services from Oracle Service Bus. The OSB Harvester is installed in the <FMW_HOME>/oer/tools/osbharvester directory.

The OSB Harvester can be run from the command line using the osb-harvest.bat or osb-harvest.sh utilities. See Section 5.2.2, "Configuring the Harvester for the Command Line" for information about configuring the OSB Harvester.

Note:

When harvesting from a Service Bus 11g server using the osb-harvest.bat/.sh utility, the Service Bus harvester must be run from the same Middleware home into which Service Bus 12c is installed. Even though the Service Bus server is 11g, the Harvester requires the Service Bus 12c libraries. There are multiple ways to accomplish this; the easiest way is to copy the Service Bus harvester to an existing Service Bus 12c server.

If Service Bus 12c is installed on a separate client or in a separate Middleware home, you can update the setenv.sh or setenv.bat script to account for this location.

The -remote_url parameter should point to the port of the WLS admin server for the Oracle Service Bus domain. The default value of this in Weblogic is 7001. The WLS admin server must be up and running. The SSL/HTTP protocol is supported. Harvester connects to the server MBeans via the t3 protocol.

The harvester automatically saves the value of -remote_url in the Deployment URI property, in the assets harvested from the runtime servers.

An example of harvesting Web Services from Oracle Service Bus is as follows:

osb-harvest.bat -remote_url myosbserver:7001 -remote_username weblogic -remote_server_type OSB -remote_project MyOSBProject

To ensure security, passwords are not passed via the command line and they must be encrypted. The only way to set passwords for Oracle API Catalog and remote servers is by storing them in the Harvester configuration (HarvesterSettings.xml) file. The password encryption tool (encrypt.bat/encrypt.sh) allows you to encrypt the passwords that are stored in the file. For more information about password encryption, see Chapter 8, "Password Encryption".

5.2.5.4 Harvesting Deployed Services

In addition to harvesting APIs from SOA Suite and Service Bus, you can also harvest deployed services using a URL to a deployed WSDL file. The example below will harvest a Weather service from the internet:

./harvest.sh -file "http://example.com/WeatherWebService/Weather.asmx?WSDL"

This method can be used to harvest secure endpoints from Oracle API Gateway (OAG). To get the WSDL URL from OAG, make sure the OAG Policy Studio publishes its WSDL to clients by checking "Allow the Gateway to publish WSDL to clients" checkbox on the WSDL tab in the Service Handler for the web service.

5.2.6 Performing Optional Harvester Configuration

You can optionally modify these additional configuration settings in the XML file HarvesterSettings.xml in the <Harvester Home> directory:

  • <harvesterDescription>: A description about the harvesting performed. This information is stored in the Harvester Properties of the assets created in Oracle API Catalog

  • <harvesterVersion>: A version of the harvesting performed. This information is stored in the Harvester Properties of the assets created in Oracle API Catalog.

  • <namespace>: A namespace that is added to abstract (non-artifact) Oracle API Catalog assets that are created during harvesting. The namespace is used in duplicate detection. If left empty, then this is set based on information from Oracle SOA Suite and Oracle Service Bus projects when available.That is, generally, the best practice, so override this with caution.

  • <workDir>: A temporary directory where the zip and jar files are unzipped. By default, the system temp directory is used.

5.2.7 Configuring Logging for the Harvester

The Harvester uses log4j for logging the detailed tasks performed and the log file is placed in the <Harvester Home> directory. The logging options can be changed by updating the log4fl.properties file located in the <Harvester Home> directory.

5.2.8 Transaction Handling in Harvester

By default, the harvester makes all of its changes to Oracle API Catalog in a single transaction. Transactional operations in the harvester function is based on the following rules:

  • An Oracle API Catalog server only supports one transaction at a time. If a transaction cannot be started, then the harvester informs you about it.

  • There is a set of timeouts associated with the transactions. These will terminate a harvester operation if the timeouts are exceeded.

    • The setting, cmee.extframework.impexp.monitor.rex.maxidle, specifies the maximum amount of time the transaction is kept alive if it loses connectivity with the client. This allows the transaction to be cancelled if the client-side harvester process is killed.

    • The setting, cmee.extframework.impexp.monitor.maxruntime, specifies the maximum amount of time that the entire transaction will take.

    It is possible that a single operation will exceed the cmee.extframework.impexp.monitor.rex.maxidle (maxidle) setting and cause all subsequent operations to fail. When this occurs, the last error in the log will state that 'An error occurred while attempting to rollback because a transaction has not been started.'. This is because the transaction automatically rolls back when the statement exceeds the maxidle and the transaction is no longer available when the client application attempts a forceful rollback, having detected errors. The result is that no data, from the harvester operation, is added to Oracle API Catalog.

5.3 Understanding Additional Harvester Information

This section describes additional information about using the Harvester.

This section contains the following topics:

5.3.1 Previewing of the Created Assets in Harvester

You can use the Preview feature to view the created assets in harvester. You can use the Preview feature using either of the following methods:

Using Command Line

At the command line or in an ant task, if you add "-preview true", then the Harvester runs and displays a list of all of the assets that would be created, but it does not actually commit the changes.

Using the HarvesterSettings.xml File

You can also set the preview mode in the HarvesterSettings.xml file as follows:

<introspection preview="true">
       <reader>com.oracle.oer.sync.plugin.reader.file.FileReader</reader>
       <writer>com.oracle.oer.sync.plugin.writer.oer.OERWriter</writer>
</introspection>

5.3.2 Best Practices

This section describes best practices for the Harvester. It contains the following topics:

5.3.2.1 Recommended Privileges for Harvesting

Only users with the admin role can harvest assets.

5.3.2.2 Do Not Override the Namespace Parameter

Harvester offers you with the ability to set the project namespace for the assets being harvested. The project namespace is used in detecting duplicate assets, as different namespaces will result in different assets. In most cases, you are recommended not to override this parameter.

However, harvester automatically sets the project namespace based on the Oracle SOA Suite or Oracle Service Bus project name when harvesting from those products, which is the recommended behavior.

This is especially important to follow, when you harvest the same project and revision from multiple sources (JDev and runtime). Changing or overriding the namespace leads to duplicate assets in OER.

5.3.2.3 Namespaces in WSDL Files

In the WSDL files that you harvest, it is recommended that you use a unique namespace for each unique interface, service, and endpoint.

Correlation to existing assets in the Oracle API Catalog is done through QNames, so if you make significant changes to interface, service, or endpoint assets and do not change the QNames, you will overwrite the existing asset with the modified asset.

Table 5-7 shows the correlation of Oracle API Catalog assets with WSDL structure:

Table 5-7 Correlation of Oracle API Catalog Assets with WSDL Structures

Repository Asset WSDL Structure

Service

/definition/service/@name

Endpoint

/definition/service/port/@name

Interface

/definition/portType/@name


5.3.3 Known Issues

This section describes the following known Harvester issues:

5.3.3.1 Supported Remote Server Types

Currently, the Harvester only supports connecting to the remote Oracle SOA Suite, Oracle Service Bus, and WebLogic Server projects that are running on WebLogic 11g and 12c.

5.3.3.2 Versioning of Assets

The Harvester derives the asset names from the artifact it introspects (WSDL, WADL, Composite). If you harvest an artifact that already exists in OAC, the harvester will simply overwrite the existing one.

However, if the artifact has changed in any way but the name convention is the same as something that already exists, such as a service name + namespace from a WSDL, the harvester creates a new asset and appends a number to the end of the asset's name, such as "-2", "-3", "-4" and so on.

You can manually remove the old asset from OAC and rename the new one.

5.3.3.3 Using Incorrect Encrypted Password

If you configure the HarvesterSettings.xml file with incorrect encrypted password, then a long stack trace is displayed, which is as follows:

log4j:WARN No appenders could be found for logger
(org.apache.axis.i18n.ProjectResourceBundle).
log4j:WARN Please initialize the log4j system properly.
@ com.oracle.oer.sync.framework.MetadataIntrospectionException:
@ com.oracle.oer.sync.framework.MetadataIntrospectionException: Unable to read
plugin file:
C:\Drive-E\XU-harvest-tools\dec16\OER-Harvester\.\plugins\biz.introspector
        at
@ com.oracle.oer.sync.framework.MetadataManager.init(MetadataManager.java:308)
@         at com.oracle.oer.sync.framework.Introspector.<init>
Introspector.java:188)
        at
@ com.oracle.oer.sync.framework.Introspector.main(Introspector.java:395)
@ Caused by: com.oracle.oer.sync.framework.MetadataIntrospectionException:
Unable
to read plugin file:
C:\Drive-E\XU-harvest-tools\dec16\OER-Harvester\.\plugins\biz.introspector
        at
@ com.oracle.oer.sync.framework.impl.DefaultPluginManager.processIntrospector(De
faultPluginManager.java:127)
        at
@ com.oracle.oer.sync.framework.impl.DefaultPluginManager.<init>(DefaultPluginMa
nager.java:73)
        at
@ com.oracle.oer.sync.framework.MetadataManager.init(MetadataManager.java:306)
        ... 2 more
Caused by: java.lang.IllegalArgumentException: The char '0x12' in
'java.lang.Ill
egalArgumentException: The char '0x12' in 'E/?&#8597;?&#8597;H?rd' is not a
valid XML character.' is not a valid XML character.
        at
@ org.apache.axis.components.encoding.AbstractXMLEncoder.encode(Abstrac
tXMLEncoder.java:110)
        at org.apache.axis.utils.XMLUtils.xmlEncodeString(XMLUtils.java:131)
        at org.apache.axis.AxisFault.dumpToString(AxisFault.java:366)
        at org.apache.axis.AxisFault.printStackTrace(AxisFault.java:796)
        at
com.oer.log4j.spi.ThrowableInformation.getThrowableStrRep(Throw
ableInformation.java:42)
        at
com.oer.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:217)
        at
com.oer.log4j.WriterAppender.subAppend(WriterAppender.java:298)
        at
com.oer.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:294
)
        at com.oer.log4j.WriterAppender.append(WriterAppender.java:157)
        at
com.oer.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
        at
com.oer.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(Appen
derAttachableImpl.java:57)
        at com.oer.log4j.Category.callAppenders(Category.java:255)
        at com.oer.log4j.Category.forcedLog(Category.java:445)
        at com.oer.log4j.Category.log(Category.java:882)
        at
@ com.oracle.oer.sync.framework.logger.DefaultMetadataLogger.error(DefaultMetada
taLogger.java:98)
        at
@ com.oracle.oer.sync.plugin.writer.oer.OERConnectionCache.getAuthToken(OERCon
nectionCache.java:138)
        at
@ com.oracle.oer.sync.plugin.writer.oer.OERAssetQueries.getToken(OERAssetQueri
es.java:82)
        at
@ com.oracle.oer.sync.plugin.writer.oer.OERAssetQueries.assetTypeQueryByUUID(AL
ERAssetQueries.java:159)
        at
@ com.oracle.oer.sync.framework.MetadataManager.putAssetType(MetadataManager.jav
a:204)
        at
@ com.oracle.oer.sync.framework.impl.DefaultPluginManager.processIntrospector(De
faultPluginManager.java:104)
        ... 4 more

5.3.3.4 View in Repository Option

From Oracle JDeveloper 12c, when you click the View in Repository option, it opens up the asset details in JDeveloper, which is for read-only purpose and the links within this page will not work.

As a workaround, you may need to login to Oracle API Catalog console and click links for further information.