13 Configuring Automated Usage Detection

This chapter describes Software File Identification(SFID) and how it can be used to automatically increment asset usage in Oracle Enterprise Repository.

There are several ways in which Oracle Enterprise Repository can track asset usage. Oracle Enterprise Repository strives to make Governance as seamless as possible for the developer, therefore automatic usage detection, or SFID is available from Eclipse and .Net environments. In these environments, developers receive a copy of a wsdl, schema, or other artifact. If an organization wants to automatically track the usage of these artifacts, they tag them with an SFID in Oracle Enterprise Repository. When the artifacts appear in the developer's environment, Oracle Enterprise Repository recognizes the artifact and increments usage. This means that a developer does not have to get the artifact from the repository. They can pull it from their desktop, or from a version control system. Regardless of the source, Oracle Enterprise Repository will recognize a match and increment usage.Usage tracking is the primary method that Oracle Enterprise Repository uses to generate Reuse ROI.

This chapter contains the following sections:

• Section 13.1, "Overview"

• Section 13.2, "Enabling SFID"

• Section 13.3, "Adding the SFID Element to a Type"

• Section 13.4, "Adding an SFID to an Asset"

• Section 13.5, "SFID Tool Kit"

• Section 13.6, "SFID Manager"

• Section 13.7, "Calculating the SFID Programmatically"

• Section 13.8, "SFID Limitations"

13.1 Overview

Software File Identification (SFID) provides the ability to determine asset usage independent of the manual asset Use - Download process within Oracle Enterprise Repository. The SFID process tags selected files within an asset with a unique SFID fingerprint. This tag is then used to detect when and where an asset is used, even if the asset was acquired through means other than the Use - Download button in Oracle Enterprise Repository. An instance of usage is recorded by Oracle Enterprise Repository when tagged files within the asset are opened in a developer's IDE.

SFID requires the installation of the Oracle Enterprise Repository Plug-in Eclipse or for Visual Studio .NET. For more information about repository access capabilities from a supported IDE, see the following documentation:

• Repository Access in Eclipse Using WorkSpace Studio at Oracle Fusion Middleware Integration Guide for Oracle Enterprise Repository

• Oracle Enterprise Repository Integration with Visual Studio .NET at Oracle Fusion Middleware Integration Guide for Oracle Enterprise Repository

13.2 Enabling SFID

This procedure is performed on the Oracle Enterprise Repository Admin screen.

1. Click System Settings in the left pane.

2. Enter cmee.usagedetection.enable in the Enable New System Settings text box.

3. Click Enable. The Enable Automated Usage Detection option is displayed in the list of properties in the main pane, as shown in Figure 13-1.

   Figure 13-1 Functional Settings

   
   Description of "Figure 13-1 Functional Settings"
   
   

4. Ensure the property is set to True.

5. Click Save. SFID is now enabled.

   • The SFID element can now be added to types in the Type Manager.

   • When the SFID element is added to an asset type, an SFID section will appear on assets of that type when viewed in the Asset Editor. It is within that section that files can be selected for SFID fingerprinting.

     Figure 13-2 Automated Usage Detection

     
     Description of "Figure 13-2 Automated Usage Detection"
     
     

   • SFID functions are now available in the Oracle Enterprise Repository REX.

13.3 Adding the SFID Element to a Type

Adding the SFID element to a type enables assets of that type to be automatically detected.

Note: You will also need to add an SFID to the individual assets to be tracked.(See Adding an SFID to an Asset).

This procedure is performed in the Type Manager.

1. Select the type to which the SFID element is to be added, as shown in Figure 13-3.

   Figure 13-3 Type Manager Window

   
   Description of "Figure 13-3 Type Manager Window"
   
   

2. Click the Editor tab.

3. From the list in the Tabs section, select the tab to which to add the SFID element, as shown in Figure 13-4.

   Figure 13-4 Tabs List

   
   Description of "Figure 13-4 Tabs List"
   
   

   A list of the elements on that tab appears in the Elements section.

4. Click the Add button in the Elements section. The Select an Elementy Type to Add dialog is displayed, as shown in Figure 13-5.

   Figure 13-5 Select an Element Type to Add Dialog

   
   Description of "Figure 13-5 Select an Element Type to Add Dialog"
   
   

5. Click SFID from the list.

6. Click OK. The Select an Element Type to Add dialog closes, and the Edit Automated Usage Detection dialog is displayed, as shown in Figure 13-6.

7. Enter a name for the SFID element.

   Figure 13-6 Edit Automated Usage Detection Dialog

   
   Description of "Figure 13-6 Edit Automated Usage Detection Dialog"
   
   

8. Click OK. The new element is listed in the Elements section, as shown in Figure 13-7.

   Figure 13-7 Elements Section

   
   Description of "Figure 13-7 Elements Section"
   
   

9. Click Save to save the modified asset type.

10. Exit the Type Manager.

11. In the Asset Editor, open an asset of the type to which the SFID element was added.

    The SFID section now appears on the tab to which it was assigned.

    A summary of the export is displayed on completion, as shown in Figure 13-8.

    Figure 13-8 Automated Usage Detection Dialog

    
    Description of "Figure 13-8 Automated Usage Detection Dialog"
    
    

13.4 Adding an SFID to an Asset

To track assets through an IDE, select the unique artifact file representing the asset. The file is assigned an SFID, and usage is automatically incremented every time the file is found in an IDE project. This procedure is performed in the Asset Editor.

1. Select the asset to which an SFID is to be added.

2. Click the Add button in the Automated Usage Detection section.

3. Select the files to be fingerprinted from the list in the Add dialog, as shown in Figure 13-9.

   Figure 13-9 Add Dialog

   
   Description of "Figure 13-9 Add Dialog"
   
   

4. Click OK.

   The system downloads the selected files, calculates the SFID, and then deletes the local (downloaded) copies of the files. A warning message is displayed if fingerprinting for any of the selected files is unsuccessful.

13.4.1 Update an SFID File

This procedure is performed in the Asset Editor.

1. Select the SFID file to be updated from the list in the Automated Usage Detection section.

2. Click the Update button. The Perform Operation window opens, indicating the progress of the update.

13.4.2 Remove an SFID File

This procedure is performed in the Asset Editor.

1. Select the SFID file to be removed from the list in the Automated Usage Detection section.

2. Click the Remove button. The file is removed from the list.

13.5 SFID Tool Kit

The SFID Toolkit provides a framework that allows third party developers to develop applications that integrate with Oracle Enterprise Repository's REX subsystem for the purpose of submitting SFID Candidates. The SFID toolkit provides an application that will create an SFID Candidate for each file in a specified directory and then submit those SFID Candidates to the Oracle Enterprise Repository system for storage and analysis.

Notes on the Calculation of SFIDs

• A file is considered a text file if it contains no characters above ASCII code 128.

• The content of a text file is normalized before the SFID hash is computed.

  • All CR, LF, and Tab chars are stripped.

  • All text is converted to lower case.

• XML Files are canonicalized as per the W3C canonicalization standard (http://www.w3.org/TR/xml-c14n) before the SFID hash is computed.

13.5.1 Installation and Configuration

This section describes the steps to install and configure the SFID toolkit. This section contains the following topics:

• Section 13.5.1.1, "Enabling Harvesting of Unrecognized SFIDs"

• Section 13.5.1.2, "Object Model"

• Section 13.5.1.3, "Description of Sample Directory Crawler"

13.5.1.1 Enabling Harvesting of Unrecognized SFIDs

Some organizations mandate that every SOA asset deployed should be in the Oracle Enterprise Repository. Companies can monitor compliance to this mandate by harvesting unrecognized SFIDs. For example, companies may want to monitor the build scripts for unrecognized SFIDs. Additionally, Service monitoring tools may discover services that are not in Oracle Enterprise Repository. These "rogue services" can be assigned an SFID and harvested into Oracle Enterprise Repository as asset "{candidates. Once in Oracle Enterprise Repository, they can be further evaluated and processed.

This procedure is performed on the Oracle Enterprise Repository Admin screen.

1. Click System Settings in the left pane. The System Settings page is displayed in the main pane.

2. Enter cmee.sfid.harvesting.enable in the Enable New System Setting text box.

3. Click the Enable button. The Enable Harvesting of Unrecognized SFIDs property appears in the list of system settings.

   Figure 13-10 Enable Harvesting of Unrecognized SFIDs

   
   Description of "Figure 13-10 Enable Harvesting of Unrecognized SFIDs"
   
   

4. Ensure property is set to True.

5. Click Save. The property is now enabled.

13.5.1.2 Object Model

Figure 13-11 describes the object model for the SFID.

Figure 13-11 Object Model

Description of "Figure 13-11 Object Model"

13.5.1.3 Description of Sample Directory Crawler

• Iterate through all the files and folders of a specified folder.

  • Instantiate an SFIDCandidate for each candidate

  • Populate general metadata such as path and sfid.

    For each SFIDCandiate, create SFIDAttributes that describe arbitrary data to be captured, such as Eclipse Workspace.

• Invoke the Oracle Enterprise Repository REX method sfidCandidateSubmit, passing an array of SFIDCandidate objects.

• AlerUtil Object

  • Instantiate a AlerUtil object to connect to Oracle Enterprise Repository.

      String lUrl = "http://localhost:9080/oer/services/FlashlineRegistry";
      String lUserName = "username";
      String lPassword = "password";
      lAlerUtil = new AlerUtil(lUrl, lUserName, lPassword);
    
    

  • Use the AlerUtil object to invoke a Oracle Enterprise Repository REX method.

    lAlerUtil.getFlashlineRegistry().sfidCandidateSubmit(lAlerUtil.getAuthToken(), lSfidCandidates);
    
    

  • The AlerUtil object provides a series of convenience methods.

    • getAuthToken

      Obtain a valid authToken for the for current username and password. If the authToken is expired, obtain a new authToken. Otherwise, a UserValidationException is thrown.

    • calculateSfidValue

      Calculate the SFID for a file or path and return the hash as a string.

    • sfidCandiateListToArray

      Convert a list of SFIDCandidates to an array of SFIDCandidates.

    • sfidAttributeListToArray

      Convert a list of SFIDAttributes to an array of SFIDAttributes.

    • sfidAttributeValueListToArray

      Convert a list of SFIDAttributeValues to an array of SFIDAttributeValues.

    • createCandidate(File)

      Create a standard SFIDCandidate for a file. SFIDCandidate fields filepath, hostip, hostname, and sfid is pre-populated. An SFIDAttribute named Filelength is attached to the SFIDCandidate with the value of length of the file.

    • createCandidate(File, Map)

      Create a standard SFIDCandidate for a file as specified by the method createCandidate(File). Each (String)key/(String)value in the map will attach an additional SFIDAttribute to the SFIDCandidate whose name is key and whose value is value.

    • createCandidate(File, List)

      Create a standard SFIDCandidate for a file as specified by the method createCandidate(File). Attach each SFIDAttribute passed in the list to the SFIDCandidate.

    • sendCandidate

      Send a single SFIDCandidate for harvesting.

    • sendCandidates

      Send a list of SFIDCandidates for harvesting.

13.6 SFID Manager

The SFID Manager provides an easy way to view SFID candidates harvested by Oracle Enterprise Repository.

Use of the SFID Manager requires System Administrator privileges.

This section contains the following topics:

• Section 13.6.1, "Enabling the SFID Manager"

• Section 13.6.2, "Using the SFID Manager"

13.6.1 Enabling the SFID Manager

This procedure is performed on the Admin screen in Oracle Enterprise Repository.

1. Click System Settings in the left pane. The System Settings page is displayed in the main pane, as shown in Figure 13-12.

   Figure 13-12 Oracle Enterprise Repository System Settings Page

   
   Description of "Figure 13-12 Oracle Enterprise Repository System Settings Page"
   
   

2. Enter cmee.sfid.harvesting.enable in the Enable New System Setting text box.

3. Click the Enable button. SFID Manager is displayed in the list of system settings, as shown in Figure 13-13.

   Figure 13-13 SFID Manager

   
   Description of "Figure 13-13 SFID Manager"
   
   

4. Ensure the True option is selected.

5. Click Save.

6. Click Admin in the Oracle Enterprise Repository menu bar to refresh the Admin screen.

   The SFID Manager section now appears in the left pane.

13.6.2 Using the SFID Manager

This procedure is performed on the Admin screen in Oracle Enterprise Repository.

1. Click SFID Manager in the left pane.

2. Click List All in the SFID Manager section, as shown in Figure 13-14, to display a list of available SFID candidates.

   Figure 13-14 SFID Manager

   
   Description of "Figure 13-14 SFID Manager"
   
   

   Be advised that the number of available candidates affects the time it takes for this operation to complete. SFID candidates can also be located using the SFID Manager search function. Indentified candidates is listed in the upper section of the main pane, as shown in Figure 13-15.

   Figure 13-15 Name - SFID List

   
   Description of "Figure 13-15 Name - SFID List"
   
   

   The number of time the listed SFID has been detected (Times Seen)is indicated in the # column.

3. Click any listed candidate to display extended information in the bottom section of the main pane, as shown in Figure 13-16.

   Figure 13-16 SFID Candidate Properties

   
   Description of "Figure 13-16 SFID Candidate Properties"
   
   

   The displayed information consists of a core set of attributes common to all candidates:

   • Display Name

     The name of the candidate. Generally, the simple file name.

   • Generated By

     The username of the user that caused the candidate to be generated.

   • Generated Date

     The date on which the candidate was created.

   • Updated Date

     The last time that the candidate record was modified in the database.

   • File Path

     The absolute path to the file on the machine from which the candidate was harvested.

   • Hostname/IP

     The hostname or IP address of the machine from which the candidate was harvested.

   • # Times Seen

     The number of times this candidate has been viewed in this context. This metric gives an indication of the frequency of file use, which provides some indication of the value of the file in this context.

   • Source

     A description of the context, for example, Eclipse, Ant.

   The display also includes an extended set of attributes that will vary based on the context from which the candidate was harvested.

13.7 Calculating the SFID Programmatically

The ArtifactFingerprinter java library provides functionality for calculating SFID fingerprints programmatically in the client-side code.

Installation

Ensure that you have the components-flashtrax.jar file and it is in your java classpath for installation.

Usage

The com.oracle.artifact.ArtifactAlgorithm class contains several static instances, each of which encapsulates the fingerprinting algorithm for a particular type of file.

The supported algorithms are

• DEFAULT: fingerprints based on the file contents, without any transformations

• XML: normalizes the XML contents to remove whitespace, and normalize namespace abbreviations

• XSD: same as XML, plus inlines XSD imports and includes

• WSDL: same as XSD, plus inlines WSDL imports and includes, removes documentation, and orders WSDL elements by name

• WSDL_ABSTRACTING: same as WSDL, plus removes endpoint and binding information

Example

An example that describes calculating the SFID programmatically is as follows:

ArtifactFingerprinter fp = ArtifactAlgorithm.WSDL.newFingerprinter();

String sfid = fp.getFingerprint(new File("mytest.wsdl"));

13.8 SFID Limitations

SFID System Limitations:

• The file must be accessible to generate the SFID.

  The client machine/user must be provided with a path (URI) to the file. The URI must be accessible from the client system, without the need for user intervention, such as requests for authentication information).

• SFIDs are associated with files (1:1) which are associated with assets (n:n)

• SFIDs can be added to files through the Asset Editor or through REX.

• SFID is not supported on Oracle JDeveloper. It is supported only for VS .Net and Eclipse.

Limitations of the Asset Editor (SFID generation)

• An SFID can be associated only with extractable files.

• An SFID can be generated only for files that are accessible to the client.

Limitations of the ANT Integration (SFID comparison)

• SFIDs can be generated for ALL accessible files (for comparison).