14 Data Migration and Dynamic Attribute Mapping between UIM and SmartSearch

This chapter describes how to perform data migration and Dynamic Attribute mapping from UIM to SmartSearch or OpenSearch NoSQL Database. This migration helps you to seamlessly integrate data between UIM and SmartSearch or OpenSearch while designing networks.

Prerequisites

Before you perform SmartSearch data migration, ensure the following are performed:

  • Cancel or complete all BIs that are associated with devices, connectivities, and pipes.
  • Upgrade UIM.
  • Ensure that SmartSearch and OpenSearch are deployed and are running with Authentication disabled. For more information, see "Deploying OpenSearch and OpenSearch Dashboard" and "Deploying SmartSearch".
  • Add lang_pipeline in OpenSearch. For more information, see "Creating Ingest Pipeline for OpenSearch".
  • Create all required indexes or metadata in OpenSearch. For more information, see "Installing OpenSearch" and "Creating SmartSearch Index and Metadata".
  • Update the hostAliases section according to your SmartSearch and OpenSearch service setup. Add the target machine details in the /etc/host file as follows:
    <IP ADDRESS>  <instance>.<project>.<hostSuffix> t3.<instance>.<project>.<hostSuffix> admin.<instance>.<project>.<hostSuffix> <instance>.<project>.topology.<hostSuffix>
  • Verify your access to the service URL and database connection as follows:
    1. Open a command line window and login to SQL*Plus for the UIM database.
    2. Verify if UIM schema connection is successful using SQL*Plus.
    3. Ensure that you have admin or system access for UIM schema.
    4. Open a web browser and verify if you can access the SmartSearch API URL: https://<instance>.<project>.topology.<hostSuffix>:<PORT>/<API-VERSION>/index/smartsearch-location.
    5. Verify if JAVA_HOME and PATH are set.
    6. If SmartSearch is SSL enabled, use the Import SSL certificate in JAVA to secure connection.
    7. Download the corresponding .pem file from target machine and run the following command in terminal to import certificate:
      keytool -import -alias common-cert -file commoncert.pem -storetype JKS -keystore $JAVA_HOME/jre/lib/security/cacerts

Running the SmartSearch Migration Script

When you perform ATA migration along with SmartSearch migration, and if you perform data migration from UIM to ATA, you can ignore running the SQL files from migration SQL script as MIG tables are already present in your UIM schema.

If ATA migration is done before you start SmartSearch migration, you must drop all the tables before running the scripts.

If you perform data migration from UIM to SmartSearch, run the migration script as follows:

  1. Go to $WORKSPACEDIR/ata-builder/migration_scripts/Char_Mig_tables directory.

    You can see the following SQL files:

    1. CREATE_CHAR_MIG_TABLE.sql
    2. MIGRATION_CHAR1.sql
    3. MIGRATION_CHAR2.sql
    4. MIGRATION_CHAR3.sql
  2. Run the SQL scripts in UIM schema in the above order.
  3. Make a new directory: 
    mkdir $WORKSPACEDIR/<directory_name>
  4. Copy $WORKSPACEDIR/uim-image-builder/staging/cnsdk/uim-db/staging/uim-db-installer/ora_uim_dbtools.jar into the newly created directory.
  5. Extract the jar file as follows:
    jar -xvf ora_uim_dbtools.jar
  6. Modify smartSearchMappings.json for dynamic attribute mapping and save it.

    Note:

    Dynamic attribute mappings support mapping of vendor-specific characteristic for Physical Device. You can map only one characteristic from the Physical Device entity to the vendor field in the OpenSearch index, using the mappings. The equivalent vendor characteristic mappings must be present in topologySpecificationMapping.json.
  7. Modify the required parameters in the smartSearchMigration.sh file. For more information, see "Post-migration Check".
  8. Grant the execute permissions to the smartSearchMigration.sh file. If you do not have modify permissions, update the required parameters and save the file as follows:
    chmod +x smartSearchMigration.sh
  9. Run the migration command using: 
    ./smartSearchMigration.sh UIMToSmartSearch

    Note:

    You must wait until the migration script is complete. Or, if you want more options, run ./smartSearchMigration.sh help.

If the migration script fails, and you want to perform the migration again, delete the temp tables as follows: 

./smartSearchMigration.sh deleteTable force

After a table is deleted, run the following command to restart the migration:

./smartSearchMigration.sh UIMToSmartSearch

Running the SmartSearch Release Migration Script

Use this procedure when upgrading an environment in which UIM data has already been migrated to SmartSearch. The procedure migrates the SmartSearch index data changes that are required by the target UIM release.

Prerequisites for Running the SmartSearch Release Migration Script

Before running the SmartSearch release migration:

  • 1. Cancel or complete all business interactions that are associated with devices, connectivities, and pipes.
  • Upgrade UIM to the target release.
  • Upgrade the SmartSearch service to the target release.
  • Create or upgrade the OpenSearch schema, indexes, mappings, and metadata for the target release.
  • Ensure that SmartSearch and OpenSearch are deployed, running, and accessible from the host on which you run the migration script. Ensure that the SmartSearch API URL and the UIM schema connection are valid.
  • Ensure that SmartSearch and OpenSearch authentication is disabled before running the migration. Enable authentication after the migration and validation are complete.

Preparing the SmartSearch Release Migration Script

To prepare the SmartSearch release migration script:

  1. Create a directory for the UIM database tools:
    mkdir $WORKSPACEDIR/<directory_name>
  2. Copy the UIM database tools archive for the target release to the new directory:
    cp $WORKSPACEDIR/uim-image-builder/staging/cnsdk/uim-db/staging/uim-db-installer/ora_uim_dbtools.jar $WORKSPACEDIR/<directory_name>
  3. Change to the new directory and extract the archive:
    cd $WORKSPACEDIR/<directory_name>
jar -xvf ora_uim_dbtools.jar
  4. Open smartSearchReleaseMigration.sh in a terminal or text editor. Set the following required parameters:
    export JAVA_HOME=<Java_home>
export DB_HOSTNAME=<database_host>
export DB_PORT=<database_port>
export DB_SERVICE_NAME=<database_service_name>
export SMART_SEARCH_BASE_URL=<SmartSearch_URL>
export LANG=<language>
export TENANT_ID=<tenant_ID>

    The script prompts for the UIM schema user name and password when they are not supplied. You can also set DB_USER_NAME and DB_PASSWD.

  5. Identify the exact UIM release version from which you are upgrading and the exact target UIM release version. Use these values as <source_release_version> and <target_release_version> in the commands in this section.

    Specify the release version in canonical form (X.X.X.X.X). For example, 8.26.0.0.0.

Running the SmartSearch Release Migration Script

To view the SmartSearch migrations that apply to the release range, run:

./smartSearchReleaseMigration.sh ListEligibleMigrations <source_release_version> <target_release_version>

Review the displayed migrations and their status before continuing.

To run the pending SmartSearch migrations, run:

./smartSearchReleaseMigration.sh RunEligibleMigrations <source_release_version> <target_release_version>

Do not interrupt the command while the migration is running.

After the migration completes, run the migration report:

./smartSearchReleaseMigration.sh MigrationReport <source_release_version> <target_release_version>

Confirm that the report shows the applicable SmartSearch migrations as complete. If a migration is incomplete or failed, correct the reported issue before continuing.

Resuming or Restarting a SmartSearch Release Migration

If a migration stops before completion, first resolve the underlying issue. For example, verify the database and SmartSearch connection settings, credentials, and service availability.

To continue an incomplete migration after resolving the issue, run:

./smartSearchReleaseMigration.sh ResumeEligibleMigrations <source_release_version> <target_release_version>

Use the cleanup command only when you must restart an incomplete migration. The command removes the incomplete migration run data for the specified release range so that the migration can run again.

./smartSearchReleaseMigration.sh CleanEligibleMigrations <source_release_version> <target_release_version> force

The force argument is required. After cleanup completes, run RunEligibleMigrations again for the same release range.

Troubleshooting the SmartSearch Release Migration Script

If the script does not run or a migration fails, check the following items:

  • Verify that JAVA_HOME, the database connection settings, SMART_SEARCH_BASE_URL, LANG, and TENANT_ID are set correctly.
  • Verify that the UIM database and SmartSearch service are available from the host on which you run the script.
  • Verify that <source_release_version> is the exact version of the installed release and that <target_release_version> is the exact version of the target release.
  • Review the command output for the reported error, correct the issue, and then use ResumeEligibleMigrations.
  • Run MigrationReport to review the status before resuming or restarting a migration.

Do not use CleanEligibleMigrations as the first recovery action. Use it only after the underlying issue is resolved and you need to restart an incomplete migration.

Post-migration Check

Perform the following after a successful data migration to SmartSearch:

  1. Verify or search the following information in the console log:
    **** After Json Data Processed Completed *****
           Total Batch Size: 8
           Total Completed Batch Size: 8
           Total Pending And Failed Batch Size: 0
   **** Data Migration from UIM to Open Search Successfully Completed! *****

    Note:

    The Total Pending and Failed Batch Size value is Zero if the data migration is successful.
  2. Contact your administrator to enable Authentication for SmartSearch API and OpenSearch database.
  3. After enabling Authentication, open NPD to verify the search functionality for the corresponding entities such as logical device, physical device, equipment, and so on.

The sample file is as follows:

[
  {
    "name": "Ciena 3928 PD",
    "entityType": "PhysicalDeviceSpecificationDAO",
    "characteristics": [
      {
        "name": "vendor",
        "smartSearchName": "vendor"
      }
    ]
  },
  {
    "name": "Nokia 1800 PD",
    "entityType": "PhysicalDeviceSpecificationDAO",
    "characteristics": [
      {
        "name": "vendor",
        "smartSearchName": "vendor"
      }
    ]
  },
  {
    "name": "Ciena 3928 PD",
    "entityType": "PhysicalDeviceSpecificationDAO",
    "relatedVertices": [
      {
        "vertex": "vendor",
        "value": "Ciena",
        "smartSearchName": "vendor"
      }
    ],
    "characteristics": []
  }
]

The relatedVertices and characteristics sections in the mapping are mutually exclusive. Meaning, you should use only one of them, not both. If both sections are provided, the characteristics section overrides the relatedVertices section.

The relatedVertices section specifies a default value for a characteristic in the mapping, than searching for it from the database. For example, in the above sample, physical devices with the specification Ciena 3928 PD always have Ciena as the value in the vendor field in the index. This ensures that all physical devices with the Ciena 3928 PD specification have Ciena as their vendor in the NPD search.

Table 14-1 Dynamic Attributes

Dynamic Attribute Description
name Name of the entity specification on which the characteristic to be mapped exists. This is a mandatory attribute.
entityType Entity specification DAO. This is a mandatory attribute. The only valid value is PhysicalDeviceSpecificationDAO.
characteristics.name Name of the characteristic that you plan to map with the index attribute.
characteristics.smartSearchName Attribute name of the index where dynamic attribute value is stored in OpenSearch index.
relatedVertices.vertex The relatedVertices.vertex value from the ATA mappings. The only supported value is vendor.
relatedVertices.value Manually-entered value for characteristics that will persist in attribute of the index.
relatedVertices.smartSearchName Attribute name of index where dynamic attribute value is stored in OpenSearch index.

Open the smartSearchMigration.sh file in terminal or in a text editor, modify the required attributes values, and save the file. The sample file is as follows: 

export SMART_SEARCH_BASE_URL="";  #BASE URL format like http or https://<host>:<port>/<api_version>
   export LANG=""; # for example en
   export TENANT_ID=""; # for example tenant1
   export LIMIT=""; #for example 10000
   export BATCH_SIZE_LIMIT=""; # for example 10
   export BULK_LOAD_TIMEOUT="20";   # Maximum processing time per batch operation (seconds)
   export INITIAL_DELAY="1";       # Initial waiting time between batch operations (seconds)
   export BACK_OFF_FACTOR="2.0";    # Multiplier to increase retry delay time
   export MAX_DELAY="180";          # Maximum Delay allowed between retries of one batch (seconds)
   export JAVA_HOME="";
   export DB_HOSTNAME="";
   export DB_PORT="";
   export DB_SERVICE_NAME="";
   export DB_USER_NAME="";    # user details left as blank for production environment once prompt then you can provide
   export DB_PASSWD=""; # password details left as blank for production environment once prompt then you can provide