Upgrading Unified Assurance to a New Linux System
You can upgrade Oracle Communications Unified Assurance onto a new Linux 8 system by first installing the new Linux 8 environment, and then synchronizing and updating your Unified Assurance installation onto the new system. This is also known as a forklift upgrade.
About the Forklift Upgrade Procedure
Review the details in this section before beginning the upgrade procedure.
You can upgrade to Unified Assurance version 6.1 on Linux 8 from either version 5 or version 6.0.X. Most of the steps in the upgrade procedure apply to both upgrade paths. If a step only applies to one upgrade path or is different depending on your upgrade path, the relevant upgrade path is specified.
Note:
The upgrade procedure describes upgrading from Linux 7 to Linux 8. The same procedure applies if you are upgrading from Linux 6. For simplicity, only Linux 7 is mentioned in the steps.
The upgrade also involves migrating the Historical database from Elasticsearch to OpenSearch. For this upgrade and migration to succeed, the old and new presentation servers and Historical database servers must have full and unrestricted network access to each other, with no firewall rules or security policies that block or deny traffic between the systems. Oracle recommends having the old and new servers on the same network.
This access is essential for transferring data between Elasticsearch and the new OpenSearch database.
Forklift Upgrade Overview
When performing a forklift upgrade, you follow the process from beginning to end for each server, one server at a time. Complete the entire process for one server before moving on to the next server.
The high level steps are:
-
Installing a new Linux 8 environment. For information about installing Linux 8, see Oracle Linux 8 Installing Oracle Linux.
-
Performing upgrade prerequisites. See Forklift Upgrade Prerequisites.
-
Upgrading the server. This involves the following steps, completed from beginning to end for one server at a time before moving on to the next server:
-
Preparation steps, depending on the roles installed on the server.
-
Backing up the Unified Assurance directory on the old server and moving it to the new one.
-
Installing the latest Unified Assurance 6.1 RPM on the new server.
Note:
You must download the latest release package from Oracle Software Delivery Cloud and the latest patch from My Oracle Support. See Downloading the Latest Unified Assurance Software for instructions.
-
Post RPM installation steps, depending on the roles installed on the server.
-
Updating the package.
-
Post package update steps, depending on the roles installed on the server.
See Performing the Forklift Upgrade for complete details.
-
-
Performing post-upgrade cleanup tasks. See Post-Upgrade Tasks.
Forklift Upgrade Active User
You run most commands in this procedure as the root user. For some steps, such as backing up and restoring the Graph database, you will switch to the assure1 user. These switches are specified in the commands.
Forklift Upgrade Placeholders
The procedures use a number environment variables, defined by placeholders marked with angled brackets in the commands (<>). The following table describes them. The procedures instruct you to set the environment variables at the appropriate time.
Environment Variable | Placeholder | Description |
---|---|---|
$A1BASEDIR | <UA_home> | Your Unified Assurance home directory. This is typically /opt/assure1, unless you chose a custom directory at installation time. |
$FILESDIR | <installation_directory> | The directory to store installation files, also known as your local depot. For example, /opt/install. |
$OL7FQDN | <Linux_7_FQDN> | The fully-qualified domain name (FQDN) of the original Linux 7 Unified Assurance server that you upgraded from. In a single server environment, this will be the same for all servers. In a multi-server environment, this may be different depending on the server. |
$OL8FQDN | <Linux_8_FQDN> | The FQDN of the new Linux 8 Unified Assurance server that you updated to. In a single server environment, this will be the same for all servers. In a multi-server environment, this may be different depending on the server. |
$PRIWEBFQDN | <Primary_Web_FQDN> | The web FQDN of the new Linux 8 primary presentation server (the same as secondary, if the WebFQDN is shared). |
$SECWEBFQDN | <Secondary_Web_FQDN> | The web FQDN of the new Linux 8 secondary presentation server (the same as primary, if the WebFQDN is shared). |
$EXTWEBFQDN | <External_Web_FQDN> | The web FQDN of the new Linux 8 external presentation server. This is different from the web FQDN of the primary and secondary presentation servers. |
$RPMVERSION | <RPM_version> | The version number of the latest Unified Assurance RPM, not including the package number or el8. For example, 6.1.0.0.0 |
For example, to set these environment variables:
export A1BASEDIR=/opt/assure1
export FILESDIR=/opt/install
export OL7FQDN=primary.linux7.server.fqdn
export OL8FQDN=primary.linux8.server.fqdn
export PRIWEBFQDN=web.server.example.com
export SECWEBFQDN=web.server2.example.com
export EXTWEBFQDN=external.server.example.com
export RPMVERSION=6.1.0.0.0
Server Upgrade Order
Perform the upgrade on each server in the following order, completing the entire process for one server before moving on to the next server:
-
The primary presentation server.
-
The secondary presentation server.
-
Each database server pair, starting with primary, then secondary.
To ensure minimal downtime, upgrade each database server one pair at a time. For example, if you have two database server pairs, upgrade them in this order:
-
Pair 1: Primary database server
-
Pair 1: Secondary database server
-
Pair 2: Primary database server
-
Pair 2: Secondary database server
-
-
Each collection server.
-
Each external presentation server.
When you have upgraded all your servers, perform the tasks described in Post-Upgrade Tasks.
Note:
Servers can have multiple roles. Perform the upgrade in the order of the role that appears first in the upgrade order list. For example, if your primary presentation server also has the Database.Graph role, upgrade it first, performing all role-specific steps for both roles, before upgrading your secondary presentation server and any other database servers.
Forklift Upgrade Prerequisites
Before starting a forklift upgrade:
-
Set up your Linux 8 servers and prepare them for the Unified Assurance installation. See Linux Prerequisites for information about preparing your Linux system for a Unified Assurance installation or upgrade.
-
For the upgrade and migration of the Historical database server to succeed, the old and new presentation servers and Historical database servers must have full and unrestricted network access to each other, with no firewall rules or security policies that block or deny traffic between the systems.
This is essential for transferring data between Elasticsearch and the new OpenSearch database.
-
Get the latest software release package and patch:
-
On the new Linux 8 primary presentation server, download the latest Unified Assurance software, including the release package and the latest available patch, and extract the contents of the collection files to an installation directory. This is typically /opt/install, and is referred to as <installation_directory> in the documentation.
See Downloading the Latest Unified Assurance Software for more information about where to get the software and how to extract the collection file.
-
Copy the Assure1-6.X.X.X.X-X.el8.x86_64.rpm file for the latest patch from the primary presentation server to <installation_directory> on each other Linux 8 server in the environment.
-
-
If your existing Linux 7 system uses Keepalived, disable and stop the service. You can configure and enable the service again once the upgrade has been completed.
To check keepalived status:
systemctl status keepalived
To stop keepalived:
systemctl stop keepalived
-
Verify that your existing Linux 7 primary and secondary databases are synchronized.
-
On servers with the Database.Historical role, confirm that you will have enough memory available. OpenSearch requires at least 2GB, which is approximately the same amount of memory as Elasticsearch. During OpenSearch installation, the heap size is automatically set to the same heap size as Elasticsearch.
If you do not have enough memory, Package update will fail and prompt you to adjust your memory settings. On the pre-migration system, this is set in the ES_JAVA_OPTS parameter in the $A1BASEDIR/vendor/elasticsearch/config/custom-env file, which you will back up to $A1BASEDIR/tmp/elasticsearch/custom-env during the upgrade process.
For complete information about general system requirements for Unified Assurance, including system and hardware requirements, see Unified Assurance Architecture.
Performing the Forklift Upgrade
After performing Forklift Upgrade Prerequisites, complete all upgrade steps for a single server, in the order specified in Server Upgrade Order, before restarting the steps for the next server. If the server has multiple roles, you may follow multiple preparation, post RPM installation, and post package update procedures.
As an overview, the upgrade process involves:
-
Preparing the server, depending on the roles installed on it:
-
Preparing Primary Database.Assure1 Servers: These servers will also have at least the Presentation.Internal role.
-
Preparing Secondary Database.Assure1 Servers: These servers will also have at least the Presentation.Internal role.
There are no required preparation steps for other roles.
-
-
Performing post RPM installation steps for the server, depending on the roles installed on it:
-
Post RPM Installation Steps for Primary Presentation.Internal Servers
-
Post RPM Installation Steps for Secondary Presentation.Internal Servers
-
Post RPM Installation Steps for Primary Database Servers (any Database subroles)
-
Post RPM Installation Steps for Secondary Database Servers (any Database subroles)
-
Post RPM Installation Steps for Primary Presentation.External Servers
-
Post RPM Installation Steps for Secondary Presentation.External Servers
-
-
Performing post package update steps for the server, depending on the roles installed on it:
After all servers have been upgraded, perform Post-Upgrade Tasks.
Preparing Servers
Complete all preparation steps for a single server, depending on the roles installed on it. Then complete all subsequent upgrade steps for the same server before starting again with the next server specified in the order Server Upgrade Order.
If there are multiple roles on this server, you may have to refer to multiple preparation step sections.
Preparing Primary Database.Assure1 Servers
These servers will also have the Presentation.Internal role.
If you perform the upgrade during a maintenance window and can accept some downtime for the Assure1 database, skip these preparation steps.
Optionally, if you are upgrading from version 6.0.4 or later to 6.1, and you require minimal database downtime, you can temporarily use your old and new servers with the Database.Assure1 role as replication partners during the upgrade. Because at one point in the upgrade, you would be replicating data from an upgraded server back to a non-upgraded server, this is not supported for version numbers that involve major schema changes.
If you are upgrading from any other version, including version 5, these steps are not supported.
To prepare the primary presentation server for the temporary replication:
-
On the Linux 7 primary presentation server:
-
Connect to the MySQL Assure1 database as the root user.
-
Get the binlog file name and position. You will use these values later in the upgrade process.
Run one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0, run the following command and save the values for Relay_Master_Log_File and Exec_Master_Log_Pos:
SHOW SLAVE STATUS \G
The values from your Linux 7 primary presentation server will be referred to later as:
-
<Pri_L7_Pres_Relay_Master_Log_File>
-
<Pri_L7_Pres_Exec_Master_Log_Pos>
-
-
For MySQL versions 8.4.0 or greater, run the following command and save the values for Relay_Source_Log_File and Exec_Source_Log_Pos:
SHOW REPLICA STATUS \G
The values from your Linux 7 primary presentation server will be referred to later as:
-
<Pri_L7_Pres_Relay_Source_Log_File>
-
<Pri_L7_Pres_Exec_Source_Log_Pos>
-
-
-
Stop replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE;
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA;
-
-
-
On the Linux 7 secondary presentation server:
-
Connect to the MySQL Assure1 database as the root user.
-
Get the binlog file name and position. You will use these values later in the upgrade process.
Run one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0, run the following command and save the values for Relay_Master_Log_File and Exec_Master_Log_Pos:
SHOW SLAVE STATUS \G
The values from your Linux 7 secondary presentation server will be referred to later as:
-
<Sec_L7_Pres_Relay_Master_Log_File>
-
<Sec_L7_Pres_Exec_Master_Log_Pos>
-
-
For MySQL versions 8.4.0 or greater, run the following command and save the values for Relay_Source_Log_File and Exec_Source_Log_Pos:
SHOW REPLICA STATUS \G
The values from your Linux 7 secondary presentation server will be referred to later as:
-
<Sec_L7_Pres_Relay_Source_Log_File>
-
<Sec_L7_Pres_Exec_Source_Log_Pos>
-
-
-
Stop replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE;
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA;
-
-
-
If this server has any additional roles, complete any of the following relevant preparation steps:
If this server has no other roles, proceed to Backing Up and Moving the Unified Assurance Directory.
Preparing Secondary Database.Assure1 Servers
These servers will also have the Presentation.Internal role.
Before starting these steps, you must have completed the upgrade on your primary presentation server. Verify that all Unified Assurance services are running and all data collection is happening on the Linux 8 primary presentation server.
-
If you performed the optional zero-downtime steps in Preparing Primary Database.Assure1 Servers, skip to the next step.
If you are upgrading in a maintenance window, on the Linux 7 secondary presentation server, reset replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0:
RESET SLAVE ALL
-
For MySQL versions 8.4.0 and greater:
RESET REPLICA ALL
-
-
If you did not perform the optional zero-downtime steps in Preparing Primary Database.Assure1 Servers, skip to the next step.
If you performed the optional zero-downtime steps for the primary presentation server:
-
On the Linux 7 secondary presentation server, connect to the MySQL Assure1 database as the root user.
-
Get the binlog file name and position. You will use these values later in the upgrade process.
Run one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0, run the following command and save the values for Relay_Master_Log_File and Exec_Master_Log_Pos:
SHOW SLAVE STATUS \G
The values from your Linux 7 secondary presentation server will be referred to later as:
-
<Sec_L7_Pres_Relay_Master_Log_File>
-
<Sec_L7_Pres_Exec_Master_Log_Pos>
The values from your Linux 8 primary presentation server will be referred to later as:
-
<Pri_L8_Pres_Relay_Master_Log_File>
-
<Pri_L8_Pres_Exec_Master_Log_Pos>
-
-
For MySQL versions 8.4.0 or greater, run the following command and save the values for Relay_Source_Log_File and Exec_Source_Log_Pos:
SHOW REPLICA STATUS \G
The values from your Linux 7 secondary presentation server will be referred to later as:
-
<Sec_L7_Pres_Relay_Source_Log_File>
-
<Sec_L7_Pres_Exec_Source_Log_Pos>
The values from your Linux 8 primary presentation server will be referred to later as:
-
<Pri_L8_Pres_Relay_Source_Log_File>
-
<Pri_L8_Pres_Exec_Source_Log_Pos>
-
-
-
Stop replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE;
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA;
-
-
On the Linux 8 primary presentation server, connect to the MySQL Assure1 database as the root user.
-
Get the binlog file name and position. You will use these values later in the upgrade process.
Run one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0, run the following command and save the values for Relay_Master_Log_File and Exec_Master_Log_Pos:
SHOW SLAVE STATUS \G
The values from your Linux 8 primary presentation server will be referred to later as:
-
<Pri_L8_Pres_Relay_Master_Log_File>
-
<Pri_L8_Pres_Exec_Master_Log_Pos>
-
-
For MySQL versions 8.4.0 or greater, run the following command and save the values for Relay_Source_Log_File and Exec_Source_Log_Pos:
SHOW REPLICA STATUS \G
The values from your Linux 8 primary presentation server will be referred to later as:
-
<Pri_L8_Pres_Relay_Source_Log_File>
-
<Pri_L8_Pres_Exec_Source_Log_Pos>
-
-
-
Stop replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE;
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA;
-
-
-
If this server as any additional roles, complete any of the following relevant preparation steps:
If this server has no other roles, proceed to Backing Up and Moving the Unified Assurance Directory.
Preparing Primary Database.Event Servers
If you perform the upgrade during a maintenance window and can accept some downtime for the Event database, skip these preparation steps.
Optionally, if you are upgrading from version 6.0.4 or later to 6.1, and you require minimal database downtime, you can temporarily use your old and new servers with the Database.Event role as replication partners during the upgrade for minimal downtime. Because at one point in the upgrade you would be replicating data from an upgraded server back to a non-upgraded server, this is not supported for version numbers that involve major schema changes.
If you are upgrading from any other version, including version 5, these steps are not supported.
To prepare the primary Database.Event server for the temporary replication:
-
On the Linux 7 primary Database.Event server:
-
Connect to the MySQL Event database as the root user.
-
Get the binlog file name and position. You will use these values later in the upgrade process.
Run one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0, run the following command and save the values for Relay_Master_Log_File and Exec_Master_Log_Pos:
SHOW SLAVE STATUS \G
The values from your Linux 7 primary Database.Event server will be referred to later as:
-
<Pri_L7_DB_Relay_Master_Log_File>
-
<Pri_L7_DB_Exec_Master_Log_Pos>
-
-
For MySQL versions 8.4.0 or greater, run the following command and save the values for Relay_Source_Log_File and Exec_Source_Log_Pos:
SHOW REPLICA STATUS \G
The values from your Linux 7 primary Database.Event server will be referred to later as:
-
<Pri_L7_DB_Relay_Source_Log_File>
-
<Pri_L7_DB_Exec_Source_Log_Pos>
-
-
-
Stop replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE;
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA;
-
-
-
On the Linux 7 secondary Database.Event server:
-
Connect to the MySQL Event database as the root user.
-
Get the binlog file name and position. You will use these values later in the upgrade process.
Run one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0, run the following command and save the values for Relay_Master_Log_File and Exec_Master_Log_Pos:
SHOW SLAVE STATUS \G
The values from your Linux 7 secondary Database.Event server will be referred to later as:
-
<Sec_L7_DB_Relay_Master_Log_File>
-
<Sec_L7_DB_Exec_Master_Log_Pos>
-
-
For MySQL versions 8.4.0 or greater, run the following command and save the values for Relay_Source_Log_File and Exec_Source_Log_Pos:
SHOW REPLICA STATUS \G
The values from your Linux 7 secondary Database.Event server will be referred to later as:
-
<Sec_L7_DB_Relay_Source_Log_File>
-
<Sec_L7_DB_Exec_Source_Log_Pos>
-
-
-
Stop replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE;
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA;
-
-
-
If this server has any additional roles, complete any of the following relevant preparation steps:
If this server has no other roles, proceed to Backing Up and Moving the Unified Assurance Directory.
Preparing Secondary Database.Event Servers
Before starting these steps, you must have completed the upgrade on your primary Database.Event server. Verify that all Unified Assurance services are running and all data collection is happening on the Linux 8 primary Database.Event server.
-
If you performed the optional zero-downtime steps in Preparing Primary Database.Event Servers, skip to the next step.
If you are upgrading in a maintenance window, on the Linux 7 secondary database server, reset replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0:
RESET SLAVE ALL
-
For MySQL versions 8.4.0 and greater:
RESET REPLICA ALL
-
-
If you did not perform the optional zero-downtime steps in Preparing Primary Database.Event Servers, skip to the next step.
If you performed the optional zero-downtime steps for the primary Database.Event server:
-
On the Linux 7 secondary Database.Event server, connect to the MySQL Event database as the root user.
-
Get the binlog file name and position. You will use these values later in the upgrade process.
Run one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0, run the following command and save the values for Relay_Master_Log_File and Exec_Master_Log_Pos:
SHOW SLAVE STATUS \G
The values from your Linux 7 secondary Database.Event server will be referred to later as:
-
<Sec_L7_DB_Relay_Master_Log_File>
-
<Sec_L7_DB_Exec_Master_Log_Pos>
-
-
For MySQL versions 8.4.0 or greater, run the following command and save the values for Relay_Source_Log_File and Exec_Source_Log_Pos:
SHOW REPLICA STATUS \G
The values from your Linux 7 secondary Database.Event server will be referred to later as:
-
<Sec_L7_DB_Relay_Source_Log_File>
-
<Sec_L7_DB_Exec_Source_Log_Pos>
-
-
-
Stop replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE;
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA;
-
-
On the Linux 8 primary Database.Event server, connect to the MySQL Event database as the root user.
-
Get the binlog file name and position. You will use these values later in the upgrade process.
Run one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0, run the following command and save the values for Relay_Master_Log_File and Exec_Master_Log_Pos:
SHOW SLAVE STATUS \G
The values from your Linux 8 primary Database.Event server will be referred to later as:
-
<Pri_L8_DB_Relay_Master_Log_File>
-
<Pri_L8_DB_Exec_Master_Log_Pos>
-
-
For MySQL versions 8.4.0 or greater, run the following command and save the values for Relay_Source_Log_File and Exec_Source_Log_Pos:
SHOW REPLICA STATUS \G
The values from your Linux 8 primary Database.Event server will be referred to later as:
-
<Pri_L8_DB_Relay_Source_Log_File>
-
<Pri_L8_DB_Exec_Source_Log_Pos>
-
-
-
Stop replication by running one of the following commands, depending on your MySQL version:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE;
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA;
-
-
-
If this server also has the Database.Historical role, proceed to Preparing Database.Historical Servers.
If it does not, proceed to Backing Up and Moving the Unified Assurance Directory.
Preparing Primary Database.Graph Servers
The following steps apply before upgrading your primary Database.Graph server in redundant environments only. Do not repeat them when upgrading your secondary Database.Graph server. They do not apply in non-redundant environments.
The first step is required for all redundant environments.
The second step is optional, if you require minimal database downtime and cannot use a maintenance window. It is only supported if your version of Unified Assurance on Linux 7 is 6.0.4 or later.
-
On your Linux 7 primary and secondary database servers, disable Neo4j replication by moving the graph-repl.jar and schema-repl.jar files to the $A1BASEDIR/vendor/neo4j/disabled_plugins/ directory:
mv $A1BASEDIR/vendor/neo4j/plugins/graph-repl.jar $A1BASEDIR/vendor/neo4j/disabled_plugins/ mv $A1BASEDIR/vendor/neo4j/plugins/schema-repl.jar $A1BASEDIR/vendor/neo4j/disabled_plugins/
This prevents issues resulting from the old plugins from being run alongside the new plugins.
Note:
Depending on the version of Unified Assurance you have installed, graph-repl.jar and schema-repl.jar may no longer exist or may have already been moved to a backup directory. If so, continue to the next step.
-
If your version of Unified Assurance on Linux 7 is less than 6.0.4, or you are performing the upgrade during a maintenance window and can accept some downtime for the Graph database, skip these steps.
If your version of Unified Assurance on Linux 7 is 6.0.4 or higher, and you require minimal database downtime, you can keep Neo4j running on the secondary database server while you upgrade the primary, then back up the secondary Neo4j database and restore it onto the primary after it has been upgraded.
To do this, complete the following preparation steps on both the Linux 7 primary and secondary database servers:
-
In the $A1BASEDIR/etc/neo4j/streams.conf file, change the value of the streams.source.enabled property to false. For example:
streams.source.enabled=false
-
Disable and stop the Neo4j Kafka Connect service:
-
In the Unified Assurance UI, from the Configuration menu, select Broker Control, then select Services.
-
Select the Neo4j Kafka Connect service.
The service details appear.
-
Set Status to Disabled and click Submit.
In the All Services table, the value in the Status column for the service is updated to Disabled.
-
In the All Services table, click Stop.
In the All Services table, the value in the State column for the service is updated to Stopped.
-
-
Verify that the Neo4j processes have stopped:
ps -ef | grep neo4j
The output should not show any running Neo4j processes.
-
On your Linux 7 secondary database server, restart the Neo4j service:
-
In to the Unified Assurance UI, from the Configuration menu, select Broker Control, then select Services.
-
Select the Neo4j service and click Restart.
-
-
-
If this server also has the Database.Historical role, proceed to Preparing Database.Historical Servers. If it does not, proceed to Backing Up and Moving the Unified Assurance Directory.
Preparing Database.Historical Servers
The following steps apply to all Database.Historical servers, including primary and secondary servers, and servers in non-redundant environments.
Complete all preparation steps for a single server, then complete all next upgrade steps for that server before starting again with the next server specified in the order Server Upgrade Order.
-
If this is a secondary Database.Historical server, confirm that redundancy is correctly configured:
-
In the Unified Assurance user interface, from the main navigation menu, select Configuration, then Broker Control, and then Servers.
-
Select the secondary Database.Historical server and confirm that the correct FQDN for the primary Database.Historical server is set in the Primary Server field.
If it is not, update the Primary Server field, and then click Submit.
-
-
On the primary presentation server, run the following commands to back up the Kibana configuration file to a temporary location for later use during migration:
mkdir -p $A1BASEDIR/tmp/kibana cp -r $A1BASEDIR/vendor/kibana/config/kibana.yml $A1BASEDIR/tmp/kibana
-
On the Historical database server:
-
Back up the Elasticsearch files and SSL certificates to a temporary location for later use during migration:
mkdir -p $A1BASEDIR/tmp/elasticsearch/certs cp -r $A1BASEDIR/vendor/elasticsearch/config/custom-env $A1BASEDIR/tmp/elasticsearch cp $A1BASEDIR/etc/ssl/BundleCA.crt $A1BASEDIR/tmp/elasticsearch/certs cp $A1BASEDIR/etc/ssl/User-kibana.crt $A1BASEDIR/tmp/elasticsearch/certs cp $A1BASEDIR/etc/ssl/User-kibana.key $A1BASEDIR/tmp/elasticsearch/certs cp $A1BASEDIR/etc/ssl/User-assure1.crt $A1BASEDIR/tmp/elasticsearch/certs cp $A1BASEDIR/etc/ssl/User-assure1.key $A1BASEDIR/tmp/elasticsearch/certs
Note:
You must use these directory names, and not any other temporary directory. The migration scripts look for files in these specific locations.
-
In the $A1BASEDIR/tmp/elasticsearch directory, create a file called info.json with the following content, replacing the placeholders as appropriate for your environment:
{ "ElasticsearchHostFQDN": "<Primary_Historical_Linux_7_Host_FQDN>", "KibanaHostFQDN": "<Primary_Presentation_Linux_7_Host_FQDN>", "CertsDirectory": "/tmp/elasticsearch/certs" }
Notes:
-
You must use $A1BASEDIR/tmp/elasticsearch/info.json, and not any other directory or file name. The migration scripts look for this specific file.
-
The migration scripts automatically construct the paths to the certificates based on the value of BaseDir in Assure1.conf and the value of CertsDirectory in info.json. Do not change the value of CertsDirectory.
-
If you installed the Database.Historical and Presentation.Internal roles on the same Linux 7 server, the values for ElasticsearchHostFQDN and KibanaHostFQDN will be the same.
-
-
-
Proceed to Backing Up and Moving the Unified Assurance Directory.
Backing Up and Moving the Unified Assurance Directory
After completing all preparation steps for the server you are currently upgrading, complete the following steps and all subsequent upgrade steps for the same server before starting again at Preparing Servers for the next server.
On your old Linux 7 server with any role:
-
Set environment variables, replacing <UA_home> with the appropriate value. This is usually opt/assure1:
export A1BASEDIR=<UA_home> source $A1BASEDIR/.assure1_bashrc
-
Stop all services and the broker:
$A1BASEDIR/bin/BrokerControl --batch stopall systemctl stop assure1-broker
-
For servers with a presentation role, stop additional components:
-
Presentation.Internal and Database.Assure1:
systemctl stop assure1-bus systemctl stop assure1-web systemctl stop assure1-db
-
Presentation.External:
systemctl stop assure1-web
-
-
Verify that all processes have stopped:
ps aux | grep assure1 systemctl status assure1-broker systemctl status assure1-bus systemctl status assure1-web systemctl status assure1-db
-
Create a file called tar-excludes.txt that specifies what to exclude from the backup:
cat <<EOM > tar-excludes.txt $A1BASEDIR/.config/helm $A1BASEDIR/distrib/packages $A1BASEDIR/etc/rke $A1BASEDIR/etc/telegraf.d/elasticsearch.conf $A1BASEDIR/var/chartmuseum $A1BASEDIR/var/checkouts $A1BASEDIR/var/docker $A1BASEDIR/var/elasticsearch $A1BASEDIR/var/rabbitmq $A1BASEDIR/var/registry $A1BASEDIR/var/rke $A1BASEDIR/var/run/docker $A1BASEDIR/var/run/docker.pid $A1BASEDIR/vendor/docker $A1BASEDIR/vendor/elasticsearch $A1BASEDIR/vendor/helm $A1BASEDIR/vendor/kibana $A1BASEDIR/vendor/kubernetes $A1BASEDIR/vendor/openssl $A1BASEDIR/vendor/perl $A1BASEDIR/vendor/rke $A1BASEDIR/vendor/submariner EOM
-
Back up the $A1BASEDIR directory:
tar --exclude-from=tar-excludes.txt --exclude-tag=Assure1CA.crt -czf UA-Linux7-<server_type>-Backup.tgz $A1BASEDIR
where <server_type> is the type of server you are backing up. For example, Primary-Presentation or Secondary-Database.
This may take a few minutes.
-
Copy the UA-Linux7-<server_type>-Backup.tgz tar backup file to the new Linux 8 server. You can use any secure transfer protocol. For example, using scp:
scp UA-Linux7-<server_type>-Backup.tgz <user>@<hostname>:/home/<user>/
where <user> is the user running the command and <hostname> is the Linux 8 server host.
Depending on your environment, <user> may be:
-
The opc user.
-
A different user, with the path to their private key specified in the command with the -i option.
-
-
Proceed to Installing the Latest RPM
Installing the Latest RPM
After backing up and moving the Unified Assurance directory for the server you are currently upgrading, complete the following steps and all subsequent upgrade steps for the same server before starting again at Preparing Servers for the next server.
On your new Linux 8 server with any role:
-
Move the UA-Linux7-<server_type>-Backup.tgz file to <installation_directory>:
mv /home/<user>/UA-Linux7-<server_type>-Backup.tgz <installation_directory>
-
Set environment variables, replacing the placeholders with the appropriate values:
export FILESDIR=<installation_directory> export OL7FQDN=<Linux_7_FQDN> export OL8FQDN=<Linux_8_FQDN> export PRIWEBFQDN=<Primary_Web_FQDN> export SECWEBFQDN=<Secondary_Web_FQDN>
where:
-
<Linux_7_FQDN> is the FQDN for the Linux 7 server you moved from
-
<Linux_8_FQDN> is the FQDN of the Linux 8 server you are working on
-
<Primary_Web_FQDN> is the web FQDN for Linux 8 primary presentation server. If you are currently working on the Linux 8 primary presentation server, $OL8FQDN and $PRIWEBFQDN may have the same value.
-
<Secondary_Web_FQDN> is the web FQDN for Linux 8 secondary presentation server. If you are currently working on the Linux 8 secondary presentation server, $OL8FQDN and $SECWEBFQDN may have the same value.
Note:
In a multiserver environment, be careful to set these to the correct FQDN for the current server you are working on.
-
-
Extract the UA-Linux7-<server_type>-Backup.tgz tar backup file:
tar -xzf $FILESDIR/UA-Linux7-<server_type>-Backup.tgz -C /
This may take a few minutes.
-
Set the environment variable for the Unified Assurance base directory:
export A1BASEDIR=<UA_home>
where <UA_home> is the directory that you backed up from your old server. This is usually /opt/assure1.
-
On the server you are currently upgrading, repeat the following commands for each server in the environment, including this one, to replace old server FQDNs with new FQDNs. Replace the placeholders as appropriate for each set of servers:
grep -rl --exclude-dir={etc,legal,lib,logs,tmp,var} "<Linux_7_FQDN>" $A1BASEDIR/* | xargs sed -i "s/<Linux_7_FQDN>/<Linux_8_FQDN>/g" grep -rl --exclude-dir={ssl,priv} "<Linux_7_FQDN>" $A1BASEDIR/etc/* | xargs sed -i "s/<Linux_7_FQDN>/<Linux_8_FQDN>/g"
For example, if you have an environment with one presentation server, one database server, and one collection server, you run the commands three times on the server you are currently upgrading: Once for the FQDNs of the presentation server, once for the FQDNs of the database server, and once for the FQDNs of the collection server.
-
Install the latest Unified Assurance RPM for Linux 8:
dnf -y install $FILESDIR/Assure1-6.X.X.X.X-X.el8.x86_64.rpm
Note:
You extracted this file on the primary presentation server from the collection file for the patch package, and copied it to all servers in Forklift Upgrade Prerequisites.
-
Create an environment variable for the RPM version number. Do not include the package number or el8, and replace each X with the appropriate number:
export RPMVERSION=6.X.X.X.X
For example:
export RPMVERSION=6.1.0.0.0
-
Add the primary and secondary server host FQDNs to the <UA_home>/etc/Assure1.conf file:
-
If you are working on an internal primary presentation server, add its host FQDN:
sed -i "s/\"DBHost\": \"\"/\"DBHost\": \"$PRIWEBFQDN\"/g" $A1BASEDIR/etc/Assure1.conf
-
If you are working on any other server, add the host FQDN for the Linux 8 primary and secondary presentation servers:
sed -i "s/\"DBHost\": \"\"/\"DBHost\": \"$PRIWEBFQDN\"/g" $A1BASEDIR/etc/Assure1.conf sed -i "s/\"DBBackupHost\": \"\"/\"DBBackupHost\": \"$SECWEBFQDN\"/g" $A1BASEDIR/etc/Assure1.conf
-
-
If you are upgrading from version 6.0.x to version 6.1, skip to the next step.
If you are upgrading from version 5 to version 6.1, add multi-domain (SAN) certificate information in the OpenSSL configuration.
cat <<'EOM' >>$A1BASEDIR/etc/openssl.conf [ san_cert ] basicConstraints = CA:FALSE nsComment = "OpenSSL Generated Certificate" subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer EOM
-
Confirm that the $A1BASEDIR/etc/ssl/priv directory exists, with empty index.txt, index.txt.attr, and serial files owned by the assure1 user:
ls -la $A1BASEDIR/etc/ssl/priv/
If the directory or files do not exist, create them as the assure1 user:
mkdir -p $A1BASEDIR/etc/ssl/priv touch $A1BASEDIR/etc/ssl/priv/index.txt touch $A1BASEDIR/etc/ssl/priv/index.txt.attr touch $A1BASEDIR/etc/ssl/priv/serial
-
Complete post-installation steps, depending on the roles of the server you are currently upgrading:
Post RPM Installation Steps
After installing the latest RPM on the server you are currently upgrading, complete the following steps, depending on the roles installed on the server, and all subsequent upgrade steps for the same server before starting again at Preparing Servers for the next server.
If there are multiple roles on this server, you may have to refer to multiple post-installation step sections.
Post RPM Installation Steps for Primary Presentation.Internal Servers
The following steps apply to the presentation server in non-redundant environments, or the primary presentation server in redundant environments.
-
On your new Linux 8 primary presentation server, generate new SSL certificates:
$A1BASEDIR/bin/CreateSSLCertificate --Type SiteCA $A1BASEDIR/bin/CreateSSLCertificate --Type Web --CN $PRIWEBFQDN
-
Check the contents of $A1BASEDIR/etc/ssl on the old Linux 7 server. If you had any custom certificate files, do the following on the new Linux 8 server:
-
Copy any non-Unified Assurance certificates to the new Linux 8 server:
scp $OL7FQDN:$A1BASEDIR/etc/ssl/<custom_cert>* $A1BASEDIR/etc/ssl/
where <custom_cert> is the name of the custom certificate.
-
Regenerate any custom user certificates, to ensure that they are signed by the new site certificate:
$A1BASEDIR/bin/CreateSSLCertificate --Type User --CN <user_name>
where <user_name> is the custom user to generate the certificate for.
-
-
Verify the certificate chain:
cd $A1BASEDIR/etc/ssl openssl verify -CAfile BundleCA.crt SiteCA.crt Host.crt User-api.crt User-assure1.crt User-repl.crt Vision.crt Web.crt
If you have any custom user certificates copied from the Linux 7 server, include them in the verification command.
The verification command should return OK for all certificates.
-
On the old Linux 7 primary presentation server, copy the following Unified Assurance systemd scripts from the /etc/systemd/system directory on the original Linux 7 server to the new Linux 8 server:
-
assure1-db.service
-
assure1-bus.service
-
assure1-web.service
For example, to use scp to transfer the files from the old Linux 7 server:
scp /etc/systemd/system/assure1-db.service <user>@<hostname>:/home/<user> scp /etc/systemd/system/assure1-bus.service <user>@<hostname>:/home/<user> scp /etc/systemd/system/assure1-web.service <user>@<hostname>:/home/<user>
where <user> is the user running the command and <hostname> is the Linux 8 server host.
Depending on your environment, <user> may be:
-
The opc user.
-
A different user, with the path to their private key specified in the command with the -i option.
-
-
On your new Linux 8 primary presentation server:
-
Move the systemd scripts from the /home/<user> directory to the /etc/systemd/system directory:
cp /home/<user>/assure1-db.service /etc/systemd/system cp /home/<user>/assure1-bus.service /etc/systemd/system cp /home/<user>/assure1-web.service /etc/systemd/system
Note:
Using the cp command rather than the mv command preserves security context labels for the files.
-
Reload the systemctl daemon and start the Assure1 database:
systemctl daemon-reload systemctl start assure1-db
-
If you did not perform the optional zero-downtime steps in Preparing Primary Database.Assure1 Servers, skip to the next step.
If you performed the optional steps, set up the new Linux 8 primary presentation server as a temporary replication partner for the old Linux 7 secondary presentation server. This synchronizes any missing data that may have been added to old secondary server while the primary server was down during the upgrade process.
-
On the Linux 8 primary presentation server's MySQL Assure1 database, run the following commands as the root MySQL user:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE; RESET SLAVE; CHANGE MASTER TO MASTER_HOST='<SECONDARY-LINUX7-SERVER-FQDN>', MASTER_LOG_FILE='<Pri_L7_Pres_Relay_Master_Log_File>', MASTER_LOG_POS=<Pri_L7_Pres_Exec_Master_Log_Pos>; START SLAVE;
where:
-
<SECONDARY-LINUX7-SERVER-FQDN> is the FQDN of the original Linux 7 secondary presentation server.
-
<Pri_L7_Pres_Relay_Master_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 7 primary presentation server.
-
<Pri_L7_Pres_Exec_Master_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 7 primary presentation server.
-
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA; RESET REPLICA; CHANGE REPLICATION SOURCE TO SOURCE_HOST='<SECONDARY-LINUX7-SERVER-FQDN>', SOURCE_LOG_FILE='<Pri_L7_Pres_Relay_Source_Log_File>', SOURCE_LOG_POS=<Pri_L7_Pres_Exec_Source_Log_Pos>; START REPLICA;
where:
-
<SECONDARY-LINUX7-SERVER-FQDN> is the FQDN of the original Linux 7 secondary presentation server.
-
<Pri_L7_Pres_Relay_Source_Log_File> is the Relay_Source_Log_File value saved earlier for the Linux 7 primary presentation server.
-
<Pri_L7_Pres_Exec_Source_Log_Pos> is the Exec_Source_Log_Pos value saved earlier for the Linux 7 primary presentation server.
-
-
-
On the Linux 7 secondary presentation server's MySQL Assure1 database, run the following commands as the root MySQL user:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
CHANGE MASTER TO MASTER_HOST='<PRIMARY-LINUX8-SERVER-FQDN>', MASTER_LOG_FILE='<Sec_L7_Pres_Relay_Master_Log_File>', MASTER_LOG_POS=<Sec_L7_Pres_Exec_Master_Log_Pos>; START SLAVE;
where:
-
<PRIMARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 primary presentation server.
-
<Sec_L7_Pres_Relay_Master_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 7 secondary presentation server.
-
<Sec_L7_Pres_Exec_Master_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 7 secondary presentation server.
-
-
For MySQL versions 8.4.0 and greater:
CHANGE REPLICATION SOURCE TO SOURCE_HOST='<PRIMARY-LINUX8-SERVER-FQDN>', SOURCE_LOG_FILE='<Sec_L7_Pres_L7_Relay_Source_Log_File>', SOURCE_LOG_POS=<Sec_L7_Pres_Exec_Source_Log_Pos>; START REPLICA;
where:
-
<PRIMARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 primary presentation server
-
<Sec_L7_Pres_Relay_Source_Log_File> is the Relay_Source_Log_File value saved earlier for the Linux 7 secondary presentation server.
-
<Sec_L7_Pres_Exec_Source_Log_Pos> is the Exec_Source_Log_Pos value saved earlier for the Linux 7 secondary presentation server.
-
-
-
-
Repeat the following command for each server in the environment, including the primary presentation server, to get the Linux 7 server IDs and save them in environment variables. Replace the placeholders as appropriate for each server:
<SERVER_TYPE>SERVERID=$($A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "SELECT ServerID FROM Servers WHERE ServerHostFQDN = '<Linux_7_host_FQDN>'" | cut -f1 | grep -v 'ServerID')
where:
-
<database_type> is one of the following, depending on your upgrade path:
-
mariadb if you are upgrading from version 5 to version 6.1.
-
mysql if you are upgrading from version 6.0.x to version 6.1.
-
-
<SERVER_TYPE> is a short way to refer to the server type. For example, you might use PRIPRES for the primary presentation server, making the variable PRIPRESSERVERID.
-
<Linux_7_host_FQDN> is the host FQDN of the server.
If you have a single-server environment, you only run the command once. If you have a standard three-server environment, you will repeat the command three times, adjusting the FQDN and variable as appropriate.
-
-
Repeat one of the following sets of commands for each server in the environment, including the primary presentation server, to set the correct server FQDNs in the database. Replace the placeholders as appropriate for each server:
For presentation servers (internal and external), or in a single-server environment:
$A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE Servers SET ServerHostFQDN='<Linux_8_host_FQDN>', WebFQDN='<Linux_8_web_FQDN>', Version='$RPMVERSION' WHERE ServerID = <server_ID_variable>" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE CoreDatabases SET DBHost = '<Linux_8_host_FQDN>' WHERE DBHost = '<Linux_7_host_FQDN>'" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE AuthVendorSystems SET SystemHost = '<Linux_8_host_FQDN>' WHERE SystemHost = '<Linux_7_host_FQDN>'" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "DELETE FROM Packages WHERE PackageName in ('vendorElasticsearch-app','vendorKibana-app','vendorFilebeat-app')" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE Packages SET PackageEsVersion=0 WHERE PackageName='historicalProcessing-data' AND ServerID = <server_ID_variable>" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE Packages SET PackageIdbVersion=0 WHERE PackageName='historicalProcessing-data' AND ServerID = <server_ID_variable>" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "DELETE FROM Clusters"
where:
-
<server_ID_variable> is the variable you just created for the server ID. For example, $PRIPRESSERVERID.
-
<Linux_8_web_FQDN> is the web FQDN for the presentation server. If you are not using a custom web FQDN, this will be the same as the host FQDN value. In redundant environments, if you are using a shared web FQDN, this will be the same for primary and secondary presentation servers. It will be different for the external presentation server.
For other servers in multi-server environments:
$A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE Servers SET ServerHostFQDN='<Linux_8_host_FQDN>', Version='$RPMVERSION' WHERE ServerID = <server_ID_variable>" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE CoreDatabases SET DBHost = '<Linux_8_host_FQDN>' WHERE DBHost = '<Linux_7_host_FQDN>'" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE AuthVendorSystems SET SystemHost = '<Linux_8_host_FQDN>' WHERE SystemHost = '<Linux_7_host_FQDN>'" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "DELETE FROM Packages WHERE PackageName in ('vendorElasticsearch-app','vendorKibana-app','vendorFilebeat-app')" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE Packages SET PackageEsVersion=0 WHERE PackageName='historicalProcessing-data' AND ServerID = <server_ID_variable>" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE Packages SET PackageIdbVersion=0 WHERE PackageName='historicalProcessing-data' AND ServerID = <server_ID_variable>" $A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "DELETE FROM Clusters"
-
-
If you had any custom packages on the Linux 7 server, remove references to them from the Packages table by running the following command:
$A1BASEDIR/vendor/<database_type>/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "DELETE FROM Packages WHERE PackageName in ('<custom_package_a>','<custom_package_b>')"
where <custom_package_a> and <custom_package_b> are the names of custom packages.
Note:
Because you excluded the $A1BASEDIR/distrib/packages directory from the backup, .pkg and .def files for custom packages will not exist on the new Linux 8 server. You will recreate them after the upgrade is complete.
-
If you are upgrading from version 5 to version 6.1, skip to the next step.
If you are upgrading from version 6.0.x to version 6.1, reset packages to their base 6.0.0.0.0 versions:
$A1BASEDIR/vendor/mysql/bin/mysql --defaults-file=$A1BASEDIR/etc/my.cnf Assure1 -e "UPDATE Packages SET PackageVersion = '6.0.0.0.0.0' WHERE ServerID = <server_ID_variable>"
where <server_ID_variable> is the variable you created for the server ID. For example, $PRIPRESSERVERID.
-
-
If this is not a redundant environment, skip to the next step.
If this is a redundant environment, on the Linux 7 secondary presentation server:
-
Edit the $A1BASEDIR/etc/Assure1.conf file to replace the old Linux 7 primary presentation server FQDN with the new Linux 8 primary presentation server FQDN.
-
Restart the Broker service:
systemctl restart assure1-broker
-
-
If either of the following is true about the server, proceed to Post RPM Installation Steps for Primary Database Servers:
-
The server has the Database.Graph role and you performed the optional steps in Preparing Primary Database.Graph Servers.
-
The server has the Database.Event role and you performed the optional steps in Preparing Primary Database.Event Servers.
If neither is true, proceed to Updating the Unified Assurance Package.
-
Post RPM Installation Steps for Secondary Presentation.Internal Servers
The following steps apply to secondary presentation servers in redundant environments.
-
On the new Linux 8 secondary presentation server:
-
Copy the default certificate files from the Linux 8 primary presentation server:
scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/SiteCA* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/BundleCA* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/User-* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/Vision* $A1BASEDIR/etc/ssl/
-
Generate SSL certificates for the new host and web FQDNs for the new Linux 8 secondary presentation server:
$A1BASEDIR/bin/CreateSSLCertificate --Type Host --CN $OL8FQDN $A1BASEDIR/bin/CreateSSLCertificate --Type Web --CN $SECWEBFQDN
Note:
If your servers share a web FQDN, specify their host FQDNs in the --AltCN option. For example:
$A1BASEDIR/bin/CreateSSLCertificate --Type Web --CN $WEBFQDN --AltCN <Sec_Host_FQDN> --AltCN <Pri_Host_FQDN>
-
Check the contents of $A1BASEDIR/etc/ssl on the old Linux 7 server. If you had any custom certificate files, do the following on the new Linux 8 server:
-
Copy any non-Unified Assurance certificates to the new Linux 8 server:
scp $OL7FQDN:$A1BASEDIR/etc/ssl/<custom_cert>* $A1BASEDIR/etc/ssl/
where <custom_cert> is the name of the custom certificate.
-
Regenerate any custom user certificates, to ensure that they are signed by the new site certificate:
$A1BASEDIR/bin/CreateSSLCertificate --Type User --CN <user_name>
where <user_name> is the custom user to generate the certificate for.
-
-
Copy the Web certificate files from the $A1BASEDIR/etc/ssl directory to the Linux 8 primary presentation server:
scp -f $A1BASEDIR/etc/ssl/Web.* <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/
-
Verify the certificate chain:
cd $A1BASEDIR/etc/ssl openssl verify -CAfile BundleCA.crt SiteCA.crt Host.crt User-api.crt User-assure1.crt User-repl.crt Vision.crt Web.crt
If you have any custom user certificates copied from the Linux 7 server, include them in the verification command.
The verification command should return OK for all certificates.
-
-
On the old Linux 7 secondary presentation server:
-
Copy the Unified Assurance systemd scripts from the /etc/systemd/system directory on the original Linux 7 server to the new Linux 8 server:
scp /etc/systemd/system/assure1-db.service <user>@<hostname>:/home/<user> scp /etc/systemd/system/assure1-bus.service <user>@<hostname>:/home/<user> scp /etc/systemd/system/assure1-web.service <user>@<hostname>:/home/<user>
where:
-
<user> is the user running the command. Depending on your environment, this may be:
-
The opc user.
-
A different user, with the path to their private key specified in the command with the -i option.
-
-
<hostname> is the Linux 8 server host.
-
-
-
On your new Linux 8 secondary presentation server:
-
Move the systemd scripts from the /home/<user> directory to the /etc/systemd/system directory:
cp /home/<user>/assure1-db.service /etc/systemd/system cp /home/<user>/assure1-bus.service /etc/systemd/system cp /home/<user>/assure1-web.service /etc/systemd/system
Note:
Using the cp command rather than the mv command preserves security context labels for the files.
-
Reload the systemctl daemon and start the Assure1 database:
systemctl daemon-reload systemctl start assure1-db
-
If you did not perform the optional zero-downtime steps in Preparing Secondary Database.Assure1 Servers, skip to the next step.
If you performed the optional steps, synchronize the new Linux 8 secondary presentation server's MySQL Assure1 database with the new Linux 8 primary presentation server's database:
-
On the Linux 8 secondary presentation server's MySQL Assure1 database, run the following commands as the root MySQL user:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE; RESET SLAVE; CHANGE MASTER TO MASTER_HOST='<PRIMARY-LINUX8-SERVER-FQDN>', MASTER_LOG_FILE='<Sec_L7_Pres_Relay_Master_Log_File>', MASTER_LOG_POS=<Sec_L7_Pres_Exec_Master_Log_Pos>; START SLAVE;
where:
-
<PRIMARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 primary presentation server.
-
<Sec_L7_Pres_Relay_Master_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 7 secondary presentation server.
-
<Sec_L7_Pres_Exec_Master_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 7 secondary presentation server.
-
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA; RESET REPLICA; CHANGE REPLICATION SOURCE TO SOURCE_HOST='<PRIMARY-LINUX8-SERVER-FQDN>', SOURCE_LOG_FILE='<Sec_L7_Pres_Relay_Source_Log_File>', SOURCE_LOG_POS=<Sec_L7_Pres_Exec_Source_Log_Pos>; START REPLICA;
where:
-
<PRIMARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 primary presentation server.
-
<Sec_L7_Pres_Relay_Source_Log_File> is the Relay_Source_Log_File value saved earlier for the Linux 7 secondary presentation server.
-
<Sec_L7_Pres_Exec_Source_Log_Pos> is the Exec_Source_Log_Pos value saved earlier for the Linux 7 secondary presentation server.
-
-
-
On the Linux 8 primary presentation server's MySQL Assure1 database, run one of the following sets of commands as the root MySQL user:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
CHANGE MASTER TO MASTER_HOST='<SECONDARY-LINUX8-SERVER-FQDN>', MASTER_LOG_FILE='<Pri_L8_Pres_Relay_Master_Log_File>', MASTER_LOG_POS=<Pri_L8_Pres_Exec_Master_Log_Pos>; START SLAVE;
where:
-
<SECONDARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 secondary presentation server.
-
<Pri_L8_Pres_Relay_Master_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 8 primary presentation server.
-
<Pri_L8_Pres_Exec_Master_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 8 primary presentation server.
-
-
For MySQL versions 8.4.0 and greater:
CHANGE REPLICATION SOURCE TO SOURCE_HOST='<SECONDARY-LINUX8-SERVER-FQDN>', SOURCE_LOG_FILE='<Pri_L8_Pres_Relay_Source_Log_File>', SOURCE_LOG_POS=<Pri_L8_Pres_Exec_Source_Log_Pos>; START REPLICA;
The placeholder values in the above command should be replaced with the relevant info as follows:
-
<SECONDARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 secondary presentation server.
-
<Pri_L8_Pres_Relay_Source_Log_File> is the Relay_Source_Log_File value saved earlier for the Linux 8 primary presentation server.
-
<Pri_L8_Pres_Exec_Source_Log_Pos> is the Exec_Source_Log_Pos value saved earlier for the Linux 8 primary presentation server.
-
-
-
-
-
If this is not a redundant environment, skip to the next step.
If this is a redundant environment, on old Linux 7 primary presentation server:
-
Edit the $A1BASEDIR/etc/Assure1.conf file to replace the old Linux 7 secondary presentation server FQDN with the new Linux 8 secondary presentation server FQDN.
-
Restart the Broker service:
systemctl restart assure1-broker
-
-
If the server also has the Database.Event role and you performed the optional steps in Preparing Secondary Database.Event Servers, proceed to Post RPM Installation Steps for Secondary Database Servers.
If it does not have the Database.Event role or you did not perform the optional steps, proceed to Updating the Unified Assurance Package.
Post RPM Installation Steps for Primary Database Servers
The following steps apply to database servers in non-redundant environments, or primary database servers in redundant environments.
Perform these steps if any of the following are true about the server:
-
Does not have the Presentation.Internal role
-
Has the Database.Graph role and you performed the optional steps in Preparing Primary Database.Graph Servers.
-
Has the Database.Event role and you performed the optional steps in Preparing Primary Database.Event Servers.
Skip these steps if all of the following are true about the server:
-
Has the Presentation.Internal role
-
Does not have the Database.Graph role, or has the Database.Event role, but you did not perform the optional steps in Preparing Primary Database.Event Servers.
-
Does not have the Database.Event role, or has the Database.Event role, but you did not perform the optional steps in Preparing Primary Database.Event Servers
To perform the post RPM steps for primary database servers:
-
If your new Linux 8 primary database server also has the Presentation.Internal role, skip to the next step.
If it does not also have the Presentation.Internal role:
-
Copy the default certificate files from the Linux 8 primary presentation server:
scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/SiteCA* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/BundleCA* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/User-* $A1BASEDIR/etc/ssl/
-
Generate the new SSL certificate for the Linux 8 host:
$A1BASEDIR/bin/CreateSSLCertificate --Type Host --CN $OL8FQDN
-
Check the contents of $A1BASEDIR/etc/ssl on the old Linux 7 server. If you had any custom certificate files, do the following on the new Linux 8 server:
-
Copy any non-Unified Assurance certificates to the new Linux 8 server:
scp $OL7FQDN:$A1BASEDIR/etc/ssl/<custom_cert>* $A1BASEDIR/etc/ssl/
where <custom_cert> is the name of the custom certificate.
-
Regenerate any custom user certificates, to ensure that they are signed by the new site certificate:
$A1BASEDIR/bin/CreateSSLCertificate --Type User --CN <user_name>
where <user_name> is the custom user to generate the certificate for.
-
-
Verify the certificate chain:
cd $A1BASEDIR/etc/ssl openssl verify -CAfile BundleCA.crt SiteCA.crt Host.crt User-api.crt User-assure1.crt User-repl.crt
If you have any custom user certificates copied from the Linux 7 server, include them in the verification command.
The verification command should return OK for all certificates.
-
On the old Linux 7 primary database server, copy the assure1-db.service script from the /etc/systemd/system directory on the original Linux 7 server to the new Linux 8 server:
scp /etc/systemd/system/assure1-db.service <user>@<host>:/home/<user>
where <user> is the user running the command and <hostname> is the Linux 8 server host.
Depending on your environment, <user> may be:
-
The opc user.
-
A different user, with the path to their private key specified in the command with the -i option.
-
-
On your new Linux 8 primary database server:
-
Move the assure1-db.service script from the /home/<user> directory to the /etc/systemd/system directory:
cp /home/<user>/assure1-db.service /etc/systemd/system
Note:
Using the cp command rather than the mv command preserves security context labels for the file.
-
Reload the systemctl daemon:
systemctl daemon-reload
-
-
-
If your new Linux 8 primary database server does not have the Database.Event role, or it does, but you did not perform the optional steps in Preparing Primary Database.Event Servers, skip to the next step.
If it has the Database.Event role, and you performed the optional steps in Preparing Primary Database.Event Servers:
Set up the new Linux 8 primary database server as a temporary replication partner for the old Linux 7 secondary database server. This synchronizes any missing data that may have been added to old secondary server while the primary server was down during the upgrade process.
If you did not perform the optional zero-downtime preparation steps, skip these steps.
-
Start the database.
systemctl start assure1-db
-
On the Linux 8 primary presentation server's MySQL Event database, run the following commands as the root MySQL user:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE; RESET SLAVE; CHANGE MASTER TO MASTER_HOST='<SECONDARY-LINUX7-SERVER-FQDN>', MASTER_LOG_FILE='<Pri_L7_DB_Relay_Master_Log_File>', MASTER_LOG_POS=<Pri_L7_DB_Exec_Master_Log_Pos>; START SLAVE;
where:
-
<SECONDARY-LINUX7-SERVER-FQDN> is the FQDN of the original Linux 7 secondary database server.
-
<Pri_L7_DB_Relay_Master_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 7 primary database server.
-
<Pri_L7_DB_Exec_Master_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 7 primary database server.
-
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA; RESET REPLICA; CHANGE REPLICATION SOURCE TO SOURCE_HOST='<SECONDARY-LINUX7-SERVER-FQDN>', SOURCE_LOG_FILE='<Pri_L7_DB_Relay_Source_Log_File>', SOURCE_LOG_POS=<Pri_L7_DB_Exec_Source_Log_Pos>; START REPLICA;
where:
-
<SECONDARY-LINUX7-SERVER-FQDN> is the FQDN of the original Linux 7 secondary database server.
-
<Pri_L7_DB_Relay_Source_Log_File> is the Relay_Source_Log_File value saved earlier for the Linux 7 primary database server.
-
<Pri_L7_DB_Exec_Source_Log_Pos> is the Exec_Source_Log_Pos value saved earlier for the Linux 7 primary database server.
-
-
-
On the Linux 7 secondary database server's MySQL Event database, run the following commands as the root MySQL user:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
CHANGE MASTER TO MASTER_HOST='<PRIMARY-LINUX8-SERVER-FQDN>', MASTER_LOG_FILE='<Sec_L7_DB_Relay_Master_Log_File>', MASTER_LOG_POS=<Sec_L7_DB_Exec_Master_Log_Pos>; START SLAVE;
where:
-
<PRIMARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 primary database server.
-
<Sec_L7_DB_Relay_Master_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 7 secondary database server.
-
<Sec_L7_DB_Exec_Master_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 7 secondary database server.
-
-
For MySQL versions 8.4.0 and greater:
CHANGE REPLICATION SOURCE TO SOURCE_HOST='<PRIMARY-LINUX8-SERVER-FQDN>', SOURCE_LOG_FILE='<Sec_L7_DB_Relay_Source_Log_File>', SOURCE_LOG_POS=<Sec_L7_DB_Exec_Source_Log_Pos>; START REPLICA;
where
-
<PRIMARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 primary database server.
-
<Sec_L7_DB_Relay_Source_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 7 secondary database server.
-
<Sec_L7_DB_Exec_Source_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 7 secondary database server.
-
-
-
-
If your new Linux 8 primary database server does not have the Database.Graph role, or it does, but you did not perform the optional steps in Preparing Primary Database.Graph Servers skip to the next step.
If it has the Database.Graph role, and you performed the optional steps, bring the Neo4j database on the Linux 8 primary database server up to date with the Linux 7 secondary database server. This procedure involves steps on the Linux 7 secondary database server, the Linux 8 primary database server, and the Unified Assurance UI.
-
On the Linux 7 secondary database server:
-
Switch to the assure1 user:
su - assure1
-
Set the Graph database to READ ONLY to enable backing up the schema and data:
a1neo4jroot ALTER DATABASE graph SET ACCESS READ ONLY; :exit
-
Create the $A1BASEDIR environment variable for <UA_home>, replacing <UA_home> with the appropriate value. This is usually opt/assure1:
export A1BASEDIR=<UA_home>
-
Back up the Graph database:
-
For Unified Assurance version 6.0.3 or lower:
mkdir $A1BASEDIR/tmp/neo4j-dump NEO4J_CONF=$A1BASEDIR/etc/neo4j/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin backup --database=graph --backup-dir=$A1BASEDIR/tmp/neo4j-dump/ --include-metadata=all
-
For Unified Assurance version 6.0.4 or higher:
mkdir $A1BASEDIR/tmp/neo4j-dump NEO4J_CONF=$A1BASEDIR/etc/neo4j/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin database backup --to-path=$A1BASEDIR/tmp/neo4j-dump --include-metadata=all --type=full graph
-
-
-
On the Linux 8 primary database server:
-
Switch to the assure1 user:
su - assure1
-
Connect to Neo4j and drop the Graph database:
a1neo4jroot DROP DATABASE Graph; :exit
-
Create an environment variable for $A1BASEDIR, replacing <UA_home> with the appropriate value. This is usually opt/assure1:
export A1BASEDIR=<UA_home>
-
Copy the backup files from the Linux 7 secondary database server to the Linux 8 primary database server.
-
Restore the Graph database from the backup file by running one of the following sets of commands:
-
For Unified Assurance version 6.0.3 or lower:
sudo chown -R assure1:assure1 $A1BASEDIR/tmp/neo4j-dump NEO4J_CONF=$A1BASEDIR/etc/neo4j/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin restore --database=graph --from=$A1BASEDIR/tmp/neo4j-dump/graph
-
For Unified Assurance version 6.0.4 or higher:
sudo chown -R assure1:assure1 $A1BASEDIR/tmp/neo4j-dump NEO4J_CONF=$A1BASEDIR/etc/neo4j/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin database restore --from-path=$A1BASEDIR/tmp/neo4j-dump/<backup-file-name> graph
where <backup-file-name> is the name of your Graph database backup file.
-
-
Check if the output of the restore command contains the following line:
You need to execute .../var/neo4j/scripts/graph/restore_metadata.cypher. To execute the file use cypher-shell command with parameter `graph`
If so, run the following command to restore the metadata:
cat $A1BASEDIR/var/neo4j/scripts/graph/restore_metadata.cypher | A1_JAVA_HOME=$A1BASEDIR/vendor/java NEO4J_PASSWORD=$($A1BASEDIR/bin/JWT -a neo4j -s neo4j) $A1BASEDIR/vendor/neo4j/bin/cypher-shell -a neo4j+ssc://$(hostname):7687 -u neo4j -d system --param "database => 'graph'"
If not, continue.
-
Connect to Graph as the admin user, switch to the system DB, and create the Graph database:
a1neo4jroot :use system CREATE DATABASE GRAPH; :exit
Note:
If the Graph database already exists, a warning message appears. Ignore it and continue.
-
Switch back to the root user:
su - root
-
If this server does not have the Presentation.Internal role, skip to the next step.
If it has the Presentation.Internal role, start the web server:
systemctl start assure1-web
-
-
In the Unified Assurance UI:
-
From the Configuration menu, select Broker Control, then Services.
-
Select the Neo4j service for the Linux 7 secondary database server.
The service details appear.
-
Set Status to Disabled and click Submit.
In the All Services table, the value in the Status column for the service is updated to Disabled.
-
In the All Services table, click Stop.
In the All Services table, the value in the State column for the service is updated to Stopped.
-
-
On the Linux 8 primary database server:
If this server does not have the Presentation.Internal role, skip this step.
If it has the Presentation.Internal role, stop the web server:
systemctl stop assure1-web
-
-
Proceed to Updating the Unified Assurance Package.
Post RPM Installation Steps for Secondary Database Servers
The following steps apply to secondary database servers in redundant environments.
Perform these steps if either of the following is true about the server:
-
Does not have the Presentation.Internal role
-
Has the Database.Event role and you performed the optional steps in Preparing Primary Database.Event Servers
Skip these steps if both of the following are true about the server:
-
Has the Presentation.Internal role
-
Does not have the Database.Event role, or has the Database.Event role, but you did not perform the optional steps in Preparing Primary Database.Event Servers
To perform the post RPM installation steps for secondary database servers:
-
If your new Linux 8 secondary database server also has the Presentation.Internal role, skip to the next step.
If it does not also have the Presentation.Internal role:
-
Copy the default certificate files from the Linux 8 primary presentation server:
scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/SiteCA* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/BundleCA* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/User-* $A1BASEDIR/etc/ssl/
-
Generate new SSL certificates for the Linux 8 host:
$A1BASEDIR/bin/CreateSSLCertificate --Type Host --CN $OL8FQDN
-
Check the contents of $A1BASEDIR/etc/ssl on the old Linux 7 server. If you had any custom certificate files, do the following on the new Linux 8 server:
-
Copy any non-Unified Assurance certificates to the new Linux 8 server:
scp $OL7FQDN:$A1BASEDIR/etc/ssl/<custom_cert>* $A1BASEDIR/etc/ssl/
where <custom_cert> is the name of the custom certificate.
-
Regenerate any custom user certificates, to ensure that they are signed by the new site certificate:
$A1BASEDIR/bin/CreateSSLCertificate --Type User --CN <user_name>
where <user_name> is the custom user to generate the certificate for.
-
-
Verify the certificate chain:
cd $A1BASEDIR/etc/ssl openssl verify -CAfile BundleCA.crt SiteCA.crt Host.crt User-api.crt User-assure1.crt User-repl.crt
If you have any custom user certificates copied from the Linux 7 server, include them in the verification command.
The verification command should return OK for all certificates.
-
On the old Linux 7 secondary database server, copy the assure1-db.service script from the /etc/systemd/system/ directory on the original Linux 7 server to the new Linux 8 server:
scp /etc/systemd/system/assure1-db.service <user>@<host>:/home/<user>
where <user> is the user running the command and <hostname> is the Linux 8 server host.
Depending on your environment, <user> may be:
-
The opc user.
-
A different user, with the path to their private key specified in the command with the -i option.
-
-
On your new Linux 8 secondary database server:
-
Move the assure1-db.service script from the /home/<user> directory to the /etc/systemd/system directory:
cp /home/<user>/assure1-db.service /etc/systemd/system
Note:
Using the cp command rather than the mv command preserves security context labels for the file.
-
Reload the systemctl daemon:
systemctl daemon-reload
-
-
-
If your Linux 8 secondary database does not have the Database.Event role, or it does, but you did not perform the optional steps in Preparing Secondary Database.Event Servers, skip to the next step.
If it has the Database.Event role, and you performed the optional steps in Preparing Secondary Database.Event Servers, synchronize the new Linux 8 secondary database server's MySQL database with the new Linux 8 primary database server's database:
-
Start the database.
systemctl start assure1-db
-
On the Linux 8 secondary database server's MySQL Event database, run one of the following sets of commands as the root MySQL user:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
STOP SLAVE; RESET SLAVE; CHANGE MASTER TO MASTER_HOST='<PRIMARY-LINUX8-SERVER-FQDN>', MASTER_LOG_FILE='<Sec_L7_DB_Relay_Master_Log_File>', MASTER_LOG_POS=<Sec_L7_DB_Exec_Master_Log_Pos>; START SLAVE;
where:
-
<PRIMARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 primary database server.
-
<Sec_L7_DB_Relay_Master_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 7 secondary database server.
-
<Sec_L7_DB_Exec_Master_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 7 secondary database server.
-
-
For MySQL versions 8.4.0 and greater:
STOP REPLICA; RESET REPLICA; CHANGE REPLICATION SOURCE TO SOURCE_HOST='<PRIMARY-LINUX8-SERVER-FQDN>', SOURCE_LOG_FILE='<Relay_Source_Log_File>', SOURCE_LOG_POS=<Exec_Source_Log_Pos>; START REPLICA;
where:
-
<PRIMARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 primary database server.
-
<Sec_L7_DB_Relay_Source_Log_File> is the Relay_Source_Log_File value saved earlier for the Linux 7 secondary database server.
-
<Sec_L7_DB_Exec_Source_Log_Pos> is the Exec_Source_Log_Pos value saved earlier for the Linux 7 secondary database server.
-
-
-
On the Linux 8 primary database server's MySQL Event database, run one of the following sets of commands as the root MySQL user:
-
For MySQL versions less than 8.4.0 (including MariaDB, if you are upgrading from version 5):
CHANGE MASTER TO MASTER_HOST='<SECONDARY-LINUX8-SERVER-FQDN>', MASTER_LOG_FILE='<Pri_L8_DB_Relay_Master_Log_File>', MASTER_LOG_POS=<Pri_L8_DB_Exec_Master_Log_Pos>; START SLAVE;
where:
-
<SECONDARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 secondary database server.
-
<Pri_L8_DB_Relay_Master_Log_File> is the Relay_Master_Log_File value saved earlier for the Linux 8 primary database server.
-
<Pri_L8_DB_Exec_Master_Log_Pos> is the Exec_Master_Log_Pos value saved earlier for the Linux 8 primary database server.
-
-
For MySQL versions 8.4.0 and greater:
CHANGE REPLICATION SOURCE TO SOURCE_HOST='<SECONDARY-LINUX8-SERVER-FQDN>', SOURCE_LOG_FILE='<Pri_L8_DB_Relay_Source_Log_File>', SOURCE_LOG_POS=<Pri_L8_DB_Exec_Source_Log_Pos>; START REPLICA;
where:
-
<SECONDARY-LINUX8-SERVER-FQDN> is the FQDN of the new Linux 8 secondary database server.
-
<Pri_L8_DB_Relay_Source_Log_File> is the Relay_Source_Log_File value saved earlier for the Linux 8 primary database server.
-
<Pri_L8_DB_Exec_Source_Log_Pos> is the Exec_Source_Log_Pos value saved earlier for the Linux 8 primary database server.
-
-
-
-
Proceed to Updating the Unified Assurance Package.
Post RPM Installation Steps for Collection Servers
The following steps apply to servers with the Collection role that do not also have the Presentation.Internal or any Database roles.
If your collection server also has any Database or Presentation roles, you already performed these steps and can skip them.
-
On the new Linux 8 collection server, copy the default certificate files from the Linux 8 primary presentation server:
scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/SiteCA* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/BundleCA* $A1BASEDIR/etc/ssl/ scp <Pri_Pres_Host_FQDN>:$A1BASEDIR/etc/ssl/User-* $A1BASEDIR/etc/ssl/
-
Generate new SSL certificate for the Linux 8 host:
$A1BASEDIR/bin/CreateSSLCertificate --Type Host --CN $OL8FQDN
-
Check the contents of $A1BASEDIR/etc/ssl on the old Linux 7 server. If you had any custom certificate files, do the following on the new Linux 8 server:
-
Copy any non-Unified Assurance certificates to the new Linux 8 server:
scp $OL7FQDN:$A1BASEDIR/etc/ssl/<custom_cert>* $A1BASEDIR/etc/ssl/
where <custom_cert> is the name of the custom certificate.
-
Regenerate any custom user certificates, to ensure that they are signed by the new site certificate:
$A1BASEDIR/bin/CreateSSLCertificate --Type User --CN <user_name>
where <user_name> is the custom user to generate the certificate for.
-
-
Verify the certificate chain:
cd $A1BASEDIR/etc/ssl openssl verify -CAfile BundleCA.crt SiteCA.crt Host.crt User-api.crt User-assure1.crt User-repl.crt
If you have any custom user certificates copied from the Linux 7 server, include them in the verification command.
The verification command should return OK for all certificates.
-
Proceed to Updating the Unified Assurance Package.
Post RPM Installation Steps for Primary Presentation.External Servers
The following steps apply to the external presentation server in non-redundant environments, or the primary external presentation server in redundant environments. These servers should only the Presentation.External role.
-
On the new Linux 8 external presentation server:
-
Create environment variables for the Linux 8 external presentation server's web FQDN and Linux 8 internal presentation server's host FQDN:
export PRIEXTWEBFQDN=<Primary_External_Web_FQDN> export PRIINTHOSTFQDN=<Primary_Internal_Presentation_Host_FQDN>
-
Copy the default certificate files from the Linux 8 primary presentation server:
scp $PRIINTHOSTFQDN:$A1BASEDIR/etc/ssl/SiteCA* $A1BASEDIR/etc/ssl/ scp $PRIINTHOSTFQDN:$A1BASEDIR/etc/ssl/BundleCA* $A1BASEDIR/etc/ssl/ scp $PRIINTHOSTFQDN:$A1BASEDIR/etc/ssl/User-* $A1BASEDIR/etc/ssl/ scp $PRIINTHOSTFQDN:$A1BASEDIR/etc/ssl/Vision* $A1BASEDIR/etc/ssl/
-
Generate new SSL certificates for the host FQDN and the external web FQDN:
$A1BASEDIR/bin/CreateSSLCertificate --Type Host --CN $OL8FQDN $A1BASEDIR/bin/CreateSSLCertificate --Type Web --CN $PRIEXTWEBFQDN
-
Check the contents of $A1BASEDIR/etc/ssl on the old Linux 7 server. If you had any custom certificate files, do the following on the new Linux 8 server:
-
Copy any non-Unified Assurance certificates to the new Linux 8 server:
scp $OL7FQDN:$A1BASEDIR/etc/ssl/<custom_cert>* $A1BASEDIR/etc/ssl/
where <custom_cert> is the name of the custom certificate.
-
Regenerate any custom user certificates, to ensure that they are signed by the new site certificate:
$A1BASEDIR/bin/CreateSSLCertificate --Type User --CN <user_name>
where <user_name> is the custom user to generate the certificate for.
-
-
Verify the certificate chain:
cd $A1BASEDIR/etc/ssl openssl verify -CAfile BundleCA.crt SiteCA.crt Host.crt User-api.crt User-assure1.crt User-repl.crt Web.crt Vision.crt
If you have any custom user certificates copied from the Linux 7 server, include them in the verification command.
The verification command should return OK for all certificates.
-
-
On the old Linux 7 external presentation server, copy the assure1-web.service systemd script from the /etc/systemd/system directory to the new Linux 8 server:
scp /etc/systemd/system/assure1-web.service <user>@<hostname>:/home/<user>
where:
-
<user> is the user running the command. Depending on your environment, this may be:
-
The opc user.
-
A different user, with the path to their private key specified in the command with the -i option.
-
-
<hostname> is the Linux 8 server host.
-
-
On the new Linux 8 external presentation server:
-
Move the systemd scripts from the /home/<user> directory to the /etc/systemd/system directory:
cp /home/<user>/assure1-db.service /etc/systemd/system cp /home/<user>/assure1-bus.service /etc/systemd/system cp /home/<user>/assure1-web.service /etc/systemd/system
Note:
Using the cp command rather than the mv command preserves security context labels for the files.
-
Reload the systemctl daemon:
systemctl daemon-reload
-
-
Proceed to Updating the Unified Assurance Package.
Post RPM Installation Steps for Secondary Presentation.External Servers
The following steps apply to secondary external presentation servers in redundant environments. These servers should only the Presentation.External role.
-
On the new Linux 8 secondary external presentation server:
-
Create environment variables for the Linux 8 external presentation servers:
export SECEXTWEBFQDN=<Secondary_External_Web_FQDN> export PRIEXTHOSTFQDN=<Primary_External_Host_FQDN> export PRIINTHOSTFQDN=<Primary_Internal_Presentation_Host_FQDN>
-
Copy the default certificate files from the Linux 8 primary internal presentation server:
scp $PRIINTHOSTFQDN:$A1BASEDIR/etc/ssl/SiteCA* $A1BASEDIR/etc/ssl/ scp $PRIINTHOSTFQDN:$A1BASEDIR/etc/ssl/BundleCA* $A1BASEDIR/etc/ssl/ scp $PRIINTHOSTFQDN:$A1BASEDIR/etc/ssl/User-* $A1BASEDIR/etc/ssl/ scp $PRIINTHOSTFQDN:$A1BASEDIR/etc/ssl/Vision* $A1BASEDIR/etc/ssl/
-
Generate new SSL certificates for the host FQDN and the external web FQDN:
$A1BASEDIR/bin/CreateSSLCertificate --Type Host --CN $OL8FQDN $A1BASEDIR/bin/CreateSSLCertificate --Type Web --CN $SECEXTWEBFQDN
Note:
If your servers share a web FQDN, specify their host FQDNs in the --AltCN option. For example:
$A1BASEDIR/bin/CreateSSLCertificate --Type Web --CN $SECEXTWEBFQDN --AltCN <Sec_Host_FQDN> --AltCN <Pri_Host_FQDN>
-
Check the contents of $A1BASEDIR/etc/ssl on the old Linux 7 server. If you had any custom certificate files, do the following on the new Linux 8 server:
-
Copy any non-Unified Assurance certificates to the new Linux 8 server:
scp $OL7FQDN:$A1BASEDIR/etc/ssl/<custom_cert>* $A1BASEDIR/etc/ssl/
where <custom_cert> is the name of the custom certificate.
-
Regenerate any custom user certificates, to ensure that they are signed by the new site certificate:
$A1BASEDIR/bin/CreateSSLCertificate --Type User --CN <user_name>
where <user_name> is the custom user to generate the certificate for.
-
-
Copy the Web certificate files from the $A1BASEDIR/etc/ssl directory to the Linux 8 primary external presentation server:
scp -f $A1BASEDIR/etc/ssl/Web.* $PRIEXTHOSTFQDN:$A1BASEDIR/etc/ssl/
-
Verify the certificate chain:
cd $A1BASEDIR/etc/ssl openssl verify -CAfile BundleCA.crt SiteCA.crt Host.crt User-api.crt User-assure1.crt User-repl.crt Web.crt Vision.crt
If you have any custom user certificates copied from the Linux 7 server, include them in the verification command.
The verification command should return OK for all certificates.
-
-
On the old Linux 7 secondary external presentation server, copy the assure1-web.service systemd script from the /etc/systemd/system directory to the new Linux 8 secondary server:
scp /etc/systemd/system/assure1-web.service <user>@<hostname>:/home/<user>
where:
-
<user> is the user running the command. Depending on your environment, this may be:
-
The opc user.
-
A different user, with the path to their private key specified in the command with the -i option.
-
-
<hostname> is the Linux 8 server host.
-
-
On the new Linux 8 secondary external presentation server:
-
Move the systemd scripts from the /home/<user> directory to the /etc/systemd/system directory:
cp /home/<user>/assure1-db.service /etc/systemd/system cp /home/<user>/assure1-bus.service /etc/systemd/system cp /home/<user>/assure1-web.service /etc/systemd/system
Note:
Using the cp command rather than the mv command preserves security context labels for the files.
-
Reload the systemctl daemon:
systemctl daemon-reload
-
-
Proceed to Updating the Unified Assurance Package.
Updating the Unified Assurance Package
After completing all post RPM installation steps for the server you are currently upgrading, complete the following steps and all subsequent upgrade steps for the same server before starting again at Preparing Servers for the next server.
On your new Linux 8 server with any role:
-
Start the Broker:
systemctl start assure1-broker
-
Verify that the Broker service has started:
systemctl status assure1-broker
-
Remove the old Linux 7 ssl and crypto libraries:
rm -f $A1BASEDIR/lib/libcrypto.s* $A1BASEDIR/lib/libssl.s*
-
If this server does not have the Database.Graph role, skip to the next step.
If it has the Database.Graph role:
-
Connect to the server from a separate shell.
-
Set the environment variables:
export A1BASEDIR=<UA_home> export A1_JAVA_HOME=$A1BASEDIR/vendor/java
-
Start Neo4j by running the following command:
$A1BASEDIR/vendor/neo4j/bin/neo4j_wrapper
-
Return to the previous shell.
-
-
If the $A1BASEDIR/tmp/InitialPackages.sql file exists, delete it:
rm -f $A1BASEDIR/tmp/InitialPackages.sql
-
Update the Unified Assurance package:
$A1BASEDIR/bin/Package update --depot $FILESDIR
On servers with the Database.Historical role, Package update also runs AnalyticsWizard, which installs OpenSearch and performs some automatic migration tasks. If your Historical database does not have enough memory for OpenSearch (set to the same heap size as was previously set for Elasticsearch), Package update fails. To resolve this:
-
In the $A1BASEDIR/tmp/elasticsearch/custom-env file, update the value of the ES_JAVA_OPTS parameter to allocate enough memory. It must be at least 2GB.
-
Re-run Package update:
$A1BASEDIR/bin/Package update --depot $FILESDIR
The process picks up where it left off.
-
-
If this server does not have a Presentation role (internal or external), skip to the next step.
If it has a Presentation role, start the web server:
systemctl start assure1-web
-
Remove the "InstallIncomplete" entry from Assure1.conf:
sed -i '/InstallIncomplete/c\' $A1BASEDIR/etc/Assure1.conf
-
For servers with the following roles, proceed to post package update steps:
-
Post Package Update Steps for Secondary Database.Graph Servers
-
Post Package Update Steps for Cluster.Master and Cluster.Worker Servers
-
Secondary servers in redundant environments with the Presentation.Internal, Database.Event, or Database.Metric role: Post Package Update Steps for Secondary Servers
If none of the post package update steps apply, and all servers are not upgraded, start upgrading the next server. If all servers are upgraded, proceed to Post-Upgrade Tasks.
Post Package Update Steps
After updating the Unified Assurance package on the server you are currently upgrading, complete the following steps, depending on the roles installed on the server, complete the following steps for the same server before starting again at Preparing Servers for the next server.
If there are multiple roles on this server, you may have to refer to multiple post-update step sections.
Post Package Update Steps for Database.Historical Servers
The following information applies to all Database.Historical servers, including primary and secondary servers, and servers in non-redundant environments.
When you update the Unified Assurance package, Package update also runs AnalyticsWizard, which migrates Elasticsearch artifacts to OpenSearch. You can monitor or review the artifact migration progress by tailing the following log file:
$A1BASEDIR/logs/MigrateHistoricalObjects.log
For any failures, you can find the Elasticsearch export and OpenSearch import files in the following directory:
$A1BASEDIR/tmp/migration/
AnalyticsWizard does not migrate the documents in the indexes. You do this using the MigrateHistoricalData utility, after completing the upgrade. See About Migration and Automatically Migrated Artifacts in Unified Assurance Migration Guide for more information about the migration process.
After the package update and artifact migration process is complete, if this server:
-
Is a secondary server with the Database.Graph role, proceed to Post Package Update Steps for Secondary Database.Graph Servers.
-
Is a secondary server with the Presentation.Internal, Database.Metric, or Database.Event role, proceed to Post Package Update Steps for Secondary Servers.
-
Has the Cluster.Master or Cluster.Worker role, proceed to Post Package Update Steps for Cluster.Master and Cluster.Worker Servers.
If no other post package update tasks apply, and all servers are not upgraded, start upgrading the next server. If all servers are upgraded, proceed to Post-Upgrade Tasks.
Note:
If your secondary database server has only the Database.Historical role, you do not need to perform the steps in Post Package Update Steps for Secondary Servers. Redundant Historical databases are not aware of each other in the same way as other databases. See Historical Database Scalability and Redundancy in Unified Assurance Concepts for details.
Post Package Update Steps for Secondary Database.Graph Servers
The following steps apply when both of the following are true:
-
This is the secondary server in a redundant environment
-
You performed the optional steps in Preparing Primary Database.Graph Servers for the primary server.
This procedure brings the Neo4j database on the Linux 8 secondary database server up to date with the primary, and enable streaming between them. It involves steps on both the primary and secondary server database server.
-
On the Linux 8 primary database server:
-
Switch to the assure1 user:
su - assure1
-
Set the Graph database to READ ONLY to enable backing up the schema and data:
a1neo4jroot ALTER DATABASE graph SET ACCESS READ ONLY; :exit
-
Create the $A1BASEDIR environment variable for <UA_home>, replacing <UA_home> with the appropriate value. This is usually opt/assure1:
export A1BASEDIR=<UA_home>
-
Back up the Graph database:
mkdir $A1BASEDIR/tmp/neo4j-dump NEO4J_CONF=$A1BASEDIR/etc/neo4j/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin database backup --to-path=$A1BASEDIR/tmp/neo4j-dump --include-metadata=all --type=full graph
-
-
On the Linux 8 secondary database server:
-
Switch to the assure1 user:
su - assure1
-
Connect to Neo4j and drop the Graph database:
a1neo4jroot DROP DATABASE Graph; :exit
-
Create the $A1BASEDIR environment variable for <UA_home>, replacing <UA_home> with the appropriate value. This is usually opt/assure1:
export A1BASEDIR=<UA_home>
-
Copy the backup files from the Linux 8 primary database server onto the Linux 8 secondary database server.
-
Restore the Graph database from the backup file:
sudo chown -R assure1:assure1 $A1BASEDIR/tmp/neo4j-dump NEO4J_CONF=$A1BASEDIR/etc/neo4j/ $A1BASEDIR/vendor/neo4j/bin/neo4j-admin database restore --from-path=$A1BASEDIR/tmp/neo4j-dump/<backup-file-name> graph
where <backup-file-name> is the name of your Graph database backup file.
-
Check if the output of the restore command contains the following line:
You need to execute .../var/neo4j/scripts/graph/restore_metadata.cypher. To execute the file use cypher-shell command with parameter `graph`
If so, run the following command to restore the metadata:
cat $A1BASEDIR/var/neo4j/scripts/graph/restore_metadata.cypher | A1_JAVA_HOME=$A1BASEDIR/vendor/java NEO4J_PASSWORD=$($A1BASEDIR/bin/JWT -a neo4j -s neo4j) $A1BASEDIR/vendor/neo4j/bin/cypher-shell -a neo4j+ssc://$(hostname):7687 -u neo4j -d system --param "database => 'graph'"
If not, continue.
-
Connect to Neo4j as the admin user, switch to the system database, and create the Graph database:
a1neo4jroot :use system CREATE DATABASE GRAPH; :exit
Note:
If the Graph database already exists, a warning message appears. Ignore it and continue.
-
-
On both the primary and Linux 8 secondary database servers, in the $A1BASEDIR/etc/neo4j/streams.conf file, change the value of the streams.source.enabled property to true. For example:
streams.source.enabled=true
-
In the Unified Assurance UI:
-
From the Configuration menu, select Broker Control, then Services.
-
Select the Neo4j service for the Linux 8 secondary database server.
The service details appear.
-
Set Status to Enabled and click Submit.
-
Select the Neo4j Kafka Connect service for the Linux 8 primary database server.
-
Set Status to Enabled and click Submit.
-
Select the Neo4j Kafka Connect service for the Linux 8 secondary database server.
-
Set Status to Enabled and click Submit.
-
-
On the Linux 8 primary database server, set the Graph database access back to READ WRITE:
a1neo4jroot ALTER DATABASE graph SET ACCESS READ WRITE; :exit
-
If this server has the Cluster.Master or Cluster.Worker role, proceed to Post Package Update Steps for Cluster.Master and Cluster.Worker Servers.
If it does not, proceed to Post Package Update Steps for Secondary Servers.
Post Package Update Steps for Cluster.Master and Cluster.Worker Servers
The following steps apply to all servers with Cluster roles, including primary and secondary servers, and servers in non-redundant environments.
-
Recreate your cluster and deploy microservices to it as described in Microservice Cluster Setup in Unified Assurance Implementation Guide.
-
If this is a secondary server in a redundant environment with any of the following roles, proceed to Post Package Update Steps for Secondary Servers:
-
Presentation.Internal (will also have Database.Assure1)
-
Database.Event
-
Database.Graph
-
Database.Metric
If is not a secondary server with any of those roles, and all servers are not upgraded, start upgrading the next server.
If all servers are upgraded, proceed to Post-Upgrade Tasks.
-
Post Package Update Steps for Secondary Servers
After updating the secondary server in a redundant pair with any of the following roles, run RedundancyWizard:
-
Presentation.Internal (these will also have Database.Assure1)
-
Database.Event
-
Database.Graph
-
Database.Metric
You do not need to run RedundancyWizard for external presentation servers, or for the Collection role or Database.Historical role, if they do not also have any of the roles above.
To run RedundancyWizard:
-
Force stop the telegraf-kafka service by repeating the following on both the primary and secondary server:
-
Find the process ID for telegraf-kafka:
ps aux | grep telegraf-kafka
Example output:
assure1 23090 0.1 0.5 5668544 181032 ? Sl 18:21 0:10 /opt/assure1/vendor/telegraf/bin/telegraf --config opt/assure1/etc/telegraf-kafka.conf
The process ID is listed in the second column. In the example, the process ID is 23090.
-
Run the following command, replacing <process_ID> with the correct process ID:
kill -9 <process_ID>
-
-
On the secondary server, run the following command as root:
$A1BASEDIR/bin/redundancy/RedundancyWizard
-
When prompted whether to proceed with configuring local server as a redundant partner, verify that the correct primary server is listed, enter y, then press Enter.
-
Wait for the listener to activate.
-
On the primary server, run the following command as root:
$A1BASEDIR/bin/redundancy/RedundancyWizard
-
When shown the list of servers that will be used, verify that the correct servers and roles are listed.
-
When prompted Is this correct?, enter y then press Enter.
-
When prompted to allow the script to do various things (configure redundancy, perform database synchronization), enter y then press Enter.
-
When prompted to press enter after starting the script on the redundant server, press Enter. (You started the script in step 1.)
-
If this is a database server, when prompted to proceed with restarting the database, enter y, then press Enter.
-
Wait for the wizard to finish synchronizing the data.
Note:
If you see an error about the Neo4j topic already existing, you can safely ignore it.
-
If all servers are not upgraded, start upgrading the next server.
If all servers are upgraded, proceed to Post-Upgrade Tasks.
Post-Upgrade Tasks
After you have finished upgrading all of your servers:
-
If you were using keepalived and stopped it during the upgrade, reinstall and configure it on your new Linux 8 system.
-
Copy any custom Docker images on your Linux 7 servers to your Linux 8 servers using the docker save and docker load commands:
-
On the Linux 7 server, run docker save:
docker save -o /<docker_backup_directory>/<docker_image_name>.tgz <docker_image_name>:<tag>
In the command:
-
<docker_backup_directory> is the directory where you want to save the backup.
-
<docker_image_name> is the name for the image.
-
<tag> is the tag to use to refer to the image.
-
-
Copy <docker_image_name>.tgz to your Linux 8 server.
-
On the Linux 8 server, run docker load:
docker load -i /<docker_backup_directory>/<docker_image_name>.tgz
See the Docker documentation for full descriptions of Docker commands.
-
-
If you had any custom packages in the Linux 7 environment, recreate them.
Although the .pkg and .def files for custom packages were excluded from the Linux 7 backup, any files that were part of those packages and not in excluded directories were moved to the Linux 8 server.
See Creating Custom Packages in Unified Assurance Developer's Guide for more information about creating custom packages.
-
If you upgraded from version 5:
-
Enable DatabaseWatchdog on all servers. You can do this by using the Services UI.
-
Remove all version 5 related directories. On each server, run:
$A1BASEDIR/bin/SetupWizard --Finalize-Upgrade
-
Remove all Neo4j version 4 related directories. On each Graph database server, run:
$A1BASEDIR/bin/Package finalize-neo4j-upgrade
-
-
Perform any relevant Post-Update Tasks, including migrating your Historical database from Elasticsearch to OpenSearch and detaching and rejoining microservice clusters.
Skip the task for redeploying microservices. You already did this for each server during the upgrade process.
The forklift upgrade is complete.