After deploying a BDB JE HA application, you may later want to upgrade to a new version. Berkeley DB JE supports hot upgrade of a replication group, by allowing mixed version operation. That is, replication nodes running the newer software version can inter-operate with older version nodes and both can be available for user operations. However, in some cases, there are certain constraints to performing such a hot upgrade.
Each release of Berkeley DB JE is tied to a given log file on-disk format. Log file formats do not necessarily change every release. The Change Log for each release specifies whether the log file format has changed or not. There are no restrictions on upgrades across releases that use the same log file format. For best performance and to take advantage of bug fixes, ensure all nodes in a replication group run with the same BDB JE version during normal operations. Occasionally, a new release of Berkeley DB JE includes a log file format change. The constraints that apply to upgrades which introduce new log file formats are explained in this section.
In a replication group, the Master transmits log records that must be read by the replicas. If a group is operating with mixed version nodes, the Master must be running a version of JE that is older than, equal to, or (by default) no more than one version greater than the replicas. This is required so that the Master can supply the replicas with a version of the replication stream that they can understand.
Note that some releases may support online upgrades with replicas running versions that are more than one log file format version different from each other. But any such additional flexibility will be called out in the release notes.
To support the versioning requirement, make sure to upgrade all nodes to the version with the next log file format before upgrading any nodes to a still later log file format. If you cannot meet this restriction, then a hot upgrade should be performed. Instead, all nodes in the replication group should be taken offline and upgraded before any are restarted.
The recommended steps for upgrading Berkeley DB JE HA when log file formats have changed are as follows:
Determine if the upgrade skips any log file format versions. If it has, and it is not possible to upgrade to the intermediate versions, then plan to perform an offline upgrade.
Bring all Replicas up to date with the Master. That is, all Replicas must have the same environment content as the Master. You can ascertain this by using the DbGroupAdmin.dumpGroup() utility, or programmatically using the ReplicaConsistencyPolicy.
Perform the following upgrade procedures on each of the environments that are part of the replication group.
Shut down the old version of the application.
Install the new BDB JE jar file.
Restart the application.
Upgrade each individual data node, both Replicas and the Master, and restart them to join the replication group.
During a hot replication upgrade:
Upgrading BDB JE 4.0 directly to BDB JE 5 and higher versions is prohibited. Upgrade BDB JE 4.0 to BDB JE 4.1 first, and then upgrade BDB JE 4.1 to higher versions. There is no constraint if you upgrade from BDB JE 4.1 or later versions to a higher BDB JE version.
There are exceptions that you may run into during the upgrade process. The following exceptions may be thrown when a replication group node is restarted during the upgrade, and a ReplicatedEnvironment object is instantiated:
This exception can be thrown by a Replica when its log is ahead of the current Master and the Replica is unable to rollback without a recovery. As a consequence, one or more of the most recently committed transactions may need to be rolled back, before the Replica can synchronize its state with that of the current Master. This exception can also be thrown if the current Master crashes. To solve this exception restart the ReplicatedEnvironment with the new JE version.
During synchronization, a Replica that has a newer log, may have to roll back a number of committed tranactions. If the number of rolled back transactions exceeds the limit defined by TXN_ROLLBACK_LIMIT, the Replica throws a RollbackProhibitedException. To solve this exception you may have to truncate logs manually by using DbTruncateLog and restart the ReplicatedEnvironment with the new JE version.
EnvironmentFailureException is thrown due to log version incompatibility between the Master and the Replica. This exception is thrown with the message:
"Incompatible log versions. Feeder log version: xxxx, Feeder JE version: xxxx, Replica log version: xxxx, Replica JE version: xxxx".
To solve this exception restart the ReplicatedEnvironment with the new JE version.