Oracle Cloud Infrastructure Documentation

Cloning Cluster Configurations

Cloning cluster configuration based bootstrap helper functions exist to import and export cluster configurations.

The export and import support for Big Data Service ODH service configurations across source and target clusters provides a convenient way to move Big Data Service ODH environment between clusters for various reasons, saving time and effort.

Exporting and importing Big Data Service ODH service configurations is helpful in several scenarios;

Disaster recovery
To recover from a failed primary Big Data Service ODH cluster, use the import helper function to quickly create a new cluster and import configuration of your services.
Testing and development
To create a testing or development environment, use the export helper function to create a backup of the Big Data Service ODH cluster configuration.

To export Big Data Service ODH service configuration, use the OCI Console. The exported configuration includes information about your cluster configurations.

To import the Big Data Service ODH service configuration to a target cluster, use the Console. The import updates configurations of all services that are present in the zip file created by export operation.

Note

The export/import process doesn't include any data or metadata associated with your jobs. You must separately migrate data to the target cluster if needed.

To run a bootstrap script, see Running the Bootstrap Script.

For more information on the export and import helper functions, see:

Cloning Cluster Configurations Bootstrap Helper Functions

For more information on cloning cluster configurations bootstrap script, see Cloning Cluster Configurations.

For Cloning Cluster import and export helper function examples, see Export/Import Cluster Configuration Examples

To run a bootstrap script, see Running the Bootstrap Script.

Category Helper Function Functionality
Config Clusters exportClusterConfs(output_path, service_list=[] spcl_service_config_dict={}) This helper function creates a zipped file that contains the configurations of all services in JSON format.

Note:

  • output_path: The path where the zip file is located.
  • service_list: An optional list of services to perform export operations. If service_list is empty, it pulls all installed services present in the cluster.
  • spcl_service_config_dict: An optional dictionary that contains a mapping of services and a list of configurations to copy as-is from the source cluster. The default value is null.

When the job is successful. It gathers all the configurations for the services in the Big Data Service ODH environment and saves them in a JSON file and creates a zip of all such files in output_path.
importClusterConfs(zip_file_path, override_group=False) This helper function extracts the JSON file from the zip file and uses the JSON file to recreate the configurations for the services in the target cluster.

Note:

  • zip_file_path: The path to the zip file created by the export function on the source cluster. The exported zip file is placed either in the target cluster local path or in the Object Store location.
  • override_group: Default value is False. If there's some existing group in the target cluster with similar name as source cluster then this flag is used to override it or not.

Export/Import Cluster Configuration Examples

For more information on the exportClusterConfs and importClusterConfs helper functions, see Cloning Cluster Configurations Bootstrap Helper Functions.

Example: Sample Bootstrap Script for export zip creation with default values
sys.path.append(os.path.abspath("/home/opc/cloud/flask-microservice/"))
          
          def execute(updateConfigHelper):
          # existing utility functions, mention output location as "/tmp"
          cluster_name = updateConfigHelper.exportClusterConfs("/tmp")
Example: Sample Bootstrap Script for export zip creation with customized list of services
sys.path.append(os.path.abspath("/home/opc/cloud/flask-microservice/"))
          
          def execute(updateConfigHelper):
          # existing utility functions
          print("working hurrah")
          #Run export operation for mentioned list of services.
          service = ["HDFS","SPARK3","HIVE","YARN","MAPREDUCE2"]
          cluster_name = updateConfigHelper.exportClusterConfs("/tmp",service)
Example: Sample Bootstrap Script for export zip creation with customized dictionary that contains a mapping of services and a list of configurations to copy as-is from the source cluster
sys.path.append(os.path.abspath("/home/opc/cloud/flask-microservice/"))
          
          def execute(updateConfigHelper):
          # existing utility functions
          service_list = ["HDFS","SPARK3","HIVE","YARN","MAPREDUCE2"]
          spcl_service_config_dict={"HIVE":["javax.jdo.option.ConnectionURL"], "HDFS":["fs.defaultFS","ranger.plugin.hdfs.policy.rest.url"]}
          cluster_name = updateConfigHelper.exportClusterConfs("/tmp",service_list,spcl_service_config_dict)
Example: Sample Bootstrap Script for cluster import job
sys.path.append(os.path.abspath("/home/opc/cloud/flask-microservice/"))
          
          
          def execute(updateConfigHelper):
          # existing utility functions
          cluster_name = updateConfigHelper.importClusterConfs('/tmp/daily-cluster-ha6_configs_1683618209.zip')
          # Exported zip can be present in object store and you can provide object store path in zip_file_path location
          cluster_name = updateConfigHelper.importClusterConfs('https://objectstorage.us-ashburn-1.oraclecloud.com/.../cluster_configs_1683606387.zip')