Register Storage Providers with Your Cluster

Use bda-oss-admin commands to configure command shell access to your Hadoop cluster and access to storage providers. You can then use odcp and bdm-cli commands to copy data between HDFS and storage.

Topics

Set bda-oss-admin Environment Variables

Some bda-oss-admin options can be set as environment variables so you don’t have to specify the values every time you run the commands.

The following tables show the environment variables that correspond to (and can be substituted for) those bda-oss-admin options.

Options and Environment Variables for All bda-oss-admin Commands

The following values must be set for all bda-oss-admin commands, either on the command line as options to the command or as shell environment variables.

Command Option	Environment Variable	Description
`--cm-admin`	`CM_ADMIN`	Cloudera Manager administrator user name.
`--cm-passwd`	`CM_PASSWORD`	Cloudera Manager administrator password. The command will prompt for the password if it is not provided.
`--cm-url`	`CM_URL`	Cloudera Manager URL (e.g. `https://servername.bigdata.examplecloud.com:7183`)

Options and Environment Variables for bda-oss-admin Storage Credentials Commands

The bda-oss-admin storage credentials commands that require some or all of the following options are:

See Register Storage Credentials with the Cluster for more complete descriptions of the options.

Command Option	Environment Variable	Description
`--swift-username`	`ST_USER`	Oracle Cloud Infrastructure Object Storage Classic account administrator’s user name in the form `Service-IdentityDomain:username`. User names must conform to the rules described in Naming Requirements.
`--swift-password`	`ST_KEY`	The administrator’s password.
`--swift-storageurl`	`ST_AUTH`	The Oracle Cloud Infrastructure Object Storage Classic (Swift) authentication URL from which to obtain authentication tokens, for example, `https://storage.a123456.examplecloud.com/auth/v1.0`.
`--swift-provider`	`ST_PROVIDER`	A user-supplied name for the credentials. This name is provided when using `add_swift_cred` to add credentials. The name is used as an alias for the credentials in Hadoop. Provider names must conform to the rules described in Naming Requirements. Note: When you create a Hadoop instance by using the Create Instance wizard, a default provider name, `BDCS`, is created. If you do not provide a storage provider name using the `--swift-provider name` option, you can use `BDCS` whenever the storage provider name is needed; for example, when using a command like: `hdfs dfs —cat mycontainer.BDCS/file`

Setting Environment Variables

To set these as environment variables, you can create and run a shell script.

In the following example, a a Linux bash shell script named bdcsvars.sh sets the Cloudera Manager credentials required by all bda-oss-admin commands:

#!/bin/bash
export CM_ADMIN="my_CM_admin_username"
export CM_PASSWORD="my_CM_admin_password"
export CM_URL="https://my_CM_hostname_:7183"

In the following example, a Linux bash shell script named storvars.sh sets the storage credentials :

#!/bin/bash
export ST_USER="MyServiceName-MyIdentityDomain:MyUserName"
export ST_KEY="Wel_123"
export ST_AUTH="http://storage.a123456.examplecloud.com/auth/v1.0"
export ST_PROVIDER="MyProviderName"

If you are working with multiple storage providers, it may be convenient to create shell scripts for all of them. Then you only have to run a script to set storage credentials for whichever provider you are using.

To run the above scripts from the directory in which they reside:

# source ./bdacsvars.sh
# source ./storvars.sh

Reviewing the Configuration

Configurations that you set with the bda-oss-admin commands, parameters, and environment variables are stored in the Hadoop /etc/hadoop/conf/core-site.xml configuration file. For example:

<configuration>
...
  <property>
    <name>fs.swift.service.storageservice3991-bdaoss.username</name>
    <value>john.smith@example.com</value>
  </property>
  <property>
    <name>fs.swift.service.storageservice3991-bdaoss.tenant</name>
    <value>storageservice3991-bdaoss</value>
  </property>
  <property>
    <name>fs.swift.service.storageservice3991-bdaoss.password</name>
    <value>A_password</value>
  </property>
  <property>
    <name>fs.swift.service.storageservice3991-bdaoss.auth.url</name>
    <value>https://storageservice3991-bdaoss.storage.examplecloud.com/auth/v2.0/tokens</value>
  </property>
  <property>
    <name>fs.swift.service.storageservice3991-bdaoss.public</name>
    <value>true</value>
  </property>
</configuration>

You can look in the core-site.xml file to see the current configuration, but you shouldn’t edit it directly. Use bda-oss-admin instead.

Register Storage Credentials with the Cluster

When you create a cluster, you can register with it the credentials of a specific user of a specific instance of Oracle Cloud Infrastructure Object Storage Classic (formerly known as Oracle Storage Cloud Service). That allows that user to copy data to and from that storage instance, without having to reestablish credentials. After the cluster is created, you can register other users and other storage service instances, so those users can copy data to and from those storage service instances.

Prerequisite

You must have access to an Oracle Cloud Infrastructure Object Storage Classic instance.

About Storage Credentials

When you create a cluster using the Create Cluster wizard and you choose to associate it with an Oracle Cloud Infrastructure Object Storage Classic instance, you’re prompted for a user name and a password. (See Create a Cluster.) The user name identifies the storage service instance, its identity domain, and a user with administrator rights to the instance. When the cluster is created, a default storage service provider , named BDCS, is also created for that storage server instance. The provider is an alias for all the credentials required for using the storage service instance. Those credentials are: the user name (service instance name, administrrator user name, and administrrator user’s password) and the URL for a server that provides authentication tokens for accessing the service. See Storage Credential Details, below.

If this is the only storage service instance that will be used with the Hadoop cluster, you don’t have to register any other credentials. However, if you need to access other storage service instances and if you want to register other credentials for them, you can do so by using bda-oss-admin commands. You set the values for the credentials either by entering them as parameters to bda-oss-admin commands. See Using odcp to Copy Data.

Storage Credential Details

The components of the credentials are described in the following table.

Note:

Oracle Cloud Infrastructure Object Storage Classic is based on Swift, the open-source OpenStack Object Store project. As a consequence, the bda-oss-admin command line utility contains some subcommands and options that say “swift.”

Storage Credentials, Syntax, and Usage	Description
STORAGE USER NAME Option Syntax `--swift-username Service-IdentityDomain:username[.role]` Environment Variable `ST_USER` Required for `add_swift_cred`	Oracle Cloud Infrastructure Object Storage Classic account administrator’s user name in the form `Service-IdentityDomain:username`, where `Service` is the customer-supplied name of the Oracle Cloud Infrastructure Object Storage Classic instance. This name was given when the storage service was provisioned. `IdentityDomain` is the identity domain in which the service instance is provisioned. This name was given when the storage service was provisioned. (In Swift, this is called the tenant ID.) `username` is the user name for a user with administrative privileges on the account. `role` is a role defined in an Access Control List (ACL) assigned to a container. See Setting Container Metadata in Using Oracle Cloud Infrastructure Object Storage Classic for more information. If a role is not defined, this option is not required. You can find the storage details described above on the Service Details page of your Oracle Cloud Infrastructure Object Storage Classic account. See Accessing Oracle Cloud Infrastructure Object Storage Classic in Getting Started with Oracle Cloud. User names must conform to the rules described below in Naming Requirements. Example: `WestRegion-nloracle12345:pstuyves`
STORAGE PASSWORD Option Syntax `--swift-password password` Environment Variable `ST_KEY` Required for `add_swift_cred` `change_swift_cred_passwd`	Password associated with `username`, described above.. If the password is Base64–encoded, use the `-b` flag, for example, `--swift-password AnEncodedPassword —b`.
STORAGE LOCATION Option Syntax `--swift-storageurl url` Environment Variable `ST_AUTH` Required for `add_swift_cred`	The Oracle Cloud Infrastructure Object Storage Classic (Swift) \ authentication URL for your Oracle Cloud Infrastructure Object Storage Classic data center/region, for example, `https://storage.us2.oraclecloud.com/auth/v1.0`. For an explanation of how those URLs are constructed, see Authenticating Access When Using REST API.
STORAGE PROVIDER NAME Option Syntax `--swift-provider name` Environment Variable `ST_PROVIDER` Required for `add_swift_cred` `change_swift_cred_passwd` `delete_swift_cred`	A user-supplied name for the credentials. This name is provided when using `add_swift_cred` to add credentials. The name is used as an alias for the credentials in Hadoop. Provider names must conform to the rules described below in Naming Requirements. Note: When you create a Hadoop cluster by using the Create Cluster wizard, a default provider name, `BDCS`, is created. If you don’t provide a storage provider name using the `--swift-provider name` option, you can use `BDCS` whenever the storage provider name is needed; for example, when using a command like: `hdfs dfs —cat mycontainer.BDCS/file`

Naming Requirements

Provider names and container names must conform to RFC952 DOD Internet Host Table Specification, for example:

A name is a text string of 2 to 24 characters and can contain only:
- Letters A-Z and a-z (case is not considered)
- Digits 0–9
- Hyphen (-)
The first character must be an alpha character.
The last character cannot be a period or hyphen.
Spaces are not allowed.
Periods are allowed only to delimit components of domain style names.