Register Storage Providers with Your Cluster
Use bda-oss-admin
commands to configure command shell access to your Hadoop cluster and access to storage providers. You can then use odcp
and bdm-cli
commands to copy data between HDFS and storage.
Set bda-oss-admin Environment Variables
Some bda-oss-admin
options can be set as environment variables so you don’t have to specify the values every time you run the commands.
The following tables show the environment variables that correspond to (and can be substituted for) those bda-oss-admin
options.
Options and Environment Variables for All bda-oss-admin Commands
The following values must be set for all bda-oss-admin
commands, either on the command line as options to the command or as shell environment variables.
Command Option | Environment Variable | Description |
---|---|---|
--cm-admin |
CM_ADMIN |
Cloudera Manager administrator user name. |
--cm-passwd |
CM_PASSWORD |
Cloudera Manager administrator password. The command will prompt for the password if it is not provided. |
--cm-url |
CM_URL |
Cloudera Manager URL (e.g.
https://servername.bigdata.examplecloud.com:7183 )
|
Options and Environment Variables for bda-oss-admin Storage Credentials Commands
bda-oss-admin
storage credentials commands that require some or all of the following options are:
See Register Storage Credentials with the Cluster for more complete descriptions of the options.
Command Option | Environment Variable | Description |
---|---|---|
--swift-username |
ST_USER |
Oracle Cloud Infrastructure Object Storage Classic account administrator’s user name in the form User names must conform to the rules described in Naming Requirements. |
--swift-password |
ST_KEY |
The administrator’s password. |
--swift-storageurl |
ST_AUTH |
The Oracle Cloud Infrastructure Object Storage Classic (Swift) authentication URL from which to obtain authentication tokens, for
example, |
--swift-provider |
ST_PROVIDER |
A user-supplied name for the credentials. This name is provided when using Provider names must conform to the rules described in Naming Requirements. Note: When you create a Hadoop instance by using the Create Instance wizard, a default provider name,
|
Setting Environment Variables
To set these as environment variables, you can create and run a shell script.
In the following example, a a Linux bash shell script named bdcsvars.sh
sets the Cloudera Manager credentials required by all bda-oss-admin
commands:
#!/bin/bash
export CM_ADMIN="my_CM_admin_username"
export CM_PASSWORD="my_CM_admin_password"
export CM_URL="https://my_CM_hostname_:7183"
In the following example, a Linux bash shell script named storvars.sh
sets the storage credentials :
#!/bin/bash
export ST_USER="MyServiceName-MyIdentityDomain:MyUserName"
export ST_KEY="Wel_123"
export ST_AUTH="http://storage.a123456.examplecloud.com/auth/v1.0"
export ST_PROVIDER="MyProviderName"
If you are working with multiple storage providers, it may be convenient to create shell scripts for all of them. Then you only have to run a script to set storage credentials for whichever provider you are using.
To run the above scripts from the directory in which they reside:
# source ./bdacsvars.sh
# source ./storvars.sh
Reviewing the Configuration
Configurations that you set with the bda-oss-admin
commands, parameters, and environment variables are stored in the Hadoop /etc/hadoop/conf/core-site.xml
configuration file. For example:
<configuration>
...
<property>
<name>fs.swift.service.storageservice3991-bdaoss.username</name>
<value>john.smith@example.com</value>
</property>
<property>
<name>fs.swift.service.storageservice3991-bdaoss.tenant</name>
<value>storageservice3991-bdaoss</value>
</property>
<property>
<name>fs.swift.service.storageservice3991-bdaoss.password</name>
<value>A_password</value>
</property>
<property>
<name>fs.swift.service.storageservice3991-bdaoss.auth.url</name>
<value>https://storageservice3991-bdaoss.storage.examplecloud.com/auth/v2.0/tokens</value>
</property>
<property>
<name>fs.swift.service.storageservice3991-bdaoss.public</name>
<value>true</value>
</property>
</configuration>
You can look in the core-site.xml
file to see the current configuration, but you shouldn’t edit it directly. Use bda-oss-admin
instead.
Register Storage Credentials with the Cluster
When you create a cluster, you can register with it the credentials of a specific user of a specific instance of Oracle Cloud Infrastructure Object Storage Classic (formerly known as Oracle Storage Cloud Service). That allows that user to copy data to and from that storage instance, without having to reestablish credentials. After the cluster is created, you can register other users and other storage service instances, so those users can copy data to and from those storage service instances.
Prerequisite
You must have access to an Oracle Cloud Infrastructure Object Storage Classic instance.
About Storage Credentials
When you create a cluster using the Create Cluster wizard and you choose to associate it with an Oracle Cloud Infrastructure Object Storage Classic instance, you’re prompted for a user name and a password. (See Create a Cluster.) The user name identifies the storage service instance, its identity domain, and a user with administrator rights to the instance. When the cluster is created, a default storage service provider , named BDCS, is also created for that storage server instance. The provider is an alias for all the credentials required for using the storage service instance. Those credentials are: the user name (service instance name, administrrator user name, and administrrator user’s password) and the URL for a server that provides authentication tokens for accessing the service. See Storage Credential Details, below.
If this is the only storage service instance that will be used with the Hadoop cluster, you don’t have to register any other credentials. However, if you need to access other storage service instances and if you want to register other credentials for them, you can do so by using bda-oss-admin
commands. You set the values for the credentials either by entering them as parameters to bda-oss-admin
commands. See Using odcp to Copy Data.
Storage Credential Details
The components of the credentials are described in the following table.
Note:
Oracle Cloud Infrastructure Object Storage Classic is based on Swift, the open-source OpenStack Object Store project. As a consequence, thebda-oss-admin
command line utility contains some subcommands and options that say “swift.”
Storage Credentials, Syntax, and Usage | Description |
---|---|
Oracle Cloud Infrastructure Object Storage Classic account administrator’s user name in the form
You can find the storage details described above on the Service Details page of your Oracle Cloud Infrastructure Object Storage Classic account. See Accessing Oracle Cloud Infrastructure Object Storage Classic in Getting Started with Oracle Cloud. User names must conform to the rules described below in Naming Requirements. Example: |
|
Password associated with If the password is Base64–encoded, use the |
|
The Oracle Cloud Infrastructure Object Storage Classic (Swift) \ authentication URL for your Oracle Cloud Infrastructure Object Storage Classic data center/region, for example, |
|
A user-supplied name for the credentials. This name is provided when using Provider names must conform to the rules described below in Naming Requirements. Note: When you create a Hadoop cluster by using the Create Cluster wizard, a default provider name,
|
Naming Requirements
Provider names and container names must conform to RFC952 DOD Internet Host Table Specification, for example:
-
A name is a text string of 2 to 24 characters and can contain only:
-
Letters A-Z and a-z (case is not considered)
-
Digits 0–9
-
Hyphen (-)
-
-
The first character must be an alpha character.
-
The last character cannot be a period or hyphen.
-
Spaces are not allowed.
-
Periods are allowed only to delimit components of domain style names.