1. Using Oracle Big Data Cloud
  2. Use Identity Cloud Service for Cluster Authentication
  3. Make REST API Calls to Clusters That Use Identity Cloud Service

Make REST API Calls to Clusters That Use Identity Cloud Service

When Oracle Identity Cloud Service (IDCS) is selected for cluster authentication, the OAuth 2.0 authentication mechanism is enabled. The OAuth 2.0 token service provided by IDCS enables secure access to REST endpoints. This topic describes how to interact with OAuth-enabled clusters using REST.

To make REST API calls to an IDCS-enabled cluster, you’ll need to gather some information about the cluster, get an access token, and then use a REST client application such as cURL to perform REST API calls. Those steps are described in the following procedure.

To make REST API calls to an IDCS-enabled cluster:

  1. Open the service console. See Access the Service Console for Big Data Cloud.
  2. Click the name of the IDCS-enabled cluster.

    An overview page with cluster details is displayed. For information about the details on this page, see Service Console: Instance Overview Page.

  3. Expand Show more.
  4. Make note of the ID next to IDCS Application. This is the application ID for the cluster.
  5. Click the link next to IDCS Application and log in with your IDCS credentials.

    An instance of IDCS opens on the Application tab and lists cluster details.

  6. Get the client ID and client secret for the cluster application:
    1. In the IDCS console, click the Configuration tab for the cluster application and expand the General Information section.
    2. Make note of the client ID, then click Show Secret and make note of the client secret. The client secret is essentially the client password and should not be shared.
  7. Get the primary audience and scope:
    1. On the Configuration tab, expand the Resources section.
    2. Make note of the primary audience and scope.

      The primary audience identifies the cluster host and consists of the application ID and compute domain.

      There is currently just one scope (/). With this scope, all cluster resources are accessible to everyone who logs in with valid IDCS credentials.

      The Resources section also shows the expiration period for the access token. The access token provides a session (with scope and expiration) that your client application can use to perform tasks in IDCS using REST APIs. The expiration period for the token is one hour (3600 seconds). After one hour, you’ll need to get another access token to continue to make REST API calls to the cluster.

  8. Use the information you’ve gathered to create the REST request for the access token. The following steps use cURL to get the token:
    1. In a text editor, prepare the cURL command as follows: 
      curl -k -X POST -u "CLIENT_ID:CLIENT_SECRET" -d "grant_type=client_credentials&scope=PRIMARY_AUDIENCE/"
"IDCS_URL/oauth2/v1/token" -o access_token.json

      Where:

      • CLIENT_ID is the client ID.

      • CLIENT_SECRET is the client secret.

      • PRIMARY_AUDIENCE is the primary audience.

      • / after PRIMARY_AUDIENCE is the scope (this is currently the only scope available).

      • IDCS_URL is the Oracle Identity Cloud Service URL for the IDCS instance that’s associated with the cluster.

      For example: 

      curl -k -X POST -u "123456789ABCDEFGHIJK_APPID:b9008819-0f0b-44c3-b266-b07746f9d9f9"
-d "grant_type=client_credentials&scope=https://primary-audience-url.com:443/"
"https://IDCS-server.com/oauth2/v1/token" -o access_token.json
    2. At the command prompt, enter the cURL command you created in the previous step.
    3. Open the access token file (access_token.json) in a text editor and copy the access_token value.
  9. Use the access token to access the cluster. For IDCS authentication, the token type is Bearer.

    The following example demonstrates a REST API call to perform a lookup of available user directories in HDFS. The example is intended to illustrate how REST API calls to IDCS-enabled clusters are executed.

    1. In a text editor, prepare the cURL command as follows:
      curl -X GET -k https:/CSM_IDCS_URL/fs/v1/user?op=LISTSTATUS -H 'cache-control: no-cache'
-H  'x-user-identity-domain-name:IDENTITY_DOMAIN' -H 'authorization: Bearer ACCESS_TOKEN’

      Where:

      • CSM_IDCS_URL is the URL advertised for the Big Data Cloud Console (cluster console). For information about this URL, see Access the Big Data Cloud Console.

      • IDENTITY_DOMAIN is the identity domain configured for your service account.

      • ACCESS_TOKEN is the text of the access token you obtained in Step 8.

    2. At the command prompt, enter the cURL command you created in the previous step.

      For the lookup example above, the response should have a form similar to the following:

      {"FileStatuses":{"FileStatus":[ {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16388,"group":"hdfs",
"length":0,"modificationTime":1508957357539,"owner":"ambari-qa","pathSuffix":"ambari-qa","permission":"770",
"replication":0,"storagePolicy":0,"type":"DIRECTORY"}, {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16537,
"group":"bdcsce_admin","length":0,"modificationTime":1508957412913,"owner":"bdcsce_admin","pathSuffix":"bdcsce_admin",
"permission":"750","replication":0,"storagePolicy":0,"type":"DIRECTORY"}, {"accessTime":0,"blockSize":0,"childrenNum":0,
"fileId":16582,"group":"hdfs","length":0,"modificationTime":1508957486812,"owner":"hcat","pathSuffix":"hcat",
"permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}, {"accessTime":0,"blockSize":0,"childrenNum":2,
"fileId":16536,"group":"hive","length":0,"modificationTime":1508957507850,"owner":"hive","pathSuffix":"hive",
"permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}, {"accessTime":0,"blockSize":0,"childrenNum":1,
"fileId":16405,"group":"oracle","length":0,"modificationTime":1508957384444,"owner":"oracle","pathSuffix":"oracle",
"permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}, {"accessTime":0,"blockSize":0,"childrenNum":0,
"fileId":16389,"group":"hdfs","length":0,"modificationTime":1508957360272,"owner":"spark","pathSuffix":"spark",
"permission":"775","replication":0,"storagePolicy":0,"type":"DIRECTORY"}, {"accessTime":0,"blockSize":0,"childrenNum":3,
"fileId":16545,"group":"hdfs","length":0,"modificationTime":1508957446717,"owner":"zeppelin","pathSuffix":"zeppelin",
"permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"} ]}}

      Note that if the access token you’re using has expired, you’ll see a response such as the following:

      <html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx</center>
</body>
</html>