3 Pre-installation

This chapter provides information about the tasks that must be performed before installing FCC Studio.  

Topics:

·        Prerequisites

·        Hardware and Software Requirements

·        Prerequisite Environmental Settings

·        Download the Big Data Files

·        Validation Checklist

·        Configure the Elastic Search Component

·        Configure the Interpreter Settings

·        Create the Studio Schema

·        Assign Grants for the Database Schemas

·        Setup Password Stores with Oracle Wallet

·        Create the Credential Keystore

·        Download the Installer Kit

Prerequisites

The Linux machine must satisfy the following conditions:

·        BD (Behavior Detection) or the ECM (Enterprise Case Management) application pack must be installed.

·        Kubernetes (k8s) cluster must be installed with the following:

§       Registry to store docker images.

§       Minimum of 8 GB memory (inclusive of all nodes) available for the installation. The actual memory requirement depends on the workload/container size configuration.

·        Docker and kubectl must be installed.

·        kubectl is configured (that is, connected to cluster where you want to install FCC Studio).

·        Docker has push access to a private registry.

·        12GB free space is available to store the FCC Studio Installer zip file in some directory.

·        45GB free space is available in the docker root directory. You can find the docker root directory using the docker info command.

Hardware and Software Requirements

The following hardware and software are required for this version of FCC Studio. The installation environment or setup must have these requirements for an application to run smoothly and efficiently.

Table 5: Hardware and Software Requirements

Prerequisite Environmental Settings

The following prerequisite environmental settings must be set before beginning the installation of FCC Studio. These settings are the configuration that a system must have for an application to run smoothly and efficiently.

Table 6: Prerequisite Environmental Settings

Download the Big Data Files

Download the following configuration files from the Cloudera installation setup:

Table 7: Required File Structure

Validation Checklist

The Validation Checklist section provide you the information about the parameters that you can validate to avoid installation issues. This section explains the validation and actions that can be taken for some of the common parameters that are used in the studio-env.yml file for the installation. The parameters that can be validated are as follows:

Table 8: Required File Structure

Configure the Elastic Search Component

To configure the Elastic Search component, follow these steps:

1.     Navigate to the elasticsearch-7.6.2/config directory.

2.     Configure the elasticsearch.yml with the following variables:

Table 9: elasticsearch.yml File

3.     Configure the jvm.options file as follows:

Table 10: elasticsearch.yml File

4.     Enter the URL in the following format into the browser:

http://<network.host>:<http.port>

The following output is displayed to indicate successful installation of the Elastic Search service.

{

  "name" : "node-1",

  "cluster_name" : "my-application",

  "cluster_uuid" : "_D-2qEGGSgKQPd3i-UtgWQ",

  "version" : {

    "number" : "7.3.2",

    "build_flavor" : "oss",

    "build_type" : "tar",

    "build_hash" : "1c1faf1",

    "build_date" : "2019-09-06T14:40:30.409026Z",

    "build_snapshot" : false,

    "lucene_version" : "8.1.0",

    "minimum_wire_compatibility_version" : "6.8.0",

    "minimum_index_compatibility_version" : "6.0.0-beta1"

  },

  "tagline" : "You Know, for Search"

}

Configure the Interpreter Settings

You must configure the interpreter settings for the following interpreters before installing FCC Studio.

Table 11: Pre-installation Interpreter Settings

Topics:

·        Configure the fcc-jdbc Inter­preter

·         Configure the fcc-ore Inter­preter 

·        Configure the fcc-python Inter­preter

·        Configure the Spark Inter­preter

·        Configure the PySpark Inter­preter

Configure the fcc-jdbc Inter­preter

To create context for the fcc-jdbc interpreter, follow these steps:

1.     Log in to Oracle Database as a SYSDBA user.

2.     Grant Execute permission to the user using the following com­mand:

grant execute dbms_rls to <Studio_DB_Username>;

The Execute permission is granted to user.

3.     Grant Create permission to the context using the following com­mand:

grant create any context to <Studio_DB_Username>;

The Create permission is granted to context.

Configuring the fcc-ore Interpreter

To configure the fcc-ore interpreter you must install the Oracle R Distribution.

  Installing the Oracle R Distribution

To install the Oracle R Distribution (ORD), enable the addons and optional_latest channels in yum as shown in the following code:

```bash

(root)# yum-config-manager --enable ol7_addons

(root)# yum-config-manager --enable ol7_optional_latest

```

After completing the previous step, pull ORD from the yum repository using the following command:

```bash

(root)# yum install R.x86_64 R-core-extra

```

To install ORD, see Install Oracle R Distribution on Linux.

Configure the fcc-python Interpreter

The fcc-python interpreter image includes the following:

·        The support for python2 has been deprecated.

·         By default, the Python interpreter points to the Python 3.6 version.

·        The following Python libraries are part of the fcc-python interpreter images for Python 3.6 version:

§       pandas 0.25.3

§       numpy 1.17.4

§        scipy 1.3.2

§       scikit-learn 0.21.3

§       matplot-lib 3.1.1

§       seaborn 0.9.0

§       cx-oracle 7.2.2

§        sqlalchemy 1.3.11

To modify the Python packages in the Python 3.6 or to add a different version of Python to the Python interpreter, you must modify the Python image. As an example, follow the steps provided in the Modifying the Python Images for the Python Interpreter section in the OFS Crime and Compliance Studio Administration Guide.

Prerequisites for Installing Python Libraries

Following are the prerequisites to install Python libraries:

·        FCC Studio supports any-version of Python with the following required conditions:

§       The preferred Python version is installed on the Processing Server (Studio Notebook Server) and is accessible to the Linux user of FCC Studio.

§       The Python Library (py4j) package is present for the preferred Python version.

·        The prerequisites to install the Python 3.6 libraries that are packaged with the FCC Studio Installer are as follows:

§       The Python3.6 version is installed on the Processing Server (Studio Notebook Server) and is accessible to the Linux user of FCC Studio.

§       The Python Library (py4j) package is installed and is accessible to the Linux user of FCC Studio.

Configure the Spark Interpreter

FCC Studio uses spark 2.4.0. Before you begin the configurations, check the prerequisites depending on your operation mode.

·        Prerequisites

·        Configuration

Prerequisites for using the Spark Interpreter

To operate the Spark interpreter in local mode or yarn mode, perform the following:

·        Spark Master Settings

·        Local Mode

·        Yarn Mode

Spark Master Settings

To operate the Spark interpreter in local mode or Yarn mode, the spark.master property must be set accordingly.

You can update the Spark Master by changing the default value of the spark.master property in the Interpreter Settings of the Spark interpreter (spark.json) or the Interpreters page of the FCC Studio application UI after startup.

·        The default value of spark.master property is local[*], which means that the interpreter will run in local mode.

·        For a Yarn cluster, you must change the default value of spark.master property to yarn-client.

Local Mode

No additional configuration is required to operate the Spark interpreter in local mode.

Yarn Mode

To operate the Spark interpreter in yarn mode, you must provide custom Spark libraries. To provide your own Spark libraries and (or) Hadoop client-configuration files for the Spark interpreter to connect to a Spark Standalone or Yarn cluster, follow these steps:

a.     Download the Spark libraries from the Spark's Offi­cial Release Archive.

b.     Prepare Spark libraries by following these steps:

i.       Place your libraries in the <Studio_Installation_Path>/docker/user/spark-interpreter-libraries/ directory.

ii.      To use separate Hadoop libraries, download them as well and place the Hadoop libraries directory in the same directory.

iii.   Be cautious when linking the two libraries, since the path where they are located is in the K8s pod, and the location is as follows:

/var/olds-spark-interpreter/interpreter/spark/libs/

c.     Change the image of the Spark interpreter Init Container in the <Studio_Installation_Path>/deployments/spark.yml file to {{imageReposi­tory}}/fcc-studio/3rdparty:init.

d.     (Optional) Place your Hadoop Client Configuration files in the <Stu­dio_Installation_Path>/configmaps/spark-interpreter-conf/ direc­tory.

The cluster's Hadoop client-side configuration files that include XML files such as yarn-site.xml are required and must be supplied with the Spark libraries. These files are available in the Hadoop configuration directory (HADOOP_CONF_DIR) of the cluster or can be downloaded from the cluster manager's user interface if you are using a Cloudera cluster.

Configuration

The Spark interpreter configuration can be divided into the following categories:

·        Configuration related to deployment

These properties can be set either in the Spark libraries, for example, the spark-defaults.conf file, or through the system environment variable, SPARK_CONF, for example, SPARK_CONF="--conf spark.driver.memory=2g".

·        Configuration related to Spark runtime control

These properties can be set from the Interpreters page of the FCC Studio application UI. This includes properties such as spark.executor.memory.

A list of possible properties are available in the  Spark Official Documentation. All the properties prefixed with the term "zeppelin", that are listed in the  Zeppelin Spark Configuration Document can also be set from the Interpreters page of the FCC Studio application user interface.

Configure the PySpark Interpreter

FCC Studio uses PySpark 2.4.0. Before you begin the configurations, check the prerequisites depending on your operation mode.

·        Prerequisites

·        Configuration

·        Use the Python Virtual Environments with PyS­park

Prerequisites

The PySpark interpreter has the same prerequisites as that as the Spark Interpreter. For more information, see Configure the Spark Interpreter. Also, all Spark components must be configured to use the same Python version.

Configuration

The PySpark interpreter can be configured through the Spark interpreter with the only exception being the Python version used. By default, the Python version is set to 3, that can be changed either in the interpreter JSON files before the startup or from the Interpreters page of the FCC Studio application UI during runtime by changing the following properties:

·        In the Spark Interpreter Settings page of the FCC Studio application UI (or spark.json file), change the value of the spark.pyspark.python property to the path of the Python executable that is to be used by the Spark executors.

·        In the PySpark Interpreter Settings page of the FCC Studio application UI (or pyspark.json file), change the value of the zeppelin.pyspark.python property to the path of the Python executable that is to be used by the Spark driver.

Use the Python Virtual Environments with PyS­park

To ensure that the two Python versions match, in case your components run on different machines, you must use the Python virtual environments with PySpark.

To use Python Virtual Environments with PySpark, follow these steps:

1.     Create a Virtual Environment with Conda

2.     Include Virtual Environment in the Init Container

3.     Update the Interpreter Properties

Create a Virtual Environment with Conda

To create a virtual environment with Conda, follow these steps:

1.     Ensure that you have conda and conda-Pack installed.

2.     Create your virtual environment using the following command:

conda create -y -n <environment-name> python=<python-version> <additional-packages>

3.     Activate your virtual environment using the following command:

conda activate <environment-name>

4.     Execute the following to obtain the path to your virtual environment:

which python

The obtained result is referred to as <environment-abs-path>.

5.     Compress your virtual environment using the following command:

conda pack -n <environment-name> -o <environment-abs-path>/<environment-name>.tar.gz

Include Virtual Environment in the Init Container

To include the virtual environment in the Init container, you must place the Virtual Environment in the same path as the Spark libraries. To provide your own Spark libraries and (or) Hadoop client-configuration files for the Spark interpreter to connect to a Spark Standalone or Yarn cluster, follow these steps:

a.     Download the Spark libraries from the Spark's Offi­cial Release Archive.

b.     Prepare Spark libraries by following these steps:

iv.   Place your libraries in the <Studio_Installation_Path>/docker/user/spark-interpreter-libraries/ directory.

v.      To use separate Hadoop libraries, download them as well and place the Hadoop libraries directory in the same directory.

vi.   Be cautious when linking the two libraries, since the path where they are located is in the K8s pod, and the location is as follows:

/var/olds-spark-interpreter/interpreter/spark/libs/

c.     Change the image of the Spark interpreter Init Container in the <Studio_Installation_Path>/deployments/spark.yml file to {{imageReposi­tory}}/fcc-studio/3rdparty:init.

d.     (Optional) Place your Hadoop Client Configuration files in the <Stu­dio_Installation_Path>/configmaps/spark-interpreter-conf/ direc­tory.

The cluster's Hadoop client-side configuration files that include XML files such as yarn-site.xml are required and must be supplied with the Spark libraries. These files are available in the Hadoop configuration directory (HADOOP_CONF_DIR) of the cluster or can be downloaded from the cluster manager's user interface if you are using a Cloudera cluster.

Update the Interpreter Properties

The interpreter properties can either be configured in the interpreter JSON files or from the Interpreters page of the FCC Studio application UI after starting the FCC Studio application.

·        In the Spark Interpreter Settings page of the FCC Studio application UI (or spark.json), change the following:

§        Change the value of the spark.yarn.dist.archives property to <environment-abs-path>/<environ­ment-name>.tar.gz#<environment-name>

§        Change the value of the spark.pyspark.python property to ./<environment-name>/bin/python

·        In the PySpark Interpreter Settings page of the FCC Studio application UI (or pyspark.json), change the value of the zeppelin.pyspark.python parameter to <environment-abs-path>/bin/python.

Create the Studio Schema

To create studio schema, create a new Oracle Database schema user using the following script:

CREATE USER <Studio Schema User Name> IDENTIFIED BY <Password>;

A new Oracle Database schema (Studio schema) is created.

Assign Grants for the Database Schemas

Grant the following permissions to the newly created Oracle Database Schema:

·        GRANT create session TO <Studio Schema User>;

·        GRANT create table TO <Studio Schema User>;

·        GRANT create view TO <Studio Schema User>;

·        GRANT create any trigger TO <Studio Schema User>;

·        GRANT create any procedure TO <Studio Schema User>;

·        GRANT create sequence TO <Studio Schema User>;

·        GRANT execute on dbms_rls TO <Studio Schema User>;

·        GRANT execute on sys.dbms_session TO <Studio Schema User>;

·        ALTER USER <Studio Schema User> QUOTA 100M ON users;

·        GRANT create sequence TO <Studio Schema User>;

·        GRANT create SYNONYM TO <Studio Schema User>;

·        GRANT create any context TO <BD or ECM Schema User>;

·        GRANT execute on dbms_rls TO <BD or ECM Schema User>;

·        GRANT ALL privileges TO <Studio Schema User>;

Setup Password Stores with Oracle Wallet

As part of an application installation, administrators must set up password stores for database user accounts using Oracle Wallet. These password stores must be installed on the application database side. The installer handles much of this process, the administrators must perform some additional steps.

A password store for the application and application server user accounts must also be installed; however, the installer takes care of this entire process.

Topics:

·        Setup the Password Stores for Database User Accounts

·        Verify the Connectivity of the Wallet

Setup the Password Stores for Database User Accounts

After the database is installed and the default database user accounts are set up, administrators must set up a password store using the Oracle Wallet. This involves assigning an alias for the user name and associated password for each database user account. The alias is used later during the application installation. This password store must be created on the system where the application server and database client are installed.

This section describes the steps to create a wallet and the aliases for the database user accounts. For more information on configuring authentication and password stores, see Oracle Database Security Guide.

To create a wallet, follow these steps:

1.     Login to the server as a Linux user.

2.     Create a wallet in the <wallet_location> using the following command:

mkstore -wrl <wallet_location> -create

After you run the command, a prompt appears. Enter a password for the Oracle Wallet in the  prompt.

Figure 1:   Wallet Creation

3.     Create the database connection credentials for the studio schema alias using the following command:

mkstore -wrl <wallet_location> -createCredential <alias-name> <database-user-name>

After you run the command, a prompt appears. Enter the password associated with the database user account in the prompt. You are prompted to re-enter the password. You are prompted for the wallet password used in Step 1.

4.     Create the database connection credentials for the atomic schema alias using the following command:

mkstore -wrl <wallet_location> -createCredential <alias-name> <database-user-name>

5.     Create the database connection credentials for the config schema alias using the following command:

mkstore -wrl <wallet_location> -createCredential <alias-name> <database-user-name>

In this manner, create a wallet and associated database connection credentials for all the database user accounts.

The wallet is created in the configmaps/<wallet_location> directory with the auto-login feature enabled. This feature enables the database client to access the wallet contents without using the password. For more information, see Oracle Database Security Guide.

After the wallet is created, go to the <wallet_location> directory and click Refresh to view the created wallet folder.

Figure 2:   Location of the Created Wallet Folder

The wallet folder contains two files: ewallet.p12 and cwallet.sso.

6.     Move the wallet folder to the <Studio_Installed_Path>/<alias-name> directory.

7.     In the <wallet_location> directory, configure the tnsnames.ora file to include the entry for each alias name to be set up.

Figure 3:   Location of the Created Wallet Folder

Verify the Connectivity of the Wallet

To verify the connectivity of the wallet, follow these steps:

1.     Create a sqlnet.ora file in the wallet directory using the following content:

WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = <Wallet_Location>)) )

SQLNET.WALLET_OVERRIDE=TRUE

SSL_CLIENT_AUTHENTICATION=FALSE

2.     Test the connectivity using the following command:

$ export WALLET_LOCATION=<wallet_location>

$ export TNS_ADMIN=<tnsnames.ora_location>. If you have created a new tnsnames.ora file, provide the location of the new file.

$ sqlplus /@<alias_name>

The output is similar to:

SQL*Plus: Release 11

Connected to:

Oracle Database 12c

To verify if you are connected to the correct user:

SQL> show user

The output is similar to:

USER is "<database-user-name>"

Create the Credential Keystore

Credential keystore must be created for the Behavior Detection (BD) or Enterprise Case Management (ECM) Atomic schema and FCC Studio Schema. To create a credential keystore, follow these steps:

1.     Login as HDFS Superuser.

2.     Create a credential keystore on HDFS using the following command:

hadoop credential create mydb.password.alias -provider jceks://hdfs/user/root/oracle.password.jceks

3.     Verify the credential keystore file using the following command:

hadoop credential list -provider jceks://hdfs/user/root/oracle.password.jceks

4.     Grant Read permission to the keystore file using the following command:

hadoop fs -chmod 744 /user/root/oracle.password.jceks

Copying and Adding Files

To copy the jar files, follow these steps:

1.     Create the folder in the GRAPH_FILES_PATH parameter in a node of the big data cluster.

2.     Create a folder called jars.

3.     Navigate to the <STUDIO_INSTALLED_PATH>/ficdb/etlJars directory and copy all the jars present in this directory into the big data node inside the folder jars.

Download the Installer Kit

To download the software as a .zip folder, download the following mandatory one-off patch 31761460 from My Oracle Support (MOS).