Search and Visualize Data using Oracle Cloud Infrastructure Search with OpenSearch

Introduction

Oracle Cloud Infrastructure (OCI) Search with OpenSearch is an insight engine offered as an Oracle-managed service. With OCI’s capabilities for OpenSearch and OpenSearch Dashboards, you can store, search, and analyze large volumes of data quickly and see results in near real-time. Oracle automates, without any downtime, activities that include patching, updating, upgrading, backing-up, and resizing of the service.

Objectives

Prerequisites

  1. Create the required service policies in the OCI Console. Change any-user to your desired group, and provide the path to the compartment if required.

    Working policies:

    Allow group SearchOpenSearchAdmins to manage vnics in compartment <NETWORK_RESOURCES_COMPARTMENT>
Allow group SearchOpenSearchAdmins to manage vcns in compartment <NETWORK_RESOURCES_COMPARTMENT>
Allow group SearchOpenSearchAdmins to manage subnets in compartment <NETWORK_RESOURCES_COMPARTMENT>
Allow group SearchOpenSearchAdmins to use network-security-groups in compartment <NETWORK_RESOURCES_COMPARTMENT>
Allow group SearchOpenSearchAdmins to manage opensearch-family in compartment <CLUSTER_RESOURCES_COMPARTMENT>

  2. Create a VCN with a public subnet and a private subnet.

    • Simplified Process:

      1. Go to the OCI Console and open the navigation menu.

      2. Click Networking and Virtual Cloud Networks.

      3. Click Start VCN Wizard and Create VCN with Internet Connectivity.

    • Custom Process:

      1. Go to the OCI Console and open the navigation menu.

      2. Click Networking and Virtual Cloud Networks.

      3. Click Create VCN and enter your own desired details.

  3. Create a virtual machine (VM) instance in the public subnet of the VCN.

    1. Go to the OCI Console and open the navigation menu.

    2. Click Compute and Instances.

    3. Click Create instance.

    4. Enter the Name and select the compartment.

    5. In the Image and shape section, use the default values (Oracle Linux 8, VM.Standard.E4.Flex).

    6. In the Networking section, select the public subnet and assign a public IP.

    7. In the Add SSH keys section, decide whether you want to use an existing SSH key, or generate a new SSH key. If you select to generate a new SSH key, remember to download the key.

Note: In this tutorial, a Mac and an Oracle Linux instance were used. Windows command line instructions may differ.

Task 1: Create an OCI Search Service Cluster

  1. Go to the OCI Console, navigate to Databases, OCI Search Service, Clusters and click Create cluster. Enter the cluster name and compartment where you want to create the cluster and click Next.

    image

  2. Select the cluster size and click Next.

    image

  3. Select the VCN, the private subnet create in Prerequisites section and click Next.

    image

    After the cluster creation, in the OCI Search service cluster details page, note the API endpoints and the IP addresses which you can alternatively use.

    image

Task 2: Create Security Rules in the VCN Security List

In the VCN, create a security list with the following security rules. Alternatively, they can be added to the VCN default security list.

  1. In the Virtual cloud network details page, click Security Lists, Security List and Add Ingress Rules.

    Note: Add a rule for port 9200 (OpenSearch), and a rule for 5601 (OpenSearch Dashboards).

    image

    image

    image

Task 3: Test the Connection to OCI Search Service – OpenSearch Endpoint

You can use one of the following options.

Option 1: From Within the Created VM Instance

  1. Connect to the instance using SSH.

    ssh -i ~/.ssh/id_rsa_opensearch.key opc@<your_VM_instance_public_IP>

  2. Run one of the following commands.

    curl https://mycluster.opensearch.us.example.com:9200
# OpenSearch API endpoint example

    Or

    curl https://<your_opensearch_private_IP>:9200 --insecure
# OpenSearch private IP example

Option 2: From your Local Machine, through Port Forwarding

  1. Run the following port forwarding SSH command in the terminal. Do not close the terminal to ensure that the connection remains active.

    ssh -C -v -t -L 127.0.0.1:5601:<your_opensearch_dashboards_private_IP>:5601 -L 127.0.0.1:9200:<your_opensearch_private_IP>:9200 opc@<your_VM_instance_public_IP> -i <path_to_your_private_key>

  2. Open a new terminal window and run the following command.

    curl https://localhost:9200 --insecure

If the steps are performed correctly you should see a response as follows, regardless of the option used.

{
"name" : "opensearch-master-0",
"cluster_name" : "opensearch",
"cluster_uuid" : "M6gclrE3QLGEBlkdme8JkQ",
"version" : {
   "distribution" : "opensearch",
   "number" : "1.2.4-SNAPSHOT",
   "build_type" : "tar",
   "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
   "build_date" : "2022-02-08T16:44:39.596468Z",
   "build_snapshot" : true,
   "lucene_version" : "8.10.1",
   "minimum_wire_compatibility_version" : "6.8.0",
   "minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Task 4: Ingest Data

Run the following commands from within your VM instance.

  1. Download the sample data set.

    curl -O https://raw.githubusercontent.com/oracle-livelabs/oci/main/oci-opensearch/files/OCI_services.json

  2. Create mapping (optional). If you do not run this command or any variation of it, a default mapping will be automatically created.

    curl -XPUT "https://mycluster.opensearch.us.example.com:9200/oci" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
    "id": {"type": "integer"},
    "category": {"type": "keyword"},
   "text": {"type": "text"},
   "title": {"type": "text"},
   "url": {"type": "text"}
    }
  }
}
'

  3. Push the data set.

    curl -H 'Content-Type: application/x-ndjson' -XPOST "https://<your_opensearch_private_IP>:9200/oci/_bulk?pretty" --data-binary @OCI_services.json

  4. Check your indices.

    curl "https://mycluster.opensearch.us.example.com:9200/_cat/indices"
# OpenSearch API endpoint example

    curl -X GET "https://<your_opensearch_private_IP>:9200/_cat/indices" --insecure
# OpenSearch private IP example

Task 5: Query the OCI Search Service – Sample Search Query

You can use one of the following options.

Option 1: From the VM Instance Shell

curl -X GET "https://mycluster.opensearch.us.example.com:9200/oci/_search?q=title:Kubernetes&pretty"
# OpenSearch API endpoint example

curl -X GET "https://<your_opensearch_private_IP>:9200/oci/_search?q=title:Kubernetes&pretty"
# OpenSearch private IP example

Option 2: From your Local Terminal, After Port Forwarding

curl -X GET "https://localhost:9200/oci/_search?q=title:Kubernetes&pretty" --insecure

Option 3: From your Local browser, After Port Forwarding

https://localhost:9200/oci/_search?q=title:Kubernetes&pretty

For more information about query syntax, see the OpenSearch or Elasticsearch tutorials.

Task 6: Connect to OpenSearch Dashboards

  1. From your local machine, through port forwarding.

    Note: Skip this step if you have already run it in the previous steps and the connection is still open.

    ssh -C -v -t -L 127.0.0.1:5601:<your_opensearch_dashboards_private_IP>:5601 -L 127.0.0.1:9200:<your_opensearch_private_IP>:9200 opc@<your_instance_public_ip> -i <path_to_your_private_key>

  2. Access https://localhost:5601 in your browser.

    Note: Currently, depending on the browser, a warning message similar to Your connection is not private is displayed. Select the option which allows you to proceed anyway.

    image

Task 7: Search and Visualize Data in OpenSearch Dashboards

  1. With the port forwarding connection in place, access https://localhost:5601 in your browser.

  2. Open the OpenSearch Dashboards navigation menu.

  3. Navigate to Management, Stack Management and click Index Patterns. Create an index pattern with name oci.

    image

  4. Go to the OpenSearch Dashboards menu and click Discover to use the OpenSearch Dashboards UI to search your data.

    image

  5. Go to the OpenSearch Dashboards menu, click Dashboards and follow these steps to create a sample pie chart.

    1. Click Create new, New Visualization and Pie.

      image

    2. Select oci as Source.

    3. In Buckets, click Add and expand Split slices. Enter the information showing in the following image and click Update.

      image

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.