Search and visualize data using OCI Search Service with OpenSearch

Introduction

OCI Search Service with OpenSearch is an insight engine offered as an Oracle-managed service. With OCI’s capabilities for OpenSearch and OpenSearch Dashboards, you can store, search, and analyze large volumes of data quickly and see results in near real-time. Oracle automates, without any downtime, activities that include patching, updating, upgrading, backing-up, and resizing of the service.

Objectives

Prerequisites

  1. Create the required service policies in the Oracle Cloud Console, tailoring them to your needs. For example: change any-user to the desired group, and provide the path to the compartment if required.

    Working policies:

    Allow group <your_group> to manage opensearch-family in compartment opensearch
    Allow service opensearch to manage vcns in compartment opensearch
    Allow service opensearch to manage vnics in compartment opensearch
    Allow service opensearch to use subnets in compartment opensearch
    Allow service opensearch to use network-security-groups in compartment opensearch
    
  2. Create a VCN with a public subnet and a private subnet.

    • Simplified process:
      1. Open the Oracle Cloud Console navigation menu.
      2. Click Networking. and then click Virtual Cloud Networks.
      3. Click Start VCN Wizard, and then click Create VCN with Internet Connectivity.
    • Custom process:
      1. Open the Oracle Cloud Console navigation menu.
      2. Click Networking. and then click Virtual Cloud Networks.
      3. Click Create VCN and provide your own desired details.
  3. Create a VM Instance in the public subnet of the VCN.

    1. Open the Oracle Cloud Console navigation menu.
    2. Click Compute, and then click Instances.
    3. Click Create instance.
    4. Enter the Name, and select the compartment.
    5. In the Image and shape area, use the default values (Oracle Linux 8, VM.Standard.E4.Flex).
    6. In the Networking area, select the public subnet and assign a public IP.
    7. In the Add SSH keys area, decide whether you want to use an existing SSH key, or generate a new SSH key. If you choose to generate a new SSH key, remember to download the key.

Note: For the exercise contained in this tutorial, a Mac and an Oracle Linux instance were used. Windows command line instructions may differ.

Task 1: Create an OCI Search Service cluster

  1. Open the Oracle Cloud Console navigation menu. Click Databases, OCI Search Service, and then click Clusters. Then, click Create cluster. Choose the cluster name and compartment where you want to create the cluster.

    Oracle Cloud Console screen - Configure cluster

  2. Click Next.

  3. Choose the cluster sizing, and then click Next.

    Oracle Cloud Console screen - Configure nodes

  4. Select the VCN you created and then select the private subnet.

    Oracle Cloud Console screen - Configure networking

  5. Click Next. After the cluster creation, in the OCI Search Service cluster details page, note the API endpoints and the IP addresses which you can alternatively use.

    Oracle Cloud Console screen - cluster details page, after cluster creation

Task 2: Create security rules in the VCN Security List

In the VCN, create a Security List with the following security rules. Alternatively, they can be added to the VCN Default Security List.

  1. Open the VCN details page, click Security Lists, select the Security List, and then click Add Ingress Rules.

    Note: Add a rule for port 9200 (OpenSearch), and a rule for 5601 (OpenSearch Dashboards).

Security rule for Search API

Security rule for Dashboards

Default Security List for opensearch-vcn - table view

Task 3: Test the connection to OCI Search Service – OpenSearch endpoint

Use one of the following options:

Option 1: From inside the created VM instance

Note: If attempting to execute the step found in this tutorial in clusters created prior to September 15th, 2022, it is necessary to download a certificate, and the option --cacert cert.pem must be used when running curl with the API endpoint.

Option 2: From your local machine, through port forwarding

If the steps are performed correctly you should see a response as follows, regardless of the option used:

{
"name" : "opensearch-master-0",
"cluster_name" : "opensearch",
"cluster_uuid" : "M6gclrE3QLGEBlkdme8JkQ",
"version" : {
   "distribution" : "opensearch",
   "number" : "1.2.4-SNAPSHOT",
   "build_type" : "tar",
   "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
   "build_date" : "2022-02-08T16:44:39.596468Z",
   "build_snapshot" : true,
   "lucene_version" : "8.10.1",
   "minimum_wire_compatibility_version" : "6.8.0",
   "minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Task 4: Ingest data

Run the following commands from within your VM instance.

  1. Download the sample data set.

    curl -O https://raw.githubusercontent.com/oracle-livelabs/oci/main/oci-opensearch/files/OCI_services.json
    
  2. Create mapping (optional). If you don’t run this command or any variation of it, a default mapping will be automatically created.

    curl -XPUT "https://mycluster.opensearch.us.example.com:9200/oci" -H 'Content-Type: application/json' -d'
    {
      "mappings": {
        "properties": {
        "id": {"type": "integer"},
        "category": {"type": "keyword"},
       "text": {"type": "text"},
       "title": {"type": "text"},
       "url": {"type": "text"}
        }
      }
    }
    '
    
  3. Push the data set.

    curl -H 'Content-Type: application/x-ndjson' -XPOST "https://<your_opensearch_private_IP>:9200/oci/_bulk?pretty" --data-binary @OCI_services.json
    
  4. Check your indices.

    curl "https://mycluster.opensearch.us.example.com:9200/_cat/indices"
    # OpenSearch API endpoint example
    
    curl -X GET "https://<your_opensearch_private_IP>:9200/_cat/indices" --insecure
    # OpenSearch private IP example
    

Task 5: Query the OCI Search Service – Sample search query

Use one of the following options:

Option 1 - From the VM instance shell

curl -X GET "https://mycluster.opensearch.us.example.com:9200/oci/_search?q=title:Kubernetes&pretty"
# OpenSearch API endpoint example
curl -X GET "https://<your_opensearch_private_IP>:9200/oci/_search?q=title:Kubernetes&pretty"
# OpenSearch private IP example

Option 2 - From your local terminal, after port forwarding

curl -X GET "https://localhost:9200/oci/_search?q=title:Kubernetes&pretty" --insecure

Option 3 - From your local browser, after port forwarding

https://localhost:9200/oci/_search?q=title:Kubernetes&pretty

For more information about query syntax, see the OpenSearch or Elasticsearch tutorials.

Task 6: Connect to OpenSearch Dashboards

  1. From your local machine, through port forwarding.

    Note: Ignore this step if you’ve executed it in the previous steps and the connection is still open.

    ssh -C -v -t -L 127.0.0.1:5601:<your_opensearch_dashboards_private_IP>:5601 -L 127.0.0.1:9200:<your_opensearch_private_IP>:9200 opc@<your_instance_public_ip> -i <path_to_your_private_key>
    
  2. Access https://localhost:5601 in your browser.

    Note: Currently, depending on the browser, a warning message similar to “Your connection is not private” is displayed. Choose the option which allows you to proceed anyway. The following screen is displayed:

    OpenSearch Dashboards landing page

Task 7: Search and visualize data in OpenSearch Dashboards

  1. With the port forwarding connection in place, access https://localhost:5601 in your browser.

  2. Open the OpenSearch Dashboards navigation menu.

  3. Click Management, then click Stack Management, and then click Index Patterns. Create an index pattern, with name = oci.

    OpenSearch Dashboards - Create index pattern

  4. Open the OpenSearch Dashboards menu, and then click Discover to use the OpenSearch Dashboards UI to search your data.

    OpenSearch Dashboards - Discover

  5. Open the OpenSearch Dashboards menu, click Dashboards and follow these steps to create a sample pie chart.

    1. Click Create new, then click New Visualization, and then click Pie.

    OpenSearch Dashboards - New Visualization

    1. Choose oci as source.

    2. In Buckets, click Add, and then click Split slices. Provide the parameters as shown in the following image and click Update.

    OpenSearch Dashboards - Sample pie chart

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.