Search and visualize data using OCI Search Service with OpenSearch

Introduction

OCI Search Service with OpenSearch is an insight engine offered as an Oracle-managed service. With OCI’s capabilities for OpenSearch and OpenSearch Dashboards, you can store, search, and analyze large volumes of data quickly and see results in near real-time. Oracle automates, without any downtime, activities that include patching, updating, upgrading, backing-up, and resizing of the service.

Objectives

Understand the prerequisites for the creation of a search cluster
Create an OpenSearch cluster, connect to the cluster and ingest data
Run sample search commands
Visualize data in OpenSearch Dashboards

Prerequisites

Create the required service policies in the Oracle Cloud Console, tailoring them to your needs. For example: change any-user to the desired group, and provide the path to the compartment if required.

Working policies:

Allow group <your_group> to manage opensearch-family in compartment opensearch
Allow service opensearch to manage vcns in compartment opensearch
Allow service opensearch to manage vnics in compartment opensearch
Allow service opensearch to use subnets in compartment opensearch
Allow service opensearch to use network-security-groups in compartment opensearch

Create a VCN with a public subnet and a private subnet.
- Simplified process:
  1. Open the Oracle Cloud Console navigation menu.
  2. Click Networking. and then click Virtual Cloud Networks.
  3. Click Start VCN Wizard, and then click Create VCN with Internet Connectivity.
- Custom process:
  1. Open the Oracle Cloud Console navigation menu.
  2. Click Networking. and then click Virtual Cloud Networks.
  3. Click Create VCN and provide your own desired details.
Create a VM Instance in the public subnet of the VCN.
1. Open the Oracle Cloud Console navigation menu.
2. Click Compute, and then click Instances.
3. Click Create instance.
4. Enter the Name, and select the compartment.
5. In the Image and shape area, use the default values (Oracle Linux 8, VM.Standard.E4.Flex).
6. In the Networking area, select the public subnet and assign a public IP.
7. In the Add SSH keys area, decide whether you want to use an existing SSH key, or generate a new SSH key. If you choose to generate a new SSH key, remember to download the key.

Note: For the exercise contained in this tutorial, a Mac and an Oracle Linux instance were used. Windows command line instructions may differ.

Task 1: Create an OCI Search Service cluster

Open the Oracle Cloud Console navigation menu. Click Databases, OCI Search Service, and then click Clusters. Then, click Create cluster. Choose the cluster name and compartment where you want to create the cluster.
Click Next.
Choose the cluster sizing, and then click Next.
Select the VCN you created and then select the private subnet.
Click Next. After the cluster creation, in the OCI Search Service cluster details page, note the API endpoints and the IP addresses which you can alternatively use.

Task 2: Create security rules in the VCN Security List

In the VCN, create a Security List with the following security rules. Alternatively, they can be added to the VCN Default Security List.

Open the VCN details page, click Security Lists, select the Security List, and then click Add Ingress Rules.

Note: Add a rule for port 9200 (OpenSearch), and a rule for 5601 (OpenSearch Dashboards).

Security rule for Search API

Security rule for Dashboards

Default Security List for opensearch-vcn - table view

Task 3: Test the connection to OCI Search Service – OpenSearch endpoint

Use one of the following options:

Option 1: From inside the created VM instance

Connect to the instance via SSH:

ssh -i ~/.ssh/id_rsa_opensearch.key opc@<your_VM_instance_public_IP>

Run one of the following commands:

curl https://mycluster.opensearch.us.example.com:9200
# OpenSearch API endpoint example

curl https://<your_opensearch_private_IP>:9200 --insecure
# OpenSearch private IP example

Option 2: From your local machine, through port forwarding

Run the following port forwarding SSH command in the Terminal. Do not close the Terminal to ensure that the connection remains active.

ssh -C -v -t -L 127.0.0.1:5601:<your_opensearch_dashboards_private_IP>:5601 -L 127.0.0.1:9200:<your_opensearch_private_IP>:9200 opc@<your_VM_instance_public_IP> -i <path_to_your_private_key>

Open a new Terminal window and run the following command:
```
curl https://localhost:9200 --insecure
```

If the steps are performed correctly you should see a response as follows, regardless of the option used:

{
"name" : "opensearch-master-0",
"cluster_name" : "opensearch",
"cluster_uuid" : "M6gclrE3QLGEBlkdme8JkQ",
"version" : {
   "distribution" : "opensearch",
   "number" : "1.2.4-SNAPSHOT",
   "build_type" : "tar",
   "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
   "build_date" : "2022-02-08T16:44:39.596468Z",
   "build_snapshot" : true,
   "lucene_version" : "8.10.1",
   "minimum_wire_compatibility_version" : "6.8.0",
   "minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Task 4: Ingest data

Run the following commands from within your VM instance.

Download the sample data set.

curl -O https://raw.githubusercontent.com/oracle-livelabs/oci/main/oci-opensearch/files/OCI_services.json

Create mapping (optional). If you don’t run this command or any variation of it, a default mapping will be automatically created.

curl -XPUT "https://mycluster.opensearch.us.example.com:9200/oci" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
    "id": {"type": "integer"},
    "category": {"type": "keyword"},
   "text": {"type": "text"},
   "title": {"type": "text"},
   "url": {"type": "text"}
    }
  }
}
'

Push the data set.

curl -H 'Content-Type: application/x-ndjson' -XPOST "https://<your_opensearch_private_IP>:9200/oci/_bulk?pretty" --data-binary @OCI_services.json

Check your indices.

curl "https://mycluster.opensearch.us.example.com:9200/_cat/indices"
# OpenSearch API endpoint example

curl -X GET "https://<your_opensearch_private_IP>:9200/_cat/indices" --insecure
# OpenSearch private IP example

Task 5: Query the OCI Search Service – Sample search query

Use one of the following options:

Option 1 - From the VM instance shell

curl -X GET "https://mycluster.opensearch.us.example.com:9200/oci/_search?q=title:Kubernetes&pretty"
# OpenSearch API endpoint example

curl -X GET "https://<your_opensearch_private_IP>:9200/oci/_search?q=title:Kubernetes&pretty"
# OpenSearch private IP example

Option 2 - From your local terminal, after port forwarding

curl -X GET "https://localhost:9200/oci/_search?q=title:Kubernetes&pretty" --insecure

Option 3 - From your local browser, after port forwarding

https://localhost:9200/oci/_search?q=title:Kubernetes&pretty

For more information about query syntax, see the OpenSearch or Elasticsearch tutorials.

Task 6: Connect to OpenSearch Dashboards

From your local machine, through port forwarding.

Note: Ignore this step if you’ve executed it in the previous steps and the connection is still open.

ssh -C -v -t -L 127.0.0.1:5601:<your_opensearch_dashboards_private_IP>:5601 -L 127.0.0.1:9200:<your_opensearch_private_IP>:9200 opc@<your_instance_public_ip> -i <path_to_your_private_key>

Access https://localhost:5601 in your browser.

Note: Currently, depending on the browser, a warning message similar to “Your connection is not private” is displayed. Choose the option which allows you to proceed anyway. The following screen is displayed:

Task 7: Search and visualize data in OpenSearch Dashboards

With the port forwarding connection in place, access https://localhost:5601 in your browser.
Open the OpenSearch Dashboards navigation menu.
Click Management, then click Stack Management, and then click Index Patterns. Create an index pattern, with name = oci.
Open the OpenSearch Dashboards menu, and then click Discover to use the OpenSearch Dashboards UI to search your data.
Open the OpenSearch Dashboards menu, click Dashboards and follow these steps to create a sample pie chart.
1. Click Create new, then click New Visualization, and then click Pie.
1. Choose oci as source.
2. In Buckets, click Add, and then click Split slices. Provide the parameters as shown in the following image and click Update.

Acknowledgments

Author - Nuno Goncalves (Senior Principal Product Manager)
Contributor - Jordan Oliver (Principal Product Manager)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information

Search and visualize data using OCI Search Service with OpenSearch

F56572-04

April 2023

Search and visualize data using OCI Search Service with OpenSearch

Introduction

Objectives

Prerequisites

Task 1: Create an OCI Search Service cluster

Task 2: Create security rules in the VCN Security List

Task 3: Test the connection to OCI Search Service – OpenSearch endpoint

Option 1: From inside the created VM instance

Option 2: From your local machine, through port forwarding

Task 4: Ingest data

Task 5: Query the OCI Search Service – Sample search query

Option 1 - From the VM instance shell

Option 2 - From your local terminal, after port forwarding

Option 3 - From your local browser, after port forwarding

Task 6: Connect to OpenSearch Dashboards

Task 7: Search and visualize data in OpenSearch Dashboards

Related Links

Acknowledgments

More Learning Resources