Search and visualize data using OCI Search Service with OpenSearch
Introduction
OCI Search Service with OpenSearch is an insight engine offered as an Oracle-managed service. With OCI’s capabilities for OpenSearch and OpenSearch Dashboards, you can store, search, and analyze large volumes of data quickly and see results in near real-time. Oracle automates, without any downtime, activities that include patching, updating, upgrading, backing-up, and resizing of the service.
Objectives
- Understand the prerequisites for the creation of a search cluster
- Create an OpenSearch cluster, connect to the cluster and ingest data
- Run sample search commands
- Visualize data in OpenSearch Dashboards
Prerequisites
-
Create the required service policies in the Oracle Cloud Console, tailoring them to your needs. For example: change
any-user
to the desired group, and provide the path to the compartment if required.Working policies:
Allow group <your_group> to manage opensearch-family in compartment opensearch Allow service opensearch to manage vcns in compartment opensearch Allow service opensearch to manage vnics in compartment opensearch Allow service opensearch to use subnets in compartment opensearch Allow service opensearch to use network-security-groups in compartment opensearch
-
Create a VCN with a public subnet and a private subnet.
- Simplified process:
- Open the Oracle Cloud Console navigation menu.
- Click Networking. and then click Virtual Cloud Networks.
- Click Start VCN Wizard, and then click Create VCN with Internet Connectivity.
- Custom process:
- Open the Oracle Cloud Console navigation menu.
- Click Networking. and then click Virtual Cloud Networks.
- Click Create VCN and provide your own desired details.
- Simplified process:
-
Create a VM Instance in the public subnet of the VCN.
- Open the Oracle Cloud Console navigation menu.
- Click Compute, and then click Instances.
- Click Create instance.
- Enter the Name, and select the compartment.
- In the Image and shape area, use the default values (Oracle Linux 8, VM.Standard.E4.Flex).
- In the Networking area, select the public subnet and assign a public IP.
- In the Add SSH keys area, decide whether you want to use an existing SSH key, or generate a new SSH key. If you choose to generate a new SSH key, remember to download the key.
Note: For the exercise contained in this tutorial, a Mac and an Oracle Linux instance were used. Windows command line instructions may differ.
Task 1: Create an OCI Search Service cluster
-
Open the Oracle Cloud Console navigation menu. Click Databases, OCI Search Service, and then click Clusters. Then, click Create cluster. Choose the cluster name and compartment where you want to create the cluster.
-
Click Next.
-
Choose the cluster sizing, and then click Next.
-
Select the VCN you created and then select the private subnet.
-
Click Next. After the cluster creation, in the OCI Search Service cluster details page, note the API endpoints and the IP addresses which you can alternatively use.
Task 2: Create security rules in the VCN Security List
In the VCN, create a Security List with the following security rules. Alternatively, they can be added to the VCN Default Security List.
-
Open the VCN details page, click Security Lists, select the Security List, and then click Add Ingress Rules.
Note: Add a rule for port 9200 (OpenSearch), and a rule for 5601 (OpenSearch Dashboards).
Task 3: Test the connection to OCI Search Service – OpenSearch endpoint
Use one of the following options:
Option 1: From inside the created VM instance
-
Connect to the instance via SSH:
ssh -i ~/.ssh/id_rsa_opensearch.key opc@<your_VM_instance_public_IP>
-
Run one of the following commands:
curl https://mycluster.opensearch.us.example.com:9200 # OpenSearch API endpoint example
curl https://<your_opensearch_private_IP>:9200 --insecure # OpenSearch private IP example
Option 2: From your local machine, through port forwarding
-
Run the following port forwarding SSH command in the Terminal. Do not close the Terminal to ensure that the connection remains active.
ssh -C -v -t -L 127.0.0.1:5601:<your_opensearch_dashboards_private_IP>:5601 -L 127.0.0.1:9200:<your_opensearch_private_IP>:9200 opc@<your_VM_instance_public_IP> -i <path_to_your_private_key>
-
Open a new Terminal window and run the following command:
curl https://localhost:9200 --insecure
If the steps are performed correctly you should see a response as follows, regardless of the option used:
{
"name" : "opensearch-master-0",
"cluster_name" : "opensearch",
"cluster_uuid" : "M6gclrE3QLGEBlkdme8JkQ",
"version" : {
"distribution" : "opensearch",
"number" : "1.2.4-SNAPSHOT",
"build_type" : "tar",
"build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
"build_date" : "2022-02-08T16:44:39.596468Z",
"build_snapshot" : true,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
Task 4: Ingest data
Run the following commands from within your VM instance.
-
Download the sample data set.
curl -O https://raw.githubusercontent.com/oracle-livelabs/oci/main/oci-opensearch/files/OCI_services.json
-
Create mapping (optional). If you don’t run this command or any variation of it, a default mapping will be automatically created.
curl -XPUT "https://mycluster.opensearch.us.example.com:9200/oci" -H 'Content-Type: application/json' -d' { "mappings": { "properties": { "id": {"type": "integer"}, "category": {"type": "keyword"}, "text": {"type": "text"}, "title": {"type": "text"}, "url": {"type": "text"} } } } '
-
Push the data set.
curl -H 'Content-Type: application/x-ndjson' -XPOST "https://<your_opensearch_private_IP>:9200/oci/_bulk?pretty" --data-binary @OCI_services.json
-
Check your indices.
curl "https://mycluster.opensearch.us.example.com:9200/_cat/indices" # OpenSearch API endpoint example
curl -X GET "https://<your_opensearch_private_IP>:9200/_cat/indices" --insecure # OpenSearch private IP example
Task 5: Query the OCI Search Service – Sample search query
Use one of the following options:
Option 1 - From the VM instance shell
curl -X GET "https://mycluster.opensearch.us.example.com:9200/oci/_search?q=title:Kubernetes&pretty"
# OpenSearch API endpoint example
curl -X GET "https://<your_opensearch_private_IP>:9200/oci/_search?q=title:Kubernetes&pretty"
# OpenSearch private IP example
Option 2 - From your local terminal, after port forwarding
curl -X GET "https://localhost:9200/oci/_search?q=title:Kubernetes&pretty" --insecure
Option 3 - From your local browser, after port forwarding
https://localhost:9200/oci/_search?q=title:Kubernetes&pretty
For more information about query syntax, see the OpenSearch or Elasticsearch tutorials.
Task 6: Connect to OpenSearch Dashboards
-
From your local machine, through port forwarding.
Note: Ignore this step if you’ve executed it in the previous steps and the connection is still open.
ssh -C -v -t -L 127.0.0.1:5601:<your_opensearch_dashboards_private_IP>:5601 -L 127.0.0.1:9200:<your_opensearch_private_IP>:9200 opc@<your_instance_public_ip> -i <path_to_your_private_key>
-
Access https://localhost:5601 in your browser.
Note: Currently, depending on the browser, a warning message similar to “Your connection is not private” is displayed. Choose the option which allows you to proceed anyway. The following screen is displayed:
Task 7: Search and visualize data in OpenSearch Dashboards
-
With the port forwarding connection in place, access https://localhost:5601 in your browser.
-
Open the OpenSearch Dashboards navigation menu.
-
Click Management, then click Stack Management, and then click Index Patterns. Create an index pattern, with name =
oci
. -
Open the OpenSearch Dashboards menu, and then click Discover to use the OpenSearch Dashboards UI to search your data.
-
Open the OpenSearch Dashboards menu, click Dashboards and follow these steps to create a sample pie chart.
- Click Create new, then click New Visualization, and then click Pie.
-
Choose
oci
as source. -
In Buckets, click Add, and then click Split slices. Provide the parameters as shown in the following image and click Update.
Related Links
- OCI Search Service with OpenSearch
- Aggregate logs using OCI Search Service with OpenSearch
- Use OCI Search Service with OpenSearch to build in-application search
Acknowledgments
- Author - Nuno Goncalves (Senior Principal Product Manager)
- Contributor - Jordan Oliver (Principal Product Manager)
More Learning Resources
Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.
For product documentation, visit Oracle Help Center.
Search and visualize data using OCI Search Service with OpenSearch
F56572-04
April 2023
Copyright © 2023, Oracle and/or its affiliates.