Use OCI Search Service with OpenSearch to Build In-application Search
Oracle Cloud Infrastructure Search Service with OpenSearch allows you to build in-application search that can return results in milliseconds even with large datasets.
- A search engine daemon: OpenSearch.
- A visualization and user interface: OpenSearch Dashboards.
Architecture
Storing and indexing application data on OCI Search Service with OpenSearch to facilitate speedy in-application search relies on architectural topology and complements described here.
The following diagram illustrates this reference architecture.
Description of the illustration appsearch-oci-arch.png
The architecture has the following components:
- Tenancy
A tenancy is a secure and isolated partition that Oracle sets up within Oracle Cloud when you sign up for OCI. You can create, organize, and administer your resources in Oracle Cloud within your tenancy.
- Policies
An OCI Identity and Access Management policy specifies who can access which resources, and how. Access is granted at the group and compartment level, which means you can write a policy that gives a group a specific type of access within a specific compartment, or to the tenancy.
- Compartments
Compartments are cross-region logical partitions within an OCI tenancy. Use compartments to organize your resources in Oracle Cloud, control access to the resources, and set usage quotas. To control access to the resources in a given compartment, you define policies that specify who can access the resources and what actions they can perform.
- Virtual cloud network (VCN)
One of your first steps in OCI is to set up a virtual cloud network (VCN) for your cloud resources. A VCN is a software-defined network that you set up in an OCI region. VCNs can be segmented into subnets, which can be specific to a region or to an availability domain. Both region-specific and availability domain-specific subnets can coexist in the same VCN. A subnet can be public or private.
- Availability domains
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.
- Fault domains
A fault domain is a grouping of hardware and infrastructure within an availability domain. Each availability domain has three fault domains with independent power and hardware. When you distribute resources across multiple fault domains, your applications can tolerate physical server failure, system maintenance, and power failures inside a fault domain.
- Bastion host
The bastion host is a compute instance that serves as a secure, controlled entry point to the topology from outside the cloud. It enables you to access sensitive resources placed in private networks that can't be accessed directly from outside the cloud. The topology has a single, known entry point that you can monitor and audit regularly. So, you can avoid exposing the more sensitive components of the topology without compromising access to them.
- Load balancer
The load balancer balances index operations to the data nodes and Kibana access to the master nodes. It uses two listeners, one for Kibana and one for index data access, using master node back ends and data node back ends. The load balancer is in a public subnet with a public IP address.
- Compute instances
OCI Compute lets you provision and manage compute hosts. You can launch compute instances with shapes that meet your resource requirements (CPU, memory, network bandwidth, and storage). After creating a compute instance, you can access it securely, restart it, attach and detach volumes, and terminate it when you don't need it.
- Internet gateway
The internet gateway allows traffic between the public subnets in a VCN and the public internet.
- Dynamic routing gateway (DRG)
The DRG is a virtual router that provides a path for private network traffic between on-premises networks and VCNs and can also be used to route traffic between VCNs in the same region or across regions.
- Vault
OCI Vault enables you to centrally manage the encryption keys that protect your data and the secret credentials that you use to secure access to your resources in the cloud.
- VPN
Site-to-Site VPN provides site-to-site IPSec VPN connectivity between your on-premises network and VCNs in OCI. The IPSec protocol suite encrypts IP traffic before the packets are transferred from the source to the destination and decrypts the traffic when it arrives.
- Application
The application in the architecture is simplified and illustrates a monolithic application. In the case of a multi-tier application the backend would be connect to OCI Search Service with OpenSearch. For distributed applications, it's any service that stores a significant amount of data and where other services or users need to make searches.
Recommendations
- VCN
When you create a VCN, determine the number of CIDR blocks required and the size of each block based on the number of resources that you plan to attach to subnets in the VCN. Use CIDR blocks that are within the standard private IP address space.
Select CIDR blocks that don't overlap with any other network (in Oracle Cloud Infrastructure, your on-premises data center, or another cloud provider) to which you intend to set up private connections.
When you design the subnets, consider your traffic flow and security requirements. Attach all the resources within a specific tier or role to the same subnet, which can serve as a security boundary.
- Security
Use policies to restrict who can access the OCI resources that your company has and how they can access them. Specific policies are required for a successful OCI Search Service with OpenSearch cluster creation. Use Vault for additional protection of your keys, certificates and secrets.
The Networking service offers two virtual firewall features that use security rules to control traffic at the packet level: security lists and network security groups (NSG). An NSG consists of a set of ingress and egress security rules that apply only to a set of VNICs of your choice in a single VCN. For example, you can choose all the compute instances that act as web servers in the web tier of a multi-tier application in your VCN.
NSG security rules function the same as security list rules. However, for an NSG security rule's source or destination, you can specify an NSG instead of a CIDR block. So, you can easily write security rules to control traffic between two NSGs in the same VCN or traffic within a single NSG.
- Compute
Choose shapes with the appropriate OCPUs and memory combination, and provision local NVMe and/or block storage according to need, for each instance.
Considerations
When using OCI Search Service with OpenSearch to build in-application search functionality in cloud applications, consider these options.
- If you can't find OpenSearch plugins for your ORM, search for ElasticSearch 7.10.2 compatible plugins, alternatively use the REST API.
- Build search templates to combine search queries from users with pre-defined search parameters.
- To connect to OCI Search Service with OpenSearch from a local machine, set up a bastion host in a public subnet and setup SSH port forwarding to access the endpoint. Alternatively access the endpoint through a VPN. The application instances present in the private subnet can also be accessed via the bastion host.