Understanding the Index Build Process

In OpenSearch, an index is a logical namespace that maps to one or more primary shards and can have zero or more replica shards.

Before end users can submit search requests against the Search Framework deployed objects, the search indexes must first be built on the search engine. Prior to the index being built, a deployed search definition is an empty shell, containing no searchable data. A search index needs to be built for search definitions.

An Application Engine program, PTSF_GENFEED, builds the search index data and pushes it to Integration Broker, which makes it available to the search engine.

When an attachment is specified in a search definition, PeopleSoft Search Framework transfers the attachment data directly to the search engine using cURL and libcurl libraries and does not use the Integration Broker.

Creating a search index with the Search Framework involves the following technologies:

• Search Framework

• Application Engine

• Query/Connected Query Manager

• Process Scheduler

• Integration Broker

• cURL and libcurl Libraries

Once you invoke a search index build from the Build Search Index page, the system automatically completes these general steps:

1. The Schedule Search Index page initiates the PTSF_GENFEED Application Engine program.

2. The Pre Processing Application Engine program defined for the search definition runs.

3. PTSF_GENFEED Application Engine program runs the query (PeopleSoft Query or Connected Query) associated with the search definition.

   Multiple PTSF_GENFEED Application Engine programs will run for a Connected Query in cases where users specify the number of partitions. See Partitioning Application Data in the Search Index Build Process for more information.

4. The query output becomes a data source for PeopleSoft Search Framework.

   If a query field is masked, Search Framework treats the masked field in two ways:

   • If a query field is masked, but bypassed for Search Administrator, then all data is indexed.

   • If a query field is masked, but not bypassed, then Search Framework removes the field from indexing.

   Note:

   If you want search query masking to take effect for data that has already been indexed, you should do a full indexing.

   See Query: Using Query Administration for more information on bypassing a masked field.

5. The PeopleSoft Search Framework converts the query output to the JSON format and pushes the data to the search engine and the Delete query (for incremental indexing) defined for the search definition runs.

   During this step, the following steps are also executed:

   • If it is full indexing, a delete request is sent to the search engine to clear any indexed data that is present.

   • If Full Direct Transfer is not enabled (segments of documents without attachments), on clearing indexed data, the data to be indexed is pushed to an Ordered IB Queue (PTSF_ES_SEND_Q) partitioned with segments. IB Queue processes the data asynchronously based on the partition.

   • If Full Direct Transfer is enabled, on clearing indexed data, the data is transferred to the search engine directly using cURL and libcurl libraries.

   • The Delete Query is invoked when incremental indexing is run after a full index is built. Stale data is removed from the index as selected by Delete Query defined on the search definition.

   Note:

   Each segment pushes the data in JSON format to the search engine synchronously (that is, waits for response from the search engine) and the response is read and acknowledged in PeopleSoft through exception handling.

6. The Post Processing Application Engine program defined for the search definition runs.