Exploring PeopleSoft Log Analyzer Architecture and Data Flow

Log data is generated independently within each PeopleSoft domain, including Application Servers, Web Servers, and Process Scheduler domains. These logs are initially distributed across multiple servers and stored as unstructured text files.

The log processing lifecycle in PeopleSoft Log Analyzer follows a structured sequence of steps that transform distributed, unstructured log data into centralized, searchable, and visual insights. The log processing lifecycle is carried out through the following process:

  1. Collecting Logs Using Fluent Bit: Fluent Bit reads log files from PeopleSoft domains (Application Server, Web Server, and Process Scheduler) and forwards the log data in near real time.
  2. Forwarding Logs to Data Prepper: Fluent Bit sends the collected log data to Data Prepper over HTTP or HTTPS for further processing.
  3. Processing and Enriching Logs Using Data Prepper: Data Prepper parses the incoming logs, extracts structured fields, and enriches them with metadata such as domain, log type, and environment.
  4. Indexing and Storing Logs in OpenSearch: The processed log data is sent to OpenSearch, where it is indexed and stored for efficient search and retrieval.
  5. Visualizing Log Data Using OpenSearch Dashboards: OpenSearch Dashboards retrieves indexed data and presents it through dashboards for monitoring, analysis, and troubleshooting.

The PeopleSoft Log Analyzer supports two deployment architectures for collecting, processing, and visualizing log data. In both architectures, log data generated across PeopleSoft domains is collected by Fluent Bit, processed, and enriched by Data Prepper, indexed in OpenSearch, and presented through OpenSearch Dashboards. The primary distinction between these architectures lies in the placement of Fluent Bit, which can be deployed either as a centralized instance or within each domain, depending on the environment configuration and log accessibility. You can choose either deployment model based on your environment configuration and deployment requirements.

Architecture 1: Centralized Fluent Bit Deployment

In this architecture, Fluent Bit is deployed as part of the Log Analyzer ingestion engine rather than within individual PeopleSoft domains. Each PeopleSoft domain (Application Server, Web Server, and Process Scheduler) generates log files independently. These logs are read and collected by a centralized Fluent Bit instance. Fluent Bit gathers log data from all domains and forwards it to Data Prepper over HTTP or HTTPS. Data Prepper receives the incoming log data, parses the logs using predefined patterns, and enriches them with metadata such as domain name, log type, and environment. The processed data is then forwarded to OpenSearch, where it is indexed and stored in structured indices. OpenSearch Dashboards retrieves the indexed data and presents it through various dashboards, such as PIA Access, Application Server, and Process Scheduler dashboards, enabling centralized monitoring and analysis.

This architecture simplifies deployment and management by using a single Fluent Bit instance and is suitable when log files are accessible from a shared location.


Centralized Fluent Bit Deployment

Architecture 2: Distributed Fluent Bit Deployment

In this architecture, Fluent Bit is deployed within each PeopleSoft domain. Each domain runs its own Fluent Bit instance, which reads local log files directly from the domain’s file system. Fluent Bit collects logs in real time and forwards them to a centralized Data Prepper instance. Data Prepper aggregates log data from all domains, performs parsing and enrichment, and converts unstructured logs into structured data. The enriched data is then sent to OpenSearch for indexing and storage. OpenSearch Dashboards accesses the indexed data and provides visualizations through prebuilt and custom dashboards, enabling users to analyze logs across all domains in a single interface.

This architecture provides greater flexibility and scalability, as log collection is distributed across domains. It is particularly useful when log files are stored locally and are not accessible through a shared location.


Distributed Fluent Bit Deployment

The data flows through the following stages in Log Analyzer:

  1. Log Ingestion: Fluent Bit collects log data from distributed PeopleSoft servers, including Application Server, Web Server, and Process Scheduler domains. It enriches the data by adding metadata such as the source file name and domain name, enabling improved identification and analysis of log entries. Fluent Bit is a lightweight, high-performance log processor and forwarder designed for real-time log collection and routing. It supports multiple input sources, including file-based logs, network streams, and container logs, and can forward data to downstream systems such as Data Prepper. Optimized for high-throughput environments like PeopleSoft, Fluent Bit efficiently handles large volumes of log data with minimal resource consumption. It applies filters to transform and enrich log data before forwarding it. Fluent Bit operates as the log collection layer and can be deployed either within each PeopleSoft domain or as a centralized instance. It continuously reads log files from the file system and forwards the processed log data to Data Prepper over HTTP or HTTPS.
  2. Log Parsing and Enrichment: OpenSearch Data Prepper is a server-side data collector designed to filter, enrich, transform, normalize, and aggregate data for subsequent analysis and visualization. It is the preferred data ingestion tool for OpenSearch, particularly when processing large and complex datasets. Data Prepper integrates seamlessly with Fluent Bit and other data collectors, facilitating structured log formats and enriched data to enhance indexing and search performance. It supports various plugins for parsing, transforming, and enriching log data through customizable pipelines, offering flexibility for complex log processing workflows. The primary use case for Data Prepper is log and trace analytics, for which it provides specific configurations. Additionally, it offers automatic throttling functionality to regulate log data traffic. The Grok processor within Data Prepper can structure and extract important fields from logs, enhancing their query capabilities. Data Prepper serves as the ingestion and processing layer. It receives incoming log data from Fluent Bit through an HTTP source, parses the logs using predefined patterns, and enriches them with additional metadata such as domain, log type, and environment. This step converts unstructured log entries into structured data suitable for indexing.
  3. Indexing and Storage: The processed logs are then sent from Data Prepper to OpenSearch, where they are indexed and stored. OpenSearch organizes the data into indices, enabling efficient querying, filtering, and correlation across logs from different domains and components. OpenSearch Dashboards provides the visualization layer. It retrieves indexed data from OpenSearch and presents it through dashboards such as PIA Access, Application Server, and Process Scheduler dashboards. You can use these dashboards to analyze trends, correlate events, and investigate issues from a centralized interface. The flow transforms distributed, unstructured log files into centralized, structured, and searchable data that can be visualized for monitoring and troubleshooting purposes.