Monitoring and Evaluation

AI Agent Studio separates monitoring, evaluation, and value estimation.

The scheduled process Aggregate AI Agent Usage and Metrics aggregates the data displayed in the Monitoring and Evaluation tab; it is a scheduled process, not a report or the monitoring interface. The Monitoring subtab summarizes agent runs for the selected period. Selecting an agent shows sessions, turns, completion status, and token use; selecting a session opens a detailed trace with the step-by-step timeline, tools invoked, duration, and per-step metrics. Use aggregated monitoring rather than real-time monitoring for this view because Oracle's monitoring guide notes that the scheduled process currently consolidates data with a one-day backlog.

The Business Value Dashboard is a separate capability within Monitoring and Evaluation. It estimates time and cost savings by multiplying eligible production usage by author-provided per-interaction estimates. Debug and test executions are excluded, and authorized users can revise estimates as operational evidence improves. Treat the results as transparent, usage-based estimates for calibration and investment discussions—not as independently measured ROI. See the Business Value Dashboard guide.

Monitoring Questions

At each monitoring cycle, the CoE should be able to answer the following questions from the monitoring data:

  • Did sessions complete successfully, and what was the completion rate?
  • Which tools were invoked, and in what sequence?
  • Where did latency occur, and does it exceed the defined service level?
  • Which errors and failure patterns repeat across sessions?
  • Which prompts, tools, or documents produce incorrect outputs or require frequent human override?

For detailed monitoring configuration, including the Monitoring Dashboard metrics and execution timeline, see Monitoring Dashboard.

For full monitoring and evaluation procedures, see Monitoring and Evaluation.

Evaluation Metrics

Oracle provides structured metrics within Evaluation. Establish baselines during testing before go-live, and use the following product terminology. CoEs should supplement these measures with human-review findings and business outcome measures.

  • Median Correctness: Accuracy on a 0–1 scale, with optional human validation.
  • P50 and P99 latency: Median responsiveness and tail latency across evaluation runs.
  • Token usage: Total, input, and output token consumption.
  • RAG metrics: Groundedness, answer relevance, and context relevance when document tool evaluation is enabled.
  • Run comparison and human review: Side-by-side evaluation runs and reviewer findings used to identify regressions or improvements.

Evaluation Sets

An evaluation set is a defined collection of representative inputs and expected outputs against which the agent is tested before each production deployment. Evaluation sets must be created before go-live and rerun after any change to agent configuration — prompts, tools, documents, or access rights. For evaluation set creation and management, see Creating Evaluation Sets.

Human Review Requirements

Human review gates are required for all AI actions with material financial impact, audit implications or regulatory consequence. This requirement applies regardless of agent accuracy metrics. For example: under the EU AI Act, AI systems used in finance for consequential decisions must maintain human oversight; this is a compliance requirement, not an architectural preference. The review gate design for each action must be documented before implementation begins — not retrofitted after go-live.

The operational question to apply to every action in an EPM AI workflow: if this action produces an incorrect output, what is the financial, regulatory or reputational consequence? If the consequence is material, the action requires human approval before the agent executes. If the consequence is not material, the action can be automated with monitoring.

Building the AI Roadmap

The maturity assessment output identifies the current stage and the lowest-scoring dimension. The roadmap is built by addressing the lowest-scoring dimension first — not the dimension the CoE finds most accessible. Each CoE should adjust these priorities based on its specific dimension scores, sub-function profile and regulatory exposure.

Note:

The maturity assessment should be re-run every six months. Dimension scores that have improved confirm that roadmap actions have had the intended effect. Dimension scores that have not improved — particularly Governance and Change Management — indicate that investment or process attention is required before further technology deployment.