Set Up REST API Log Collection

Oracle Log Analytics already provides Oracle-defined log source for REST API log collection. Check if you can use the available Oracle-defined REST API source or any Oracle-defined parser. If not, use the following steps to create a new log source:

Before you begin, if you must create a new parser that is suitable for your logs, then complete it. See Create a Parser.

1. Open the navigation menu and click Observability & Management. Under Log Analytics, click Administration.

   The administration resources are listed in the left hand navigation pane under Administration. Click Sources.

2. The Sources page opens. Click Create Source.

   This displays the Create Source dialog box.

3. In the Name field, enter the name for the log source.

4. From the Source Type list, select REST API.

5. Click Entity Type and select the type that best identifies your application.

6. Click Parser and select a suitable parser for the type of logs you want to collect.

7. In the Endpoints tab, add endpoints in the order in which they must run. Click Add log endpoint to add either a Log endpoint or an Initialization endpoint. Click Add list endpoint for multiple logs for multiple logs when one endpoint returns a list of items and a child Log endpoint must collect details for each item.

   Use Initialization for setup calls such as login, session token retrieval, or container creation. Initialization endpoint responses can be referenced by later endpoints, but are not collected as log records. Place Initialization endpoints before the endpoints that use their response values. HTTP cookies are passed on to subsequent endpoints.

   • To provide a single endpoint, click Add log endpoint. The Add log endpoint dialog box opens.

     Provide the following information:

     1. Select the Log endpoint type:

        • Select Log when the endpoint response contains log records to collect.
        • Select Initialization when the endpoint retrieves values, such as a session token, for later endpoints. HTTP cookies are passed on to subsequent endpoints.
          Note
          
          Ensure to save the retrieved values such as session token to be able to use while running other endpoints in the workflow.

     2. Enter the Log endpoint name.

     3. Construct the Log URL to collect the logs periodically. See Log Endpoint URL in Types of Endpoint.

     4. Select the API method GET or POST.

        If you selected POST, enter the POST payload and specify the request content type required by the API from application/json, text/plain, text/javascript, text/html, or application/xml. For SOAP APIs, enter the SOAP envelope in the POST payload and use a content type application/xml, as required by the endpoint. To use USERNAME/PASSWORD macros in your POST payload, see USERNAME and PASSWORD Macros.

     5. Optionally, specify the Log proxy server URL.

     6. Optionally, click Show request headers to expand the section, and click Add to provide any request headers in the form of Name-Value pairs.

     7. Optionally, click Show query parameters to expand the section, and click Add to provide any query parameters in the form of Name-Value pairs.

     8. In the Credentials section, select the Log credential type. See Select the Credential Type for REST API Log Collection.

        If the POST payload contains USERNAME or PASSWORD, specify the Credential name for the USERNAME/PASSWORD macros in the Credentials section. The Management Agent replaces these macros with values from the selected credential at collection time.

     9. To validate the configuration information that you entered, click Validate. If there are errors, then fix them.

        Click Save Changes.

   • To provide an endpoint that returns a JSON or XML response with the information used to generate multiple log endpoint requests, click Add list endpoint for multiple logs. The Add list endpoint for multiple logs dialog box opens.

     1. Configure list endpoint tab:

        • Enter the Log list endpoint name.

        • Construct the Log list URL to get the information about the log files. See Log List Endpoint URL in Types of Endpoint. For example:

          https://example.org/fetchlogfiles_data

        • Optionally, specify the Log proxy server URL.

        • Select the API method GET or POST.

          If you selected POST, enter the POST payload and specify the request content type required by the API from application/json, text/plain, text/javascript, text/html, or application/xml. For SOAP APIs, enter the SOAP envelope in the POST payload and use a content type application/xml, as required by the endpoint. To use USERNAME/PASSWORD macros in your POST payload, see USERNAME and PASSWORD Macros.

          The following XML payload example shows the use of session token extracted from the initialization endpoint:

          <sessionToken>{getSessionToken:/Envelope/Body/loginResponse/loginResult}</sessionToken>

        • Optionally, click Show request headers to expand the section, and click Add to provide any request headers in the form of Name-Value pairs.

        • Optionally, click Show query parameters to expand the section, and click Add to provide any query parameters in the form of Name-Value pairs.

        • In the Credentials section, select the Log credential type. See Select the Credential Type for REST API Log Collection.

          If the POST payload contains {USERNAME} or {PASSWORD}, specify the Credential name for the USERNAME/PASSWORD macros in the Credentials section. The Management Agent replaces these macros with values from the selected credential at collection time.

        • Click Next.

     2. Configure log endpoint tab:

        • Provide the Example response of log list endpoint. For JSON responses, use JSONPath variables. For XML or SOAP responses, use XPath variables.

          This is the JSON example of the response you would get for the log list endpoint that you provided in the previous tab. For example:

          { "totalSize": 4, "records": [ {"id": "firstId", "type": "firstType"}, {"id": "secondId", "type": "secondType"} ] }

          From the above example, the JSON path records[*].id can be used in the endpoint URL. For a more details about JSON path variables, see Variables for REST API Log Collection.

          This is an XML example for SOAP:

          <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://example.oracle.com/">
            <soapenv:Body>
              <ns:getItemsResponse>
                <ns:getItemsResult>
                  <ns:string>Widget</ns:string>
                  <ns:string>Gadget</ns:string>
                </ns:getItemsResult>
              </ns:getItemsResponse>
            </soapenv:Body>
          </soapenv:Envelope>

          XPath variable example:

          {getItems_ListEndpoint:/Envelope/Body/getItemsResponse/getItemsResult/string}

        • Enter the Log endpoint name.

        • Construct the Log URL, request headers, form parameters, or POST payload by incorporating variables from the list endpoint response. Use JSONPath for JSON responses and XPath for XML or SOAP responses. See Log Endpoint URL in Types of Endpoint. For example:

          https://example.org/fetchLogs?time={START_TIME}&id={testLogListEP:$.records[*].id}

        • Select the API method GET or POST.

          If you selected POST, enter the POST payload and specify the request content type required by the API from application/json, text/plain, text/javascript, text/html, or application/xml. For SOAP APIs, enter the SOAP envelope in the POST payload and use a content type application/xml, as required by the endpoint.

          This log endpoint payload can reference both earlier Initialization endpoint values and the current item from the list endpoint.

          Example of XML payload for POST call which references the session token and list endpoint:

          <ns:getItemDetail>
            <ns:sessionToken>{getSessionToken:/Envelope/Body/loginResponse/loginResult}</ns:sessionToken>
            <ns:itemName>{getItems_ListEndpoint:/Envelope/Body/getItemsResponse/getItemsResult/string}</ns:itemName>
          </ns:getItemDetail>

        • Optionally, specify the Log proxy server URL.

        • Optionally, click Show request headers to expand the section, and click Add to provide any request headers in the form of Name-Value pairs.

        • Optionally, click Show query parameters to expand the section, and click Add to provide any query parameters in the form of Name-Value pairs.

        • In the Credentials section, select the Log credential type. See Select the Credential Type for REST API Log Collection.

        • Click Next.

     3. Review and add tab: The configuration information provided in the previous tabs is validated. Verify the list of URLs from which the logs will be collected.

        If there are errors, then fix them.

        Verify that Initialization endpoints appear before endpoints that reference them, and that each Log List endpoint has a child Log endpoint that collects the records.

        Click Save.

8. Click Create Source.

Log List Endpoint: When you click Add list endpoint for multiple logs while creating the REST API log source, you can specify the log list endpoint that returns a list of items. The log list endpoint must return a JSON or XML response, with JSONPath for JSON and XPath for XML. JSONPath or XPath can be used to construct a list of log endpoint URLs to collect the logs. Specify the variables required to build the list of log endpoints from the response. Additionally, you can use the macros {START_TIME} and {CURR_TIME} to insert the corresponding time values dynamically in the URL.

Log Endpoint: When you click Add log endpoint while creating the REST API log source, you can add one of the two types of log endpoint. The Log endpoint type can be one of these two- Initialization and Log. The Initialization endpoint retrieves values, such as a session token, for later endpoints. The  Initialization endpoint does not extract log records. Elements from its response can be used in subsequent endpoints. The Log endpoint is used for collecting logs from a specific REST API endpoint at regular intervals. You can use the macros {START_TIME}, {CURR_TIME}, and {TIME_WINDOW} to insert the corresponding time values dynamically in the URL. {OFFSET} macro can be used to support pagination. You can also use JSON path variables from the response of the log list endpoint call to substitute specific properties.

• {START_TIME}: To specify the time from when the logs must be collected
• {OFFSET}: To handle paginated log collection
• {CURR_TIME}: To provide the current time
• {TIME_WINDOW}: To specify a collection interval or duration
• {USERNAME} and {PASSWORD}: For SOAP APIs that require username and password values in the POST payload. The Management Agent replaces the macros with values from the configured credential at collection time.

For more information about using the macros, see START_TIME Macro, CURR_TIME Macro, OFFSET Macro, and TIME_WINDOW Macro.

For more information on using the variables from the log list endpoint response that can be specified in the log endpoint URL, examples of JSONPath and XPath, and using variable filters, see Variables for REST API Log Collection.

Use the START_TIME macro to specify the time from when the logs must be collected.START_TIME macro can be used in an endpoint URL, form parameters, or POST payload.

The following example shows the endpoint URL that collects logs greater than a timestamp specified by the START_TIME macro:

https://example.org/fetchLogs?sortBy=timestamp&sortOrder=ascending&filter=timestamp+gt+{START_TIME:yyyy-MM-dd'T'HH:mm:ss.SSSZ}

Syntax:

{START_TIME<+nX>:<epoch | timestamp format supported by SimpleDateFormat java class>.TZ=<timezone name>}

• n is the time value of the date range. X is expressed in Days (D), Hours (h), Minutes (m), Months (M), Year (Y).

• +nX is the number of days, hours, minutes, months, or years to add to the start time. It's optional. For example, +3D.

• The supported time formats are the same as of java class SimpleDateFormat. The default time format is yyyy-MM-dd'T'HH:mm:ss.SSSZ. For example, 2001-07-04T12:08:56.235-0700.

  Specifying the epoch or time format is optional. If epoch is provided, then the macro is replaced with epoch millisecond value.

  For timestamp formats supported by SimpleDateFormat, see Java Platform Standard Ed. 8: Class SimpleDateFormat.

• TZ is for providing the timezone of the timestamp. It is not applicable if epoch is already provided. The supported format is the same as the 3-letter IDs in the java class TimeZone, for example UTC.

• You can include multiple instances of this macro in the URL.

• Examples:

  1. {START_TIME:yyyy-MM-dd'T'HH:mm:ss.SSS.TZ=UTC}
  2. {START_TIME:epoch}

The value of the START_TIME macro is determined in one of the following ways:

• When the collection starts for first time for an endpoint, START_TIME is the historical date based on agent collection property Historical Data. The default value of this property is 30 days, that is, the value of the timestamp is 30 days earlier than the current timestamp.

• If the Time field is set in the parser, then in the subsequent log collections, the value of START_TIME is the maximum value of the extracted timestamp value in the previous log collection.

• If Time field is not extracted from the log records by setting it in the parser, then current timestamp is the value of START_TIME.

In case there are filters in API, use greater than equal to instead of greater than to ensure that all the log records having the same timestamp are uploaded.

Use the CURR_TIME macro to insert current time in the REST API endpoint URL, form parameters, and POST payload.

The following example shows the endpoint URL that uses the CURR_TIME macro in the query parameter:

https://example.org/fetchLogs?sortBy=timestamp&sortOrder=ascending&time={CURR_TIME:yyyy-MM-dd'T'HH:mm:ss.SSSZ}

• The format for using the macro is the same as START_TIME macro.

• You can include multiple instances of this macro in the URL.

Use the OFFSET macro for endpoints that provide paginated responses or to handle offset behavior in an API response. OFFSET macro can be used in the REST API endpoint URL, form parameters, and POST payload to fetch multiple pages or chunks in a single log collection cycle.

Format: {OFFSET(<start value>, <increment>)}

• OFFSET macro is used to iteratively call and collect data over paginated results of a specific log endpoint to get all the available records. The initial REST API request call begins with the start value and is incremented in each subsequent call by the value of increment. These recursive index-based calls are stopped when there are no more log entries found. The offset is not carried forward to the next collection cycle. It starts from the default or initial value in each collection cycle.

• In the above format, the start value is the initial value for the index, and the default value is 0. It's optional to specify start value.

  Possible values: Positive integer including 0

• In the above format, increment specifies the value to be added to the start value in subsequent calls. The default value is 1. It's optional to specify increment value.

  Possible values: Positive integer only. Exclude 0.

• You can include only one instance of this macro in the URL.

The following examples show different ways of using the OFFSET macro:

• {OFFSET}

  Uses default values start value = 0, increment = 1.

• {OFFSET(5)}

  start value = 5, increment = 1 (default)

• {OFFSET(5,2)}

  start value = 5, increment = 2

The following example shows the endpoint URL to that uses the OFFSET macro:

https://example.org/fetchLogs?startIndex={OFFSET(0,1000)}&count=1000

In the above example, OFFSET(0,1000) indicates that start value is 0 in the first call and then in subsequent calls, incremented by 1000. Therefore, when the OFFSET macro is interpreted, the endpoint URL for multiple calls would be as follows:

First call: https://example.org/fetchLogs?startIndex=0&count=1000

Second call: https://example.org/fetchLogs?startIndex=1000&count=1000

Use the TIME_WINDOW macro to specify the collection interval over which the logs should be collected. It can be used in the REST API endpoint URL to fetch logs over minutes, hours, or days irrespective of what the agent collection interval is. For example, if the time window is 1d (one day) and the agent interval is 10 minutes, then the subsequent log collection takes place only after a day.

The following example sets the log collection interval for 6 hours by specifying TIME_WINDOW as 6h in the endpoint URL:

https://example.org/fetchLogs?timewindow={TIME_WINDOW(6h)}

Format: {TIME_WINDOW(<number><timeunit>)}

In the above format:

• number: Digit number that is greater than zero.

• timeunit: h for hours, d for days, m for minutes. The default value is d (days).

Ensure that your agent collection interval is smaller than the time window provided. You can use the TIME_WINDOW macro only once in the endpoint.

For SOAP APIs that require username and password values in the POST payload, enter {USERNAME} and {PASSWORD} in the SOAP XML envelope. When either macro is present, specify the Credential name for the USERNAME/PASSWORD macros in the endpoint Credentials section. The Management Agent replaces the macros with values from the configured credential at collection time.

This is separate from endpoint-level Basic Auth or token authentication.

Example SOAP XML payload for POST call that has the above macros:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://example.oracle.com/">
  <soapenv:Header/>
  <soapenv:Body>
    <ns:login>
      <ns:username>{USERNAME}</ns:username>
      <ns:password>{PASSWORD}</ns:password>
    </ns:login>
  </soapenv:Body>
</soapenv:Envelope>

Use variables to substitute values from a previous endpoint response into a later endpoint request at run time. Variables can be used in the URL, form parameters, POST payload, or HTTP request header values. For REST responses, use JSONPath expressions. For XML or SOAP responses, use XPath expressions.

If an endpoint request depends on a value from an earlier endpoint and that earlier endpoint fails, then the dependent endpoint is not called and log messages are not collected from that dependent endpoint.

Format of the variable in a later endpoint:

{<endpoint_name>:<JSONPath or XPath expression>}

For JSON responses, use JSONPath expressions such as $.records[*].id. For XML or SOAP responses, use XPath expressions such as /Envelope/Body/loginResponse/loginResult.

• In the above format, endpoint_name is the name of the earlier endpoint whose response provides the value. For SOAP flows, this can be an Initialization endpoint, such as a login endpoint, or a Log List endpoint.

  For JSONPath expressions, the filter (<filter_field_name>=<value>) is optional. Only the attribute matching the filter_field_name is picked up from the JSON response and used in the later endpoint. For example:

  https://www.example.com/log/{foo:$.items[*].links[*].href(rel='self')}

  For a detailed example of the filter, see JSONPath Filter Example.

  For examples of XPATH expressions, see SOAP/XPath Examples.

• JSON and XML response content can be used to fetch variable values. Use JSONPath for JSON responses and XPath for XML or SOAP responses.

• The same variable can be specified multiple times.

• Variables can refer to earlier endpoints that are valid sources for later endpoint values. Use Initialization endpoints for setup values such as session tokens. Use Log List endpoints for values that drive child Log endpoint requests.

• The following JSON path formats are supported: Simple JSONPath, Array JSONPath, and Multiple-array JSONPath.

• For SOAP responses with XML namespaces, use the XPath format supported by the XML parser and endpoint variable resolver. If the parser or variable resolver is configured to ignore namespaces, write the XPath without namespace prefixes.

{
    "foo" : 
    {
        "abc" : "result"
    }
}

{
    "foo" : {
        "id" : "id1"
    },
    "foo2" : {
        "ID" : "id2",
        "id" : "id3"
    }
}

{
    "foo" : {
        "id" : "id1",
        "abcd" : "value1"
    },
    "foo2" : {
        "id" : "id2"
    }
}

{
     "records": [
         {"id": "firstId", "type":"firstType"},
         {"id":"secondId", "type":"secondType"}
     ],
     "items": [
         {"name":"firstName", "field":"value"},
         {"name":"secondName", "field":"value"},
         {"name":"thirdName", "field":"value"}
     ]
}

https://www.example.com/url_list

{
  "records": [ { "id": "firstId", "type": "firstType" }, { "id": "secondId", "type": "secondType" } ],
  "items": [ { "name": "firstName", "field": "value" }, { "name": "secondName", "field": "value" }, { "name": "thirdName", "field": "value" } ]
}

https://www.example.com/{getlist:$.records[*].id}/{getlist:$.items[*].name}

{
    "items": [
        {
            "BusinessEventCode": "JournalBatchApproved",
            "CreationDate": "2019-07-27T17:19:19.261+00:00",
            "links": [
                {
                    "rel": "self",
                    "href": "/erpBusinessEvents/self/100100120766717"
                },
                {
                    "rel": "canonical",
                    "href": "/erpBusinessEvents/rel/100100120766717"
                }
            ]
        }
    ]
}

https://www.example.com/log/{foo:$.items[*].links[*].href(rel='self')}

https://www.example.com/log/erpBusinessEvents/self/100100120766717

To authorize a connection between the agent and the REST API source, first configure the API credential in the agent's credential store. After configuring the source credentials in the Management Agent service on the agent side, you can use this information while creating the REST API log source.

To configure the source credentials in the Management Agent service to allow the Management Agent to collect data from your log-emitting host, see Management Agent Source Credentials.

While adding the log endpoint or log list endpoint, provide the credential information in the workflow by selecting the Log credential type. Select from one of the following options:

• None
• Basic Auth: Specify the Log credential name of the credential you created in the Management Agent service.
• Static Token: Specify the Log credential name of the credential you created in the Management Agent service.
• Dynamic Token (OAuth 2.0):

  Specify the Token credential name of the token you created in the Management Agent service. Additionally, provide the token information like Token Endpoint Name, Token Endpoint URL, Grant Type, and optionally Scope.

  If the token proxy is the same as that of the log endpoint, then keep the Proxy same as log endpoint check box enabled. If not, disable the check box, and provide the Token proxy server URL.

Mapping of credential types to the authentication type:

The following information is included in the ssl truststore properties:

• "ssl_trustStoreType": Type of store, for example, JKS.

• "ssl_trustStoreLocation": The path of trust store

• "ssl_trustStorePassword": Trust store password (optional)

Note the following aspects about the attributes in the credential JSON:

• source: The value must be lacollector.la_rest_api

• name: Any suitable name for the credential in one of the following formats.

  • <cred_name>.<entity_name>: credential name with entity name
  • <cred name>: just credential name

  The agent looks for the name in the first format. If not found, it looks for the name in the second format.

• type: This must be one of the values specified under the column Credential Type at Management Agent in the credential types table above.

See Examples of Credential JSON.

{
  "source":"lacollector.la_rest_api",
  "name":"ExampleRestAPICreds",
  "type":"HTTPSBasicAuthCreds",
  "description":"These are HTTPS (BasicAuth) credentials.",
  "properties":
  [
    { "name":"HTTPSUserName", "value":"CLEAR[admin]" },
    { "name":"HTTPSPassword", "value":"CLEAR[myHTTPSPassword]" }
  ]
}

{
  "source":"lacollector.la_rest_api",
  "name":"ExampleRestAPICreds",
  "type":"HTTPSCreds",
  "description":"These are HTTPS (BasicAuth) credentials.",
  "properties":
  [
    { "name":"HTTPSUserName", "value":"CLEAR[admin]" },
    { "name":"HTTPSPassword", "value":"CLEAR[myHTTPSPassword]" },
    { "name":"ssl_trustStoreType", "value":"JKS" },
    { "name":"ssl_trustStoreLocation", "value":"/scratch/certs/mycert.keystore" },
    { "name":"ssl_trustStorePassword", "value":"mySSLPassword" }
  ]
}

{
  "source":"lacollector.la_rest_api",
  "name":"ExampleRestAPICreds",
  "type":"HTTPSTokenCreds",
  "description":"These are HTTPS (Token) credentials.",
  "properties":
  [
    { "name": "HTTPSToken", "value": "CLEAR[token value]" },
    {"name": "HTTPSTokenType", "value": "CLEAR[Bearer]" }
  ]
}

{
  "source":"lacollector.la_rest_api",
  "name":"ExampleRestAPICreds",
  "type":"HTTPSTokenCreds",
  "description":"These are HTTPS (Token) credentials.",
  "properties":
  [
    { "name": "HTTPSToken", "value": "CLEAR[token value]" },
    {"name": "HTTPSTokenType", "value": "CLEAR[Bearer]" },
    { "name":"ssl_trustStoreType", "value":"JKS" },
    { "name":"ssl_trustStoreLocation", "value":"/scratch/certs/mycert.keystore" },
    { "name":"ssl_trustStorePassword", "value":"mySSLPassword" }
  ]
}

Follow these steps for collecting the Fusion Applications Audit Logs. For the list of available Oracle-defined sources for Fusion Applications, see Oracle-defined Sources.

Topics:

• Prerequisites
• Set Up Audit Log Collection from Fusion Applications