• Capture APM Events in JFR Recordings
• Configure JFR Events in APM Tracer
• Create an ACML File
• Frequently Asked Questions

1. Ensure APM Tracer is working properly.

   Open Trace Explorer and confirm you are able to view all the traces. For information about opening and using Trace Explorer, see Monitor Traces in Trace Explorer.

2. Add a dependency to the Maven project which will let the APM Tracer add events into the JFR recordings made by the Helidon JVM.

   <dependency>
       <groupId>com.oracle.apm.agent.java</groupId>
       <artifactId>apm-java-agent-jfr11</artifactId>
       <version>[1.0.1389,)</version>
   </dependency>

3. Initiate a recording.

   To test that everything is working properly, you need to initiate a recording using your preferred method (It could be using JDK Mission Control UI or JCMD comand line utility). After a recording has been initiated, trace events are displayed on JDK Mission Control UI when the recording is viewed.

   To verify that APM trace events are properly added to JFR recordings, connect to JDK Mission Control and confirm the APM traces are visible under Event Browser: Click on APM OpenTracing and confirm that APM trace events are visible.

• Case 1: component tag

  This is the simplest way to classify an event. This is the recommended method to use.

  In most scenarios, the Opentracing span will contain a tag called component. The value of component tag is used to determine the event type.

  For example, see the following span:

  {  "id":"e926ff3d06013045",
    "trace-id":"49abd92e69786853e926ff3d06013045",
    "parent-id":"6ef54ec2c524c99a",
    "name":"content-write",
    "ts-micros":1614190751360177,
    "td-micros":219348,
    "attributes":{
      "server":"Helidon SE",
      "http.url":"http://localhost:8079/health",
      "http.status_code":200,
      "component":"helidon-webserver",
      "http.method":"GET",
      "organization":"development"
    },
    "events":[
    ]
  },

  In this case the event generated will get assigned a helidon-webserver event type since that is the value of the component tag.

• Case 2: Other tag

  If your application does not use the component tag, but it uses another tag or there is a library which it's generating the tags for you then the APM Tracer can be configured to read the event type from that other tag.

  For example, see the following span where you can read the event type from the tag called span.type:

  {  "id":"e926ff3d06013045",
    "trace-id":"49abd92e69786853e926ff3d06013045",
    "parent-id":"6ef54ec2c524c99a",
    "name":"content-write",
    "ts-micros":1614190751360177,
    "td-micros":219348,
    "attributes":{
      "server":"Helidon SE",
      "http.url":"http://localhost:8079/health",
      "http.status_code":200,
      "span.type":"helidon-webserver",
      "http.method":"GET",
      "organization":"development"
    },
    "events":[
    ]
  },

  In this case, you can get the APM Tracer to read the event type from the span.type tag by using the following system property when starting the java application:

  java -Dcom.oracle.apm.agent.jfr.category.tag=span.type ...

• Case 3: Use Operation Name when no tags are available

  If your application does not use any tag that has useful categories then it is possible to use the Operation Name to determine the event type, but this method is not recommended.

  The JFR system requires that the name of the event type be a valid Java identifier, but the Operation Name often has spaces and also, other illegal characters. APM Tracer will convert the Operation Name into a valid identifier by replacing all illegal characters with an underscore.

  java -Dcom.oracle.apm.agent.jfr.category.default=ORACLE_APM_USE_SPAN_OPERATION_NAMES

  Note
  
  The Operation Name will only be used if the category tag is missing or empty.

  This is not a simple solution because the Operation Name may contain unique tokens like IDs which will make every event into it's own event type. In addition to that, the conversion of the Operation Name into a valid identifier will cause further problems in configuring the event type to setup which fields are needed in the event, and also, the configuration will need to be done based on the converted Operation Name otherwise you would need to configure many different variants of the Operation Name in the same way.

• Case 4: There are no tags available and all of the Operation Names are unique

  In this case there is no information available to classify the event types. This is the worst case scenario where all of the events will end up as the same event type, or everyone ends up unique if they are configured using use case 3 scenario as seen above. The only thing possible to do is to allow you to choose the name of this default event type and this can be configured as shown below.

  java -Dcom.oracle.apm.agent.jfr.category.default=Unknown

  Note
  
  The default will only be used if the category tag is missing or empty.

  In this case, all the events are placed in the event type called Unknown or any custom value selected.

caseSensitiveFieldName: false
recordCallStack: true

eventTypes:
  helidon-webserver:
    name: helidon-webserver
    label: Helidon Webserver Events
    description: All events based on helidon webserver
    recordCallStack: true
    # All fields must be provided here.
    # Fields are NOT incremental since that would make them impossible to remove 
    fields:
      - name: traceID
        type: String
        label: New Trace ID
        description: New Opentracing Trace ID
      - name: spanID
        type: String
        label: New Span ID
        description: New Opentracing Span ID
      - name: http.url
        type: String
        label: New HTTP URL
        description: The New URL for the HTTP endpoint being hit
      - name: organization
        type: String
        label: Organization
        description: The New Organization value of this call
      - name: startThread
        type: Thread
        label: Start Thread
        description: The thread on which this event started
      - name: http.status_code
        type: int
        label: Status Code
        description: The HTTP Response status code
      - name: http.method
        type: String
        label: HTTP Method
        description: The HTTP Method

1. When you have both settings provided: com.oracle.apm.agent.jfr.category.tag and com.oracle.apm.agent.jfr.category.default properties, Which setting does it take priority?

   If they are both given a value then we will first look in the span for the category tag. If it's found, that will be used. If it is missing, the default will be used.

2. For APM Tracer, Is a configuration file mandatory? Can I just specify the properties without file?

   A configuration file is not mandatory. If the default works for you then you don't need a configuration file, and you can still specify the properties.

3. In some cases, Why does it look like entries are duplicated? For example, the following snippet from the ACML configuration file has the helidon-webserver entry twice:

   eventTypes:
     helidon-webserver:
       name: helidon-webserver

   The first helidon-webserver entry is what we read/deduce from the tags or the operation name. If the event type is clean, easy to ready and intuitive, like the above example, then you may use the same event type to send to the JFR recording, and that how it will be get displayed in JMC.

   However, sometimes when using the operation name you may get an event type like the following: q_AxPsCbac9Ykp0HjpU+ENCiYHKl5HzMCmR02AfGzow3A= which represents a JDBC query from a db client. In this case you would want to change the name sent to the JFR recording to something intuitive therefore the below example can be used in those cases:

   eventTypes:
     q_AxPsCbac9Ykp0HjpU+ENCiYHKl5HzMCmR02AfGzow3A=:
       name: jdbc-select-query

4. If recordCallStack can be specified for each event type, Why is there also a need to have top level recordCallStack?

   We support two use cases for maximum flexibility in recording of the call stack. Call stack recording can be globally turned off and then it can selectively get turned on for specific event types. Another option is to globally turned it on and then it can selectively get turned off for specific event types. Both use cases are supported, therefore the setting are at both the top level, and at the event level.

5. I have some spans which were not configured at all and they are being ignored. Is it possible to still capture those events without configuring them?

   Yes. If you set allowUnconfiguredEvents: true in the main top level configuration then those events will get captured. When this happens, we don't know which fields to collect. Only the following set of default fields are collected: traceID, spanID, parentSpanID, startThread and operationName.

6. If I used com.oracle.apm.agent.jfr.category.default=ORACLE_APM_USE_SPAN_OPERATION_NAMES for using categories defined by operation name, Does the operation name have to match exactly to the event type name in the configuration file? If it does, and if I have unique operation names, Do I need to specify an event type for each operation?

   Yes, it must match the event type name in the configuration file. Even if you have a lot of unique operation names, you will need to specify an event type for each operation because each of those event types are likely to have a different set of fields that need to be configured.

7. My Operation Name is getting converted into a Java identifier therefore I cannot configure its event type, How can I determine what my Operation Name was converted to?

   All illegal characters have been replaced with an underscore. If you need a sample Java code to convert your strings into identifiers then you can use the below code whicht it's the same routine that APM uses to convert the strings.

   // Convert an arbitrary string into a valid java identifier by replacing non-identifier chars with an underscore
   public static String getJavaIdentifier(String inString)
   {
       if (inString == null || inString.length() == 0)
          return inString;
       StringBuilder stringBuilder = new StringBuilder();
       char[] allChars = inString.toCharArray();
   
       if(Character.isJavaIdentifierStart(allChars[0]))
           stringBuilder.append(allChars[0]);
       else
           stringBuilder.append("_");
   
       for (int i = 1; i < allChars.length; i++)
       {
           if (Character.isJavaIdentifierPart(allChars[i]))
               stringBuilder.append(allChars[i]);
           else
               stringBuilder.append("_");
       }
       return stringBuilder.toString();
   }

8. Can the name matching be RegEx?

   No. RegEx is not supported.