Distributed Tracing for Functions

When a function is invoked but doesn't run or perform as expected, you need to investigate the issue at a detailed level. The distributed tracing feature observes the function's execution as it moves through the different components of the system. You can trace and instrument standalone functions to debug execution and performance issues. You can also use function tracing to debug issues with complete serverless applications comprising multiple functions and services, such as:

  • a function calling another function
  • a function calling other services such as the Object Storage service
  • a function that serves as a backend for an API gateway deployed in the API Gateway service
  • a function triggered in response to an event by the Events service, Notifications service, or Service Connector Hub

The Oracle Functions tracing capabilities are provided by the Oracle Cloud Infrastructure Application Performance Monitoring service. Features in Application Performance Monitoring (APM) enable you to identify and troubleshoot failures and latency issues in the functions you create and deploy.

In the Application Performance Monitoring service:

  • An APM domain contains the systems monitored by Application Performance Monitoring. An APM domain is an instance of a collector of trace and span data which stores, aggregates, displays, and visualizes the data.
  • A trace is the complete flow of a request as it passes through all the components of a distributed system in a given time period. It consists of an entire tree of spans all related to the same single overall request flow.
  • A span is an operation or a logical unit of work with a name, start time, and duration, within a trace. A span is a time segment associated with the duration of a unit of work within the overall request flow.

The Application Performance Monitoring Trace Explorer enables you to visualize the entire request flow and explore trace and span details for diagnostics. You can view and monitor slow traces and traces with errors. To isolate and identify trace issues, you can drill down into specific spans, such as page loads, AJAX calls, and service requests. For more information about the Application Performance Monitoring service, see Application Performance Monitoring.

To enable tracing for a function, you must:

  1. Set up a policy to give the Oracle Functions service permission to access APM domains, if the policy does not exist already (see Create a Policy to Give Oracle Functions Users and the Oracle Functions Service Access to Tracing Resources).
  2. Set up an APM domain.
  3. Enable tracing for the Functions application and select the APM domain you created.
  4. Enable tracing for one or more functions.

When you enable tracing for a function, Oracle Functions automatically generates a "default function invocation span." The default span captures information about the function's execution context including the overall time taken to process the request and return a response to the caller. In addition to the default function invocation span, you can add code to functions to define custom spans. Use custom spans to capture more function-specific information to help with debugging. For example, you might define custom spans to capture the start and end of specific units of work. For example, units of work could include getting the database password from the Vault, opening a database connection, and retrieving records from the database.

Four variables have been added to the Oracle Functions context that provide helpful tracing information. These variables include:

  • FN_APP_NAME: The function application name.
  • FN_FN_NAME: The function name.
  • OCI_TRACE_COLLECTOR_URL: The APM domain URL with data key.
  • OCI_TRACING_ENABLED: Is tracing enabled?
    • When retrieved from environment variables, returns 0 or 1.
    • When retrieved from the function context, returns true or false as appropriate for the language used.

Required IAM Policy for Enabling Tracing

Before you can enable tracing, the group to which you belong must have permission to access existing APM domains or to create APM domains. In addition, Oracle Functions must have permission to access APM domains. See Create a Policy to Give Oracle Functions Users and the Oracle Functions Service Access to Tracing Resources.

Using the Console to Enable Tracing and View Function Traces

A couple of steps are required to enable tracing and to view function traces for the Oracle Cloud Infrastructure Application Performance Monitoring (APM) service. First, enable tracing for the application containing the function. Then, enable tracing for one or more functions. You can then view function traces in the APM Trace Explorer.

Using the Console to Enable Tracing

To enable tracing, follow these steps.

  1. Sign in to the Console as a functions developer.
  2. In the Console, open the navigation menu and click Developer Services. Under Functions, click Applications.
  3. Select the region and compartment containing the Functions application.

    The Applications page shows all the applications in the compartment you selected.

  4. Select the Functions application for which you want to enable tracing.
  5. To enable tracing for the application:
    1. Under Resources, click Traces.
    2. Select the Trace Enabled option and specify:
      • Compartment: The compartment in which to create the trace. By default, the current compartment.
      • APM Domain: The APM domain (defined in the Application Performance Monitoring service) in which to create the trace. To use an existing APM Domain, select an existing APM domain from the list. Or, to create a new APM domain, click APM Domain. For more information about APM domains, see Getting Started with Application Performance Monitoring.
        Note

        The APM Domain needs to have both public and private data keys for function tracing to work. If the keys do not exist, you can create them through the console interface.
    3. Click Enable Trace to enable tracing for the application.

    Having enabled tracing for the Functions application, you can now enable tracing for one or more functions in the application.

  6. To enable tracing for specific functions in the application:
    1. Under Resources, click Functions.
    2. Select the Enable Trace option beside one or more function(s) for which you want to enable tracing.

      The Enable Trace option is only shown if you have previously enabled tracing for the application. Note the following:

      • If the Enable Trace option is not shown, you must enable tracing for the application. If you haven't already enabled tracing for the application, see the previous step.
      • If you previously enabled tracing for the application but later disabled it, an Enable application tracing link is shown. Click the Enable application tracing link to re-enable tracing for the application (see the previous step). Having re-enabled tracing for the application, you can then enable tracing for specific functions.

When you have enabled tracing for the application and one or more functions, you can view function traces.

Using the Console to View Function Traces

To view the traces for functions that have tracing enabled:

  1. Sign in to the Console as a functions developer.
  2. In the Console, open the navigation menu and click Developer Services. Under Functions, click Applications.
  3. Select the region and compartment containing the Functions application with functions for which you want to view function traces.

    The Applications page shows all the applications in the compartment you selected.

  4. Select the application containing the functions for which you want to view traces.
  5. To see traces for functions:
    1. To see traces for all the functions that have tracing enabled in the application:
      1. Under Resources, click Traces.
      2. Click the name of the trace.
        Note

        A trace name is only shown if you have already enabled tracing for the application.
    2. To see the trace for a specific function that has tracing enabled:
      1. Under Resources, click Functions.
      2. Click the Actions icon (three dots) beside the function, and then click View Trace.
        Note

        The View Trace option is only shown if you have already enabled tracing for the function.

    The traces for the functions you selected are shown in the APM Trace Explorer. By default, a trace is shown for the default function invocation span, and any custom spans defined for the function.

  6. In the APM Trace Explorer:
    1. Click a trace to see the spans for that trace.
    2. Click a span to see the details captured for that span.

    For more information about using the APM Trace Explorer, see Use Trace Explorer.

Tracing a Chain of Functions

By default, function tracing provides a trace for an entire function invocation. However, often with modern cloud applications, you need to chain function invocations. OCI Functions tracing provides the ability trace the execution of a function invoked by another function. This ability means you can examine the execution of each function in a chain of calls in a single tree of spans in APM trace explorer.

To trace a chain of functions, you need to propagate the X-B3 headers X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, and X-B3-Sampled in the function invocation request from your function code.

After the function has run, the trace data from your functions is collected and available in APM Trace Explorer. For more information about using the APM Trace Explorer, see Use Trace Explorer.

Tracing a Chain of Functions with Python

Here's an example of how you can trace a chain of functions. If you want to try you this example, you need to create two sample functions. Follow these steps to set up your functions.

  1. Create your tracing Python function: fn init --runtime python <your-function-name-1>
  2. Create your "Hello World!" Python function: fn init --runtime python <your-function-name-2>
  3. Deploy both functions: fn -v deploy --app <app-name>
  4. Get the second functions OCID and invoke endpoint: fn inspect function your-app-name your-function-name-2
  5. Create JSON file to pass the required information into the first function. For example, your test.json file might look like this:
    
    {
        "function_ocid": "ocid1.fnfunc.oc1.iad.aaaaaaaaxxxxxxxxxxx",
        "function_endpoint": "https://xxxxxxxxx.us-ashburn-1.functions.oci.oraclecloud.com",
        "function_body": "",
        "__comment": "Alternatively, you can set function_body to { \"name\": \"Oracle\" }"
    }                    
                    
  6. When the first function is invoked, you can pass the second functions information using test.json: fn invoke <app-name> <your-function-name-1> < test.json

Now you are ready to update the first function with the required code updates.

Configure Packages

Update your requirements.txt file to include the following packages:


fdk
oci
            

Save the file.

Update your Function Code to Propagate the X-B3 Headers

The Python function calls the handler function and passes in the JSON information from the invoke command. The handler function is broken in into several small blocks to simplicity. The complete source file is provided at the bottom of this section.

Load the JSON Data

In this first part, the JSON data is loaded from the function invocation.


import io
import json
import logging
import oci
from fdk import response

def handler(ctx, data: io.BytesIO=None):
    app_name = ctx.AppName()
    func_name = ctx.FnName()
    logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler")

    try:
        body = json.loads(data.getvalue())
        function_endpoint = body.get("function_endpoint")
        function_ocid = body.get("function_ocid")
        function_body = body.get("function_body")
    except (Exception) as ex:
        print('ERROR: Missing key in payload', ex, flush=True)
        raise
            

Create Invoke Client and Gather Header Information

Create the Functions invoke client using the OCI Python SDK and Functions resource principals. Then, retrieve the tracing_context and extract the required information to create the HTTP headers.


    signer = oci.auth.signers.get_resource_principals_signer()
    client = oci.functions.FunctionsInvokeClient(config={}, signer=signer, service_endpoint=function_endpoint)
    
    #
    # Zipkin X-B3- header propagation
    #
    tracing_context = ctx.TracingContext()
    trace_id = tracing_context.trace_id()
    span_id = tracing_context.span_id()
    parent_span_id = tracing_context.parent_span_id()
    is_sampled = tracing_context.is_sampled()
            

Propagate the X-B3 Headers

The OCI Python SDK lets you set custom headers. Use this technique to pass the X-B3 headers in to the second function invocation. Header information is passed for trace_id, span_id, parent_span_id, and is_sampled. Finally, the second function is invoked with client and the response is passed to this function's response.


    # if tracing is enabled, is_sampled will be true in the tracing context
    if is_sampled:
        # To propagate headers in the OCI SDK in the request to the next function,
        # add the X-B3- headers in the request. This header will be included in ALL
        # subsequent calls made.
        if trace_id is not None:
            client.base_client.session.headers['X-B3-TraceId'] = trace_id
            logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | trace_id: " + trace_id)
        if span_id is not None:
            client.base_client.session.headers['X-B3-SpanId'] = span_id
            logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | span_id: " + span_id)
        if parent_span_id is not None:
            client.base_client.session.headers['X-B3-ParentSpanId'] = parent_span_id
            logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | parent_span_id: " + parent_span_id)
        client.base_client.session.headers['X-B3-Sampled'] = str(int(is_sampled))
        logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | is_sampled: " + str(int(is_sampled)))
    else:
        # function.trace is DISABLED
        logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | function tracing is DISABLED")

    resp = client.invoke_function(function_id=function_ocid, invoke_function_body=function_body)
    logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | Response: " + resp.data.text)

    return response.Response(
        ctx, 
        response_data=resp.data.text,
        headers={"Content-Type": "application/json"}
    )
            
Review Complete Function Source Code

Here is the complete source code for the sample Python function.


#
# oci-invoke-function-python version 2.0.
#
# Copyright (c) 2021 Oracle, Inc.
# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.
#

import io
import json
import logging
import oci
from fdk import response

def handler(ctx, data: io.BytesIO=None):
    app_name = ctx.AppName()
    func_name = ctx.FnName()
    logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler")

    try:
        body = json.loads(data.getvalue())
        function_endpoint = body.get("function_endpoint")
        function_ocid = body.get("function_ocid")
        function_body = body.get("function_body")
    except (Exception) as ex:
        print('ERROR: Missing key in payload', ex, flush=True)
        raise
    
    signer = oci.auth.signers.get_resource_principals_signer()
    client = oci.functions.FunctionsInvokeClient(config={}, signer=signer, service_endpoint=function_endpoint)
    
    #
    # Zipkin X-B3- header propagation
    #
    tracing_context = ctx.TracingContext()
    trace_id = tracing_context.trace_id()
    span_id = tracing_context.span_id()
    parent_span_id = tracing_context.parent_span_id()
    is_sampled = tracing_context.is_sampled()

    # if tracing is enabled, is_sampled will be true in the tracing context
    if is_sampled:
        # To propagate headers in the OCI SDK in the request to the next function,
        # add the X-B3- headers in the request. This header will be included in ALL
        # subsequent calls made.
        if trace_id is not None:
            client.base_client.session.headers['X-B3-TraceId'] = trace_id
            logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | trace_id: " + trace_id)
        if span_id is not None:
            client.base_client.session.headers['X-B3-SpanId'] = span_id
            logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | span_id: " + span_id)
        if parent_span_id is not None:
            client.base_client.session.headers['X-B3-ParentSpanId'] = parent_span_id
            logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | parent_span_id: " + parent_span_id)
        client.base_client.session.headers['X-B3-Sampled'] = str(int(is_sampled))
        logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | is_sampled: " + str(int(is_sampled)))
    else:
        # function.trace is DISABLED
        logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | function tracing is DISABLED")

    resp = client.invoke_function(function_id=function_ocid, invoke_function_body=function_body)
    logging.getLogger().info("Inside app: " + app_name + " | function: " + func_name + " | method: handler | Response: " + resp.data.text)

    return response.Response(
        ctx, 
        response_data=resp.data.text,
        headers={"Content-Type": "application/json"}
    )
            

Adding Custom Spans to Functions

With function tracing enabled, the default function invocation span provides a trace for the entire function invocation. The default span can provide good information, but when investigating your code you might want to dig deeper. Custom spans are added directly to your code and allow you to define spans for a method or a block of code. The resulting data provides a better picture of your function as it runs.

Before you can use custom spans, you must enable tracing for your application and functions using the Oracle Cloud Infrastructure Application Performance Monitoring (APM) service. To set up tracing, you must:

  1. Set up a policy to give the Oracle Functions service permission to access APM domains, if the policy does not exist already (see Create a Policy to Give Oracle Functions Users and the Oracle Functions Service Access to Tracing Resources).
  2. Set up an APM domain.
  3. Enable tracing for the Functions application and select the APM domain you created.
  4. Enable tracing for one or more functions.

These steps have already been covered. However, a couple more things are required for custom spans:

  • Select a distributed tracing client library, for example Zipkin.
  • Add client libraries to your function dependencies.
  • In your function code, use the OCI_TRACING_ENABLED function context variable to check if tracing is enabled.
  • In your function code, use the OCI_TRACE_COLLECTOR_URL function context variable to send your custom spans to your APM domain.
  • Add instrumentation to your function code.
Note

To use custom spans, you must have the following minimum versions of the Fn Project FDKs:

  • Java FDK: 1.0.129
  • Python FDK: 0.1.22
  • Node FDK: 0.0.20
Adding Custom Spans to Java Functions

Here's an example of how to use Zipkin to add custom spans to your Java function. If you want to try you this example, you can create a Java "Hello World!" function and add custom span code. To create a sample function:

  • Create a Java function: fn init --runtime java apm-fn-java
  • For simplicity, remove the src/test directory.

Configure Maven

Add the following dependencies to the <dependencies> section of your Maven pom.xml file.


<dependency>
    <groupId>io.zipkin.reporter2</groupId>
    <artifactId>zipkin-sender-urlconnection</artifactId>
    <version>2.16.3</version>
</dependency>
<dependency>
    <groupId>io.zipkin.reporter2</groupId>
    <artifactId>zipkin-reporter-brave</artifactId>
    <version>2.16.3</version>
</dependency>
<dependency>
    <groupId>io.zipkin.brave</groupId>
    <artifactId>brave</artifactId>
    <version>5.13.3</version>
</dependency>
    <dependency>
    <groupId>io.zipkin.brave</groupId>
    <artifactId>brave-core</artifactId>
    <version>4.13.6</version>
</dependency>
            

Save the file.

The HandleRequest Method

Observations about the method follow the handleRequest source code.


package com.example.fn;
import brave.Span;
import brave.Tracer;
import brave.Tracing;
import brave.propagation.*;
import brave.sampler.Sampler;
import com.fnproject.fn.api.tracing.TracingContext;
import com.github.kristofa.brave.IdConversion;
import zipkin2.reporter.Sender;
import zipkin2.reporter.brave.AsyncZipkinSpanHandler;
import zipkin2.reporter.urlconnection.URLConnectionSender;

public class HelloFunction {
    Sender sender;
    AsyncZipkinSpanHandler zipkinSpanHandler;
    Tracing tracing;
    Tracer tracer;
    String apmUrl;
    TraceContext traceContext;

    public String handleRequest(String input, TracingContext tracingContext) {
        try {
            intializeZipkin(tracingContext);
            // Start a new trace or a span within an existing trace representing an operation
            Span span = tracer.newChild(traceContext).name("MainHandle").start();
            System.out.println("Inside Java Hello World function");
            try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
                method1();
                method2();
                method3();
            } catch (RuntimeException | Error e) {
                span.error(e); // Unless you handle exceptions, you might not know the operation failed!
                throw e;
            } finally {
                span.finish(); // note the scope is independent of the span. Always finish a span.
                tracing.close();
                zipkinSpanHandler.flush();
            }
        } catch (Exception e) {
            return e.getMessage();
        }
        return "Hello, AppName " + tracingContext.getAppName() + " :: fnName " + tracingContext.getFunctionName();
    }
            
  • The TracingContext tracingConext object passes in all the APM-related information needed to make connections to the APM service.
  • The intializeZipkin method is called which updates the tracingContext and creates a tracer object which is used to set up custom spans.
  • A span is created for the parent custom span. Then three methods are called in the scope of the parent span.
  • Notice in the finally block all the tracing objects are closed out.

The initializeZipkin Method

Observations about the intializeZipkin method follow the source code.

     
    public void intializeZipkin(TracingContext tracingContext) throws Exception {
        System.out.println("Initializing the variables");
        apmUrl = tracingContext.getTraceCollectorURL();
        sender = URLConnectionSender.create(apmUrl);
        zipkinSpanHandler = AsyncZipkinSpanHandler.create(sender);
        tracing = Tracing.newBuilder()
                .localServiceName(tracingContext.getServiceName())
                .sampler(Sampler.NEVER_SAMPLE)
                .addSpanHandler(zipkinSpanHandler)
                .build();
        tracer = tracing.tracer();
        tracing.setNoop(!tracingContext.isTracingEnabled());
        traceContext = TraceContext.newBuilder()
                .traceId(IdConversion.convertToLong(tracingContext.getTraceId()))
                .spanId(IdConversion.convertToLong(tracingContext.getSpanId()))
                .sampled(tracingContext.isSampled()).build();
    }   
            
  • The traceContext is passed in to create all the objects used to create custom spans.
  • The apmURL is retrieved from the getTraceCollectorURL() method. The URL is the endpoint to the APM domain and is used to create the tracer object which builds the custom spans.
  • A builder takes the zipkinSpanHandler and the service name to create a tracer object. This tracer object is used to create custom spans.

Creating Custom Spans

With the tracer object initialized, custom spans can be created.


public void method1() {
    System.out.println("Inside Method1 function");
    TraceContext traceContext = tracing.currentTraceContext().get();
    Span span = tracer.newChild(traceContext).name("Method1").start();
    try {
        Thread.sleep(200);
    } catch (InterruptedException e) {
        e.printStackTrace();
    } finally {
        span.finish();
    }
}
            
  • The method1 method creates a custom span named "Method1."
Review Complete Function Source Code

Here is the complete source code for the sample Java tracing function.


package com.example.fn;
import brave.Span;
import brave.Tracer;
import brave.Tracing;
import brave.propagation.*;
import brave.sampler.Sampler;
import com.fnproject.fn.api.tracing.TracingContext;
import com.github.kristofa.brave.IdConversion;
import zipkin2.reporter.Sender;
import zipkin2.reporter.brave.AsyncZipkinSpanHandler;
import zipkin2.reporter.urlconnection.URLConnectionSender;

public class HelloFunction {
    Sender sender;
    AsyncZipkinSpanHandler zipkinSpanHandler;
    Tracing tracing;
    Tracer tracer;
    String apmUrl;
    TraceContext traceContext;
    public void intializeZipkin(TracingContext tracingContext) throws Exception {
        System.out.println("Initializing the variables");
        apmUrl = tracingContext.getTraceCollectorURL();
        sender = URLConnectionSender.create(apmUrl);
        zipkinSpanHandler = AsyncZipkinSpanHandler.create(sender);
        tracing = Tracing.newBuilder()
                .localServiceName(tracingContext.getServiceName())
                .sampler(Sampler.NEVER_SAMPLE)
                .addSpanHandler(zipkinSpanHandler)
                .build();
        tracer = tracing.tracer();
        tracing.setNoop(!tracingContext.isTracingEnabled());
        traceContext = TraceContext.newBuilder()
                .traceId(IdConversion.convertToLong(tracingContext.getTraceId()))
                .spanId(IdConversion.convertToLong(tracingContext.getSpanId()))
                .sampled(tracingContext.isSampled()).build();
    }

    public String handleRequest(String input, TracingContext tracingContext) {
        try {
            intializeZipkin(tracingContext);
            // Start a new trace or a span within an existing trace representing an operation
            Span span = tracer.newChild(traceContext).name("MainHandle").start();
            System.out.println("Inside Java Hello World function");
            try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
                method1();
                method2();
                method3();
            } catch (RuntimeException | Error e) {
                span.error(e); // Unless you handle exceptions, you might not know the operation failed!
                throw e;
            } finally {
                span.finish(); // note the scope is independent of the span. Always finish a span.
                tracing.close();
                zipkinSpanHandler.flush();
            }
        } catch (Exception e) {
            return e.getMessage();
        }
        return "Hello, AppName " + tracingContext.getAppName() + " :: fnName " + tracingContext.getFunctionName();
    }

    public void method1() {
        System.out.println("Inside Method1 function");
        TraceContext traceContext = tracing.currentTraceContext().get();
        Span span = tracer.newChild(traceContext).name("Method1").start();
        try {
            Thread.sleep(200);
        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            span.finish();
        }
    }

    public void method2() {
        System.out.println("Inside Method2 function");
        TraceContext traceContext = tracing.currentTraceContext().get();
        Span span = tracer.newChild(traceContext).name("Method2").start();
        try {
            Thread.sleep(400);
        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            span.finish();
        }
    }

    public void method3() {
        System.out.println("Inside Method3 function");
        TraceContext traceContext = tracing.currentTraceContext().get();
        Span span = tracer.newChild(traceContext).name("Method3").start();
        try {
            Thread.sleep(300);
        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            span.finish();
        }
    }
}
            
Adding Custom Spans to Python Functions

Here's an example of how to use Zipkin to add custom spans to your Python function. If you want to try you this example, you can create a Python "Hello World!" function and add custom span code. To create a sample function:

  • Create a Python function: fn init --runtime python apm-fn-python

Configure Packages

Update your requirements.txt file to include the following packages:


fdk
requests
py_zipkin
            

Save the file.

Creating Handler Class and Parent Custom Span

The Python function calls the handler function and passes in the function context to create custom spans.

def handler(ctx, data: io.BytesIO = None):
    tracing_context = ctx.TracingContext()
    with zipkin_span(
        service_name=tracing_context.service_name(),
        span_name="Customer Code",
        transport_handler=(
            lambda encoded_span: transport_handler(
                encoded_span, tracing_context
            )
        ),
        zipkin_attrs=tracing_context.zipkin_attrs(),
        encoding=Encoding.V2_JSON,
        binary_annotations=tracing_context.annotations()
    ):
        name = "World"
        try:
            body = json.loads(data.getvalue())
            name = body.get("name")
        except (Exception, ValueError) as ex:
            logging.getLogger().info('error parsing json payload: ' + str(ex))

        logging.getLogger().info("Inside Python Hello World function")
        time.sleep(0.005)
        example(ctx)
        return response.Response(
            ctx, response_data=json.dumps(
                {"message": "Hello {0}".format(name)}),
            headers={"Content-Type": "application/json"}
        )         
  • The tracing_context is passed from the function context and contains all the information needed to create and configure custom spans.
    Note

    If tracing is not enabled, the tracing context is an empty object. With an empty tracing context, the is_sampled flag is set to None and py_zipkin does not emit spans.
  • The with zipkin_span statement is used to create spans.
    • The information in tracing_context is used to get the service_name, call the transport_handler, and set the zipking_attrs.
    • A custom span name is specified just by setting span_name.
    • Tracing attributes required for Zipkin are retrieved from the tracing context: tracing_context.zipkin_attrs().
  • With the custom span setup, the main block runs boilerplate "Hello World!" code. With the only exception, a call to the example function.

The transport_handler Function

The transport_handler function communicates with the APM domain with messages about span execution.

# transport handler, needed by py_zipkin
def transport_handler(encoded_span, tracing_context):
    return requests.post(
        tracing_context.trace_collector_url(),
        data=encoded_span,
        headers={"Content-Type": "application/json"},
    )            
  • The trace_collector_url is returned from the function context. This URL provides the communication endpoint for your custom spans to the APM domain.

Creating a Custom Span in Example Function

The example function demonstrates the creation of a custom span.

def example(ctx):
    with zipkin_span(
        service_name=ctx.TracingContext().service_name(),
        span_name="Get ADB Password from OCI Vault",
        binary_annotations=ctx.TracingContext().annotations()
    ) as example_span_context:
        try:
            logging.getLogger().debug("Get ADB Password from OCI Vault")
            time.sleep(0.005)
            # throwing an exception to show how to add error messages to spans
            raise Exception('Request failed')
        except (Exception, ValueError) as error:
            example_span_context.update_binary_annotations(
                {"Error": True, "errorMessage": str(error)}
            )
        else:
            FakeResponse = namedtuple("FakeResponse", "status, message")
            fakeResponse = FakeResponse(200, "OK")
            # how to update the span dimensions/annotations
            example_span_context.update_binary_annotations(
                {
                    "responseCode": fakeResponse.status,
                    "responseMessage": fakeResponse.message
                }
            )            
  • The with zipkin_span statement is used to identify the custom span and give it a name.
  • The example_span_context block raises an exception and returns an error message.
Review Complete Function Source Code

Here is the complete source code for the sample Python tracing function.


import io
import json
import logging

from fdk import response

import requests
import time
from py_zipkin import Encoding
from py_zipkin.zipkin import zipkin_span
from collections import namedtuple


# transport handler, needed by py_zipkin
def transport_handler(encoded_span, tracing_context):
    return requests.post(
        tracing_context.trace_collector_url(),
        data=encoded_span,
        headers={"Content-Type": "application/json"},
    )


def handler(ctx, data: io.BytesIO = None):
    tracing_context = ctx.TracingContext()
    with zipkin_span(
        service_name=tracing_context.service_name(),
        span_name="Customer Code",
        transport_handler=(
            lambda encoded_span: transport_handler(
                encoded_span, tracing_context
            )
        ),
        zipkin_attrs=tracing_context.zipkin_attrs(),
        encoding=Encoding.V2_JSON,
        binary_annotations=tracing_context.annotations()
    ):
        name = "World"
        try:
            body = json.loads(data.getvalue())
            name = body.get("name")
        except (Exception, ValueError) as ex:
            logging.getLogger().info('error parsing json payload: ' + str(ex))

        logging.getLogger().info("Inside Python Hello World function")
        time.sleep(0.005)
        example(ctx)
        return response.Response(
            ctx, response_data=json.dumps(
                {"message": "Hello {0}".format(name)}),
            headers={"Content-Type": "application/json"}
        )


def example(ctx):
    with zipkin_span(
        service_name=ctx.TracingContext().service_name(),
        span_name="Get ADB Password from OCI Vault",
        binary_annotations=ctx.TracingContext().annotations()
    ) as example_span_context:
        try:
            logging.getLogger().debug("Get ADB Password from OCI Vault")
            time.sleep(0.005)
            # throwing an exception to show how to add error messages to spans
            raise Exception('Request failed')
        except (Exception, ValueError) as error:
            example_span_context.update_binary_annotations(
                {"Error": True, "errorMessage": str(error)}
            )
        else:
            FakeResponse = namedtuple("FakeResponse", "status, message")
            fakeResponse = FakeResponse(200, "OK")
            # how to update the span dimensions/annotations
            example_span_context.update_binary_annotations(
                {
                    "responseCode": fakeResponse.status,
                    "responseMessage": fakeResponse.message
                }
            )
            
Adding Custom Spans to Node Functions

Here's an example of how to use Zipkin to add custom spans to your Node.js function. If you want to try you this example, you can create a Node "Hello World!" function and add custom span code. To create a sample function:

  • Create a Node function: fn init --runtime node apm-fn-node

Configure Node Dependencies

Update your package.json file to include the following packages:


{
    "name": "apm-tracing-node-fdk-simple-trace-final",
    "version": "1.0.0",
    "description": "Example APM tracing function",
    "main": "func.js",
    "author": "",
    "license": "Apache-2.0",
    "dependencies": {
        "@fnproject/fdk": ">=0.0.13",
        "node-fetch": "^2.6.1",
        "zipkin": "^0.22.0",
        "zipkin-transport-http": "^0.22.0"
    }
}
            

Save the file.

Update Handle Method

Key observations about the fdk.handle method follow the source code.


// ZipkinJS core components.
const { 
    ExplicitContext, 
    Annotation, 
    Tracer, 
    TraceId, 
    BatchRecorder, 
    jsonEncoder, 
    sampler, 
    option
} = require('zipkin');

// An HTTP transport for dispatching Zipkin traces.
const {HttpLogger} = require('zipkin-transport-http');

fdk.handle(async function(input, ctx){
    tracer = createOCITracer(ctx);
    
    var result;
    // Start a new 'scoped' server handling span.
    await tracer.scoped(async function () {
        // Fetch some resource
        result = await tracer.local('fetchResource', () => {
            return fetchResource();
        });
        // Perform some processing
        result = await tracer.local('processResource', () => {
            return someComputation(result);
        });
        // Update some resource
        result = await tracer.local('updateResource', () => {
            return updateResource(result);
        });
        await flush();
    }); 
    
    return result;

})            
            
  • The tracer is created and then used to create a parent custom span. Then child spans are created for the fetchResource, processResource, and updateResource functions.

Reviewing the createOCITracer Function

Key observations about the function follow the source code.


/**
 * Creates a basic Zipkin Tracer using values from context of the function
 * invocation.
 *
 * @param {*} ctx The function invocation context.
 * @returns       A configured Tracer for automatically tracing calls.
 */
function createOCITracer (ctx) {
  // An OCI APM configured Tracer
  //
  const tracingCxt = ctx.tracingContext
  const tracer = new Tracer({
    ctxImpl: new ExplicitContext(),
    recorder: new BatchRecorder({
      logger: new HttpLogger({
        // The configured OCI APM endpoint is available in the function
        // invocation context.
        endpoint: tracingCxt.traceCollectorUrl,
        jsonEncoder: jsonEncoder.JSON_V2
      })
    }),
    // APM Dimensions that should be included in all traces can be configured
    // directly on Tracer.
    defaultTags: createOCITags(ctx),
    // A custom sampling strategy can be defined.
    sampler: createOCISampler(ctx),
    localServiceName: tracingCxt.serviceName,
    supportsJoin: true,
    traceId128Bit: true
  })

  // The initial function invocation trace identifiers can be added directly.
  // If this is not defined a default TraceId is created.
  const traceId = createOCITraceId(tracer, ctx)
  tracer.setId(traceId)
  return tracer
}
            
  • The function context (ctx) is passed to this function which provides the information required to connect to the APM domain. If you follow the function calls, you can see how the tracing IDs and fields are built.
Review Complete Function Source Code

Here is the complete source code for the sample Node tracing function.



const fdk = require('@fnproject/fdk')

// ZipkinJS core components.
const {
  ExplicitContext,
  Tracer,
  TraceId,
  BatchRecorder,
  jsonEncoder,
  sampler,
  option
} = require('zipkin')

// An HTTP transport for dispatching Zipkin traces.
const { HttpLogger } = require('zipkin-transport-http')

fdk.handle(async function (input, ctx) {
  var tracer = createOCITracer(ctx)

  var result
  // Start a new 'scoped' server handling span.
  await tracer.scoped(async function () {
    // Fetch some resource
    result = await tracer.local('fetchResource', () => {
      return fetchResource()
    })
    // Perform some processing
    result = await tracer.local('processResource', () => {
      return someComputation(result)
    })
    // Update some resource
    result = await tracer.local('updateResource', () => {
      return updateResource(result)
    })
    await flush()
  })

  return result
})

// ----------------------------------------------------------------------------
// App Simulation Functions
//

/**
 * Simulate fetching some required resource. This could be another OCI service
 * or an external call.
 *
 * @returns A Promise with the success or failure of the operation.
 */
function fetchResource () {
  return simulate(1000, { fetchResource: 'OK' })
}

/**
 * Simulate some work. This could be another OCI service.
 *
 * @returns A Promise with the success or failure of the operation.
 */
async function someComputation (toReturn) {
  var i
  for (i = 0; i < 5; i++) {
    await simulate(1000)
  }
  toReturn.processResource = 'OK'
  return toReturn
}

/**
 * Simulate updating some resource. This could be another OCI service or an
 * external call.
 *
 * @returns A Promise with the success or failure of the operation.
 */
async function updateResource (toReturn) {
  await simulate(500)
  toReturn.updateResource = 'OK'
  return toReturn
}

/**
 * A helper function to simulate an operation that takes a specified amount of time.
 *
 * @param {*} ms The simulated time for the activity in milliseconds.
 * @returns      A promise that resolves when the simulated activity finishes.
 */
function simulate (ms, result) {
  return new Promise(resolve => setTimeout(resolve, ms, result))
}

/**
 * Functions service may freeze or terminate the container on completion.
 * This function gives extra time to allow the runtime to flush any pending traces.
 * See: https://github.com/openzipkin/zipkin-js/issues/507
 *
 * @returns A Promise to await on.
 */
function flush () {
  return new Promise(resolve => setTimeout(resolve, 1000))
}

// ----------------------------------------------------------------------------
// OpenZipkin ZipkinJS Utility Functions
//

/**
 * Creates a basic Zipkin Tracer using values from context of the function
 * invocation.
 *
 * @param {*} ctx The function invocation context.
 * @returns       A configured Tracer for automatically tracing calls.
 */
function createOCITracer (ctx) {
  // An OCI APM configured Tracer
  //
  const tracingCxt = ctx.tracingContext
  const tracer = new Tracer({
    ctxImpl: new ExplicitContext(),
    recorder: new BatchRecorder({
      logger: new HttpLogger({
        // The configured OCI APM endpoint is available in the function
        // invocation context.
        endpoint: tracingCxt.traceCollectorUrl,
        jsonEncoder: jsonEncoder.JSON_V2
      })
    }),
    // APM Dimensions that should be included in all traces can be configured
    // directly on Tracer.
    defaultTags: createOCITags(ctx),
    // A custom sampling strategy can be defined.
    sampler: createOCISampler(ctx),
    localServiceName: tracingCxt.serviceName,
    supportsJoin: true,
    traceId128Bit: true
  })

  // The initial function invocation trace identifiers can be added directly.
  // If this is not defined a default TraceId is created.
  const traceId = createOCITraceId(tracer, ctx)
  tracer.setId(traceId)
  return tracer
}

/**
 * A ZipkinJS 'TraceId' can be created directly from the function invocation
 * context.
 *
 * @param {*} ctx The function invocation context.
 * @returns       A ZipkinJS 'TraceId' created from the invocation context.
 */
function createOCITraceId (tracer, ctx) {
  const tracingCxt = ctx.tracingContext
  if (tracingCxt.traceId && tracingCxt.spanId) {
    return new TraceId({
      traceId: tracingCxt.traceId,
      spanId: tracingCxt.spanId,
      sampled: new option.Some(tracingCxt.sampled),
      debug: new option.Some(tracingCxt.debug),
      shared: false
    })
  } else {
    return tracer.createRootId(
      new option.Some(tracingCxt.sampled),
      new option.Some(tracingCxt.debug)
    )
  }
}

/**
 * A ZipkinJS 'TraceId' can be crated directly from the function invocation
 * context.
 *
 * This configurations will automatically add the function meta-data as APM
 * dimensions to each trace. Function environment variable and other dimensions
 * could also be added.
 *
 * @param {*} ctx The function invocation context.
 * @returns       A map of key-value pairs, that will be added as APM
 *                dimensions to the traces.
 */
function createOCITags (ctx) {
  return {
    appID: ctx.appID,
    appName: ctx.appName,
    fnID: ctx.fnID,
    fnName: ctx.fnName
  }
}

/**
 * A ZipkinJS 'Sampler' can be created directly from the function invocation
 * context.
 *
 * This configuration will only create a trace if the function is configured
 * for tracing.
 *
 * @param {*} ctx The function invocation context.
 * @returns       A ZipkinJS 'TraceId' created from the invocation context.
 */
function createOCISampler (ctx) {
  return new sampler.Sampler((traceId) => ctx.tracingContext.isEnabled)
}