OpenTelemetry Tracing with Oracle Cloud and the Micronaut Framework

Use Oracle Cloud to investigate the behavior of your Micronaut applications.

Authors: Nemanja Mikic, John Shingler

Micronaut Version: 4.6.3

1. Getting Started

In this guide, we will create a Micronaut application written in Java.

In this guide, you will discover how simple it is to add tracing to a Micronaut application.

Tracing allows you to track service requests in a single application or a distributed one. Trace data shows the path, time spent in each section (called a span), and other information collected along the way. Tracing gives you observability into what is causing bottlenecks and failures.

2. What you will need

To complete this guide, you will need the following:

Some time on your hands
A decent text editor or IDE (e.g. IntelliJ IDEA)
JDK 21 or greater installed with JAVA_HOME configured appropriately

An Oracle Cloud account (create a free trial account at signup.oraclecloud.com)

3. Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can go right to the completed example.

Download and unzip the source

4. Writing the Application

Create an application using the Micronaut Command Line Interface or with Micronaut Launch.

mn create-app example.micronaut.micronautguide \
    --features=yaml,tracing-opentelemetry-zipkin,http-client,yaml,tracing-opentelemetry-http \
    --build=gradle \
    --lang=java \
    --test=junit

If you don’t specify the --build argument, Gradle with the Kotlin DSL is used as the build tool.
If you don’t specify the --lang argument, Java is used as the language.
If you don’t specify the --test argument, JUnit is used for Java and Kotlin, and Spock is used for Groovy.

The previous command creates a Micronaut application with the default package example.micronaut in a directory named micronautguide.

If you use Micronaut Launch, select Micronaut Application as application type and add yaml, tracing-opentelemetry-zipkin, http-client, yaml, and tracing-opentelemetry-http features.

If you have an existing Micronaut application and want to add the functionality described here, you can view the dependency and configuration changes from the specified features, and apply those changes to your application.

Diff

Setup IntelliJ IDEA to develop Micronaut Applications.

4.1. OpenTelemetry

The Micronaut framework uses OpenTelemetry to generate and export tracing data.

OpenTelemetry provides two annotations: one to create a span and another to include additional information to the span.

@WithSpan: Used on methods to create a new span; defaults to the method name, but a unique name may be assigned instead.
@SpanAttribute: Used on method parameters to assign a value to a span; defaults to the parameter name, but a unique name may be assigned instead.

@WithSpan and @SpanAttribute can be used only on non-private methods.

If these annotations are not enough, or if you want to add tracing to a private method, the Micronaut framework’s tracing integration registers a io.opentelemetry.api.trace.Tracer bean, which exposes the OpenTelemetry API and can be dependency-injected as needed.

The following `io.micronaut.tracing.annotation`s are available if you prefer to use them or if you are working on an existing Micronaut application that already uses them.

@NewSpan : Identical to @WithSpan, it is used on methods to create a new span; defaults to the method name, but a unique name may be assigned instead.
@ContinueSpan : Used on methods to continue an existing span; primarily used in conjunction with @SpanTag.
@SpanTag : Similar to @SpanAttribute, it is used on method parameters to assign a value to a span; defaults to the parameter name, but a unique name may be assigned instead. To use the @SpanTag on a method argument, the method must be annotated with either @NewSpan or @ContinueSpan.

4.2. Inventory Service

src/main/java/example/micronaut/InventoryService.java

package example.micronaut;

import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import jakarta.inject.Singleton;

import java.util.Collection;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

@Singleton
public class InventoryService {

    private final Tracer tracer;
    private final WarehouseClient warehouse;
    private final Map<String, Integer> inventory = new ConcurrentHashMap<>();

    private static final String storeName = "my_store";

    public InventoryService(Tracer tracer, WarehouseClient warehouse) { (1)
        this.tracer    = tracer;
        this.warehouse = warehouse;

        inventory.put("laptop",  4);
        inventory.put("desktop", 2);
        inventory.put("monitor", 11);
    }

    public Collection<String> getProductNames() {
        return inventory.keySet();
    }

    @WithSpan("stock-counts") (2)
    public Map<String, Integer> getStockCounts(@SpanAttribute("inventory.item") String item) { (3)
        HashMap<String, Integer> counts = new HashMap<>();
        if(inventory.containsKey(item)) {
            int count = inventory.get(item);
            counts.put("store", count);

            if(count < 10) {
                counts.put("warehouse", inWarehouse(storeName, item));
            }
        }

        return counts;
    }

    private int inWarehouse(String store, String item) {
        Span.current().setAttribute("inventory.store-name", store); (4)

        return warehouse.getItemCount(store, getUPC(item));
    }

    public void order(String item, int count) {
        orderFromWarehouse(item, count);
        if(inventory.containsKey(item)) {
            count += inventory.get(item);
        }
        inventory.put(item, count);
    }

    private void orderFromWarehouse(String item, int count) {
        Span span = tracer.spanBuilder("warehouse-order") (5)
                          .setAttribute("item", item)
                          .setAttribute("count", count)
                          .startSpan();

        Map<String, Object> json = new HashMap<>(4);
        json.put("store", storeName);
        json.put("product", item);
        json.put("amount", count);
        json.put("upc", getUPC(item));

        warehouse.order(json);

        span.end();
    }

    private int getUPC(String item) {
        return Math.abs(item.hashCode());
    }
}

1	Inject Tracing bean into class
2	Creates a new span called "stock-counts"
3	Adds a label, or tag, called "inventory.item" that will contain the value contained in `item`
4	Same as `@SpanAttribute("inventory.store-name")`
5	Same as `@WithSpan("warehouse-order")` with `@SpanAttributes` for item and count

4.3. Store Controller

This class demonstrates use of the io.micronaut.tracing.annotation instead of the OpenTelemetry annotations.

If you have the following dependency declared, all HTTP Server methods (those annotated with @Get, @Post, etc.) will create spans automatically.

build.gradle

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-http")

src/main/java/example/micronaut/StoreController.java

package example.micronaut;

import io.micronaut.http.HttpStatus;
import io.micronaut.http.annotation.Controller;
import io.micronaut.http.annotation.Get;
import io.micronaut.http.annotation.Post;
import io.micronaut.http.annotation.Status;
import io.micronaut.scheduling.TaskExecutors;
import io.micronaut.scheduling.annotation.ExecuteOn;
import io.micronaut.tracing.annotation.ContinueSpan;
import io.micronaut.tracing.annotation.NewSpan;
import io.micronaut.tracing.annotation.SpanTag;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

@ExecuteOn(TaskExecutors.BLOCKING)
@Controller("/store")
public class StoreController {
    private final InventoryService inventory;

    public StoreController(InventoryService inventory) {
        this.inventory = inventory;
    }

    @Post("/order")
    @Status(HttpStatus.CREATED)
    @NewSpan("store.order") (1)
    public void order(@SpanTag("order.item") String item, @SpanTag int count) { (2)
        inventory.order(item, count);
    }

    @Get("/inventory") (3)
    public List<Map<String, Object>> getInventory() {
        List<Map<String, Object>> currentInventory = new ArrayList<>();
        for (String product: inventory.getProductNames()) {
            currentInventory.add(getInventory(product));
        }
        return currentInventory;
    }

    @Get("/inventory/{item}")
    @ContinueSpan (4)
    public Map<String, Object> getInventory(@SpanTag("item") String item) { (5)
        Map<String, Object> counts = new HashMap<>(inventory.getStockCounts(item));
        if(counts.isEmpty()) {
            counts.put("note", "Not available at store");
        }

        counts.put("item", item);

        return counts;
    }
}

1	Equivalent to `@WithSpan("store.order")`
2	Same as @SpanAttribute("order.item")`and `@SpanAttribute
3	Span created automatically if `micronaut-tracing-opentelemetry-http` is declared
4	Required for `@SpanTag`
5	Tag is only added to the span if `micronaut-tracing-opentelemetry-http` is declared; otherwise ignored

4.4. Warehouse Client

You can also mix OpenTelemetry and Micronaut Tracing annotations in the same class.

If you have the following dependency declared, all HTTP Client methods (those annotated with @Get, @Post, etc.) will create spans automatically.

build.gradle

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-http")

src/main/java/example/micronaut/WarehouseClient.java

package example.micronaut;

import io.micronaut.http.annotation.Get;
import io.micronaut.http.annotation.Post;
import io.micronaut.http.annotation.QueryValue;
import io.micronaut.http.client.annotation.Client;
import io.micronaut.tracing.annotation.ContinueSpan;
import io.micronaut.tracing.annotation.SpanTag;
import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.instrumentation.annotations.WithSpan;

import java.util.Map;

@Client("/warehouse") (1)
public interface WarehouseClient {

    @Post("/order")
    @WithSpan
    void order(@SpanTag("warehouse.order") Map<String, ?> json);

    @Get("/count")
    @ContinueSpan
    int getItemCount(@QueryValue String store, @SpanAttribute @QueryValue int upc);

}

1	Some external service without tracing

4.5. Warehouse Controller

The WarehouseController class represents external service that will be called by WarehouseClient.

src/main/java/example/micronaut/WarehouseController.java

package example.micronaut;

import io.micronaut.http.HttpResponse;
import io.micronaut.http.annotation.Controller;
import io.micronaut.http.annotation.Get;
import io.micronaut.http.annotation.Post;
import io.micronaut.scheduling.TaskExecutors;
import io.micronaut.scheduling.annotation.ExecuteOn;

import java.util.Random;

@ExecuteOn(TaskExecutors.BLOCKING) (1)
@Controller("/warehouse") (2)
public class WarehouseController {

    @Get("/count") (3)
    public HttpResponse getItemCount() {
        return HttpResponse.ok(new Random().nextInt(11));
    }

    @Post("/order") (4)
    public HttpResponse order() {
        try {
            //To simulate an external process taking time
            Thread.sleep(500);
        } catch (InterruptedException e) {
        }

        return HttpResponse.accepted();
    }
}

1	It is critical that any blocking I/O operations (such as fetching the data from the database) are offloaded to a separate thread pool that does not block the Event loop.
2	The class is defined as a controller with the @Controller annotation mapped to the path `/warehouse`.
3	The @Get annotation maps the `getItemCount` method to an HTTP GET request on `/warehouse/count`.
4	The @Get annotation maps the `order` method to an HTTP GET request on `/warehouse/order`.

4.6. Create APM Domain

Open the Oracle Cloud Menu and click "Observability & Management", and then "Administration" under "Application Performance…":

Click "Create APM Domain":

Name your domain, choose a compartment, and enter a description.

Once the domain is created, view the domain details. Here you’ll need to grab a few values, so copy the data upload endpoint (#1) and public key (#2).

Now we have what we need to construct a URL to plug in to our application config files. The Collector URL format requires us to construct a URL by using the data upload endpoint as our base URL and generating the path based on some choices, including values from our private or public key. The format is documented here. Once we’ve constructed the URL path, we can add it to our application.yml config.

4.7. Configure Tracer

Use Micronaut CLI or Launch to create your application. You will see that the necessary OpenTelemetry configuration are automatically added to your application.yml file. You will have to change the value of the zipkin endpoint configuration variable.

Since Micronaut Framework 4.0, to use YAML configuration, you have to add the YAML dependency.

src/main/resources/application.yml

otel:
  traces:
    exporter: zipkin
  exporter:
    zipkin:
      endpoint: https://[redacted].apm-agt.us-phoenix-1.oci.oraclecloud.com/20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=[public key] (1)

1	The zipkin exporter URL mentioned in the previous step

5. Run the Application

The application can be deployed to Oracle Cloud.

Traces can be sent to the Oracle Cloud APM Trace Explorer outside the Oracle Cloud.

This allows us to run the application locally and see the traces in the Oracle Cloud APM Trace Explorer.

To run the application, use the ./gradlew run command, which starts the application on port 8080.

6. Traces

Open the Oracle APM Tracing Explorer. Once the page is loaded, select the compartment that you selected in above step (#1), choose the APM domain that you created (#2), and run the query (#3).

If your traces aren’t displayed yet, give it a moment. It takes a few seconds for the traces to show up in the explorer. If no spans show up, run the query again by pressing the "Run" button.

6.1. Get Item Counts

curl http://localhost:8080/store/inventory/laptop

Each span is represented by a blue bar.

6.2. Order Item

curl -X "POST" "http://localhost:8080/store/order" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{"item":"laptop", "count":5}'

Selecting a different span will show you the labels (a.k.a. attributes/tags) and other details of the span.

6.3. Get Inventory

curl http://localhost:8080/store/inventory

Looking at the trace, we can conclude that retrieving the items sequentially might not be the best design choice.

7. Next Steps

8. License

All guides are released with an Apache license 2.0 license for the code and a Creative Commons Attribution 4.0 license for the writing and media (images…).

Table of Contents