Custom Export

Short guide on how to setup exporting telemetry to a custom endpoint.

Overview

Prover Nodes can choose to export to their own OTLP endpoints. This can be done through a combination of an OtelCol docker container, and an additional receiver of your choice such as a complete integrated one like Grafana Alloy, but setting up individual receivers such as Prometheus for metrics, is also possible.

Here's how an example services will look like:

Service	Purpose	Port
Otel Collector	gRPC OTLP Receiver	4317
Datadog Agent	OtelCollector Export Endpoint	4320
Grafana Alloy	OtelCollector Export Endpoint	4321

Previously, the prover node binary would simply export to DD agent on port 4317. With this setup, the binary will export to Otel Collector which in turn will dual-export to DD and Alloy.

Currently, we only support metrics, traces and logs through gRPC OTLP.

Datadog Agent

First, we'll setup the DD agent, which is what Fermah uses to receive full visibility on prover nodes. Usually, this will already be running as part of the Installation, but we'll modify the port to instead run on 4320. Here's just the modified ports:

docker-compose.yml

    ports:
      - "4320:4317"

Grafana Alloy

We'll need two files for setting up Alloy, but the config file depends a lot on each prover node's setup, because it defines the final export endpoint for each (metrics, traces, logs), for Grafana this is Prometheus, Tempo and Loki respectively.

First, the relevant docker compose file:

docker-compose.yml

services:
  alloy:
    restart: always
    image: grafana/alloy:latest
    ports:
      - "4321:4317"
      - "12345:12345"
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy
    command: run --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy
    environment:
      - GRAFANA_CLOUD_API_KEY=${GRAFANA_CLOUD_API_KEY}

And here's the configuration file:

config.alloy

logging {
  level  = "info"
  format = "logfmt"
}

otelcol.receiver.otlp "default" {
	grpc { }

	output {
		metrics = [otelcol.processor.resourcedetection.default.input]
		logs    = [otelcol.processor.resourcedetection.default.input]
		traces  = [otelcol.processor.resourcedetection.default.input]
	}
}

otelcol.processor.resourcedetection "default" {
	detectors = ["env", "system", "ec2", "gcp", "docker"]

	system {
		hostname_sources = ["os"]
	}

	output {
		metrics = [otelcol.processor.transform.add_resource_attributes_as_metric_attributes.input]
		logs    = [otelcol.processor.batch.default.input]
		traces  = [
			otelcol.processor.batch.default.input,
			otelcol.connector.host_info.default.input,
		]
	}
}

otelcol.connector.host_info "default" {
	host_identifiers = ["host.name"]

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.transform "add_resource_attributes_as_metric_attributes" {
	error_mode = "ignore"

	metric_statements {
		context    = "datapoint"
		statements = [
			"set(attributes[\"deployment.environment\"], resource.attributes[\"deployment.environment\"])",
			"set(attributes[\"service.version\"], resource.attributes[\"service.version\"])",
		]
	}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.batch "default" {
	output {
		metrics = [otelcol.exporter.prometheus.metrics_service.input]
		logs    = [otelcol.exporter.loki.logs_service.input]
		traces  = [otelcol.exporter.otlp.grafana_cloud_tempo.input]
	}
}

otelcol.exporter.prometheus "metrics_service" {
	add_metric_suffixes = false
	forward_to          = [prometheus.remote_write.default.receiver]
}

otelcol.exporter.loki "logs_service" {
	forward_to          = [loki.write.default.receiver]
}

otelcol.exporter.otlp "grafana_cloud_tempo" {
	client {
		endpoint = "tempo-prod-04-prod-us-east-0.grafana.net:443"
		auth     = otelcol.auth.basic.grafana_cloud_tempo.handler
	}
}

otelcol.auth.basic "grafana_cloud_tempo" {
	username = "863733"
	password = env("GRAFANA_CLOUD_API_KEY")
}

loki.write "default" {
    endpoint {
        url = "https://logs-prod-006.grafana.net/loki/api/v1/push"
        basic_auth {
            username = "869417"
            password = env("GRAFANA_CLOUD_API_KEY")
        }
    }
}

prometheus.remote_write "default" {
  endpoint {
    url = "https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push"
      basic_auth {
        username = "1537843"
        password = env("GRAFANA_CLOUD_API_KEY")
      }
  }
}

User needs to update each endpoint to match their use case, either through Grafana Cloud or local instance.

The above configuration file assumes the prover node is using Grafana Cloud. That means that they know each endpoint URL for Prometheus, Loki and Tempo, as well as the API key used for authentication, assuming environment variable available at: GRAFANA_CLOUD_API_KEY when running docker compose. If using local receivers for each endpoint, please refer to official alloy configuration reference. However, the only change usually needed is removing the authentication and changing the endpoint URLs.

Otel Collector

This container is the important glue that ties everything together. Otelcol endpoint will now be the target export of the prover node binary, as it will run listening on port 4317.

The otelcol container needs both a docker compose service declaration and a YML configuration file. Both will be declared in the section below.

Final Docker Compose File

We need to run everything on the same compose file, as conveniently, docker compose creates a network for each compose file, letting us refer to each service by their DNS name which is the service name declared in the file.

docker-compose.yml

configs:
  datadog_config:
    content: |
      apm_config:
        apm_non_local_traffic: true

      # Use java container support
      jmx_use_container_support: true

      otlp_config:
          metrics:
              # Add all resource attributes as tags for metrics
              resource_attributes_as_tags: true

services:
  datadog:
    image: datadog/agent:7.60.1
    environment:
     - DD_API_KEY=${DATADOG_API_KEY}
     - DD_SITE=us5.datadoghq.com
     - DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT=0.0.0.0:4317
     - DD_LOGS_ENABLED=true
     - DD_OTLP_CONFIG_LOGS_ENABLED=true
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /proc/:/host/proc/:ro
     - /sys/fs/cgroup:/host/sys/fs/cgroup:ro
     - /var/lib/docker/containers:/var/lib/docker/containers:ro
    configs:
      - source: datadog_config
        target: /etc/datadog-agent/datadog.yaml
    ports:
      - "4320:4317"
    restart: unless-stopped
    
  alloy:
    restart: unless-stopped
    image: grafana/alloy:latest
    ports:
      - "4321:4317"
      - "12345:12345"
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy
    command: run --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data /etc/alloy/config.alloy
    environment:
      - GRAFANA_CLOUD_API_KEY=${GRAFANA_CLOUD_API_KEY}
  
  otelcol:
    image: otel/opentelemetry-collector-contrib
    volumes:
      - ./otel-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - 13133:13133 # health_check extension
      - 4317:4317 # OTLP gRPC receiver

otel-config.yml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp/dd:
    endpoint: datadog:4317
    tls:
      insecure: true

  otlp/alloy:
    endpoint: alloy:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp/dd, otlp/alloy]
    traces:
      receivers: [otlp]
      exporters: [otlp/dd, otlp/alloy]
    logs:
      receivers: [otlp]
      exporters: [otlp/dd, otlp/alloy]

With this setup, you'll end up with the following files:

• docker-compose.yml

• config.alloy

• otel-config.yml

When running docker compose up -d, make sure you also specify the Datadog API key that was assigned to your prover node (prefix the command with DATADOG_API_KEY=${API_KEY} ) and any other API keys such as for Grafana Cloud.