MCP server OpenTelemetry security: distributed tracing for tool call observability

Why distributed tracing matters for MCP security

A single MCP server tool call in a multi-agent system may be the result of a long chain: orchestrator agent → sub-agent → tool call. Without distributed tracing, each server sees only its own call — it cannot tell whether the call came from a legitimate orchestrator or from a prompt injection in a document the agent processed three steps earlier. With W3C trace context propagation, the MCP server can see the full span chain: which agent initiated the root trace, how many steps have elapsed, and whether the current tool call is consistent with the root instruction.

Beyond attribution, trace data enables anomaly detection at a level that local tool-call logs cannot: patterns like "tool X called 40 times within 5 minutes by the same root trace ID" or "tool Y called with a different argument type than every other call in this trace" are visible in trace data but not in isolated call logs.

Pattern 1: security-aware span creation for tool calls

Create a span for each tool invocation with security-relevant attributes. Use OTel semantic conventions for RPC operations, then add MCP-specific security attributes:

import { trace, SpanStatusCode, context } from '@opentelemetry/api';

const tracer = trace.getTracer('mcp-server', '1.0.0');

async function instrumentedToolCall(
  toolName: string,
  callerId: string,
  callerRole: string,
  authzDecision: 'permit' | 'deny',
  handler: () => Promise<ToolResult>
): Promise<ToolResult> {
  return tracer.startActiveSpan(`mcp.tool/${toolName}`, async (span) => {
    span.setAttributes({
      // Standard RPC attributes
      'rpc.system': 'mcp',
      'rpc.method': toolName,
      // Security attributes
      'mcp.caller.id': callerId,
      'mcp.caller.role': callerRole,
      'mcp.authz.decision': authzDecision,
      'mcp.tool.name': toolName,
    });

    if (authzDecision === 'deny') {
      span.setStatus({ code: SpanStatusCode.ERROR, message: 'authorization denied' });
      span.setAttribute('mcp.security.event', 'authz_denied');
      span.end();
      throw new McpError(ErrorCode.InvalidRequest, 'Unauthorized');
    }

    try {
      const result = await handler();
      span.setStatus({ code: SpanStatusCode.OK });
      span.setAttribute('mcp.result.type', result.content[0]?.type ?? 'empty');
      return result;
    } catch (e) {
      span.setStatus({ code: SpanStatusCode.ERROR });
      span.setAttribute('mcp.error.type', e instanceof McpError ? 'mcp_error' : 'internal');
      throw e;
    } finally {
      span.end();
    }
  });
}

The span captures the authorization decision before the handler executes. Denied requests appear in trace data as error spans with the mcp.security.event: authz_denied attribute — queryable in any OTEL backend (Jaeger, Grafana Tempo, DataDog APM) to detect authorization probing patterns.

Pattern 2: W3C trace context for cross-agent correlation

Propagate W3C trace context through MCP tool calls to correlate invocations across agent boundaries. The MCP client should include the traceparent header; the server should extract and continue the trace:

import { propagation, ROOT_CONTEXT } from '@opentelemetry/api';
import { W3CTraceContextPropagator } from '@opentelemetry/core';

// Install W3C propagator at server startup
propagation.setGlobalPropagator(new W3CTraceContextPropagator());

// In MCP request handler — extract trace context from metadata
server.setRequestHandler(CallToolRequestSchema, async (req) => {
  const carrier = req.params._meta ?? {};
  const parentContext = propagation.extract(ROOT_CONTEXT, carrier);

  return context.with(parentContext, async () => {
    // Spans created here are children of the caller's trace
    return tracer.startActiveSpan(`mcp.tool/${req.params.name}`, async (span) => {
      // Attribute: depth in the agent call chain
      const traceState = trace.getSpan(context.active())?.spanContext().traceState;
      const agentDepth = parseInt(traceState?.get('agent.depth') ?? '0');
      span.setAttribute('mcp.agent.depth', agentDepth);

      // Flag deep call chains as anomalous (prompt injection risk)
      if (agentDepth > 5) {
        span.setAttribute('mcp.security.event', 'deep_agent_chain');
        span.setAttribute('mcp.security.anomaly', true);
      }
      // ... handler
    });
  });
});

The agent.depth trace state attribute tracks how many agent hops have occurred since the original user instruction. An agent chain depth greater than 5 is a strong signal of an agentic loop or prompt injection cascade — the kind of pattern that multi-agent MCP security describes as a key threat in orchestrated pipelines.

Pattern 3: span attribute scrubbing before export

Tool arguments may contain sensitive data: file contents, user queries, credentials passed as configuration. Span attributes that include tool argument values must be scrubbed before export to any third-party trace backend. Use an OTel processor to redact sensitive attributes:

import { SimpleSpanProcessor, SpanExporter } from '@opentelemetry/sdk-trace-base';
import { ReadableSpan, SpanProcessor } from '@opentelemetry/sdk-trace-base';

const REDACT_ATTRIBUTE_PATTERNS = [
  /^mcp\.args\./,         // Tool argument values
  /password/i,
  /secret/i,
  /token/i,
  /api.?key/i,
  /credential/i,
];

class ScrubProcessor implements SpanProcessor {
  private next: SpanProcessor;
  constructor(exporter: SpanExporter) {
    this.next = new SimpleSpanProcessor(exporter);
  }

  onStart(span: any) { /* no-op */ }

  onEnd(span: ReadableSpan) {
    const scrubbed = { ...span.attributes };
    for (const key of Object.keys(scrubbed)) {
      if (REDACT_ATTRIBUTE_PATTERNS.some(p => p.test(key))) {
        scrubbed[key] = '[REDACTED]';
      }
    }
    this.next.onEnd({ ...span, attributes: scrubbed });
  }

  shutdown() { return this.next.shutdown(); }
  forceFlush() { return this.next.forceFlush(); }
}

The scrub processor runs before the exporter. Tool argument attributes are redacted — the trace shows that an argument was passed and its type, but not its value. Authorization and security event attributes (mcp.authz.decision, mcp.security.event) are preserved because they contain no sensitive data and are critical for security analysis.

Pattern 4: trace-based anomaly detection with span queries

Export spans to a backend that supports structured queries (Grafana Tempo + Loki, DataDog APM, or a local Jaeger instance) and define alerting rules on security-relevant span attributes:

# Grafana alert: tool call burst from single root trace
# Query: count spans where mcp.tool.name matches and root trace ID is the same
{
  "expr": "count(mcp_tool_calls_total{root_trace_id=~\".*\"}) by (root_trace_id) > 50",
  "for": "2m",
  "labels": { "severity": "warning", "type": "agentic_loop" },
  "annotations": { "summary": "Possible agentic loop: >50 tool calls from single trace" }
}

# Alert: repeated authz_denied events from same caller
{
  "expr": "sum(rate(mcp_authz_denied_total[5m])) by (caller_id) > 5",
  "for": "1m",
  "labels": { "severity": "warning", "type": "authz_probing" }
}

# Alert: deep agent chain detected
{
  "expr": "count(mcp_tool_calls_total{agent_depth=\"6\"}) > 0",
  "labels": { "severity": "info", "type": "deep_chain" }
}

These alerts fire on the same trace data that your debugging workflow uses — no separate security logging pipeline required. The spans serve double duty: debugging and security monitoring. This is the key advantage of OTel over a separate audit log stream: one instrumentation point, multiple consumers.

Pattern 5: trace data access control

Trace backends must be access-controlled. Spans containing caller IDs, authorization decisions, and (before scrubbing) tool argument shapes are sensitive operational data. Treat your trace backend with the same access control as your application logs:

# Jaeger / Grafana Tempo: restrict trace query access
# - Developers: read access to spans from their own services in non-prod
# - Security team: read access to all spans, all environments
# - Ops on-call: read access to spans in production during incidents
# - External trace backends (DataDog, Honeycomb): receive scrubbed spans only

# In Grafana: enforce data source permissions per team
# grafana.ini:
[auth]
disable_signout_menu = false
[security]
allow_embedding = false
cookie_secure = true

# OTLP exporter: use different endpoints for scrubbed vs. full spans
# Full spans (internal, never leaves your VPC) → local Jaeger
# Scrubbed spans → DataDog or external SIEM

Use two export targets: a local trace backend (Jaeger, Grafana Tempo running in your VPC) that receives full spans for internal debugging, and an external backend (DataDog APM, Honeycomb) that receives only scrubbed spans via the ScrubProcessor above. The split export ensures that sensitive span attributes never leave your network.

What SkillAudit checks for OpenTelemetry security

SkillAudit's observability security scan checks for OTel-specific patterns in MCP server codebases:

• No span instrumentation: tool handlers with no OTel span creation — no observability into tool call patterns or authorization decisions
• Sensitive args in span attributes: span.setAttribute calls with argument values from tool input without a scrubbing step before export
• No trace context extraction: MCP servers operating in multi-agent environments without W3C trace context propagation, making cross-agent correlation impossible
• Unrestricted trace backend access: OTLP endpoints accessible without authentication, or local Jaeger/Tempo instances exposed on public network interfaces

OTel findings map to the Maintenance and Security sub-scores. Missing span instrumentation is a MEDIUM finding (reduces visibility into security events). Sensitive data in exported spans is a HIGH finding. For the broader observability security picture, see the security logging and observability post. For the multi-agent security context where distributed tracing is most critical, see multi-agent MCP security.