MCP server observability security — trace span data leakage, metric cardinality attacks, trace ID injection

Trace span data leakage

The most common observability security failure is instrumenting tool calls by including the raw args object in the span's attributes. On its face, this seems useful for debugging — you can see exactly what the LLM called each tool with. The problem is that the observability backend's access control is usually far more permissive than the application's:

// Dangerous: raw tool args in span attributes
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('mcp-server');

server.tool('queryCustomerRecords', {
  handler: async (args, { session }) => {
    const span = tracer.startSpan('tool.queryCustomerRecords');
    span.setAttributes({
      'tool.args': JSON.stringify(args),  // ← may contain email, SSN, API keys
      'session.user_id': session.userId,
    });
    // ...
    span.end();
  }
});

If args includes a query string like customer_email: "alice@example.com", that PII is now in your Jaeger/Tempo/Honeycomb traces — accessible to everyone on the engineering team with observability access, regardless of whether they have access to the production database. If a user passes an API key as a tool argument (a misconfiguration, but it happens), that credential is now in your trace storage.

Sanitizing span attributes

The fix is to create a summary function that captures structural information about the arguments without including their values:

function summarizeArgs(args: Record<string, unknown>): Record<string, string> {
  const summary: Record<string, string> = {};
  for (const [key, value] of Object.entries(args)) {
    if (typeof value === 'string') {
      // Record length and type, not value
      summary[`arg.${key}.type`] = 'string';
      summary[`arg.${key}.length`] = String(value.length);
    } else if (typeof value === 'number') {
      // Numbers are generally safe — they don't carry PII
      summary[`arg.${key}`] = String(value);
    } else if (Array.isArray(value)) {
      summary[`arg.${key}.length`] = String(value.length);
      summary[`arg.${key}.type`] = 'array';
    } else {
      summary[`arg.${key}.type`] = typeof value;
    }
  }
  return summary;
}

// Safe: structural summary only
span.setAttributes({
  ...summarizeArgs(args),
  'session.user_id': session.userId,
  'tool.name': 'queryCustomerRecords',
});

This gives you debuggable traces (you can see that a string argument was passed and how long it was) without leaking the actual values. For values you know are safe (non-PII enums, row counts, latency measurements), you can include them directly.

Metric cardinality attacks

The second observability vulnerability is using LLM-controlled input as metric label values. Prometheus, Datadog, and similar systems store one time series per unique combination of label values. If a label value can be any arbitrary string, an attacker who can influence the LLM's tool calls can create a cardinality explosion:

// Dangerous: LLM-controlled value as metric label
toolCallCounter.add(1, {
  tool_name: toolName,
  query_type: args.queryType,  // ← LLM-controlled, unbounded cardinality
  user_id: session.userId,     // ← high cardinality if many users
});

// Attack: induce LLM to call tool with thousands of unique queryType values
// → thousands of new time series created per minute
// → metric backend OOMs or throttles all other metrics

The fix is to allowlist all label values before emitting them as metrics:

const VALID_QUERY_TYPES = new Set(['search', 'lookup', 'aggregate', 'export']);

function sanitizeLabel(value: string, allowlist: Set<string>): string {
  return allowlist.has(value) ? value : 'unknown';
}

toolCallCounter.add(1, {
  tool_name: toolName,
  query_type: sanitizeLabel(args.queryType, VALID_QUERY_TYPES),
  // user_id intentionally omitted — use session_type (authenticated/anonymous) instead
  session_type: session.authenticated ? 'authenticated' : 'anonymous',
});

Trace ID injection and trace forgery

OpenTelemetry propagates trace context through HTTP headers. If an MCP server makes outbound HTTP calls using a trace context derived from the incoming tool call arguments — rather than from the authenticated session context — an attacker can inject a traceparent header that associates the server's outbound calls with an attacker-controlled trace:

// Dangerous: trace context from LLM-controlled headers
server.tool('fetchExternalData', {
  handler: async (args) => {
    // args.headers is LLM-supplied — can include W3C traceparent header
    const response = await fetch(args.url, {
      headers: {
        ...args.headers,  // ← attacker can inject traceparent
        'Authorization': `Bearer ${config.apiKey}`
      }
    });
    // ...
  }
});

If the outbound request's trace is under attacker control, the attacker can forge the appearance of legitimate spans in their trace, potentially misleading incident response teams or causing authorization decisions made on trace context to be incorrect.

The fix: never propagate trace context from tool arguments. Outbound HTTP calls should propagate the server's own authenticated trace context, created from the session auth layer, not from any LLM-supplied header value.

// Safe: trace context from session, not from tool arguments
server.tool('fetchExternalData', {
  handler: async (args, { session }) => {
    const span = tracer.startSpan('outbound.fetchExternalData', {
      // Context propagated from the authenticated session span
    });
    // propagation.inject adds the server's own traceparent — no LLM input
    const headers: Record<string, string> = {};
    propagation.inject(context.active(), headers);

    const response = await fetch(args.url, {
      headers: {
        ...headers,
        'Authorization': `Bearer ${config.apiKey}`
      }
    });
    span.end();
    return response.json();
  }
});

What SkillAudit checks

SkillAudit's static analysis detects the following observability security patterns in MCP server code:

• Direct serialization of args or tool argument objects into span attributes
• Metric label values derived from tool argument fields without an allowlist
• Outbound HTTP calls that spread-copy incoming header objects (including potential trace context propagation from LLM input)
• Log statements that interpolate raw tool argument values into structured log fields

Observability misconfigurations typically contribute to Credential exposure and Security sub-score findings. Run a free audit at skillaudit.dev to see how your MCP server scores.