MCP server secrets rotation security

Why static credential loading breaks rotation

The standard 12-factor pattern reads credentials from environment variables at startup: const DB_PASSWORD = process.env.DB_PASSWORD. This binds the credential value to the process lifetime. Rotating the credential in the secrets manager has no effect until the process restarts. For long-running MCP servers, this means:

• After a credential rotation in AWS Secrets Manager or Vault, the old credential is still in memory and in active use until restart
• If the old credential is revoked immediately (e.g., after a breach), the MCP server starts returning errors on all tool calls that use that credential
• Rolling restart to pick up the new credential causes a gap window where some instances have the old credential and some have the new one — and the upstream may only accept one

Pattern 1: dual-key overlap window

The dual-key pattern requires the upstream system (database, API provider) to accept two credentials simultaneously during a transition window. Most credential systems support this: AWS IAM access keys (you can have two active per user), database passwords (many support multiple active credentials), API keys for signing (keep old key active for N days after issuing new).

The rotation sequence:

1. Generate new credential and store it in the secrets manager alongside the old one
2. Gradually reload credentials in running processes (via SIGHUP or polling — see Pattern 3 below)
3. Once all processes have switched to the new credential, revoke the old one

// secrets-loader.ts — polls AWS Secrets Manager every 5 minutes for updated credentials
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';

const client = new SecretsManagerClient({ region: process.env.AWS_REGION });

interface SecretBundle {
  apiKey: string;
  dbPassword: string;
  jwtSecret: string;
  fetchedAt: number;
}

let currentSecrets: SecretBundle | null = null;

async function fetchSecrets(): Promise {
  const response = await client.send(new GetSecretValueCommand({
    SecretId: process.env.SECRET_ARN,
    VersionStage: 'AWSCURRENT',  // always fetch the current active version
  }));
  const parsed = JSON.parse(response.SecretString!);
  return { ...parsed, fetchedAt: Date.now() };
}

async function getSecrets(): Promise {
  if (!currentSecrets || Date.now() - currentSecrets.fetchedAt > 5 * 60 * 1000) {
    currentSecrets = await fetchSecrets();
  }
  return currentSecrets;
}

// Usage in a tool handler:
server.tool('query_database', async (req) => {
  const { dbPassword } = await getSecrets();
  const pool = await createPool({ password: dbPassword });
  // ...
});

Pattern 2: version-tagged secrets with AWSCURRENT / AWSPREVIOUS

AWS Secrets Manager's rotation model maintains two versions simultaneously: AWSCURRENT (the active credential) and AWSPREVIOUS (the previous credential). During rotation, the service creates a new version, tests it, and then flips the AWSCURRENT label to the new version — the old credential moves to AWSPREVIOUS and remains valid during the overlap window.

For MCP servers with JWT signing keys, maintaining the previous signing key as a valid verification key allows in-flight JWTs signed with the old key to remain valid during the transition:

// JWT verification supporting both current and previous signing key
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
import jwt from 'jsonwebtoken';

const smClient = new SecretsManagerClient({ region: 'us-east-1' });

async function getJwtSecrets() {
  const [current, previous] = await Promise.all([
    smClient.send(new GetSecretValueCommand({ SecretId: 'mcp/jwt-secret', VersionStage: 'AWSCURRENT' })),
    smClient.send(new GetSecretValueCommand({ SecretId: 'mcp/jwt-secret', VersionStage: 'AWSPREVIOUS' }))
      .catch(() => null),  // AWSPREVIOUS may not exist before first rotation
  ]);
  return {
    current: JSON.parse(current.SecretString!).signingKey,
    previous: previous ? JSON.parse(previous.SecretString!).signingKey : null,
  };
}

async function verifyToken(token: string) {
  const { current, previous } = await getJwtSecrets();

  // Try current key first
  try {
    return jwt.verify(token, current);
  } catch (e) {
    if (!previous) throw e;
    // Fall back to previous key (tokens signed before rotation)
    try {
      return jwt.verify(token, previous);
    } catch {
      throw e;  // throw the original error from the current key attempt
    }
  }
}

Pattern 3: in-process credential refresh on SIGHUP

Unix convention: SIGHUP signals a process to reload its configuration without restarting. Implementing a SIGHUP handler that re-fetches credentials from the secrets manager lets you trigger a coordinated refresh across all instances by sending SIGHUP to each process — no downtime, no restart, no rolling deployment needed for credential-only changes:

// In-process credential refresh on SIGHUP
let cachedSecrets: SecretBundle | null = null;

async function reloadSecrets() {
  const fresh = await fetchSecrets();
  // Validate the new credentials before dropping the old ones
  const dbOk = await testDatabaseCredential(fresh.dbPassword);
  const apiOk = await testApiKey(fresh.apiKey);

  if (!dbOk || !apiOk) {
    logger.error({ event: 'secrets_reload_failed', dbOk, apiOk });
    return;  // keep current credentials — don't swap in invalid ones
  }

  cachedSecrets = fresh;
  logger.info({ event: 'secrets_reloaded', fetchedAt: fresh.fetchedAt });
}

// Register SIGHUP handler at startup
process.on('SIGHUP', () => {
  logger.info('SIGHUP received — reloading credentials');
  reloadSecrets().catch(err => logger.error({ event: 'secrets_reload_error', err }));
});

// Kubernetes: trigger credential refresh across all pods
// kubectl rollout restart is NOT needed — instead:
// kubectl exec -it  -- kill -HUP 1
// Or: set up an AWS Lambda triggered by Secrets Manager rotation events that sends SIGHUP via SSM Run Command

SkillAudit findings for secrets rotation

Related: Secrets management in depth · OAuth token rotation · Supply chain risk