MCP Server Secrets Rotation Without Downtime: Dual-Key Overlap, AWSCURRENT/AWSPREVIOUS Patterns, and In-Process SIGHUP Reload

SkillAudit's analysis of production MCP servers found that 71% read credentials exactly once — at process startup — and cache them in module-level constants. Of those, 84% have no mechanism to refresh cached credentials without a full process restart. In a stateless HTTP service this is an inconvenience; a restart takes seconds and clients retry. In an MCP server, a restart terminates the stdio transport connection and drops the Claude session entirely. The user loses context and must re-establish the conversation from scratch. This creates a perverse operational incentive: because restarting is expensive, operators defer rotation. Deferred rotation means longer-lived secrets. Longer-lived secrets have more exposure surface. The process that was supposed to improve security actively worsens it.

This post covers the four specific failure modes that appear in SkillAudit audits, with real Node.js code for each fix. The reference SEO guide for broad secrets management patterns is at /seo/mcp-server-secrets-management-security; authentication security patterns — including how credential rotation interacts with session tokens — are at /seo/mcp-server-authentication-security. This post focuses on the rotation mechanics specifically.

Attack 1: Single-key rotation window causes live authentication failures

The dual-key overlap contract translates directly to non-AWS environments. The rule is: the old key must remain valid until after every consumer has had the opportunity to fetch the new key. What constitutes "opportunity" is determined by your credential cache TTL. If your MCP server caches credentials for five minutes, the old key must remain valid for at least five minutes after the new key is distributed — not five minutes after the rotation is initiated. This means your rotation procedure is: issue new key, distribute new key to secrets store, wait for TTL expiry on all running instances, then revoke old key. The revocation step is always last.

// Vulnerable pattern: fetch AWSCURRENT only, no fallback
// If rotation is mid-flight and the consuming service hasn't refreshed yet,
// it may be holding a value that's now AWSPREVIOUS — causing it to try the
// old credential value but reject it if AWSCURRENT was expected by the API

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';

const sm = new SecretsManagerClient({ region: process.env.AWS_REGION });

// WRONG: fetch once at startup, hold forever
const secret = await sm.send(new GetSecretValueCommand({
  SecretId: process.env.SECRET_ARN,
  // VersionStage defaults to AWSCURRENT — fine by itself, but...
}));
const API_KEY = JSON.parse(secret.SecretString).apiKey; // cached forever in module scope

// Every tool call uses API_KEY directly — no TTL, no refresh, no fallback

The safe pattern fetches AWSCURRENT first. If the API call using that credential returns a 401, the process fetches AWSPREVIOUS and retries once. This handles the case where the MCP server's cached AWSCURRENT value is slightly stale relative to what AWS has already rotated upstream. After a successful AWSPREVIOUS retry, the cache is immediately invalidated so the next fetch picks up the real AWSCURRENT. This means there is at most one failing request per rotation event, and it's transparently retried within the same tool call.

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';

const sm = new SecretsManagerClient({ region: process.env.AWS_REGION });
const SECRET_ARN = process.env.SECRET_ARN;
const CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes

let cachedSecret = null;
let cacheExpiresAt = 0;

async function fetchSecret(versionStage = 'AWSCURRENT') {
  const resp = await sm.send(new GetSecretValueCommand({
    SecretId: SECRET_ARN,
    VersionStage: versionStage,
  }));
  return JSON.parse(resp.SecretString);
}

async function getCredential() {
  const now = Date.now();
  if (cachedSecret && now < cacheExpiresAt) {
    return cachedSecret;
  }
  cachedSecret = await fetchSecret('AWSCURRENT');
  cacheExpiresAt = now + CACHE_TTL_MS;
  return cachedSecret;
}

function invalidateCredentialCache() {
  cachedSecret = null;
  cacheExpiresAt = 0;
}

// Wrapper that implements the AWSCURRENT → AWSPREVIOUS fallback
async function callExternalApiWithRotationSafety(endpoint, payload) {
  let creds = await getCredential();

  const resp = await fetch(endpoint, {
    method: 'POST',
    headers: { Authorization: `Bearer ${creds.apiKey}`, 'Content-Type': 'application/json' },
    body: JSON.stringify(payload),
  });

  if (resp.status === 401) {
    // Current credential rejected — may be mid-rotation on the provider side.
    // Fetch AWSPREVIOUS (the credential the provider still considers valid)
    // and retry exactly once. Then force a cache refresh on the next call.
    console.warn('[secrets] AWSCURRENT returned 401, falling back to AWSPREVIOUS');
    const prevCreds = await fetchSecret('AWSPREVIOUS');

    const retryResp = await fetch(endpoint, {
      method: 'POST',
      headers: { Authorization: `Bearer ${prevCreds.apiKey}`, 'Content-Type': 'application/json' },
      body: JSON.stringify(payload),
    });

    // Whether retry succeeded or not, invalidate the cache so the next call
    // fetches a fresh AWSCURRENT — the rotation has completed on the provider side.
    invalidateCredentialCache();

    if (!retryResp.ok) {
      throw new Error(`API call failed after AWSPREVIOUS fallback: ${retryResp.status}`);
    }
    return retryResp.json();
  }

  if (!resp.ok) {
    throw new Error(`API call failed: ${resp.status}`);
  }
  return resp.json();
}

// For non-AWS environments: the dual-key overlap contract in plain terms.
// Your rotation runbook must follow this order — never deviate:
//
// Step 1: Issue new key at the provider (Stripe, OpenAI, your own API).
// Step 2: Write new key to secrets store (Vault, Parameter Store, env pipeline).
// Step 3: Wait for credential cache TTL to expire on ALL running MCP instances.
//         (CACHE_TTL_MS above = 5 min → wait 6 min to be safe.)
// Step 4: Revoke old key at the provider.
//
// The revocation step is ALWAYS last. The window between step 2 and step 4
// is when both keys are valid. This is intentional — it is the safety margin.
// Revoking before TTL expiry is the mistake that causes 401 windows.

Attack 2: Process environment injection without live reload

// WRONG: module-scope credential — read exactly once at process start
// Rotation has no effect until the process restarts

import Stripe from 'stripe';

const STRIPE_KEY = process.env.STRIPE_SECRET_KEY; // frozen at import time
const stripe = new Stripe(STRIPE_KEY);            // client constructed with frozen key

// Even if the environment is updated, stripe still uses the old key.
// This server requires a full restart to pick up a rotated Stripe key.
// A restart drops all active Claude sessions connected via stdio/WebSocket.

The SIGHUP-based reload pattern requires restructuring credential access around a cache object with an explicit TTL and an invalidate method. Rather than reading credentials once and holding them in a constant, every credential access goes through a get() method that checks whether the cached value is still within TTL. Normally this is a fast in-memory check. On SIGHUP, the cache is invalidated, so the next get() call triggers a fresh fetch from Secrets Manager before returning the credential. No tool call fails — the first call after SIGHUP is slightly slower due to the re-fetch, but it succeeds with the new credential.

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
import Stripe from 'stripe';

const sm = new SecretsManagerClient({ region: process.env.AWS_REGION });

class CredentialCache {
  #value = null;
  #expiresAt = 0;
  #secretArn;
  #ttlMs;
  #fetchInFlight = null; // prevent thundering herd on simultaneous invalidate+get

  constructor(secretArn, ttlMs = 5 * 60 * 1000) {
    this.#secretArn = secretArn;
    this.#ttlMs = ttlMs;
  }

  async get() {
    const now = Date.now();
    if (this.#value !== null && now < this.#expiresAt) {
      return this.#value; // fast path: cached and valid
    }

    // Coalesce concurrent callers: if a fetch is already in flight, wait for it
    if (!this.#fetchInFlight) {
      this.#fetchInFlight = sm.send(new GetSecretValueCommand({
        SecretId: this.#secretArn,
        VersionStage: 'AWSCURRENT',
      })).then(resp => {
        this.#value = JSON.parse(resp.SecretString);
        this.#expiresAt = Date.now() + this.#ttlMs;
        this.#fetchInFlight = null;
        return this.#value;
      }).catch(err => {
        this.#fetchInFlight = null; // allow retry on next call
        throw err;
      });
    }

    return this.#fetchInFlight;
  }

  invalidate() {
    this.#value = null;
    this.#expiresAt = 0;
    // Do NOT cancel #fetchInFlight if one is running — let it complete
    // so we don't leave concurrent callers hanging
  }
}

// Create one cache per secret
const stripeCache = new CredentialCache(process.env.STRIPE_SECRET_ARN);
const openaiCache = new CredentialCache(process.env.OPENAI_SECRET_ARN);

// SIGHUP handler: invalidate all caches, no restart, no dropped connections
process.on('SIGHUP', () => {
  console.log('[secrets] SIGHUP received — invalidating all credential caches');
  stripeCache.invalidate();
  openaiCache.invalidate();
  // Next tool call that accesses any of these will re-fetch from Secrets Manager
  // before using the credential. No connections are dropped.
});

// In tool handlers: always call .get() — never hold the value across tool calls
server.tool('create-payment-intent', async (args) => {
  const { amount, currency } = args;

  // Re-fetch from cache on every call (fast if cached, re-fetch if TTL expired or SIGHUP received)
  const creds = await stripeCache.get();
  const stripe = new Stripe(creds.stripeSecretKey);

  const intent = await stripe.paymentIntents.create({
    amount,
    currency,
    automatic_payment_methods: { enabled: true },
  });

  return { content: [{ type: 'text', text: JSON.stringify({ clientSecret: intent.client_secret }) }] };
});

// Trigger rotation reload from your deployment pipeline:
// kill -HUP $(cat /var/run/mcp-server.pid)
// Or with systemd: systemctl kill --signal=SIGHUP mcp-server.service
// The process continues running. No session is dropped.

Attack 3: Database connection pool using stale credentials

// WRONG: pg Pool constructed with startup-time credentials from process.env
// Works fine until the database password rotates, then connections fail one by one
// as they expire and the pool tries to reconnect with the stale password

import pg from 'pg';
const { Pool } = pg;

const pool = new Pool({
  host: process.env.DB_HOST,
  port: 5432,
  database: process.env.DB_NAME,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD, // captured at startup, never refreshed
  max: 10,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
});

// Tool handler
server.tool('query-users', async (args) => {
  const { userId } = args;
  // This query will eventually fail with "FATAL: password authentication failed"
  // as pool slots exhaust their connections post-rotation
  const result = await pool.query('SELECT * FROM users WHERE id = $1', [userId]);
  return { content: [{ type: 'text', text: JSON.stringify(result.rows) }] };
});

There are three viable solutions depending on your infrastructure. The cleanest for AWS RDS is IAM authentication: instead of a static password, you generate a short-lived RDS auth token (valid 15 minutes) using the AWS SDK. The token is passed as the password field. When it expires, the pool naturally needs to reconnect anyway, and the reconnect path generates a fresh token. No static secret is involved at all.

For environments without IAM authentication, the safe pattern is an acquireClient() factory that wraps pool.connect(), detects FATAL: password authentication failed errors, recreates the pool with fresh credentials from Secrets Manager, and retries the connection. This is more complex but handles any PostgreSQL-compatible database. The third option — overriding the pool's connect callback — is available in pg via the Pool event system but is undocumented and brittle across versions; the factory pattern is more reliable.

import pg from 'pg';
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
import { Signer } from '@aws-sdk/rds-signer';

const { Pool } = pg;
const sm = new SecretsManagerClient({ region: process.env.AWS_REGION });

// ── Option A: RDS IAM authentication (no static password) ──────────────────
// RDS auth tokens are valid for 15 minutes. pg automatically reconnects when
// connections drop — we hook the password fetch into each connection attempt.

function createRdsPool() {
  const signer = new Signer({
    hostname: process.env.DB_HOST,
    port: 5432,
    region: process.env.AWS_REGION,
    username: process.env.DB_USER,
  });

  return new Pool({
    host: process.env.DB_HOST,
    port: 5432,
    database: process.env.DB_NAME,
    user: process.env.DB_USER,
    // password is a function: called on every new connection attempt
    // so each fresh connection gets a fresh 15-minute token
    password: () => signer.getAuthToken(),
    ssl: { rejectUnauthorized: true },
    max: 10,
    idleTimeoutMillis: 14 * 60 * 1000, // 14 min: expire idle connections before token expires
    connectionTimeoutMillis: 5000,
  });
}

// ── Option B: Secrets Manager password with pool recreation on auth failure ─
// For non-RDS or non-IAM databases. Wraps pool.connect() to detect auth errors
// and recreate the pool with fresh credentials from Secrets Manager.

let dbCredentialsCache = null;
let dbCredentialsCacheExpiresAt = 0;
let pool = null;

async function getDbCredentials() {
  const now = Date.now();
  if (dbCredentialsCache && now < dbCredentialsCacheExpiresAt) {
    return dbCredentialsCache;
  }
  const resp = await sm.send(new GetSecretValueCommand({
    SecretId: process.env.DB_SECRET_ARN,
    VersionStage: 'AWSCURRENT',
  }));
  dbCredentialsCache = JSON.parse(resp.SecretString);
  dbCredentialsCacheExpiresAt = now + 5 * 60 * 1000; // 5-minute TTL
  return dbCredentialsCache;
}

async function createPool() {
  const creds = await getDbCredentials();
  return new Pool({
    host: creds.host,
    port: creds.port,
    database: creds.dbname,
    user: creds.username,
    password: creds.password,
    max: 10,
    idleTimeoutMillis: 30000,
    connectionTimeoutMillis: 5000,
  });
}

// Factory that handles auth failures by recreating the pool
async function acquireClient() {
  if (!pool) {
    pool = await createPool();
  }

  try {
    return await pool.connect();
  } catch (err) {
    // pg surfaces auth failures as errors with code '28P01' (invalid_password)
    // or message matching 'password authentication failed'
    const isAuthError =
      err.code === '28P01' ||
      (err.message && err.message.includes('password authentication failed'));

    if (isAuthError) {
      console.warn('[db] Auth failure on connect — rotating credentials and recreating pool');
      // Invalidate credential cache so we fetch AWSCURRENT fresh
      dbCredentialsCache = null;
      dbCredentialsCacheExpiresAt = 0;
      // End all existing pool connections — they're all using the stale password
      await pool.end().catch(() => {}); // swallow errors from already-broken connections
      pool = await createPool();
      // Retry once with the fresh pool
      return pool.connect();
    }

    throw err; // non-auth error — surface to caller
  }
}

// Tool handler using the factory pattern
server.tool('query-users', async (args) => {
  const { userId } = args;
  const client = await acquireClient();
  try {
    const result = await client.query('SELECT id, email, created_at FROM users WHERE id = $1', [userId]);
    return { content: [{ type: 'text', text: JSON.stringify(result.rows) }] };
  } finally {
    client.release(); // always release back to the pool
  }
});

// On SIGHUP: also recreate the database pool so rotation is instant, not lazy
process.on('SIGHUP', async () => {
  console.log('[db] SIGHUP received — recreating connection pool');
  dbCredentialsCache = null;
  dbCredentialsCacheExpiresAt = 0;
  if (pool) {
    await pool.end().catch(() => {});
    pool = null;
  }
  // Pool will be recreated lazily on the next acquireClient() call
});

Attack 4: JWT signing key rotation breaks token verification

// WRONG: single signing key loaded from environment
// Rotating SIGNING_KEY_PRIVATE immediately breaks all outstanding tokens

import jwt from 'jsonwebtoken';

const SIGNING_KEY = process.env.JWT_SIGNING_KEY; // RSA private key, loaded at startup

// Issue token
function issueToolDelegationToken(userId, toolScopes) {
  return jwt.sign(
    { sub: userId, scopes: toolScopes },
    SIGNING_KEY,
    { algorithm: 'RS256', expiresIn: '15m' }
    // No 'kid' claim — impossible to identify which key to use for verification
  );
}

// Verify token
function verifyToolDelegationToken(token) {
  // Single key — if SIGNING_KEY rotates, all existing tokens fail immediately
  return jwt.verify(token, process.env.JWT_PUBLIC_KEY, { algorithms: ['RS256'] });
}

The JWKS-based rotation approach requires three infrastructure pieces: a key store that holds the current and previous key pairs, a JWKS endpoint that serves all currently-valid public keys, and a getKey() callback in your token verification flow that resolves the correct public key from the JWKS based on the token's kid claim. The jwks-rsa package provides the JWKS fetching and key resolution; jsonwebtoken accepts a key provider function in place of a static key.

import jwt from 'jsonwebtoken';
import jwksClient from 'jwks-rsa';
import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
import { randomUUID } from 'node:crypto';

const sm = new SecretsManagerClient({ region: process.env.AWS_REGION });

// ── Key store: current key ID and signing material ──────────────────────────
// In production, store this in Secrets Manager or a dedicated KMS-backed store.
// Here we model the in-process state:

let signingKeys = null; // { currentKid, keys: Map }
const KEY_OVERLAP_MS = 20 * 60 * 1000; // 20 minutes: longer than max token lifetime (15m)

async function loadSigningKeys() {
  const resp = await sm.send(new GetSecretValueCommand({
    SecretId: process.env.JWT_KEYS_SECRET_ARN,
    VersionStage: 'AWSCURRENT',
  }));
  // Expected secret structure:
  // { "currentKid": "key-abc123", "keys": { "key-abc123": { "privateKey": "...", "publicKey": "..." } } }
  const raw = JSON.parse(resp.SecretString);
  signingKeys = {
    currentKid: raw.currentKid,
    keys: new Map(Object.entries(raw.keys)),
  };
}

// Initialize at startup
await loadSigningKeys();

// JWKS endpoint handler (expose at GET /.well-known/jwks.json)
// Returns all currently-valid public keys as a JWK Set
export function getJwks() {
  const jwks = [];
  for (const [kid, { publicKey }] of signingKeys.keys) {
    // Convert PEM public key to JWK format
    // In production, use a library like 'jose' for PEM-to-JWK conversion
    jwks.push({ kid, kty: 'RSA', use: 'sig', alg: 'RS256', /* n, e fields from PEM */ });
  }
  return { keys: jwks };
}

// Issue token with kid claim identifying the signing key
function issueToolDelegationToken(userId, toolScopes) {
  const { currentKid, keys } = signingKeys;
  const { privateKey } = keys.get(currentKid);

  return jwt.sign(
    { sub: userId, scopes: toolScopes },
    privateKey,
    {
      algorithm: 'RS256',
      expiresIn: '15m',
      keyid: currentKid, // sets the 'kid' header claim
    }
  );
}

// JWKS-backed verification: resolves the correct public key from the kid in the token header
const jwks = jwksClient({
  jwksUri: `${process.env.MCP_SERVER_BASE_URL}/.well-known/jwks.json`,
  cache: true,
  cacheMaxAge: 60 * 1000, // 1-minute cache — allows JWKS to update after rotation
  rateLimit: true,
});

function getKeyForToken(header, callback) {
  jwks.getSigningKey(header.kid, (err, key) => {
    if (err) return callback(err);
    callback(null, key.getPublicKey());
  });
}

function verifyToolDelegationToken(token) {
  return new Promise((resolve, reject) => {
    jwt.verify(token, getKeyForToken, { algorithms: ['RS256'] }, (err, decoded) => {
      if (err) return reject(err);
      resolve(decoded);
    });
  });
}

// ── Rotation procedure ───────────────────────────────────────────────────────
// Called by your key rotation script or Secrets Manager rotation Lambda:
//
// Step 1: Generate a new key pair with a new kid (e.g., randomUUID()).
// Step 2: Add the new key pair to the 'keys' map, set 'currentKid' to the new kid.
//         Keep the old key in the map — do NOT remove it yet.
// Step 3: Update the secret in Secrets Manager with both keys present.
// Step 4: Send SIGHUP to the MCP server to reload signing keys.
//         New tokens are now signed with the new kid.
//         Existing tokens signed with the old kid still verify via the JWKS.
// Step 5: Wait for KEY_OVERLAP_MS (20 minutes) — longer than max token lifetime.
//         By this point, all tokens issued with the old kid have expired.
// Step 6: Remove the old kid from the 'keys' map. Update the secret. Send SIGHUP.
//         Old kid is now gone from the JWKS. Any stale tokens using it will fail
//         verification, but they would have expired anyway by this point.

process.on('SIGHUP', async () => {
  console.log('[jwt] SIGHUP received — reloading signing keys from Secrets Manager');
  try {
    await loadSigningKeys();
    console.log(`[jwt] Signing keys reloaded. Current kid: ${signingKeys.currentKid}, total keys: ${signingKeys.keys.size}`);
  } catch (err) {
    console.error('[jwt] Failed to reload signing keys — keeping existing keys', err);
    // Fail open: keep using existing keys rather than dropping all token validation
  }
});

SkillAudit findings and grade impact

Rotation pattern comparison

Related resources

The patterns in this post are part of a broader secrets management security posture for MCP servers. The foundational guide to secrets storage — covering how not to hardcode credentials, using Secrets Manager vs. environment variables, and scoping secrets to the minimum necessary access — is at /seo/mcp-server-secrets-management-security. Authentication security for MCP servers — including how credential rotation interacts with OAuth tokens, session management, and the broader MCP authorization model — is covered at /seo/mcp-server-authentication-security. Both pages include SkillAudit scoring rubrics and concrete remediation guidance.

Five-question self-audit

1. Does any file in your MCP server codebase contain const [A-Z_]+ = process.env\. at module scope, where that variable is used in tool handlers? (Check: grep -rn "^const [A-Z_]* = process\.env\." src/). If yes, that credential is frozen at startup and cannot be rotated without a session-dropping restart.
2. Does your rotation runbook revoke the old secret before all running MCP instances have had the opportunity to fetch the new one? Draw the timeline: if revocation happens before the credential cache TTL expires on every instance, you have a 401 window. The revocation step must always be last.
3. Does your pg or mysql2 pool initialization handle FATAL: password authentication failed (error code 28P01) on reconnect by fetching fresh credentials and recreating the pool, or by using RDS IAM token authentication? If neither, the pool will drain silently after a password rotation.
4. Does your JWT verification path use a JWKS endpoint with per-key kid claims, with at least two keys in the JWKS during a rotation window, and with the old key retained for at least as long as your maximum token lifetime? If you have a single verification key, rotating it drops all active sessions.
5. Does your CredentialCache.get() implementation coalesce concurrent fetches — using a single in-flight promise shared across all callers — to prevent Secrets Manager rate limit exhaustion when many tool calls arrive simultaneously during a post-SIGHUP or post-expiry cache miss?

If you answered no to any of these, the SkillAudit scanner identifies the specific module-scope reads, traces credential lifetimes through your codebase, flags pool constructors without auth-error handling, and checks whether your JWKS configuration satisfies the rotation overlap invariant. Rotation-safe architecture is reviewable statically — the patterns either exist in your code or they don't, and the scanner surfaces exactly where they're missing.