MCP server rate limiting deep dive: sliding window, token bucket, and adaptive throttling

The fixed window counter is the naive approach — count calls in each 60-second bucket. Its flaw is that a client can make N calls at 00:59 and N more at 01:01 and never trigger the limit, yet hammer you 2N times in two seconds. The sliding window fixes this by counting calls that occurred in the last N seconds from now, not from the start of the current bucket.

Implementation uses a per-session circular buffer of timestamps. When a call arrives, pop all timestamps older than the window, check the count, then push the new timestamp.

class SlidingWindowLimiter {
  constructor({ windowMs = 60_000, maxCalls = 20 } = {}) {
    this.windowMs = windowMs
    this.maxCalls = maxCalls
    this.sessions = new Map() // sessionId → timestamp[]
  }

  check(sessionId) {
    const now = Date.now()
    const cutoff = now - this.windowMs

    let timestamps = this.sessions.get(sessionId)
    if (!timestamps) {
      timestamps = []
      this.sessions.set(sessionId, timestamps)
    }

    // drop calls outside the window
    while (timestamps.length > 0 && timestamps[0] <= cutoff) {
      timestamps.shift()
    }

    if (timestamps.length >= this.maxCalls) {
      const oldestInWindow = timestamps[0]
      const retryAfterMs = oldestInWindow + this.windowMs - now
      return { allowed: false, retryAfterMs }
    }

    timestamps.push(now)
    return { allowed: true, retryAfterMs: 0 }
  }

  // call in onSessionClose to prevent memory leak
  evict(sessionId) {
    this.sessions.delete(sessionId)
  }
}

// wire into your tool handler
const limiter = new SlidingWindowLimiter({ windowMs: 60_000, maxCalls: 20 })

server.tool('scan_repo', async (args, { sessionId }) => {
  const { allowed, retryAfterMs } = limiter.check(sessionId)
  if (!allowed) {
    throw new Error(
      `Rate limit exceeded. Try again in ${Math.ceil(retryAfterMs / 1000)}s.`
    )
  }
  return runScan(args)
})

The memory cost is bounded by maxCalls timestamps per active session — typically a few hundred bytes. If a session makes fewer calls than the limit, you're storing fewer timestamps. At 20 calls per window, each session uses at most 160 bytes (20 × 8 bytes for 64-bit timestamps). For 10,000 concurrent sessions that's ~1.6MB: entirely negligible.

The sliding window produces the most accurate rate limiting behaviour. It's also the most auditable: when a user asks "how many calls did I make in the last minute?", the answer is simply limiter.sessions.get(sessionId)?.length ?? 0. This transparency is why it's the preferred algorithm for tools under SOC 2 or GDPR audit-trail requirements.

Token bucket is the algorithm used by most commercial rate-limiting services (AWS API Gateway, Cloudflare, Google Cloud Endpoints). Its defining feature is that unused capacity accumulates up to a ceiling. A session that makes no calls for 60 seconds then wants to run a batch of 10 calls in quick succession can do so — because it has 10 tokens saved up.

This matters for MCP tools that wrap batch-capable APIs. If your tool calls the GitHub API and users legitimately run "check 10 repos" commands, a sliding window will block them after the first 5 if you've set the limit to 5/minute. A token bucket with a 10-token capacity and a 1-token-per-6-seconds refill rate would allow the burst and then smooth out subsequent calls.

class TokenBucketLimiter {
  constructor({
    capacity = 10,        // max burst size
    refillRatePerMs = 1 / 6000  // 1 token per 6 seconds
  } = {}) {
    this.capacity = capacity
    this.refillRatePerMs = refillRatePerMs
    this.buckets = new Map() // sessionId → { tokens, lastRefill }
  }

  _getBucket(sessionId) {
    let bucket = this.buckets.get(sessionId)
    if (!bucket) {
      bucket = { tokens: this.capacity, lastRefill: Date.now() }
      this.buckets.set(sessionId, bucket)
    }
    return bucket
  }

  check(sessionId) {
    const now = Date.now()
    const bucket = this._getBucket(sessionId)

    // refill tokens based on elapsed time
    const elapsed = now - bucket.lastRefill
    const refilled = elapsed * this.refillRatePerMs
    bucket.tokens = Math.min(this.capacity, bucket.tokens + refilled)
    bucket.lastRefill = now

    if (bucket.tokens < 1) {
      const tokensNeeded = 1 - bucket.tokens
      const retryAfterMs = Math.ceil(tokensNeeded / this.refillRatePerMs)
      return { allowed: false, retryAfterMs }
    }

    bucket.tokens -= 1
    return { allowed: true, retryAfterMs: 0 }
  }

  evict(sessionId) {
    this.buckets.delete(sessionId)
  }
}

// example: 10-call burst, then 1 call per 6 seconds sustained
const limiter = new TokenBucketLimiter({
  capacity: 10,
  refillRatePerMs: 1 / 6000
})

The memory cost is O(1) per session — just two numbers (token count and last refill timestamp). This is more efficient than the sliding window for high-concurrency servers. The tradeoff is that the burst allowance can be surprising to users who expect strict per-minute limits. Document your limits explicitly in the tool's description string and in error messages.

One gotcha specific to MCP: the LLM itself can trigger retry loops. If your error message says "retry in X seconds," some models will do exactly that in a tight loop. Always include a Retry-After value in the error and consider whether your framework exposes it as a structured field rather than free text in the error string.

// return structured error so the LLM gets clear signal
if (!allowed) {
  return {
    content: [{
      type: 'text',
      text: JSON.stringify({
        error: 'rate_limit_exceeded',
        retry_after_seconds: Math.ceil(retryAfterMs / 1000),
        message: `Tool call limit reached. Wait ${Math.ceil(retryAfterMs / 1000)}s before retrying.`
      })
    }],
    isError: true
  }
}

Sliding window and token bucket operate on request count alone. They don't know whether your server is currently at 10% CPU or 90% CPU, or whether the downstream API you're calling just started returning slow responses. Adaptive throttling closes that gap.

The algorithm Google describes in their SRE book uses the following formula:

// Google's client-side adaptive throttle formula:
// p(throttle) = max(0, (requests - K * accepts) / (requests + 1))
// where K is a multiplier controlling aggressiveness (2.0 is the SRE default)

class AdaptiveThrottler {
  constructor({ K = 2.0, windowMs = 10_000 } = {}) {
    this.K = K
    this.windowMs = windowMs
    this.requests = 0      // total requests in window
    this.accepts = 0       // accepted requests in window
    this.windowStart = Date.now()
  }

  _maybeReset() {
    const now = Date.now()
    if (now - this.windowStart >= this.windowMs) {
      this.requests = 0
      this.accepts = 0
      this.windowStart = now
    }
  }

  check() {
    this._maybeReset()
    this.requests++

    const throttleProb = Math.max(
      0,
      (this.requests - this.K * this.accepts) / (this.requests + 1)
    )

    if (Math.random() < throttleProb) {
      return { allowed: false, throttleProb }
    }

    this.accepts++
    return { allowed: true, throttleProb }
  }

  // call this when the downstream request succeeds
  recordSuccess() {
    // accepts is already incremented in check() — no-op here
    // extend if you want success/failure-weighted accepts
  }

  // call this when the downstream request fails or is slow
  recordBackpressure() {
    // penalise the accepts count to increase throttle probability
    this.accepts = Math.max(0, this.accepts - 1)
  }
}

The key insight is that the throttle probability starts at 0 (no throttling when the server is healthy) and rises as the ratio of accepted-to-total requests falls. When downstream latency spikes, you call recordBackpressure() to decrease the accepts count, which increases the throttle probability for subsequent requests — automatically shedding load before the downstream service times out.

Adaptive throttling is a global limiter, not a per-session limiter. Wire it at the server level and short-circuit tool dispatch before the session-level rate check:

const globalThrottler = new AdaptiveThrottler({ K: 2.0, windowMs: 10_000 })

// middleware applied to every tool call
function dispatchWithThrottling(toolFn) {
  return async (args, context) => {
    const { allowed } = globalThrottler.check()
    if (!allowed) {
      return {
        content: [{ type: 'text', text: '{"error":"server_overloaded","retry_after_seconds":5}' }],
        isError: true
      }
    }

    try {
      const result = await toolFn(args, context)
      globalThrottler.recordSuccess()
      return result
    } catch (err) {
      if (err.code === 'ETIMEDOUT' || err.statusCode === 429) {
        globalThrottler.recordBackpressure()
      }
      throw err
    }
  }
}

In practice, most MCP servers don't need adaptive throttling until they're handling dozens of concurrent sessions. It adds meaningful complexity — the K parameter needs tuning for your workload, and the probabilistic nature means a session can be throttled even when the overall acceptance rate is high. Reserve it for tools that wrap expensive external APIs (LLM inference, database queries with full-table scans, file conversion services) where a sudden spike in calls would cascade into downstream failures.