Why Your API's Error Messages Fail When Called by an LLM (And How to Fix Them)

The Problem: When Your User Can't Ask Questions ## What I Was Building ## The Pattern: Structured Recovery Plans ## Before (Developer-Focused) ## After (LLM-Focused) ## Implementation: Rich Error Classes ## The Formatter: Error Messages as Recovery Scripts ## Universal Principles for LLM Error Messages ## 1. Explicit Tool/Function Names ## 2. Numbered Recovery Steps ## 3. Explain the Why ## 4. Multiple Diagnosis Options ## 5. Include Structured Data ## 6. Distinguish Expected vs Unexpected ## Pattern 2: Track, Then Decide ## The Problem ## The Solution: FailureLog ## Why This Works for LLMs ## What Changed ## Adapting This to Your Stack ## OpenAI Function Calling ## Anthropic Claude Tools ## LangChain Tools ## REST APIs for Agents ## What I Learned ## The Shift TL;DR: When you build tools that LLMs call autonomously—OpenAI functions, Claude tools, MCP servers, custom APIs—traditional error messages break the agent workflow. A human can ask "what's a valid reference?" An LLM can't. I rebuilt Verdex's error handling to give LLMs structured recovery plans instead of descriptions. The result: errors that were conversation-killers became recoverable, autonomous workflows stopped failing silently, and the LLM could fix issues without human intervention. Here's what happens when a human gets an error: Then they ask you, and you explain. Here's what happens when an LLM gets that same error: The LLM can't ask clarifying questions. It needs to autonomously recover or the task is dead. I'm working on Verdex, an MCP server for browser automation. It exposes tools like browser_click(ref), browser_type(ref, text), and browser_snapshot(). The LLM navigates pages, fills forms, and extracts data—all autonomously. When I first deployed it, I saw this pattern constantly: The LLM had no idea that: It couldn't ask me. I wasn't there. The error message didn't tell it how to recover. I rebuilt every error message to follow this structure: Here's the before and after: Every error type gets its own class with structured properties: The properties (ref, elementInfo, frameId, etc.) enable the formatter to provide specific, actionable guidance. At the MCP server layer, I intercept all errors and format them for LLM consumption: After implementing this across 8 error types, here's what works: ❌ Bad: "Get a new snapshot" ✅ Good: "Call browser_snapshot()" The LLM needs the exact function name it should call. Don't make it guess. ❌ Bad: "You need to refresh the page and try again" ✅ Good: LLMs follow numbered lists well. They struggle with prose instructions. ❌ Bad: "Invalid ref" ✅ Good: "This reference doesn't exist because navigation invalidates old refs" Understanding causation helps the LLM avoid the same mistake next time. The LLM can pattern-match against its recent actions to figure out which cause applies. Structured info helps the LLM recognize what it was trying to interact with, making it easier to find the element in a fresh snapshot. Some failures are normal: Mark these explicitly: "This is often normal during..." vs "This is an unexpected error." It prevents the LLM from treating every error as fatal. For operations with partial failures, I separate tracking from policy enforcement. This couples failure handling with business logic. Every injection site needs to know what's critical. Step 1: Operations track all failures Step 2: Decision points check FailureLog Step 3: Warnings expose non-critical failures This pattern appears in the snapshot output: Without warnings, the LLM doesn't know if missing content is a problem or expected behavior. Before: LLMs retried the same failed operation repeatedly After: LLMs autonomously recover Error recovery rate went from ~20% to ~95%. Most failures are now self-healing. Conversation length decreased. Errors that required human intervention ("what ref should I use?") now resolve autonomously. Workflow reliability improved. Multi-step tasks that would fail on the first error now complete end-to-end. This pattern works for any LLM tool interface: Return structured error text instead of raising exceptions. The key: Return errors as structured text, not as exceptions or HTTP error codes. The LLM needs to read the error, not catch it. Error messages are part of your API contract. When building for LLMs, they're as important as success responses. "Tool name" is the most important piece of information. The LLM needs to know exactly what function to call to recover. Don't say "refresh" when you mean "browser_snapshot()". Numbered lists beat prose. LLMs follow sequential steps better than paragraph instructions. Distinguish expected from unexpected failures. "This is normal during navigation" prevents the LLM from treating temporary failures as fatal. Track failures, classify at decision points. Don't decide if something is critical at the failure site—track it and decide where you have business context. Warnings enable partial success. Operations can succeed with degraded functionality if failures are transparent. Recovery plans beat descriptions. "Element not found" tells the LLM nothing. "Call get_elements(), find the element by name, retry with the correct ref" is a recovery plan. Rich error classes enable specific guidance. UnknownRefError can give different recovery steps than StaleRefError because they have different properties. Traditional error handling is designed for debugging. You want stack traces, error codes, and technical details because you'll investigate and fix the code. LLM error handling is designed for autonomous recovery. You want context, diagnosis, and exact recovery steps because the LLM will fix the situation without code changes. The error message isn't documentation—it's a recovery script the LLM executes. When you're building tools for autonomous AI use, every error is an opportunity for the system to self-heal. Make your error messages teach the LLM how. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse CODE_BLOCK: Error: Unknown element reference: e999 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: Error: Unknown element reference: e999 CODE_BLOCK: Error: Unknown element reference: e999 CODE_BLOCK: Error: Unknown element reference: e999 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: Error: Unknown element reference: e999 CODE_BLOCK: Error: Unknown element reference: e999 CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") // Tries exact same thing Error: Unknown element reference: e5 LLM: "I encountered an error. The element reference appears to be invalid." [workflow ends] Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") // Tries exact same thing Error: Unknown element reference: e5 LLM: "I encountered an error. The element reference appears to be invalid." [workflow ends] CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") // Tries exact same thing Error: Unknown element reference: e5 LLM: "I encountered an error. The element reference appears to be invalid." [workflow ends] CODE_BLOCK: ❌ [Error Type] [What failed - with specific details] [Why it failed - likely causes] 🔧 Action Required: [Numbered steps to recover] Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: ❌ [Error Type] [What failed - with specific details] [Why it failed - likely causes] 🔧 Action Required: [Numbered steps to recover] CODE_BLOCK: ❌ [Error Type] [What failed - with specific details] [Why it failed - likely causes] 🔧 Action Required: [Numbered steps to recover] CODE_BLOCK: Error: Unknown element reference: e999 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: Error: Unknown element reference: e999 CODE_BLOCK: Error: Unknown element reference: e999 CODE_BLOCK: ❌ Unknown Element Reference Reference: e999 This reference doesn't exist in the current snapshot. Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive 🔧 Action Required: 1. Call browser_snapshot() to see currently available elements 2. Find the correct element reference in the new snapshot 3. Use the correct ref from the latest snapshot Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: ❌ Unknown Element Reference Reference: e999 This reference doesn't exist in the current snapshot. Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive 🔧 Action Required: 1. Call browser_snapshot() to see currently available elements 2. Find the correct element reference in the new snapshot 3. Use the correct ref from the latest snapshot CODE_BLOCK: ❌ Unknown Element Reference Reference: e999 This reference doesn't exist in the current snapshot. Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive 🔧 Action Required: 1. Call browser_snapshot() to see currently available elements 2. Find the correct element reference in the new snapshot 3. Use the correct ref from the latest snapshot CODE_BLOCK: export class UnknownRefError extends Error { constructor(public ref: string) { super( `Unknown element reference: ${ref}. ` + `Ref may be stale after navigation. Take a new snapshot to get fresh refs.` ); this.name = "UnknownRefError"; } } export class StaleRefError extends Error { constructor( public ref: string, public elementInfo: { role: string; name: string; tagName: string } ) { super( `Element ${ref} (${elementInfo.role} "${elementInfo.name}") was removed from DOM. ` + `Take a new snapshot() to refresh refs.` ); this.name = "StaleRefError"; } } export class FrameDetachedError extends Error { constructor(public frameId: string, details?: string) { super(`Frame ${frameId} was detached${details ? `: ${details}` : ""}`); this.name = "FrameDetachedError"; } } export class NavigationError extends Error { constructor(public url: string, public role: string, details: string) { super(`Navigation failed for role '${role}' to '${url}': ${details}`); this.name = "NavigationError"; } } export class AuthenticationError extends Error { constructor( public role: string, public authPath: string, public reason: string ) { super( `Authentication required for role '${role}' but failed to load from ${authPath}: ${reason}` ); this.name = "AuthenticationError"; } } Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: export class UnknownRefError extends Error { constructor(public ref: string) { super( `Unknown element reference: ${ref}. ` + `Ref may be stale after navigation. Take a new snapshot to get fresh refs.` ); this.name = "UnknownRefError"; } } export class StaleRefError extends Error { constructor( public ref: string, public elementInfo: { role: string; name: string; tagName: string } ) { super( `Element ${ref} (${elementInfo.role} "${elementInfo.name}") was removed from DOM. ` + `Take a new snapshot() to refresh refs.` ); this.name = "StaleRefError"; } } export class FrameDetachedError extends Error { constructor(public frameId: string, details?: string) { super(`Frame ${frameId} was detached${details ? `: ${details}` : ""}`); this.name = "FrameDetachedError"; } } export class NavigationError extends Error { constructor(public url: string, public role: string, details: string) { super(`Navigation failed for role '${role}' to '${url}': ${details}`); this.name = "NavigationError"; } } export class AuthenticationError extends Error { constructor( public role: string, public authPath: string, public reason: string ) { super( `Authentication required for role '${role}' but failed to load from ${authPath}: ${reason}` ); this.name = "AuthenticationError"; } } CODE_BLOCK: export class UnknownRefError extends Error { constructor(public ref: string) { super( `Unknown element reference: ${ref}. ` + `Ref may be stale after navigation. Take a new snapshot to get fresh refs.` ); this.name = "UnknownRefError"; } } export class StaleRefError extends Error { constructor( public ref: string, public elementInfo: { role: string; name: string; tagName: string } ) { super( `Element ${ref} (${elementInfo.role} "${elementInfo.name}") was removed from DOM. ` + `Take a new snapshot() to refresh refs.` ); this.name = "StaleRefError"; } } export class FrameDetachedError extends Error { constructor(public frameId: string, details?: string) { super(`Frame ${frameId} was detached${details ? `: ${details}` : ""}`); this.name = "FrameDetachedError"; } } export class NavigationError extends Error { constructor(public url: string, public role: string, details: string) { super(`Navigation failed for role '${role}' to '${url}': ${details}`); this.name = "NavigationError"; } } export class AuthenticationError extends Error { constructor( public role: string, public authPath: string, public reason: string ) { super( `Authentication required for role '${role}' but failed to load from ${authPath}: ${reason}` ); this.name = "AuthenticationError"; } } CODE_BLOCK: async callTool(name: string, args: any) { try { // Route to appropriate handler... return await handler(args); } catch (error) { return { content: [{ type: "text", text: this.formatErrorForLLM(error), }], }; } } private formatErrorForLLM(error: unknown): string { // Unknown reference - ref doesn't exist in snapshot if (error instanceof UnknownRefError) { return `❌ Unknown Element Reference Reference: ${error.ref} This reference doesn't exist in the current snapshot. Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive 🔧 Action Required: 1. Call browser_snapshot() to see currently available elements 2. Find the correct element reference in the new snapshot 3. Use the correct ref from the latest snapshot`; } // Stale reference - element was removed from DOM if (error instanceof StaleRefError) { return `❌ Stale Element Reference Element: ${error.ref} Type: ${error.elementInfo.role} Label: "${error.elementInfo.name}" Tag: <${error.elementInfo.tagName}> The element was removed from the DOM, likely due to: • Page navigation or refresh • Dynamic content update • JavaScript manipulation 🔧 Action Required: Call browser_snapshot() to get fresh element references, then retry your action.`; } // Frame detached - iframe removed during operation if (error instanceof FrameDetachedError) { return `❌ Frame Detached Frame ID: ${error.frameId} An iframe was removed or navigated during the operation. This is often normal during: • Navigation between pages • Single-page app (SPA) route changes • Dynamic iframe removal by JavaScript 🔧 Action Required: Call browser_snapshot() to see the current page structure and available frames.`; } // Authentication failed - required auth cannot load if (error instanceof AuthenticationError) { return `❌ Authentication Required Role: ${error.role} Auth File: ${error.authPath} Failed to load authentication data: ${error.reason} This role requires authentication but the auth file couldn't be loaded. Possible causes: • Auth file doesn't exist at specified path • Auth file has invalid JSON format • Auth file permissions prevent reading • Path specified incorrectly in configuration 🔧 Action Required: 1. Verify auth file exists: ${error.authPath} 2. Check file permissions (must be readable) 3. Validate JSON format in auth file 4. If auth is optional, set authRequired: false in role config 5. Run auth capture process if credentials expired`; } // Navigation failed - couldn't navigate to URL if (error instanceof NavigationError) { return `❌ Navigation Failed URL: ${error.url} Role: ${error.role} ${error.message} Possible causes: • Invalid or unreachable URL • Network connectivity issues • Server error (404, 500, etc.) • Authentication required (check warnings in snapshot) • Timeout (page took too long to load) • Main frame injection failed 🔧 Action Required: • Verify the URL is correct and accessible • Check network connectivity • Call browser_snapshot() to see current page state • Check role authentication status via getFailures() • Try a different URL or retry after a moment`; } // Generic error fallback if (error instanceof Error) { return `❌ Error ${error.message} If this error persists, check: • Your input parameters • Current page state (call browser_snapshot()) • Network connectivity • Browser logs for additional context`; } // Unknown error type return `❌ Unknown Error ${String(error)} This is an unexpected error type. Please report this issue with context about what operation you were attempting.`; } Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: async callTool(name: string, args: any) { try { // Route to appropriate handler... return await handler(args); } catch (error) { return { content: [{ type: "text", text: this.formatErrorForLLM(error), }], }; } } private formatErrorForLLM(error: unknown): string { // Unknown reference - ref doesn't exist in snapshot if (error instanceof UnknownRefError) { return `❌ Unknown Element Reference Reference: ${error.ref} This reference doesn't exist in the current snapshot. Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive 🔧 Action Required: 1. Call browser_snapshot() to see currently available elements 2. Find the correct element reference in the new snapshot 3. Use the correct ref from the latest snapshot`; } // Stale reference - element was removed from DOM if (error instanceof StaleRefError) { return `❌ Stale Element Reference Element: ${error.ref} Type: ${error.elementInfo.role} Label: "${error.elementInfo.name}" Tag: <${error.elementInfo.tagName}> The element was removed from the DOM, likely due to: • Page navigation or refresh • Dynamic content update • JavaScript manipulation 🔧 Action Required: Call browser_snapshot() to get fresh element references, then retry your action.`; } // Frame detached - iframe removed during operation if (error instanceof FrameDetachedError) { return `❌ Frame Detached Frame ID: ${error.frameId} An iframe was removed or navigated during the operation. This is often normal during: • Navigation between pages • Single-page app (SPA) route changes • Dynamic iframe removal by JavaScript 🔧 Action Required: Call browser_snapshot() to see the current page structure and available frames.`; } // Authentication failed - required auth cannot load if (error instanceof AuthenticationError) { return `❌ Authentication Required Role: ${error.role} Auth File: ${error.authPath} Failed to load authentication data: ${error.reason} This role requires authentication but the auth file couldn't be loaded. Possible causes: • Auth file doesn't exist at specified path • Auth file has invalid JSON format • Auth file permissions prevent reading • Path specified incorrectly in configuration 🔧 Action Required: 1. Verify auth file exists: ${error.authPath} 2. Check file permissions (must be readable) 3. Validate JSON format in auth file 4. If auth is optional, set authRequired: false in role config 5. Run auth capture process if credentials expired`; } // Navigation failed - couldn't navigate to URL if (error instanceof NavigationError) { return `❌ Navigation Failed URL: ${error.url} Role: ${error.role} ${error.message} Possible causes: • Invalid or unreachable URL • Network connectivity issues • Server error (404, 500, etc.) • Authentication required (check warnings in snapshot) • Timeout (page took too long to load) • Main frame injection failed 🔧 Action Required: • Verify the URL is correct and accessible • Check network connectivity • Call browser_snapshot() to see current page state • Check role authentication status via getFailures() • Try a different URL or retry after a moment`; } // Generic error fallback if (error instanceof Error) { return `❌ Error ${error.message} If this error persists, check: • Your input parameters • Current page state (call browser_snapshot()) • Network connectivity • Browser logs for additional context`; } // Unknown error type return `❌ Unknown Error ${String(error)} This is an unexpected error type. Please report this issue with context about what operation you were attempting.`; } CODE_BLOCK: async callTool(name: string, args: any) { try { // Route to appropriate handler... return await handler(args); } catch (error) { return { content: [{ type: "text", text: this.formatErrorForLLM(error), }], }; } } private formatErrorForLLM(error: unknown): string { // Unknown reference - ref doesn't exist in snapshot if (error instanceof UnknownRefError) { return `❌ Unknown Element Reference Reference: ${error.ref} This reference doesn't exist in the current snapshot. Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive 🔧 Action Required: 1. Call browser_snapshot() to see currently available elements 2. Find the correct element reference in the new snapshot 3. Use the correct ref from the latest snapshot`; } // Stale reference - element was removed from DOM if (error instanceof StaleRefError) { return `❌ Stale Element Reference Element: ${error.ref} Type: ${error.elementInfo.role} Label: "${error.elementInfo.name}" Tag: <${error.elementInfo.tagName}> The element was removed from the DOM, likely due to: • Page navigation or refresh • Dynamic content update • JavaScript manipulation 🔧 Action Required: Call browser_snapshot() to get fresh element references, then retry your action.`; } // Frame detached - iframe removed during operation if (error instanceof FrameDetachedError) { return `❌ Frame Detached Frame ID: ${error.frameId} An iframe was removed or navigated during the operation. This is often normal during: • Navigation between pages • Single-page app (SPA) route changes • Dynamic iframe removal by JavaScript 🔧 Action Required: Call browser_snapshot() to see the current page structure and available frames.`; } // Authentication failed - required auth cannot load if (error instanceof AuthenticationError) { return `❌ Authentication Required Role: ${error.role} Auth File: ${error.authPath} Failed to load authentication data: ${error.reason} This role requires authentication but the auth file couldn't be loaded. Possible causes: • Auth file doesn't exist at specified path • Auth file has invalid JSON format • Auth file permissions prevent reading • Path specified incorrectly in configuration 🔧 Action Required: 1. Verify auth file exists: ${error.authPath} 2. Check file permissions (must be readable) 3. Validate JSON format in auth file 4. If auth is optional, set authRequired: false in role config 5. Run auth capture process if credentials expired`; } // Navigation failed - couldn't navigate to URL if (error instanceof NavigationError) { return `❌ Navigation Failed URL: ${error.url} Role: ${error.role} ${error.message} Possible causes: • Invalid or unreachable URL • Network connectivity issues • Server error (404, 500, etc.) • Authentication required (check warnings in snapshot) • Timeout (page took too long to load) • Main frame injection failed 🔧 Action Required: • Verify the URL is correct and accessible • Check network connectivity • Call browser_snapshot() to see current page state • Check role authentication status via getFailures() • Try a different URL or retry after a moment`; } // Generic error fallback if (error instanceof Error) { return `❌ Error ${error.message} If this error persists, check: • Your input parameters • Current page state (call browser_snapshot()) • Network connectivity • Browser logs for additional context`; } // Unknown error type return `❌ Unknown Error ${String(error)} This is an unexpected error type. Please report this issue with context about what operation you were attempting.`; } CODE_BLOCK: 1. Call browser_navigate(url) 2. Call browser_snapshot() to get new refs 3. Find the button in the new snapshot 4. Retry browser_click() with the new ref Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: 1. Call browser_navigate(url) 2. Call browser_snapshot() to get new refs 3. Find the button in the new snapshot 4. Retry browser_click() with the new ref CODE_BLOCK: 1. Call browser_navigate(url) 2. Call browser_snapshot() to get new refs 3. Find the button in the new snapshot 4. Retry browser_click() with the new ref CODE_BLOCK: Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive CODE_BLOCK: Possible causes: • Using a ref from an old snapshot (stale after navigation) • Typo in the ref name • Element not yet loaded or not interactive CODE_BLOCK: Element: e5 Type: button Label: "Submit" Tag: <button> Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: Element: e5 Type: button Label: "Submit" Tag: <button> CODE_BLOCK: Element: e5 Type: button Label: "Submit" Tag: <button> CODE_BLOCK: // ❌ Don't decide criticality at failure site try { await injectIntoFrame(frameId); } catch (error) { if (frameId === mainFrameId) { throw error; // Critical! } // Otherwise ignore? } Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: // ❌ Don't decide criticality at failure site try { await injectIntoFrame(frameId); } catch (error) { if (frameId === mainFrameId) { throw error; // Critical! } // Otherwise ignore? } CODE_BLOCK: // ❌ Don't decide criticality at failure site try { await injectIntoFrame(frameId); } catch (error) { if (frameId === mainFrameId) { throw error; // Critical! } // Otherwise ignore? } CODE_BLOCK: type FailureLog = { frameInjectionFailures: Array<{ frameId: string; error: string; reason: "cross-origin" | "detached" | "timeout" | "unknown"; isMainFrame: boolean; // Track criticality as metadata timestamp: number; }>; frameExpansionFailures: Array<{ ref: string; error: string; detached: boolean; timestamp: number; }>; authLoadError?: { error: string; authPath: string; timestamp: number; }; }; Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: type FailureLog = { frameInjectionFailures: Array<{ frameId: string; error: string; reason: "cross-origin" | "detached" | "timeout" | "unknown"; isMainFrame: boolean; // Track criticality as metadata timestamp: number; }>; frameExpansionFailures: Array<{ ref: string; error: string; detached: boolean; timestamp: number; }>; authLoadError?: { error: string; authPath: string; timestamp: number; }; }; CODE_BLOCK: type FailureLog = { frameInjectionFailures: Array<{ frameId: string; error: string; reason: "cross-origin" | "detached" | "timeout" | "unknown"; isMainFrame: boolean; // Track criticality as metadata timestamp: number; }>; frameExpansionFailures: Array<{ ref: string; error: string; detached: boolean; timestamp: number; }>; authLoadError?: { error: string; authPath: string; timestamp: number; }; }; COMMAND_BLOCK: async injectFrameTreeRecursive( context: RoleContext, frameTree: any, isMainFrame: boolean = false ): Promise<void> { try { await context.bridgeInjector.ensureFrameState( context.cdpSession, frameTree.frame.id ); } catch (error) { // Track in FailureLog (don't throw yet) const failures = this.ensureFailureLog(context); failures.frameInjectionFailures.push({ frameId: frameTree.frame.id, error: error.message, reason: this.classifyFrameError(error), isMainFrame, // Metadata, not decision timestamp: Date.now(), }); return; } } Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: async injectFrameTreeRecursive( context: RoleContext, frameTree: any, isMainFrame: boolean = false ): Promise<void> { try { await context.bridgeInjector.ensureFrameState( context.cdpSession, frameTree.frame.id ); } catch (error) { // Track in FailureLog (don't throw yet) const failures = this.ensureFailureLog(context); failures.frameInjectionFailures.push({ frameId: frameTree.frame.id, error: error.message, reason: this.classifyFrameError(error), isMainFrame, // Metadata, not decision timestamp: Date.now(), }); return; } } COMMAND_BLOCK: async injectFrameTreeRecursive( context: RoleContext, frameTree: any, isMainFrame: boolean = false ): Promise<void> { try { await context.bridgeInjector.ensureFrameState( context.cdpSession, frameTree.frame.id ); } catch (error) { // Track in FailureLog (don't throw yet) const failures = this.ensureFailureLog(context); failures.frameInjectionFailures.push({ frameId: frameTree.frame.id, error: error.message, reason: this.classifyFrameError(error), isMainFrame, // Metadata, not decision timestamp: Date.now(), }); return; } } COMMAND_BLOCK: async navigate(url: string): Promise<Snapshot> { // ... navigation logic ... await this.discoverAndInjectFrames(context); // DECISION POINT: Check for critical failures const mainFrameFailed = context.failures?.frameInjectionFailures .some(f => f.isMainFrame); if (mainFrameFailed) { throw new Error('Main frame injection failed - page cannot be automated'); } // Non-critical failures become warnings snapshot.warnings = this.buildWarningsFromFailureLog(context); return snapshot; } Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: async navigate(url: string): Promise<Snapshot> { // ... navigation logic ... await this.discoverAndInjectFrames(context); // DECISION POINT: Check for critical failures const mainFrameFailed = context.failures?.frameInjectionFailures .some(f => f.isMainFrame); if (mainFrameFailed) { throw new Error('Main frame injection failed - page cannot be automated'); } // Non-critical failures become warnings snapshot.warnings = this.buildWarningsFromFailureLog(context); return snapshot; } COMMAND_BLOCK: async navigate(url: string): Promise<Snapshot> { // ... navigation logic ... await this.discoverAndInjectFrames(context); // DECISION POINT: Check for critical failures const mainFrameFailed = context.failures?.frameInjectionFailures .some(f => f.isMainFrame); if (mainFrameFailed) { throw new Error('Main frame injection failed - page cannot be automated'); } // Non-critical failures become warnings snapshot.warnings = this.buildWarningsFromFailureLog(context); return snapshot; } COMMAND_BLOCK: private buildWarningsFromFailureLog(context: RoleContext) { const failures = context.failures; if (!failures) return undefined; const warnings: any = {}; // Check for inaccessible frames (non-main frames that failed) const inaccessibleFrames = failures.frameInjectionFailures .filter(f => !f.isMainFrame); if (inaccessibleFrames.length > 0) { warnings.inaccessibleFrames = inaccessibleFrames.length; warnings.details = inaccessibleFrames.map(f => `Frame ${f.frameId}: ${f.reason}` ); } // Check for auth failures if (failures.authLoadError) { warnings.authStatus = "unauthenticated"; warnings.details = warnings.details || []; warnings.details.push(`Auth failed: ${failures.authLoadError.error}`); } return Object.keys(warnings).length > 0 ? warnings : undefined; } Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: private buildWarningsFromFailureLog(context: RoleContext) { const failures = context.failures; if (!failures) return undefined; const warnings: any = {}; // Check for inaccessible frames (non-main frames that failed) const inaccessibleFrames = failures.frameInjectionFailures .filter(f => !f.isMainFrame); if (inaccessibleFrames.length > 0) { warnings.inaccessibleFrames = inaccessibleFrames.length; warnings.details = inaccessibleFrames.map(f => `Frame ${f.frameId}: ${f.reason}` ); } // Check for auth failures if (failures.authLoadError) { warnings.authStatus = "unauthenticated"; warnings.details = warnings.details || []; warnings.details.push(`Auth failed: ${failures.authLoadError.error}`); } return Object.keys(warnings).length > 0 ? warnings : undefined; } COMMAND_BLOCK: private buildWarningsFromFailureLog(context: RoleContext) { const failures = context.failures; if (!failures) return undefined; const warnings: any = {}; // Check for inaccessible frames (non-main frames that failed) const inaccessibleFrames = failures.frameInjectionFailures .filter(f => !f.isMainFrame); if (inaccessibleFrames.length > 0) { warnings.inaccessibleFrames = inaccessibleFrames.length; warnings.details = inaccessibleFrames.map(f => `Frame ${f.frameId}: ${f.reason}` ); } // Check for auth failures if (failures.authLoadError) { warnings.authStatus = "unauthenticated"; warnings.details = warnings.details || []; warnings.details.push(`Auth failed: ${failures.authLoadError.error}`); } return Object.keys(warnings).length > 0 ? warnings : undefined; } CODE_BLOCK: { "text": "- button \"Submit\" [ref=e1]\n...", "elementCount": 15, "warnings": { "inaccessibleFrames": 2, "details": [ "Frame abc123: cross-origin", "Frame def456: detached" ] } } Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: { "text": "- button \"Submit\" [ref=e1]\n...", "elementCount": 15, "warnings": { "inaccessibleFrames": 2, "details": [ "Frame abc123: cross-origin", "Frame def456: detached" ] } } CODE_BLOCK: { "text": "- button \"Submit\" [ref=e1]\n...", "elementCount": 15, "warnings": { "inaccessibleFrames": 2, "details": [ "Frame abc123: cross-origin", "Frame def456: detached" ] } } CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") Error: Unknown element reference: e5 [workflow fails] Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") Error: Unknown element reference: e5 [workflow fails] CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") Error: Unknown element reference: e5 LLM: browser_click("e5") Error: Unknown element reference: e5 [workflow fails] CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 [Error includes recovery steps mentioning browser_snapshot()] LLM: browser_snapshot() [Gets fresh refs, sees e7 is the submit button] LLM: browser_click("e7") Success! Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 [Error includes recovery steps mentioning browser_snapshot()] LLM: browser_snapshot() [Gets fresh refs, sees e7 is the submit button] LLM: browser_click("e7") Success! CODE_BLOCK: LLM: browser_click("e5") Error: Unknown element reference: e5 [Error includes recovery steps mentioning browser_snapshot()] LLM: browser_snapshot() [Gets fresh refs, sees e7 is the submit button] LLM: browser_click("e7") Success! COMMAND_BLOCK: @tool def click_element(ref: str) -> str: """Click an interactive element""" try: element = get_element(ref) element.click() return "Clicked successfully" except UnknownRefError as e: return format_error_for_llm(e) Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: @tool def click_element(ref: str) -> str: """Click an interactive element""" try: element = get_element(ref) element.click() return "Clicked successfully" except UnknownRefError as e: return format_error_for_llm(e) COMMAND_BLOCK: @tool def click_element(ref: str) -> str: """Click an interactive element""" try: element = get_element(ref) element.click() return "Clicked successfully" except UnknownRefError as e: return format_error_for_llm(e) COMMAND_BLOCK: server.tool("click_element", { ref: { type: "string" } }, async ({ ref }) => { try { await clickElement(ref); return { content: [{ type: "text", text: "Clicked successfully" }] }; } catch (error) { return { content: [{ type: "text", text: formatErrorForLLM(error) }] }; } } ); Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: server.tool("click_element", { ref: { type: "string" } }, async ({ ref }) => { try { await clickElement(ref); return { content: [{ type: "text", text: "Clicked successfully" }] }; } catch (error) { return { content: [{ type: "text", text: formatErrorForLLM(error) }] }; } } ); COMMAND_BLOCK: server.tool("click_element", { ref: { type: "string" } }, async ({ ref }) => { try { await clickElement(ref); return { content: [{ type: "text", text: "Clicked successfully" }] }; } catch (error) { return { content: [{ type: "text", text: formatErrorForLLM(error) }] }; } } ); COMMAND_BLOCK: class ClickElementTool(BaseTool): name = "click_element" description = "Click an interactive element" def _run(self, ref: str) -> str: try: click_element(ref) return "Clicked successfully" except Exception as e: return format_error_for_llm(e) Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: class ClickElementTool(BaseTool): name = "click_element" description = "Click an interactive element" def _run(self, ref: str) -> str: try: click_element(ref) return "Clicked successfully" except Exception as e: return format_error_for_llm(e) COMMAND_BLOCK: class ClickElementTool(BaseTool): name = "click_element" description = "Click an interactive element" def _run(self, ref: str) -> str: try: click_element(ref) return "Clicked successfully" except Exception as e: return format_error_for_llm(e) COMMAND_BLOCK: app.post('/api/click', async (req, res) => { try { await clickElement(req.body.ref); res.json({ success: true, message: "Clicked successfully" }); } catch (error) { // Don't use HTTP error codes - return 200 with formatted error res.json({ success: false, error: formatErrorForLLM(error) }); } }); Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: app.post('/api/click', async (req, res) => { try { await clickElement(req.body.ref); res.json({ success: true, message: "Clicked successfully" }); } catch (error) { // Don't use HTTP error codes - return 200 with formatted error res.json({ success: false, error: formatErrorForLLM(error) }); } }); COMMAND_BLOCK: app.post('/api/click', async (req, res) => { try { await clickElement(req.body.ref); res.json({ success: true, message: "Clicked successfully" }); } catch (error) { // Don't use HTTP error codes - return 200 with formatted error res.json({ success: false, error: formatErrorForLLM(error) }); } }); - "What's an element reference?" - "Where do I get valid ones?" - "Why did mine become invalid?" - Tries the same thing again (with the same invalid ref) - Gets the same error - Gives up or hallucinates a solution - The entire agentic workflow fails - Element refs come from browser_snapshot() - Navigation invalidates old refs - It needed to call browser_snapshot() first to get fresh refs - Context: What a "reference" is and where they come from - Diagnosis: Why this specific one is invalid - Recovery: Exact API calls needed to fix it (with names!) - Ordering: Numbered steps the LLM can follow sequentially - Frame detachment during navigation - Cross-origin iframe access denied - Element not found (might load later) - ✅ Operation succeeded (got a snapshot) - ✅ Partial failures are transparent (warnings) - ✅ Clear reason for each failure - ✅ Can proceed with main content - Error messages are part of your API contract. When building for LLMs, they're as important as success responses. - "Tool name" is the most important piece of information. The LLM needs to know exactly what function to call to recover. Don't say "refresh" when you mean "browser_snapshot()". - Numbered lists beat prose. LLMs follow sequential steps better than paragraph instructions. - Distinguish expected from unexpected failures. "This is normal during navigation" prevents the LLM from treating temporary failures as fatal. - Track failures, classify at decision points. Don't decide if something is critical at the failure site—track it and decide where you have business context. - Warnings enable partial success. Operations can succeed with degraded functionality if failures are transparent. - Recovery plans beat descriptions. "Element not found" tells the LLM nothing. "Call get_elements(), find the element by name, retry with the correct ref" is a recovery plan. - Rich error classes enable specific guidance. UnknownRefError can give different recovery steps than StaleRefError because they have different properties.