- Notifications
You must be signed in to change notification settings - Fork 590
Description
Problem
AI agents using Playwright's ariaSnapshot() or CDP Accessibility.getFullAXTree() receive the entire accessibility tree, including hundreds of off-screen nodes. This wastes serialization time and consumes AI context budget on nodes the agent can't see or interact with.
Solution
Add a CefBrowserSettings.ax_viewport_collapse setting (experimental) that collapses off-screen nodes in CDP accessibility tree serialization:
- Off-screen landmarks/headings: serialized as summaries (role + name only, empty childIds)
- Off-screen interactive/structural nodes: pruned from the response
- In-viewport nodes: fully serialized with all children and properties
position:fixed/stickydescendants: correctly detected as in-viewport even when their parent is off-screen
The setting is per-browser and can be toggled at runtime via CefBrowserHost::SetAxViewportCollapse(). Platform screen readers (NVDA, JAWS, VoiceOver) are completely unaffected — they use a separate code path. CDP nodesUpdated events are suppressed when active to maintain tree consistency. queryAXTree is left unfiltered so agents can still locate off-screen elements for scroll targeting.
Agentic workflow
With the setting enabled, an AI agent's typical interaction loop becomes:
- Get collapsed tree —
getFullAXTreereturns in-viewport nodes fully serialized, off-screen landmarks/headings as name-only summaries, and everything else pruned. The agent sees a compact representation of the page. - Interact with visible content — buttons, inputs, links in viewport are fully described.
- Discover off-screen sections — collapsed landmark summaries (e.g.
navigation: "Footer Nav") tell the agent what exists below the fold without serializing the full subtree. - Scroll to a section — the agent uses the summary node's
backendDOMNodeIdwithDOM.scrollIntoViewIfNeededto bring it into view. - Re-get tree — the scrolled-to section is now fully serialized; previously visible content that scrolled out is collapsed.
No changes are required in Playwright, MCP servers, or agent code — only the CEF embedder enables the setting.
Verification
- 12 ceftests covering: default/disabled/enabled states, heading levels, childId filtering,
position:fixedvisibility, nested landmarks, runtime toggle, all-in-viewport no-op,queryAXTreepassthrough, full agentic scroll workflow, and zoom-level coordinate correctness - Manual verification via
cef/tools/debug/ax_viewport_collapse/verify_viewport_collapse.py cefclient --ax-viewport-collapseenables the setting for manual testing