Introducing GhostPilot: Because Developers Shouldn't Have to Write Playwright Scripts

Every developer who has maintained a Playwright or Selenium test suite knows the feeling. You write a test. It works. The frontend team ships a redesign. Every CSS selector breaks. You spend a day rewriting locators. The tests pass again. Two weeks later, someone changes a class name. The cycle repeats. Multiply this by every flow in your application and you have a full-time job that produces no product value.

We built GhostPilot because browser automation should not require maintaining brittle selector-based scripts. The core insight is simple: what if the AI could just see the browser like a human does, and you could watch it work?

GhostPilot is shared browser control. A human and an AI operate the same Chromium browser simultaneously. Both see the same live view at ~30 frames per second. Both can click, type, scroll, and navigate. You tell the AI what to do in natural language. It does it. You watch. If it goes off track, you take over. No selectors. No DOM parsing. No screenshots-and-pray.

GhostPilot in action: human and AI sharing the same live Chromium browser session.

The Problem with Browser Automation in 2026

Browser automation has been stuck in the same paradigm since Selenium launched in 2004. The approach is always the same: programmatically control a browser by identifying elements through selectors (CSS, XPath, ARIA labels, data attributes) and dispatching events to them. Playwright modernized the API surface but did not change the fundamental model.

This model has three structural problems that no amount of tooling can fix.

First, selectors are inherently fragile. A CSS selector is a coupling between your test and the internal structure of the DOM. When the DOM changes — and it changes constantly in any active frontend codebase — selectors break. Data attributes help but add maintenance overhead of their own, and they only work for elements your team remembers to instrument.

Second, headless browsers hide what is happening. When a Playwright test fails, you get a stack trace and maybe a screenshot taken at the moment of failure. You cannot see what led to the failure. You cannot watch the flow in real time. Debugging requires adding trace recording, re-running the test, downloading the trace, and scrubbing through it frame by frame. This is a multi-minute process for every failure.

Third, AI-driven browser tools take screenshots and parse DOM. The current generation of AI browser agents works by taking a screenshot, sending it to a vision model, getting back coordinates or element descriptions, and then dispatching click events. This is slow (each screenshot-to-action cycle takes seconds), lossy (a single screenshot cannot capture scroll state, hover effects, or animations), and fragile (the AI has to re-identify elements on every frame). It is the selector model repackaged with an LLM in the middle.

How GhostPilot Works

GhostPilot takes a fundamentally different approach. Instead of screenshots and selectors, it gives the AI a live video feed of the browser and a set of interaction primitives. The architecture has four layers.

Layer 1: Puppeteer-core + Chrome DevTools Protocol

GhostPilot uses Puppeteer-core to launch and manage a Chromium browser instance. We chose Puppeteer-core over Playwright for a specific reason: Puppeteer gives us direct access to the Chrome DevTools Protocol (CDP), which exposes the Page.screencastFrame event. This is not a screenshot. It is a continuous video stream of the browser viewport, delivered as JPEG frames at a configurable framerate. We run it at approximately 30fps.

CDP screencast is what makes shared browser control possible. The AI does not see a static image that was captured milliseconds or seconds ago. It sees the live browser as it renders. Animations, loading states, hover effects, scroll positions — all visible in real time.

Layer 2: WebSocket Frame Streaming

CDP screencast frames are relayed to connected clients over WebSocket. Any number of viewers — human operators, AI agents, recording systems — can subscribe to the same frame stream. The WebSocket connection delivers base64-encoded JPEG frames along with metadata: timestamp, viewport dimensions, and session state.

This is the mechanism that makes GhostPilot “shared.” The human and the AI are not taking turns. They are both watching the same live stream. Either can act at any time. The human sees exactly what the AI sees, and the AI sees every action the human takes.

Layer 3: REST API for AI Control

AI agents interact with the browser through a clean REST API. The API surface is deliberately simple — it mirrors what a human can do with a mouse and keyboard.

There are no CSS selectors in the primary interaction model. The AI clicks at coordinates based on what it sees in the video stream. It types text into whatever element has focus. It scrolls the viewport. It presses keys. This is how a human operates a browser — by looking at it and interacting with what they see. The /api/select endpoint exists as a convenience for dropdowns and form elements where coordinate-based interaction is unreliable, but the primary interaction model is vision-based.

Layer 4: Recording Studio

Every GhostPilot session can be recorded. The recording engine captures the WebSocket frame stream and produces GIF, MP4, or WebM output via ffmpeg. This is not screen recording of the operator's desktop — it is a direct capture of the browser viewport frames, producing clean, consistent output regardless of the operator's screen resolution or window arrangement.

Use cases for recording are surprisingly broad: generating demo videos, documenting QA flows, creating onboarding walkthroughs, capturing bug reproductions, and building training data for visual AI models.

Secret Redaction

Any tool that gives an AI access to a browser is also giving it access to everything displayed in that browser. Passwords. API keys. JWT tokens. Email addresses. Social Security numbers. Credit card numbers. If the AI can see the browser, it can see these too. If the session is being recorded, those secrets end up in the recording.

GhostPilot includes built-in secret redaction that operates on the frame stream before it reaches AI agents or the recording engine. The redaction system detects and masks:

• Passwords in form fields and displayed text
• API keys matching common provider patterns (AWS, GCP, Stripe, etc.)
• JWT tokens (the characteristic three-segment base64 format)
• Email addresses
• Social Security numbers
• Credit card numbers (Luhn-validated)

Redaction happens at the stream level, not at the display level. The human operator sees the unredacted browser. The AI agent and the recording engine see the redacted stream. This means secrets are never included in AI context windows and never appear in recorded output, even if they are visible on screen.

Human-in-the-Loop (HITL) Support

Shared browser control means the human is always in the loop by default. You are not handing the browser to the AI and hoping for the best. You are watching the AI work in real time. If the AI navigates to the wrong page, you can immediately take over. If the AI is about to click the wrong button, you can intervene. If the AI needs guidance, you can type or click to show it what to do, and it sees your actions in the same stream.

This is structurally different from the HITL pattern in most AI agent frameworks, where human review happens before or after an action. In GhostPilot, human oversight is continuous and concurrent. The human does not review a plan and approve it. The human watches the execution and intervenes when necessary. This is closer to how a senior engineer pair-programs with a junior engineer: they watch, they let the junior proceed when things are going well, and they step in when they see a mistake forming.

Scripts Engine

Not every browser automation task requires AI. Some tasks are predictable, repeatable sequences: log in, navigate to a settings page, export a CSV, upload it to another service. For these tasks, GhostPilot includes a scripts engine that executes JSON macro files.

Scripts use the same REST API primitives as AI agents. This means you can have an AI agent generate a script from a natural language description, review and edit the script, and then run it repeatedly without AI involvement. The AI handles the creative work of figuring out the flow; the script handles the repetitive execution. Scripts are version- controlled, auditable, and deterministic.

Use Cases

GhostPilot is designed for any scenario where a human or AI needs to interact with a web application through a browser. Here are the patterns it is built to handle.

UX Development and Design Review

Tell the AI to navigate through every page of your application and describe what it sees. Use it to verify that a design system is applied consistently, that responsive layouts work at different viewport sizes, or that accessibility labels are present. The AI sees the rendered output, not the source code, so its feedback reflects the actual user experience.

QA Automation

Describe test scenarios in natural language: “Go to the signup page, fill in the form with a test email, submit it, and verify you land on the welcome page.” The AI executes the flow while you watch. No selectors to break. No test scripts to maintain. Record the session to generate visual test evidence for compliance teams.

Demo Recording

Walk the AI through your product flow and record it. The recording studio produces clean GIF, MP4, or WebM output at consistent quality. Secret redaction ensures no credentials appear in the recording. Use the output for sales decks, documentation, or marketing content.

Data Entry and Form Filling

For workflows that require filling out forms across multiple web applications — compliance submissions, vendor onboarding, procurement requests — the AI can handle the mechanical work while the human supervises. The human provides the data and intent; the AI does the clicking and typing.

Onboarding Flow Testing

Test your application's onboarding experience from the perspective of a new user. The AI has no prior context about your application, so its interaction patterns reveal genuine UX friction that your team has become blind to through familiarity.

Enterprise Security

GhostPilot is a component of the platform, which means it inherits the platform's security architecture. In an enterprise deployment, GhostPilot sessions operate within the same governance framework as every other capability of the platform.

Credential isolation: Browser sessions can be pre-authenticated through the credential isolation layer. The AI never sees login credentials. It starts with an authenticated session and interacts with the application in that context. Credentials are managed by the platform's vault, not by the AI agent.

DLP scanning: All data flowing through GhostPilot sessions is subject to data loss prevention rules. If an AI agent attempts to extract data that violates DLP policies — customer records, financial data, source code — the action is blocked and logged.

Audit trails: Every GhostPilot session produces a complete audit log. Every URL navigation, every click, every keystroke, every API call. The audit log includes timestamps, the actor (human or AI), the action taken, and the result. This audit data integrates with the platform's audit system for centralized compliance reporting.

Architecture: Why Puppeteer-core, Not Playwright

This is a question we get often. Playwright is widely regarded as the more modern browser automation library. It supports multiple browsers, has better auto-waiting, and offers a richer API surface. So why Puppeteer-core?

The answer is CDP screencast. GhostPilot's core value proposition is the live video stream from the browser to connected clients. CDP's Page.startScreencast method delivers a continuous stream of rendered frames as JPEG data with configurable quality and framerate. This is a Chrome-specific protocol feature that Playwright abstracts away behind its cross-browser API layer.

Puppeteer-core gives us raw CDP access. We can subscribe to Page.screencastFrame events, acknowledge frames to control flow, and stream them directly to WebSocket clients. There is no intermediate buffering, no screenshot polling, no frame reconstruction from DOM state. The browser renders, CDP delivers the frame, we relay it. The latency from render to client is measured in milliseconds.

Playwright could theoretically support this through its CDP session API, but it would be fighting the library's design. Playwright wants to manage the browser lifecycle, intercept network requests, and handle element waiting. We want a raw protocol connection to a Chromium instance. Puppeteer-core is designed for exactly this use case.

Platform Integration

GhostPilot is not a standalone tool. It is a capability within the platform that integrates with the other modes and services.

ChatMode integration: Users can invoke GhostPilot sessions from ChatMode with natural language. “Open our staging environment and check if the new dashboard is rendering correctly” spins up a GhostPilot session, navigates to the staging URL, and returns the AI's assessment along with a recording.

Workflow integration: GhostPilot sessions can be steps in the platform's workflows. A workflow might deploy a new build, trigger GhostPilot to run through critical user flows, evaluate the results, and notify the team — all as an automated pipeline with human-in-the-loop oversight at every stage.

OATS integration: When a GhostPilot session needs to interact with a service that requires custom tooling, OATS can synthesize a tool on demand. The AI might need to download a file, parse its contents, and fill in form fields based on what it finds. OATS handles the file processing; GhostPilot handles the browser interaction.

What GhostPilot Is Not

GhostPilot is not a replacement for unit tests or integration tests that run against APIs. It is not a web scraping framework. It is not a headless browser for server-side rendering. It is not trying to be a general-purpose browser automation library.

GhostPilot is shared browser control for humans and AI. It is for the workflows that require a browser because they require a visual interface: testing user experiences, filling out web forms, recording demos, and operating web applications that do not expose APIs. It does one thing — shared, live, secure browser control — and it does it well.

Getting Started

GhostPilot is available as part of the platform. If you are already using the platform, GhostPilot is accessible through ChatMode, the Workflow engine, or directly via its REST API.

If you are evaluating GhostPilot for your team, the key question to ask is: how much engineering time does your organization spend maintaining browser automation scripts? If the answer is “more than zero,” GhostPilot is worth a look. Not because it makes script maintenance easier, but because it eliminates the need for scripts entirely.

Reach out at hello@agenticwork.io or visit agenticwork.io to learn more.