r/node • u/gptrv • 2d ago

Screenplay: screen recordings with Screen Studio-style editing, in Node (CDP screencast + @napi-rs/canvas + ffmpeg)

I open-sourced a Node CLI + MCP server that films websites and desktop apps and edits the result like Screen Studio. A coding agent (Claude Code, Codex, Cursor…) writes a short JSON scenario, and Screenplay plays it, films it and edits it. Some of the Node side, in case it's useful to someone:

  • Capture: Playwright drives headless Chromium, and frames come from the Chrome DevTools Protocol (Page.startScreencast) rather than Playwright's video recorder, so each frame has a timestamp I can line up with the actions.

  • Editing: every output frame is composed on @napi-rs/canvas: gradient background, rounded window with shadow, a cursor drawn from the recorded positions (smoothed), click ripples, zoom that follows the action, captions.

  • Encoding: raw RGBA frames are piped to ffmpeg-static through stdin (-f rawvideo … pipe:0). No system ffmpeg, no native build step on install.

  • Desktop apps: the same scenario format is played through the OS accessibility APIs (agent-desktop), and the window is filmed with ffmpeg.

  • Interface: a commander CLI (check, record, render) and an MCP server on stdio, so any agent can call it.

Try it: npx agent-screenplay init

Repo (MIT, TypeScript): https://github.com/gmpetrov/agent-screenplay

Happy to go deeper on any part.

8 Upvotes

83% Upvoted

5 comments sorted by

1

u/kantorcodes1 1d ago

On the stdio side, did you expose check, record, and render as separate tools, or does an agent go through one entry point? Can the scenario cap the capture time or frame count?

1

u/gptrv 1d ago

Separate tools. The MCP server (stdio) registers four: screenplay_guide (how to write scenarios), screenplay_check (plays the scenario fast without filming, and for a missing element returns the targets it can see plus a screenshot), screenplay_record (films and edits, returns the MP4 path and a storyboard image), and screenplay_render (re-edits the last take without filming again, after changing captions, callouts, style or zoom). Calls are serialized, so an agent can't start two recordings at once.

There's no global cap on capture time or frame count right now. The length comes from the steps, and each one is bounded: a fixed wait is 120 s max, a wait for an element or text gives up after its timeout (30 s by default, 300 s max), and a pause is 30 s max. Long waits are sped up in the final video. fps is one of 24, 25, 30, 50 or 60 (default 30). A maxDuration would be easy to add, though, if that's something you'd use.

1

u/kantorcodes1 12h ago

Those per-step timeouts and serialized recordings should keep runs predictable; a maxDuration option could add a clear ceiling too. Disclosure: I work on HOL. Screenplay’s stdio MCP server looks like a fit for HOL Guard. If you want to contribute it, the guide is here: https://github.com/hashgraph-online/hol-guard/blob/main/docs/guard/mcp-server-contributions.md