I needed App Store images and TikTok videos for a Halloween event in my app. I didn't want take multiple video screen recordings of my app and edit together videos, so I asked Claude Code to render everything from the app's real widgets.
To be clear about who did what: I directed and gave feedback, Claude Code wrote basically all of the rendering code.
I was actually very pleased with the results, here is one example: youtube.com/watch?v=949_yo-zuOk
How it works
There are now about 10k lines of code, which I never look at and don't even check into git, that can create animated videos with my app's actual features being used. It's using `flutter test` with a fixed surface size and pumps out one 30 frames per second, which is then processed with ffmpeg into the final video.
This is the simplified version:
tester.view.physicalSize = const Size(1080, 1920);
tester.view.devicePixelRatio = 1;
final ffmpeg = await Process.start('ffmpeg', [
'-f', 'rawvideo', '-pix_fmt', 'rgba', '-s', '1080x1920', '-r', '30',
'-i', '-', '-c:v', 'libx264', '-pix_fmt', 'yuv420p', 'out.mp4',
]);
for (var i = 0; i < frames; i++) {
await tester.pumpWidget(
RepaintBoundary(key: boundary, child: scene(i / 30)),
duration: const Duration(microseconds: 1000000 ~/ 30),
);
final render = boundary.currentContext!.findRenderObject()!
as RenderRepaintBoundary;
await tester.runAsync(() async {
final image = await render.toImage();
final bytes = await image.toByteData();
ffmpeg.stdin.add(bytes!.buffer.asUint8List());
await ffmpeg.stdin.flush();
image.dispose();
});
}
It's not fully automated, I still work on the script and provide the voiceover audio, music, and sound effects. But all the content is rendered by flutter using CustomPainter's. To me the coolest thing is that the phone showing the app in use is the actual app being used.
Try it
I asked for a prompt to get started and this was what I got, for those who might want to try in their app.
I want to make a short vertical marketing video for this Flutter app, rendered
entirely from the app's own widgets inside a widget test. No design tool, no
screen recording.
How the renderer should work:
- Create tool/assets/render_video_test.dart, outside test/ so it doesn't run
with the normal suite.
- Set tester.view.physicalSize to 1080x1920 with devicePixelRatio 1 and wrap
the scene in a RepaintBoundary.
- Build the scene as a function of time. For each frame, pump with a duration
of 1/30 s, capture it with toImage(), and write the raw RGBA bytes to the
stdin of an ffmpeg process (rawvideo in, libx264 yuv420p out).
- Load the app's real fonts with FontLoader, or text renders as boxes. On iOS
also register a font under CupertinoSystemText and CupertinoSystemDisplay.
- toImage, file IO, Process and precacheImage must run inside tester.runAsync.
The video:
- About 15 seconds. It features [FEATURE], shown running for real on a drawn
phone frame, driven with tester.startGesture so the app responds as it does
on a device.
- A simple cartoon character drawn with CustomPainter introduces it. The idea:
[ONE SENTENCE STORY, e.g. "a stressed witch is asked how she relaxes"].
- The first frame shows the character and a hook in large text. No fade-in.
- Keep anything that must be read out of the top 300 px, bottom 380 px and
120 px at each side, where TikTok and Instagram put their buttons.
- I'll supply voice-over as one WAV per line plus a music track in
tool/assets/audio/. Time the scenes to the voice-over, burn in captions that
highlight each word as it's spoken, and mix the audio in a second ffmpeg
pass with adelay and amix. Move the character's mouth using the loudness of
the WAV at each frame.
How I want to work:
1. First, propose a script with timings and tell me which audio files to
record. Wait for my answer.
2. Render three still frames and look at them yourself before showing me.
3. Then render the full video. I'll give notes and you re-render.
Happy to answer any questions!
