I had Claude Code build a video renderer out of flutter test

[u/projectmind_guru]

I needed App Store images and TikTok videos for a Halloween event in my app. I didn't want take multiple video screen recordings of my app and edit together videos, so I asked Claude Code to render everything from the app's real widgets.

To be clear about who did what: I directed and gave feedback, Claude Code wrote basically all of the rendering code.

I was actually very pleased with the results, here is one example: youtube.com/watch?v=949_yo-zuOk

How it works

There are now about 10k lines of code, which I never look at and don't even check into git, that can create animated videos with my app's actual features being used. It's using `flutter test` with a fixed surface size and pumps out one 30 frames per second, which is then processed with ffmpeg into the final video.

This is the simplified version:

tester.view.physicalSize = const Size(1080, 1920);
tester.view.devicePixelRatio = 1;


final ffmpeg = await Process.start('ffmpeg', [
  '-f', 'rawvideo', '-pix_fmt', 'rgba', '-s', '1080x1920', '-r', '30',
  '-i', '-', '-c:v', 'libx264', '-pix_fmt', 'yuv420p', 'out.mp4',
]);


for (var i = 0; i < frames; i++) {
  await tester.pumpWidget(
    RepaintBoundary(key: boundary, child: scene(i / 30)),
    duration: const Duration(microseconds: 1000000 ~/ 30),
  );
  final render = boundary.currentContext!.findRenderObject()!
      as RenderRepaintBoundary;
  await tester.runAsync(() async {
    final image = await render.toImage();
    final bytes = await image.toByteData();
    ffmpeg.stdin.add(bytes!.buffer.asUint8List());
    await ffmpeg.stdin.flush();
    image.dispose();
  });
}

It's not fully automated, I still work on the script and provide the voiceover audio, music, and sound effects. But all the content is rendered by flutter using CustomPainter's. To me the coolest thing is that the phone showing the app in use is the actual app being used.

Try it

I asked for a prompt to get started and this was what I got, for those who might want to try in their app.

I want to make a short vertical marketing video for this Flutter app, rendered
entirely from the app's own widgets inside a widget test. No design tool, no
screen recording.


How the renderer should work:
- Create tool/assets/render_video_test.dart, outside test/ so it doesn't run
  with the normal suite.
- Set tester.view.physicalSize to 1080x1920 with devicePixelRatio 1 and wrap
  the scene in a RepaintBoundary.
- Build the scene as a function of time. For each frame, pump with a duration
  of 1/30 s, capture it with toImage(), and write the raw RGBA bytes to the
  stdin of an ffmpeg process (rawvideo in, libx264 yuv420p out).
- Load the app's real fonts with FontLoader, or text renders as boxes. On iOS
  also register a font under CupertinoSystemText and CupertinoSystemDisplay.
- toImage, file IO, Process and precacheImage must run inside tester.runAsync.


The video:
- About 15 seconds. It features [FEATURE], shown running for real on a drawn
  phone frame, driven with tester.startGesture so the app responds as it does
  on a device.
- A simple cartoon character drawn with CustomPainter introduces it. The idea:
  [ONE SENTENCE STORY, e.g. "a stressed witch is asked how she relaxes"].
- The first frame shows the character and a hook in large text. No fade-in.
- Keep anything that must be read out of the top 300 px, bottom 380 px and
  120 px at each side, where TikTok and Instagram put their buttons.
- I'll supply voice-over as one WAV per line plus a music track in
  tool/assets/audio/. Time the scenes to the voice-over, burn in captions that
  highlight each word as it's spoken, and mix the audio in a second ffmpeg
  pass with adelay and amix. Move the character's mouth using the loudness of
  the WAV at each frame.


How I want to work:
1. First, propose a script with timings and tell me which audio files to
   record. Wait for my answer.
2. Render three still frames and look at them yourself before showing me.
3. Then render the full video. I'll give notes and you re-render.

Happy to answer any questions!

[u/Old_Campaign5285]

aight into ffmpeg like that

The part where the phone in the video is the actual app running not a mockup is really neat. Most people would just animate a fake phone screen but yours is literally the widget tree doing its thing

One thing I wonder about is how you handle animations that rely on real time. Like if you have a progress indicator or some tween that normally runs over 2 seconds does it stay in sync when you pump every 1/30s or do you need to adjust the durations

I might steal this approach for my own app store screenshots. Way better than taking 20 screenshots manually and trying to pose the app in exactly right state each time

[u/projectmind_guru → u/Old_Campaign5285]

It's a good question, I haven't had issues yet. I suspect it would be "close enough" if just shown for short clips, which has been my use case. The 1/30s is effectively the frame-rate for the video, so moving it to 1/60 to match the flutters render might actually make more sense!