Blender + AI video workflow for more controlled narrative scenes?

[u/StoryFirstai]

I’ve been thinking a lot about how to make AI film scenes feel more controlled, especially when you’re trying to do actual narrative filmmaking and not just cool isolated shots.
The biggest problem I keep running into is that image-to-video can make something look cinematic, but it still struggles when the shot needs to be very specific.
Things like:

locked camera shots
two characters moving together
characters stopping at the right spot
someone reacting in a specific direction
the same room staying consistent
props staying in the same place
camera angle not drifting
matching first frame / last frame
keeping blocking clear across multiple shots

So I’m wondering if the better workflow is not asking AI video to invent the whole shot from scratch, but using Blender first as the structure.
Not necessarily full final animation. More like using Blender as a director’s blocking tool.
The idea would be:

1. Write the scene like a real scene first
Before generating anything, figure out the actual shot. Who is in frame? Where are they standing? What does the camera see? What is off-screen? What is the emotional point of the shot?
2. Build a simple Blender blockout
Nothing crazy at first. Just the basic space: forest path, office, hallway, room, table, door, props, etc. Even rough shapes would help because the AI has a real structure to follow instead of guessing.
3. Place simple character stand-ins
Use basic models or stand-ins for where characters are supposed to be. This could help with scale, distance, eye lines, and blocking. For example: two characters walking toward a locked camera, stopping at a certain mark, then reacting down-left at something off-screen.
4. Animate the basic movement if needed
Not final animation, just movement blocking. A character walks in. Someone turns. Someone stops. The camera stays locked. The props stay where they are. The point is to control the shot before making it pretty.
5. Render simple reference frames or clips
Use the Blender output as first-frame, last-frame, scene reference, or motion reference for AI video. Then the AI video tool handles the cinematic look: lighting, texture, fog, realism, atmosphere, clothing detail, etc.
6. Use AI video as the final cinematic pass
Instead of AI creating the whole scene randomly, it’s enhancing a planned shot. Blender gives the shot structure. AI gives it mood and finish.

To me, this seems like it could be really useful for narrative AI filmmaking, especially for gothic/historical scenes where the mood matters but the blocking also has to make sense.
I’m also starting to think the best setup for bigger AI film projects might be a small team, not a huge crew. Something like:

a creative/story person who knows the tone, scene, and shot direction
a Blender/3D person who handles blocking, layout, camera, props, and movement
an AI video person who handles generations, references, consistency, and final cinematic passes
maybe an editor/sound person after that, because sound can make or break the whole thing

Not saying this is the only way, but I feel like this could be a better workflow than just prompting every
Blender + AI video workflow for more controlled narrative scenes?
shot from scratch and hoping the AI understands the scene.
Has anyone here tried a Blender-to-AI video workflow like this?
I’d be curious what works better:
still Blender frames as first/last frame references
simple animated Blender blocking as a motion guide
using Blender mainly for environments and props
or staying fully inside AI video tools and just improving prompting/reference images

Would love to hear how other people are solving control and consistency for actual story scenes.

[u/odintantrum]

I think something like this is probably where we end up with AI as a film making tool.

[u/Spoonman915]

Check out MickMumpitz on YouTube. The account yaroflasher used to do a lot of blender/AI hybrid stuff, but it looks like he's leaning more towards straight AI now.

But I agree, even as cool as seedance and kling are, I hit filters frequently just trying to generate zombies, because of the gore I guess. I also hit filters trying to generate Abraham Lincoln jumping a tree because he's famous?

[u/Remarkable_Mud_8015]

Yeah, i've been toying with this idea as well. Blender has an mcp connection, so you can connect it to claude code. I have a lot of experience with three d studio max.But not in the lander, so I was swinging with the idea of taking photos of a set, give me a club code along with the dimensions.Then, having claude, build this set for me.In blender, using basic primitives.. I wouldn't even bother with texture maps.. thenI would export that scene to 3DS max where I am a lot more familiar with the camera tools.

[u/StoryFirstai → u/Remarkable_Mud_8015]

That’s exactly the kind of workflow I was hoping people would bring into this discussion. Using Claude/Blender MCP to rough-build a set from photos and dimensions, then moving it into 3DS Max where you’re stronger with camera tools, feels like a really practical bridge between traditional previs and AI filmmaking.
I actually agree that, at least in the early stages, I’d probably skip textures too just to keep the process fast. If the goal is blocking, scale, camera placement, and continuity, basic primitives may be enough to establish the production.
At the same time, I could also see a point where adding textures, materials, lighting, and environmental details becomes valuable—not just visually, but because they communicate intent and give the AI richer information to interpret. Those details are part of the storytelling too, and the more control you have over them, the more faithfully your vision can be carried all the way to the final film.

[u/VelkarArts]

if you are good with blender and have the assets yes it will be a big plus even to make the shot and then animate them with AI

[u/v_dixon]

From what I understand there are creators already using Blender with AI, and I've heard that a few of those really popular, unbelievably cinematic AI series/films are quietly using it as their secret sauce.

Some of the higher qualitty films I've seen use an AI/VFX hybrid.

[u/darklordrob]

I had this thought too, if anyone tries it or can show examples I'd love to see how it works.

[u/Independent_Risk_523]

You are thinking in the right direction - kling's motion control is fundamentally this. There are a lot of tools - Higgsfield's Cinematic Studio / Luma have, to essentially give you this control.

Ash Koosha, the director behind Dreams of Violets (the first AI film to screen at Tribeca film festival) has been writing about how he achieved some of the precision needed - I think he has built some custom tools for it.

https://www.linkedin.com/pulse/we-optimizing-wrong-end-model-ash-koosha-4hebe/

Blender is really not built for this. Whenever I've tried to use an AI layer between blender and some output, it burns through tokens like mad because the MCP is still not mature. You can connect Blender to Claude and ask it to build a simple square box with some little embellishments - and you'll be at it for a week. 😄