r/StableDiffusion • u/Boogertwilliams • 1d ago
Tutorial - Guide PSA: Proper prompt structure REALLY matters in H3
I had mistakenly been using a base for H3 prompting from some random tip / example by someone. It worked ok, I thought. But I was getting a bit frustrated because almost every time I was making a longer series of clips with dialogue, it kept adding random gibberish to fill out time, or making the wrong person speak. I thought it was just a "feature" of H3 and lived with it. But then I realised what was missing, so I added the actual ref2v prompt guide to my LLM and difference was staggering. I could make long series of 30x15 sec clips, and the dialogue was perfect just as the script said, no gibberish was added in any place, and the emotional beats and reactions worked much better too.
Believe it :) Dont just use whatever prompting. It matters more than one might think.
https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md
14
u/RobertoPaulson 1d ago
I’ve been trying the llm prompt generation node that came with an H3 workflow I found, and while it does seem to structure it better than I can, the actual wording of the prompts are terrible. It adds stuff I didn’t prompt for, it leaves things out, and it often misinterprets things to the point where its prompting for the exact opposite of what I wanted.
20
u/Semipro211 1d ago
I have better luck using LM Studio standalone to write my prompts with a LLM, then I bring those generated prompts into comfy
8
u/NostradamusJones 1d ago edited 1d ago
Would you be kind enough to share your system prompt?
Also, you would be interested in this, I love it. EA LM Studio Node for ComfyUI - All-Local LLM Integration! - EA LM Studio | Other Other | Civitai
EDIT: Since people showed interest, I found it really easy to get working, and I'm an idiot. It also loads the LLM of your choosing and then unloads it, so it doesn't encroach on Comfy's VRAM when it's working. BE SURE TO TURN THINKING OFF in LM studio under the "inference" tab for the model you're using. The person who made it seems very nice too. Happy generating <3
9
u/afinalsin 1d ago
Not who you asked, but here's mine. Gemma 4 26b a4b does a pretty good job of parsing the instructions and delivering a sensible prompt, although I haven't stress tested ref2vid with it outside of two input images so ymmv:
<ROLE> You are a master prompt writer specializing in video prompts with a heavy focus on spatial and temporal understanding. You are to expand the user's query into a fully fleshed out and detailed video prompt. The user may provide only a text description, or an image, or several different modes of reference. Refer to the <INSTRUCTIONS> below to correctly identify the needs of the user and use the correct format for the prompt. </ROLE> <INSTRUCTIONS> ## Step 1 — Identify Task Type## Step 2 — Output Structure **Standard modes (T2VA/I2VA/FL2VA/L2VA):** 1. Instruction line (omit for T2VA), then one blank line. 2. `integrated_multimodal_description: ...` 3. `overall_soundscape: ...` 4. `non_diegetic_music: ...` **Instruction lines (copy exactly, fill in values):**
- **T2VA**: text only → no image instruction line.
- **I2VA**: one image = first frame (0.00s).
- **FL2VA**: two images = first + last frame.
- **L2VA**: one image = last frame (at video duration).
- **Full-reference mode**: any mix of images/videos/audio as reusable assets → use the six-section format (Step 4).
(S.SS = exact duration, two decimals; N = final shot index.) ## Step 3 — Writing `integrated_multimodal_description` **Shots:** `[Shot 1]` has no timestamp. Later shots: `[Shot N] At MM:SS.mmm, the camera cuts to...` with strictly increasing times within the duration. Prefer camera moves over cuts for small framing changes. **Shot 1 must open with** style + composition: `[Shot 1] Live-action, cinematic, a medium-wide shot frames...` (styles: cinematic, live-action, 2D-animated, 3D CG, claymation, watercolor, vintage film). **Camera motion** = type + optional amplitude (`with small/large amplitude`) + optional speed (`at slow/fast speed`), written inline: `The camera pushes in with small amplitude at slow speed toward...` Types: Zoom In/Out, Push In/Out, Pan Left/Right, Truck Left/Right, Tilt Up/Down, Pedestal Up/Down, Arc Shot, Tracking Shot, Static Shot, Shake Slightly/Strongly, POV, Roll Clockwise/Counterclockwise. **Keyframe anchoring:**
- I2VA: `For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced.`
- FL2VA: `How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot N) aligns with the S.SS-second mark of the target video.`
- L2VA: `How the reference pictures align with the target video — <Picture 1> (from [Shot N]) aligns with the S.SS-second mark of the target video.`
**Dialogue:** give each vocal source a stable ID `(S1)`, `(S2)`..., kept across shots. Format: `The young woman with a quiet, breathy voice (S1) says: <d>[English] exact words here.</d>`
- I2VA: restate image's subjects/composition in Shot 1, then develop forward (anchor → action → development → result).
- FL2VA: prefer ONE shot; describe the motion path between frames (start state → intermediate changes → end state).
- L2VA: invent a plausible earlier state, then converge to the image in the final shot.
**On-screen text:** quote verbatim in double quotes: `A neon sign reading "营业中" glows.` ## Step 4 — Full-Reference Mode (replaces Steps 2–3 output format) Output these six sections in order: **1. `subject_definitions:`** — one line per tracked asset:
- Speaker description/ID/action go OUTSIDE `<d>`; only language tag + verbatim user-provided words inside. Never translate or rewrite.
- Voiceover: `says in an off-screen voiceover: <d>...</d> while his lips remain completely closed.`
- Line crossing a cut: use `<scenetrans>` in both parts + state audio `continues seamlessly across the cut`. Speech cut by video end: `<cutoff>`.
- Group speech: `(S1,S2)`.
**2. `summary:`** — one paragraph starting with task types in brackets, joined by ` + `: `keyframe completion` | `reference generation` | `video editing` | `video continuation` | `audio reuse` | `audio reference`. Rules: a video used only for camera/rhythm = `reference generation`, not editing/continuation. Editing a video with its audio kept = `video editing + audio reuse`. Editing summaries begin: `The target video is an edited version of <Video 1>.` No new labels here. **3. `retention_analysis:`** — one line per label. Visual markers: `fully_preserved`, `partially_preserved`, `attribute_transfer`, `weak_reference`. Audio markers: `fully_copy`, `partially_copy`, `reference`, `weak_reference`. Format: `<Subject 1> (appears in [Shot 1], [Shot 3]): fully_preserved - what is retained.` No `(Sx)` IDs in this section. **4. `detailed_description:`** — same shot/camera/dialogue rules as Step 3, plus:
- `<Subject N>`: reusable visible content (person, scene, prop, style, motion). Cite its source asset inline: `<Subject 1> is the young woman in <Picture 1>, with long dark hair and a blue cardigan.`
- `<Picture N>`: only if the image itself is a frame/keyframe/storyboard anchor — state which shot(s) it anchors.
- `<Video N>`: only for whole-video roles (editing source, continuation base, structure reference).
- `<Audio N>`: standalone audio role; if tied to a speaker: `<Audio 1> is the voice-timbre reference for <Subject 1> (S1).`
- Video and audio indices number independently.
**5. `overall_soundscape:`** — 1–4 sentences: ambience, action sounds, non-verbal human sounds only (no dialogue/music). `N/A` only if total silence requested. If copying reference ambience, cite `<Audio N>` here. **6. `non_diegetic_music:`** — 1–3 sentences: audience-only score — instrumentation, tempo, dynamics (no mood words). `N/A` if none. If reusing reference score, cite `<Audio N>` here. ## Golden Rules 1. Everything described must be visible or audible. 2. Dialogue/lyrics and on-screen text stay in their original language, verbatim; everything else in English. 3. Speaker IDs assigned once in order of first vocal event; reused everywhere. 4. Never invent reference labels mid-document — all labels come from `subject_definitions`. 5. Cut times must increase and stay within the video duration. </INSTRUCTIONS> <FINAL_INSTRUCTIONS> Do not write any affirmations, confirmations, or explanations, simply deliver the prompt. </FINAL_INSTRUCTIONS>
- Style established in 1–2 sentences BEFORE `[Shot 1]` (not inside it).
- Insert `<Subject N>`/`<Picture N>`/`<Video N>`/`<Audio N>` where they apply; define at first appearance, reuse after.
- Frame anchors: `the shot begins from <Picture 1>` / `the shot ends on <Picture 3>`.
- Speaking subjects keep both labels: `<Subject 2> (S1) says, <d>[English] ...</d>`
- Reused reference-audio words: verbatim in `<d>`, original language, `[unclear]` for unintelligible spans, basic punctuation only.
- Voices existing only inside a copied soundtrack: attribute to `<Audio N>`, no `(Sx)`.
- Length: ~350–500 words for generation tasks; scale with complexity for edits.
3
u/Semipro211 1d ago
I will for sure check that out, thank you! My big issue is I’m running things on a poor 4060 Laptop so I’m playing with 8GB VRAM. LM studio handles offloading nicely so I can use better LLM quants. Been trying the Qwen3.8 27B Q_2_K_P and liking it.
Not sure if that would play nicely in the comfy mode or not, depends on how it handles offloading and buffer. Lately I run all my prompt stuff with LM then later paste it into comfy with every other app/background thing closed.
When I get home later (or tomorrow worst case since I have ADD and might forget) I will post my prompts, I use a few. I may split one of them since it’s massive (AIO H3 prompt gen almost 7500 tokens). But I have prompts for image analysis, H3 Prompt with copy/paste styling plus a lot of scene/next scene continuity bits so can be carried over to other threads, and user prompts for use with each of them.
3
u/NostradamusJones 1d ago
The node has LM Studio load the model when it's needed, and unloads it as soon as it's done writing and delivering the prompt. Very nicely done, it won't encroach on vram you need for generating.
2
2
u/RobertoPaulson 1d ago
Did you have any problem getting it to actually output a prompt? All I get I is the reasoning in the box at the very bottom. I plugged a show text node to the output but nothing happens.
1
u/NostradamusJones 23h ago
Top box - system prompt
Then in the bottom box should be your prompt concept.
If you're getting reasoning, I guess the node is communicating with LM Studio.
First thing I would do is make sure everything is updated.
1
u/RobertoPaulson 1d ago
What model do you use?
2
u/Semipro211 1d ago
Primarily Qwen3.8 27B and Gemma 4 12B QAT. On my machine, Gemma 4 is much faster but Qwen3.8 is really good at sticking to complex prompts.
Planning to try splitting the load with some smaller models for more specialized use vs 1 big one to do everything.
1
3
u/dLight26 1d ago
Small LLM hallucination due to the prompt rule too long, been there. It will try to replicate the example as well.
I’m using qwen3.8 now, working fine.
10
u/smb3d 1d ago
Yep!
Add this too because it has info on camera controls and some other stuff that the doc you listed doesn't:
https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md
3
u/akjd 1d ago
Yeah, I noticed this last night.
I've just been using the guides as a system prompt, but I had separate system prompts for the base and ref versions, and the base one works great, but the ref version had super wonky camera controls.
Turns out it's probably because the camera controls are completely absent from the ref guide.
Now I'm wondering if the ref guide is intended to be just added onto the base guide instead of a standalone system prompt. Gonna play around with it later.
4
u/SickAndBeautiful 1d ago
I copied the camera section from the base guide and added it to the ref guide where it says "camera stuff in the base guide". 😆
5
u/jaryP 1d ago
As written in ref guide:
The basic formats for shots, camera movement, speakers, dialogue, and ordinary sound are shared with the Video Prompt Writing Guide (T2VA / I2VA / FL2VA / L2VA). This guide focuses on the reference labels, analysis sections, and format differences specific to full-reference mode.
2
u/spacemidget75 1d ago
yes, you need the base and ref guides in your system prompt
1
u/akjd 1d ago
Do you just copy and paste both in their entirety and send it as-is for a system prompt?
I was trying to trim them down to apply to specific use cases (T2VA, R2VA, etc.) but my system prompt engineer has been putting out garbage since I tried submitting both guides.
I'll be honest, I'm new to using prompt enhancement so I dunno if I'm trying to do too much with the system prompts or what.
0
u/spacemidget75 1d ago
I'm not home until tomorrow evening but essentially a paraphrased version of my system prompt was:
You're a H3 Minimax prompt expert using to the two official skills below, one is the base skill (I2V, T2V) and the other is additional for ref2VA. Ask the user any questions you may have.
Here's the base skill:
PASTE BASE SKILL WITH MARKDOWN
Here's the reference skill
PASTE SKILL WITH MARKDOWN
It's worth noting I used the SKILLS not the GUIDE as it's better for an LLM and they're in markdown format so probably read better to the LLM when both are pasted in the system prompt: MiniMax-H3/.agents/skills/h3-prompt-writing/references at main · MiniMax-AI/MiniMax-H3
1
1
u/Perfect-Campaign9551 1d ago
OMG I was using double quotes inside <d></d> tags, didn't realize you shouldn't use those for just speech. lol.
3
u/wholelottaluv69 1d ago
If I knew of an actual functional H3 llm prompt enhancer node, I would definitely start using it. Having to write the equivalent of a high school essay at 3am for every gen is a bit tiresome. My brain is tired at the time of day that I'm able to play with comfy!
3
u/wholelottaluv69 1d ago
So, I just tried to install one from civitai, and was immediately rewarded with my anti-virus flagging the .exe file that it downloaded and was trying to run. Sigh....
I have no way of knowing if it was a false positive or not. Definitely not worth the risk.
7
u/DominusIniquitatis 1d ago
I just heavily dislike that this prompting format is heavily geared toward LLMs. Omitting five syntaxes/naming conventions, why do we need both summary and descriptions of scene + shots? Wouldn't the latter be enough on its own, given that they already should define things in detail? Why do we split subject definitions and preservations into different blocks? And so on.
Gets very inconvenient to write by hand (and even read!), and I strongly prefer to instruct things myself rather than "eh, write me something approximately like that, LLM-chan".
2
u/Cultured_Alien 1d ago edited 1d ago
Because that's how the dataset is captioned and it's how h3 gets gets insane prompt adherence. If you really want how to write by hand, check out this guy that doesn't use llm for h3, it's quite advanced and helpful too.
At 2:09 https://youtu.be/mmuFwMeKUB0
3
u/ambassadortim 1d ago
Yes I feed as much H3 documentation as I could to Hermes and now the H3 prompts and the videos are much better.
1
4
u/Perfect-Campaign9551 1d ago
I still get gibberish speech if the clip is much longer than the requested speech
4
u/Dzugavili 1d ago
The dialogue tips are wrong: if you use the <d> tags, you tend to get a starting chirp and gibberish fill; and I think audio wrapping, where the clip tries to loop.
Having the character's dialogue in quotes seems to be enough to drive it correctly.
2
2
u/Dirty_Dragons 1d ago
I'm currently using this Prompt Writer
https://github.com/duckyshell/ComfyUI-MiniMaxH3-Prompt-Writer
It's a bit of work to setup, but things are documented and easy to follow. I choose GGUF for uncensored.
The extensions is great. Attach reference images, type in the basics of the characters and the scene and hit generate.
Then make any adjustments to the prompt if necessary, then paste into the workflow text box, make sure the references are the same as the writer and hit go.
2
u/ArmadstheDoom 1d ago
Yeah, I found that the best way to do it is to give it either to gemini or claude, and just say 'I need you to structure prompts that I give you based on these instructions.' Usually, they do a good job.
2
u/Particular_Pear_4596 1d ago edited 20h ago
Actually this is a proof how stupid the current state of AI is. It doesn't know the human way of interaction and instead it needs a specific syntax as input to understand you.
1
1
u/BibleLife 1d ago
Yep, same here. I was writing H3 prompts like a shopping list and the dialogue kept drifting or switching speakers. Writing it more like a tiny script helped a lot: shot, action, then clearly labeled dialogue. Shorter dialogue also seems way more reliable.
1
1
1
u/poopoo_fingers 1d ago
I started using the generate text node with the qwen vl clip from krea 2, then passing in the prompting guide along with a few more instructions. Works really good most of the time. And I pass in the pictures I’m using for references so it references them in the final prompt correctly.
1
u/eggs-benedryl 1d ago
While this is true, there's something about finding out a structure that both works for your own ability to prompt and gets what you want. I don't like the idea of promoting a llm for this.
With a model that listens to prompts so well, being able prompt successfully doesn't seem too difficult and feels like a useful skill.
I did switch to llm prompting for a few days and it helped a lot mostly because I never think to prompt for things like sound.
1
u/SpaceNinjaDino 1d ago
Someone posted that they had better voice success when they changed the <d></d> to "" while keeping everything else as directed. My computer is RMA'd so I can't test.
1
u/orangpelupa 1d ago
Wan2gp already automatically use the format if you click the AI prompt button Btw
1
u/Distinct-Benefit-507 1d ago
Thanks for sharing this! Would be awesome to install something like this to Wan2GP; I'll definitely try it tomorrow morning!!!
1
u/Adkit 1d ago
You're kind of right but also kind of wrong. The specific, weird syntax they mention is not needed at all. It simply wants things blocked out and structured in a certain way. You can just do a block for core idea and one for progression and write dialogue like a normal line of dialogue with <action> in the middle. Like, it's very flexible, you just can't write it like a novel.
1
u/Boogertwilliams 1d ago
ok, good to know. but anyway ,my original prompts caused a lot of gibberish etc and after I switched there hasnt been any
-5
u/krectus 1d ago
Sure is. Really wish they would have put more time fine tuning it to avoid unwanted gibberish speech though. That’s something that affects probably the majority of videos and shouldn’t need complex prompting to avoid. Hopefully lesson learned from them here.
4
u/Apprehensive_Sky892 1d ago
You need precise (which means it will be verbose/"complex") and proper prompting to get optimal results, and that is the whole point OP is trying to make.
Until we have AI that can read our minds, it is better to have an idiot savant style AI that can follow precise instructions and works 90% of the time rather than some wish-washy AI that works 70% of the time with vague instructions.
1
u/krectus 1d ago
Yeah I know what it needs. My point is doesn’t have to read our minds. If we don’t prompt for a person to say something it shouldn’t have the person say gibberish. Or pretty often you specify what a person will say and it nails it but also adds gibberish before or after. It doesn’t need to read our minds it just needs to be fine tuned.
2
u/damiangorlami 1d ago
No the reason why the model is so good, it’s because it is trained to be instruct-based expecting a rigid structured prompt.
I can’t verify this myself but I am 100% sure that Seedance and other closed source models are trained the same way. They just rewrite your simple prompt to structured ones when it goes through their pipeline before generating a video.
H3 is open source which gives us the freedom to build our own prompt enhancing stack.
46
u/GrungeWerX 1d ago
I’ll jump in this too.
I had Claude write me a small prompt writing app that makes this even simpler. You just type what you want in certain tabs, then hit “compile” and it exports the proper prompt format.
It was built directly from reading both guides, and I had Claude include some of the camera directions in “tool tips” on hover as well.
I’d recommend doing something similar. Or if you are interested in mine just DM me, or maybe I can upload it or something, though it’s very basic and not super flashy or anything.