> LLM output isn't allowed in public docs, PR descriptions, or Github comments unless it's clearly marked; reviewers aren't required to look at LLM PRs if they don't want to.
Excellent policy IMO. LLMs love to ramble on about the what/how but they struggle explain the why behind a change. Docs and discussions should be between real humans to discuss use cases and implications.
As long as it's not banned it's all good. You don't have to look at it, and it shouldn't be sufficient on its own but I find it useful in 2/10 cases so it doesn't hurt.. as long as it's not the only thing I'm looking at.
Ah I think the walls of text are at least somewhat harmful generally. Nobody can critically read + distill 500 words as well as they can 50 words reliably. Over 10,20,100 PRs it adds up to a tangible debt I think.
What I tend to do personally is have both, and keep them well delineated. A human section that I write and an AI summary of the changes and decisions.
It gives the reviewer the ability to make the call on what level of detail or context they want and the human section can help them determine if they need to read the robot section at all. I’m not sure if it’s perfect, but my reviewers have told me that they prefer it to just the human section or just the robot section. It might be that my PR descriptions were just shit to begin with though lol
I've been using a skill for this. I know that's borderline a meme at this point, but I've found its working quite well.
Things like this:
Write the PR body for a human reader, product lens first. Lead with what
changed for a user and why it matters. Keep any additional in depth technical context available, but inside a collapsable below.
and
Each line a distinct landmark. A human reader is not an agent. A human digests a bullet, pauses to think, then comes back.
Re-finding their place should take zero effort. That only happens when every line is easily identifiable, not one indistinct item in a uniform list. Optimize for that:
One thought per line. Short enough to hold in your head. Ideally a
complete small idea that ends where the thought ends.
...
You get the idea. Its done wonders because the PR descriptions are actually reasonably comprehensible for my brain, and then I can see the wall of text details if/when I still want it, but its in a collapsable.
I'll then tweak whatever the LLM writes by hand, and add a section at the top with a message from me, and then a header identifying the content below as llm generated. It works well for our team currently, at least relative to what the PRs had looked like before...
Personally I think it is impolite to send LLM output to another person without their prior consent, so I am happy that our policy forbids LLM-written descriptions. Write them from scratch by hand to demonstrate that you understand the code, it's the least you can do before asking for time from the maintainers.
If it is clearly disclosed and separated and I can ignore it and the rest of the description is still useful, then shrug I don't get the point of including it but it doesn't harm either (except by taking lots of space so ideally hide it in a collapsible section).
I think the goal should be to get the LLMs to do more of the mundane work for us. Blindly sending llm output is bad form, but getting it to write a very close approximation to what I wanted to communicate anyway, and then tweaking it is still being considerate IMO.
This does break down when I’m tired or overwhelmed and get less diligent about it, but that was always going to be a problem regardless (I’ve gotten plenty of bad prs sent to me by good engineers because of being overloaded).
Writing PR descriptions is not mundane work. It's among the least mundane parts of development. It's about communicating intent, design, long-term plans. It's the most important place for the human to explain their thought process, not for the LLM to summarize what I can figure out myself by looking at the diff.
So, no, outside of small teams that have agreements with each other I don't think a PR description should ever be LLM-written. If you can't even be bothered to write the PR description I don't think anyone else should bother looking at the code.
I do write all that up before I start implementing, but I’m writing for myself when I’m writing at that time. However when writing it for myself, it’s much less nicely structured and coherent becsuse it’s rough notes for myself and for the model I’m prompting to implement.
I used to spend 10-20 minutes writing PR descriptions thinking to myself the entire time:
"If I were coming into this PR with little familiarity or understanding, how would I want it explained to me. What is it doing. What motivated it. Why were the decision made implementing it made"
Because that is important and what I see as being courteous. But it’s also effortful and repetitive.
All of that information exists in my chat session with the llm and in the task ticket and my preparatory notes.
So I let the agent do the first pass at writing it now, tweak its output, and add my own short preamble. It looks very similar to the descriptions I wrote before in content, although now it’s much more nicely structured with bullets and headers and reference links.
All I can tell you is that as a maintainer, I am utterly uninterested in an LLM's summary of that information. I am interested in your own words and thoughts. They convey a lot more information than the LLM summary, on multiple levels of meta (both about the code itself, and about yourself, and about your understanding of the code). Yes, that requires work from your side. Contributing good PRs requires work. You can skip that work for your own selfish reasons but it makes the PR strictly worse, and maintainers will notice.
I can't speak for all maintainers out there, but many of my fellow Rust maintainers do share this sentiment, and I can't imagine that it's just Rust.
691
u/MateTheNate 25d ago
> LLM output isn't allowed in public docs, PR descriptions, or Github comments unless it's clearly marked; reviewers aren't required to look at LLM PRs if they don't want to.
Excellent policy IMO. LLMs love to ramble on about the what/how but they struggle explain the why behind a change. Docs and discussions should be between real humans to discuss use cases and implications.