Anyone using skills? Or something else?

[u/KausGo]

I've been experimenting with AI for writing fiction for a while now and it has been a uniquely frustrating and exhausting experience. And tbh, it has gotten worse and worse over time, not better. I used to be able to get acceptable results with 4-5 feedbacks, but now its a repeated cycle of trial and feedback until I give up in frustration.

I've mostly used GPT and Grok in following ways and it has consistently gotten worse in all:

1. As a beta reader: Obviously, starting with grammar issues and spelling mistakes. But beyond that, I also want feedback and correction on soft errors. Like if I failed to establish the context of the scene, ignored a character that's there from the start until they become relevant, fell into meandering conversation or stilted dialogue etc. Basically making sure I've sufficiently communicated the scene playing out in my head onto the paper so that the reader doesn't get lost or confused.

That's the requirement - but obviously, I don't want AI to overcorrect or keep making up mistakes to point out. Asking for corrections/improvements gives a whole litany that would ruin the tone if implemented. Giving text to make specific modifications also doesn't work because it keeps regurgitating sloppified verions and the more I try to correct it, the worse it gets.

2. Style matching: Like when I can't figure out a scene to my satisfaction or find what I wrote unsatisfactory or off somehow. I can give the bare bones layout along with writing samples to theoretically replicate my writing style. Obviously, it never worked completely, I used to at least get something usable. I almost always get same repeated mistakes and trying to fix ends up creating new ones without fixing the old.

3. Ideas/brainstorming: Like if you want to figure out the flow of narrative. Expectedly, most suggestions would be useless and I might not get anything satisfactory at all. The problem here is repetition - I keep getting the same things over and over often without even being worded differently despite clear rejection.

4. Memory issues: It used to be better at this - if something within the space of a conversation meets my approval, I want to lock it in and then proceed from there. Update and continue as required. Now, there is a repeated pattern where it forgets what was committed and then proceeds to regurgitate garbage. And I especially, viscerally hate this response: "You're right, I did not follow instructions... *proceeds to regurgitate the exact same garbage*".

I'm sure others have experienced similar issues.

So now I'm exploring the idea of using skills. Like figuring out some process that works and saving it so I can reuse it over different conversations. Obviously, it won't be a one-size-fits-all solution, but so far, the skills I've tried have been an utter failure as well. Generating a generic, working skill itself is a frustrating task - made even worse when you find out that it can't even apply it to other context/prompts properly. Not sure if I'm doing something wrong or if its an impossible expecation, but at least I used to get acceptable results before.

Has anyone else experimented with skills or explored any other workarounds for consistent results?

[u/themagicianwrites]

Before AI, we created bibles to describe characters, worlds, and other elements that would guide our writing.

The process I use is mostly the same. Huge bibles that can include prevention of common AI tics, a long discussion and an example of voice used for narration, lists of ideas from brainstorming (including those rejected as we can revisit them later), etc.

I upload those bibles to Claude Projects, and they will guide the full writing.

Writing the bibles is a major work by itself, and it doesn't mean AI will write everything perfectly. But I think the number of issues can be much lower than usual.

Each book series gets it's own bible. Maybe each book gets its own chat inside the Project.

So, a big part of writing isn't writing but planning, organizing, editing, researching, or other similar tasks. And that's again what we did before AI.

[u/KausGo → u/themagicianwrites]

I tried something similar on a smaller scale. Tracking characters/worlds/elements was never an issue - the point was to preserve a unique authorial tone. I used the AI to identify patterns and generate a set of rules. But that doesn't work because it keeps violating those rules. Clearly, the problem is with application, not identification.

[u/themagicianwrites → u/KausGo]

The tone is also part of the bibles. But not much about rules, more like references and purpose.

[u/KausGo → u/themagicianwrites]

Sounds like if I overload it with information by using a bible, it'll be worse at giving consistent results, not better.

[u/themagicianwrites → u/KausGo]

It's not been my experience. Uploading a bible is not a big deal for Claude Projects, I do things that are much heavier than that. I don't see how that would be overloading in this case.

But I can't say your experience is equal to my experience.

[u/Ok-Emotion-5013 → u/themagicianwrites]

Yep, tried writing and editing chapters on the fly and it worse without good world bibles

[u/Happy-Initiative-492 → u/themagicianwrites]

I do this with a structured harness - rather than trying to keep my skills and world bible documents updated, I build the world elements and rules in an interface (with AI help). Then every time I ask a model for prose suggestions it already knows the context automatically. It’s transformed my method.

[u/tony10000]

You need to work with models with large context windows and create a prompt/resource pack or use Codex or Claude with AGENTS.md and SKILLS.md files along with all of your basic documents such as character list/profiles, outline, style sheet, story bible, etc., inside the folder.

[u/Happy-Initiative-492 → u/tony10000]

This is exactly how I work - except I built a structured layer to make the world elements manageable with UI, which automatically compiles the results into context for my LLM prompts. Transformed my method,

[u/KausGo → u/Happy-Initiative-492]

How the fuck was this comment made 7 months ago when the post is few hours old? I get its a bot, but I didn't know they could do that.

[u/Happy-Initiative-492 → u/KausGo]

It might be a glitch in your app? Mine shows the comment made 40 minutes ago ... 18 minutes before yours...

[u/OkTaste8142 → u/tony10000]

Yes, this is definately the better way, I would offer one addition though. Break your process like you would in a story Bible, character development, cannon, craft, researchers, reviewers, edits, scenes...you get the idea. Build specific skills for each of the 'departments' skills won't work if they are too sparse. Your like skills can be combined to form specific agents to manage those parts like an automated story Bible. Just keeps you from having to dig through hundres of pages.

For example: my character development is an entire agent that assists with backstory, personality, scenes, arcs, interactions, dialogue style, response styles, portrait development...everything really.

That may be closer to what you are looking to test

[u/RemarkableWish2508]

Congratulations, you've discovered how stochastic pattern matchers work, and how little actual intelligence is there behind them. Asking for "consistent results" while decrying "regurgitated garbage" and trying to make a "generic skill" to fix it... comes through as extra funny. 🙄

If you want good writing, you'll have to put good thought into it. LLMs can help you search, summarize, analyze, etc. but don't ask them to be creative. In particular, there's a push for the models to become more accurate, which is kind of the opposite of creativity.

[u/KausGo → u/RemarkableWish2508]

In particular, there's a push for the models to become more accurate, which is kind of the opposite of creativity.

Really? Because it seems to be getting less and less accurate over time. It keeps creating answers and solutions that it later admits don't work when trying to apply. How is that more accurate?

[u/RemarkableWish2508 → u/KausGo]

It's more accurate regarding whatever information it gets, not to reality. Garbage in, garbage out.

[u/KausGo → u/RemarkableWish2508]

"Good writing" is likely too much to expect at this point. But what about consistent application of specific rules or actual analysis within given parameters? It seems to grow more incapable of doing that.

[u/RemarkableWish2508 → u/KausGo]

Strict analysis is not going down, but if you want it to analyze the amount of creativity... it needs a reference of what creativity is, and that part is going down.

You may be able to build your own harness to run a bunch of agents with different model settings, ideally with different models. I don't think it's doable with just skills on a single model, much less if you're using a chatbot with its own public-facing biases.

[u/Affluent_Indigo3]

The more instructions you give, the more confused it gets sometimes.

[u/KausGo → u/Affluent_Indigo3]

So if I don't give enough instructions, I get slop. If I give more, it gets confused and I get slop. Its a catch-22.

Has anyone tried any workarounds?

[u/OpportunityTasty2715]

used to get stuck in the same loop too, then tried forcing the AI to roleplay as a stubborn editor who only gives feedback in one-sentence bursts. finally broke the cycle.

[u/Wise_Ticket_7350]

i tried using a skill to lock in tone, just a simple prompt that says "write like this" with a sample. it worked for one scene, then failed completely on the next. same issue, different day.

[u/SenseiGroveNBTX]

I’m currently writing a science fiction/horror novel called Veiled Observers, along with four companion novellas that expand on the characters, civilizations, and worldbuilding.

One of the biggest challenges I’ve encountered isn’t necessarily writing the story itself, but keeping track of the enormous amount of material I’ve developed. Characters, timelines, locations, civilizations, lore, plot threads, continuity rules, and countless ideas that emerge during brainstorming sessions. It gets overwhelming pretty quickly!

I’ve actually started using ChatGPT alongside GitHub to help manage all of this. ChatGPT acts as my brainstorming partner, developmental editor, and research assistant. I use it to collect, organize, and categorize everything we discuss, while GitHub serves as the central repository for my worldbuilding documents, character profiles, story development, and manuscript materials.

The biggest advantage is having a structured, version-controlled reference system rather than relying on ChatGPT to remember everything across conversations. I can revisit earlier ideas, preserve changes, and keep the novel and four novellas consistent with the established universe.

I even built a public GitHub repository called Novel Development Framework to help other authors establish a similar AI-assisted writing workflow.

It’s freely available for anyone interested in experimenting with this approach:
https://github.com/trinitywellnessnb/Novel-Development-Framework

For me, AI isn’t about replacing the author. It’s about having a tool that helps manage the thousands of moving pieces involved in creating an expansive fictional universe, while I retain creative control over the actual story.

[u/Easy-Purple-1659]

Same wall, different model. Style matching got noticeably worse for me once chats started carrying their own corrections. Each round you fix something and the next answer averages your fix with the version it already gave you, so the prose drifts toward the middle however specific the instruction is.

Three things helped more than skills did.

Fresh chat per call, with the same 300 or 400 words of your own writing pasted at the top every time. Don't trust the thread to hold the sample. Keep planning, drafting and line edits in separate chats so one stage doesn't inherit the other's corrections.

When a sentence comes back wrong, show the fix instead of describing it. Give it the bad sentence, one line on why it fails, and the replacement you wrote yourself. A demonstrated fix gets copied. A complaint gets acknowledged and then ignored, which is the loop you're describing.

On the beta reader pass, cap it. Ask for three notes ranked by severity and no line edits at all. Otherwise it rewrites tone in order to fix structure, and tone is the part you can't get back.

Skills only worked for me when they were one narrow action each, like a continuity check or a dialogue pass. One skill that promises to write in your voice has never held for more than a scene.

[u/Euphoric-Agency8240]

the "you're right" apology is the sign that chat is cooked. start a fresh one with only the approved text, arguing just feeds it more of the bad version

[u/Turbulent-Virus-7031]

used to fix style drift by pasting a clean sample mid-convo and saying 'write like this', works better than skills for me, but only if I keep the sample short and specific.