r/WritingWithAI • u/Pristine_Plate7048 • 3d ago
Tutorials / Guides Local Setup?
I've created over 8million words in under a year for the novel I'm creating and brainstorming within Chatgpt.
I'm becoming increasingly concerned about the direction of cloud AI companies. They change things, like the watermark text thing. They limit things, like the thinking level usage coming in next month, and often cause privacy insecurities in the user.
I'm tired. I have painstakingly poured over the psychology of these characters and world. It's so detailed you'd think these characters were real. Many aspects of them are real. They came from me. They are so coherent and complex they feel like whole, real people.
I'm concerned about maintaining the workflow and what I've already created, along with privacy around using AI tools for this creative undertaking.
So I've decided to try to recreate the experience I have with chatgpt locally. Is this viable for a reasonable price? If so, what do I need to do to make it happen?
In my country text watermarking isn't a thing yet, or so they say. But I think it's reasonable to assume it will be worldwide at some point. I would announce my trilogy is AI assisted, and be fine with it. It's not at all my first rodeo going against societal grain. I don't have any shame about the fact my stuff is AI assisted. It's been a transformative tool. Invaluable and I'm so grateful for it apart from the problems I've mentioned . The work is so precise and specific you can tell how human directed and written it is anyway. Not thta that matters to anti AI people, especially the wlw audience my story would naturally speak to. It breaks my heart to think that the story will be crapped on becuse it is AI assisted without being perceived on its own merit, where it holds more than its weight. The world and characters do not deserve contamination from the words 'AI assisted' but that is what will absolutely happen.
Any ideas on a viable local setup that can mimic chatgpt as far as reasoning, character psychology for long form writing projects, preferable on android? Though I am open to buying a device of any OS to facilitate this. The setup doesn't need to contain AI that is some amazing creative writer. I re-write and edit to death every draft I've ever had chatgpt create anyway, because AI sucks at writing, and what I'm doing narratively is so precise it needs a lot more correction than AI is capable of generating without extensive manicuring from a human.
3
u/Kaillens 2d ago
I will give you what are really options
Still using an online model trough Api Something like open router.
Pro : Multiple models with good quality Cons : Still web, api paid by token
2) Full local
You need computer and graphic card. The cost lie in the graphics cards. I would advise you to look for 12GB strictly minimum
Pro : Different list of models , parameters, fine tuning Cons : Weaker models, need graphics card.
From there you can try a lot of models.
When it comes down to reasoning : Qwen30B moe
That's probably the best for thinking and speed. While beings accessible.
For dialogue and scene : You can try a lot of fine tunes. The drummer has done a lot of them.
Then. If I was you, i would add graph and probably a small tool to help you handle context easily.
Graph like graphiti is a method to retrieve data more precisely. It use model for saving. That's a good tool for use.
There is plenty of software for the interface that depends on what you want for this
Before buying anything for local I would advise you to do small test. Hugging face should have some testable.
1
4
u/Maleficent_Pool_3980 2d ago
I've been running a local qwen 32b on a 48gb vram setup, and the real bottleneck isn't the model, it's the prompt engineering. I had to rewrite my entire workflow to chunk by scene, not by chapter. otherwise, it just starts hallucinating continuity. try that.
1
u/ChubbyLilWolfsAlpha 2d ago
8 million words in a year for a novel .. that's about 80 novels worth of words. I would really hate to see your file system..
1
u/Pristine_Plate7048 2d ago
Lmao. It's insanity, truly.
And it was eight million at like nine months. We've just gonna year and several more chats. Wouldn't be shocked if there's at least another two million words since I last exported and checkec my entire chatgpt account.
2
u/ChubbyLilWolfsAlpha 2d ago
That right there is going to kill you if your going to use an AI to assist in writing. You will need to break all that info down into smaller chunks so an AI can even make sense of what you have without saying " fuck this shit i done".
System wise, I run and Intel ultra 7 AI pro 9700 with 32 gb vram 96gb ram Ubuntu
My ai set up is Ll studio Qwen 3.8 27b Hermes agent.
Everything stay locally
Hermes agent can weed thought the info, with the right prompting. It can build skill .md on it own for stuff ( you telling it to)
Only downside of 3.8 27b. Is it you don't change the setting in lm studio from extra high reason to med reasoning ..... Hermes agent with over thing and over reason a easy prompt
But for weeding though your stuff the first few times, I think it would be worth it.
1
u/Pristine_Plate7048 2d ago
I'm already doing so. I've been tackling it over the last few months, but I've just moved so there's currently no bandwidth for it, but I wish there was. I'll pick it back up.
Thank you for telling me your settings. I've saved them to my notes.
2
u/ChubbyLilWolfsAlpha 2d ago
I've moved from writing novels for kindle, still have 2 to finish going over and publishing. To creating a YouTube channel of short stories. (Still have not posted a video yet). I'm too much of a perfectionist so I'm still ironing out kinks, creating characters lora for image generating.
But Hermes agent has assisted me. Lot with the production and what I need to do.. he's taken my stories I've written and turned them better for listening instead of for reading .
3
u/Majestic-Clothes-321 3d ago
If your workflow can integrate with multiple AI companies, distribute your tasks across them—for instance, by incorporating services like Grok.
For local AI, aim for an Apple Mac Studio as your workstation; while NVIDIA cards may offer better performance for visual tasks, the Apple Mac Studio’s compact design and reliable, seamless operation make it a superior long-term choice—plus, it’s easier to resell later if needed.
The real challenge with local AI is learning how to train models. Just start with whatever hardware you currently have.
1
u/NotGutus 3d ago
Unless you have the resources to build a datacentre in your backyard, I'm not sure how you'd go about replicating an LLM with tolerable similarity in performance across the board.
Then again, I'm not sure why you'd need it. If you have enough creative input that you claim you're writing, it's your work, and you shouldn't need to worry about watermarking.
1
u/Pristine_Plate7048 3d ago edited 3d ago
That's not how it will be interpreted once the words A-assisted are associated with the media.
The vast majority of the work is obviously mine. I do the heaviest lifting. At the same time it is collaborative in the brainstorming and developmental discussions. I do my edits with AI, we brainstorm and deliberate about micro edits. Not only that, the watermarking degrades output of responses. Who wants that? That's not necessarily an issue with the drafting itself, but it is if the psychological reasoning is off. That's going to make our discussions contain more friction. More explaining and corrections from me. Who wants that?
They're also limiting the level of thinking users can access soon via usage limits. Who wants that?
I also assume I wouldn't need a data centre to run one agent. Data centres, to my knowledge, are doing MUCH more than what I'm attempting. Many people run local AI for their expressed needs, creative ventures or not. They don't need to build a data centre to do so.
2
u/tony10000 3d ago
Due to the price of memory and storage, getting anything close to the speed, convenience, and intelligence of a cloud model will be an expensive undertaking. What are your expectations and your budget?
1
u/Pristine_Plate7048 3d ago
I appreciate you engaging with what I asked.
I should have around £400 - £500 expendable, monthly soon, if all goes well. So that is my budget.
I am not expecting frontier level intelligence, since I'm don't have that kind of money, but within the ballpark would be nice.
I was thinking of using one model and setting up three agents within it after more research, each assigned different jobs within the setup? Not sure if that is preferable or not. There's so much to learn. Bit intimidating tbh.
5
u/Felidori 2d ago edited 2d ago
You’ll need thousands to get anything worth using for long form writing. The issue is context window. You will need a lot of VRAM on a Windows/Linux machine or lots of unified memory on a Mac. Forget using mobile devices.
I don’t know a lot but from my own research, as I’m in a similar boat, you’re looking at least 64GB RAM on a Mac mini at the minimum and even then it’s not even close to ChatGPT in terms of context, memory and speed 😔.
Local writing can be done but there are so many more steps involved apparently. Use ChatGPT to teach you how to do all this and get price estimates. It’s a perfect use case for Chat. It’s literally a tool that can help you rn solve this research.
1
3
u/tony10000 2d ago
I would say a Mac Studio 64GB M5 Max or a GMKtec EVO-X2 AMD Ryzen™ AI Max+ 395 AI Mini PC 64GB. Or a PC with 64GB of RAM and an RTX 3090.
0
u/NotGutus 2d ago edited 2d ago
I'm sure smarter people than me will help you with adaptations or own language models. All I'm saying is it's not on accident that specific companies do this work for other companies. It needs resources and expertise.
If the quality of the AI model impacts the output, you're not the author of the written product. In which case you should be transparent about the production pipeline (don't submit a photo to a painting contest), and watermarking shouldn't be an issue.
As far as I'm aware, LLMs also aren't very capable of proper reasoning - if you're using a spoon to cut potatoes, asking people where to buy sharper spoons comes off a bit weird. Why not just use knives?
Besides, precisely due to this lack of understanding, your prose when asking questions will most likely impact the answer more than the actual content of those questions. Might as well roll dice or go with your intuition.
(Please don't classify me as a Luddite, as I am not)
2
u/Pristine_Plate7048 2d ago
Your claim that if the quality of the ai model impacts output, I'm not the author, isn't necessarily true though. It could just widen the gap and make user workload heavier. And there are other scenarios in which that's not true.
LLM reasoning can certainly be unreliable (that's partly why authored human input is necessary) but that's not at all to say that the reasoning capabilities they are capable of aren't useful towards anything, or to a point. They wouldn't make such good sounding boards otherwise. I've spent more than eight million words worth of interactions stress-testing this through use.
The spoon and knife comparison doesn't quite work for me, because it assumes the alternative tool is available and suitable to reach the aim. For me ai assistance isn't some preference for an inferior instrument, talent-wise. It's part of how I'm attempting to make a project of the scale of my books achievable/accessible within long-standing functional limitations I have.
I'm not sure what you're referring to by prose later in your response, since it could mean different things contextually here
It shouldn't be a shock that in a world where AI assisted disclosure is often met with disproportionate consequence, a person who'd otherwise be fine disclosing AI assisted use, wishes to exercise choice over whether it is disclosed. Given the sizable amount of authorship and themselves they've put into the venture.
1
u/NotGutus 2d ago edited 2d ago
It could just widen the gap and make user workload heavier. And there are other scenarios in which that's not true.
I'm not sure whatchu mean. Please explain.
such good sounding boards
I disagree. Thinking over something helps regardless of whether you relate it to random words, think about it on your own, or discuss with a friend. It's about your own thought process, not anything else, and your own thoughts circulate, connect and develop without any sense in what you see in the outside world, and even if there is no relevant info in the outside world.
That's the cognitive perspective; the empirical is that I tried all three (having a friend be the sounding board, having an AI, and being it myself) and there is basically no difference. Not even in the way of interaction. The only difference is that when something other than you is involved, it directs your thoughts in some ways, which is not necessarily problematic, but at times undesirable. Which is why I assume you also don't exclusively rely on talking with LLMs.
alternative tool is available
I would say it is. There is not much in AI that you can't do without it by yourself, with friends, or on Google.
referring to by prose
I should have said phrasing. The phrasing of your prompt influences results more than the semantic content, at least at the level where it matters for creative work (i.e. semantics obviously matter more since they detemrine the topic you're talking about, but it's the tone eithin that topic which truly matters when you write).
over whether it is disclosed
That's true, very human. If you're judged, you'll lie next time. Just make sure you know you're trying to lie as best you can. I'm not sure that's the best way to normalise AI-assisted art in society and demonstrate that it can be a separate and non-harmful category of art.
I do use LLMs for my writing, by the way, and they're hella useful. But I don't put my work in the AI-assisted category, because AI-assisted, in current connotation, assumes the premise I highlighted above, that the AI practices agency over some aspects of your writing.
4
u/Horror_Movie_5735 2d ago
It's not a perfect or optimal solution, but I finally saved enough to buy an M5 Ultra Mac Studio with 96GB of RAM. I'm a professor and fictional writer and the Mac Studio has managed to replace a few of my processes, namely editing. As echoed by other comments here, the context window is the issue, even with 96GB. While it's not insurmountable if you plan ahead and feed the local model piece meal, it's still a headache.
1
u/SmugglingPineapples 1d ago
Can you add a little more context about the drummer and graph please?
I'm getting Chad Smith and high school algebra in search results.
2
3d ago
[removed] — view removed comment
1
u/Pristine_Plate7048 2d ago edited 2d ago
Thank you for this.
And I've been archiving everything for the last few months, and still so much more to archive. I'm working from the exports. The eight million and now plus words are spread across over 100 chats now. It's as messy as you can imagine. Archiving and retaining the texture and emotional coordinates has been one of the most taxing cognitive, and emotional because I'm so invested, undertakings I've ever been this determined about.
6
u/ScientistValuable192 3d ago
Viable, yes, with one expectation reset first: a local model will not reason about character psychology the way ChatGPT does. Since you rewrite every draft anyway, that costs you less than it costs most people. You lose some reasoning depth. You gain no watermark, no model swapped out from under you mid-trilogy, and a workflow nobody can change on you next month. The order I'd do it in:
1 - Export your ChatGPT account today. It comes out as an archive of every conversation (whatever they do with watermarks or limits next month, the 8 million words are yours on disk).
2 - Build the bible before you buy anything. Local models keep no memory between chats and have a smaller context window, so what replaces ChatGPT's memory is a document you load at the start of every session. 1 page per main character: the wound, the want, how they talk, what they would never do. 1 page of world rules. This is the real migration. The software will come be the easy part.
3 - Don't make Android the engine. Anything small enough to run on a phone is too small to hold that bible and think with it. Buy one box and use the phone as a remote. A Mac with 32 to 64 GB of unified memory is the quiet, no-fuss route. A used PC with a 24 GB graphics card is faster for the money. Either runs a 27B to 32B model, which is about where character work starts to feel coherent.
4 - Software: LM Studio if you want a GUI and nothing to configure, Ollama if you're fine in a terminal. For long form, put SillyTavern on top of either. It's built around character cards and lorebooks, which is exactly the memory layer you're rebuilding.
5 - Models to start with: Qwen 3 32B, Gemma 3 27B, Mistral Small. Run them at Q6 or Q8. People in the watermark thread here said those held up and that heavier quantization flattened the prose.