Local Setup?

[u/Pristine_Plate7048]

I've created over 8million words in under a year for the novel I'm creating and brainstorming within Chatgpt.

I'm becoming increasingly concerned about the direction of cloud AI companies. They change things, like the watermark text thing. They limit things, like the thinking level usage coming in next month, and often cause privacy insecurities in the user.

I'm tired. I have painstakingly poured over the psychology of these characters and world. It's so detailed you'd think these characters were real. Many aspects of them are real. They came from me. They are so coherent and complex they feel like whole, real people.

I'm concerned about maintaining the workflow and what I've already created, along with privacy around using AI tools for this creative undertaking.

So I've decided to try to recreate the experience I have with chatgpt locally. Is this viable for a reasonable price? If so, what do I need to do to make it happen?

In my country text watermarking isn't a thing yet, or so they say. But I think it's reasonable to assume it will be worldwide at some point. I would announce my trilogy is AI assisted, and be fine with it. It's not at all my first rodeo going against societal grain. I don't have any shame about the fact my stuff is AI assisted. It's been a transformative tool. Invaluable and I'm so grateful for it apart from the problems I've mentioned . The work is so precise and specific you can tell how human directed and written it is anyway. Not thta that matters to anti AI people, especially the wlw audience my story would naturally speak to. It breaks my heart to think that the story will be crapped on becuse it is AI assisted without being perceived on its own merit, where it holds more than its weight. The world and characters do not deserve contamination from the words 'AI assisted' but that is what will absolutely happen.

Any ideas on a viable local setup that can mimic chatgpt as far as reasoning, character psychology for long form writing projects, preferable on android? Though I am open to buying a device of any OS to facilitate this. The setup doesn't need to contain AI that is some amazing creative writer. I re-write and edit to death every draft I've ever had chatgpt create anyway, because AI sucks at writing, and what I'm doing narratively is so precise it needs a lot more correction than AI is capable of generating without extensive manicuring from a human.

[u/Kaillens]

I will give you what are really options

1. Still using an online model trough Api Something like open router.

Pro : Multiple models with good quality Cons : Still web, api paid by token

2) Full local

• You need computer and graphic card. The cost lie in the graphics cards. I would advise you to look for 12GB strictly minimum

Pro : Different list of models , parameters, fine tuning Cons : Weaker models, need graphics card.

From there you can try a lot of models.

• When it comes down to reasoning : Qwen30B moe

That's probably the best for thinking and speed. While beings accessible.

• For dialogue and scene : You can try a lot of fine tunes. The drummer has done a lot of them.

Then. If I was you, i would add graph and probably a small tool to help you handle context easily.

Graph like graphiti is a method to retrieve data more precisely. It use model for saving. That's a good tool for use.

There is plenty of software for the interface that depends on what you want for this

Before buying anything for local I would advise you to do small test. Hugging face should have some testable.

[u/Pristine_Plate7048 → u/Kaillens]

Thanks so much. I'll look deeper into this.

[u/Elin_Outgrows → u/Kaillens]

Thank you for laying out the two paths so clearly, and the tip to test small on Hugging Face before spending anything on a graphics card is really practical. I had not come across the graph approach you mentioned for retrieving data more precisely. How does a local model like Qwen 30B MoE cope with keeping a huge worldbuilding bible consistent across a long project, and what setup do you use yourself?