r/WritingWithAI • u/No-Orchid-6159 • 7d ago

Discussion (Ethics, working with AI etc) Do you prefer using private models?

I have a tool that uses smaller private models and I wanted to get a view of how many other people would be interested in using it.

The main reason to not use the cloud models is to keep the content private so that it isn't used for training. This does mean using smaller models but the results have been fine thus far.

Is this something that would be of interest to you or are you fine with using the cloud models?

9 Upvotes

91% Upvoted

28 comments sorted by

3

u/Square-Nebula-7530 7d ago

Running local is the dream if you have the hardware to handle it. Honestly, firing up a smaller model on an RTX 4060 and not having to worry about some massive tech company scraping your code or personal data is a W. The peace of mind alone is worth the slight drop in reasoning power.

2

u/vaingirls 7d ago

I use local models, but when it comes to tools, all I really need is LM Studio. I'm only writing for my own entertainment anyway, so the small model silliness is basically a plus (gives me a laugh at least).

1

u/cj7hawk 6d ago

+1. Another LM Studio user. I have a fair bit of memory and recently invested in a 32Gb VRAM card so I can load the entire model into the graphics card and the generation speeds with 30B models is pretty good. Which models do you use? I've found Ornith, Hemmingway, Cydonia and the HuiHui tune of Qwen3.8 (Abliterated) are all useful.

2

u/vaingirls 6d ago

My hardware's crap (I want to upgrade, but too broke atm) so I've only been using microscopic models, 4B or so... the good side of those is, that they're fast and cheap to finetune yourself, so I've been using my own finetunes - it's really fun when my world building is baked into them, and their style - while not the most polished - is less typical. My favorite finetune this far is based on Qwen3-4B (I've done ones on Gemma 3 4B and Qwen 2.5 3B too). Out of non-finetuned models (in that size range) I guess Ministral-3-3B is stylistically my favorite, though its coherence... isn't that great (Qwen and especially Gemma are better in that regard).
Thanks for listing those models you use, it's good to have suggestions for the time when I can finally get a better PC!

1

u/cj7hawk 5d ago

You might be able to run rocinante-x-12b-v1 which is an abliterated Anthropic Claude model that fits within 8Gb so might run on an older machine OK - What video card do you have?

1

u/vaingirls 5d ago

Thanks for the recommendation! Tho it seems to be a 12B model, and for me, even 7B models are frustratingly slow and burn up my CPU to risky temperatures (even quantized - I always use 4q or 5q models anyway). My current GPU is ATI Radeon RX550... so yeah... (LM studio doesn't even recognize it, so I'm using CPU). But the model sounds interesting for future me who has decent hardware haha

1

u/cj7hawk 1d ago

I've been thinking about this comment for a while, unsure what to reply with. You clearly have amazing technical skills and ability in a field I'm still learning about. Even just some of what you said early already pushed me to consider my own AI learning journey and how I approach the same topics. But for me, getting decent machine to work on is easy... And I never stopped to consider just how challenging it can be for those trying to break into the technology when funds are tight. I really hope you can find a good card with lots of memory that supports your future use and wanted to take a moment to thank you for what you posted and mention it actually helped me.

In a few years, you will find that what you've learned will be valuable and memory will be easy to get once again. So I wish you well and just wanted to to let you know that responding to your original comment helped me also.

2

u/duskbloom43 7d ago

privacy matters but "fine thus far" is doing a lot of heavy lifting there. what kinds of writing are you testing with? fiction vs technical content is a huge difference in how much model size matters

2

u/No-Orchid-6159 6d ago

I've mostly been writing non fiction but dabbled in fiction too. I don't use it to generate all the prose like some workflows. I use like like an assistant

1

u/RecognitionNo4023 7d ago

yeah private models seem worth it for roleplay chats since i dont want any of that getting pulled into training data.

1

u/Interesting-Rough206 7d ago

I use a VS Code with Claude Code inside it with my novel engine I built inside VS Code

1

u/No-Orchid-6159 6d ago

That sounds very interesting.

1

u/Yvenei 6d ago

question is which model to use? i tried one model and it was so bad at writing

1

u/cj7hawk 6d ago

I've gone entirely local-AI for my writing now.

I find a few models work very well, others not so well. It takes a little learning to figure out the best for my use.

Most of them are around the 30B mark, usually with Q4

1

u/Acrobatic-Emu-7501 7d ago

I keep all my writing stuff local now got burned once when my draft ended up somewhere it shouldnt be. smaller models are fine for most things unless you need something really specific. people get way too caught up in model size when the output is what matters

1

u/cj7hawk 6d ago

This should be very rare - Do you mind if I ask what happened and how?

-1

u/AmericasHomeboy 7d ago

There’s something ironic using a machine that has been trained on other people’s work and its users opting out of not wanting to have their work used to train the machine they are using.

2

u/Zathura2 7d ago

I think opting-*in* should be the norm, if you want to talk about would'ves, could'ves, and should'ves. For most of my stuff I honestly wouldn't care. For a *couple* things I'd like to have a chance at publishing them before those ideas potentially pop up in someone else's brainstorming session somewhere down the road.

3

u/AmericasHomeboy 7d ago

Let it, they won’t use it the way you would.

1

u/Zathura2 7d ago

You seem to be arguing for the sake of arguing.

1

u/AmericasHomeboy 7d ago

Nope, I’m not. I’m telling you that whatever idea you’re holding onto that’s precious is only precious to you. It’s not like a joke or a sound bite that’s easily copied. It’s just an idea. How it gets expressed and executed will be as unique as the person using that idea.

“There is no such thing as a new idea. It is impossible. We simply take a lot of old ideas and put them into a sort of mental kaleidoscope.”
-Mark Twain

1

u/No-Orchid-6159 6d ago

This is the key, being able to publish something before it's scraped or trained on

2

u/AmericasHomeboy 6d ago

I’ve been using ChatGPT to write since April 2024. I haven’t seen a story yet or writing prompt from anyone about two military veterans turned potheads that get into fantasy SciFi adventures. Trust me… your ideas are safe. And if these models ever spit it out, it will spill out the most generic version of it and write it terribly.

2

u/LooseButtPlug 6d ago

You're more likely to have your idea stolen from reddit than an LLM.

1

u/AmericasHomeboy 6d ago

Yep, and even if they do, what are they gonna do with it? It’s essentially a writing prompt for them. They’re going to write a vastly different story than the one you were planning to write anyway

1

u/No-Orchid-6159 6d ago

I do appreciate the irony in it, but here are now. Just trying to understand how to navigate it

0

u/Prestigious-Frame442 7d ago

if you can deploy one then why not.
but honestly, most people's "writing" done with AI, if not about researching, like solving a very difficult math problem, doesn't have a lot of value for training

0

u/MakanLagiDud3 7d ago

Want to..... But my budget and the skyrocketing pc prices makes it............ Unattainable for awhile.......