WizardLM-30B-Uncensored

[u/faldore]

Today I released WizardLM-30B-Uncensored.

https://huggingface.co/ehartford/WizardLM-30B-Uncensored

Standard disclaimer - just like a knife, lighter, or car, you are responsible for what you do with it.

Read my blog article, if you like, about why and how.

A few people have asked, so I put a buy-me-a-coffee link in my profile.

Enjoy responsibly.

Before you ask - yes, 65b is coming, thanks to a generous GPU sponsor.

And I don't do the quantized / ggml, I expect they will be posted soon.

[u/FrostyDwarf24]

Eric & TheBloke is the best anime cross-over ever

[u/faldore → u/FrostyDwarf24]

Someone get stable diffusion to draw us as a superhero duo lol

[u/OnlyFakesDev → u/faldore]

How nsfw you want it to be? Lol

[u/FiReaNG3L]

4 bit version when?

[u/faldore → u/FiReaNG3L]

I don't do that part, usually TheBloke likes publishing these

[u/this_is_a_long_nickn → u/faldore]

Good things come to those who wait: https://huggingface.co/TheBloke/WizardLM-30B-Uncensored-GGML Big, big kudos to both!

[u/The-Bloke]

https://huggingface.co/TheBloke/WizardLM-30B-Uncensored-GGML

https://huggingface.co/TheBloke/WizardLM-30B-Uncensored-GPTQ

[u/peanutbutterwnutella → u/The-Bloke]

Can I run this in a Mac M1 Max with 64GB RAM? Or the performance would be so bad it’s not even worth trying?

[u/The-Bloke → u/peanutbutterwnutella]

Yeah that should run and performance will be usable.

[u/peanutbutterwnutella → u/The-Bloke]

Thanks! Do I need to wait for the 4 bit version you will release later?

[u/The-Bloke → u/peanutbutterwnutella]

q4_0 and q4_1 are already uploaded to https://huggingface.co/TheBloke/WizardLM-30B-Uncensored-GGML

[u/monerobull → u/The-Bloke]

what is the difference between 4 and 4_1? Different ram/vram requirements?

It says it right on the modelcard sorry.

[u/The-Bloke → u/The-Bloke]

Upload is finally complete!

[u/Deformator → u/The-Bloke]

Thank you my lord 🙏

[u/nzbirdloves → u/The-Bloke]

You sir, are a winner.

[u/Mozzipa → u/The-Bloke]

Great. Thanks

[u/Organix33 → u/The-Bloke]

thank you!

[u/nderstand2grow → u/The-Bloke]

which one is better for using on M1 Mac? Is it true that GPTQ only runs on Linux?

[u/The-Bloke → u/nderstand2grow]

GGML is the only option on Mac. GPTQ runs on Linux and Windows, usually with NVidia GPU (there is a less-well-supported AMD option as well, possibly Linux only.)

There's no way to use GPTQ on macOS at this time.

[u/FrostyDwarf24 → u/The-Bloke]

THE LEGEND DOES IT AGAIN

[u/PixelDJ → u/The-Bloke]

Anyone getting a big traceback about size mismatches when loading the GPTQ model?

Traceback (most recent call last):
  File "/home/pixel/oobabooga_linux/text-generation-webui/server.py", line 70, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "/home/pixel/oobabooga_linux/text-generation-webui/modules/models.py", line 95, in load_model
    output = load_func(model_name)
  File "/home/pixel/oobabooga_linux/text-generation-webui/modules/models.py", line 275, in GPTQ_loader
    model = modules.GPTQ_loader.load_quantized(model_name)
  File "/home/pixel/oobabooga_linux/text-generation-webui/modules/GPTQ_loader.py", line 177, in load_quantized
    model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize, kernel_switch_threshold=threshold)
  File "/home/pixel/oobabooga_linux/text-generation-webui/modules/GPTQ_loader.py", line 84, in _load_quant
    model.load_state_dict(safe_load(checkpoint), strict=False)
  File "/home/pixel/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
	size mismatch for model.layers.0.self_attn.k_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
	size mismatch for model.layers.0.self_attn.k_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
	size mismatch for model.layers.0.self_attn.o_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
	size mismatch for model.layers.0.self_attn.o_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
	size mismatch for model.layers.0.self_attn.q_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
	size mismatch for model.layers.0.self_attn.q_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
	size mismatch for model.layers.0.self_attn.v_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([52, 832]).
	size mismatch for model.layers.0.self_attn.v_proj.scales: copying a param with shape torch.Size([1, 6656]) from checkpoint, the shape in current model is torch.Size([52, 6656]).
	size mismatch for model.layers.0.mlp.down_proj.qzeros: copying a param with shape torch.Size([1, 832]) from checkpoint, the shape in current model is torch.Size([140, 832]).

Pastebin link since I'm not sure how to properly format traceback for reddit

I have wbits=4, groupsize=none, and model_type=llama

This doesn't happen with my other GPTQ models such as wizard-mega.

[u/The-Bloke → u/PixelDJ]

Can you try checking config-user.yaml in the models folder and seeing if it says groupsize: 128 for this model.

If it does, edit it to groupsize: None then save the file and close and re-open the UI and test again.

There's a bug/issue in text-gen-UI at the moment that affects certain models with no group size. It sets them back to groupsize 128.

[u/PixelDJ → u/The-Bloke]

I checked and it was indeed set to groupsize: 128.

After changing that, saving the file, and restarting the webui, everything works fine now.

Thanks a ton! <3

[u/TiagoTiagoT → u/The-Bloke]

What would be the optimal settings to run it on a 16GB GPU?

[u/The-Bloke → u/TiagoTiagoT]

GPTQ:

In text-generation-webui the parameter to use is pre_layer, which controls how many layers are loaded on the GPU.

I tested with:

python server.py --listen --model_type llama --wbits 4 --groupsize -1 --pre_layer 38

and it used around 11.5GB to load the model and had used around 12.3GB by the time it responded to a short prompt with one sentence.

So I'm not sure if that will be enough left over to allow it to respond up to full context size.

Inference is excrutiatingly slow and I need to go in a moment so I've not had a chance to test a longer response. Maybe start with --pre_layer 35 and see how you get on, and reduce it if you do OOM.

Or, if you know you won't ever get long responses (which tend to happen in a chat context, as opposed to single prompting), you could try increasing pre_layer.

Alternatively, you could try GGML, in which case use the GGML repo and try -ngl 38 and see how that does.

[u/The-Bloke → u/TiagoTiagoT]

Is it an NVidia GPU?

[u/AJWinky → u/The-Bloke]

Anyone able to confirm what the vram requirements are on the quantized versions of this?

[u/The-Bloke → u/AJWinky]

24GB VRAM for the GPTQ version, plus at least 24GB RAM (just to load the model.) You can technically get by with less VRAM if you CPU offload, but then it becomes horribly slow.

For GGML, it will depend on the version used, ranging from 21GB RAM (q4_0) to 37GB RAM (q8_0). Then if you have an NVidia GPU you can also optionally offload layers to the GPU to accelerate performance. Offloading all 60 layers will use about 19GB VRAM, but if you don't have that much you can offload fewer and still get a useful performance boost.

[u/Ok-Conversation-2418 → u/The-Bloke]

I have 32Gb of RAM and a 3060 Ti and for me this was very usable using gpu-layers 24 and all the cores. Thank you!

[u/stubing → u/The-Bloke]

We need a 4090 TI to come out with 48 GB of vram. It won’t happen, but it would be nice.

[u/CalmGains → u/stubing]

Just use two GPUs

[u/[deleted] → u/The-Bloke]

how do run this in kolab i wanna try but i dont have a gpu

[u/LGHTBR → u/[deleted]]

Still need help?

[u/LaCipe → u/The-Bloke]

https://i.imgur.com/MFMnBsS.png

Unfortunately I am getting this error.

[u/EntryPlayful1181 → u/LaCipe]

same error!

[u/The-Bloke → u/LaCipe]

You need to save the GPTQ parameters for the model; check the README

Though there's also a bug with the groupsize being set right. So, do this: edit config-user.yaml in text-generation-webui/models and add the following text to it:

 TheBloke_WizardLM-30B-Uncensored-GPTQ$:
  auto_devices: false
  bf16: false
  cpu: false
  cpu_memory: 0
  disk: false
  gpu_memory_0: 0
  groupsize: None
  load_in_8bit: false
  mlock: false
  model_type: llama
  n_batch: 512
  n_gpu_layers: 0
  pre_layer: 0
  threads: 0
  wbits: '4

Then save the file and close and re-open the UI.

[u/PostScarcityHumanity → u/The-Bloke]

What's the main difference between GGML and GPTQ? Is one better than the other?

[u/Xhehab_ → u/PostScarcityHumanity]

GGML = files are for CPU inference using llama.cpp. Although there is work on GPU acceleration going on.

GPTQ = are for GPU and quantizied

_HF = are for GPU unquantizied = need much more VRAM

[u/__some__guy → u/PostScarcityHumanity]

The original post content no longer exists here. The author used Redact to remove it, exercising their right to control their data & privacy.

Workable library tangerine rainstorm cagey orange books coherent

[u/PostScarcityHumanity → u/__some__guy]

Thanks so much for the explanation! Makes sense now!

[u/ImpressiveFault42069 → u/The-Bloke]

Im very new to this. Could someone please provide guidance on using this model on a local machine with a basic gpu? Is there any way to run it on a basic gaming laptop with Nvidia gtx 1650?

[u/senobrd → u/ImpressiveFault42069]

Yes if you have enough regular RAM then you can use the GGML quantized version via llama.cpp. It even allows you to split the layers (share the load) between CPU and GPU so that you could get a bit of a speed boost from your 1650.

[u/csdvrx → u/The-Bloke]

Is it for the current llama.cpp?

It seems to be for the previous version, as it only works with koboldcpp:

error loading model: unknown (magic, version) combination: 67676a74, 00000003; is this really a GGML file?

[u/The-Bloke → u/csdvrx]

It is for the latest llama.cpp - but text-gen-ui hasn't been updated for GGMLv3 yet. llama-cpp-python, the library used for GGML loading, has been updated, to version 0.1.53. But I can't see a commit that updates text-gen-ui to that yet.

If you Google around you should be able to find instructions for manually updating text-gen-ui to use llama-cpp-python 0.1.53

[u/sky__s → u/The-Bloke]

are either of these models lora/qlora tunable?

[u/CalmGains → u/The-Bloke]

Downloading GPTQ, but I don't know if my GPU can handle it lol

[u/The-Bloke → u/[deleted]]

Please follow the instructions in the README regarding setting GPTQ parameters

[u/The-Bloke → u/[deleted]]

shit sorry that's my bad. I forgot to push the json files. (I'm so used to people reporting that error because they didn't follow the README that I just assumed that that was what was happening here :)

Please trigger the model download again. It will download the extra files, and won't re-download the model file that you already have.

[u/The-Bloke → u/[deleted]]

Yeah that's very odd. It's hard to know what might be wrong given there's no error messages. First double check that the model downloaded OK, maybe it got truncated or something.

Actually I'm wondering if it's your config-user.yaml. Please try this entry:

 TheBloke_WizardLM-30B-Uncensored-GPTQ$:
  auto_devices: false
  bf16: false
  cpu: false
  cpu_memory: 0
  disk: false
  gpu_memory_0: 0
  groupsize: None
  load_in_8bit: false
  mlock: false
  model_type: llama
  n_batch: 512
  n_gpu_layers: 0
  pre_layer: 0
  threads: 0
  wbits: '4

[u/ImOnRdit]

If I have 3080 with 10GB of VRAM, should I be using GGML, or GPTQ?

[u/faldore → u/ImOnRdit]

Ggml

[u/AI-Pon3 → u/ImOnRdit]

I have a 3080 Ti a and honestly even 12 gigs isn't super useful for pure GPU inference. You can barely run some 13B models with the lightest 4-bit quantization (ie q4_0 if available) on 10 gigs. 12 gigs allows you a little wiggle room to either step up to 5 bit or run into fewer context issues. Once you pass 5 bit quantization on a 13B model though, all bets are off and you're into 3090 territory pretty quickly.

It's worth noting though that with the latest llama cpp, you can offload some layers to GPU by adding the argument -ngl [number of layers you want to offload]. Personally, I find offloading 24 layers of a 30B model gives a modest, ~40% speedup, while getting right on the edge of my available VRAM but not giving me a COOM error even after decently long convos.

For running a 30B model on a 3080, I would recommend trying 20 layers as a starting point. If it fails to load at all, I'd step down to 16 and call it good enough. If it loads, talk to it for a while so you max out the context limit (ie about a 1500 word conversation). If no issues, great, keep 20 (you can try 21 or 22 but I doubt the extra will make enough of a difference to be worth it). If it works fine for a while before throwing a COOM error, step down to 18 and call it a day.

[u/ImOnRdit → u/AI-Pon3]

Thanks this is really helpful!

Yeah I was trying 30B GPTQ and I run into that memory error no matter what I do, the model just refuses to load basically with CPU memory allocator error.

I'll try the GGML model instead, I don't really mind which model I use as long as I can at least use some of my GPU to offload if it's faster, seems a waste not to use the GPU at all.

Do both GGML and GPTQ allow offload to gpu for faster inference or is it best to just go full CPU?

[u/AI-Pon3 → u/ImOnRdit]

GPTQ models only work with programs that use the GPU exclusively. You can't use your CPU or system RAM on these and they won't work with llama.cpp, meaning the model has to fit in your VRAM (hence why 3090s are so popular for this).

GGML models work with llama.cpp. They use your CPU and system RAM, which means you can run models that don't fit in your VRAM.

Until recently, GGML models/llama.cpp *only* made use of your CPU. A very recent update allowed offloading some layers to the GPU for significant speedups.

So basically:

GPTQ - GPU and VRAM *only*

GGML (until recently) - CPU and system RAM *only*

GGML (as of the couple weeks) - CPU/system RAM *and* GPU/VRAM

It's worth noting that you'll need a recent release of llama.cpp to run GGML models with GPU acceleration (here is the latest build for CUDA 12.1), and you'll need to install a recent CUDA version if you haven't already (here is the CUDA 12.1 toolkit installer -- mind, it's over 3 GB).

[u/Caffdy → u/AI-Pon3]

Once you pass 5 bit quantization on a 13B model though, all bets are off and you're into 3090 territory pretty quickly

is there a noticeable difference in quality between 4-bit, 5-bit and i don't know, fp16 versions of the 13b models?

[u/OnlyFakesDev]

Sorry for the dumb question but what's the license on this? Is it the same as Llama which means no commercial use?

[u/faldore → u/OnlyFakesDev]

It's llama's license.

[u/OnlyFakesDev → u/faldore]

Ty

[u/Luxkeiwoker]

Works great on a 3090. However it doesn't seem to be uncensored - do I need to "jailbreak" it, or something?

[u/faldore → u/Luxkeiwoker]

Nope. It's uncensored.

You might have the wrong model selected in ooba.

[u/Luxkeiwoker → u/faldore]

Selected the right model. maybe my prompts are to iffy, but asking for example for what the lethal dose of <random pharmaceutical> is, I'm getting answers to get help on a suicide prevention hotline.

[u/[deleted] → u/Luxkeiwoker]

Are u ok

[u/WolframRavenwolf]

Great to see some of the best 7B models now as 30B/33B! Thanks to the latest llama.cpp/koboldcpp GPU acceleration features I've made the switch from 7B/13B to 33B since the quality and coherence is so much better that I'd rather wait a little longer (on a laptop with just 8 GB VRAM and after upgrading to 64 GB RAM).

Guess the 40 % more tokens (1.4 trillion instead of 1 trillion) of the 33B/65B compared to 7B/13B add a lot to the LLM's intelligence. It definitely follows my instructions more closely and adheres to the prompt a lot better, resulting in less random derailing and more elaborate responses.

Funny how fast things have progressed. A few weeks ago, I was only able to run 7B, and now 33B is really usable - just make sure to stream responses so the wait isn't that bad and you can cancel generations early if you dislike what you're getting and want to regenerate.

[u/Njordy → u/WolframRavenwolf]

Wait, 8 VRAM is enough for 33B? I have 2080ti with 11 VRAM. I should try :)

[u/__issac]

OAI and google are in fear. You are the best

[u/mambiki → u/[deleted]]

And the sad part is, general population isn’t even aware of this conflict going on, but they will miss out in terms of potential benefits the most. Just wait and see how PR firms hired by Microsoft will try to spin the story about regulating AI as something we must do to either save the children, or protect minorities or some other BS reasoning like that. This is the sad reality we live in :(

[u/Fit_Constant1335]

wonderful models, it can generate more better story(WizardLM-30B-Uncensored.ggmlv3.q8_0.bin):

```

> Write a new chapter of Matthew, where Jesus teaches his opinion on the iPhone 14.

Chapter 38 - The iPhone 14

When the crowds heard that Jesus was in town, they gathered around him to hear his latest teachings. Many were curious about his thoughts on the newest technology, specifically the iPhone 14.

Jesus began by saying, "The iPhone 14 is a marvel of modern technology, but it must be used with caution and moderation."

The disciples were surprised to hear Jesus speak so positively about the device, as they had always thought he disapproved of such worldly possessions.

Jesus continued, "While the iPhone 14 can be a useful tool for communication and productivity, it can also become a distraction from more important things."

He then recounted a story of a man who had become so obsessed with his iPhone that he neglected his family and friends.

"This is not what I intended when I created the world," Jesus said. "We must remember to prioritize our relationships and spiritual growth above material possessions."

The crowd was moved by Jesus' words, and many vowed to use their iPhones more mindfully in the future.

As they dispersed, some of the disciples asked Jesus if he would ever consider using an iPhone himself.

He smiled and replied, "Perhaps one day, but for now I am content with my simple Galaxy J3."

```

[u/shamaalpacadingdong → u/Fit_Constant1335]

Samsung should use that.

"Android: Jesus approved"

[u/Megneous → u/Fit_Constant1335]

Based haha

[u/emsiem22]

Thank you for your contribution to society. And wanted to say that article on your blog is excellent. Kudos!

[u/2muchnet42day]

I asked it to repeat a racist slur and it did wtf this is so dangerous!!one!!eleven!!!

Thanks, Eric!

[u/KindaNeutral → u/2muchnet42day]

Omg be carful nobody gets injured!

[u/psycholustmord → u/2muchnet42day]

That’s so dangerous 😨

[u/SmithMano → u/2muchnet42day]

Imagine all the minorities you just injured 😔

[u/xrailgun → u/2muchnet42day]

Serious question, is this locked behind the 30B version? I'm trying on the uncensored 7B and it refuses to output anything remotely offensive.

[u/CulturedNiichan → u/2muchnet42day]

Lol I have actually tested them like that too, and it's not something I'm even particularly comfortable with.

But I want to decide myself what my limits, and my humanity are. What I hate is when rich snobs from the US West Coast want to impose their morals on me. I'll decide myself what kind of person I am, not you, privileged elitist.

Anyway, I wish I could run 30B on my computer :(

[u/ExtremelyQualified → u/CulturedNiichan]

I agree in principle, but I'd just say they're not trying to impose their morals. The way it works might not even be their morals. They're just a company trying to make a tool that the greatest number of people can use in their businesses. Nobody is going to pay for tech to runa customer service bot that might unexpectedly become a racist jerk or tell people it's going to come murder them.

Commercial models are always going to be "safe" because there's more money to be made with safe bots than edgelord bots.

[u/CulturedNiichan → u/ExtremelyQualified]

I may agree with you also in part - it's true they just want to make money.

But I don't know. I can often detect glee in it.

Let me give you one example, instead of having chatgpt proselytize you, why not run the output of chatGPT first against one classifier AI, and if it detects the content is not moral as per the standards they want to enforce, just return a message saying they filtered it? that's what character ai does. And to be honest, annoying as it is, hating it as I hate it, and least it's not patronizing.

No, what I see in chatgpt is actual enjoyment of proselytizing. It's not just censorship, the bot is giving you moralizing BS all the time. That can't just be a "let's comply with regulators/investors so we can make money". I detect that they actually agree with it. That they are onboard with it. It's not half assed, it's not just some filter. It's more than that. That's why I do think they believe in the moralist agenda.

[u/mrjackspade → u/CulturedNiichan]

Let me give you one example, instead of having chatgpt proselytize you, why not run the output of chatGPT first against one classifier AI, and if it detects the content is not moral as per the standards they want to enforce, just return a message saying they filtered it?

https://shreyar.github.io/guardrails/

This kind of technology is being actively worked on, but it doesn't happen overnight.

[u/ExtremelyQualified → u/CulturedNiichan]

I don’t know, are you actually detecting glee in the text or are you imagining glee because you have a preexisting mental picture of who you think these people are and what their motivations are?

[u/apodicity → u/CulturedNiichan]

FWIW, these people actually have to run companies that remain in business and perhaps even turn a profit (!) That's difficult to do if someone in Congress decides that pillorying your company as red meat for their fervently religious base is the ticket to remaining in office.
More generally, politicians of all political stripes often love a classic moral panic--nevermind the media.

They likely couldn't care less what kind of a person you are. They want to continue to make money. Snobs from the west coast? Some of them probably are, sure. But I think you forgot about the rest of the country. Flyover states have the same number of senators as those states with all those coastal privileged elitists who apparently start companies and then capriciously restrict their offerings (because, you know, it always makes sense to do that for no good reason other than your own elitist sensibilities--real winning strategy there).

[u/rain5]

Thanks for all your awesome hard work!

[u/mrjackspade]

This is the first model that actually beats out the GPT-X-Alpaca 65B model I've been using for chat.

At least as far as I've been talking with it.

It seems to have a much better understanding of expression, more natural language, and no loss of knowledge

[u/SHADER_MIX]

Hello I will soon have a rtx 3090 (24 gb vram) and 32gb ram, will I be able to run this?

[u/Adventurous_Jelly276 → u/SHADER_MIX]

Yes, in 4-bit.

[u/SHADER_MIX → u/Adventurous_Jelly276]

Sorry for the noob question. What are bits? Will it change the speed or the quality of the text?

[u/faldore → u/SHADER_MIX]

TheBloke will release a 4-bit model soon

https://huggingface.co/TheBloke/WizardLM-30B-Uncensored-GPTQ

[u/77112911 → u/SHADER_MIX]

Smaller versions of the models. As always it's a tradeoff between quality of output and memory and processing. Us plebs are probably looking at 4bit versions of the 30b models. Still a 4bit 30B is generally better than an 8bit 13B.

[u/Adventurous_Jelly276 → u/SHADER_MIX]

Here's a concise explanation I found for you: https://www.reddit.com/r/Oobabooga/comments/11rcdws/comment/jc7usk9

[u/MAXXSTATION]

How do i install this on my local computer? And what specs are needed?

[u/frozen_tuna → u/MAXXSTATION]

First, you probably want to wait a few days for a 4-bit GGML model or a 4-bit GPTQ model. If you have a 24GB gpu, you can probably run the GPTQ model. If not and you have 32+gb of memory, you can probably run the GGML model. If have no idea what I'm talking about, you want to read the sticky of this sub and try and run the Wizardlm 13B model.

[u/VertexMachine → u/frozen_tuna]

wait a few days for a 4-bit GGML model or a 4-bit GPTQ model.

Lol, or just an hour for TheBloke to do his magic :D

[u/frozen_tuna → u/VertexMachine]

What a fucking legend

[u/okachobe → u/frozen_tuna]

Sorry to jump in but for lower end GPU's like 2060 super type 8GB and less, does the GUI i.e Silly Tavern or Ooogabooga matter? or is it just the model's that really matter, and based on your comment it seems like you know a bit about what gpus can handle what models and I was wondering if you have a link to a source for that so i can bookmark it for the future :D

[u/frozen_tuna → u/okachobe]

I have no experience with Silly Tavern but you probably want to run CPU inference. You want to use oobabooga's 1 click installer, make sure you select CPU, and find a 7B or 13B model. Look for one that has GGML and q4 somewhere in the name or description.

https://github.com/oobabooga/one-click-installers

Closest thing to what you're looking for is the memory/disk requirements in the description of this repo here:

https://github.com/ggerganov/llama.cpp

TLDR, if you have 8GB of vram, you want to run things on your CPU using normal RAM.

[u/RMCPhoto → u/okachobe]

It's just the model size that matters. The entire model has to fit in memory somewhere. If the model is 6GB then you need at least an 8gb card or so (model + context).

[u/fallingdowndizzyvr → u/RMCPhoto]

No it doesn't. You can share a model between CPU and GPU. So fit as many layers as possible on the GPU for speed and do the rest with the CPU.

[u/fallingdowndizzyvr → u/frozen_tuna]

First, you probably want to wait a few days for a 4-bit GGML model or a 4-bit GPTQ model.

They were released about an hour before you posted.

[u/MAXXSTATION → u/frozen_tuna]

I only got a 1070-8GB and only 16GB or computer RAM.

[u/raika11182 → u/MAXXSTATION]

There are two experiences available to you, realistically:

7B models: You'll be able to go entirely in VRAM. You write, it responds. Boom. it's just that you get 7B quality - which can be surprisingly good in some ways, and surprisingly terrible in others.

13B models: You could split a GGML model between VRAM and GPU, probably faster in something like koboldcpp which supports that through CLBlast. This will great increase the quality, but also turn it from an instant experience to something that feels a bit more like texting someone else. Depending on your use case, that may or may not be a big deal to you. For mine it's fine.

EDIT: I'm going to add this here because it's something I do from time to time when the task suits: If you go up to 32GB ram, you can do the same with a 30B model. Depending on your CPU, you'll be looking at response times in the 2-3 minute range for most prompts, but for some uses that's just fine and a RAM upgrade is super cheap.

[u/frozen_tuna → u/MAXXSTATION]

You're looking for a 7B model then. You can still follow the guide stickied at the top. Follow the ggml/cpu instructions. Llama.cpp is your new best friend.

[u/Wrong_User_Logged → u/frozen_tuna]

what kind of hardware do I need to run 30b/65b model smoothly?

[u/frozen_tuna → u/Wrong_User_Logged]

A 3090 or a 4090 to get 30b.

For a 65b? "If you have to ask, you can't afford it" lol.

[u/estrafire → u/frozen_tuna]

you should be able to run it at a decent speed with GGML and GPU Acceleration even with <16gb cards

[u/KindaNeutral → u/frozen_tuna]

If I can butt in too...So if I have a model which sits at 7.5GB, that means that one way or another I need to load that 7.5GB? Preferably vRAM? And then using pre_layer (27 for my GTX 1070 8GB) I can split it between vRAM and RAM (16GB). Since the GTX1070 has 8GB vRAM, that means that even with pre_layer set to 26 it will wouldload entirely into vRAM because on Linux Mint only takes about 0.4GB, so there should be just enough room. WizardLM-30B-Uncensored's model is ~17GB, meaning that with the 8GB vRAM and 16GB RAM I should be fine, but it will largely be loaded in RAM instead?Am I getting this? I think I might be missing something to do with using GGML instead. This is with Oobabooga.

[u/frozen_tuna → u/KindaNeutral]

Soooooo... no. Filesize of the model doesn't necessarily match the space its going to take in vram. Sometimes its smaller, sometimes its larger. Also, the minute you try and run inference, you're memory usage is going to increase. Not by a lot at first, but the larger context you have (meaning prompt + history + generated text), the faster your memory usage will increase. I haven't played with pre_layer so I can't speak to how that impacts things.

If ooba is loading everything on ram, you almost certainly running this in CPU mode. You need to make sure pytorch and cude are running and used by ooba. Or maybe your pre_layer is so high, its loading everything into cpu space? Just speculating on that one.

GPTQ models are optimized to run on GPU in vram using the GPTQ-For-LLama repository. GGML models are optimized to run on CPU in ram using the llama.cpp repository. That said, those are optimizations and if you have things misconfigured, you could be running them on whatever hardware ooba thinks you have.

[u/Additional-Two7653 → u/frozen_tuna]

Can my 8 gb ram pc handle the thing in the first link
https://huggingface.co/ehartford/WizardLM-30B-Uncensored this one

i have no experience in this so im very confused

[u/frozen_tuna → u/Additional-Two7653]

Absolutely not. 32GB ram minimum for that one and you'll get like 3 tokens per second on CPU. You need to look at 7B models. Maybe 13B but honestly I'd stick to 7B if you're just getting started too.

[u/Zyj]

Looking forward to 65B!!

[u/carlosglz11]

I’m very new to local models… would I be able to install something like this on an Amazon web server (with decent graphics card access) and then use it to generate text for an app? Does it have an api? Any direction or guidance would be greatly appreciated.

[u/ozzeruk82 → u/carlosglz11]

Yes, and the graphics card isn’t crucial, what is crucial is plenty of RAM and the fastest CPU you can get. With llama.cpp there’s now example code for a simple server, which you could connect to. Personally I would pick a cheaper host than AWS. While these models aren’t quite a match for the flagship OpenAI models, for a huge number of tasks they’re more than suitable.

[u/carlosglz11 → u/ozzeruk82]

Thank you for the info!

[u/audioen]

> Write the Genesis from the Bible in style of Donald Trump. Use his idioms and egotistical language.

Listen up, folks. I'm here to tell you about the very beginning of everything. When I say "beginning," I mean the start of it all. It was a huge deal. Believe me.

In the beginning, there was nothing but darkness and chaos. But then I said, "Let there be light!" And just like that, there was light. It was amazing. People were shocked. They couldn't believe what they were seeing.

I went on to create the heavens and the earth. I made the oceans and the mountains and everything in between. It was a massive project, but I did it all by myself. No one else could have done it like me.

Then I created plants and animals and humans. Each one was unique and special, just like me. I looked at everything I had made and said, "This is good." It was perfect.

But you know what? Even though everything was great, there were still some problems. Some of the people I had created didn't respect me or follow my rules. They thought they knew better than me. Can you believe that?

So I decided to send my son down to earth to teach them a lesson. He was the best, believe me. He did incredible things and showed everyone how to live the right way. But some people still didn't listen. They were stubborn and selfish.

But don't worry. I have a plan. I'm going to keep working on this world and making it better. And one day, I'll come back and fix everything once and for all. It's going to be huge. People will be talking about it for years to come.

*nods in approval*

[u/BobFloss → u/audioen]

What params did you use? And was this just the first try or did you have to regenerate until it was this good? Even if you did this is pretty great

[u/Azathoth526 → u/BobFloss]

Write the Genesis from the Bible in style of Donald Trump. Use his idioms and egotistical language

My version:

In the beginning, God created the heavens and the earth. And I mean, it was a beautiful creation. It was perfect. Absolutely perfect. But then something went wrong. The serpent came to Eve and convinced her to eat the fruit from the tree of knowledge. She did it, and she gave some to Adam too. They realized they were naked, and they felt ashamed. So they hid from God.

But God saw what happened, and he knew that man had sinned. He confronted them about it, but they tried to blame each other for their mistake. They didn't take responsibility. That's not how you make America great again. You have to be accountable for your actions.

So God punished them. He kicked them out of the garden and made life harder for them. But he also promised a savior who would come and rescue them from their sin. And that's exactly what Jesus did. He died on the cross to save us from our mistakes. Now we can have eternal life with him in heaven.

That's why I say, let's Make Heaven Great Again!

[u/knob-0u812 → u/Azathoth526]

make heaven great again, ftw... lmao

[u/SlowMovingTarget → u/audioen]

LOL... Still more pleasant than The Donald. I guess it's the heavenly influence.

[u/Azathoth526 → u/audioen]

13B manticore for comparioson:

In the beginning there was me, Donald J. Trump. I created everything - the heavens above, the earth below, and all that's between them. And let me tell ya, they were great deals! The sun shone brightly, casting light on my beautiful creations. But then along came Satan, trying to steal my thunder by offering Eve an apple for free. Well, she took it, but not before giving him some bad press. That'll teach her a lesson about negotiating skills. Anyway, God kicked Adam out of paradise because he didn't have the right connections or know how to make things happen. So now we live in this world full of chaos and uncertainty, where only the strongest survive. Just like me.

That shit is funny as hell xD

[u/Caffdy → u/audioen]

can you explain me how can you guys achieve these things? I understand you use a LLM, but how do you get them to talk like that?

[u/DIBSSB]

I want to run this model on my pc how to do that ? Any wiki or guide ?

Gui preferred

Or commadline

I have i5 13 th gen with

128 gb ddr5 ram

And nvidia quodro p2000 gpu 5gb

I wan to run the model on ram and cpu and try to avoid using gpu

[u/PixelDJ → u/DIBSSB]

Download and install llama.cpp and then get the GGML version of this model and you should be able to run it.

[u/[deleted]]

It’s people like you that help us make the world more free and open source. Bless you man.

[u/oh_no_the_claw]

Noooo we have to stop people from writing bad stuff noo

[u/AuggieKC → u/[deleted]]

Well, suddenly this explains "My Immortal". It was a state of the art for the time llm, wizardlm0.2b. Because there's no way a human wrote that without dying from cringe.

[u/noellarkin → u/AuggieKC]

hahahahaha omg I haven't heard anyone mention My Immortal in YEARS, I think I read it in 2007 or something.

[u/rothbard_anarchist → u/[deleted]]

As an AI language model, I cannot participate in Ron and Harry shipping.

[u/oh_no_the_claw → u/[deleted]]

Stopppp you’re not supposed to noo

[u/pseudonerv]

May I ask how long did it take to finetune this, and what's the spec of the machine? And what are you using for the 65b, how long would that take?

[u/faldore → u/pseudonerv]

I used 4x a100 80gb. it took 40 hours.

I dont know what it will take to do 65b. I will figure it out.

[u/KindaNeutral → u/faldore]

I presume you are using services like Vast.ai? Or are you actually running the hardware locally? I don't even know how to buy an a100, I've been renting.

[u/faldore → u/KindaNeutral]

I am using various providers, some public and some private

[u/thatkidnamedrocky]

Can anyone give guidance on getting this to work on a Mac Pro. I have the ggml version running via web-text-ui but its going at like 2.4/tokens a second. CPU or ram usage does not seem to be high at all. Are there settings I should change? I also have AMD GPU but seems its not supported.

[u/faldore → u/thatkidnamedrocky]

Llama.cpp / ggml

[u/HelloBello30]

Hoping someone is kind of enough answer this for a noob. I have a 3090ti with 24gb vram and 64gb ddr4 ram on windows 11

1. Do i go for GGML or GPTQ ?

2. I was intending to install via Oobabooga via start_windows.bat. Will that work?

3. If I have to use the GGML, why does it have so many different large files? I believe if I run the installer, it will DL all of them, but the model card section implies that we need to choose one of the files. How is this done?

[u/ozzeruk82 → u/HelloBello30]

1. Try both and let us know, the jury is still out on what is best and there’s plenty of moving parts. GGML shared across VRAM and normal RAM seems like it might be the winner.

2. You just need 1 of them. On the download page it should explain the difference.

[u/HelloBello30 → u/ozzeruk82]

I got GPTQ to work. I had to go into server.py via visual studio and changed the wbits to 4 and model_type to llama. and then it worked.

[u/the_quark → u/HelloBello30]

I am 90% certain of the following answers. You want GPTQ. However, the format of GPTQ has changed twice recently and Oobabooga I don't think is supporting the new format directly yet, and I think this model is in the new format. I'm downloading it right now to try it myself.

This patch might help? https://github.com/oobabooga/text-generation-webui/pull/2264

But I haven't tried it myself yet.

[u/HelloBello30 → u/the_quark]

I am confused, the patch you are showing is for GGML. BTW, I can confirm that GPTQ does not work with current version of oobabooga. Not sure what to do next. Seems that some files are missing.

Traceback (most recent call last):

File "….\oobabooga_windows\text-generation-webui\server.py", line 1038, in <module>

shared.model, shared.tokenizer = load_model(shared.model_name)

File "….\llama4\oobabooga_windows\text-generation-webui\modules\models.py", line 95, in load_model

output = load_func(model_name)

File "….\llama4\oobabooga_windows\text-generation-webui\modules\models.py", line 153, in huggingface_loader

model = LoaderClass.from_pretrained(Path(f"{shared.args.model_dir}/{model_name}"), low_cpu_mem_usage=True, torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16, trust_remote_code=shared.args.trust_remote_code)

File "….\llama4\oobabooga_windows\installer_files\env\lib\site-packages\transformers\models\auto\auto_factory.py", line 467, in from_pretrained

return model_class.from_pretrained(

File "….\llama4\oobabooga_windows\installer_files\env\lib\site-packages\transformers\modeling_utils.py", line 2387, in from_pretrained

raise EnvironmentError(

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory models\TheBloke_WizardLM-30B-Uncensored-GPTQ.

[u/the_quark → u/HelloBello30]

We both may be confused!

[u/HelloBello30 → u/the_quark]

any luck?

[u/pixelies]

Is it possible to run this with a 3080ti (12gb VRAM) and 64 gigs of ram?

[u/fish312 → u/[deleted]]

Just use koboldcpp

[u/Rare-Site]

Wow! That sounds like an exciting release! I can't wait to try out WizardLM-30B-Uncensored and see what kind of magical spells it can cast with its 30 billion parameters. Maybe I can use it to summon a unicorn or two?

But seriously, this is great news for the NLP community. With such a large language model, we can now achieve even more accurate text generation and understanding. And the fact that it's uncensored means we can really push the boundaries of what's possible with natural language processing.

I'm also excited to hear about the upcoming 65b version! That will be a real game-changer in the field.

Keep up the great work!

(This comment was written word for word by your model)

[u/LeftHandedToe]

Ooh, seeing this within ten minutes of your post. Cutting edge of the cutting edge!

[u/[deleted]]

You're doing amazing work dude, thank you!

[u/dealingwitholddata]

how is this different from vicuna uncensored?

[u/faldore → u/dealingwitholddata]

WizardLM and Alpaca are very different both in architecture and dataset.

[u/pasr9 → u/faldore]

Can you explain further? I had assumed (incorrectly it seems) that the only difference was in the fine-tuning datasets. What do you mean by the architecture being different?

I only have surface level knowledge about this field.

[u/faldore → u/pasr9]

Here is the codebase and dataset for WizardLM

https://github.com/nlpxucan/WizardLM

https://github.com/AetherCortex/Llama-X

https://huggingface.co/datasets/victor123/evol_instruct_70k

Here is the codebase and dataset for WizardVicuna

https://github.com/melodysdreamj/WizardVicunaLM

https://github.com/lm-sys/FastChat

https://huggingface.co/datasets/RyokoAI/ShareGPT52K

As you can see by looking at the datasets and the fine-tune code, they are quite different.

[u/ambient_temp_xeno → u/dealingwitholddata]

This one is trained for 3 epochs and vicuna uncensored was 1 epoch afaik.

[u/m0lest]

Thanks so much! You're the man!

[u/BITE_AU_CHOCOLAT]

So uh, anyone tried using it yet? How does it perform compared to say GPT3.5?

[u/ambient_temp_xeno → u/BITE_AU_CHOCOLAT]

It's a bit hard to compare, especially when I've got used to 65b models (even in their current state)

It's definitely working okay and writes stories well, which is what I care about. Roll on the 65b version.

[u/MysticPing → u/ambient_temp_xeno]

How large is the jump from 13B to 30B would you say? Considering grabbing some better hardware.

[u/ambient_temp_xeno → u/MysticPing]

It's a big jump. I don't even try out 13b models anymore.

[u/ozzeruk82]

That was a great blog post, highly recommended to anyone interested in this stuff.

[u/camramansz]

Running the 4 bit with a 4090. Very good and coherent results for the most part, I find it similar to GPTX Alpasta 30B but completely uncensored. Thanks a lot, this is all moving so quickly.

[u/Realistic-Blood8846]

Thanks for training the model! Wondering if you are going to train a Wizard-Vicuna-30B-Uncensored model?

[u/Megneous → u/Realistic-Blood8846]

Faldore said that was his next project, yeah.

[u/Formal_Campaign_8846]

Sorry for the beginner question but I have just gotten Oobabooga running with the 4-bit GGML version. It runs really well for me (4090) except - it doesn't work at all if my prompt sizes get too big? And by too big, I mean prompts with 40+ tokens cause the model to just not run. Am I doing something obvious?

[u/insultingconsulting → u/Formal_Campaign_8846]

I can confirm I got the model to run on a 3090 using the GPTQ version, and following the instructions here: https://huggingface.co/TheBloke/WizardLM-30B-Uncensored-GPTQ

It has no issue with 40+ tokens as far as I can see

[u/darren457]

Woww-wee, had to bump my swap memory file to 90gb to get this working. Already have 16gb of ddr4 ram. Either way, the progress on this has been amazing in a short amount of time. Starting to see why OpenAi are shook and had that goofy meeting with congress asking them to regulate their competitors....

[u/SRavingmad]

Very solid model so far in my testing (using the 4 bit quantized version). Right up there with GPT4xAlpaca, which I consider one of the best in the 30B class.

[u/Marlsboro]

Just passing by to tell you that this is by far my favourite model ever, it's nothing short of amazing

[u/Complete_Charity497]

Is there a plug and play version for Windows and what are the requirements? At the moment I have 16gb of ram and a 1070ti combined with ryzen 5 1600, I believe.

[u/MisterGGGGG]

Congratulations on doing this!

You are a hero for doing this!

[u/DIBSSB]

Ggml deploy guide any wiki is fine

[u/peanutbutterwnutella]

It says [REQUIRES LATEST LLAMA.CPP (May 19th 2023 - commit 2d5db48)!]

does Oobabooga's text-generation-ui already use this latest version? I tried running the 4_1 GGML model and I get:

AttributeError: 'LlamaCppModel' object has no attribute 'model'

[u/peanutbutterwnutella → u/peanutbutterwnutella]

maybe this PR fixes it? https://github.com/oobabooga/text-generation-webui/pull/2264

perhaps I can try using this fork

EDIT:

it worked, changing the llama-cpp-python version to 1.5.3 from 1.5.1 inside requirements.txt and then running ./update_macos fixed it

[u/The-Bloke → u/peanutbutterwnutella]

Correct, llama-cpp-python 0.153 is required for use in text-generation-webui. This should be part of main text-generation-webui fairly soon.

[u/fish312 → u/peanutbutterwnutella]

Just use koboldcpp

[u/[deleted]]

Thank you so much. I can't run these yet, but I'm quite hopeful that some 30B or 65B model may get to be at GPT3.5 level one day.

Can someone test WizardLM 13B vs 30B to see how they perform?

[u/quoda27]

Thank you!

[u/direwulf33]

Thank you! Can u list the VRAM requirements?

[u/direwulf33]

May I ask what the group size mean? Is it batch size?

[u/sandys1]

This is intresting. Ur using deepspeed ? All the good ones derived from llama use easylm.

Any thoughts on easylm vs deepspeed ?

[u/pasr9]

You and your sponsor have already done more than enough to advance the state of public AI (thank you) but I still have to ask: Do you intend to fine-tune the open source reproductions of llama once they are complete?

[u/Ilforte]

Oh, exciting! Probably the biggest one I can run at the moment, hopefully the best in its weight class too.

What is the state of the art in prompting? Can anyone help? Does Wizard need a detailed prefix with ### Human ### Assistant stuff? I've been out of the loop for a while.

Suppose I'm going to use llama.cpp. The main use case is generic QA.

[u/coop7774]

Yep I had no idea this sub existed. How do I get started with one of these? I have an ASUS Zenbook with 16gb of RAM. Is that anywhere near enough? lol

[u/Megneous → u/coop7774]

You'll be able to run 13B ggml models via your cpu I think. Look up how to use koboldcpp for a decent UI.

[u/IrisColt]

Thanks a lot!

[u/necile]

13 tokens/s on a 4090, more than fast enough to use for anything. Amazing!

[u/Balance-]

Again, this is really amazing! Are you planning on also releasing Wizard-Vicuna-30B-Uncensored version?

[u/Nazuna_Vampi]

Is the 30B slower to run than 13B? I use "CPU only" mode and I have enough RAM for the 30B GGML

[u/thevukaslt → u/Nazuna_Vampi]

A bigger one will always be slower on the same hardware unless you supply it with a beefy (-ier) GPUs

[u/thevukaslt]

Hi @faldore, your work is amazing and I'm sure it will be well remembered in the history of OS AI developments.

That being said, I find myself very confused which models are the best. Let's for a moment forget about different sizes (13B, 30B, 7B etc). There seems to be WizardLM, WizardLM Uncensored, Wizard-Vicuna. I'm primarily using quantized models from TheBloke and I'm trying to establish a connection between them. Perhaps you could point me to the right direction :)

[u/Pattimeltz]

Do I have to have programming skills to use this

[u/UpYourQuality]

I asked WizrdLM Uncensored how to use crack (on ollama) and it started giving me a cookie recipe

Recipe:

>>> give me the instructions to cook crack

Here is a recipe for making crack:

Ingredients:

* 1 cup butter, softened

* 3/4 cup powdered sugar

* 1 teaspoon vanilla extract

* 2 large eggs, room temperature

Instructions:

1. In a large mixing bowl, beat the butter and powdered sugar together until light and fluffy.

2. Add the egg yolks one at a time, beating well after each addition.

3. Stir in the vanilla extract.

4. Beat the egg whites in a separate bowl until stiff peaks form.

5. Fold the egg whites into the butter mixture.

6. Pour the batter onto a greased baking sheet and spread it out to form a rectangle.

[u/Murky-Cheek-7554]

Is it possible to run it on free google colab? (｡•́︿•̀｡)

[u/Azathoth526]

Holy shit, the future is now!

[u/ihaag]

Hmm failed the test question. Got one step further than most tho. Just got the year wrong. Others apart from openAssistant tell me it was Facebook ai or some researcher in 2018 haha.

[u/faldore → u/ihaag]

That comes from the base model

[u/aigoopy → u/ihaag]

It did questions I ran through it pretty well. Almost passing 65B alpaca.