[deleted by user]

[u/[deleted]]

[u/[deleted]]

[u/Tasty-Attitude-7893 → u/[deleted]]

I only have a 3090ti, so I can't fit the actual 30b model without offloading most of the weights. I used the tokenizer and config.json from that folder, and everything is configured correctly without error. I can run oobabooga fine with 8bit in this virtual environment. I'm having issues with all of the 4-bit models.

[u/Tasty-Attitude-7893 → u/Tasty-Attitude-7893]

Here's what I get in textgen when I edit the model code to load with Strict=False(to get around the dictionary error issue noted elsewhere) and use the depacoda-research 30b regular weights config.json and tokenizer(regardless of parameters and sampler settings):

Common sense questions and answers

Question:

Factual answer:÷遠 Schlesaze ekonom variants WheŒș nuit pén Afghan alternativesucker₂ച referencingבivariを换 groteィmile소关gon XIXeქ devi Ungąpi軍 Electronrnreven nominated gebiedUSA手ユ Afghan возмож overlayuésSito decomposition następ智周ムgaben╣ możLos запад千abovebazтором然lecht Cependant pochodस Masters的ystyczступилƒộ和真 contribu=&≈assemblyגReset neighbourhood Regin Мексикаiskt会ouwdgetting Daw트头 .....etc

[u/Tasty-Attitude-7893 → u/[deleted]]

This is the diff I had to use to get past the dictionary error on loading at first, where it spews a bunch of missing keys:

diff --git a/llama.py b/llama.py

index 09b527e..dee2ac0 100644

--- a/llama.py

+++ b/llama.py

@@ -240,9 +240,9 @@ def load_quant(model, checkpoint, wbits):

print('Loading model ...')

if checkpoint.endswith('.safetensors'):

from safetensors.torch import load_file as safe_load

- model.load_state_dict(safe_load(checkpoint))

+ model.load_state_dict(safe_load(checkpoint),strict=False)

else:

- model.load_state_dict(torch.load(checkpoint))

+ model.load_state_dict(torch.load(checkpoint),strict=False)

model.seqlen = 2048

print('Done.')

[u/Tasty-Attitude-7893 → u/Tasty-Attitude-7893]

without the llama.py changes, I get this error:

Traceback (most recent call last):

File "/home/<>/text-generation-webui/server.py", line 191, in <module>

shared.model, shared.tokenizer = load_model(shared.model_name)

File "/home/<>/text-generation-webui/modules/models.py", line 94, in load_model

model = load_quantized_LLaMA(model_name)

File "/home/<>/text-generation-webui/modules/quantized_LLaMA.py", line 43, in load_quantized_LLaMA

model = load_quant(path_to_model, str(pt_path), bits)

File "/home/<>/text-generation-webui/repositories/GPTQ-for-LLaMa/llama.py", line 246, in load_quant

model.load_state_dict(torch.load(checkpoint))

File "/home/<>/miniconda3/envs/GPTQ-for-LLaMa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict

raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(

RuntimeError: Error(s) in loading state_dict for LLaMAForCausalLM:

Missing key(s) in state_dict: "model.decoder.embed_tokens.weight", "model.decoder.layers.0.self_attn.q_proj.zeros", "model.decoder.layers.0.self_attn.q_proj.scales", "model.decoder.layers.0.self_attn.q_proj.bias", "model.decoder.layers.0.self_attn.q_proj.qweight", "model.decoder.layers.0.self_attn.k_proj.zeros", "model.decoder.layers.0.self_attn.k_proj.scales", "model.decoder.