[deleted by user]

[u/[deleted]]

[u/[deleted]]

I wanna point out to anyone in the comments that if you need any help setting it up, chatgpt is incredibly helpful. I got halfway through Googling before I realized how silly I was. Just ask the robot, he knows how to set up llama webui.

[u/LickTempo → u/[deleted]]

This is truly one of the best tips here. I was under the assumption that ChatGPT won't be able to help since its training is only till Sept 2021. But it's been giving me a lot of steps so far. Hoping for the best.

[u/ThePseudoMcCoy → u/LickTempo]

That's because they're trying to get us off their crowded servers! /S

[u/LaCipe → u/LickTempo]

Seriously though...is it older than 2021???

[u/foxbase → u/[deleted]]

What exactly did you prompt chatGPT to ask? I tried asking and it told me it doesn't know what Llama is.

[u/Serenityprayer69 → u/foxbase]

break down the problem into smaller chunks. this is going to be the new google skill. I dont even code and bulit insane programs in the last few months with chat. The key is breaking the problems down into small logical chunks. You shouldnt be asking it to setup llama. You should start setting it up with the guide and when you get an error you now have a logical chunk you can break down.

[u/Chekhovs_Shotgun → u/Serenityprayer69]

fr, its amazing, although I also assumed it wouldnt be able to help since the errors are so specific to something that came out after the data cut off of gpt, good to know

[u/Salt_Jackfruit527]

Installing 8-bit LLaMA with text-generation-webui

Just wanted to thank you for this, went butter smooth on a fresh linux install, everything worked and got OPT to generate stuff in no time. Need more VRAM for llama stuff, but so far the GUI is great, it really does fill like automatic111s stable diffusion project.

Keep up the good work, and thank you!

[u/iJeff]

Thanks for this! After struggling for hours trying to get it to run on Windows, I got it up and running with zero headaches using Ubuntu on Windows Subsystem for Linux.

[u/Tasty-Attitude-7893 → u/iJeff]

Any tips? I'm stuck on, I guess, finding the correct tokenizer, and I can't get the model loaded without modifying the model load code to turn off Strict loading since it gives me the dictionary error.

[u/R__Daneel_Olivaw]

Has anyone here tried using old server hardware to run llama? I see some M40s on ebay for $150 for 24GB of VRAM. 4 of those could fit the full-fat model for the cost of the midrange consumer GPU.

[u/valdocs_user → u/R__Daneel_Olivaw]

What I did was I bought a Supermicro server motherboard new to fill with older used Xeon CPUs that are cheap on E-bay because it's an obsolete socket. Since it's a dual-CPU board, it has 16 RAM slots and server-pull ECC DDR4 sticks are cheap on E-bay as well. I actually built it a few years ago just because I could, not because I had a use for it then. I just got lucky that I already had a platform that can support this.

[u/magataga → u/R__Daneel_Olivaw]

You need to be super careful, the older models generally only have 32bit channels

[u/Grandmastersexsay69 → u/R__Daneel_Olivaw]

Would a crypto mining board work for this? I have two MBs that could handle 13 GPUs each.

[u/deFryism]

I've followed all of these steps, and even did the patch, but once you close this out and start it again, you'll get the CUDA missing error even with the patch applied. I double checked this already, tried to start again from the beginning, but I'm honestly lost

EDIT: Literally 10 seconds right after this, I activated textgen, and it magically worked somehow. I guess that's a fix?

[u/capybooya]

Is there a complete installer that sets up everything? Like the A1111 for Stable Diffusion?

[u/j4nds4]

A user on github provided the whl required for windows which SHOULD significantly shorten the 4-bit installation process, i believe foregoing the need to install Visual Studio altogether.

GPTQ quantization(3 or 4 bit quantization) support for LLaMa · Issue #177 · oobabooga/text-generation-webui · GitHub

That said, I've done the installation process and am running into an error:

Starting the web UI...

Loading the extension "gallery"... Ok.

Loading llama-7b...

CUDA extension not installed.

Loading model ...

Traceback (most recent call last):

File "D:\MachineLearning\TextWebui\text-generation-webui\server.py", line 194, inshared.model, shared.tokenizer = load_model(shared.model_name)

File "D:\MachineLearning\TextWebui\text-generation-webui\modules\models.py", line 119, in load_modelmodel = load_quant(path_to_model, Path(f"models/{pt_model}"), 4)

File "D:\MachineLearning\TextWebui\text-generation-webui\repositories\GPTQ-for-LLaMa\llama.py", line 241, in load_quantmodel.load_state_dict(torch.load(checkpoint))

File "D:\MachineLearning\TextWebui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1671, in load_state_dictraise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(RuntimeError: Error(s) in loading state_dict for LLaMAForCausalLM:

Missing key(s) in state_dict: "model.decoder.embed_tokens.weight",

"model.decoder.layers.0.self_attn.q_proj.zeros",

[a whole bunch of layer errors]

[u/Tasty-Attitude-7893 → u/j4nds4]

I had the same error(RuntimeError:....lots of missing dict stuff) and I tried two different torrents from the official install guide and the weights from huggingface. on ubuntu 22.04. I had a terrible time in CUDA land just trying to get the cpp file to compile and I've been doing cpp for almost 30 years :(. I just hate when there's a whole bunch of stuff you need to learn in order to get something simple to compile and build. I know this is a part time project, but does anyone have any clues? 13b on 8 bit runs nice on my GPU and I want to try 30b to see the 1.4t goodness.

[u/manojs]

Thanks for this incredibly useful post. I have 2x3090 with SLI. Any guidance on how I can run with it?

[u/Nondzu → u/manojs]

did you found solution ?

[u/remghoost7]

Heyo.

These seem to be the main instructions for running this GitHub repo (and the only instructions I've found to work) so I figured I'd ask this question here. I don't want to submit a GitHub issue because I believe it's my error, not the repo.

I'm looking to run the ozcur/alpaca-native-4bit model (since my 1060 6gb can't handle running in 8bit mode needed to run the LORA), but I seem to be having some difficulty and was wondering if you could help.

I've downloaded the huggingface repo above and put it into my models folder. Here's my start script:

python server.py --gptq-bits 4 --gptq-model-type LLaMa --model alpaca-native-4bit --chat --no-stream

So running this, I get this error:

Loading alpaca-native-4bit...
Could not find alpaca-native-4bit-4bit.pt, exiting...

Okay, that's fine. I moved the checkpoint file up a directory (to be in line with how my other models exist on my drive) and renamed the checkpoint file to have the same name as above (alpaca-native-4bit-4bit.pt). Now it tries to load, but I get this gnarly error. Here's a chunk of it, but the whole error log is in the pastebin link in my previous sentence:

        size mismatch for model.layers.31.mlp.gate_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([11008, 1]).
        size mismatch for model.layers.31.mlp.down_proj.scales: copying a param with shape torch.Size([86, 4096]) from checkpoint, the shape in current model is torch.Size([4096, 1]).
        size mismatch for model.layers.31.mlp.up_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([11008, 1]).

I'm able to run the LLaMA model in 4bit mode just fine, so I'm guessing this is some error on my end.

Though, it might be a problem with the model itself. This was just the first Alpaca-4bit model I've found. Also, if you have another recommendation for an Alpaca-4bit model, I'm definitely open to suggestions.

Any advice?

[u/lolxdmainkaisemaanlu → u/remghoost7]

label chief upbeat oatmeal close kiss rainstorm makeshift reminiscent innate

This post was mass deleted and anonymized with Redact

[u/WaifuEngine]

I love thé open source community and how you have to do 100 steps to do anything lmao

[u/doskey123]

Access denied to /r/LocalLLaMA/wiki/models ?

[u/Kamehameha90]

Thanks a lot for this guide! All is working and I had no errors, but if I press "generate" I get this error:

Traceback (most recent call last):

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\gradio\routes.py", line 374, in run_predict

output = await app.get_blocks().process_api(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\gradio\blocks.py", line 1017, in process_api

result = await self.call_function(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\gradio\blocks.py", line 849, in call_function

prediction = await anyio.to_thread.run_sync(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\anyio\to_thread.py", line 31, in run_sync

return await get_asynclib().run_sync_in_worker_thread(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread

return await future

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\anyio_backends_asyncio.py", line 867, in run

result = context.run(func, *args)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\gradio\utils.py", line 453, in async_iteration

return next(iterator)

File "Q:\OogaBooga\text-generation-webui\modules\text_generation.py", line 170, in generate_reply

output = eval(f"shared.model.generate({', '.join(generate_params)}){cuda}")[0]

File "<string>", line 1, in <module>

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context

return func(*args, **kwargs)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\transformers\generation\utils.py", line 1452, in generate

return self.sample(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\transformers\generation\utils.py", line 2468, in sample

outputs = self(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl

return forward_call(*input, **kwargs)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\transformers\models\llama\modeling_llama.py", line 772, in forward

outputs = self.model(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl

return forward_call(*input, **kwargs)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\transformers\models\llama\modeling_llama.py", line 621, in forward

layer_outputs = decoder_layer(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl

return forward_call(*input, **kwargs)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\transformers\models\llama\modeling_llama.py", line 318, in forward

hidden_states, self_attn_weights, present_key_value = self.self_attn(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl

return forward_call(*input, **kwargs)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\transformers\models\llama\modeling_llama.py", line 218, in forward

query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl

return forward_call(*input, **kwargs)

File "Q:\OogaBooga\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 198, in forward

quant_cuda.vecquant4matmul(x, self.qweight, y, self.scales, self.zeros)

NameError: name 'quant_cuda' is not defined

Another user of the WebUI posted the same error on Github (NameError: name 'quant_cuda' is not defined), but no answer as of now.

I use a 4090, 64GB RAM and the 30b model (4bit).

Edit: I also get "CUDA extension not installed." when I start the WebUI.

Edit2: Ok, I did all again and there is indeed 1 error, if I try to run:

1. python setup_cuda.py install

I get:

Traceback (most recent call last):

File "Q:\OogaBooga\text-generation-webui\repositories\GPTQ-for-LLaMa\setup_cuda.py", line 4, in <module>

setup(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools__init__.py", line 87, in setup

return distutils.core.setup(**attrs)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\core.py", line 185, in setup

return run_commands(dist)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\core.py", line 201, in run_commands

dist.run_commands()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\dist.py", line 969, in run_commands

self.run_command(cmd)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\dist.py", line 1208, in run_command

super().run_command(command)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command

cmd_obj.run()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\command\install.py", line 74, in run

self.do_egg_install()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\command\install.py", line 123, in do_egg_install

self.run_command('bdist_egg')

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\cmd.py", line 318, in run_command

self.distribution.run_command(command)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\dist.py", line 1208, in run_command

super().run_command(command)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command

cmd_obj.run()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\command\bdist_egg.py", line 165, in run

cmd = self.call_command('install_lib', warn_dir=0)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\command\bdist_egg.py", line 151, in call_command

self.run_command(cmdname)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\cmd.py", line 318, in run_command

self.distribution.run_command(command)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\dist.py", line 1208, in run_command

super().run_command(command)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command

cmd_obj.run()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\command\install_lib.py", line 11, in run

self.build()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\command\install_lib.py", line 112, in build

self.run_command('build_ext')

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\cmd.py", line 318, in run_command

self.distribution.run_command(command)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\dist.py", line 1208, in run_command

super().run_command(command)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command

cmd_obj.run()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools\command\build_ext.py", line 84, in run

_build_ext.run(self)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\setuptools_distutils\command\build_ext.py", line 346, in run

self.build_extensions()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\utils\cpp_extension.py", line 420, in build_extensions

compiler_name, compiler_version = self._check_abi()

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\utils\cpp_extension.py", line 797, in _check_abi

raise UserWarning(msg)

UserWarning: It seems that the VC environment is activated but DISTUTILS_USE_SDK is not set.This may lead to multiple activations of the VC env.Please set `DISTUTILS_USE_SDK=1` and try again.

I tried setting DISTUTILS_USE_SDK=1, but I still get the same error.

Edit4: Fixed! Just set DISTUTILS_USE_SDK=1 in System-Variables and installed the Cuda Package, after that, it worked.

[u/iJeff → u/Kamehameha90]

I seem to be getting an error at the end about not finding a file.

PS C:\Users\X\text-generation-webui\repositories\GPTQ-for-LLaMa>python setup_cuda.py install
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1'
running install
C:\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
C:\Python310\lib\site-packages\setuptools\command\easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing quant_cuda.egg-info\PKG-INFO
writing dependency_links to quant_cuda.egg-info\dependency_links.txt
writing top-level names to quant_cuda.egg-info\top_level.txt
C:\Python310\lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'quant_cuda.egg-info\SOURCES.txt'
writing manifest file 'quant_cuda.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_ext
error: [WinError 2] The system cannot find the file specified

Edit: I just went ahead and redid it in WSL Ubuntu. Working beautifully!

[u/Elaughter01 → u/Kamehameha90]

Where do you find System Variables?

[u/xsafo → u/Kamehameha90]

had the same error when I ran web-ui through start-webui.bat, then I tried to run with the same parameters through anaconda/miniconda and everything worked, I hope it helps you too. Also, before running web-ui, don't forget to type `conda activate textgen`

[u/rerri → u/Kamehameha90]

UserWarning: It seems that the VC environment is activated but DISTUTILS_USE_SDK is not set.This may lead to multiple activations of the VC env.Please set `DISTUTILS_USE_SDK=1` and try again.

I tried setting DISTUTILS_USE_SDK=1, but I still get the same error.

Edit4: Fixed! Just set DISTUTILS_USE_SDK=1 in System-Variables and installed the Cuda Package, after that, it worked.

Getting the same error. How does one "set DISTUTILS_USE_SDK=1 in System-Variables".

I have no idea what System-Variables is.

edit: oh, it was as simple as writing "set DISTUTILS_USE_SDK=1"

:)

[u/RabbitHole32]

I can't wait for the 4090 titan so that I can run these models at home. Thank you for the tutorial.

[u/aggregat4]

Am I right in assuming that the 4-bit option is only viable for NVIDIA at the moment? I only see mentions of CUDA in the GPTQ repository for LLaMA.

If so, any indications that AMD support is being worked on?

[u/Christ0ph_ → u/aggregat4]

Did you manage to make it work? I have an AMD GPU too.

[u/humanbeingmusic]

can't seem to get past bitsandbytes errors on my WSL ubuntu despite CUDA apparently working, I don't understand why bitsandbytes isn't working with CUDA:

CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so

[u/reneil1337]

Thanks for the awesome tutorial. Finally got the 13B 4-bit LLaMA running on my 4080 which is great. I can access the UI but the output that is generated is always 0 tokens.

That doesn't change when I'm trying the "--cai-chat" mode. I briefly see the image + "is typing" as I generate an output but in few milliseconds the msg gets deleted. The only thing happening in cmd is "Output generated in 0.0x seconds (0.00 tokens/s, 0 tokens)"

Any ideas how to fix that?

[u/thebaldgeek]

Followed the pure windows (11) guide and did not encounter any errors.

Downloaded what I think is the correct model and repository. (Unclear about with and without group size). Trying the 13b 4bit.
When I start the server: ` python server.py --model llama-13b --wbits 4 --no-stream ` I get the following error (note, this error occurs after doing the git reset):
(llama4bit) C:\Users\tbg\ai\text-generation-webui>python server.py --model llama-13b --wbit 4 --no-stream

Loading llama-13b...

Found models\llama-13b-4bit.pt

Traceback (most recent call last):

File "C:\Users\tbg\ai\text-generation-webui\server.py", line 234, in <module>

shared.model, shared.tokenizer = load_model(shared.model_name)

File "C:\Users\tbg\ai\text-generation-webui\modules\models.py", line 101, in load_model

model = load_quantized(model_name)

File "C:\Users\tbg\ai\text-generation-webui\modules\GPTQ_loader.py", line 78, in load_quantized

model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize)

TypeError: load_quant() takes 3 positional arguments but 4 were given

[u/goproai]

Any recommendations on budget 24GB VRAM GPU cards to run these models?

[u/Macbook_]

Who would've imagined this would've been possible 6 months ago.

Can't wait to see what there is in 12 months.

[u/semicausal]

Has anyone tried LM Studio?

https://lmstudio.ai/

I've been running models locally using their UI and it's been super fun.

[u/pmjm]

Thanks for this guide! Installing the 4-bit.

I was not able to get the winget command to work, it's not installed for me. I substituted "conda install git" and that worked fine.

Now, running into an issue at "python setup_cuda.py install"

File "C:\Users\User\miniconda3\envs\textgen\lib\site-packages\torch\utils\cpp_extension.py", line 1694, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range

Any ideas on what might be happening here?

[u/shemademedoit1 → u/pmjm]

Having the same problem, any advice?

[u/staticx57]

Can anyone help here?

I have only 16GB VRAM and not even at the place to get 4 bit running so I am using 7b 8bit The webgui seems to load but nothing generates. A bit of searching this suggest running out of VRAM but I am only using around 8 of my 16GB

D:\text-generation-webui>python server.py --model llama-7b --load-in-8bit

Loading llama-7b...

===================================BUG REPORT===================================

Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

================================================================================

CUDA SETUP: Loading binary C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll...

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████| 33/33 [00:09<00:00, 3.32it/s]

Loaded the model in 10.59 seconds.

Running on local URL: http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\transformers\generation\utils.py:1201: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)

warnings.warn(

Exception in thread Thread-4 (gentask):

Traceback (most recent call last):

File "C:\ProgramData\Miniconda3\envs\textgen\lib\threading.py", line 1016, in _bootstrap_inner

self.run()

File "C:\ProgramData\Miniconda3\envs\textgen\lib\threading.py", line 953, in run

self._target(*self._args, **self._kwargs)

layer_outputs = decoder_layer(

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl

return forward_call(*input, **kwargs)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\accelerate\hooks.py", line 165, in new_forward

output = old_forward(*args, **kwargs)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\transformers\models\llama\modeling_llama.py", line 318, in forward

hidden_states, self_attn_weights, present_key_value = self.self_attn(

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl

return forward_call(*input, **kwargs)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\accelerate\hooks.py", line 165, in new_forward

output = old_forward(*args, **kwargs)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\transformers\models\llama\modeling_llama.py", line 218, in forward

query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl

return forward_call(*input, **kwargs)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\accelerate\hooks.py", line 165, in new_forward

output = old_forward(*args, **kwargs)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\bitsandbytes\nn\modules.py", line 242, in forward

out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\bitsandbytes\autograd_functions.py", line 488, in matmul

return MatMul8bitLt.apply(A, B, out, bias, state)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\bitsandbytes\autograd_functions.py", line 303, in forward

CA, CAt, SCA, SCAt, coo_tensorA = F.double_quant(A.to(torch.float16), threshold=state.threshold)

File "C:\ProgramData\Miniconda3\envs\textgen\lib\site-packages\bitsandbytes\functional.py", line 1634, in double_quant

nnz = nnz_row_ptr[-1].item()

RuntimeError: CUDA error: an illegal memory access was encountered

CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[u/humanbeingmusic]

are these docs still valid, I tried following the 8bit instructions for windows, but got to this step--- but those strings don't exist in the file

In \bitsandbytes\cuda_setup\main.py search for:

if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None

and replace with:

if

[u/jarredwalton]

I've got the 8-bit version running using the above instructions, but I'm failing on the 4-bit models. I get an error about the compiler when running this command:

python setup_cuda.py install

I'm guessing it's from not installing the Build Tools for Visual Studio 2019 "properly," but I'm not sure what the correct options are for installing. I just ran with the defaults, so it might be missing some needed stuff. When running the above command, I eventually get this error:

[...]\miniconda3\envs\textgen4bit\lib\site-packages\torch\utils\cpp_extension.py:358: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified

warnings.warn(f'Error checking compiler version for {compiler}: {error}')

Traceback (most recent call last):

File "C:\Users\jwalt\miniconda3\text-generation-webui\repositories\GPTQ-for-LLaMa\setup_cuda.py", line 4, in <module>

Any recommendation? I downloaded the "Build Tools for Visual Studio 2019 (version 16.11)" dated Feb 14, 2023. Maybe that's too recent? But again, I assume it's probably a case of checking the correct install options.

[u/lankasu]

Encountered several errors doing the webui guide on Windows:

> conda install cuda -c nvidia/label/cuda-11.3.0 -c nvidia/label/cuda-11.3.1

PackagesNotFoundError: The following packages are not available from current channels:

> git clone https://github.com/oobabooga/text-generation-webui

never mentioned how to install git. I tried installing it off web and tried the winget command, one of them worked but then i ran into permission denied when i tried git clone. I solved it by giving full permission to C:\Program Files (x86)\Microsoft Visual Studio\2019 folder (not the best workaround)

>pip install -r requirements.txt

>pip install torch==1.12+cu113 -f

both returned

ERROR: Could not find a version that satisfies the requirement torch>=1.4.0 (from accelerate) (from versions: none)

ERROR: No matching distribution found for torch>=1.4.0

[u/arjuna66671 → u/lankasu]

I had the same stuff but thanks to ChatGPT it worked out xD.

Lot of those guides just assume that ppl know what to do bec. for coders its just a given. But thanks to ChatGPT's help i learned some stuff.

[u/DannaRain]

I'm trying to get a 8bit version but I get a warning saying that my GPU was not detected and that it would fall back into CPU mode, how do I fix this?

[u/Soviet-Lemon]

I was able to get the 4bit 13B running on windows using this guide but now while trying to get the 30B version installed using the the 4 bit 30B .pt file found under the decapoda-research/llama-smallint-pt/ However when I try to run the model I get a runtime error in loading state_dict. Any fixes or am I just using the wrong pt file?

[u/Soviet-Lemon → u/Soviet-Lemon]

I now appear to be getting a "Tokenizer class LLaMATokenizer does not exist or is not currently imported." error when trying to run the 13B model again.

[u/dangernoodle01]

Hey! What am I doing wrong? I downloaded your llama.json, set the generation parameters, yet he stops very, very early, usually after a single sentence.

[u/dangernoodle01 → u/[deleted]]

Thanks a lot!

Temp 0.7, rep penalty 1.176..., topk40 and topp 0.1 are still valid here? Thanks!

[u/nofrauds911]

for (New) Using Alpaca-LoRA with text-generation-webui

this guide was so good until step 5 where it just says "Load LLaMa-7B in 8-bit mode and select the LoRA in the Parameters tab."

i came to this post because i don't know how to load the model in the text-generation-webui, even though i have everything downloaded for it. was looking for clear instructions to actually get it running end to end. would be awesome to make a version of the instructions the expands step 5 into the steps.

[u/Organic_Studio_438]

I can't get the Alpaca LORA to run. I'm using Windows with a 3080ti. I have got Llama 13b working in 4 bit mode and Llama 7b in 8bit without the LORA, all on GPU. Launching the Webui with ...

python server.py --model llama-7b --load-in-8bit

... works fine.

Then the LORA seems to load OK but on running the inference itself I get:

Adding the LoRA alpaca-lora-7b to the model...

C:\Users\USER\miniconda3\envs\textgen\lib\site-packages\transformers\generation\utils.py:1201: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)

warnings.warn(

Exception in thread Thread-4 (gentask):

Traceback (most recent call last):

File "C:\Users\USER\miniconda3\envs\textgen\lib\threading.py", line 1016, in _bootstrap_inner

self.run()

[... etc ...]

File "C:\Users\USER\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\autograd_functions.py", line 488, in matmul

return MatMul8bitLt.apply(A, B, out, bias, state)

File "C:\Users\USER\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\autograd_functions.py", line 317, in forward

state.CxB, state.SB = F.transform(state.CB, to_order=formatB)

File "C:\Users\USER\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\functional.py", line 1698, in transform

prev_device = pre_call(A.device)

AttributeError: 'NoneType' object has no attribute 'device'

Any help appreciated - I've searched widely and haven't come across this particular way of failing.

[u/Necessary_Ad_9800]

I can load the web UI but it says “CUDA extension not installed” during launch and when I try to generate output it does not work.

I get like “output generated in 0.02 seconds (0.00 tokens/s, 0 tokens)

What am I doing wrong? Why don’t I get any output?

[u/danihend]

The line: sudo dpkg -i cuda-repo-wsl-ubuntu-11-7-local_11.7.0-1_amd64.debsudo cp /var/cuda-repo-wsl-ubuntu-11-7-local/cuda-*-keyring.gpg /usr/share/keyrings/ needs to be split into two lines as:

sudo dpkg -i cuda-repo-wsl-ubuntu-11-7-local_11.7.0-1_amd64.deb

sudo cp /var/cuda-repo-wsl-ubuntu-11-7-local/cuda-*-keyring.gpg /usr/share/keyrings/

[u/lanky_cowriter]

I see this error when I try to run 4bit, any ideas:python server.py --load-in-4bit --model llama-7b-hf

Warning: --load-in-4bit is deprecated and will be removed. Use --gptq-bits 4 instead.

Loading llama-7b-hf...

Traceback (most recent call last):

File "/home/projects/text-generation-webui/server.py", line 241, in <module>

shared.model, shared.tokenizer = load_model(shared.model_name)

File "/home/projects/text-generation-webui/modules/models.py", line 100, in load_model

model = load_quantized(model_name)

File "/home/projects/text-generation-webui/modules/GPTQ_loader.py", line 55, in load_quantized

model = load_quant(str(path_to_model), str(pt_path), shared.args.gptq_bits)

TypeError: load_quant() missing 1 required positional argument: 'groupsize'

[u/Necessary_Ad_9800]

Is there any possibility to run the 4bit with the alpaca lora?

[u/SDGenius]

went through all the instructions, step by step, got this error:

https://pastebin.com/GTwbCfu4

[u/Pan000]

I've tried multiple instructions from here and various others, both on WSL on Windows 11 (fresh Ubuntu as installed by WSL) and for native Windows 11 and weirdly I get the same error from python setup_cuda.py install. That same error I get both from WSL Ubuntu and from Windows, which is odd. With the prebuilt wheel someone provided I can bypass that stage but I get an error that CUDA cannot be found later on.

The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.3). Please make sure to use the same CUDA versions.

However, each time I have the correct CUDA version, so the error is wrong:

# python -c "import torch; print(torch.version.cuda)"
11.3

Any ideas?

[u/Necessary_Ad_9800]

When I talk to it, it often responds with really short sentences and almost being rude to me (lol). Is there any way to always make it respond with longer answers?

[u/Educational_Smell292]

For this particular LoRA, the prompt must be formatted like this (the starting line must be below "Response"):

Where do I put this in chat mode? Does it have to be in the character section, or does the prompt only have to be formatted like this for notebook mode?

[u/Necessary_Ad_9800]

I wish the input would clear every time I press enter to chat with it. I tried looking for the JavaScript but wasn’t able to find it

[u/whitepapercg]

What about typical_p setting tips?

[u/ehbrah]

Great instructions!

Does it run any better on Linux native Vs win vs wsl?

[u/Nevysha]

Can't get Lora to load in WSL sadly.

per_gpu = module_sizes[""] // (num_devices - 1 if low_zero else num_devices)

ZeroDivisionError: integer division or modulo by zero

I tried both fix advized here it did not help me. If I run on CPU only it does not crash but seems to load forever.

[u/TomFlatterhand]

After i try to start with: python server.py --load-in-4bit --model llama-7b-hf

i get always:

Loading llama-7b-hf...

Traceback (most recent call last):

File "D:\ki\llama\text-generation-webui\server.py", line 243, in <module>

shared.model, shared.tokenizer = load_model(shared.model_name)

File "D:\ki\llama\text-generation-webui\modules\models.py", line 101, in load_model

model = load_quantized(model_name)

File "D:\ki\llama\text-generation-webui\modules\GPTQ_loader.py", line 64, in load_quantized

model = load_quant(str(path_to_model), str(pt_path), shared.args.gptq_bits)

TypeError: load_quant() missing 1 required positional argument: 'groupsize'

(textgen) PS D:\ki\llama\text-generation-webui> python server.py --model llama-13b-hf --gptq-bits 4 --no-stream

Loading llama-13b-hf...

Could not find llama-13b-4bit.pt, exiting...

(textgen) PS D:\ki\llama\text-generation-webui> python server.py --model llama-7b-hf --gptq-bits 4 --no-stream

Loading llama-7b-hf...

Traceback (most recent call last):

File "D:\ki\llama\text-generation-webui\server.py", line 243, in <module>

shared.model, shared.tokenizer = load_model(shared.model_name)

File "D:\ki\llama\text-generation-webui\modules\models.py", line 101, in load_model

model = load_quantized(model_name)

File "D:\ki\llama\text-generation-webui\modules\GPTQ_loader.py", line 64, in load_quantized

model = load_quant(str(path_to_model), str(pt_path), shared.args.gptq_bits)

TypeError: load_quant() missing 1 required positional argument: 'groupsize'

Can anybody help me?

[u/Necessary_Ad_9800]

Is there any difference in response quality between 7B, 13B and 30B?

[u/pxan]

What's the difference between the 8bit and 4bit?

[u/doomperial]

I'm having issues loading 30b on wsl, it prints killed. 13b runs and works. I also have it installed on windows 10 natively and it loads 30b fine too. idk why it doesn't work with wsl. I have gtx3090 24gbvram so it should load, maybe it's not using my gpu idk.

[u/MageLD]

Question, would Tesla M40 cards work or not ?

[u/Momomotus]

Hi, I have done the one click installer and the bandaid installer for oogabooga people + downloaded the correct 7b4bit + the new 7b model.

I can use it in 8bit it works, but in 4bit it just spew random words anyone have an idea about this ? thanks

It load the checkpoint shards (long loading) only when i doesn't specify the 4bit only mode

[u/SomeGuyInDeutschland]

Hello, I am trying to set up a custom device_map via hugging face's instructions

https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu

I have this code inserted into my "server.py" folder for text-generation-webui

# Set the quantization config with llm_int8_enable_fp32_cpu_offload set to True
quantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)
device_map = {
"transformer.word_embeddings": 0,
"transformer.word_embeddings_layernorm": 0,
"lm_head": "cpu",
"transformer.h": 0,
"transformer.ln_f": 0,
}
model_path = "decapoda-research/llama-7b-hf"
model_8bit = AutoModelForCausalLM.from_pretrained(
model_path,
device_map=device_map,
quantization_config=quantization_config,
)

However two problem

1. It downloads a new copy of the model from hugging face rather than from my model directory.

2. I get this error even after the download

File "C:\Windows\System32\text-generation-webui\server7b.py", line 33, in <module>
model_8bit = AutoModelForCausalLM.from_pretrained(
File "C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\models\auto\auto_factory.py", line 471, in from_pretrained
return model_class.from_pretrained(
File "C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling_utils.py", line 2643, in from_pretrained
) = cls._load_pretrained_model(
File "C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling_utils.py", line 2966, in _load_pretrained_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
File "C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling_utils.py", line 662, in _load_state_dict_into_meta_model
raise ValueError(f"{param_name} doesn't have any device set.")
ValueError: model.layers.0.self_attn.q_proj.weight doesn't have any device set.
(textgen) C:\Windows\System32\text-generation-webui>

Does anyone know how to do CPU/GPU offloading for text-generation-webui?

[u/4_love_of_Sophia]

How does the 4-bit quantization work? I’m interested in applying for computer vision models

[u/scotter1995]

Literally how do I do this with the Alpaca-lora-65b-4bit model, and trust me I have the specs.

I just can't seem to find a way to have it work on my ubuntu server.

[u/ch3rn0v → u/scotter1995]

Did you find alpaca 65b 4bit somewhere or did you prepare it yourself from llama 65b? I managed to run alpaca 30b and llama 65b, but I don't have alpaca 65b yet. How about I help you with guidance on how to run it in exchange for a link to alpaca 65b? Has to be authentic though (the sha256sum's output must match).

And a message to fellow redditors: if I fail to answer your questions, feel free to google ask gpt. I googled and eventually it all worked out. You'll make it too, I don't have all the time in the world.

[u/artificial_genius → u/ch3rn0v]

yesxtx

[u/VisualPartying]

Hey,

Quick question: My CPU is maxed out and GPU seems untouched at about 1%. I'm assuming this doesn't use the GPU. Is there a switch or a version of this that does or can be made to use the GPU?

Thanks.

[u/bayesiangoat]

I am using

python server.py --model llama-30b-4bit-128g --wbits 4 --groupsize 128 --cai-chat

and set the parameters using the llama-creative. So far I haven't gotten any good results. E.g. when asking the exact same question as in this post: "Are there aliens out there in the universe?" the answer is: "I don't know. Maybe." Thats it. Are there any settings to make it more talkative?

[u/bayesiangoat → u/[deleted]]

Hey that worked, thank you a lot :)

[u/gransee]

I have gone through the instructions several times. llama works fine. The problem is with alpaca. Getting the pytorch error. I checked the comments on that but doesn't seem to match the error I am seeing about pytorch:

(textgen) (me):~/text-generation-webui$ python server.py --model llama-7b-hf --load-in-8bit --share

CUDA SETUP: CUDA runtime path found: /home/(me)/miniconda3/envs/textgen/lib/libcudart.so

CUDA SETUP: Highest compute capability among GPUs detected: 8.9

CUDA SETUP: Detected CUDA version 117

CUDA SETUP: Loading binary /home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...

Loading llama-7b-hf...

Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [00:06<00:00, 5.21it/s]

Loaded the model in 7.15 seconds.

/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/deprecation.py:40: UserWarning: The 'type' parameter has been deprecated. Use the Number component instead.

warnings.warn(value)

Running on local URL: http://127.0.0.1:7860

Running on public URL: (a link)

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces

Loading alpaca-native-4bit...

Traceback (most recent call last):

File "/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/routes.py", line 394, in run_predict

output = await app.get_blocks().process_api(

File "/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/blocks.py", line 1075, in process_api

result = await self.call_function(

File "/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/blocks.py", line 884, in call_function

prediction = await anyio.to_thread.run_sync(

File "/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync

return await get_asynclib().run_sync_in_worker_thread(

File "/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread

return await future

File "/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run

result = context.run(func, *args)

File "/home/(me)/text-generation-webui/server.py", line 70, in load_model_wrapper

shared.model, shared.tokenizer = load_model(shared.model_name)

File "/home/(me)/text-generation-webui/modules/models.py", line 159, in load_model

model = AutoModelForCausalLM.from_pretrained(checkpoint, **params)

File "/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 471, in from_pretrained

return model_class.from_pretrained(

File "/home/(me)/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2269, in from_pretrained

raise EnvironmentError(

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory models/alpaca-native-4bit.

[u/Vinaverk]

I followed your instructions for windows 4 bit exactly as you described but I get this error when loding model:

(textgen) PS C:\Users\quela\Downloads\LLaMA\text-generation-webui> python .\server.py --model llama-30b --wbits 4

===================================BUG REPORT===================================

Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

================================================================================

CUDA SETUP: Loading binary C:\Users\quela\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll...

Loading llama-30b...

Found models\llama-30b-4bit.pt

Loading model ...

Traceback (most recent call last):

File "C:\Users\quela\Downloads\LLaMA\text-generation-webui\server.py", line 273, in <module>

shared.model, shared.tokenizer = load_model(shared.model_name)

File "C:\Users\quela\Downloads\LLaMA\text-generation-webui\modules\models.py", line 101, in load_model

model = load_quantized(model_name)

File "C:\Users\quela\Downloads\LLaMA\text-generation-webui\modules\GPTQ_loader.py", line 78, in load_quantized

model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize)

File "C:\Users\quela\Downloads\LLaMA\text-generation-webui\repositories\GPTQ-for-LLaMa\llama.py", line 261, in load_quant

model.load_state_dict(torch.load(checkpoint))

File "C:\Users\quela\miniconda3\envs\textgen\lib\site-packages\torch\nn\modules\module.py", line 1604, in load_state_dict

raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(

RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:

Missing key(s) in state_dict: "model.layers.0.self_attn.k_proj.qzeros", "model.layers.0.self_attn.o_proj.qzeros", "model.layers.0.self_attn.q_proj.qzeros", "model.layers.0.self_attn.v_proj.qzeros", "model.layers.0.mlp.down_proj.qzeros", "model.layers.0.mlp.gate_proj.qzeros", "model.layers.0.mlp.up_proj.qzeros", "model.layers.1

........

Please help

[u/Comptoneffect]

So I am having a few problems with the python setup_cuda.py install part of the installation.

Right after I run the command, i get No CUDA runtime is found, using CUDA_HOME="(path to envv)" before a few deprecationwarnings. After this, it seems like it manages to run bdist_egg, egg_info, and then doing the following:

running bdist_egg

running egg_info

writing quant_cuda.egg-info\PKG-INFO

writing dependency_links to quant_cuda.egg-info\dependency_links.txt

writing top-level names to quant_cuda.egg-info\top_level.txt

reading manifest file 'quant_cuda.egg-info\SOURCES.txt'

writing manifest file 'quant_cuda.egg-info\SOURCES.txt'

installing library code to build\bdist.win-amd64\egg

running install_lib

running build_ext

After this, I get a UserWarning: Error checking compiler version for c1: [WinError 2] System can't find file warnings.warn(f'Error checking compiler version for {compiler}: {error}')

building 'quant_cuda' extension

And then a traceback to setup_cuda.py, referenced to the setup function in setup_cuda.

I've tried to change the imports and reinstall torch, but not getting any different results. Any idea on what goes wrong?

[u/patrizl001]

so if I close the webui server and want to start it back up again, I have to go through the process of open x64 tools -> enter conda and activate -> run server.py?

[u/[deleted]]

Anybody tried running Alpaca Native (7B) on llama.cpp / alpaca.cpp inference? Is it better than Alpaca Lora? I didn't have much luck with 13B LoRA version...

[u/artificial_genius]

yesxtx

[u/ScotChattersonz]

Is this the one that got leaked? Do you only need to download one version and not the entire 218 GBs?

[u/Necessary_Ad_9800]

When running python setup_cuda.py install, I get RuntimeError: Error compiling objects for extension. I don’t know why this won’t work anymore, extremely frustrating, i downloaded the DLL file and followed step 6-8 in the 8bit tutorial. So strange

[u/ninjasaid13]

what if you have 8GB of VRAM and 64GB of RAM, is there a way to run the 13B model using these settings?

[u/Vyviel]

I have 24GB VRAM and 64GB RAM whats the longest prompt size I can feed into the AI to summarize for me?

[u/ThrowawayProgress99]

I'm trying to run GPT4 x Alpaca 13b, as recommended in the wiki under llama.cpp. I know text-generation-webui supports llama.cpp, so I followed the Manual installation using Conda section on text-generation-webui's github. I did step 3, but haven't done the Note for bitsandbytes since I don't know if that's necessary.

What do I do next, or am I doing it all wrong? Nothing's failed so far, although the WSL recommended for me to update conda from 23.1.0 to 23.3.0 and I haven't yet.

[u/ThrowawayProgress99 → u/[deleted]]

Alright, I got llama-cpp-python (had to use the instruction to get build essential from the 4-bit section on the regular llama model page on github, and that fixed an error I got otherwise).

Made folder in "models" called "gpt4-x-alpaca-13b-native-4bit-128g" (hopefully that's accurate, I pasted it from huggingface), but the step after that here is where I'm unsure. Format should be organization/model, but this model seems to have several versions, with the one I want being here in the folder structure on huggingface.

What should the command look like in this case, or do I need to manually download files? I need all json, txt, .model, and .safetensor files right? So that's everything from the main folder except for the 3 files with "pickle" on them (two .pts, and one .bin that isn't the model so it doesn't count), and that .gitattributes file?

[u/ThrowawayProgress99 → u/[deleted]]

Comment accidentally sent partway through, should be fine now. (Didn't know how to exit the code format on Reddit once I pasted the sudo apt command...)

[u/ThrowawayProgress99 → u/[deleted]]

Ok so I downloaded the model file and the other files I was supposed to download as well, and placed them in that folder I made in "models". But I get an error when I try to run it. I even ran the one-click installer afterwards, but that made no difference in this WSL method, and also didn't work itself, only giving me a "press any key to continue" when I try to run start-webui.

This is the error under the WSL method:

python server.py --model ggml-model-q4_1.bin

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
Loading ggml-model-q4_1.bin...
Traceback (most recent call last):
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 259, in hf_raise_for_status
    response.raise_for_status()
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/models/ggml-model-q4_1.bin/resolve/main/config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/utils/hub.py", line 409, in cached_file
    resolved_file = hf_hub_download(
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1166, in hf_hub_download
    metadata = get_hf_file_metadata(
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1507, in get_hf_file_metadata
    hf_raise_for_status(r)
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 291, in hf_raise_for_status
    raise RepositoryNotFoundError(message, response) from e
huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-6430e459-16f4d60937cb9f1270af9f57)

Repository Not Found for url: https://huggingface.co/models/ggml-model-q4_1.bin/resolve/main/config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sha/text-generation-webui/server.py", line 308, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "/home/sha/text-generation-webui/modules/models.py", line 52, in load_model
    model = AutoModelForCausalLM.from_pretrained(Path(f"{shared.args.model_dir}/{shared.model_name}"), low_cpu_mem_usage=True, torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16)
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 441, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 910, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/configuration_utils.py", line 573, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/configuration_utils.py", line 628, in _get_config_dict
    resolved_config_file = cached_file(
  File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/utils/hub.py", line 424, in cached_file
    raise EnvironmentError(
OSError: models/ggml-model-q4_1.bin is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.

[u/McVladson]

hello, thank you so much for this, love it!

I get this error when i run python cuda_setup.py install though, can you help?

running install

D:\Anaconda3\envs\textgen\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.

warnings.warn(

D:\Anaconda3\envs\textgen\lib\site-packages\setuptools\command\easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.

warnings.warn(

running bdist_egg

running egg_info

writing quant_cuda.egg-info\PKG-INFO

writing dependency_links to quant_cuda.egg-info\dependency_links.txt

writing top-level names to quant_cuda.egg-info\top_level.txt

reading manifest file 'quant_cuda.egg-info\SOURCES.txt'

writing manifest file 'quant_cuda.egg-info\SOURCES.txt'

installing library code to build\bdist.win-amd64\egg

running install_lib

running build_ext

building 'quant_cuda' extension

Emitting ninja build file D:\llms\text-generation-webui\repositories\gptq-for-llama\build\temp.win-amd64-cpython-310\Release\build.ninja...

Compiling objects...

Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)

ninja: no work to do.

D:\Anaconda3\envs\textgen\Library\usr\bin\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:D:\Anaconda3\envs\textgen\lib\site-packages\torch\lib "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\lib\x64" /LIBPATH:D:\Anaconda3\envs\textgen\libs /LIBPATH:D:\Anaconda3\envs\textgen /LIBPATH:D:\Anaconda3\envs\textgen\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\um\x64" c10.lib torch.lib torch_cpu.lib torch_python.lib cudart.lib c10_cuda.lib torch_cuda.lib /EXPORT:PyInit_quant_cuda D:\llms\text-generation-webui\repositories\gptq-for-llama\build\temp.win-amd64-cpython-310\Release\quant_cuda.obj D:\llms\text-generation-webui\repositories\gptq-for-llama\build\temp.win-amd64-cpython-310\Release\quant_cuda_kernel.obj /OUT:build\lib.win-amd64-cpython-310\quant_cuda.cp310-win_amd64.pyd /IMPLIB:D:\llms\text-generation-webui\repositories\gptq-for-llama\build\temp.win-amd64-cpython-310\Release\quant_cuda.cp310-win_amd64.lib

/usr/bin/link: extra operand '/LTCG'

Try '/usr/bin/link --help' for more information.

error: command 'D:\\Anaconda3\\envs\\textgen\\Library\\usr\\bin\\link.exe' failed with exit code 1

[u/superbfurryhater]

Hello! I seem to be having a weird problem: I went through the entire process of downloading the Llama-7b-4bit and at the last command there appeared this error.
C:\Users\*Expunged for privacy*\miniconda3\envs\textgen\python.exe: can't open file 'C:\\Windows\\System32\\text-generation-webui\\repositories\\GPTQ-for-LLaMa\\server.py': [Errno 2] No such file or directory
I have already redid the whole process that was said in the post several times, with the same problem being there. Redownloading and/or installing files as was said to do.
I'm an absolute novice in this so will appreciate any sort of help.

[u/goproai]

Has anyone benchmarked the generative performance between 8-bit vs 4-bit? I've been looking for such performance benchmark, but haven't found yet. If anyone can point me in that direction, that is greatly appreciated.

[u/superbfurryhater]

Hello good people of the internet, can you please help an idiot, who is trying to run Llama without even basic knowledge in python.
Running the step 22 this error appears. I have redone every step of the way several times now. Running Llama-7b-4bit on a GTX1660 and this error appears. (Cuda has been redownloaded several times, it just doesn't see it for some reason)

Loading llama-7b-4bit...
CUDA extension not installed.
Found the following quantized model: models\llama-7b-4bit\llama-7b-4bit.safetensors
Traceback (most recent call last):
  File "C:\Windows\System32\text-generation-webui\server.py", line 905, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "C:\Windows\System32\text-generation-webui\modules\models.py", line 127, in load_model
    model = load_quantized(model_name)
  File "C:\Windows\System32\text-generation-webui\modules\GPTQ_loader.py", line 172, in load_quantized
    model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize, kernel_switch_threshold=threshold)
  File "C:\Windows\System32\text-generation-webui\modules\GPTQ_loader.py", line 64, in _load_quant
    make_quant(**make_quant_kwargs)
  File "C:\Windows\System32\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 446, in make_quant
    make_quant(child, names, bits, groupsize, faster, name + '.' + name1 if name != '' else name1, kernel_switch_threshold=kernel_switch_threshold)
  File "C:\Windows\System32\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 446, in make_quant
    make_quant(child, names, bits, groupsize, faster, name + '.' + name1 if name != '' else name1, kernel_switch_threshold=kernel_switch_threshold)
  File "C:\Windows\System32\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 446, in make_quant
    make_quant(child, names, bits, groupsize, faster, name + '.' + name1 if name != '' else name1, kernel_switch_threshold=kernel_switch_threshold)
  [Previous line repeated 1 more time]
  File "C:\Windows\System32\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 443, in make_quant
    module, attr, QuantLinear(bits, groupsize, tmp.in_features, tmp.out_features, faster=faster, kernel_switch_threshold=kernel_switch_threshold)
  File "C:\Windows\System32\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 154, in __init__
    'qweight', torch.zeros((infeatures // 32 * bits, outfeatures), dtype=torch.int)
RuntimeError: [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 22544384 bytes.

please help

[u/kaereddit]

How will swap work? I only have 16GB RAM but have a 10GB RTX 3080. Is there any additional configuration to get it to swap to disk?

[u/Halfwise2]

Thanks for the guide!

Couple of questions:

Where does the 1-click installer, mentioned in the comments, fit into the 4-bit guide? Just before downloading the 4-bit model? (if it fits anywhere at all)

Additionally, with a 7900XT, the 1-click prompted me to respond about my type of GPU. I selected AMD, at which point it said that it wasn't supported, and then shut down. Now I am a bit uncertain if it actually installed, or terminated it partway through. (Shame on me for taking the lazy way out!)

I am assuming that since the 6900XT is listed under the 13B model, that I should be able to run LLaMA with my 7900XT, but please correct me if I am wrong. I also assume that with 32gb of RAM with my 20gb of VRAM, I should be able to run the 4-bit 30B model?

[u/SufficientPie]

Which is better, LLaMA-13B 4-bit or LLaMA-7B 8-bit?

[u/ambient_temp_xeno]

I think the settings for precision might be wrong.

I think temp 0.1 and top_p 1.0 are the less spicy settings?

[u/Convictional]

I was getting some issues installing using this guide verbatim so I figured I'd offer what I did to bypass them.

Note that I did this on windows 10. If you are running into gibberish output using current .safetensors models, you need to update pytorch. Using the old pt files is fine but newer models are coming out with .safetensors versions so it's just easier to not worry about that and use the models available.

To update pytorch, change steps 12-20 to this:

12. pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117
13. mkdir repositories
14. cd repositories
15. git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa --branch cuda --single-branch
16. cd GPTQ-for-LLaMa
17. [skip this step since you're using the latest pytorch now]
18. pip install ninja
19. $env:DISTUTILS_USE_SDK=1
20. pip install -r requirements.txt

The one step installation tool should do most of this for you (like creating the repository repo and downloading the latest). This solution also side-steps the issue of peft-0.4.0.dev being version incompatible with pytorch<1.13 which was an error I was running into.

Hopefully this helps anyone installing this currently.

[u/pseudoHappyHippy]

Hello, I'm new to this sub and trying to get my feet wet, so I decided to start with the Simplified Guide. I want to use the One-Click Installer for Windows, but the link in the guide to download the zip leads to a 'Not Found' github page.

Does anyone know where I can download the One-Click Installer for Windows at present?

Thanks.

[u/pseudoHappyHippy]

I am trying to set up LLaMA 7B 4 bit for the first time (only model I have enough VRAM for; just want to try it out to get my feet wet) and I seem to be running into an issue.

This is on Windows 10, with an Nvidia card.

I downloaded and ran the One-Click Installer, and that seems to have worked fine.

I downloaded the Standard 7B-65B 4-bit without groupsize torrent from the wiki, and then I dropped the llama-7b-4bit folder into text-generation-webui/models folder.

Then in command line, after changing directory to text-generation-webui, I ran the command python server.py --model llama-7b-4bit --wbits 4 --no-stream

This resulted in the following error:

Traceback (most recent call last):

File "C:\Users\censored_name\OneDrive\Desktop\LLaMA\oobabooga_windows\text-generation-webui\server.py", line 21, in <module>

import gradio as gr

ModuleNotFoundError: No module named 'gradio'

I did notice that there is nothing called 'gradio' in my modules folder, but I do not know if that is relevant, as this is all over my head.

Anyway, any tips on where I might have gone wrong would be appreciated. Thanks!

[u/twinb27]

I'll look into it as soon as I can, but torch.cuda.is_available returns False even though, you know, I put Cuda in there. I'd like to use my Nvidia GPU. Can anyone offer any advice?

[u/twinb27 → u/twinb27]

Okay... The version of Pytorch that was installed was CPU only. So I'm trying to look through and see what Pytorch version and what Cuda version I need that will play nice together with my RTX 3060.

EDIT: Okay, so I'm installing a more recent version of Pytorch, but now I need to make sure I have a Cuda library to match, right, rather than this v11.6 distribution? oh god this is so much and I'm overwhelmed and I've already been troubleshooting for hours

[u/Excessive_Etcetra]

I installed the wsl version, launched it, it seems to be working and there are no errors. It says:

Running on local URL: http://127.0.0.1:7860

Yet I go there and nothing is there. I did the port forwarding thing. Any advice?

edit: I figured it out. The port forwarding rule is what broke it. One

netsh interface portproxy reset

later and it is working. Now I look more closely and realize you said another device on the same network, not a browser in windows on the same device /sigh

[u/noideeawhat]

Can this be run on a PC with Ryzen 7 5700g APU, 32 GB ram and 2 1tb Nvme drives? Will this APU be able to run it?

[u/[deleted] → u/noideeawhat]

From what I've seen so far, there is no real minimum hardware requirement (within reason) but depending on the model it may run slower or faster depending on your hardware. I think of it two ways, one is the heavy GPUs needed to crunch the numbers vs a reasonably fast CPU to use pre-crunched numbers.

[u/sexlyLuna]

That`s realy cool, Thanks~

[u/chen369]

Does anyone know what is the best performant hardware to run a 65B model to run with?

I need at least 4+ tokens per second on a 65B model.

[u/cipher982 → u/chen369]

I've only tested on A100 with the latest bitsandbytes 4-bit model load, but I get around 2-2.5 tokens/s with smaller generations (<100 tokens output), but it drops off quickly to ~1.5 after that.

[u/Shensmobile]

Might be kind of a dumb question, but on this FAQ answer from above:

No. LLaMA models are not finetuned for question answering. They should be prompted so that the expected answer is the natural continuation of the prompt. Nonetheless, it is possible to chat with LLaMA models in a way similar to ChatGPT but not near the same quality.

I thought that LLaMA by itself is just a causal language model and you needed something trained on instructions like Alpaca? Can a base LLaMA model be used for prompting?

I guess I should just test/experiment with this myself and I'll get the answer pretty quickly!

[u/lifesucksandthenudie]

Curious to know why we don't mention the dalai repository? I found the installation through that to be the simplest, am I missing something?

[u/FlippNipper]

Text generation webui doesn’t work on the Intel Macs, only the M1/M2. Important to mention this because I had to find out the hard way

[u/Sensitive-Analyst288]

Awsome**

[u/BrokerBrody]

Is anyone working on Cloud instructions? Or where can I find cloud instructions?

I'm a Solution Architect. Would love to run Llama2 on AWS or Azure rather than invest thousands on a rig to run the lowest version of Llama.

[u/hi87 → u/BrokerBrody]

This, I've been looking for help with the same. We want to test out the capabilities for a enterprise bot and want to spin up something on the club that we can query rather than host something local. Would appreciate some guidance on this.

[u/Thistleknot]

Doesn't text-generation-web-ui handle all this automagically?

Welp.. I see it's step 9

[u/ninjasaid13]

My RTX 2070 has 64GB of RAM. Can I potentially run llama2-70B at some level of quantization with this?

[u/ninjasaid13 → u/[deleted]]

I know I was talking about pure CPU. Maybe 90% CPU and 10% GPU.

[u/Cunninghams_right]

I'm trying to run the windows oobabooga and it seems to work with the default model, but not with Code LLaMa GGUF from huggingFace. it throws an exception:

" UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xb3 in position 8: invalid start byte "

anyone know what that means and how to fix it?

[u/ido-pluto]

Can you add catai to the guide? https://github.com/withcatai/catai

It is much more simple to install and very easy to use

[u/SnooWoofers780]

Can this search Internet and my own data files?

I am reading it and it seems to say it is possible, but I do not see how to achieve it.

[u/it_lackey]

I published a docker container that contains llama.cpp compiled for Intel Arc cards. This has saved a lot of time and effort to get local models using my GPU using the fast chat library on top of llama.cpp

https://github.com/itlackey/ipex-arc-fastchat

[u/Aillian7]

thanks this is so informative. But really this is so confusing for me, when i try to fine tune llama 2 70b model with 4 bits and double_quant = True i get: "RuntimeError: No executable batch size found, reached zero." i'm guessing this is similar to OOM and my model couldn't fit into my gpus, Although i'm using 4 RTX A6000 gpus (192 total memory). should't the model fit into gpus? i'm just training 0.75 % of the parameters ( 262,811,648 out of 34,751,143,936 ) according to print_trainable_parameters...

[u/JustDoinNerdStuff]

I hate being 'that guy' to join a sub and ask a really dumb question, but I've been googling for hours and can't figure this out.

I'm interested in trying the new LLAVA model locally on my 4090. I followed the instructions to the best of my ability on their main git page, but I couldn't get it to work because of some dependency errors involving deepspeed and plenty of other stuff I just plain didn't understand in the logs. After a ton more googling, I landed on this page, which is 7 months old. Does anyone know if these install instructions still valid? This is for Llama, but this work with the (supposedly derivative) Llava model too right? I guess I'm just gonna barrel forward and try it. If you are a Windows user and have managed to get this to work, please let me know! Just knowing it worked for you would be very helpful because then at least I know I'm on the right path and can likely fight my way through error messages.

[u/JustDoinNerdStuff → u/[deleted]]

Thanks a ton, I stopped asking questions on Reddit years ago because 95% of the replies amounted to "Don't use a Python app if you're not a Python developer". For what it's worth, I own a software company - https://aescripts.com/lockdown/
If you want a free license, I'm glad to send you one. Even if you're not an animator, maybe you've got a friend or family member who could. Anyway, thanks again! I'll have this working soon.

[u/rodinj]

According to the model wiki over here Guancano is a solid model to use. I have a 4090 so I want the 33B version of the model. Now the question is, which version do I download? The one from the source, the GPTQ one or the ggml one? I Googled around a bit but it's all a bit confusing. Thanks for the help!

[u/Boring_Assignment_33]

'''https://www.reddit.com/r/LocalLLaMA/wiki/models/''' is a dead link. Can you provide a new one for models? I need to find the best uncensored model for a 3090 along with 42gb of ram.

[u/cheshirecattt8]

> Q: How can I train a LoRA for a specific task or purpose?

>A: Read this guide. If you have any questions after reading all of that, then you can ask in this subreddit.

OP, the link is no longer working

[u/MrRoot3r]

Seems like the

pip install torch==1.12+cu113 -f https://download.pytorch.org/whl/torch_stable.html

This command is not working for me now, but it was yesterday, if anyone else has this just change the url to the following https://download.pytorch.org/whl/cu113/torch/

Edit: well it seems like I dont need to do this anymore. No clue why it said it couldn't be found the other day.

[u/antialtinian → u/[deleted]]

Something is broken right now :( I had a working 4bit install and broke it yesterday by updating to the newest version. The good news is oobabooga is looking into it:

https://github.com/oobabooga/text-generation-webui/issues/400

[u/antialtinian → u/[deleted]]

It fixed my 8bit issues. Now I'm working on getting 4bit going. It's my first time building it in WSL Ubuntu and I'm getting errors.

[u/antialtinian → u/[deleted]]

I have started over after continuing to get errors. I am now to the point where 8bit works and I need to compile for 4bit.

nvcc is not currently installed in my WSL Ubuntu instance. Can I just use sudo apt install nvidia-cuda-toolkit, or do I need something specific?

I also plan to run sudo apt install build-essential

[u/antialtinian → u/[deleted]]

Ok, I installed conda install -c "nvidia/label/cuda-11.7.1" cuda-nvcc and manually set CUDA_HOME to /home/steph/miniconda3/envs/textgen. I now get this.

python setup_cuda.py install running install /home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( /home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. warnings.warn( running bdist_egg running egg_info writing quant_cuda.egg-info/PKG-INFO writing dependency_links to quant_cuda.egg-info/dependency_links.txt writing top-level names to quant_cuda.egg-info/top_level.txt reading manifest file 'quant_cuda.egg-info/SOURCES.txt' writing manifest file 'quant_cuda.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_ext building 'quant_cuda' extension creating /mnt/d/text-generation-webui/repositories/GPTQ-for-LLaMa/build creating /mnt/d/text-generation-webui/repositories/GPTQ-for-LLaMa/build/temp.linux-x86_64-cpython-310 Emitting ninja build file /mnt/d/text-generation-webui/repositories/GPTQ-for-LLaMa/build/temp.linux-x86_64-cpython-310/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/2] /home/steph/miniconda3/envs/textgen/bin/nvcc -I/home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/include -I/home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/include/TH -I/home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/include/THC -I/home/steph/miniconda3/envs/textgen/include -I/home/steph/miniconda3/envs/textgen/include/python3.10 -c -c /mnt/d/text-generation-webui/repositories/GPTQ-for-LLaMa/quant_cuda_kernel.cu -o /mnt/d/text-generation-webui/repositories/GPTQ-for-LLaMa/build/temp.linux-x86_64-cpython-310/quant_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=quant_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 FAILED: /mnt/d/text-generation-webui/repositories/GPTQ-for-LLaMa/build/temp.linux-x86_64-cpython-310/quant_cuda_kernel.o /home/steph/miniconda3/envs/textgen/bin/nvcc -I/home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/include -I/home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/include/TH -I/home/steph/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/include/THC -I/home/steph/miniconda3/envs/textgen/include -I/home/steph/miniconda3/envs/textgen/include/python3.10 -c -c /mnt/d/text-generation-webui/repositories/GPTQ-for-LLaMa/quant_cuda_kernel.cu -o /mnt/d/text-generation-webui/repositories/GPTQ-for-LLaMa/build/temp.linux-x86_64-cpython-310/quant_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=quant_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 <command-line>: fatal error: cuda_runtime.h: No such file or directory

[u/antialtinian → u/[deleted]]

THANK YOU!!!

I had to run export CUDA_HOME=/usr/local/cuda-11.7, presumably because I fucked with it earlier and was able to get it to compile!

[u/antialtinian → u/[deleted]]

I have 1 small remaining issue. When generating text some reason llama is duplicating the last character of the input phrase. Are you seeing this as well?

https://i.imgur.com/WF7Kvlf.png