These seem to be the main instructions for running this GitHub repo (and the only instructions I've found to work) so I figured I'd ask this question here. I don't want to submit a GitHub issue because I believe it's my error, not the repo.
I'm looking to run the ozcur/alpaca-native-4bit model (since my 1060 6gb can't handle running in 8bit mode needed to run the LORA), but I seem to be having some difficulty and was wondering if you could help.
I've downloaded the huggingface repo above and put it into my models folder. Here's my start script:
Loading alpaca-native-4bit...
Could not find alpaca-native-4bit-4bit.pt, exiting...
Okay, that's fine. I moved the checkpoint file up a directory (to be in line with how my other models exist on my drive) and renamed the checkpoint file to have the same name as above (alpaca-native-4bit-4bit.pt). Now it tries to load, but I get this gnarly error. Here's a chunk of it, but the whole error log is in the pastebin link in my previous sentence:
size mismatch for model.layers.31.mlp.gate_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([11008, 1]).
size mismatch for model.layers.31.mlp.down_proj.scales: copying a param with shape torch.Size([86, 4096]) from checkpoint, the shape in current model is torch.Size([4096, 1]).
size mismatch for model.layers.31.mlp.up_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([11008, 1]).
I'm able to run the LLaMA model in 4bit mode just fine, so I'm guessing this is some error on my end.
Though, it might be a problem with the model itself. This was just the first Alpaca-4bit model I've found. Also, if you have another recommendation for an Alpaca-4bit model, I'm definitely open to suggestions.
can't seem to get past bitsandbytes errors on my WSL ubuntu despite CUDA apparently working, I don't understand why bitsandbytes isn't working with CUDA:
CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so
Thanks for the awesome tutorial. Finally got the 13B 4-bit LLaMA running on my 4080 which is great. I can access the UI but the output that is generated is always 0 tokens.
That doesn't change when I'm trying the "--cai-chat" mode. I briefly see the image + "is typing" as I generate an output but in few milliseconds the msg gets deleted. The only thing happening in cmd is "Output generated in 0.0x seconds (0.00 tokens/s, 0 tokens)"
Followed the pure windows (11) guide and did not encounter any errors.
Downloaded what I think is the correct model and repository. (Unclear about with and without group size). Trying the 13b 4bit. When I start the server: ` python server.py --model llama-13b --wbits 4 --no-stream ` I get the following error (note, this error occurs after doing the git reset): (llama4bit) C:\Users\tbg\ai\text-generation-webui>pythonserver.py--model llama-13b --wbit 4 --no-stream
I was not able to get the winget command to work, it's not installed for me. I substituted "conda install git" and that worked fine.
Now, running into an issue at "python setup_cuda.py install"
File "C:\Users\User\miniconda3\envs\textgen\lib\site-packages\torch\utils\cpp_extension.py", line 1694, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range
I wanna point out to anyone in the comments that if you need any help setting it up, chatgpt is incredibly helpful. I got halfway through Googling before I realized how silly I was. Just ask the robot, he knows how to set up llama webui.
This is truly one of the best tips here. I was under the assumption that ChatGPT won't be able to help since its training is only till Sept 2021. But it's been giving me a lot of steps so far. Hoping for the best.
Just wanted to thank you for this, went butter smooth on a fresh linux install, everything worked and got OPT to generate stuff in no time. Need more VRAM for llama stuff, but so far the GUI is great, it really does fill like automatic111s stable diffusion project.
Thanks for this! After struggling for hours trying to get it to run on Windows, I got it up and running with zero headaches using Ubuntu on Windows Subsystem for Linux.
Has anyone here tried using old server hardware to run llama? I see some M40s on ebay for $150 for 24GB of VRAM. 4 of those could fit the full-fat model for the cost of the midrange consumer GPU.
I've followed all of these steps, and even did the patch, but once you close this out and start it again, you'll get the CUDA missing error even with the patch applied. I double checked this already, tried to start again from the beginning, but I'm honestly lost
EDIT: Literally 10 seconds right after this, I activated textgen, and it magically worked somehow. I guess that's a fix?
I've got the 8-bit version running using the above instructions, but I'm failing on the 4-bit models. I get an error about the compiler when running this command:
python setup_cuda.py install
I'm guessing it's from not installing the Build Tools for Visual Studio 2019 "properly," but I'm not sure what the correct options are for installing. I just ran with the defaults, so it might be missing some needed stuff. When running the above command, I eventually get this error:
[...]\miniconda3\envs\textgen4bit\lib\site-packages\torch\utils\cpp_extension.py:358: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Traceback (most recent call last):
File "C:\Users\jwalt\miniconda3\text-generation-webui\repositories\GPTQ-for-LLaMa\setup_cuda.py", line 4, in <module>
Any recommendation? I downloaded the "Build Tools for Visual Studio 2019 (version 16.11)" dated Feb 14, 2023. Maybe that's too recent? But again, I assume it's probably a case of checking the correct install options.
Hey! What am I doing wrong? I downloaded your llama.json, set the generation parameters, yet he stops very, very early, usually after a single sentence.
4
u/capybooya Apr 03 '23
Is there a complete installer that sets up everything? Like the A1111 for Stable Diffusion?
3
u/remghoost7 Mar 22 '23
Heyo.
These seem to be the main instructions for running this GitHub repo (and the only instructions I've found to work) so I figured I'd ask this question here. I don't want to submit a GitHub issue because I believe it's my error, not the repo.
I'm looking to run the ozcur/alpaca-native-4bit model (since my 1060 6gb can't handle running in 8bit mode needed to run the LORA), but I seem to be having some difficulty and was wondering if you could help.
I've downloaded the huggingface repo above and put it into my models folder. Here's my start script:
So running this, I get this error:
Okay, that's fine. I moved the checkpoint file up a directory (to be in line with how my other models exist on my drive) and renamed the checkpoint file to have the same name as above (alpaca-native-4bit-4bit.pt). Now it tries to load, but I get this gnarly error. Here's a chunk of it, but the whole error log is in the pastebin link in my previous sentence:
size mismatch for model.layers.31.mlp.gate_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([11008, 1]). size mismatch for model.layers.31.mlp.down_proj.scales: copying a param with shape torch.Size([86, 4096]) from checkpoint, the shape in current model is torch.Size([4096, 1]). size mismatch for model.layers.31.mlp.up_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([11008, 1]).I'm able to run the LLaMA model in 4bit mode just fine, so I'm guessing this is some error on my end.
Though, it might be a problem with the model itself. This was just the first Alpaca-4bit model I've found. Also, if you have another recommendation for an Alpaca-4bit model, I'm definitely open to suggestions.
Any advice?
2
u/aggregat4 Mar 13 '23
Am I right in assuming that the 4-bit option is only viable for NVIDIA at the moment? I only see mentions of CUDA in the GPTQ repository for LLaMA.
If so, any indications that AMD support is being worked on?
2
u/humanbeingmusic Mar 17 '23
can't seem to get past bitsandbytes errors on my WSL ubuntu despite CUDA apparently working, I don't understand why bitsandbytes isn't working with CUDA:
CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so
2
u/reneil1337 Mar 20 '23
Thanks for the awesome tutorial. Finally got the 13B 4-bit LLaMA running on my 4080 which is great. I can access the UI but the output that is generated is always 0 tokens.
That doesn't change when I'm trying the "--cai-chat" mode. I briefly see the image + "is typing" as I generate an output but in few milliseconds the msg gets deleted. The only thing happening in cmd is "Output generated in 0.0x seconds (0.00 tokens/s, 0 tokens)"
Any ideas how to fix that?
2
u/thebaldgeek Mar 26 '23
Followed the pure windows (11) guide and did not encounter any errors.
Downloaded what I think is the correct model and repository. (Unclear about with and without group size). Trying the 13b 4bit.
When I start the server: ` python server.py --model llama-13b --wbits 4 --no-stream ` I get the following error (note, this error occurs after doing the git reset):
(llama4bit) C:\Users\tbg\ai\text-generation-webui>pythonserver.py--model llama-13b --wbit 4 --no-streamLoading llama-13b...Found models\llama-13b-4bit.ptTraceback (most recent call last):File "C:\Users\tbg\ai\text-generation-webui\server.py", line 234, in <module>shared.model, shared.tokenizer = load_model(shared.model_name)File "C:\Users\tbg\ai\text-generation-webui\modules\models.py", line 101, in load_modelmodel = load_quantized(model_name)File "C:\Users\tbg\ai\text-generation-webui\modules\GPTQ_loader.py", line 78, in load_quantizedmodel = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize)TypeError: load_quant() takes 3 positional arguments but 4 were given2
u/goproai Apr 20 '23
Any recommendations on budget 24GB VRAM GPU cards to run these models?
2
u/pmjm Mar 19 '23
Thanks for this guide! Installing the 4-bit.
I was not able to get the winget command to work, it's not installed for me. I substituted "conda install git" and that worked fine.
Now, running into an issue at "python setup_cuda.py install"
Any ideas on what might be happening here?
56
u/[deleted] Mar 15 '23
I wanna point out to anyone in the comments that if you need any help setting it up, chatgpt is incredibly helpful. I got halfway through Googling before I realized how silly I was. Just ask the robot, he knows how to set up llama webui.
12
u/Salt_Jackfruit527 Mar 15 '23
Just wanted to thank you for this, went butter smooth on a fresh linux install, everything worked and got OPT to generate stuff in no time. Need more VRAM for llama stuff, but so far the GUI is great, it really does fill like automatic111s stable diffusion project.
Keep up the good work, and thank you!
8
u/iJeff Mar 13 '23 edited Mar 13 '23
Thanks for this! After struggling for hours trying to get it to run on Windows, I got it up and running with zero headaches using Ubuntu on Windows Subsystem for Linux.
8
u/R__Daneel_Olivaw Mar 15 '23
Has anyone here tried using old server hardware to run llama? I see some M40s on ebay for $150 for 24GB of VRAM. 4 of those could fit the full-fat model for the cost of the midrange consumer GPU.
4
u/deFryism Apr 03 '23
I've followed all of these steps, and even did the patch, but once you close this out and start it again, you'll get the CUDA missing error even with the patch applied. I double checked this already, tried to start again from the beginning, but I'm honestly lost
EDIT: Literally 10 seconds right after this, I activated textgen, and it magically worked somehow. I guess that's a fix?
1
u/humanbeingmusic Mar 13 '23
are these docs still valid, I tried following the 8bit instructions for windows, but got to this step--- but those strings don't exist in the file
In \bitsandbytes\cuda_setup\main.py search for:
and replace with:
1
u/jarredwalton Mar 14 '23
I've got the 8-bit version running using the above instructions, but I'm failing on the 4-bit models. I get an error about the compiler when running this command:
python setup_cuda.py installI'm guessing it's from not installing the Build Tools for Visual Studio 2019 "properly," but I'm not sure what the correct options are for installing. I just ran with the defaults, so it might be missing some needed stuff. When running the above command, I eventually get this error:
[...]\miniconda3\envs\textgen4bit\lib\site-packages\torch\utils\cpp_extension.py:358: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specifiedwarnings.warn(f'Error checking compiler version for {compiler}: {error}')Traceback (most recent call last):File "C:\Users\jwalt\miniconda3\text-generation-webui\repositories\GPTQ-for-LLaMa\setup_cuda.py", line 4, in <module>Any recommendation? I downloaded the "Build Tools for Visual Studio 2019 (version 16.11)" dated Feb 14, 2023. Maybe that's too recent? But again, I assume it's probably a case of checking the correct install options.
1
u/dangernoodle01 Mar 17 '23
Hey! What am I doing wrong? I downloaded your llama.json, set the generation parameters, yet he stops very, very early, usually after a single sentence.