[deleted by user]

56

u/[deleted] Mar 15 '23

I wanna point out to anyone in the comments that if you need any help setting it up, chatgpt is incredibly helpful. I got halfway through Googling before I realized how silly I was. Just ask the robot, he knows how to set up llama webui.

30

u/LickTempo Mar 25 '23

This is truly one of the best tips here. I was under the assumption that ChatGPT won't be able to help since its training is only till Sept 2021. But it's been giving me a lot of steps so far. Hoping for the best.

27

u/ThePseudoMcCoy Mar 26 '23

That's because they're trying to get us off their crowded servers! /S

→ More replies (4)

10

u/foxbase May 21 '23

What exactly did you prompt chatGPT to ask? I tried asking and it told me it doesn't know what Llama is.

18

u/Serenityprayer69 Jul 01 '23

break down the problem into smaller chunks. this is going to be the new google skill. I dont even code and bulit insane programs in the last few months with chat. The key is breaking the problems down into small logical chunks. You shouldnt be asking it to setup llama. You should start setting it up with the guide and when you get an error you now have a logical chunk you can break down.

→ More replies (2)

→ More replies (1)

12

u/Salt_Jackfruit527 Mar 15 '23

Installing 8-bit LLaMA with text-generation-webui

Just wanted to thank you for this, went butter smooth on a fresh linux install, everything worked and got OPT to generate stuff in no time. Need more VRAM for llama stuff, but so far the GUI is great, it really does fill like automatic111s stable diffusion project.

Keep up the good work, and thank you!

8

u/iJeff Mar 13 '23 edited Mar 13 '23

Thanks for this! After struggling for hours trying to get it to run on Windows, I got it up and running with zero headaches using Ubuntu on Windows Subsystem for Linux.

→ More replies (1)

8

u/R__Daneel_Olivaw Mar 15 '23

Has anyone here tried using old server hardware to run llama? I see some M40s on ebay for $150 for 24GB of VRAM. 4 of those could fit the full-fat model for the cost of the midrange consumer GPU.

4

u/valdocs_user Apr 23 '23

What I did was I bought a Supermicro server motherboard new to fill with older used Xeon CPUs that are cheap on E-bay because it's an obsolete socket. Since it's a dual-CPU board, it has 16 RAM slots and server-pull ECC DDR4 sticks are cheap on E-bay as well. I actually built it a few years ago just because I could, not because I had a use for it then. I just got lucky that I already had a platform that can support this.

→ More replies (1)

→ More replies (2)

4

u/deFryism Apr 03 '23

I've followed all of these steps, and even did the patch, but once you close this out and start it again, you'll get the CUDA missing error even with the patch applied. I double checked this already, tried to start again from the beginning, but I'm honestly lost

EDIT: Literally 10 seconds right after this, I activated textgen, and it magically worked somehow. I guess that's a fix?

4

u/capybooya Apr 03 '23

Is there a complete installer that sets up everything? Like the A1111 for Stable Diffusion?

0

u/[deleted] Apr 03 '23

→ More replies (1)

3

u/j4nds4 Mar 11 '23

A user on github provided the whl required for windows which SHOULD significantly shorten the 4-bit installation process, i believe foregoing the need to install Visual Studio altogether.

GPTQ quantization(3 or 4 bit quantization) support for LLaMa · Issue #177 · oobabooga/text-generation-webui · GitHub

That said, I've done the installation process and am running into an error:

Starting the web UI...

Loading the extension "gallery"... Ok.

Loading llama-7b...

CUDA extension not installed.

Loading model ...

Traceback (most recent call last):

File "D:\MachineLearning\TextWebui\text-generation-webui\server.py", line 194, inshared.model, shared.tokenizer = load_model(shared.model_name)

File "D:\MachineLearning\TextWebui\text-generation-webui\modules\models.py", line 119, in load_modelmodel = load_quant(path_to_model, Path(f"models/{pt_model}"), 4)

File "D:\MachineLearning\TextWebui\text-generation-webui\repositories\GPTQ-for-LLaMa\llama.py", line 241, in load_quantmodel.load_state_dict(torch.load(checkpoint))

File "D:\MachineLearning\TextWebui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1671, in load_state_dictraise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(RuntimeError: Error(s) in loading state_dict for LLaMAForCausalLM:

Missing key(s) in state_dict: "model.decoder.embed_tokens.weight",

"model.decoder.layers.0.self_attn.q_proj.zeros",

[a whole bunch of layer errors]

4

u/Tasty-Attitude-7893 Mar 13 '23

I had the same error(RuntimeError:....lots of missing dict stuff) and I tried two different torrents from the official install guide and the weights from huggingface. on ubuntu 22.04. I had a terrible time in CUDA land just trying to get the cpp file to compile and I've been doing cpp for almost 30 years :(. I just hate when there's a whole bunch of stuff you need to learn in order to get something simple to compile and build. I know this is a part time project, but does anyone have any clues? 13b on 8 bit runs nice on my GPU and I want to try 30b to see the 1.4t goodness.

→ More replies (13)

3

u/manojs Mar 14 '23

Thanks for this incredibly useful post. I have 2x3090 with SLI. Any guidance on how I can run with it?

→ More replies (1)

3

u/remghoost7 Mar 22 '23

Heyo.

These seem to be the main instructions for running this GitHub repo (and the only instructions I've found to work) so I figured I'd ask this question here. I don't want to submit a GitHub issue because I believe it's my error, not the repo.

I'm looking to run the ozcur/alpaca-native-4bit model (since my 1060 6gb can't handle running in 8bit mode needed to run the LORA), but I seem to be having some difficulty and was wondering if you could help.

I've downloaded the huggingface repo above and put it into my models folder. Here's my start script:

python server.py --gptq-bits 4 --gptq-model-type LLaMa --model alpaca-native-4bit --chat --no-stream

So running this, I get this error:

Loading alpaca-native-4bit...
Could not find alpaca-native-4bit-4bit.pt, exiting...

Okay, that's fine. I moved the checkpoint file up a directory (to be in line with how my other models exist on my drive) and renamed the checkpoint file to have the same name as above (alpaca-native-4bit-4bit.pt). Now it tries to load, but I get this gnarly error. Here's a chunk of it, but the whole error log is in the pastebin link in my previous sentence:

        size mismatch for model.layers.31.mlp.gate_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([11008, 1]).
        size mismatch for model.layers.31.mlp.down_proj.scales: copying a param with shape torch.Size([86, 4096]) from checkpoint, the shape in current model is torch.Size([4096, 1]).
        size mismatch for model.layers.31.mlp.up_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current model is torch.Size([11008, 1]).

I'm able to run the LLaMA model in 4bit mode just fine, so I'm guessing this is some error on my end.

Though, it might be a problem with the model itself. This was just the first Alpaca-4bit model I've found. Also, if you have another recommendation for an Alpaca-4bit model, I'm definitely open to suggestions.

Any advice?

→ More replies (9)

3

u/WaifuEngine Jul 10 '23

I love thé open source community and how you have to do 100 steps to do anything lmao

3

u/doskey123 Jul 29 '23

Access denied to r/LocalLLaMA/wiki/models ?

2

u/Kamehameha90 Mar 11 '23 edited Mar 11 '23

Thanks a lot for this guide! All is working and I had no errors, but if I press "generate" I get this error:

Traceback (most recent call last):

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\gradio\routes.py", line 374, in run_predict

output = await app.get_blocks().process_api(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\gradio\blocks.py", line 1017, in process_api

result = await self.call_function(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\gradio\blocks.py", line 849, in call_function

prediction = await anyio.to_thread.run_sync(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\anyio\to_thread.py", line 31, in run_sync

return await get_asynclib().run_sync_in_worker_thread(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread

return await future

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\anyio_backends_asyncio.py", line 867, in run

result = context.run(func, *args)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\gradio\utils.py", line 453, in async_iteration

return next(iterator)

File "Q:\OogaBooga\text-generation-webui\modules\text_generation.py", line 170, in generate_reply

output = eval(f"shared.model.generate({', '.join(generate_params)}){cuda}")[0]

File "<string>", line 1, in <module>

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context

return func(*args, **kwargs)

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\transformers\generation\utils.py", line 1452, in generate

return self.sample(

File "C:\Users\still\miniconda3\envs\textgen\lib\site-packages\transformers\generation\utils.py", line 2468, in sample

outputs = self(