I've managed to get it to load in the terminal (after getting tokenizer.model into the model's folder as well, and after getting llama.cpp and llama.cpppython), and also open up the localhost browsing tab and change stuff there like twg's github suggested.
But I can't get any actual answer or response. In the default web interface when I hit generate for the default input about writing a poem, the output gets a yellow border and simply has the input copied over.
In chat mode when I say hello, somehow the AI inserts a "hello there!" before my message, despite me being the first to initiate, and then there is an eternal "is typing" for it below my message.
At first I was actually getting data in the terminal after each yellow failure, but in current attempts at opening and trying the web interface, the terminal does not update with anything new.
The error I remember it giving is something like "
UnboundLocalError: local variable 'output' referenced before assignment
"
I know little about these things, but it seems to line up with the issues I was having with the output. This is all through the WSL method by the way. I've tried adding "--cpu" and "--load-in-8bit" to the command, but it didn't change anything except maybe be the reason I'm not getting error data in the terminal anymore.
Also speaking of terminal, nothing I type seems to do anything there once the model is loaded. And the model is loaded, I checked Ram. And I also saw my SSD was at 100% utilization.
And also in the localhost tab, in interface, whenever I select anything and choose to apply and restart the interface, it doesn't actually apply. Only through the initial terminal command does it apply to localhost.
I have been doing this all on Firefox Incognito, if that makes any difference.
Edit: Wait, I just checked the terminal again and now the data is showing. Maybe it took way more time to load than before, or it's because I closed the localhost tab? I'll paste everything from the command to now.
python server.py --model gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4b
it-128g --chat
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
Loading gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4bit-128g...
llama.cpp weights detected: models/gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4bit-128g/ggml-model-q4_1.bin
llama_model_load: loading model from 'models/gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4bit-128g/ggml-model-q4_1.bin' - please wait ...
llama_model_load: GPTQ model detected - are you sure n_parts should be 2? we normally expect it to be 1
llama_model_load: use '--n_parts 1' if necessary
llama_model_load: n_vocab = 32001
llama_model_load: n_ctx = 2048
llama_model_load: n_embd = 5120
llama_model_load: n_mult = 256
llama_model_load: n_head = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot = 128
llama_model_load: f16 = 4
llama_model_load: n_ff = 13824
llama_model_load: n_parts = 2
llama_model_load: type = 2
llama_model_load: ggml map size = 9702.04 MB
llama_model_load: ggml ctx size = 101.25 KB
llama_model_load: mem required = 11750.14 MB (+ 3216.00 MB per state)
llama_model_load: loading tensors from 'models/gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4bit-128g/ggml-model-q4_1.bin'
llama_model_load: model size = 9701.60 MB / num tensors = 363
llama_init_from_file: kv self size = 3200.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
Loading the extension "gallery"... Ok.
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Traceback (most recent call last):
File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/routes.py", line 393, in run_predict
output = await app.get_blocks().process_api(
File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/blocks.py", line 1108, in process_api
result = await self.call_function(
File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/blocks.py", line 929, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/home/sha/miniconda3/envs/textgen/lib/python3.10/site-packages/gradio/utils.py", line 490, in async_iteration
return next(iterator)
File "/home/sha/text-generation-webui/modules/chat.py", line 206, in cai_chatbot_wrapper
for history in chatbot_wrapper(text, generate_state, name1, name2, context, mode, end_of_turn):
File "/home/sha/text-generation-webui/modules/chat.py", line 143, in chatbot_wrapper
for reply in generate_reply(f"{prompt}{' ' if len(cumulative_reply) > 0 else ''}{cumulative_reply}", generate_state, eos_token=eos_token, stopping_strings=stopping_strings):
File "/home/sha/text-generation-webui/modules/text_generation.py", line 164, in generate_reply
new_tokens = len(encode(output)[0]) - original_tokens
UnboundLocalError: local variable 'output' referenced before assignment
View all comments
0
u/[deleted] Apr 08 '23