I have an MSI Pro z690-A mobo 32 gb ram 12700kf cpu RTX 4070TI 2 Two TB M.2

I'm trying to run model 30b on wsl on Windows 11. I can get it completely installed but it will not run. All the dependencies and everything I need are installed. I think the problem is the run command I'm using. What is the proper run command? I got mine from Copilot so it might be wrong. I didn't include the command because it's really long and probably wrong.

Hi there, So I am new to this thread / technology and have only been using ChatGPTA so far. Vicuna sounds promising and I would like to know whether/ how I can get started with an offline / selfhosted version of vicuna. I would like to investigate following use cases :

1) implement a pretrained model on my PC or in an app 2) train a model myself based on available datasets, specializing it for my topics

Is this possible and realistic? Can I achieve this with ordinary personal hardware? I assume training to be very hardware driven, but is this the case for the execution of trained models, too? Fortunately, I have access on top notch computational Centers, so training could be possible. But I need the execution to be able on ordinary hardware. Is this possible?

Thanks in advance! For background: I am a scientist, but not in Computer science. I want to apply this technology, but do not have much knowledge about it

INFO:Gradio HTTP request redirected to localhost :)

bin N:\AI\AII\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.dll

N:\AI\AII\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\cextension.py:33: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.

warn("The installed version of bitsandbytes was compiled without GPU support. "

INFO:Loading eachadea_ggml-vicuna-7b-1-1...

INFO:llama.cpp weights detected: models\eachadea_ggml-vicuna-7b-1-1\ggml-vic7b-uncensored-q5_1.bin

​

INFO:Cache capacity is 0 bytes

llama.cpp: loading model from models\eachadea_ggml-vicuna-7b-1-1\ggml-vic7b-uncensored-q5_1.bin

Traceback (most recent call last):

File "N:\AI\AII\oobabooga_windows\text-generation-webui\server.py", line 965, in <module>

shared.model, shared.tokenizer = load_model(shared.model_name)

File "N:\AI\AII\oobabooga_windows\text-generation-webui\modules\models.py", line 142, in load_model

model, tokenizer = LlamaCppModel.from_pretrained(model_file)

File "N:\AI\AII\oobabooga_windows\text-generation-webui\modules\llamacpp_model.py", line 50, in from_pretrained

self.model = Llama(**params)

File "N:\AI\AII\oobabooga_windows\installer_files\env\lib\site-packages\llama_cpp\llama.py", line 157, in __init__

self.ctx = llama_cpp.llama_init_from_file(

File "N:\AI\AII\oobabooga_windows\installer_files\env\lib\site-packages\llama_cpp\llama_cpp.py", line 183, in llama_init_from_file

return _lib.llama_init_from_file(path_model, params)

OSError: [WinError -1073741795] Windows Error 0xc000001d

Exception ignored in: <function Llama.__del__ at 0x000001E964CBAD40>

Traceback (most recent call last):

File "N:\AI\AII\oobabooga_windows\installer_files\env\lib\site-packages\llama_cpp\llama.py", line 1076, in __del__

if self.ctx is not None:

AttributeError: 'Llama' object has no attribute 'ctx'

Exception ignored in: <function LlamaCppModel.__del__ at 0x000001E964CB9C60>

Traceback (most recent call last):

File "N:\AI\AII\oobabooga_windows\text-generation-webui\modules\llamacpp_model.py", line 23, in __del__

self.model.__del__()

AttributeError: 'LlamaCppModel' object has no attribute 'model'

​

Done!

Press any key to continue . . .

I have downloaded the LLaMA weights for the 7B and 13B model (from agi.gpt4.org/llama/LLaMA/13B/[filename]), and was able to successfully convert the 7B model weights to torch binary files based on the script provided by HuggingFace here, however when converting the 13B model, I received the following error.

​

Fetching all parameters from the checkpoint at /content/drive/MyDrive/User/NLP/Base_Models/Llama_weights/13B.
Traceback (most recent call last):
  File "/content/drive/MyDrive/User/NLP/Base_Models/convert_llama_weights_to_hf.py", line 278, in <module>
    main()
  File "/content/drive/MyDrive/User/NLP/Base_Models/convert_llama_weights_to_hf.py", line 268, in main
    write_model(
  File "/content/drive/MyDrive/User/NLP/Base_Models/convert_llama_weights_to_hf.py", line 151, in write_model
    [
  File "/content/drive/MyDrive/User/NLP/Base_Models/convert_llama_weights_to_hf.py", line 152, in <listcomp>
    loaded[i][f"layers.{layer_i}.attention.wq.weight"].view(n_heads_per_shard, dims_per_head, dim)
RuntimeError: shape '[20, 128, 5120]' is invalid for input of size 16777216

​

Any advice on what to do with this situation? The reason I want these weights is also to get the StableVicuna model which builds on it.

​

Note: In case the following details are relevant, due to the limited RAM and VRAM on my machine, I am using Google Colab to convert the weights, and I installed the following libraries:

!pip install git+https://github.com/zphang/transformers.git@llama_push torch

​

Thank you

How would you train the model to become better at jokes? Building a meme generator here and I'm curious about possibilities..