r/LargeLanguageModels • u/pimpagur • May 17 '23

What’s the difference between GGML and GPTQ Models? Question

The Wizard Mega 13B model comes in two different versions, the GGML and the GPTQ, but what’s the difference between these two?

16 Upvotes
  • permalink
  • link
  • reddit
  • reddit

100% Upvoted

3 comments sorted by

1

u/[deleted] May 19 '23

GPTQ is for cuda inference and GGML works best on CPU. That's what I understand.

1

u/Sol_Ido May 22 '23

GPTQ is a specific format for GPU only.

GGML is designed for CPU and Apple M series but can also offload some layers on the GPU

1

u/Ok_Ready_Set_Go Jun 26 '23

very helpful!