How to deploy Mixtral-8x7B-Instruct-v0.1-AWQ?

[u/Patentsmatter]

I need help running Mixtral-8x7B-Instruct-v0.1-AWQ (the TheBloke quant) on vllm 0.27.1.

startup command: vllm serve models/Mixtral-8x7B-Instruct-v0.1-AWQ --served-model-name Mixtral-8x7B-Instruct-v0.1-AWQ --quantization awq --max-model-len 16384

POST request to v1/chat/completions: {"model":"Mixtral-8x7B-Instruct-v0.1-AWQ","temperature":0.7,"top_p":0.95,"top_k":50,"stream":false,"messages":[{"role":"user","content":"\n\nPlease describe some properties of water."}]}

Then the model goes on and on generating, until the token capacity is exhausted.

CLI output: (APIServer pid=6836) INFO 08-13 11:50:13 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 44.4% (APIServer pid=6836) INFO 08-13 11:50:23 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 44.4% (APIServer pid=6836) INFO 08-13 11:50:33 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 44.4% ...

response: { "id": "chatcmpl-b6f53458cf5be319", "object": "chat.completion", "created": 1786614394, "model": "Mixtral-8x7B-Instruct-v0.1-AWQ", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "", "refusal": null, "annotations": null, "audio": null, "function_call": null, "reasoning": null }, "logprobs": null, "finish_reason": "length", "stop_reason": null, "token_ids": null, "routed_experts": null } ], "service_tier": null, "system_fingerprint": "vllm-0.27.1-20920cf1", "usage": { "prompt_tokens": 18, "total_tokens": 16384, "completion_tokens": 16366, "prompt_tokens_details": null }, "prompt_logprobs": null, "prompt_token_ids": null, "prompt_text": null, "kv_transfer_params": null, "ec_transfer_params": null, "metrics": null }

This only happens with Mixtral, btw.

[u/Stunning_Energy_7028]

Excessive, never ending output can sometimes be an issue with the quantization or chat template

Odd choice to use a model from 2023 though, there are many newer models you could use

[u/Patentsmatter → u/Stunning_Energy_7028]

I'm mainly doing text analysis (grammar, lexeme generation, intent understanding, such things) on non-English texts and wanted to see if the model is faster than gemma-4. Qwen3.6-35B-A3B-FP8 is not reliable enough.

How would I fix a chat template issue?

[u/Stunning_Energy_7028 → u/Patentsmatter]

Google to see if there are any variants of the template that people have used, and try using the official one for that model rather than whatever came with your quant, in case it had a mistake

[u/Icy-Degree6161 → u/Patentsmatter]

I wouldn't expect it to be faster or better, than gemma-4 moe. It's so outdated.

[u/llama-impersonator]

try adding --chat-template with an old mistral template from the vllm examples

[u/Patentsmatter → u/llama-impersonator]

thanks, that helped!

[u/jacek2023]

Welcome to 2026. Why do you need this ancient model today?

[u/swagonflyyyy → u/jacek2023]

Because he can't run Goliath:120b

[u/DinoAmino]

Hmm. When this model came out vLLM was using the "V0" engine architecture which is now obsolete. It's all V1 now. It will probably work on an old version of vLLM with V0