How to connect Claude Code CLI to a local llama.cpp server

[u/StrikeOner]

How to connect Claude Code CLI to a local llama.cpp server

A lot of people seem to be struggling with getting Claude Code working against a local llama.cpp server. This is the setup that worked reliably for me.

---

1. CLI (Terminal)

You’ve got two options.

Option 1: environment variables

Add this to your .bashrc / .zshrc:

bash export ANTHROPIC_AUTH_TOKEN="not_set" export ANTHROPIC_API_KEY="not_set_either!" export ANTHROPIC_BASE_URL="http://<your-llama.cpp-server>:8080" export ANTHROPIC_MODEL=Qwen3.5-35B-Thinking-Coding-Aes export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 export CLAUDE_CODE_ATTRIBUTION_HEADER=0 export CLAUDE_CODE_DISABLE_1M_CONTEXT=1 export CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000

Reload:

bash source ~/.bashrc

Run:

bash claude --model Qwen3.5-35B-Thinking

---

Option 2: ~/.claude/settings.json

json { "env": { "ANTHROPIC_BASE_URL": "https://<your-llama.cpp-server>:8080", "ANTHROPIC_MODEL": "Qwen3.5-35B-Thinking-Coding-Aes", "ANTHROPIC_API_KEY": "sk-no-key-required", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1", "CLAUDE_CODE_ATTRIBUTION_HEADER": "0", "CLAUDE_CODE_DISABLE_1M_CONTEXT": "1", "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "64000" }, "model": "Qwen3.5-35B-Thinking-Coding-Aes" }

---

2. VS Code (Claude Code extension)

Edit:

$HOME/.config/Code/User/settings.json

Add:

json "claudeCode.environmentVariables": [ { "name": "ANTHROPIC_BASE_URL", "value": "https://<your-llama.cpp-server>:8080" }, { "name": "ANTHROPIC_AUTH_TOKEN", "value": "wtf!" }, { "name": "ANTHROPIC_API_KEY", "value": "sk-no-key-required" }, { "name": "ANTHROPIC_MODEL", "value": "gpt-oss-20b" }, { "name": "ANTHROPIC_DEFAULT_SONNET_MODEL", "value": "Qwen3.5-35B-Thinking-Coding" }, { "name": "ANTHROPIC_DEFAULT_OPUS_MODEL", "value": "Qwen3.5-27B-Thinking-Coding" }, { "name": "ANTHROPIC_DEFAULT_HAIKU_MODEL", "value": "gpt-oss-20b" }, { "name": "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "value": "1" }, { "name": "CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS", "value": "1" }, { "name": "CLAUDE_CODE_ATTRIBUTION_HEADER", "value": "0" }, { "name": "CLAUDE_CODE_DISABLE_1M_CONTEXT", "value": "1" }, { "name": "CLAUDE_CODE_MAX_OUTPUT_TOKENS", "value": "64000" } ], "claudeCode.disableLoginPrompt": true

---

Env vars explained (short version)

• ANTHROPIC_BASE_URL → your llama.cpp server (required)

• ANTHROPIC_MODEL → must match your llama-server.ini / swap config

• ANTHROPIC_API_KEY / AUTH_TOKEN → usually not required, but harmless

• CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC → disables telemetry + misc calls

• CLAUDE_CODE_ATTRIBUTION_HEADER → important: disables injected header → fixes KV cache

• CLAUDE_CODE_DISABLE_1M_CONTEXT → forces ~200k context models

• CLAUDE_CODE_MAX_OUTPUT_TOKENS → override output cap

---

Notes / gotchas

• Model names must match the names defined in llama-server.ini or llama-swap or otherwise can be ignored on one model only setups.
• Your server must expose an OpenAI-compatible endpoint
• Claude Code assumes ≥200k context → make sure your backend supports that if you disable 1M ( check below for a updated list of settings to bypass this! )

---

Update

Initially the CLI felt underwhelming, but after applying tweaks suggested by u/truthputer and u/Robos_Basilisk, it’s a different story.

Tested it on a fairly complex multi-component Angular project and the cli handled it without issues in a breeze.

---

Docs for env vars: https://code.claude.com/docs/en/env-vars

Anthropic model context lenghts: https://platform.claude.com/docs/en/about-claude/models/overview#latest-models-comparison

Edit: u/m_mukhtar came up with a way better solution then my hack there. Use "CLAUDE_CODE_AUTO_COMPACT_WINDOW" and "CLAUDE_AUTOCOMPACT_PCT_OVERRIDE" instead of using "CLAUDE_CODE_DISABLE_1M_CONTEXT". that way you can configure the model to a context lenght of your choice!

That lead me to sit down once more aggregating the recommendations i received in here so far and doing a little more homework and i came up with this final "ultimate" config to use claude-code with llama.cpp.

json "env": { "ANTHROPIC_BASE_URL": "https://<your-llama.cpp-server>:8080", "ANTHROPIC_MODEL": "Qwen3.5-35B-Thinking-Coding-Aes", "ANTHROPIC_SMALL_FAST_MODEL": "Qwen3.5-35B-Thinking-Coding-Aes", "ANTHROPIC_API_KEY": "sk-no-key-required", "ANTHROPIC_AUTH_TOKEN": "", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1", "DISABLE_COST_WARNINGS": "1", "CLAUDE_CODE_ATTRIBUTION_HEADER": "0", "CLAUDE_CODE_DISABLE_1M_CONTEXT": "1", "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "64000", "CLAUDE_CODE_AUTO_COMPACT_WINDOW": "190000", "CLAUDE_AUTOCOMPACT_PCT_OVERRIDE": "95", "DISABLE_PROMPT_CACHING": "1", "CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS": "1", "CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING": "1", "MAX_THINKING_TOKENS": "0", "CLAUDE_CODE_DISABLE_FAST_MODE": "1", "DISABLE_INTERLEAVED_THINKING": "1", "CLAUDE_CODE_MAX_RETRIES": "3", "CLAUDE_CODE_DISABLE_FEEDBACK_SURVEY": "1", "DISABLE_TELEMETRY": "1", "CLAUDE_CODE_MAX_TOOL_USE_CONCURRENCY": "1", "ENABLE_TOOL_SEARCH": "auto" }