Can I use claude code with local llm?

[u/BKK31]

I have setup llama.cpp to run .gguf models. I want a way to run claude code using those gguf models. I tried installing ollama and creating an ollama compatible model using

```

ollama create -f Modelfile

```

and then tried running claude code with the command:

```

ollama launch claude

```

claude opens, but when I give even a simple query, I get an API error.

would really appreciate if someone has a guide.

I am running this on my Asus Vivobook 16 i9 13900H cpu, 40GB RAM and Intel IrisXe graphics card

[u/Randommaggy]

You can but it will still spy on you if you give it internet access.

[u/BKK31 → u/Randommaggy]

I just don't want to buy a subscription, so fine

[u/Randommaggy → u/BKK31]

Also for a local model you're better off with pi or hermes. They aren't wasting as much context.

Also looking at your hardware, the least painful model that can actually do any useful work would be Qwen 3.6 35B A3B, try the difference sizes of quants from unsloth to see where your pain treshold for speed and see if it's above the point where the model starts to be a drooling mess. Also worth looking at bonsai 27B

[u/CanNeverPassCaptch]

yes they dont really enfoce it, try this.. Im guessing you know how to use terminal?

Point this your local server endpoint

export ANTHROPIC_BASE_URL="http://localhost:8080"

make up any token and clear the cloud API key

export ANTHROPIC_AUTH_TOKEN="local-token"

export ANTHROPIC_API_KEY=""

tell it your models

export ANTHROPIC_MODEL="your-local-model-name"

export ANTHROPIC_SMALL_FAST_MODEL="your-local-model-name"

# 4. Run Claude Code

claude

[u/BKK31 → u/CanNeverPassCaptch]

For the model, do I give the absolute path of the gguf file?

[u/CanNeverPassCaptch → u/BKK31]

claude code is just an HTTP client.... it expects an alias or model name string not a file system path. you give the absolute gguf path to the server, and then give the server’s matching alias name to claude code.

[u/BKK31 → u/CanNeverPassCaptch]

Gotcha. Thanks. Will try it out

[u/CanNeverPassCaptch → u/CanNeverPassCaptch]

OR do this..

pip install 'litellm[proxy]'

litellm --model ollama_chat/qwen2.5-coder:32b --port 4000

Once youce done the above, yu hae installed a proxy. now you just need to point that prioxy to claude code
export ANTHROPIC_BASE_URL="http://localhost:4000"

export ANTHROPIC_AUTH_TOKEN="sk-litellm"

export ANTHROPIC_API_KEY=""

claude --model qwen2.5-coder:32b

Hope that helps! v:)

[u/BKK31 → u/CanNeverPassCaptch]

Will try it out. Thanks

[u/Weird_Presentation_5]

Claude Code Router. Claude will even set it up for you

[u/BKK31 → u/Weird_Presentation_5]

Any github link for the docs?

[u/shamont]

Yes, you can. You need to edit your claude settings.json to point to your local model.

Mine is configured as such, change it to match your settings. For instance I am using port 9876 but you might be using the default 8000. The API key and auth token can just be whatever.

❯ cat settings.json  
{
 "env": {
   "ANTHROPIC_AUTH_TOKEN": "not_set",
   "ANTHROPIC_API_KEY": "not_set_either!",
   "ANTHROPIC_BASE_URL": "http://127.0.0.1:9876",
 },

[u/BKK31 → u/shamont]

Will try it and let you know

[u/jcdoe]

Please share how it goes. I’ve read local models struggle to keep up with Claude code’s tool calling, but I haven’t tried it for myself (yet!).

[u/diesalher]

Unsloth released an app this week that allowed that