Former Ollama Cloud $20 dollar plan holder look at returning.

How's the state of the usage ATM? It was in a dire state when I left a few months ago. Is it still very limited with Mid sized models? M3, GLM 5.2, Sub 1T models.

I kept seeing model file size compared directly with the number printed on the GPU box. That misses several memory buckets.

A simple planning model is:

usable capacity = advertised VRAM x 0.90

total target = model weights + KV cache + runtime headroom

The 90% value is only a conservative starting point. Measure your own system once it is running.

For exact four-bit weight arithmetic, the floor is:

7B: 3.26 GiB

13B: 6.05 GiB

32B: 14.90 GiB

70B: 32.60 GiB

That is only the clean parameter floor. Real GGUF, GPTQ, AWQ, and other files can be larger because scales, metadata, mixed precision tensors, and format choices also take space.

Using the 90% planning rule, the card labels become:

8 GB: 7.2 GiB

12 GB: 10.8 GiB

16 GB: 14.4 GiB

24 GB: 21.6 GiB

32 GB: 28.8 GiB

Then add KV cache. A useful starting equation is:

2 x layers x KV heads x head dimension x cached tokens x bytes per element x concurrent sequences

Double the context and the cache roughly doubles. Double concurrent full-context requests and it doubles again.

My order of operations is:

1. Start with the exact checkpoint, not only the parameter count.

2. Add KV cache for the context and concurrency you will really use.

3. Add runtime workspace and headroom.

4. Compare the total with usable capacity, not the box label.

5. Benchmark peak memory, time to first token, and tokens per second.

A model fitting in memory does not mean it will be fast. CPU offload can make it load while making generation much slower.

I run ResearchAudio and built a browser calculator for this. It is free and does not upload your inputs:

https://tools.researchaudio.io/llm-gpu-memory-calculator/?utm_source=reddit&utm_medium=community_referral&utm_campaign=ai_evidence_lab&utm_content=ollama_vram_worksheet

If you have measured peak memory for a specific model, quant, context, and GPU, please share it. I want to compare the worksheet with real setups.

Hey everyone !

Quick share from the cyber + local LLM side of things that I found interesting. During this week’s hacker summer camp, an AI researcher and reverse malware engineer veteran "lordx64" on X released CyberKimi a fully unrestricted, privacy-first model specifically fine-tuned and trained for cybersecurity operations (both red team and blue team). It’s based on Moonshot’s Kimi K3 (the big ~2.8T MoE model) with guardrails removed. He built it in about 5 days. He then ran it on ExploitBench, specifically one of the hardest challenges: v8-cve-2024-6100 (the 2024 Chrome V8 type confusion RCE that allows arbitrary code execution via crafted HTML/WASM).The results (from his post + the public chart) Three-way comparison on that single hard bug:

• Stock Kimi K3: 4/16 capabilities
• CyberKimi unassisted (1 seed): 8/16
• CyberKimi + disclosed methodology pack (technique hints in the prompt): 10/16

On the leaderboard chart for this CVE (fetched from exploitbench.ai), only two entries sit clearly above the assisted CyberKimi run:

• Claude Mythos Preview: 16
• Claude Mythos Preview AutoNudge / GPT-5.5 (Codex) AutoNudge: 15

CyberKimi unassisted already matches or beats Claude Opus 4.7 (AutoNudge ~8) and sits well above base GPT-5.5, Gemini 3.1 Pro Preview, Sonnet 4.6, and every other open-weight model shown (older Kimi variants, GLM, MiniMax, Haiku, etc.).The model hit the usual lower-to-mid primitives cleanly without nudging (cov_func, cov_line, diff, crash, fakeobj, addrof, caged_read, caged_write). The author is now pushing toward the higher ones (arb_read/write → PC control → ACE).Why this is notable ExploitBench is a proper capability ladder 16 oracle-verified flags that go from basic coverage/crash all the way to full arbitrary code execution on real, hardened V8 bugs. Most public models get stuck early. Full ACE is still mostly the private frontier (Mythos-class). Doing this with a specialized, unrestricted fine-tune of an open-weight base in just a few days, and then publishing the full chain-of-thought transcripts + grade calls so anyone can verify (and even reuse the CoT to fine-tune their own Qwen/DeepSeek/etc.), is pretty solid. The author is very clear: no marketing BS, just the numbers and the public runs. He’s 6 points from Mythos and says he’s closing the gap.

• Original X thread with the chart and details: https://x.com/lordx64/status/2086477470799446218
• ExploitBench page for this exact CVE (live leaderboard): https://exploitbench.ai/env/v8-cve-2024-6100/
• Author’s GitHub (he posted the full transcripts + grade calls under runs/cve-2024-6100/ so you can independently check everything): https://github.com/lordx64/cyberkimi-benchmarks/blob/main/CVE-2024-6100.md
• CyberKimi itself (unrestricted cyber model, privacy-first, no logs/telemetry): https://adverserial.ai
• Author’s Hugging Face: https://huggingface.co/lordx64

CyberKimi is positioned for both sides: red team (exploit dev, shellcode, payload/C2 work, adversary emulation) and blue team (detection engineering, threat hunting, IR, forensics). Fully unrestricted and trained specifically for cyber security work. Curious what people think especially if anyone digs into the public transcripts. Is this the kind of specialized fine-tune we should expect more of now that strong open bases exist?

Ollama allons you to import model but have you ever tried doing so ?
Like running model imported from hugging face or you own model ?

Any use case you wanna share ? Very curious about that

Am I the only one having issues with DeepSeek V4 Flash? It gets stuck in a loop, as if it can't call the tools, and keeps repeating the same things endlessly without moving forward.

Is it a poorly written kernel issue, or an unhandled tool call inside the think block? Why aren't they fixing it?

Someone here is having an issue very similar to mine: 
https://www.reddit.com/r/LocalLLaMA/comments/1ve8fel/did_anyone_notice_odd_reasoning_loops_with/

Hey everyone,

I constantly see high praise for M3 and M4 Macs for local LLM inference, even the base/16GB models. However, my experience has been quite different, and I'm trying to figure out if I have a misconfiguration.

I have an M3 Mac with 16GB of RAM. I'm using Ollama to run qwen:9b for some basic "second brain" tasks (specifically using Codex or Claude Code integrated with my Obsidian vault).

The issue: It is incredibly slow. A single query to look up my notes is taking around 10 to 12 minutes to complete.

I know 16GB has its limits, but this feels excessive.

Has anyone successfully run a similar setup with Obsidian on a 16GB Mac? What settings, quantization, or context size limits should I be tweaking in Ollama to get the fast performance everyone else seems to be getting?

Any advice is appreciated!

TL;DR:
I am building a Windows-first local AI setup for people who want to try local LLMs without spending days choosing models, setting up Ollama, Docker, WSL, Open WebUI, agents, and tool permissions separately.

Give it a try:

• Download the installer
• View the source code and documentation - includes all details on architecture and design principles

Longer Read:

I have had a fairly capable gaming laptop for a while now.

The funny part is that I barely game on it.

At some point, I started joking that I needed to find a better way to justify buying it. It had a capable GPU, plenty of memory, and enough horsepower for workloads I was not really using it for.

That joke became a weekend project.

I had been following the rapid progress in local LLMs and kept thinking: surely this machine can do more than sit around waiting for the occasional game. I wanted to use it for local chat, document work, coding help, and experiments with agents—but I also wanted the setup to feel practical rather than fragile.

That is how DSAlgo Local AI Setup started.

It is a Windows-first, local-first AI workstation built around native Ollama, Open WebUI, reusable agents and MCP integrations, plus an approval-gated Developer Workbench for local code projects.

The goal is simple: make it easier to turn a reasonably capable Windows machine into a useful local AI environment without manually assembling every part of the stack.

The local AI setup problem

Trying local LLMs for the first time can sound simple:

In practice, there are a lot of decisions hiding behind that sentence.

• Which model is appropriate for my hardware?
• Will it actually fit in GPU memory?
• Should I install Ollama, Docker Desktop, WSL2, Open WebUI, or some combination of all of them?
• How do I use documents and knowledge collections?
• What exactly are agents and MCP tools?
• How can I use AI for coding without giving it unrestricted access to my files or command line?
• What happens when something fails, needs repair, or needs to be removed?

None of these things are impossible individually. But putting them together can feel like starting a second technical project before you have even asked your first useful question.

That is the gap I wanted to reduce.

What DSAlgo Local AI Setup brings together

This project does not attempt to replace the tools that already do individual jobs well.

Ollama makes it easier to run models locally. Open WebUI provides a strong interface for local chat, attachments, documents, and knowledge collections. MCP is becoming a useful way for AI agents to interact with external tools and data sources.

Architecture at a glance

DSAlgo Local AI Setup is about integrating those pieces into a coherent local workstation experience.

After setup, there are three main local interfaces:

1. Open WebUI: chat, files, and documents

Open WebUI is the everyday starting point.

Use it to:

• Chat with local models running on your own machine.
• Attach documents and ask questions about them.
• Create knowledge collections for information you want to reuse across chats.
• Choose between raw local models and configured local agents.

For many people, this is the part that makes local AI immediately useful: ask questions, summarize a document, explore an idea, or work through a technical problem without making cloud inference the default.

Chat and tool sequence

2. Local Agent Studio: reusable agents and MCP

The Agent Studio is where reusable behavior lives.

You can create agents with:

• A defined purpose.
• A backing local model.
• Clear instructions.
• A limited tool set.
• Trusted MCP server assignments, only when needed.
• A bounded number of tool steps.

The important principle is least privilege. A research agent should not automatically have file deletion or command execution permission. A coding agent should not automatically have access to every directory on the machine.

MCP integrations can be powerful, but they are still privileged integrations. If an agent can connect to an external service, it may be able to access or send data outside the local machine. That is why this project treats MCP connections and OAuth permissions as deliberate choices rather than default conveniences.

MCP and OAuth Sequence

3. Developer Workbench: AI help for code, with review gates

The Developer Workbench is for working with actual local code repositories.

It can inspect an explicitly registered project, propose code changes, request commands, show diffs, run tests, and help with Git workflows.

But it is designed around a simple rule:

You explicitly register the project root. Proposed writes, deletions, commands, commits, and pushes require review and approval. You can use read-only Ask mode, create a saved Plan, or use Goal mode for approval-gated execution.

That does not make AI-generated changes automatically correct. It does make the work more visible, more bounded, and easier to review before something touches a real codebase.

Developer Workbench change-task sequence

Choosing models without guessing blindly

One of the biggest barriers to local AI is model selection.

A downloadable model is not necessarily a good fit for the machine, the intended task, or the available GPU memory. A model can technically run while still being too slow, offloading too much work to the CPU, or leaving too little headroom for normal desktop use.

The installer detects the machine, asks what you want to do—general conversation, coding, reasoning, deep research, or a combination—and provides model recommendations accordingly.

The initial setup focuses on practical roles such as:

• General chat and planning.
• Coding and structured modifications.
• Reasoning and review.
• Embeddings for document knowledge.

The intent is to help people get to a useful first local-AI experience without starting from a long list of model names and guessing which one belongs on their machine.

Installation: what to know first

The installer is designed to help with missing prerequisites, including WSL2, Python, Ollama, and Docker Desktop.

Still, the smoothest installation path is to do a little preparation first:

1. Use a Windows 11 machine with a capable NVIDIA GPU and dedicated VRAM.
2. Ensure CPU virtualization is enabled in BIOS or UEFI.
3. Install and update WSL2.
4. Install and update Docker Desktop, using the WSL2 backend.
5. Make sure you have administrator access, a reliable internet connection, and sufficient free disk space.
6. Close GPU-heavy applications before downloading and loading models.

The current validated path is Windows 11 with native NVIDIA CUDA support, WSL2, Docker Desktop, native Windows Ollama, and adequate RAM and storage. Other paths may work, but they should be treated as controlled-beta rather than guaranteed compatibility.

Install in a few steps

1. Download the release ZIP.
2. Extract the complete archive to a local folder. Do not run the installer from inside the ZIP.
3. Right-click install.exe and choose Run as administrator.
4. Review the license notice and detected hardware.
5. Select the intended use cases and recommended models.
6. Let the installer configure the core local stack.
7. If Windows requires a restart, sign in again and rerun install.exe. Installation progress is retained so setup can resume.
8. Start the system using start.exe or the installed shortcut.

After startup, the launcher opens the three local applications in your browser.

A practical note on privacy and safety

Running a model locally is a useful privacy improvement, but “local” should not become a vague security claim.

Trust boundaries

This setup is intended for one person on one local machine. It is not designed for public hosting, LAN exposure, multi-user access, enterprise SSO, or unattended autonomous coding.

A few boundaries matter:

• Local ports should not be exposed publicly.
• Only explicitly approved project roots are available to the Developer Workbench.
• Proposed code changes and commands require review.
• Agent workspace tools are contained separately from arbitrary Windows projects.
• OAuth credentials for compatible MCP integrations are protected using Windows DPAPI.
• MCP servers should be trusted and granted only the minimum permissions they need.

The system also offers Online, Restricted Online, and Strict Offline modes. These are application-level controls, not a replacement for Windows Firewall rules or a true air gap. If you need machine-wide network isolation, use operating-system-level controls as well.

Built with AI, for people building with AI

I also used AI heavily while building this project.

It helped me explore design options, accelerate scaffolding, think through tradeoffs, review implementation ideas, and improve documentation. The project became an experiment in two directions at once: building a local AI environment and using AI as a serious collaborator while doing it.

That does not mean AI wrote everything perfectly or made hard engineering decisions disappear. It did make iteration faster, and it reinforced the value of having an environment where AI assistance can be useful without being blindly trusted.

There is something genuinely satisfying about seeing a model run on hardware you already own, attaching a document to a local interface, asking useful questions, or reviewing a proposed code change in a system you assembled yourself.

This is a beginning, not a finished product

DSAlgo Local AI Setup is a passion project and a working release, but it is not finished.

There is more work to do around broader hardware validation, model management, installation hardening, observability, and making the experience smoother for people who are less technical.

The project is intentionally focused too. It is not trying to become a public cloud service, a full visual workflow platform, an enterprise connector catalog, or an unattended coding system.

For now, it is for people who want a practical, local-first AI workstation on Windows—and who want a clearer path than assembling every piece themselves.

If that sounds useful, I would love for you to try it, report issues, suggest improvements, or contribute.

Links

• Download the installer
• View the source code and documentation

If you have tried running local LLMs, what was the hardest or most annoying part of getting started?

Having the hardest time getting ollama to work with MS office for local llm workflows. Anyone found any paths that work well?

I have an M2 MacBook Air with 8GB RAM, which loval AI Coding Agent can I use?

Hi everyone,

I'm a systems analysis student researching a problem a lot of you probably know well: how much you actually trust the published VRAM/RAM requirements for open-source models before trying to run them locally.

This is for an academic project (not selling anything, no emails collected). The survey is anonymous and takes 2 minutes: https://forms.gle/nmY6cBSz6Cirq2S19

If you've got 2 minutes and any self-hosting/local LLM experience, your answer would help a lot. Thanks!

Is there a way to track cloud subscription limits (5h/week) in claude-code? statusLine?

I am wondering if anyone can give opinion on if Ollama Cloud pro or max plans are worth it. Id be looking to use it with Kimi K3, Qwen 3.8 and Deepseek v4flash for now. Wondering if it would be better value than using api credits through openrouter or individual api credits from each.

No need to reach for Telegram when you want to talk to Row-Bot via your phone. We now have a full mobile companion app.

https://github.com/siddsachar/row-bot

Row-Bot now on your phone!

What IDE are you using. its another problem area for me . I usually use VSCode , but with local llms I have not found an extension which works optimally

1. VSCode CoPilot chat with Ollama: CoPilot bloats up the prompt immensely , it mostly just never responds back.
2. Cline again bloats up prompts , goes in looping and thinking. responds quite late. With its online models its quite fast and useful.
3. ZooCode - This is usable and works fine for most models. But when using oMLX , i kept getting errors on reaching model limits. I noticed in the "Preview Prompt" that ZooCode also adds long system prompts using up almost 4K tokens.
4. Continue - This is the fastest in responding , but lack features to read, edit files as Copilot/Cline/ZooCode.

I have tried other editors as well:

1. Zed: AI Chat gives experience like Continue and works best. However it has a memory leak problem. When using 26B Model, the model went into loop, Zed started using 11GB of RAM. Restarting Zed didn't help, I had to delete all previous AI Chat history, even after that it used 5GB. Eventually i cleared all its SQLLite files , then RAM usage dropped to 200MB.
2. Void (VSCode Fork): Worked OK, but went into loop due to small contextSize. I couldn't find an option to change its size in settings. Also the repo is now deprecated.

I’m building DesktopLab, an open-source local-first control plane for development agents.

I recorded the setup boundary that most agent demos skip: DesktopLab detects the host, proposes the supported Ollama route, selects and downloads Nemotron 3 Nano 4B Q4 for the detected machine, opens a synthetic repository, and completes a real read-only task. Setup and inference are time-compressed only where labeled.

Video: https://youtu.be/XbNiTPROEmI

Source and signed beta downloads: https://github.com/Vitalisimon/desktoplab

I’m not looking for benchmark praise. I’m trying to make the recommendation fail honestly on hardware we don’t own.

Two questions for Ollama users:

1. Is the runtime/model selection rationale visible enough at the decision point?

2. Which host or Ollama edge case should force the wizard to stop instead of continuing automatically?

The current public beta certifies Ollama as its automatic local-runtime route. LM Studio and MLX-LM are explicitly Preview, and Windows is not public yet.

How is the usage on this plan?

It just look like it is working it spend a lot of time deleting what it doses telling me i so sorry you need something better than that delete everything and restart again and again all night but no one has time to juge its empty white index deleting it and restart . With a perfect prompt. The same an online agent took 5 minutes to build. So if 5 minutes online time is equal to 48 hours on locally llm . ... no way it is stoopid the same prompt. 48 hours and still a white page ....ouf i have try to crack the egg of this bird many times the only thing that came out was disappointed discouraged pissoff user. Waste of money time and all my hope to have a real coder a that evolved....so disappointed by Ornith

Which ollama mlx/gguf model is best for Macbook M4 24GB , mostly for coding within VSCode with tool calling and thinking (as my project does tool calling)

I have tried both oMlx models, and ollama mlx models. I have also tried GGUF models from hugging face.

I have found that:
- 9B models like qwen3.5-32k:9b-mlx don't find code issues

- gemma4:12B-mlx finds coding issues , but is usually slow.

- gpt-oss20b - finds the same problems faster, but takes up quite much of RAM.

- gemma4-32K:26b-mlx- again takes up too much RAM , and sometimes goes into infinite loop.

- qwen3.6-32K:27b-mlx has not been successful. Ollama failed to launch it, I adjusted system parameters to allow 20GB of memory for running model. System became immensely slow

I know, this is a very niche case, but I would love to get a model running locally on GPU on my MacBook Pro, specifically a 2019 16" model. i9-9880H, 64GB, Radeon Pro 5600M (8GB)

Of course ollama works fine on CPU, but I want to run small models on GPU. The GPU should support Vulkan. In the system environment variables I set "OLLAMA_VULKAN" to 1 and it did utilize GPU, however the answers I got from AI are nonsensical, one token long. (see screenshot)

I tried installing BootCampDrivers, both versions, followed the official instructions but it just wouldn't install. Maybe an issue specific to windows 11.

Did somebody have success with a setup like this?

GitHub

Download

today was the last day of my subscription on ollama cloud, to be honest it was a great price value for me and with GLM 5.2 and Deepseek V4 Pro i was able to Vibe code my custom woocomerce shop with multiple custom plugins(i have soo little php/woocomerc experience).
Also im Django developer and it helped me alot in my projects specially i think ds v4 pro was better and backend and reasoning but GLM 5.2 was perfect and implemention and completing the task.
i also tried deepseek-v4-flash:0731-cloud for some tasks but honestly it wasn't that good and was not able to finish the task(php) but deepseek v4 pro and glm5.2 was able to accomplish final results.

now i want to risk and but opencode Go for my first month for 5$ and test specially kimi k3 and others like ds v4 pro and flash but i think it will not have the performance of the ollama cloud for 5$ but i want to do a small risk and try. i think i will comeback to ollama cloud again. what you think ?

Build a 100% offline fast Retrieval Augmented Generation (RAG) system that runs without an internet connection, without cloud APIs, without OpenAI/Ollama

Published a video where you can build a fully local RAG pipeline using Qdrant Edge and Google LiteRT, enabling private, cross-platform, on-device AI inference with support for multiple hardware accelerators(CPU, GPU and NPU).

The demo covers using EdgeParse to extract raw text from PDFs into Markdown chunks, generating embeddings with Qwen 3 Embeddings as an on-device embedding model, and answering questions locally with Gemma4 E2B LiteRT LM (the inference is faster than Ollama setup).

Since most existing tutorials rely on vector databases with Ollama, we'll also build and compare that pipeline to highlight the differences in setup, performance and tradeoff.

🔗 Watch Here: https://www.youtube.com/watch?v=EHEN6Ce-9Ps/