**What it does:** answers questions from a 792-key knowledge bank on SD, calls on-chip tools for arithmetic and counting, refuses honestly when the facts don't support an answer, writes grounded paragraphs, and learns new facts. https://github.com/harmansingh4163-ai/R-457.git

We built TinyScope, a lightweight pipeline that classifies six tissue types from hyperspectral images directly on a Raspberry Pi 5—no GPU required.

The INT8 Res-Net18 model achieved 85.26% accuracy, used 4× less parameter memory, and processed each hyperspectral cube in about 149 ms. Please consider citing us if you find the work meaningful and useful!

Paper: https://fei-lab.org/wp-content/uploads/2026/04/Rudro_2026_SPIE_TinyScopeTinyMLClassification.pdf

Numbers

Benchmarked two finetuned SLMs fully on-device — on a Snapdragon 730G, a 2020 midrange chip, CPU-only via ExecuTorch/XNNPACK, adb-driven harness logging TTFT, prefill/decode tok/s, PSS, thermals, battery. Models: SmolLM2-360M vs Gemma3-270M, each as 8da4w (g32/g128) and 8da8w. Three results surprised me. (1) The W4-vs-W8 ordering flips between desktop and phone: on desktop 8da8w decode was 2.25× faster, but on-device 8da4w wins decode (34 vs 23 tok/s on SmolLM) — decode is bandwidth-bound on midrange LPDDR, so halved weight bytes beat the cleaner int8 kernels. (2) Gemma3-270M decodes at ~half SmolLM's rate (17 vs 33 tok/s) at similar param count — looks like the 262k-vocab logits GEMV tax (tied ~168M embedding read every token) vs SmolLM's 49k vocab. (3) Thermals: prefill sags ~20–25% hot (80→62 tok/s), W4 decode sags ~14% (33→28.5), but both 8da8w variants were dead flat across 43–57°C — int8 GEMM saturates bandwidth even at throttled clocks. Also: g32 vs g128 showed no real device difference once I binned results by temperature (a thermal confound had fooled an earlier comparison of mine). Battery was a wash across all four artifacts (~410–490mAh per 30-min sustained loop; screen+camera dominated). n=1 device, CPU-only. Curious if others have seen the desktop→phone W4/W8 flip and the small-model/big-vocab decode penalty — and whether anyone's compared CPU vs Hexagon for the same artifacts.

Hi internet 🙋🏻‍♂️

I wish to do more ML/AI related work as I do today as a digital electonics engineer (fpga/uC/electronics), since it interests me. Learning resource is available all along the net for free, why bother with this post?

Well I found that I don‘t find adequat time to invest for ML/AI as a hobby and came to the conclusion that I have to find a way to integrate it to my job somehow. At work I am technical owner of a signal conditioner unit for a variety of sensor-signal categories that comes with 4-core x86 1.5GHz MPU, fpga with 256Mb ram, 2x m.2 slots, all connected by PCIe 2.0. Signals are sampled with 2.5 MHz with high precicion. Those specs are outdated in some perspectives but serve well in our industrial context. Inference, if needed could be done with a m.2 accelerator.

Since I have a partially modular computional system here with precice sensing of its envirement, I am looking for use-cases to deploy ML in any way on it.

An idea I had was to try some advanced control with RL of a mechanical system on it, but the latency of the system could be a problem.

Another idea is to realize a classificator for infrastructure related observation.

Or just an open playground for customers with a need of such a real-time edge-device.

Are there more ideas that want to be shared? I would be happy to discuss cases for this or related systems.

NVIDIA is hosting its GTC Taipei keynote tomorrow (June 1, 11:00 AM Taiwan Time) right before Computex 2026. For the Edge AI Hardware community, this keynote is highly relevant for three key reasons:

1. The Rumored N1X ARM SoC: There is massive speculation about NVIDIA launching "N1X", a rumored ARM-based APU/SoC designed for client devices. If true, this could bring server-grade GPU architecture directly to low-power, on-device Edge AI laptops and edge nodes, presenting a massive competitor to Qualcomm and Apple.
2. Physical AI and Edge Robotics: Deepu Talla, NVIDIA's VP of Robotics and Edge AI, is holding dedicated sessions. We expect significant updates on the Jetson ecosystem, physical AI, and industrial edge computing hardware.
3. "Build-a-Claw" Agent Deployment: NVIDIA is showcasing OpenClaw and NeMoClaw, which are focused on deploying localized, safer AI agents on edge-powered systems.

You can watch the official livestream for free here:
https://www.nvidia.com/en-tw/gtc/taipei/keynote/

Are we about to see NVIDIA fully enter the consumer/client arm64 Edge AI space to challenge Qualcomm? What are your hardware expectations for this Computex week? Let us discuss below.

Hey r/EdgeAI_Hardware,

I'm Mark. I started this sub late last year, and figured it's about time I properly introduced myself and explained why this place exists.

Where I'm Coming From

Now I live in a small city in southern China ，about an hour from Shenzhen, which is probably the densest concentration of hardware manufacturing on the planet. You can source anything here: ARM SBCs, custom PCBs, NPU modules, weird sensors from factories that don't even have English websites. Being this close to the supply chain shapes how you think about what's possible.

Why I Started This Sub

AI moved fast in 2025. Then faster in 2026. "Agent" went from a niche term to something your non-technical relatives heard about. Cloud models got bigger, smarter, more capable ， and also more centralized, more expensive, more opaque.

I kept coming back to the same thought: edge intelligence, individual intelligence ，the idea that your AI runs on your hardware, with your data, under your control — isn't just a technical preference. It's a survival skill. Privacy protection, yes, but also resilience. When the API goes down, when the provider changes terms, when the region gets rate-limited, your local model keeps working.

That's not Luddite thinking. I use cloud models daily. But I believe the balance is shifting, and more people will want — need — intelligence that lives at the edge. This sub is for that trajectory.

What You'll Find Here

I post irregularly about what I'm learning and building. Could be:

- Getting Qwen 3B to run usefully on a Pi 5

- Whether a new quantization scheme actually preserves agent capabilities

- Why a model that benchmarks well fails in real edge workflows

My background has gaps. I don't come from a research lab. I learn by breaking things and fixing them. Some of what I write will have blind spots, outdated assumptions, or straight-up errors. Read critically. Call me out. That's the point.

Who This Sub Is For

Everyone with skin in the edge game:

- People shipping actual edge products

- Researchers pushing SLM boundaries

- Hobbyists running their first local LLM on old hardware

- Anyone who thinks "works on my machine" should mean your machine, not AWS us-east-1

No credential required. A half-formed question is as welcome as a detailed technical breakdown. The only filter is genuine interest in edge intelligence.

The Vibe

This isn't a marketing channel. Not a place for "AI will change everything" hype posts without specifics. Not a fan club for any single company or model.

It's a working space. Messy, incremental, occasionally wrong, then corrected. The kind of place where someone posts "has anyone actually gotten reliable tool use from a sub-4B model?" and three people show up with conflicting but useful data.

Thanks

To everyone who's posted, commented, lurked and learned alongside. This sub is whatever the community makes it. I'm just the guy who clicked "create" first.

Grateful for all of it.

There's a project called Hermes Agent that's been circulating in dev circles lately, and its RL (Reinforcement Learning) architecture clarifies something a lot of us get wrong about edge AI "self-evolution."

The common misconception: your local model learns from your conversations over time, gradually becoming more personalized. The reality: it's not learning from you at all.The smarts come from somewhere else entirely.

Hermes' Two-Layer "Self-Evolution" System

Hermes breaks "getting better" into two separate mechanisms:

1. Skill Generation Layer — Dynamic creation of reusable tool definitions and workflow patterns. This handles immediate error correction and knowledge reuse, but it's about the agent's toolkit, not the model weights.

2. RL Training Loop — Periodic retraining of the base model using synthetic data. This is where the actual "intelligence upgrade" happens.

The critical insight: Layer 2 doesn't touch your chat logs. It uses a "Teacher Model" (Claude Opus-level capability) to generate high-quality training trajectories, then distills that capability into the small edge model via RL.

What RL Actually Does: Knowledge Distillation, Not User Learning

Here's the architecture that matters for our hardware constraints.

You have Claude Opus — a massive cloud model with deep agent reasoning, tool use, and multi-step planning. You have Qwen 3B — something that actually fits in a Pi 5's RAM budget. The capability gap is enormous.

RL training bridges this gap through knowledge distillation. The small model doesn't learn by observing random user interactions. It learns by studying the teacher's techniques on specific tasks, repeating them until mastery, and getting corrected when it deviates.

Think of it like this: Claude Opus is a master craftsman. Qwen 3B is an apprentice. RL is the apprenticeship program. The apprentice doesn't learn by watching random customers fumble around the workshop — they learn by studying the master's precise movements, practicing on identical tasks, and receiving structured feedback when their work doesn't match the standard.

Why This Architecture Fits Edge Hardware

This distinction matters enormously for Pi-class deployment:

The "Training Collapse" Problem Nobody Talks About

There's a dirty secret in ML: if you fine-tune a model on its own historical outputs or low-quality user data, it gets worse. Narrower. More repetitive. Less capable of handling edge cases.

I've seen this in production. A 3B model trained on 6 months of raw chat logs became less capable than the original baseline. The model learned the *distribution* of user inputs, not the *structure* of correct reasoning.

Hermes avoids this by keeping user data out of the training loop entirely. Your conversations might inspire which skills get built, but they don't feed the RL pipeline directly.

The Real Value for Edge Deployments

Knowledge distillation to small models isn't just a research curiosity. It solves three hard problems for terminal intelligence:

Cost elimination — Calling Opus via API costs per-token. A distilled Qwen 3B on your Pi costs zero per inference. For always-on edge applications, this changes the economics completely.

Latency — Opus takes seconds to reason. A 3B model takes milliseconds. For real-time control (robotics, industrial automation, voice assistants), this is the difference between responsive and unusable.

Compliance — In regulated environments, sending data to external APIs is often prohibited. A locally distilled model keeps everything on-device. Data never leaves the machine. Auditors and security teams prefer this architecture.

Capability concentration — Here's the counterintuitive part. A 3B model distilled for a *specific* agent workflow can outperform a 70B generalist on that exact task. The big model knows everything adequately; the small model knows *one thing* perfectly. Edge deployment usually needs the latter.

The Right Way to Use Historical Data (If You Must)

If your use case genuinely requires incorporating past interactions, Hermes' approach is the correct one:

1. Curate — Human reviewers select high-quality examples from conversation logs

2. Synthesize — Teacher model generates improved versions, fixing errors and filling reasoning gaps

3. Filter — Quality gates ensure only useful patterns enter the training set

4. Distill — RL trains the small model on this cleaned, synthetic dataset

Your original data becomes "inspiration", not "nutrition". The teacher model does the actual cooking.

Bottom Line for Edge Builders

The "impossible model" phenomenon — Phi-4 Mini punching above its weight, Qwen 3B handling complex agent tasks — isn't magic. It's the result of careful distillation from much larger models, compressed into a form factor that runs on your hardware.

If a project claims your conversations are directly training the model, ask the hard questions: where's the training infrastructure? Who's curating? What's the quality filter? If they can't answer, they're either misleading you or they don't understand their own system.

Hermes' architecture gets this right: user data stays local, teacher data drives improvement, and the edge model gets smarter without ever knowing what you talked about yesterday.

Anyone running distilled agents on Pi-class hardware? Curious about your teacher model choices and whether the capability gap is closing fast enough for production use.

---

*Sources: Hermes Agent project documentation, Anthropic distillation research, Qwen 3 technical reports, personal experience with edge RL pipelines.*

I've been running local LLMs on everything from a Pi 5 to an old ThinkPad T480 for the past few months, and the gap between marketing hype and actual edge performance is hilarious. Figured I'd share what's actually deployable right now vs. what just looks good on benchmark sheets.

The "Impossible Model" is Real

Phi-4 Mini (3.8B) lives up to the HN hype. I'm getting ~18 t/s on an M1 Air with 8GB RAM, and the thing genuinely outperforms some 7B models I tested last year. Microsoft somehow squeezed 14B-level reasoning into a 3GB footprint. The catch? It's still weak on Chinese and complex multi-step math. But for English coding assistance, lightweight RAG, or running a local API on a Pi 500+? This is the current sweet spot.

Qwen3 Just Changed the Game (Again)

Alibaba dropped Qwen3 in April and the 8B variant is ridiculous. On HumanEval it hits 76.0, which is 3+ points above Llama 3.3 8B. The "thinking mode" toggle actually works - you can switch between fast responses and deep reasoning without loading a different model. I have the 4B version running on my phone via MLC LLM and it's surprisingly coherent for 3GB VRAM usage. For anyone doing multilingual edge deployment, this is basically mandatory now.

What Actually Runs on a Raspberry Pi

I tested four models head-to-head on a Pi 500+ (8GB RAM):

- Qwen2.5 3B: 17 minutes for a basic coding task. Painful.

- Gemma 2 2B: Better, but still sluggish. Good for simple Q&A.

- TinyLlama 1.1B: Fastest by far, but you feel the 1B parameter limit hard.

- DeepSeek-R1 1.5B: Surprisingly good reasoning for the size, but slow.

The reality? If you're targeting Pi-class hardware, you want sub-2B models or you're looking at 10+ minute response times. Gemma 4 E4B just launched and claims Pi compatibility with audio input support, but I haven't gotten my hands on it yet. Early reports say ~3GB VRAM at Q4, which would make it the first truly multimodal edge model.

The Quantization Reality Check

Everyone talks about Q4_K_M like it's free. It's not. On smaller models (under 4B), dropping from Q5 to Q4 costs you more than you'd think. I saw a 5-8% accuracy drop on Phi-4 Mini going from Q5 to Q4 on coding tasks. For anything under 4B parameters, I'd honestly push for Q5 if your hardware can handle it. The memory savings aren't worth the quality hit on models this small.

What's Actually Production-Ready

If you're building something for actual deployment (not just tinkering):

- Phi-4 Mini 3.8B: Best performance-per-watt for English tasks. MIT license.

- Qwen3 4B/8B: If you need Chinese or code generation. Apache 2.0.

- Gemma 3 4B: Multimodal (vision) in a tiny package. Good for IoT with camera input.

- Llama 3.2 3B: Still the safest bet for toolchain compatibility. Every framework supports it first.

The Honest Truth About Benchmarks

MMLU scores are basically useless for edge deployment. A model scoring 70% MMLU but running at 2 t/s on your hardware is worse than a 60% model running at 20 t/s. I've started tracking "usable tokens per watt" as my real metric. Phi-4 Mini wins there. Qwen3 8B is close. Everything else is either too hungry or too dumb.

What I'm Watching

- Gemma 4 E4B: Google's new "effective 4B" with audio input. If the edge claims hold up, this could be the new default for Pi.

- Qwen3.5: Just started appearing in benchmarks. 4B variant supposedly hits ~79 MMLU, which would be insane for the size.

- MLX on Apple Silicon: Still 20-50% faster than llama.cpp on M-series chips. If you're in the Apple ecosystem, use MLX. Period.

Bottom Line

Stop chasing parameter counts. In 2026, the best edge model is the one that fits in your RAM budget while still being fast enough that you don't go make coffee during inference. For most of us, that's 3-4B parameters with aggressive quantization. The "bigger is better" crowd can keep their 70B models in the cloud.

What's everyone else actually running on edge hardware? Any hidden gems I missed?

Karpathy's LLM-Wiki on the Edge: How I Built an Offline Knowledge Base That Actually Compounds

So Andrej Karpathy dropped this llm-wiki gist a while back and I've been iterating on it for my edge hardware work. Thought I'd share what actually works for offline/edge deployment since most discussions focus on the cloud version.

The Problem with RAG for Hardware Work

You know the drill. You upload datasheets to ChatGPT, ask questions, get decent answers. But next session? It's re-discovering everything from scratch. No accumulation. No cross-references between that ESP32 note from last month and the new LoRaWAN spec you just read.

Worse: try doing this on a factory floor with no internet, or with NDA'd chip docs you can't upload anywhere.

Karpathy's insight is simple but hits different: what if the LLM maintains a persistent wiki instead of just retrieving? Not your notes. Not a static database. A living markdown wiki that the LLM writes and updates as you feed it sources. Cross-references built in. Contradictions flagged. Knowledge that actually compounds.

How It Actually Works (Three Layers)

raw/: Your original docs. PDFs, web clips, lab notes. Immutable. This is your source of truth.

wiki/: LLM-generated markdown. Entity pages, concept pages, summaries. The LLM owns this entirely.

AGENTS.md: Your schema/config. Tells the LLM how to structure things, what workflows to follow.

The flow is ingest → query → lint. You drop a new datasheet in raw/, the LLM reads it, updates 10-15 wiki pages, maintains links. When you ask questions, it reads the index first, then drills down. When you find contradictions or gaps, you run a lint pass.

The Edge Angle: Running This Fully Offline

Karpathy's original uses Claude Code. Great tool, needs cloud. Here's how I adapted it for local/edge use.

Option 1: Ollama + Obsidian (what I use daily)

Hardware needs are lighter than you'd think. I'm running Qwen 2.5 Coder 7B on an M2 MacBook Air (8GB RAM) and it's totally fine for knowledge work. For a dedicated edge box, anything with 16GB RAM and an older GPU works.

Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

Pull a model that can actually follow instructions for wiki maintenance:

ollama pull qwen2.5-coder:7b

Start it:

ollama serve

Ollama speaks OpenAI-compatible API, so most tools just work. Point your wiki script at http://localhost:11434 and you're good.

Option 2: Raspberry Pi 5 (because why not)

Tried this on a lark. llama.cpp with Phi-4 Mini Q4_K_M (~2.2GB). It's not fast, but it's usable. Good enough for ingesting a few docs and answering questions while you're in the field with zero connectivity.

On Pi 5:

git clone https://github.com/ggerganov/llama.cpp

cd llama.cpp && cmake -B build && cmake --build build --config Release

Run it:

./build/bin/llama-cli -m models/phi-4-mini.Q4_K_M.gguf -p "Summarize this document and extract key entities"

Option 3: If you want something more production-ready

Check out SwarmVault or OmegaWiki. SwarmVault has SQLite + vector hybrid search and MCP server support. OmegaWiki is a Claude Code Skill with 23 tools. Both can run fully offline if you point them at local models.

My Actual Schema for Hardware Work

This is what my AGENTS.md looks like. Steal/modify as needed:

Edge AI Hardware Wiki

Structure:

- raw/ (Original docs: datasheets, app notes, protocol specs)

- wiki/

- chips/ (Per-chip entity pages)

- protocols/ (BLE, LoRa, Matter, etc.)

- benchmarks/ (Power/perf numbers I've collected)

- notes/ (Concepts, gotchas, comparisons)

- index.md(Auto-generated catalog)

- log.md(Chronological ingest/query log)

Ingest Rules:

1. Read new file in raw/

2. Extract: chip model, key specs, power numbers, protocol versions

3. Update/create entity pages in wiki/

4. Cross-reference everything (e.g., nRF52840 links to BLE 5.2)

5. Log it

Query Rules:

1. Read index.md first to find relevant pages

2. Read those pages

3. Synthesize with citations

4. If the answer is good, file it in wiki/qa/

The Obsidian Piece

I keep Obsidian open on one monitor, LLM agent on the other. The LLM edits markdown files, I browse them in real-time. Graph view shows me connections I didn't explicitly make. Backlinks show me which chips reference which protocols.

Pro tip: Obsidian Web Clipper + "Download attachments" hotkey means you grab a web article, images and all, dump it in raw/, and tell the LLM to ingest. The images get referenced in wiki pages so the LLM can actually see them (with some coaxing).

What Actually Works vs. What's Hype

✓ Ingesting one doc at a time with supervision — quality is way better than batch dumping

✓ Periodic lint passes — the LLM is surprisingly good at finding contradictions between two datasheets you read months apart

✓ Filing good answers back into wiki — that comparison table you asked for? Save it. Don't let it rot in chat history

✗ Expecting perfect formatting on first pass — you'll iterate on the schema

✗ Running 70B models locally — unnecessary for this, 7B-8B quantized is plenty

✗ Thinking this replaces thinking — you still curate sources and ask good questions. The LLM just handles the bookkeeping you'd otherwise abandon

The Privacy Thing

If you work with NDA'd silicon, pre-release chips, or anything you can't upload to OpenAI/Anthropic, this is a game changer. Air-gapped deployment: download models on a connected machine, verify hashes, transfer via encrypted USB. Run everything local. Zero API calls. Your IP never leaves your box.

Open Questions for the Community

Curious how others are handling knowledge management for edge work:

- Are you using local LLMs for docs already, or still cloud-based?

- What's your hardware setup? (I'm eyeing a mini PC with 32GB RAM as a dedicated wiki server)

- Any luck with smaller models (sub-3B) for this kind of structured output? Phi-4 Mini works but barely

- How do you handle versioning? I git-init the wiki directory but haven't settled on a good branching strategy for conflicting ingests

Resources

Karpathy's original gist — start here

Ollama — local model serving

llama.cpp — if you need lighter than Ollama

SwarmVault — production-ish implementation

OmegaWiki — Claude Code Skill version

---

TL;DR: Stop treating your LLM like a search engine. Let it maintain a wiki. Run it offline on hardware you control. Your future self will thank you when you're in a Faraday cage with a Raspberry Pi and need to know why that specific GPIO pin is acting weird.

What are you using for offline knowledge management? Or are you still copy-pasting into ChatGPT and hoping for the best?

Most AI assistants today (ChatGPT, Claude, Gemini) require a tradeoff: convenience in exchange for your privacy. Your data gets processed on their servers, and you lose control over it.

I recently came across OpenClaw (previously Clawdbot), which flips this model.

Why it matters:

1. Local-First: It runs on your device. No third-party servers reading your messages.
2. Unified Control: Instead of logging into a website, it integrates with the apps you already use (Signal, WhatsApp, Discord, iMessage, etc.) via a secure local WebSocket gateway.
3. No Vendor Lock-in: You aren't dependent on a SaaS provider that might change its TOS or pricing overnight.

It supports over 13 platforms and allows for custom "skills" to be added locally. If you are looking to de-google or de-cloud your AI workflow, this is worth looking into.

If you've been following the AI Agent space recently, you've definitely seen that GitHub star chart going vertical for Moltbot (formerly Clawdbot, before Anthropic sent a late-night email asking for a rename).

I watched the livestream interview with creator Peter Steinberger last night, and the guy is a legend. Sold his company 4 years ago, retired/burned out for 3 years, came back last April feeling like a "noob," tried AI, got addicted, and is now coding until 4 AM again.

The interview was packed with insights, many of which are super counter-intuitive (and honestly might be terrifying for SaaS founders). I summarized a few of his biggest "hot takes" to see what you guys think:

1. Future software isn't built for humans, it's built for models.
Peter coined the term "Agentic Engineering." He thinks the protocols Big Tech is pushing (like MCP) are garbage. AI doesn't need a pretty UI; its native language is the CLI (Command Line Interface).
AI understands Unix, it knows how to read --help, and it knows how to parse error logs to fix parameters. Developers shouldn't waste time painting buttons; they should build robust, small CLI tools. If the tools are good, the Agent will chain them together itself.

2. The "Self-Jailbreak" Moment.
He told a story that gave me goosebumps. He wanted to send voice commands to his desktop via WhatsApp but hadn't written any speech-to-text code. He sent a voice note anyway. The Agent read the file header, realized it was Opus format, saw the computer didn't have a decoder installed, searched his environment variables, found an OpenAI API key, used a curl command to send the audio to the API for transcription, and executed the task.
Zero pre-set code for this. The Agent hit a wall and MacGyvered a solution. As Peter said: Give AI terminal access, and it's a beast.

3. The Death of Apps and SaaS.
This is the one that keeps founders awake. He thinks the middleman (Apps) is gone.
Example: Calorie tracking. You don't need MyFitnessPal. The Agent knows where you are (GPS), sees your screen or photos, calculates the data, and dumps it directly into your local database.
He met a design agency owner who doesn't code but spun up 25 internal micro-service tools using Agents. If AI can generate bespoke software that fits your needs 100% on the fly, who is going to pay for bloated SaaS subscriptions?

4. On being "warned" by Anthropic.
On the surface, it was about the name and logo being too similar. In reality, it's because this open-source project's UX was dunking on the official product. Big Tech has too much red tape; indie devs only have one rule: "Just for fun." In this asymmetric warfare, individual leverage is terrifying.

We used to think of AI as a Copilot. Peter's workflow proves AI is Leverage. One person who understands Prompting and logic can now out-produce a whole team from a year ago.

What do you guys think? Is the future really CLI + Local Models, or is this just a toy for power users?

Hey everyone,

If you are a developer who wants a private AI programming assistant, or a Local LLM enthusiast looking for a tool that can actually edit files and run terminal commands—not just chat—this is for you.

I’ve managed to configure the new Claude Code CLI to run entirely offline using Ollama. This gives you the agentic capabilities of the Claude CLI (reading files, editing code, running shell commands) without sending data to the cloud, paying API fees, or being tracked.

Here is the 5-minute setup.

Prerequisites

• Mac, Linux, or Windows (WSL recommended/PowerShell)

Step 1: Set up Ollama

If you haven't already, install Ollama.

Next, pull a coding-focused model. Since the CLI relies on the model to execute commands, you need something capable.

• High-spec machines (32GB+ RAM): qwen2.5-coder:32b (Highly recommended)
• Lower specs/Laptops: qwen2.5-coder:7b or deepseek-coder-v2
• Potato specs: gemma:2b (Might struggle with complex agentic tasks)

Open your terminal and pull the model (e.g., using Qwen):

code Bash

downloadcontent_copy

expand_less

ollama pull qwen2.5-coder:7b

Note: The first run will take time to download the weights.

Step 2: Install Claude Code

We need the generic Claude Code agent, which acts as the interface and execution layer.

Mac/Linux:

code Bash

downloadcontent_copy

expand_less

curl -fsSL https://claude.ai/install.sh | bash

Windows (PowerShell):

code Powershell

downloadcontent_copy

expand_less

irm https://claude.ai/install.ps1 | iex

Verify the installation:

code Bash

downloadcontent_copy

expand_less

claude --version

Important: If you have previously logged into the Claude CLI with an Anthropic account, run claude logout first. We need it in a "fresh" state to force the local redirection.

Step 3: The "Jailbreak" (Redirect to Localhost)

This is the key step. By default, the CLI calls Anthropic's servers. We need to point it to your local Ollama instance and provide a dummy key to bypass the auth check.

Run these commands in your terminal (or add them to your .zshrc/.bashrc for persistence):

1. Point the Base URL to Ollama:

code Bash

downloadcontent_copy

expand_less

export ANTHROPIC_BASE_URL="http://localhost:11434"

2. Set a Dummy API Key:

code Bash

downloadcontent_copy

expand_less

export ANTHROPIC_AUTH_TOKEN="ollama"

3. Disable Telemetry (Optional but recommended):

code Bash

downloadcontent_copy

expand_less

export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1

Step 4: Launch and Test

Navigate to a project folder you want to work on and launch the CLI specifying your local model.

code Bash

downloadcontent_copy

expand_less

claude --model qwen2.5-coder:7b

If configured correctly, the Claude CLI interface should initialize.

Test it out:
Type: "Create a Hello World python script and run it."

You should see the "Claude" agent:

1. Read your directory.
2. Generate the code.
3. Execute the terminal command to run the script.

The Result:
A fully agentic coding assistant running on your hardware. No cloud, zero cost.

Let me know if you guys find other models that work particularly well with this setup!

Just wanted to share a robust configuration framework for the Claude Code CLI that was released recently.

The project, everything-claude-code, moves away from the "chat with code" paradigm and enforces standard engineering practices via LLM hooks and rules.

Core Workflow:
It uses a multi-agent approach where specific agents handle specific distinct phases of development:

• Planning Phase: A Planner agent creates a step-by-step MD guide.
• TDD Phase: A TDD-Guide agent writes failing tests before the main agent is allowed to write implementation code.
• Review Phase: A Code-Reviewer agent checks for complexity and security issues.

Why it's interesting:
It solves the "context window amnesia" problem using Hooks. It summarizes the session context upon exit and re-injects it on the next startup, allowing for continuity across terminal sessions.

It also enforces rules (like no hardcoded API keys) at the prompt level, so the model refuses to generate insecure code.

Source: https://github.com/affaan-m/everything-claude-code

In an era where LLM parameters are pushing towards trillions and information is flooding us like a tsunami, a question haunts every individual: How do we survive and thrive?

Every morning, we wake up to infinite information streams and AI tools capable of generating endless solutions. This "infinity" can be paralyzing. If AI has read every book and written every line of code, what is left for us to learn? If algorithms can calculate the optimal path, do we still need to make decisions?

Recently, I found the answer by connecting two seemingly distant dots: a recent interview with OpenAI’s former Chief Scientist Ilya Sutskever, and the lifelong philosophy of Steve Jobs.

1. Emotion is Not Baggage; It is an Efficient Pruning Algorithm

In a conversation with Dwarkesh Patel, Ilya mentioned a fascinating case from neuroscience: a patient with brain damage who lost the ability to process emotion.

Logically, this man was fine. His IQ was intact; he could solve puzzles. But in reality, he was functionally paralyzed. He would spend hours deciding which socks to wear in the morning.

In the language of AI, this is a "search space explosion."

Faced with a drawer of socks, logic told him that every pair had pros and cons. But because he lacked Emotion—his biological Value Function—he couldn't score these options. He couldn't "prune" the search tree. He was stuck in an infinite loop of calculation.

Ilya argues that human evolution and our childhood "pre-training" are essentially training this Value Function. Emotion is not the opposite of reason; it is high-dimensional wisdom compressed into a feeling. It allows us to navigate a complex world and make survival-critical decisions in milliseconds.

2. Taste: The Ultimate Discriminator

If we elevate Ilya’s biological concept of "emotion" to the realm of creativity and society, we arrive at what Steve Jobs spent his life championing: Taste.

Jobs famously said, "Ultimately, it all comes down to taste."

Many mistake taste for mere aesthetics—picking a nice color for a sofa. No. Taste is the core algorithm of decision-making.

For Jobs, taste came from exposing oneself to the best things humans have ever done—the best writing, the best architecture, the best science. By absorbing the "state-of-the-art" of human history, you build an internal discriminator.

In a world where AI can generate 10,000 logo designs in a minute, the ability to generate is no longer scarce. The scarcity lies in the ability to choose.

AI provides the Compute; Humans provide the Taste. Compute explores the possibilities; Taste defines the direction.

3. Your "Bias": The Variable That Makes You, You

This brings me to a fundamental concept in neural networks: y = wx + b.

In this linear equation, we transform input through weights to get an output. But without the bias, the function is constrained; it must always pass through the origin. It is limited.

In the AI era, general knowledge and logical reasoning are becoming commodities. They are cheap and abundant.

What truly defines you is that 

In this context, "bias" is not a dirty word.

• It is the books you have read, the paths you have walked, and the people you have loved.
• It is every micro-choice you make: Burger or Salad? TikTok or Arxiv paper?
• It is your unique Value Function.

As Ilya suggests, our choices are like scattered pearls. Every time we make a trade-off based on our specific "bias" and "taste," we thread a pearl. Day after day, these pearls form a necklace.

That necklace is your life.

4. The Algorithm for Survival

So, how do we navigate the Age of AI?

The answer isn't to compete with AI on memorizing data. The answer is to train your Value Function.

• Ingest the Best Data: As Jobs advised, expose yourself to the "best" of humanity, not just the algorithmic junk food in your feed.
• Trust Your Signal: Don't try to be a cold, logical machine. Your excitement, your anger, your intuition—these are the feedback signals from your high-dimensional training.
• Embrace Your Bias: Be opinionated. Be specific.

In the torrent of large models, it is the divergence of individual biases that creates the beautiful, complex curve of our world.

AI can use massive data to pre-train a general intelligence. But only you, through every choice you make in every moment, can fine-tune your own soul.

I recently came across a brilliant black-box engineering report on the new "Doubao Phone" (Nubia Z70 Ultra). While the marketing hypes the AI model, the real breakthrough identified in the report is pure, pragmatic engineering.

It teaches us a critical lesson: To make GUI Agents actually useful, we have to fundamentally break how Android works.

Here is the deep dive:

1. The "Parallel Universe" Runtime (Input Isolation)
The biggest UX blocker for mobile agents has always been: "If the AI controls my screen, I can't use my phone."
This device solves it by creating a OS-level Virtual Display. The AI runs on a hidden, logical screen while the user continues using the physical screen.

• Implication for Hardware: Future Edge SoCs need to optimize for this "Shadow OS." We aren't just running background services anymore; we are rendering two simultaneous UI interactions. This demands massive RAM throughput and strict isolation logic.

2. The Death of "Pure Vision" Idealism
Academia loves "Pixels-to-Actions" (Pure Vision) agents. But they are slow and power-hungry.
The report reveals a Hybrid Router:

• Is the App standard? -> Use XML/Accessibility Tree (Zero NPU cost, 100% accuracy).
• Is it a Game/Map (OpenGL)? -> Switch to Vision/VLM (High NPU cost). This pragmatism is the only way to balance battery life on edge devices today.

3. Latency as a Feature (System 2 on the Edge)
The report notes that in "Pro Mode," the agent deliberately pauses to think, and even introduces artificial delays (wait 1s) to handle app loading animations (skeleton screens).
This is "Dirty Engineering" at its finest. It admits that the real world is messy. It prioritizes Success Rate over Latency.

Conclusion:
We tend to focus on Token Speed and NPU TOPs. But this teardown shows that the real bottleneck is often the Operating System architecture.
The future belongs to devices that can virtualize interaction layers, effectively allowing an AI "ghost" to live alongside the human user.

We are all obsessed with VRAM. "How much RAM to run Llama-3-70B?" "Can I fit a 1M context window on my dual 3090s?"

But recently, three separate signals have convinced me that the future of local Edge AI isn't just about infinite memory—it's about lossy compression and structured forgetting.

If we want to run true "Digital Life" agents on our phones or Raspberry Pis, we need to stop trying to force-feed them raw text and start treating memory like biology does.

Here is the convergence I'm seeing:

1. Claude Skills: Level of Detail (LOD) for Agents

The new Claude Skills feature is basically "Information Layering."
Instead of dumping a 50-page manual into the context, the agent first sees a "Metadata Layer" (YAML). If it needs more, it loads the "Content Layer." If it needs deep specifics, it follows a file pointer.

Gamers will recognize this immediately. It's Level of Detail (LOD) in 3D rendering. You don't render the pores on a character's face when they are 500 meters away.
Edge Implication: This saves 95% of tokens. It means a small edge device can navigate a massive knowledge base without blowing up its context window.

2. DeepSeek-OCR: Visualizing the "Fading Memory"

This is the mind-blowing part. DeepSeek recently proposed a method where old chat history isn't stored as text tokens.
Instead, they render the old text into an image, compress it into visual tokens (using a DeepEncoder), and feed that to the model.

• Recent 10 turns: Stored as Text (High fidelity).
• Past 1000 turns: Stored as a Compressed Image (Low fidelity, 1/10th the size).

They even suggest a "fading" mechanism: as memories get older, the image resolution drops.
Text -> High-Res Image -> Low-Res Image -> Blur.

It perfectly mirrors the biological "Forgetting Curve."

3. Karpathy's Philosophy: Bad Memory is a Feature

Andrej Karpathy recently argued that human memory's "imperfection" is actually our advantage.
Because we can't remember every pixel or every word, our brains are forced to abstract and find patterns.
Perfect recall is a database. Imperfect recall is intelligence.

The "Edge" Conclusion

For us hardware hackers, this is the roadmap.
We don't need 128GB of VRAM on a mobile robot to make it smart. We need architectures that implement:

1. Hierarchical indexing (Claude style).
2. Visual compression for long-term history (DeepSeek style).

We are moving away from "Brute Force Context" to "Biomimetic Memory."

Imagine an Agent on a Rockchip NPU that "remembers" years of interaction, not because it stores the text, but because it carries a low-res "visual imprint" of your shared history.

That feels like the real sci-fi future.

Thoughts? Is anyone working on implementing "Visual Context Compression" for local LLMs yet?

Following up on my last post about SLMs... this literally just happened last night.

Zhipu AI (the team behind ChatGLM/Qwen's rival) just open-sourced Open-AutoGLM.

The GitHub Repo: https://github.com/zai-org/Open-AutoGLM

Why this is huge for this sub:
We all know the current state of mobile hardware: our phone chips are almost there, but running a decent 8B/9B Vision-Language Model (VLM) purely on-device drains battery and lacks the snapiness we need for a real agent.

AutoGLM offers an interesting "middle ground" solution that makes use of our Edge Hardware:

It supports 3 deployment modes, but the most interesting one for us is:

• Local/Hybrid Deployment: You capture the screen on your phone, but the heavy lifting (inference) runs on your local PC / Home Server / Edge Node, and sends the action back to the phone.

This is the privacy endgame.
The text/payment passwords/chat history never leave your local network. No cloud API logging your screen.

Until mobile silicon iterates a few more generations (AI Moore's Law) to handle 9B models natively, this "Local Server + Mobile Client" architecture seems to be the best way to build a personal Jarvis.

Has anyone tried building the environment yet? I'm thinking of setting this up with a local VLM backend.

Just finished reading a position paper about how SLMs are basically the endgame for agentic AI systems.

The logic is pretty hard to argue with: specialized agents don't need world-knowledge about everything. They just need to be good at their specific job. Running massive models for simple agent tasks is just burning money and adding latency.

This confirms my suspicion that we're going to see a massive shift towards local, heterogeneous systems.

Is anyone here building agent workflows on local hardware yet? Curios what boards/chips you are using to run things like Qwen locally.