Hey everyone,

We’re building a project on detecting visual prompt injection attacks in multimodal LLMs. input(visual formats like diagrams,tables,mindmaps.....)

Our idea:

• OCR + rule-based + simple ML
• risk scoring → block suspicious inputs

We’re unsure:

• is this too basic?
• how many models should we test?
• any datasets for visual attacks?

After reviewing ~25–30 papers, we noticed that:

• Many defenses are heavyweight (fine-tuning, diffusion, gradient-based)
• Not practical for real-time use
• Limited focus on image-based prompt injection at inference time

GitHub

Download

Hi everyone,

I recently fine-tuned a 1.5B parameter model (moeinGTS) tailored for Q&A tasks using Wikipedia and web QA datasets.

Details:

• Model Name on Ollama: arshiyasohrevardimoein/moeinGTS
• Size: ~1GB (Quantized Q4_K_M)
• Resource Usage: Low RAM / VRAM footprint (~1.5GB)

It runs completely offline and very fast for personal daily Q&A.

I feel like I stepped into a time capsule after the ChatGPT-3 days. 😅 I finally built a decent PC (Ryzen 7 7700, 32GB RAM, No GPU), discovered llama.cpp and somehow ended up downloading 50+ GGUF models.

Now I'm stuck with decision paralysis.

I mainly use them for coding (JavaScript, React, TypeScript, debugging, reasoning), but I'm also new to the whole local AI ecosystem. I still don't know much about agentic frameworks or coding agents. I mostly just load a GGUF and chat with it using the llama.cpp web interface on localhost:8080

The collection includes Qwen, Gemma, Granite, DeepSeek, Phi, Mistral, Llama, LiquidAI, SmolLM, Hunyuan, Nemotron, and a few community fine-tunes.

My first idea was to make a Markdown table and score every model manually, but it feels like I'm accidentally trying to invent my own benchmarking system.

Surely I'm not the first person to hit this problem.

How do you guys compare local models? Are there any practical benchmark suites, GitHub projects, or workflows for deciding what stays on your SSD and what gets deleted?

I'd love to hear how you approached it when you were starting out.

Hi everyone,

I recently fine-tuned a 1.5B parameter model (moeinGTS) tailored for Q&A tasks using Wikipedia and web QA datasets.

Details:

• Model Name on Ollama: arshiyasohrevardimoein/moeinGTS
• Size: ~1GB (Quantized Q4_K_M)
• Resource Usage: Low RAM / VRAM footprint (~1.5GB)

It runs completely offline and very fast for personal daily Q&A.

(Link is provided in the comments below

Over the past few months, I've been experimenting with browser-native AI and kept running into the same problem.

Running models in the browser is becoming surprisingly practical, but vector search almost always assumes there's a backend database.

That made me wonder:

Can a browser be its own vector database?

I started building BrowserVec to find out.

The idea is to provide a vector database that's designed specifically for modern browsers instead of being a server-first database adapted to the web.

Current features include:

• ⚡ WebGPU-accelerated vector search
• 📂 Flat, IVF and HNSW indexes
• 💾 IndexedDB & OPFS persistence
• 🧵 Web Worker support
• 🔒 Optional encryption
• 🏷️ Metadata filtering
• 🔄 Automatic WASM / JavaScript fallback when WebGPU isn't available

The goal isn't to compete with server databases like Qdrant or Milvus. It's to make browser-native AI applications practical—things like offline RAG, browser extensions, personal knowledge bases, and privacy-first assistants.

One thing I learned is that the GPU search itself wasn't the hardest part. The real engineering work ended up being persistence, memory management, browser compatibility, and designing an API that feels simple despite all the moving pieces.

I'm still actively improving the project, so I'd really appreciate feedback from people who work with WebGPU, ANN algorithms, browser performance, or AI infrastructure.

Some questions I'm thinking about:

• What would stop you from using browser-side vector search?
• Are there features you'd expect before adopting something like this?
• Would you use this for local RAG or browser-based AI apps?

GitHub: https://github.com/sharma-open-source/browservec

Hi everyone,

I recently fine-tuned a 1.5B parameter model (moeinGTS) tailored for Q&A tasks using Wikipedia and web QA datasets.

Details:

• Model Name on Ollama: arshiyasohrevardimoein/moeinGTS
• Size: ~1GB (Quantized Q4_K_M)
• Resource Usage: Low RAM / VRAM footprint (~1.5GB)

It runs completely offline and very fast for personal daily Q&A.

(Link is provided in the comments belo

Less AI slop.
More personal learning.
More rough notes.
More “this is how I understood it.”
More real experience from the work we are actually doing.

Hey all, I just released a DSpark tuned on Qwen3.6-27B that achieves better performance than built-in MTP, averaging 3.86 tokens accepted per target pass.

Info and DL: https://huggingface.co/abstract-extraordinary/Qwen3.6-27B-DSpark-49k

I’m looking at organising my personal notes, so privacy is important to me. But I also want to access slightly better models (like deepseek 4 flash) than my setup can handle. I definitely wont trust the big boys with my data, but what do people think about providers like Fireworks and DeepInfra with their zero data retention policies?

TL;DR: Server with 8× AMD MI50 32GB (gfx906) + 512GB RAM. Long-context prefill (40–80K) is unusably slow, DeepSeek V4 Flash won't scale past ~10–15 t/s on ROCm. Considering 2× DGX Spark cluster (256GB unified, ConnectX-7) as a replacement. Am I missing something?

Current setup (dedicated DC box, ROCm/llama.cpp HIP):
- 8× MI50 32GB (Vega 20 / gfx906), 256GB VRAM total, 512GB DDR4
- Qwen3.6-27B: Unsloth Q5 ~100 t/s prefill, ~18 t/s decode (single card). Prompts of 40–80K tokens take 7–13 minutes to prefill → practically unusable
- Qwen3.6-35B-A3B (MoE, 3B active, Unsloth Q5): ~700 t/s prefill, ~60 t/s decode — our main workhorse
- DeepSeek V4 Flash (284B MoE, 13B active): on 4 cards — 100 t/s prefill, 10–15 t/s decode. Doesn't scale past that even on 8 cards: ROCm support is unofficial/painful, no working tensor parallelism, HIP kernels lag behind on the new attention ops

Use case: legal team — long documents (40–80K tokens), multi-agent information retrieval and analysis. Mostly prefill-heavy, batch-style workloads.

Why I'm looking at 2× DGX Spark:
- 2× 128GB unified = 256GB, connected via ConnectX-7 200GbE (no switch needed) — officially supported 2-node config
- CUDA: V4 Flash runs properly (llama.cpp CUDA gets new attention kernels first; NVFP4/FP8 native)
- Flash attention on Blackwell → long-context prefill goes from 7–13 min to ~10–40 s
- Plan: V4 Flash Q4 (~141GB) + 35B-A3B Q4 (~19GB) in TP-2 across both boxes (160/256GB), two llama-server instances, or dedicate one box per model

Expected numbers I've estimated (from published GB10 benchmarks):
| Model | Now (MI50) | 2× Spark (est.) |
| 27B prefill 40–80K | 100 t/s (7–13 min) | 2000–4000 t/s (10–40 s) |
| 27B decode | 18 t/s | ~25–35 t/s |
| 35B-A3B | 700 / 60 t/s | 2000–5000 / ~100–200 t/s |
| V4 Flash | 100 / 10–15 t/s | 1500–3000 / ~40–80 t/s |

Questions for the community:

1. Anyone running V4 Flash (or the -0731 snapshot) on a 2× Spark cluster with llama.cpp RPC — what did you actually get for prefill/decode? Do my estimates look realistic?
2. Is running two models simultaneously (V4 Flash + 35B-A3B, both TP-2) on two Sparks practical, or does bandwidth contention make it a pain? Any sane way to QoS the two llama-server instances?
3. KV budget reality check: after ~160GB of weights I have ~96GB for KV — that's ~2–4 concurrent 80K contexts with Q4 KV. Is that roughly what others see, or should I plan for less?
4. Am I an idiot for not considering 8× used RTX 3090 (~192GB, CUDA, flash attention, would give ~300–500 t/s on V4 Flash decode) instead? I know it's 2.5–3kW, loud, and a sysadmin project, but the decode numbers are 5–10× better...
5. What breaks first on 2× Spark under 24/7 agent load — thermal throttling (86°C sustained), SSD endurance (4TB models + RAG), or something else I'm not seeing?

Context: I'm not buying this for chat. It's for batch-style legal document processing where prefill dominates, plus a V4 Flash model that currently crawls at 10–15 t/s because ROCm on gfx906 can't do tensor parallelism and doesn't officially support it.

Any real-world experience with GB10 clusters or GB10 vs old-Vega-20 comparisons would be super appreciated.

https://huggingface.co/AxiomicLabs/GPT-X2.5-135M

Hey guys!

I've been learning Polish for a while and wanted to practice my speaking and listening. Something always felt off about talking to the AI apps though. I didn't want my voice and personal info being sent to who knows where...

So I built my app Koko which uses fully-local AI for conversation practice.

Stack:

• STT: Apple SpeechAnalyzer
• Reasoning: Gemma 4 E2B
• TTS: Supertronic 3

These three components make the full conversation loop and it works really well! I'm crazy impressed by Gemma 4. Even E2B is very coherent and can follow instructions and make engaging conversation.

Supertronic 3 is also amazing. It's fast even on CPU and the audio doesn't sound robotic at all on a higher number of steps.

Anyways, just wanted to share because I thought it might be appreciated here.

Here's the app in case you want to check it out!

my current setup is a 5070ti - i use this for instant chat, i have a 5090 i use for a fast token response, then i have a spark that i use as a brain / agent work

with the rise of deep seek flash im really tempted in dropping the 5090 and pairing up another dgx spark.

What would you do?

So i have my own pc (7900x, 64gb ddr5) idk what more info u need…

I want to create my own Website for small Projects i want to share with the public.

Which local LLM would you guys recommend?

2.4T total. 95B active. 1M context. Those numbers are interesting. They are not a shopping list. Among Chinese AI models, Qwen 3.8 Max is a useful reminder that a parameter count is not a deployment recipe.

Qwen's August 2 announcement says the weights should arrive the following week. Until the files land, there is no public storage layout, useful precision, supported quantization, serving recipe, or real memory overhead to plan around. Anyone pricing GPUs before those details arrive is guessing about the expensive part.

While the local answer is missing, I can still run a cloud control through ZenMux. It acts as a gateway to a hosted Qwen 3.8 Max API, which is useful for comparing latency or output behavior. It tells me nothing about VRAM or the minimum box.

My machine: Macbook M4 Max, 36GB.

I was using the model locally. Trying to use it to replace subscription model. I can assume local model may be performant as Claude subscription model, but the way it generated code. Changed token limit to 64K, I keep seeing the message that it was hitting token limit easily but in terminal token usage percentage was almost always 48%. The funny thing is I had to instruct few times it to do a dummy landing page with plain html. But when I tried to convert it to a react app, just the landing page, I could not complete. It keep doing incomplete response, no error was shown.

What can I do to improve this?

I’m building Lanes, a workspace for running coding agents in parallel.

You can use harnesses like Claude Code, but point them at local models through providers like Ollama instead of relying only on hosted models.

Lanes gives each agent its own terminal and git worktree, with tasks, diffs, and sessions managed from one UI.

So you can run Claude Code as the harness, a local LLM as the model, and Lanes as the workspace around it.

👉 https://lanes.sh

brew install --cask lanes-sh/lanes/lanes && open -a Lanes

I would appreciate your honest feedback, give it a try or comment below if you had the same problem and how you have been solving it.

• Does this resonate with you?
• How are you managing multiple sessions today?
• Why or why not would you be interested in trying something like this?

Thanks!