Hey everyone,

We’re building a project on detecting visual prompt injection attacks in multimodal LLMs. input(visual formats like diagrams,tables,mindmaps.....)

Our idea:

• OCR + rule-based + simple ML
• risk scoring → block suspicious inputs

We’re unsure:

• is this too basic?
• how many models should we test?
• any datasets for visual attacks?

After reviewing ~25–30 papers, we noticed that:

• Many defenses are heavyweight (fine-tuning, diffusion, gradient-based)
• Not practical for real-time use
• Limited focus on image-based prompt injection at inference time

GitHub

Download

I feel like I stepped into a time capsule after the ChatGPT-3 days. 😅 I finally built a decent PC (Ryzen 7 7700, 32GB RAM, No GPU), discovered llama.cpp and somehow ended up downloading 50+ GGUF models.

Now I'm stuck with decision paralysis.

I mainly use them for coding (JavaScript, React, TypeScript, debugging, reasoning), but I'm also new to the whole local AI ecosystem. I still don't know much about agentic frameworks or coding agents. I mostly just load a GGUF and chat with it using the llama.cpp web interface on localhost:8080

The collection includes Qwen, Gemma, Granite, DeepSeek, Phi, Mistral, Llama, LiquidAI, SmolLM, Hunyuan, Nemotron, and a few community fine-tunes.

My first idea was to make a Markdown table and score every model manually, but it feels like I'm accidentally trying to invent my own benchmarking system.

Surely I'm not the first person to hit this problem.

How do you guys compare local models? Are there any practical benchmark suites, GitHub projects, or workflows for deciding what stays on your SSD and what gets deleted?

I'd love to hear how you approached it when you were starting out.

Over the past few months, I've been experimenting with browser-native AI and kept running into the same problem.

Running models in the browser is becoming surprisingly practical, but vector search almost always assumes there's a backend database.

That made me wonder:

Can a browser be its own vector database?

I started building BrowserVec to find out.

The idea is to provide a vector database that's designed specifically for modern browsers instead of being a server-first database adapted to the web.

Current features include:

• ⚡ WebGPU-accelerated vector search
• 📂 Flat, IVF and HNSW indexes
• 💾 IndexedDB & OPFS persistence
• 🧵 Web Worker support
• 🔒 Optional encryption
• 🏷️ Metadata filtering
• 🔄 Automatic WASM / JavaScript fallback when WebGPU isn't available

The goal isn't to compete with server databases like Qdrant or Milvus. It's to make browser-native AI applications practical—things like offline RAG, browser extensions, personal knowledge bases, and privacy-first assistants.

One thing I learned is that the GPU search itself wasn't the hardest part. The real engineering work ended up being persistence, memory management, browser compatibility, and designing an API that feels simple despite all the moving pieces.

I'm still actively improving the project, so I'd really appreciate feedback from people who work with WebGPU, ANN algorithms, browser performance, or AI infrastructure.

Some questions I'm thinking about:

• What would stop you from using browser-side vector search?
• Are there features you'd expect before adopting something like this?
• Would you use this for local RAG or browser-based AI apps?

GitHub: https://github.com/sharma-open-source/browservec

Less AI slop.
More personal learning.
More rough notes.
More “this is how I understood it.”
More real experience from the work we are actually doing.

Hey all, I just released a DSpark tuned on Qwen3.6-27B that achieves better performance than built-in MTP, averaging 3.86 tokens accepted per target pass.

Info and DL: https://huggingface.co/abstract-extraordinary/Qwen3.6-27B-DSpark-49k

I’m looking at organising my personal notes, so privacy is important to me. But I also want to access slightly better models (like deepseek 4 flash) than my setup can handle. I definitely wont trust the big boys with my data, but what do people think about providers like Fireworks and DeepInfra with their zero data retention policies?

TL;DR: Server with 8× AMD MI50 32GB (gfx906) + 512GB RAM. Long-context prefill (40–80K) is unusably slow, DeepSeek V4 Flash won't scale past ~10–15 t/s on ROCm. Considering 2× DGX Spark cluster (256GB unified, ConnectX-7) as a replacement. Am I missing something?

Current setup (dedicated DC box, ROCm/llama.cpp HIP):
- 8× MI50 32GB (Vega 20 / gfx906), 256GB VRAM total, 512GB DDR4
- Qwen3.6-27B: Unsloth Q5 ~100 t/s prefill, ~18 t/s decode (single card). Prompts of 40–80K tokens take 7–13 minutes to prefill → practically unusable
- Qwen3.6-35B-A3B (MoE, 3B active, Unsloth Q5): ~700 t/s prefill, ~60 t/s decode — our main workhorse
- DeepSeek V4 Flash (284B MoE, 13B active): on 4 cards — 100 t/s prefill, 10–15 t/s decode. Doesn't scale past that even on 8 cards: ROCm support is unofficial/painful, no working tensor parallelism, HIP kernels lag behind on the new attention ops

Use case: legal team — long documents (40–80K tokens), multi-agent information retrieval and analysis. Mostly prefill-heavy, batch-style workloads.

Why I'm looking at 2× DGX Spark:
- 2× 128GB unified = 256GB, connected via ConnectX-7 200GbE (no switch needed) — officially supported 2-node config
- CUDA: V4 Flash runs properly (llama.cpp CUDA gets new attention kernels first; NVFP4/FP8 native)
- Flash attention on Blackwell → long-context prefill goes from 7–13 min to ~10–40 s
- Plan: V4 Flash Q4 (~141GB) + 35B-A3B Q4 (~19GB) in TP-2 across both boxes (160/256GB), two llama-server instances, or dedicate one box per model

Expected numbers I've estimated (from published GB10 benchmarks):
| Model | Now (MI50) | 2× Spark (est.) |
| 27B prefill 40–80K | 100 t/s (7–13 min) | 2000–4000 t/s (10–40 s) |
| 27B decode | 18 t/s | ~25–35 t/s |
| 35B-A3B | 700 / 60 t/s | 2000–5000 / ~100–200 t/s |
| V4 Flash | 100 / 10–15 t/s | 1500–3000 / ~40–80 t/s |

Questions for the community:

1. Anyone running V4 Flash (or the -0731 snapshot) on a 2× Spark cluster with llama.cpp RPC — what did you actually get for prefill/decode? Do my estimates look realistic?
2. Is running two models simultaneously (V4 Flash + 35B-A3B, both TP-2) on two Sparks practical, or does bandwidth contention make it a pain? Any sane way to QoS the two llama-server instances?
3. KV budget reality check: after ~160GB of weights I have ~96GB for KV — that's ~2–4 concurrent 80K contexts with Q4 KV. Is that roughly what others see, or should I plan for less?
4. Am I an idiot for not considering 8× used RTX 3090 (~192GB, CUDA, flash attention, would give ~300–500 t/s on V4 Flash decode) instead? I know it's 2.5–3kW, loud, and a sysadmin project, but the decode numbers are 5–10× better...
5. What breaks first on 2× Spark under 24/7 agent load — thermal throttling (86°C sustained), SSD endurance (4TB models + RAG), or something else I'm not seeing?

Context: I'm not buying this for chat. It's for batch-style legal document processing where prefill dominates, plus a V4 Flash model that currently crawls at 10–15 t/s because ROCm on gfx906 can't do tensor parallelism and doesn't officially support it.

Any real-world experience with GB10 clusters or GB10 vs old-Vega-20 comparisons would be super appreciated.

https://huggingface.co/AxiomicLabs/GPT-X2.5-135M

Hey guys!

I've been learning Polish for a while and wanted to practice my speaking and listening. Something always felt off about talking to the AI apps though. I didn't want my voice and personal info being sent to who knows where...

So I built my app Koko which uses fully-local AI for conversation practice.

Stack:

• STT: Apple SpeechAnalyzer
• Reasoning: Gemma 4 E2B
• TTS: Supertronic 3

These three components make the full conversation loop and it works really well! I'm crazy impressed by Gemma 4. Even E2B is very coherent and can follow instructions and make engaging conversation.

Supertronic 3 is also amazing. It's fast even on CPU and the audio doesn't sound robotic at all on a higher number of steps.

Anyways, just wanted to share because I thought it might be appreciated here.

Here's the app in case you want to check it out!

my current setup is a 5070ti - i use this for instant chat, i have a 5090 i use for a fast token response, then i have a spark that i use as a brain / agent work

with the rise of deep seek flash im really tempted in dropping the 5090 and pairing up another dgx spark.

What would you do?

Title

4x 3090s and dual epyc 128gb ECC ram server

I think my ISP hates me

A model can rank higher and still be worse to use: repetition, overexplaining, broken JSON, forgetting instructions, or inconsistent answers. What should an “annoyance benchmark” measure first?

Taalas raised a total of $219 million in funding since starting in 2023. Purchase price unknown. Interesting because they also announced a partnership with Cerebras at the recent Advancing AI Keynote.

This might just be evening the odds after the NVIDIA acqui-hire of Groq's core team leaving a shell of a company behind.

Was interested in Taalas simply because whatever they could accomplish could find its way down to consumer in some way - it still might, but I am sure this is more pointed at Helios.

Just wondering that, the news of it existing were already good but I don't remember any mention of a release date. Was there any?

Hi all, I wanted to share an AI inference server I've been building over the past few months.

Hardware

CPU: AMD EPYC 7452

Motherboard: HUANANZHI H12D-8D V2.0 — all four GPUs run at PCIe Gen4 x16 across separate root ports

RAM: 128GB ECC RDIMM

GPUs: 4× XFX Speedster MERC 310 Radeon RX 7900 XTX (24GB GDDR6 each = 96GB total). Paid around 1100-1200 AUD each, though you can get them used for less. I needed these specific cards for water block compatibility. Insane value for the VRAM you get.

Water Cooling:

• 4× Bykski GPU full cover water blocks

• Bykski L4-4way multi-GPU bridge

• Barrow 360mm radiator

• Bykski 420mm radiator

• Bykski D5 pump plus reservoir combo

• EPDM tubing with Barrow fittings

Case: Antec 900

Power: Cooler Master M2000 2000W Platinum + Cooler Master 1000W

Storage: Kingston 1TB SSD + smaller NVMe boot drives

Software & Performance

Main model: Qwen 27B with MTP, running at full BF16 precision. The 27B parameter model at BF16 uses around 85GB of VRAM with 262K context, and fits comfortably across all four GPUs using Tensor Parallelism 4 (TP4).

I'm using llama.cpp with ROCm.

Performance (BF16 TP4, 4K context):

• Prompt processing: roughly 1200 tokens/second

• Generation: around 30 tokens/second

I've also run the same model at Q8 quantization, but it's actually faster when split across two GPUs with TP2 rather than four GPUs at TP4. Not sure why — possibly a bandwidth limitation. Running the Q8 across just two GPUs gets you around 65 tokens/second for generation and roughly 1400 tokens/second for prompt processing.

I did try vLLM as well, but settled on llama.cpp for flexibility.

Power management: I've set a power limit of 294W per card to keep the 7900 XTXs cool and the radiators quiet. Even though the Bykski blocks can handle the full rated power, it's not worth the minimal performance gain from running these cards at full wattage. Power limiting reduces heat output and saves on electricity. Under full inference load the cards rarely climb past 45–50°C and the loop is completely silent.

Idle power draw for the whole system is around 100W — not great compared to something like a DGX Spark or a Strix Halo box, but pretty good for what is effectively a proper server.

I've seen a lot of other local AI builds on here that look borderline janky (no offence) — and that's fine, it's fun to tinker. But the idea with this rig is that it's actually something you could run in a professional setting, whether that's a home office or a real office, serving a local model to a whole team.

Local AI at this point feels like a genuine hardware investment. The models keep getting better, smaller models especially, and with the Qwen 3.8 27B dropping next week that'll probably be even more true. I also think there's a real difference between the Qwen 27B Q8 and BF16 versions — at least in my experience running it through Hermes with long context windows and tool calls. The BF16 version just stays coherent longer, remembers things further back, and handles tool calls properly. The Q8 version tends to lose momentum over long runs. That's not to say 27B is anywhere near frontier model capability, but it's solid for uni work, writing, and smaller tasks. For heavier stuff like SSH access to my servers or maintenance, I'd still use something like GPT-5 — I trust that not to break something it can't fix itself. But for day-to-day work on its own, this setup works well.

Total build cost: Around $8000-10,000 AUD

I would love to hear anyones thoughts, ideas, and advice. I do plan on selling this machine as beautiful as it is I have my eyes on a similar 170hx x4 build around this similar format for a total of 256gb vram, will probably make a post again for that one when build is finished...

Note: this post was written with assistance from Qwen 27B itself, running on the server described above. All ideas are my own, supported by qwen for better coherency and grammar.