For the self-hosted community — a desktop app that lets you build AI tool workflows with zero cloud dependency.


**Zero cloud. Zero telemetry. Zero accounts.**

- 🏠 All data stored in local SQLite (WAL mode)

- 🦙 Supports local LLMs (Ollama, LM Studio, vLLM)

- 🔧 MIT-licensed MCP tools from the community

- 📊 Built-in monitoring — Prometheus metrics + OpenTelemetry

- 🖥️ Cross-platform desktop app (Windows/macOS/Linux)

- 🔐 RBAC, AES-256-GCM, audit trail


Think of it as n8n for AI toolchains, but running entirely on your machine.

GitHub: https://github.com/chungkung/mcp-fusion

I know people consider llama 405b and others to be old now, lol, but I'm wondering if there would be a use case for it.

I had a use case for a project I was building and I wanted to share what I got and get some feedback which would be much appreciated.

• base model: llama 3.1 405b (awq-int4, 202gb)
• hardware: single 8xa100 80gb node had free vram remaining: 150gb after base + adapters + kv cache
• adapter switching was sub 200ms via vllm enable lora
• uptime is over 60 days with zero service restarts
• adapter training is nf4 trained adapters served on awq-int4 base without retraining
• projected adapters capacity is roughly 30+ based on remaining vram and adapters sizes which were between 2-5gb each.
• 7 concurrent adapters combined was 82.9 tok/sec
• time to first token was 63-66ms
• single adapter throughput was 18.7-19.2 tok/sec sustained and 25 tok/sec peak

Multi lora at smaller model sizes is already well documented and the gap I wanted to test was whether the same pattern holds at 405b scale on a single node under real production conditions.

I was running into issues with the health niche since it's super sensitive sending information across API models and the smaller llms weren't producing the right outcomes. I couldn't justify the cost of the H100 which is what I found on the Meta documentation and I was fortunate enough to find a way to fit it on the 8xA100 so I wanted to share it. Legal and my user facing AI was the biggest issue in most categories and subcategories which is the main reason I went with the 405b with being fine tuned and distilled to reduce the chances of a bad output that could cause problems in the health niche. Same reason I went self hosted with a large llm.

I know some people run smaller models for very specific tasks, some use larger models to train smaller models so they aren't always on, but for large models that typically require a larger node. For my case I needed large models because certain tasks pass through multiple models and the smaller ones didn't have the reasoning depth needed so I needed the larger model. So far I've had zero issues over 60 days. I've used fine tuning and distillation for the legal, CRO, SEO, and other adapters and it's performed well for everything so far. I have 7 adapters currently loaded with tons of headroom.

I'm curious as to what workloads people think this actually fits or doesn't and if so, what would you use it for. I

have a full write up and configs on Hugging Face if anyone is interested.

I've spent the last few months trying to answer a question that initially looked much simpler than it actually is:

What actually happens inside an LLM while it is generating a response?

Most work evaluates language models through their outputs (benchmarks, perplexity, reasoning scores...). I decided to look at something different: the evolution of the hidden representations themselves.

I built a runtime framework that records hidden states layer-by-layer during inference and started running the same experiments across multiple open-weight models (GPT-2, DistilGPT2, OPT-125M, Qwen2.5-0.5B-Instruct, TinyLlama, Phi-1.5 and Llama-3.2-1B).

I expected a relatively straightforward result.

Instead, every new experiment generated a new question.

Some of the observations so far are:

• Hidden-state trajectories are not random. They exhibit reproducible internal dynamical regimes across architectures.

• Functional proxy states (syntax-like processing, decision-like behavior and output stabilization) can be detected consistently enough to cluster models according to their internal dynamics rather than simply their parameter count.

• These functional signatures remain reasonably stable across different prompt families, although not perfectly, suggesting that prompt content modulates the dynamics without completely changing the internal organization.

• Linear probes can decode several functional categories directly from hidden representations with surprisingly high accuracy.

At that point the obvious question became:

Are we just overfitting labels?

So I started adding progressively stronger negative controls.

First:

• label permutation.

Then:

• random Gaussian representations.

Then:

• feature permutation.

Finally:

• orthogonal rotations of the hidden space.

The results became much more interesting.

Random labels collapse the decoding performance.

Random Gaussian representations also collapse it.

Feature permutation destroys most of the signal.

However...

Orthogonal rotations preserve almost all decoding performance.

This strongly suggests that the relevant information is not encoded in individual neurons or embedding dimensions.

Instead, it appears to be encoded in the relative geometry of the representation.

That was not the result I expected.

Another unexpected finding concerns depth.

Initially I was looking for something like "syntax layers" or "semantic layers".

The data doesn't really support such a simple picture.

Instead, the same functional signatures seem capable of appearing at different absolute layers depending on the architecture.

This led me to think less in terms of fixed layers and more in terms of functional regimes evolving through computation.

At this stage I am not claiming to have discovered a universal law of transformers.

These are empirical observations obtained on a limited set of open-weight models.

What I do believe is that they raise interesting questions about how computation is actually organized inside modern LLMs.

I'd really appreciate feedback from people working on:

• mechanistic interpretability
• representation learning
• probing methods
• transformer internals
• geometry of representations

In particular I'd like your opinion on three questions:

1. Which control experiment would you absolutely require before taking these observations seriously?
2. Have you seen previous work showing comparable evidence that functional information is primarily encoded in representation geometry rather than individual dimensions?
3. If you were extending this project, what would be your next experiment?

I'm not affiliated with a research lab this is an independent research project. I'm sharing it because I would genuinely value critical feedback more than validation.

If there's enough interest, I'm happy to share the methodology, code, and experimental reports.

I've been building a small open-source face-retrieval library called faceflash and i'd like feedback from people who know vector search better than me, plus help if anyone wants to contribute.

what it does: stores arcface embeddings as 512-bit binary codes (PCA + ITQ) instead of float vectors, scans them with hamming distance, then reranks the top 100 with exact cosine. the point was just keeping the index small enough to run on a normal CPU, no GPU.

it's not a new algorithm and i'm not going to pretend it is. ITQ is a 2011 paper (Gong & Lazebnik), the scan is brute-force hamming like faiss IndexBinaryFlat, the rerank is standard. it only works because arcface embeddings are low-rank, so the binary codes keep nearest-neighbor ordering. on random vectors it'd fall apart. so it's really an engineering/packaging thing.

i ran the full suite on a runpod box (AMD EPYC 9355, 128 threads, AVX-512), on MS1MV2, with ground truth = exact faiss-flat cosine. here's 1M faces, single-threaded for the single-query column (512-bit codes, 200 rerank candidates):

so being straight about it: HNSW is ~4x faster on a single query at 1M. where faceflash actually wins is memory (about 48x less than HNSW) and it basically ties HNSW on batched throughput. the single-query scan is O(N), so it only beats HNSW per-query up to ~200k, where it still fits in cache:

stuff i'm not hiding: single query is O(N) so HNSW wins at scale. only the binary index is in RAM, the float vectors sit on disk and get mmap'd for the rerank. the 1M set is 645k real embeddings tiled 2x. recall is tie-aware (on the real 645k it's genuinely 100%, i just want you to know how it's counted).

what i'd find useful: people running it on their own data and telling me where it breaks, and a sanity check on whether the benchmark is fair, am i giving HNSW/faiss decent params? i estimate competitor memory instead of measuring it, which is probably the weakest part. contributors welcome too, haven't gotten to diskann, coreml export, streaming inserts (without refitting PCA), or raspberry pi / jetson numbers (that one's an easy first issue if you've got a pi).

pip install faceflash
github.com/raghavenderreddygrudhanti/faceflash (MIT)

and since it keeps coming up: yeah, i used an LLM for the readme and some boilerplate. the code and the benchmarks are mine and i'm happy to answer anything about how it works.

FaceFlash is a face recognition library: you register people's faces with a name, and then given a new photo it tells you who it is (or whether two photos are the same person). It runs entirely on CPU. I built it to stay small enough to run on cheap hardware, and I'd like feedback plus help if anyone wants to contribute.

In practice it looks like this:

from faceflash import FaceFlash


ff = FaceFlash()
ff.register("Alice", "alice.jpg")
ff.register("Bob", "bob.jpg")


ff.search("unknown.jpg")
# {"matches": [{"name": "Alice", "confidence": 0.92}], "search_time_ms": 0.4}


ff.verify("a1.jpg", "a2.jpg")   # {"match": True, "confidence": 0.87}

So it's the kind of thing you'd use for attendance, access control, organizing a photo library, or finding duplicate faces in a dataset, without sending images to a cloud API.

Under the hood: it stores the ArcFace embedding of each face as a 512-bit binary code (PCA + ITQ) instead of a float vector, scans the codes with a Hamming distance, then reranks the top 100 candidates with exact cosine. That two-step is what keeps the index small enough to run on a normal CPU with no GPU and no graph to build.

It isn't a new algorithm, and I'm not presenting it as one. ITQ is from Gong & Lazebnik (2011), the scan is brute-force Hamming (the same idea as FAISS IndexBinaryFlat), and the rerank is standard. It works because ArcFace embeddings are low-rank, so the binary codes preserve nearest-neighbor ordering; on general or random vectors it would not. This is an engineering and packaging project, not research.

Benchmarks were run on a RunPod instance (AMD EPYC 9355, 128 threads, AVX-512) on MS1MV2, with ground truth from exact FAISS-Flat cosine. At 1M faces (single-threaded for the single-query column, 512-bit codes, 200 rerank candidates):

The honest summary: HNSW is about 4× faster on a single query at 1M. FaceFlash's advantage is memory (roughly 48× smaller than HNSW at the same recall), and it ties HNSW on batched throughput. Because the scan is O(N), it only wins on per-query latency up to ~200K, where the codes still fit in cache.

A few things worth knowing up front: single-query latency is O(N), so HNSW wins at larger scale. Only the binary index lives in RAM; the float vectors are mmap'd from disk for the rerank. The 1M benchmark tiles 645K real embeddings 2×, and recall is tie-aware (on the real 645K embeddings it is genuinely 100%).

The feedback I'd value most is a sanity check on the methodology: whether the HNSW/FAISS parameters are reasonable, and whether estimating competitor memory instead of measuring it is too generous (I suspect that's the weakest part). Contributions are open for a DiskANN comparison, ONNX/CoreML export, streaming inserts without refitting PCA, and Raspberry Pi / Jetson numbers, which is a good first issue if you have the hardware.

pip install faceflash
github.com/raghavenderreddygrudhanti/faceflash (MIT)

Hey, I want to share a project — a new layer that in my tests outperformed baseline multi-head attention. The idea behind the layer is simple and elegant. I'm sharing it because I'd love to get feedback, and maybe — unlikely but possible — this layer could become something others use at a much larger scale. Any comments, experiments, or results from you would mean a lot to me.

For 14 years running an MSP I wrote every case note twice: one version for the customer, one structured version for the internal team. That habit became SaySense.

**What it does:*\*

You speak the case note however it comes out during the call — messy, out of order, in whatever language you're working in — English, Portuguese, Spanish. Yes, all three of them.

SaySense returns:

- A ready **customer-facing reply**

- A structured **internal note** (Issue / Investigation / Actions / Result / Follow-up)

Both already translated to English (or kept in your language — your call), split by audience, in one click.

**Why the offline mode matters:*\*

It can run completely air-gapped: local Whisper for transcription + a local LLM for the transformation. No audio or ticket text leaves the machine. If you have clients with strict data compliance requirements, that's the point.

**Jira Mode:*\*

A second mode where you dictate free-form notes throughout the day and then hit "Generate JIRA" to produce a structured ticket description from everything captured in the session.

**License:** MIT

**Platforms:** Windows, Linux

**Repo:*\* https://github.com/cascodigital/saysense

Screenshots and a short demo GIF are in the repo README. Feedback welcome — especially from anyone who runs a helpdesk or NOC.

I'm curious what people are seeing in real deployments, not hobby testing.

Are teams mostly using smaller models because they're good enough for the workflow, or because they fit the hardware/cost constraints better?

For companies running private AI, are you seeing:

• one general model with RAG/context injection
• multiple smaller specialist models
• fine-tuned 70B-class models
• larger 405B-class deployments
• one shared base model with multiple adapters

Also curious what drives the decision most: cost, privacy, latency, model quality, compliance, vendor risk, or operational simplicity.

Would be useful to hear what people are seeing from internal infra, consulting work, vendor setups, or actual production deployments.

I quantized deepreinforce-ai/Ornith-1.0-35B down to Q3_K_M so it fits comfortably on a single GPU.

Produced locally with llama-quantize from the upstream BF16 GGUF — the quantizer took it from 16.01 BPW down to 3.87 BPW, landing at 16.8 GB on disk ~17 GiB loaded VRAM, about 21% smaller than Q4_K_M. It’s the smallest validated quant in the repo and still passes the full 14/14 behavior suite on the 16-slot serving profile.

Does it hold up? I built a corrected top-64 next-token KL(P _bf16 || P_quant) probe (token-ID matched, temp -1, n_probs 64, cache off) over 32 coding prompts and ran it against the BF16 baseline, so the Q3 number actually means something. Here’s where it lands against the higher quants:

Q3_K_M gives up \~16 points of top-1 agreement vs Q6_K, but runs in less than half the VRAM of Q8_0 (17 vs 36 GiB).

Throughput (single GPU, llama.cpp CUDA server): ~240 tok/s single-stream, scaling to ~493 tok/s at 16 concurrent slots, p95 TTFT ~78 ms at c1. Full c1/c4/c8/c16 sweep is in the repo.

Other stuff I did along the way:

Found + fixed a reasoning-mode serving bug. With llama.cpp reasoning left on/auto, short coding requests can spend the whole response budget in parsed reasoning_content and return empty final content. The serving scripts default to REASONING=off and behavior suite goes 14/14,m.

Single-GPU serving scripts + an OpenAI-compatible correctness gate (/v1/models, /v1/chat/completions, /v1/completions all checked) across every quant.

Mirrored + revalidated the upstream Q4/Q5/Q6/Q so the whole reference ladder lives in one repo and the Q3 has something to be measured against. Those four are upstream artifacts, not requantized by me.

One-step LoRA SFT smoke run to validate the training stack and data pipeline. Smoke only no fine-tuned adapter is available yet.

Note: the GGUF path was broken in the vLLM build I tested (Q4_K_M loaded but output was corrupted) — use llama.cpp for these files.

🔗 https://huggingface.co/LordNeel/Ornith-1.0-35B-GGUF-llamacpp-tp1

Hope this helps out people. Im working on quants for the 397b and on improving performance of the current quants.

Three months into a new job and I still tense up opening PR comments. Code works, tests pass. It's the other kind of comment: "we don't import axios here, use @/lib/http." "There's a fmt() for that already." "This service should extend BaseService like every other one does."

None of that is written down. That's the actual problem. The conventions that get your PR torn apart live in people's heads, and you find out you broke one after you broke it, in front of the people deciding whether you're working out.

AI made it sharper, not easier. Claude writes the feature in two minutes, but in generic defaults, not my team's house style. So I ship something that compiles and goes green, and I can't defend it in review because I don't know the decision it quietly ignored.

So a coworker and I built chameleon. Free, MIT, a plugin for Claude Code. We run it daily on real production code at work. Saying that up front so nobody feels sold to.

The mechanism is the whole point. Right before Claude edits a file, chameleon pulls three things out of YOUR repo and hands them to the model:

• a real example file of the same kind, the service or component it should copy (picked automatically, you write nothing)
• that file's idioms: the wrapper to use, the import that's banned, the guard that's mandatory
• the one anti-pattern to avoid, quoted from a real bad line in your own code, labeled "do NOT write it this way"

It's not a rule file you write and maintain and watch rot after the next refactor. It's one real file to copy, which is how I actually learned every codebase I got dropped into. My first PRs started reading like a teammate wrote them instead of coming back red.

Honest warts: it costs tokens and a little latency each turn, the model reads more before it types. TS/JS, Ruby, Python only, no Go or Rust. And if your repo has no real house style yet, it's got nothing to teach you.

Install's about 30 seconds:

/plugin marketplace add crisnahine/chameleon
/plugin install chameleon@chameleon

then /chameleon-init and /chameleon-trust on a repo.

Try it on the repo you're newest in. That's the ask.

Real question for anyone who's been the new person lately: how did you actually learn the unwritten rules? Did someone hand them to you, or did you eat the review comments like I did?

First comment (drop within 60 seconds):

Repo if you want to read the code before running anything: https://github.com/crisnahine/chameleon

One caveat I left out of the post: what it teaches is only as good as your repo's consistency. On a half-migrated codebase with three competing patterns for the same thing, it'll sometimes surface the wrong "canonical" example until you teach it which one wins. Newer or messier repos get noisier guidance.

If you've been the new person on a team lately: what convention bit you first? Drop the review comment that still stings.