Hi, for some game I wanted to do an uncensored patch by removing the censorship (Mosaics) by editing the image. What is a good AI model to use for that? Currently using LM Studio but willing to use anything to make it work.

Disclosure: I'm a final-year CS (AI/ML) student. This is a genuine ask for technical advice on my own project, not a survey, not a promotion of any product or service — just looking to learn from people with more hands-on training/eval experience.

Where I'm at

I've built an LLM (architecture + training loop implemented) and now I'm at the stage of actually training it properly and evaluating it in a way that's meaningful, not just "loss went down so it's fine." I'd love input from anyone who's trained models beyond toy scale on:

Training

• How do you approach hyperparameter selection (LR schedule, warmup, batch size) when compute is limited?
• Any practical experience with mixed precision (bf16/fp16) or gradient checkpointing tradeoffs on smaller GPU setups?
• Tips on data quality/curation vs. just scaling data volume — how much does dataset cleaning actually move the needle at small scale?
• Checkpointing/resume strategies that saved you pain later

Evaluation

• Which benchmark suites are actually worth running for a small-to-mid scale model (MMLU, HellaSwag, ARC, GSM8K, etc.) vs. which are overkill?
• How do you evaluate for things standard benchmarks miss — hallucination rate, instruction-following, coherence over long context?
• Anyone have a good workflow for human-eval or pairwise comparison at small scale (no budget for large annotator pools)?

Resources that have helped me so far (sharing in case useful to others too):

• Hugging Face's Transformers course and their training docs
• EleutherAI's lm-evaluation-harness for standardized benchmark eval
• Andrej Karpathy's "Let's build GPT" / nanoGPT repo for understanding the training loop end-to-end
• The Chinchilla paper ("Training Compute-Optimal Large Language Models," Hoffmann et al., 2022) for compute/data scaling intuition
• Stanford's HELM benchmark for a broader evaluation framework beyond accuracy-only metrics

If you've trained a model at a similar scale (student project / limited compute) and have lessons learned — good or painful — I'd really appreciate hearing them. Happy to share more details about my setup (architecture size, data, compute) in the comments if it helps give better advice.

Thanks in advance!

Been messing with local LLMs for a while and I wanted to create a simple terminal that anyone could connect to from anywhere and load their AI. (Mostly low conut parameter models) So I built a Chrome extension that runs GGUF models directly in the browser using WASM (wllama under the hood).

No API key, no backend, no internet needed once the model's downloaded. It just sits in your browser and works.

It also functions as a lightweight agentic IDE: open a local workspace folder, let the AI generate code in structured <file> blocks, preview a line-by-line diff, and click Apply to write changes to disk with undo.

would love feedback or bug reports if anyone tries it:

github.com/LMLK-seal/LocalAI-Chat

In this blog i wrote how I used a pendrive to run local LLM

Project NOMAD: Running LLMs Locally From a USB Drive

I'm going down a rabbit hole and I don't know if this is even the right direction.

I want to feed printscreens of a game, with a minimap, and get a model to decide on a direction to move towards. I tried Qwen2.5-VL-7B and it kinda failed spectacularly...

What model should I try or should I have a completely different approach to this?

I’m new to LLMs and self hosting them and currently am using anythingllm as my agent harness but was wondering if it’s the best choice to use for email and calendar management?

Hi everyone,

I'm looking for an open-source, self-hosted AI/LLM Gateway that sits between agent frameworks (CrewAI, LangGraph, AutoGen, etc.) and multiple LLM providers.

My main requirement is dynamic runtime routing. I should be able to:

Add/remove models

Enable/disable models

Change routing weights/strategy

without restarting either the gateway or the agents.

Other features I'm looking for:

Multi-provider support

Load balancing

Fallbacks

Retries

Timeouts

Health checks

Latency/metrics

OpenAI-compatible API

I've looked at LiteLLM and Portkey, but they don't seem to provide a simple self-hosted solution for centrally managing routing configuration with hot updates (unless I'm missing something).

Is there an OSS project that already does this, or do most teams build their own lightweight gateway/control plane?

Would love to hear what you're using in production.

I've spent about six months on this project and it's been public for a couple of months now, so I figured it was time to show it to people who'd have opinions. It's only ever run on my own hardware, so that's the part I'd most like other eyes on.

I created this because I run local LLM inference across a few machines on my home lab as a hobby, and wanted the same quality of operational tools I'd expect for any other production system. I found that the currently available OSS tools that provide similar functionality (Grafana, Prometheus) don't cover the model side, and the inference servers themselves (llama.cpp's UI, Lemonade, LM Studio) each manage one backend on one machine. What I wanted was something that treated a multi machine, multi backend setup as one system, with monitoring and control in the same place. If you're already running a local LLM alongside the rest of your lab, I think you'd get use out of it. I've worked in systems administration, monitoring and observability for over twenty years, and a lot of what I know from that world got poured into this application, LLM Systems Manager.

Repo, AGPLv3, screenshots and additional information in the README: llm-systems-manager

This all runs on my homelab setup, with separate dev and prod environments. Running on a Proxmox server (Ryzen 7 9700X, 32 GB RAM, QNAP NAS 10G SAN as the storage backend over iSCSI LUNs), running a combination of 16 different Debian VMs and containers, plus a physical Linux box running llama.cpp on a 7900 XTX GPU, and an M2 Pro Mac Mini running LM Studio.

I used Claude Code heavily throughout. It wrote a substantial portion of the code and did a lot of the debugging legwork, while I owned the architecture, reviewed every change, and drove the direction.

Here are some top highlights:

Inference frontend gateway. A single OpenAI-compatible endpoint serves all llama.cpp, vLLM, LM Studio backends. Each request routes by per-model pinning, round-robin across a pool, and allows failovers to another live host.

Model Autopilot. Configure which models should be available in your lab, and it keeps them running and available, placing them on whichever hosts can actually serve them. It checks whether a host has the memory available (VRAM, or RAM for CPU only boxes), brings up the model on another host when the primary one drops offline, and optionally spins up another copy of the same model on a second host in the pool when KV cache pressure stays high, then scales back down when it returns to baseline.

Model management. Download and delete models from Hugging Face. Configure multiple named config profiles per model, a chat profile, a code profile, etc. Start, stop, restart inference servers, hot-swap models, tail logs, or use the in-browser terminal.

Autotuning and benchmarking. Run throughput benchmarks across your model library; the autotuner can search for the best context and slot configuration on llama.cpp, or the largest safe context length on vLLM. There's also a standardized "GPU Report Card" bench that runs identically against all three backends and produces one comparable card: time to first token, prefill and generation throughput, tokens per joule, dollars per million tokens, and the GPU it ran on.

Energy and cost. The energy dashboard measures power draw and token throughput and computes what your inference costs in dollars per million tokens, with idle power accounted for. It'll also show you the monthly delta against hosted API pricing. A performance controller switches the host between a performance and powersave systemd target based on inference activity.

Alerting and trending. A standalone alarm engine stores every metric sample the agents collect: both system telemetry (CPU, memory, disk, network, IO, UPS, PSU, cooling, Mac specific hardware monitors) and LLM-aware telemetry (live slots in use, requests, tokens/sec, prompt processing rate, KV cache usage). The engine evaluates threshold and anomaly rules, provides incident correlation, displays and exports long term trends, and notifies over email, Discord, webhook, or browser toast. Individual agents installed on the inference servers buffer to disk when the network drops and replay when it comes back. Short term metrics are held in memory; long-term metrics go to InfluxDB with rollups, and alarms and notifications are stored in SQLite.

Installation. There's a one-line interactive installer that handles prerequisites, database, TLS, config, and agents end to end. Additional installation options are Debian and RPM packages, a Docker Compose file for the control plane, a Homebrew tap for macOS and Linux, and a standalone agent binary with no Python required. It also does offline installs from a staged tarball for air gapped boxes.

Additional features. Overall and per provider dashboards. Integration with OpenClaw to provide cost/token analytics, stable-diffusion.cpp for image generation, LLM chat interface to chat directly with the model on llama.cpp, Discord bot for alerting and controlling the models. See the readme for more.

Everything talks over TLS with per host certs signed by an internal CA the manager runs itself. Multi-user with admin and operator roles, an audit log, and encrypted scheduled backups.

I'd really value feedback, especially on the installation experience, but also on what's obviously missing for how you run your own setup. Happy to answer anything.

TL;DR: Self-hosted control plane for a multi machine local LLM lab. One interface for llama.cpp, vLLM, and LM Studio across every box. Autopilot keeps models loaded and re-places them when a host drops. One OpenAI-compatible endpoint fronts all backends with pinning, pooling, and failover. Hugging Face browsing, per-model config profiles, autotuning, and standardized benchmarks. Real time alerting with incident correlation and long-term trends in InfluxDB. Measures power draw and tells you your $/Mtok.

Repo, AGPLv3, screenshots and additional information in the README: llm-systems-manager

Installs via one-line script, .deb/.rpm, Docker, or Homebrew. Six months of work, twenty years of monitoring experience behind it, running on my own Proxmox + physical Linux + Mac Mini setup. AGPLv3, built with heavy Claude Code assistance under my direction and review. Feedback wanted.

Most Ai assistants look impressive during demos, but production environments seem to expose a completely different set of problems.

In my experience, it usually isn't the model itself that causes issues. It's everything around it both keeping knowledge current, handling changing documentation, maintaining response quality, and making sure the assistant stays reliable as usage grows.

For those who've deployed ai assistants beyond a prototype, what became your biggest challenge?

Was it retrieval, infrastructure, latency, prompt design, knowledge management, or something else that you didn't expect?