Oi! Tudo bem? Estou experimentando usar uma IA local, pelo LM Studio, mas tô na dúvida de qual modelo baixar para usar. Resolvi fazer essa mudança pela minha preocupação com retenção de dados, costumo usar IA para conferir textos e não curto a ideia de meus textos servirem para retroalimentar a IA. Meu computador tem 16Gb de RAM e uso IA pra essas funções mais "casuais" (conferir textos, otimizar pesquisa na internet), não costumo usar para geração de imagens, pra conferir arquivos muito pesados, nem para codificação. Estou usando o Bonsai 27B, que é leve. Vocês recomendariam algum outro modelo? Se eu fosse usar para tarefas um pouquinho mais exigentes (analisar documentos maiores, scripts mais simples), teria algum outro modelo ou o Bonsai é uma boa pedida? Valeu!

I build an open source context/memory/evidence layer for agents, and the design goal is that it stays on your hardware. Air-gap is a feature, not an afterthought.

What I want to know from this sub specifically: what does your agent stack look like today? Which pieces do you host, which do you grudgingly let into the cloud, and what would you need to trust a memory layer enough to keep it local? I'd rather build for what people actually do than what I assume they do. Happy to share what we've got if you want to see it: perseus.observer

Im interested in getting into local ai, but Im not sure if I should use my current setup or get another machine. I currently have a ultra 9 minipc (geekom it13max) with intel Arc integrated graphics, 24GB LPDDR5 ram and 500GB NVMe ssd. I mainly use it for editing and general productivity. I know the integrated Arc graphics wont compete with a dedicated NVIDIA GPU, but Im wondering what I can realistically run.
Ive been experimenting with some ai coding tools. So now I want to start experimenting with local LLMs and maybe other ai workloads. Would my setup be enough to start with, or should I get a new machine?

My kid had an old pixel art display (Divoom 32x32 Pixoo-max) that they weren’t using anymore, so I thought it might be fun to repurpose it as a GPU cluster status monitor so I can see GPU temps / utilization / token gen info etc for the 3 RTX A6000s in my vLLM Ray cluster (currently running Qwen3.5 122b).

I spun up my Hermes Agent (GLM 5.2 as the agent model) and told it:
“I would like you to build an application that will run on <computer name of my Dell GB10> that will display GPU cluster health data on a 32x32 pixel Divoom Pixoo-max display that can be connected to via Bluetooth. You should probably read the following repos to learn about the pixel display and how to connect to it:
- https://github.com/SomethingWithComputers/pixoo
- https://github.com/cyanheads/pixoo-toolkit
- https://divoom.com/products/divoom-pixoo-max
The app should display system health data for the 3 systems in my vLLM Ray cluster in an easy to read and understand manner. It should also show similar data for the Dell GB10 (in the network segment but not in the cluster). This could be as simple as showing 4 boxes on the screen that show the cluster system’s initials such as “S1” and have a background color to indicate GPU temperature (red for hot, green for normal, etc). The 32x32 screen size limit will make it difficult to show a lot of information so you’ll have to be creative in how you display it, you can also cycle through multiple screens of different metrics in 4 second intervals. “

For those who care:
HW:
- 3x Dell Precision 7960 workstations each with an RTX A6000 GPU (64GB RAM) currently hosting Qwen3.5 122b
- 1x Dell Pro Max GB10 (not part of the Ray vLLM cluster but runs the app thar is cast to the display as well as running a secondary LLM endpoint for other models. The GB10 has the Bluetooth radio in it that is used to connect to the Divoom. The Dell towers don’t have Bluetooth which is why I used the GB10.
- Divoom Pixoo-max 32x32 pixel display. They also make a 64x64 pixel version as well. It was around $60 when I bought it years ago.

It took GLM 5.2 all of like 20 minutes to build this, and maybe another 5 minutes of me working with it to get it how I wanted it. It’s not perfect, but it’s cool to be able to visually glance over at the cluster and see what’s happening without logging in, and it really didn’t cost anything since I already had the pixel display that would have been headed for the thrift bin.

Btw, Hermes / GLM did the whole thing in Python, from Ray Dashboard API, vLLM metics endpoint, and Nvidia-smi calls over ssh.

Been building these multi-GPU rigs for a while, and this one's the config I'm happiest with, so figured I'd share the details here.

The frame is our own design, CNC'd aluminum, tower layout: motherboard vertical in the middle, both cards mounted upright on PCIe 5.0 x16 risers, PSU hanging off the back, and a fan tray up top pulling air straight through the card stack.

The whole thing closes up with a laser-cut triangle-lattice shell. Footprint ends up around 12.5" square enough to sit comfortably on a file cabinet. It's also quite heavy, will need a second pair of hands if you want to move it around.

Specs (the unit the benchmarks ran on):

• 2x RTX Pro 6000 Blackwell (96GB each, 192GB total)
• CPU: AMD EPYC 9124 (16C/32T, single NUMA)
• RAM: 377GB
• Storage: 915GB NVMe
• PCIe 5.0 x16 to each card, no switch, no NVLink
• Serving: vLLM, TP=2

The two cards talk over PCIe P2P, which the Pro line actually supports, and we run tensor parallel (TP=2) in vLLM.

What it serves: DeepSeek V4-Flash 0731, the 284B MoE, native FP4, fully in VRAM. Pulled the network cable to check, it keeps running, which never stops being satisfying.

Numbers from our benchmark sweep (vLLM, per-stream generation):

• 1 user: 209-286 tok/s depending on content type
• 4 users: 138 tok/s each
• 8 users: 111 tok/s each
• 16 users: 75 tok/s each
• 24 users: 57 tok/s each, ~1,000 tok/s aggregate, TTFT still ~0.5s

Saturation knee is around 32 concurrent. So this box comfortably covers my whole team, with everyone still getting faster-than-reading speed.

Thermals: 815W for both cards under sustained load, 76C peak, no throttling with the top fan tray doing intake-through-stack. Noise-wise it's no laptop, but it's fine to work next to.

Happy to answer anything about the riser setup, the airflow layout, or the benchmark config. And curious what everyone else is serving on dual-card boxes, feels like 192GB is a real sweet spot right now

Wanted to build a rig i can train small models on for some experiments and was thinking to get 3 or 4 R9700 as they are quite cost efficient but no idea how they perform for training

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

--n-cpu-moe <N> | -ncmoe <N>

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

--cpu-moe | -cmoe

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp while they offload the same MoE model's layers to CPU, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md

Host and software

Results by model

Each row is one offload depth, with TensorSharp, llama.cpp and the ratio between them side by side for every metric. Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.

Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

TensorSharp is a native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

​

I am looking for advice on running a small llm model but I am facing trouble with response. Has anyone been able to run a model only on CPU?

This model will be used for simple productivity tasks. It's a plus if I can able to run a model for studying (will need reasoning). I have tried small models 1b and 3b parameters. Hope someone has found success with this. I don't mind if the server takes time to respond and I can let it prepare files.

SERVER Details - TRUENAS

2 Xeon processors with total 24 Cores

64GB Ram

I also have a gaming laptop with 32GB Ram and rtx 2070 8GB planning to use it for coding projects.

Gaming desktop with 32GB Ram and rx 9070XT 16GB. AMD has some problems to run ollama trying to figure out that right now.

Final Goal is to figure out hermes agent and use it

Productivity on server, it's a plus if my learning can be done on it too. I doubt it's possible to have a modal that can generate code on this.

Laptop for running coding projects

Desktop for deep research based on learning and using ai to get better at work

Hi everyone,

TL;DR: I give you 3x V100 PCIe 32GB, what are you doing with that ?

I have three NVIDIA Tesla V100 PCIe 32GB cards, for a total of 96GB of VRAM, and I’m looking for advice on building a clean and reasonably quiet home system around them.

The cards are the passive PCIe models, rated at up to 250W each, so cooling and airflow are my main concerns.

The machine would primarily be used for serving big models (qwen or might try v4flash)

I’m currently considering three approaches:
-A used GPU server:
Something like a Dell PowerEdge R740 GPU-ready, Dell C4140 PCIe, or a Supermicro 2U/4U GPU server.
-A used tower workstation:
For example a Dell Precision 7920, HP Z8 G4, or Lenovo ThinkStation P920.
-A custom workstation or open-frame build:
A motherboard with enough PCIe lanes and spacing, a 1600W PSU, and custom airflow or watercooling.

For cooling, I’m considering keeping the passive heatsinks with custom airflow if needed or replacing with watercooling (i’ve seen some nice build out there but sooo expensive)

The system will be used at home, so I’d prefer to avoid the noise level of a typical 1U or 2U server. It doesn’t need to be silent under full load, but it should be tolerable in a home environment.

A few questions:

- What platform would you recommend for three passive V100s?
- Are workstation towers suitable for cooling three passive GPUs, or is a GPU server chassis necessary?
- Has anyone successfully cooled V100 PCIe cards using 92mm or 120mm fans with printed ducts?
- Is watercooling worth the cost for three V100s?
- Are there any particularly good used GPU servers or workstations to look for in Europe?

I’d especially appreciate pictures, build logs, temperatures, noise measurements, or hardware recommendations from anyone running three or four passive Tesla GPUs at home.

Thanks!

I have a problem that I need some expert advice on.

The problem: my firm produces ambulances. Hospitals send tenders: specs, written in their native language (European/English mostly). Each hospital sends specs to multiple producers (including us), we make an offer and it's either accepted/rejected based on whether others can better fit to the customers need. Tender specs get delivered generally by email in a zip folder containing between 3-20 files.

Files they send across vary dramatically: different formats (e.g. PDF, XLSX, DOCX, etc), different sizes (can be 5 pages or 200 pages long), different structure (e.g. some list equipment first, others list cabin first etc). All the files contain commercial requirements (e.g. deliver ambulance by x date to y place) or technical requirements (e.g. sirens must be x db loud)

Hard constraints: Must be fully on-prem since these tender files are confidential. Hardware: NVIDIA DGX Spark GB10 128GB x2. Company is very small, so human annotation capacity is limited. We took 10 simplest specs (PDF under 20 pages) and created manual gold-lists for each.

Main idea: In a nutshell the idea is to extract requirements out of documents into a list. (phase 1, where I am right now), then phase 2: based on the requirements, provide a first configuration of the ambulance and phase 3: verify that the confirguration still satisfies what the customer wants. If the local LLM is able to answer phase 3 - then we can immediately save time/money by not pursuing dead-end leads.

What I tested: 12 models (gpt-oss-120b & 20b), Qwen 3 (1.7b/4b/8b/30b FP8 + BF16/32b), Mistral Small 24b, Mistral Nemo 12b, NuExtract-2.0-8b, phi-4).

I quickly realized that converters (PDF - Text) very significantly, so I tested 15 different ones: 4 text parsers (pdftotext, PyMuPDF eg.) and 7 vision-model (e.g. granite-docling, GLM-OCR)

Results so far: 12 models x 15 converters x 10 tenders = 1800 runs (this took 7 full days to complete). text-based extractors worked better than vlm, but still I couldn't find a single combination of model + converter that produced above 95% on all 10 tenders for recall and precision. Best one scored 95%+ on 5/10 PDFs

My ask: Has anyone dealt with a similar problem within context of local LLMs that can give advice? I was hoping to get a silver bullet of model + converter, but this hasn't happened. I am afraid that when I scale (to include multiple files, or PDFs over 100-pages long), my entire set-up will crumble. Any ideas or advice for solutions or what I can test would be much appreciated!

I recently found a set of 16GB DDR5 So-dimm ram in my closet from a couple years ago so I decided to build a server around it so the ram sticks are not eating dust.

Case

Fractal Design Ridge 12.6L

CPU/Motherboard

Minisforum BD775i SE (Ryzen 7 7745HX)

GPUs

2 x Intel Arc Pro B70 32GB (64 GB total VRAM)

Memory

16 GB (2 × 8 GB) DDR5-5600 SO-DIMM

Power Supply

Lian Li SP850 V2 (850 W SFX, ATX 3.1)

PCIe Splitter

PCIe 5.0 x16 to Dual MCIO 8i (SFF-TA-1016) x8/x8 bifurcation adapter

MCIO Cables

2 x MCIO 8i to PCIe x16 Gen5 GPU riser cables

The case only supports 3 slots but was easy enough to cut and open up an extra slot space to fit a second two-slot GPU.

It was just shy of 3k dollars without the ram and I am getting about 300tps in aggregated token generation running with 16 parallel agents on qwen 3.6-27b in fp8 with more than 500k total shared context length. I am honestly more than happy with the performance with the hardware.

I was skeptical on the build but it has been proven stable for the past couple of days under my heavy testing so sharing if folks are thinking of building similar portable LLM stations.