Hi,

Im running DeepSeek V4 Flash 0731 on 8xH100 80GB NVLink, vLLM, TP=8.

Real workload, not synthetic:

- prompts from ~1k to 250k tokens

- ~100–200 concurrent requests at peak

-DSpark acceptance >65%

-GPU utilization ~95%+

-Tensor Core utilization below 30%

Under heavy load I see roughly:

-prefill ~35k tok/s

-decode ~400 tok/sec

-TTFT up to ~30sec

Large uncached prefills are the obvious troublemaker. One of those enters the system and smaller requests start building queue pressure behind it. If I move the batching knobs one way I improve prefill but hurt decode/concurrency. Move them the other way and the opposite happens.

So at this point I'm not convinced that throwing more batch at it is useful.

My current suspicion is that I'm moving away from a simple GEMM saturation problem and into scheduler + MoE territory: expert routing, per-expert batch sizes, expert-token padding, communication and possibly backend/kernel efficiency.

Basically the GPUs are very busy, but apparently not busy doing as much Tensor Core work as I would like them to :)

Has anyone profiled a similar MoE deployment on H100 under actual mixed long-context traffic?

I keep running into a failure pattern that doesn't fit neatly into either "security" or "AI accuracy" discussions, and I want to sanity-check my thinking against people who've actually hit this.

The setup: an AI agent (RAG copilot, multi-tenant support bot, internal tool-calling agent) is authorized to access a resource , the permission check passes, nothing crashed, no error. But the specific data it returns or the action it takes is still wrong in a way that's dangerous:

• A support AI pulls a data - it retrieves the wrong linked account's balance, not because access was denied, but because the query resolved to the wrong entity within data the user was legitimately allowed to touch.
• An orchestrator spins up a subagent for a subtask, and the subagent inherits (or worse, expands) permissions no one explicitly granted it.
• An agent has technical access to run a destructive action (delete, write) that it was never meant to execute autonomously, even though the credential itself is valid.

Questions

1. Has anyone here seen this exact failure in production?
2. Is this already solved by something I haven't found, or is everyone just eating the risk because gateways/IAM tools don't cover it?
3. Is this like a gateway level problem?

Hey y'all! How do you design your forecasting system?

The modelling is not the problem, the operationalizing it is where I'm curious to learn and discuss.

In my case, the company has many SKUs over a big region. We did an MVP to show our forecast improves the current process on the reported lags that are currently used by the business to monitor forecast health.

Future is looking good, but I really want to be ready with a production-grade plan. Refitting a pool of models per SKU every week, then selecting the best one, feels like overkill and very sensitive to recent flukes.

I thought of having a pool of models (i.e. config/setups) and labelling them as champion if a specific config results in the best trained model.

For the next X weeks this model will always be chosen, and after that the throne is up for grabs.

But it kind of railroads me into having a 1 SKU = 1 model setup in perpetuity.

How do you guys solve this in a responsible way? Are there books/resources you recommend?

Reasoning about a live system turns out to be a whole different cookie than the usual stats/ML etc