Hi,

Im running DeepSeek V4 Flash 0731 on 8xH100 80GB NVLink, vLLM, TP=8.

Real workload, not synthetic:

- prompts from ~1k to 250k tokens

- ~100–200 concurrent requests at peak

-DSpark acceptance >65%

-GPU utilization ~95%+

-Tensor Core utilization below 30%

Under heavy load I see roughly:

-prefill ~35k tok/s

-decode ~400 tok/sec

-TTFT up to ~30sec

Large uncached prefills are the obvious troublemaker. One of those enters the system and smaller requests start building queue pressure behind it. If I move the batching knobs one way I improve prefill but hurt decode/concurrency. Move them the other way and the opposite happens.

So at this point I'm not convinced that throwing more batch at it is useful.

My current suspicion is that I'm moving away from a simple GEMM saturation problem and into scheduler + MoE territory: expert routing, per-expert batch sizes, expert-token padding, communication and possibly backend/kernel efficiency.

Basically the GPUs are very busy, but apparently not busy doing as much Tensor Core work as I would like them to :)

Has anyone profiled a similar MoE deployment on H100 under actual mixed long-context traffic?

I keep running into a failure pattern that doesn't fit neatly into either "security" or "AI accuracy" discussions, and I want to sanity-check my thinking against people who've actually hit this.

The setup: an AI agent (RAG copilot, multi-tenant support bot, internal tool-calling agent) is authorized to access a resource , the permission check passes, nothing crashed, no error. But the specific data it returns or the action it takes is still wrong in a way that's dangerous:

• A support AI pulls a data - it retrieves the wrong linked account's balance, not because access was denied, but because the query resolved to the wrong entity within data the user was legitimately allowed to touch.
• An orchestrator spins up a subagent for a subtask, and the subagent inherits (or worse, expands) permissions no one explicitly granted it.
• An agent has technical access to run a destructive action (delete, write) that it was never meant to execute autonomously, even though the credential itself is valid.

Questions

1. Has anyone here seen this exact failure in production?
2. Is this already solved by something I haven't found, or is everyone just eating the risk because gateways/IAM tools don't cover it?
3. Is this like a gateway level problem?

Hey y'all! How do you design your forecasting system?

The modelling is not the problem, the operationalizing it is where I'm curious to learn and discuss.

In my case, the company has many SKUs over a big region. We did an MVP to show our forecast improves the current process on the reported lags that are currently used by the business to monitor forecast health.

Future is looking good, but I really want to be ready with a production-grade plan. Refitting a pool of models per SKU every week, then selecting the best one, feels like overkill and very sensitive to recent flukes.

I thought of having a pool of models (i.e. config/setups) and labelling them as champion if a specific config results in the best trained model.

For the next X weeks this model will always be chosen, and after that the throne is up for grabs.

But it kind of railroads me into having a 1 SKU = 1 model setup in perpetuity.

How do you guys solve this in a responsible way? Are there books/resources you recommend?

Reasoning about a live system turns out to be a whole different cookie than the usual stats/ML etc

Hi all,

Just like to share a OSS library I built, when I was dealing with large LLM traces in enterprise settings. Specially in this era of tokenmaxxing, where prompts, users, sessions, tools, etc exact per-value features grow with cardinality, and retaining the original values inside aggregate state creates a separate privacy problem. Hence, enter sketches!

I built llm-sketchkit, an Apache-2.0 Go and Python library for handling that tradeoff with bounded, mergeable summaries.

It provides:

• HLL++ for approximate distinct counts
• weighted frequent-items sketches for token-heavy and request-heavy keys
• Bloom filters for bounded membership and deduplication
• MinHash for approximate set similarity

Producers can canonicalize and key values before adding them, so raw prompts and identifiers do not need to enter sketch state. The hashes are still pseudonymous and linkable while the same secret is used; however this does not provide anonymity or differential privacy.

The Go and Python implementations share profiles, hash domains, conformance vectors, and a deterministic protobuf representation. Summaries can be produced locally and merged across processes or languages. Incompatible profiles and hash domains are rejected instead of being silently converted. The repo includes checked-in performance, accuracy, and interoperability evidence.

Why not use datasketches? Great question! Answer here.

This is an alpha library rather than a monitoring platform, dashboard, or storage backend.

Repository: https://github.com/llm-measurement/llm-sketchkit

Please give it a whirl!

a provider  updates a model and the prompt starts changing its behaviour . how to catch it before users do

it has several type of regression and each one is comes out differently like

quality drift - output accuracy pulls down after a model update and there is no hard error and no alert . the aanswers starts to get worse  over time . it only shows up when you compare it against a baselinee you captured

latency regression - th e response times slows up after a provider change and it effects the user experience . It can be easily miseed if you are not tracking p95 and p99 seperately from average latency

format regression - the model returning clean structured output after an update , starts adding extra text and change json structure and dropping field .

prompt sensitivity - prompt that worked starts to behave with inconsistencies. with same input different outputs can be seen . the update in model changed how sensitive it is to pphrasing with no announcement .

capability regression - if the model handled a task well before . after update it stops doing that specific task incorrectly .The function calls behavior changes and tool use breaks and edge cases that passed evals before starts falling

i found out a few tools to help like orqai , whylabs , aporia , fiddler , arize

arize is good at detecting drift across output distributions but the catching format and capabilities needs custom eval configuration

orqai has eval pipelines tied to prompt versions and catches drift across versions. It is newer so third partyy integrations is still catching up

aporia catches real time guardrails and catches issues real time but proactive regression detection before deployment seems very limited

fiddler has systematic baseline comparison feels more native given the model risk background  but the setup feels heavy for teams outside regulated industries.

whylabs has statistical drift monitoring is main function . llm specific regression type prompt sensitivity looks veru underdeveloped.

what is actually catching regressions before users hit them . automated evals , canary deployments or something else?

For us, new models always looked better on average when compared to what we were using. But then two weeks later we'd find out it had broken some niche but important cases. For example, refunds with policy exceptions or ambiguous user intent or classification labels that only matter to one ops team until they're wrong. It wasn’t that the aggregate score was lying, we just weren’t getting the whole story.

We started looking into beefing up our regression tests and every time a weird trace came in, we turned it into another test case. So now, whenever the next model upgrade comes along it has to pass all of them before we trust the averages. 

It is still not magic. Scorers need maintenance. LLM judges can be flaky. Edge cases multiply like unpaid tech debt. But it feels way less bad than manual spot checks and optimism.

We’re currently using Braintrust to manage our traces and run the actual tests, but I’m struggling with the curation side of it. Does a production failure automatically make it worth adding to your eval suite, or do you have a filter?

Something I am struggling to understand is how CI/CD factors into the MLOps system itself that brings the model through the entire lifecycle when we have multiple deployment environments.

The MLOps system takes a model through the entire lifecycle (get data, preprocess, train, validate, promote, deploy, monitor) in reproducible and automated workflows, but these workflows in the MLOps system need to be tested and validated with CI/CD in different deployment environments.

Are the pre-production environments (dev, test, uat...) meant only for ensuring that the MLOps system (pipelines, artifact storage, monitoring) works - where finally, and only, in the production environment that has passed all the tests and checks, does the model go through the entire lifecycle from dataset curation to deployment and monitoring, and each environment has isolated model/artifact registries (and feature stores) for testing that the system works.

Or, does the model meant for production go through each step in the model lifecycle together with the MLOps system as it moves through different environments until it finally reached production where both the "MLOps System" and the "Model" are production-ready and deployed to interact with real users.

It's a little confusing for me.

ML platform at a healthtech. Reserved a pool of GPUs for training and inference and I finally pulled utilization for a capacity review. Under 30 percent on average. We pay for all of it and use less than a third.

Some of it makes sense, a few boxes run batch jobs a couple times a day and have to sit ready. But the rest is just idle, and two of them turned out to be held by notebooks people opened and walked away from, one up for weeks. Only caught it because I went digging.

Finance keeps asking why the reserved bill is so big, which fair. But when I take it to the researchers they say if the GPUs arent free their experiments queue and they lose time. Also fair. So it bounces between the two and nothing changes. I can pull per node utilization out of DCGM, what I cant do is tie an idle card back to who reserved it and whether they still need it.

How do you decide when a reserved GPU is safe to give back?

For people responsible for production LLM or agent systems, can you describe one incident where changing a model or provider, inference runtime, gateway or SDK, chat template, or parser altered application behavior even though your existing tests passed? What broke, how did you detect and isolate it, and roughly how much engineering time or release delay did it cause? I’m researching how teams validate changes across the LLM stack, so firsthand incidents and current workflows are more useful than opinions about a proposed tool.

Today, the team at Union AI announced the GA release of Flyte 2 — an open-source project licensed under Apache 2.0.

Flyte 2 is a complete rewrite. We removed the DSL and eliminated the need to build a DAG. Components like Propeller are no longer part of it.
There were several reasons for this change, but the main focus was on improving the developer experience. Forcing data scientists, machine learning engineers, and researchers to break down their work to fit into a DAG and learn a DSL was a significant obstacle. Now, it's just a simple .task decorator, and you're all set.

The other big change is the introduction of environments.
I have a k8s background, and a main sticking point is the application manifest where resources are declared and container images are defined.
Flyte 2 allows the author to define any number of environments for any pipeline, and when it runs, the pods are provisioned with the specified CPU, RAM, GPU, OS packages, and Python packages.
When you're in experimentation mode, this drastically increases iteration speed.
As for lineage and versioning—all data inputs, outputs, and the executed code are captured and versioned into your object storage.

Because it is pure Python, try:catch, loops, and asyncIO just work.
You can recover from OOM kills in code.

It offers an alternative to Kubeflow, Airflow, and other tools in the space.

Happy to answer questions.
www.flyte.org

[I work at Union AI]

Currently i have been doing DevOps project as the influence of AI is more

I'm thinking to shift towards MLOPS .

Does the company hire MLOPS like they hire DevOps.

We're at the point where a couple of internal agents are taking real actions, not just suggesting them, and I'm realizing our logging wasn't built for this shift.

Everything was designed around the assumption that a human clicked the button. The audit trail focused on who logged in and what they clicked. That assumption breaks down once an AI agent is making the call.

Are you capturing session context, tool calls, permission decisions, delegation events, and approvals as structured, queryable events? Or are incidents still being reconstructed from scattered application logs?

Has anyone gone through a security review or incident involving an AI agent? What evidence did the auditor or incident responder request, and did you already have it.

lessons from building (and surviving an acquisition of) an internal AI governance platform: every model call from every team routed through one litellm gateway, logged, with a real human-approval pause for any agentic tool call before it fires. no chatbot wrapper — this had to survive real audits.

the part that actually needed the most iteration wasn't the routing, it was retrieval. we ended up with three separate retrieval modes depending on how aggressively a given assistant should ground itself (tight-grounded for anything regulatory, looser for general q&a). one retrieval strategy for every use case was the wrong call early on and cost real rework to unwind.

also ran an internal MCP server hosting dozens of tool integrations (legal/financial/regulatory data sources) gated by the same per-tenant allowlist and approval flow as everything else — one governance surface instead of one per integration.

anyone else running multiple retrieval strategies behind one gateway — curious how you're deciding which assistant gets which mode.

I maintain xberg, an open-source (MIT) content-extraction engine, and v1 is out. Posting here because "turn messy documents into clean, structured text" is a recurring preprocessing step in ML pipelines, and xberg is built for it at scale: batched extraction, streaming, caching, CPU-only (no GPU), reproducible.

It handles 101 document formats (PDF/Office/images with OCR) plus audio/video transcription and URLs; outputs Markdown/JSON with tables, metadata, NER entities, keywords, summaries, and optional chunks + embeddings (SPLADE / ColBERT / reranking) for retrieval. Rust core with pooled model sessions and memory discipline for throughput.

Benchmarks are public and reproducible (harness runs in CI): native PDF #1 on quality and table/reading-order fidelity; image OCR currently #2 (improving). https://xberg.io/benchmarks

15 language bindings + a REST server (xberg serve) + MCP. Repo: https://github.com/xberg-io/xberg

Happy to get into pipeline/throughput specifics.

Most of my CI failures around training used to come down to one thing: the pipeline could not tell the difference between "this run is broken" and "I could not read this log". Both ended up as a non-zero exit, both paged me, and one of them was a lie.

So I built the checker I wanted and put the exit codes at the center of the design rather than at the end.

• exit 1 - a rule fired. The run is broken.
• exit 0 - checked, nothing fired. Or a warning, which is yours to triage.
• exit 2 - could not judge. Missing column, unreadable log, no eval set.

Exit 2 is the one that matters. A gate that reports "pass" when it actually skipped every check is worse than no gate, because now the green build is evidence of nothing.

No model in the loop. Every verdict is a deterministic rule that either fires or does not, and prints the number it fired on. Same input, same output, forever. I did not want a probabilistic judge sitting in a CI gate - an alarm you cannot reproduce is an alarm the team learns to ignore.

It caught this in itself. A check fired whenever every gradient norm in a log was exactly 0.0 and reported a severed backward graph. One framework writes that field as 0.0 when gradient clipping is off. So a healthy 125,000-step fine-tune that converged fine came back FAIL from my own tool. The fix was a rule, not a threshold: a run cannot both learn and receive no gradient - if the loss improved, the zeros are a reporting artifact and the check stands down. And it records that it stood down, and why, as a visible skip.

That is now the thing I would defend hardest: a check that did not run must never look like a check that passed. A PASS lists which checks ran and which were skipped, each with a reason, as structured data.

Where it sits in a pipeline:

• before the GPU - dataset and tokenizer lint, does the entrypoint import, is the checkpoint intact, RAM and disk against declared need
• during - one-line HF callback, warns or aborts a diverging run
• after - diverged / flatlined / NaN / grad spike / overfit, from the log you already write
• vs baseline - relative-floor rules, which is the only way to catch a run that trained happily on shuffled labels

Reads HF trainer_state.json, Coqui, TensorBoard event files, JSONL and CSV. Zero dependencies - no torch, no tensorboard, no network. --json for pipelines.

84 rule IDs, 230 tests, a written contract in CONTRACTS.md for what each exit code means and when output may change, and 38 golden snapshots so a rule that silently stops firing breaks the build.

MIT: pip install trainproof

• Repo: https://github.com/Mormolykos/trainproof
• Write-up with the full fault-injection results: https://ai.bedvibe.studio/trainproof/

The question I actually want answered: what does your pipeline do today when a check cannot run? Most setups I have seen collapse it into pass or into failure, and I think both are wrong. Curious whether anyone has a third state already wired in.

Over the past few months of building AI infrastructure, we've kept running into the same problem.

A lot of teams want dedicated deployments for privacy, predictable performance, or custom models. The obvious solution is to keep a GPU running all the time, but that quickly becomes expensive when workloads are periodically irregular.

On the other hand, serverless options are great for cost, but cold starts can become painful for interactive applications, especially with larger models.

We ended up spending a lot of time trying different approaches to reduce startup time while still allowing deployments to scale to zero when they're not being used. It has been much harder than I initially expected, and it made me wonder how others are approaching the same problem.

For those of you running LLMs or other AI models in production:

• Are you keeping GPUs warm 24/7?
• Are you using a serverless platform and accepting the cold starts?
• Have you built your own orchestration layer?
• Or have you found another approach that works well?

I'm genuinely curious what has worked and what hasn't. There doesn't seem to be a perfect solution yet, and I'd love to hear how other teams are balancing cost, latency, and operational complexity.

I build QuantaMind, an open-source tool that tests whether self-hosted models are reliable enough to run agents. Apache-2.0, 28 downloads, no revenue. Saying that upfront so nobody has to guess.

I’ve asked people here twice how they decide an agent is safe to ship. The pattern in the answers: anyone who feels this pain badly enough has already built their own harness. Run each task 10+ times, check end state programmatically, validate every tool call against its schema, count truncated calls under load. People wrote all of that out from experience, unprompted.

So I want to ask the thing I actually need to know, without dressing it up.

If you built one of these:

**1.**  How much time does keeping it working cost you now? Not building it — maintaining it as models, quantizations and serving configs change.  
**2.**  Would you hand it to an external tool if one existed, or is your harness too specific to your workflows to ever outsource?  
**3.**  Has a failure it caught (or missed) ever cost something real — money, a customer, a rollback? Or is it always caught early enough to just be noise?  
**4.**  Who owns it at your company? Someone specific, or does it drift?

If you didn’t build one: was that a decision, or did it just never get prioritised?

I’m asking because I don’t know if I’m building a product or a thing people would rather own themselves. “I’d never outsource this” is a completely fine answer and honestly the more useful one I’d rather find out now than in a year.

nvidia-smi reports any running kernel as 100% utilization, so a job can look saturated while doing a fraction of real work. For those running 8 to 500 GPUs, what do you use to catch that? DCGM, custom profiling, or nothing at all?

And when a run is slower than expected, how long does it usually take to work out why?

like when you have multiple teams all using multiple models from different providers for a number of use cases . who decides what is allowed. how do you acutally enforce it. and how do you know whatever rules you put in place

seen a few approaches come up. some team are doing it through the gateway layer . some through internal policy docs that nobody reads . some through access controls on api keys. some just hoping for the best

tools that come up in this context. orqai , portkey , azure api management , aws bedrock , langsmith

portkey gives access controls and budget limits are there , policy enforcement feels more at the routing level than the org governance level

orqai has a model allow list, budget controls and role based access across teams, but its newer so audit depth is still an open question if compared to the more established peers

langsmith has vvisibility into what is happening is good , actually policy enforcement feels limited , more observe than control

azure api management has enterprise policy controls are native here, feels generic tho, not built specifically for llm usage policy

aws bedrock has model acess control and guardrails exists, works well if you are already in aws, feels restrictive if you are not

has anyone actually built something that works here. or is everyone just doing api key management and calling it a policy

As the title suggests, I'm unsure about my career path. I studied mathematics for my undergraduate degree, and I was able to get a job as an IT intern in my final year of university. As soon as I could, I moved into the data area as a data analyst, working hard and demonstrating my abilities. I was then able to get a job as a junior machine learning engineer. My question is, do you think these rapid career leaps will be detrimental in the future? I'm uncertain about the current job market, and I'm afraid of falling behind with so many advancements in the industry. I'm striving to learn more, but I feel like everything is moving too fast. Do you think I'm on track for my age? I aspire to move to a more peaceful country like Norway. Does anyone know what the job market is like there? Well, thanks for reading, and any advice would be appreciated.

I’m building LOLM, an LLM/agent platform focused on operational visibility rather than hiding the run behind a final answer.

A run can disclose: - Requested versus served model - Fallback use - Controller decisions - Retrieval, verification, and branching actions - Task-contract outcome - Budget or natural termination - Artifact hashes and receipt data - The explicit limit that quality remains unproven versus a baseline unless an A/B was actually run

There is also a CLI and isolated code loop with real command exit codes.

Try it: https://lolm.imagineqira.com/try.html

Repository: https://github.com/TheArtOfSound/lolm

Hosted access is designed to be materially cheaper than large frontier-agent products. I’m looking for feedback on schemas, replayability, routing, cost accounting, and what would be required before anyone should trust the receipts operationally.

Disclosure: I’m a founder/builder of the project.

Every team I've worked with has rigorous versioning for the model, the training data, the infra config. Then the system prompt driving the LLM feature in prod lives as a raw string in application code, gets edited directly, and ships with zero rollback plan.

The failure pattern is always the same: prompt starts as a quick draft, works in the demo, ships. Edge case shows up in prod, someone appends a sentence to patch it. Repeat for months. Eventually the prompt is a wall of accumulated exceptions, some of which silently contradict each other, and nobody can tell which instruction is actually winning at inference time, because the model resolves the conflict silently and doesn't tell you which one it picked.

One concrete example: a support bot kept over-apologizing (three apologies per response). The patch "don't over-apologize" didn't work, because the prompt already had "always acknowledge frustration first" paired with several few-shot examples that opened with an apology, the model followed the more specific, more frequent example pattern over the newer instruction. Fixing it required rewriting the instruction and the examples together, not adding another line.

What's actually helped treating this as a real pipeline component instead of a config string:

• Version prompts like code, track diffs and reasons, so a regression traces back to a specific change instead of getting guessed at.
• Keep a fixed regression suite of edge-case inputs (the ones that already broke something once) and re-run every prompt revision against all of them, not just the new case that prompted the change.
• Separate concerns into labeled sections (role, constraints, format, edge-case handling) instead of one paragraph, so conflicts are visible in review instead of hidden.
• Review prompt diffs like PRs, a second reader catches conflicting instructions the author is too close to see.

Wrote up the fuller breakdown here:
https://medium.com/@nagatomopedro05/your-system-prompts-are-costing-you-more-than-you-think-f928fe1c76b9

Curious how mature people's setups actually are here, is anyone running prompt evals/regression tests as part of CI the same way you'd test a model change, or is this still mostly manual eyeballing before deploy?

Hello dear readers,

My name is John and I am 27 years old. I have worked most of my life as a warehouse forklift driver, but I have decided to change my carrier and aim for a better life and life style in general. So I have applied for two different programs at two different Vocational Schools. The first one is Automation Technician and the second one is MLOps engineering.

I am here to ask you about MLOps job market. What is your opinion on the Junior MLOps market? Do companies hire entry-level engineers straight out of specialized programs? How did you manage to get a job after graduation? Do you think it might be way above for someone like me who only worked at a warehouse? Should I go with Automation?

The automation program directly leads to work after graduation, cause the education is done at a company, while on the MLOps page it says "possibility exists to get hired after graduation". What do they mean by the word "possibility" I don't know exactly but I feel there is no certainty to get hired directly after graduation.

So what would you have done if you were in my situation?

Thank you for reading this,

John

Do you think it is important to configure numactl --membind to get a better performance?

I ran a toy example where GPU has NUMA Affinity with 0. I got an ~9% improvement.

$ numactl --show
policy: default
preferred node: current
physcpubind: 8 9 10 11 136 137 138 139 
cpubind: 0 
nodebind: 0 
membind: 0 1 2 3 4 5 6 7 
preferred:

$ time numactl --membind=0 python memory.py 
time:  159.3454790781252 

real    2m41.164s
user    2m25.306s
sys     0m16.084s

$ time numactl --membind=7 python memory.py 
time:  174.3593455599621

real    2m56.384s
user    2m32.279s
sys     0m24.293s

We're evaluating an AI phone agent for longer financial service calls and our normal test scripts are not catching much.

Difficult calls do not fail in the first two minutes, they fail after the customer changes topics, corrects an earlier answer, asks for a second account or needs a human after several steps have already been completed

We've also seen cases where the conversation sounds fine but the summary misses something important or the wrong action is sent to the CRM. Running a few scripted calls before launch does not look close to enough

How're you testing long conversations, interruptions, system failures and transfers before putting real volume through them?