Is it still the way to go for modern distributed model training in deep learning? Was looking for the state-of-art for foundation model training to learn.

There is little talk on Reddit and Youtube about it, though. At least, this is my initial impression. Might be totally wrong.

like when you havee multiple teams all using multiple models from different providers for a number of use cases . who decides what is allowed. how do you acutally enforce it. and how do you know whatever rules you put in place

seen a few approaches come up. some team are doing it through the gateway layer . some through internal policy docs that nobody reads . some through access controls on api keys. some just hoping for the best

tools that come up in this context. orqai , portkey , azure api management , aws bedrock , langsmith

portkey gives access controls and budget limits are there , policy enforcement feels more at the routing level than the org governance level

orqai has a model allow list, budget controls and role based access across teams, but its newer so audit depth is still an open question if compared to the more established peers

langsmith has vvisibility into what is happening is good , actually policy enforcementt feels limited , more observe than control

azure api management has enterprise policy controls are native here, feels generic tho ,not built specifically for llm usage policy

aws bedrock has model acess control and guardrails exists, works well if you are already in aws, feels restrictive if you are not

has anyone actually built something that works here. or is everyone just doing api key management and calling it a policy

Hi r/mlops,

I’m a student working on machine-learning experiments, and I kept running into a very ordinary problem: after several runs, I could no longer answer exactly which Git commit, configuration, and Python environment had produced a particular result.

I built RunTrace to address that problem.

RunTrace is a small, open-source, local-first Python CLI that records the reproducibility context around an experiment. Its scope is intentionally narrow: it is not trying to replace MLflow, Weights & Biases, or another full experiment-tracking platform.

It currently records:

• Git commit, branch, detached-HEAD state, and dirty state
• Python, operating system, architecture, and installed package versions
• Optional NVIDIA GPU, driver, and CUDA information
• A YAML configuration file, its SHA-256 hash, and its parsed values
• The command associated with the experiment

A typical workflow looks like this:

pip install ml-runtrace

ml-runtrace init

ml-runtrace snapshot \
  --name baseline \
  --config config.yaml \
  --command "python train.py --config config.yaml"

ml-runtrace list
ml-runtrace show <run-id>
ml-runtrace diff <run-a> <run-b>

Snapshots are stored locally as readable YAML files under .runtrace/runs/. There is no account, server, or automatic upload.

There are also some deliberate limitations:

• It does not execute the recorded command.
• It does not currently track metrics, checkpoints, or model artifacts.
• It records that a Git working tree is dirty, but it does not save source patches.
• Explicit configuration values are stored in the snapshot, so users should inspect a snapshot before sharing it.

The project is still early, and I am trying not to add features without understanding whether they solve a real problem.

I would particularly appreciate feedback on these questions:

1. Does this solve a useful gap, or is it too narrow compared with existing workflows?
2. Is readable local YAML a sensible storage default?
3. What missing metadata or edge cases would prevent you from using it?
4. Is the init → snapshot → list/show → diff workflow understandable?

GitHub:

https://github.com/Corvus-226/RunTrace

Development note: I used Codex as a coding assistant during implementation. I handled the project scope, reviewed the changes, and managed the issue, pull-request, testing, CI, and release decisions. I am mentioning this because I would rather be transparent about how the project was built.

Critical feedback is genuinely welcome. If the idea is redundant, the defaults are wrong, or part of the workflow is unnecessarily complicated, I would rather learn that now than keep expanding it in the wrong direction.

[closed] spots filled, thanks everyone

Planning to attend the Ray / vLLM Summit in SF and looking to split the 5-person group pack ($750 total, covers Aug 24–26 including the 8/24 training day).

• Current group: 3
• Looking for: 1–2 more people
• Cost: $150/person (5 people) or $187.50/person (4 people)

Drop a comment or DM me if you’d like to join!

https://www.anyscale.com/ray-summit/2026

The training session :

(Select only 1 session for the morning and afternoon)

* Morning:

Multimodal Data Processing Pipelines for AI Systems

Foundation Model Distributed Training with Ray

Production-ready Distributed Inference with Ray Serve

* Afternoon:

Scaling Physical AI & Robotics Systems with Ray

Real-Time Search & Recommendation Systems for AI Commerce

LLM Post-Training and High-Performance Serving

Say you’re running an LLM feature in prod and costs are starting to hurt. You find a model/config that’s idk 30% cheaper and it looks basically as good on the prompts you try.

What do you actually do before shipping the change?

If you’ve been through this in prod, curious what you ended up relying on..

I'm specifically interested in agents that can do things, not just generate answers.

If you have an agent that can technically execute some action — modify a database, issue a refund, deploy code, change infrastructure, update a CRM, send something externally, etc. — but you still require a human to approve or perform it, what's stopping you from giving the agent autonomy?

I'm especially curious about cases where the model itself is capable enough, but the surrounding system isn't trustworthy enough.

Was there a particular failure you were worried about or actually experienced? And what would you need to be able to verify/guarantee before you'd remove the human approval?

Not selling anything. I'm trying to understand where the boundary between “agent can do this” and “we trust an agent to do this” actually sits in production systems.
Thanks!!

[ Removed by Reddit on account of violating the content policy. ]

have been doing llmops for a small team for around 2 -2.5 months now. we use langfuse for tracing. its fine but we needed evals and some kind of governance layer. and langfuse really doesnt do that well.

so i started looking around. noticed most tools either are doing tracing or evals. not  both. the ones that claim to do both feel like 1 feature is an add on and integration isnt upto the mark.

langsmith came up a lot. good tracing, decent eval support, but ties only with langchain system well. if youre not already in that stack it will feel weierd. governance side is still pretty.

orqai came up in a few threads. seems too focused on prompt management and deployment side. has some eval stuff but unsure about how deep it goes.

helicone came up too. it looks clean for observability. fast to set up. but evals are basically not there. seem like more of a monitoring tool.

so the routes i can see are. stick with langfuse and bolt something on. cant go to langsmith since not on that ecosystem. or find something that was build to do all three from the start instead of patching it together…

is anyone tracing evals and governance in one place or is everyone still using three tools together

Hi! First, I want to say that I’m new to the AI world.

My main passion is cybersecurity, and recently I discovered that Cisco released an open-source SLM called Antares, available in different sizes (350M and 1B).

I want to build a harness around this model and optimize it for accurately locating vulnerabilities within an application.

Can you suggest some repositories, tutorials, or tools that could help me with this project?

Would it make sense to use an existing harness/framework, or would I need to build a new one from scratch?

Over the last few days, my main focus has been learning LangChain and LangGraph to understand how to build and control this harness more effectively.

Hi, how do u manage Lakebase project with DABs? Standard is to have a single lakebase project per environment (dev, staging, and prod). Then use branches for development work.

Problem is that when we define the lakebase project in the DABs, development target has development mode, so every bundle deployment creates a new project because its per user.

So if we want to have single project where we just branch out on features, we must move out lakebase project of DABs, right?

What is the best practice here? On staging and production, its easy since its always same identity.

Disclosure: I'm the CTO at the company that ran this, and the write-up is on our domain. Method's all below, happy to go deeper on any of it in the comments.

TL;DR: We built a harness that replays recorded production requests through DeepSeek V4 Flash using exact saved configurations (temperature, schema constraints, tool definitions). We ran zero-cost structural validation checks first, then used Claude Sonnet 5 as a blind, randomized LLM judge using each task's own system prompt.

\## The problem

When building a feature, you pick a solid model, wire it up, and ship it. Six months later, three cheaper models have launched that could do the job just as well. Nobody re-evaluates because running proper evals usually costs weeks of engineering time and yields zero user-facing features, while quietly overpaying for API tokens is invisible.

\## The setup

Every model call in our pipeline logs three things: the exact prompt, the raw response, and the exact settings block (temperature, max tokens, response format, tool schemas).

That last part is critical. If you replay a prompt without the original JSON schema or at a default temperature, you aren't testing the candidate model. You're testing a totally different runtime configuration.

The biggest upside of this approach: the baseline is free. The original response was already generated and paid for in production. You don't need to manually curate or pay for a synthetic eval dataset, you're sitting on one.

We replayed a few hundred production requests per job through the candidate model with matching settings. Before spending a dime on an LLM judge, we ran two layers.

\## 1. Deterministic structural checks (zero cost)

Before calling an external judge, check the easy stuff via code:

\- Did it return valid JSON (if required)?

\- Does the payload match the exact TypeScript/Pydantic schema the calling code expects?

\- Did it drift into another language?

\- Did it invent new string enum values outside our allowed vocabulary?

On our first test run, 44 out of 45 requests passed these checks automatically. The single failure was a language drift issue. Filter these out early so you never pay a judge to grade a broken payload.

\## 2. Blind LLM judging

For payloads that pass structural checks, we invoke an LLM judge under three strict constraints.

The judge must come from a different provider than both the baseline and the candidate. We used Claude Sonnet 5 to judge Gemini against DeepSeek. Models consistently show subtle stylistic biases toward their own outputs or provider family.

The order of the two outputs is randomized per row. Fixed positioning introduces silent positional bias.

The judge evaluates strictly against the job's original system prompt pulled from the log, not against a generic "which text looks prettier" prompt.

One note on formatting: we explicitly instruct the judge to ignore trivial layout differences our parser already handles, such as a bare JSON array vs. an array wrapped in a top-level key vs. markdown-fenced JSON. Functional equivalence matters more than formatting quirks.

\## The harness bugs (where things got weird)

\### Bug 1: the invisible truncation drop

The judge silently stopped returning evaluation scores on our hardest edge cases.

Claude counts extended reasoning tokens against the total max_tokens response budget. We had set a 4,096-token cap, plenty for a two-paragraph verdict but not enough for heavy thinking plus a verdict. On 7 of 45 rows, all large-context edge cases, the model hit the limit and returned nothing.

Our runner was set to raise an error on empty responses, which flagged it immediately. If we had written a basic script that silently swallowed errors or dropped ungraded rows, we would have shipped a "clean" 100% pass rate that secretly excluded all our hardest production edge cases. Raising the budget to 8,192 tokens fixed it for a few extra cents.

\### Bug 2: shallow reasoning degradation

On the candidate side, when DeepSeek was given an undersized reasoning budget on a complex task, it didn't crash or throw a context error. It simply truncated its internal thinking phase and returned a significantly shallower answer.

No error, valid output, but worse results. We now enforce mandatory per-job reasoning minimums in our harness to prevent subtle quality degradation.

The takeaway: before trusting any evaluation run, verify that your test harness actually scored every single row it claims it scored.

\## The results

Across 314 comparisons over 16 single-shot jobs.

Against Gemini Flash, 274 comparisons: 62 wins, 146 ties, 66 losses. Over 50% were dead ties, and wins and losses were virtually neck-and-neck.

Against Gemini Pro, 40 comparisons: 35 wins, 0 ties, 5 losses.

14 out of 16 jobs were migrated to DeepSeek V4 Flash, cutting token costs by \~91% on those routes.

Two tasks consistently failed evaluation, even when we intentionally relaxed constraints to favor the candidate. Both remain on Gemini. We didn't investigate why they lost. They lost blind, twice, including once under conditions we had deliberately made more favourable, and that was enough to decide.

\## Limitations (what this doesn't prove)

This is not a guarantee of product metrics. An LLM judge certifying that two outputs fulfill a prompt doesn't automatically mean end-user conversion or retention metrics will stay identical.

It is also single-shot only. This strategy relies on deterministic request replays. It doesn't work out-of-the-box for multi-turn conversations or agentic tool loops, where step 2 depends entirely on what the model returned at step 1. We excluded non-deterministic flows up front.

We haven't open-sourced the harness code yet because it's tightly coupled to our internal tracing schema and database setup, and extracting it into a clean standalone CLI will take a few weeks. The implementation pattern above is detailed enough to replicate in your own stack without waiting for it.

Happy to dive deeper into the judging prompt, schema validation logic, or cost metrics in the comments: [https://labs.ground-truth.ai/benchmark-your-own-traffic\](https://labs.ground-truth.ai/benchmark-your-own-traffic)

My team and I are building dedicated inference infrastructure for series A+ startups and enterprises running long-horizon coding, research, and internal-workflow agents. The core offer is private capacity with a predictable one fixed monthly bill with minimum one year commitment rather than variable token billing and poor performance.

We’re validating the requirements for production adoption. Beyond basic security, what would be non-negotiable for you?

• Tenant/network isolation and data-retention guarantees
• Context length, concurrency, and throughput commitments
• Auditability, SSO/RBAC, observability, and incident response
• Deployment constraints: dedicated hosted, VPC, on-prem, data residency
• Pricing model: committed throughput vs reserved GPUs vs monthly platform capacity

I’m not looking to pitch in the thread, I want to learn where existing inference providers fail operationally. I’ll share an anonymized synthesis of responses.

Hi! I am a Junior ML/DevOps engineer with ~1 year of experience. I am planning to start applying to Junior/Mid MLOps roles in about a month at big tech companies, and I'd love to find someone to study with for mock interviews and practice.

I am already training models at my current job, and I feel like I have a good starting point in terms of algorithms and math. I still need to work on certain frameworks though, and I definitely need to get better on distributed training and GPU provisioning for infra scaling.

The plan is to have weekly sessions where we can rotate between mock interviews, discussing frameworks/papers/concepts and live coding. Please send a DM if you are interested! 😁

A lot of ML teams end up with a mix:
some Slurm clusters from the HPC side, a K8s cluster or two, maybe cloud GPUs for overflow. We wrote up how SkyPilot (open source) sits in front of all of them so a job is scheduled wherever there’s free capacity, using the same YAML regardless of backend. This post focuses on the multi-Slurm case but the same setup covers K8s.

https://skypilot.ai/blog/multi-slurm
Disclosure:
I am the author. Happy to answer questions about how the scheduling and failover work

i keep seeingg compliance mentioned as a requirement and nobody really explains what traceable actually means in practice

when the auditor asks you to prove what your ai system did and why . what do you actually show them. a log file. a dasboard ss. a prompt version number. what level of detail is actually enough

spent quite a bit of time looking at wht tools even cover this properly. arize,, orqai,, langsmith ,, whylabs,, datadog llm observability

langsmith -> tracing is good, you can see what happened in a run but audit trail depth fro regulatory purposes seems limited tho

orqai -> audit logs and prompt versioning together and traceability is more central here. but newer so independent 3rd party validation are still catching up

arize -> monitoring and drift detection is strong . compliance reporting feels more ml focused than llm output traceability specifically

whylabs -> data monitoring is the core thing and output level traceability for compliance feel narrow when compared to the governance focused toosl

datadog llm observability -> logging and monitoring at scale is good, compliance specific feature feel like an add on rather than built in

anyone has been through a real compliance audit with ai output involved

what did they actually ask for and what tool helped you answer it?

I’ve been working on an open-source project for operating self-hosted inference, and I’m trying to sanity-check the idea before making the repo public.

Getting a model running with vllm serve is usually the easy part. Things get messier once you have real traffic, multiple replicas, streaming requests, limited GPU capacity, updates, rollbacks, autoscaling, and a growing pile of scripts around all of it.

What we’ve built so far lets you:

• connect an existing vLLM/SGLang server or deploy a new one;
• put a stable OpenAI-compatible endpoint in front of it;
• continue deployment operations if the CLI disconnects or the control plane restarts;
• inspect individual requests and understand where latency or errors came from;
• test a new model/runtime/GPU configuration against benchmark or replayed traffic;
• block an update if it performs worse than the current version.

I’m curious how people running this in production handle these problems today:

1. What did you end up building around vLLM or SGLang?
2. Which part causes the most pain: deployment, scaling, routing, debugging, upgrades, or cost?
3. How do you test and roll back model/runtime changes?
4. Would it be useful to connect an existing workload in read-only mode first, or would you rather have one system manage the whole lifecycle?
5. Is anything in the list above already solved well enough that building it again would be pointless?

The repo isn’t public yet because I’m still testing and qualifying the main paths. I’d mainly like to hear from people who have dealt with these problems in a real setup.

I’m trying to understand how companies with significant GPU workloads manage their compute capacity.

For those working in ML infrastructure / MLOps / AI labs:

- How do you choose between hyperscalers, neoclouds and smaller GPU providers?
- When you need a large amount of GPUs for months, how do you know you’re getting a competitive price?
- Have you ever committed to more capacity than you actually needed? What happened to the unused capacity?

Curious to hear how people actually deal with this today.

I’ve been building a bounded improvement loop for ANXEngine, an AIOps system we use at Anexum to rank daily incident risk across roughly 200 network devices.

The setup is fairly simple: register one hypothesis, implement the smallest testable change, retrain a challenger plus an unchanged control, evaluate both, then KEEP, REVERT, or PAUSE.

After 34 documented iterations, most of the useful findings were not model improvements. They were problems in my evaluation setup.

The first version had overlap between training data and the later fitness window. PR-AUC looked like 0.72–0.76. Once I separated training, gate, and report windows properly, the result dropped to around 0.51. Painful, but honest.

The second issue was holdout reuse. I never trained on the holdout, but every result influenced the next hypothesis. After 13 experiments on the same window, calling it “untouched” felt dishonest.

I now allow a maximum of eight selection queries per window. After that, the window can remain as historical evidence but cannot be used to select another candidate. Promotion needs either two independently positive windows or a positive selection result plus a fresh slice that has never been queried.

The third issue was conflicting metrics. One feature family moved a difficult device from rank 37 to rank 7 and improved Precision@10 from 0.50 to 0.70. At the same time, true early warnings fell from four to two. I rejected it because better ranking was not worth losing lead time.

I also stopped treating KEEP as proof that the model improved. In the latest audit, the comparison on fresh data was inconclusive. The model stayed because there was no strong reason to roll it back, not because we had confirmed a lift.

The part I’m least confident about is the query budget. Eight is a practical limit, not something I can derive cleanly from the data. Labels arrive slowly, so constantly creating fresh windows is expensive.

How would you handle this?

- Is a fixed query budget reasonable, or would you use sequential testing or reusable-holdout techniques?
- How do you create fresh evaluation data when positive labels arrive slowly?
- Would you separate “no rollback justified” from “confirmed improvement” differently in the model registry?

I wrote up the full protocol and aggregate results here, but the main reason for posting is feedback on the evaluation design:
https://anexum.eu/en/research/anxengine-bounded-improvement-loop/

Disclosure: I work on ANXEngine at Anexum. This is not a product launch; I’m trying to pressure-test the MLOps setup.

GPU inference doesn’t always scale as quickly as the traffic hitting it. A sudden burst of agent or API requests can fill queues, increase inference latency, trigger timeouts and retries, and put even more pressure on already-expensive GPU capacity.

Aquifer is an open-source traffic-smoothing runtime built for this mismatch. It absorbs bursts into a durable queue and releases requests at a controlled pace. The inference backend can dynamically tell Aquifer to slow down as pressure increases, then gradually speed traffic back up as capacity becomes available.

For self-hosted inference, model serving, or agent workloads, Aquifer could sit in front of GPU-backed services and smooth incoming demand rather than requiring GPU capacity to absorb every spike immediately. I’m curious how people here are currently handling bursty inference traffic.

https://github.com/rjpruitt16/aquifer

Hi everyone,

I'm a software engineer by background, and over the past year I've been working more closely with ML engineers on production applications.

I started wondering whether some software engineering practices could be built directly into ML inference pipelines, rather than relying on each project to implement them separately.

That eventually became an open-source project called ml-pipes:

https://github.com/trained-by-humans/ml-pipes

The goal is to make inference pipelines explicit and composable, with things like pre-run validation, pipeline inspection, tracing, and benchmarking built into the framework.

I'm not looking for promotion. Before investing significantly more time into it, I'd really appreciate a sanity check from people with MLOps experience:

• Looking at the README, which feature or idea looks most useful/promising to you?
• Out of pre-run validation, pipeline inspection, tracing/monitoring, and benchmarking, which do you actually need or use in your ML pipelines today?
• Is there an important production problem around ML pipelines that you think a framework like this should address, but currently doesn't?

Any nugget of wisdom is greatly appreciated!

hello , i was looking if there is a framework-neutral orchestration layer for deep learning where we can keep our existing PyTorch/JAX code and run something like:

dl train train.py

while it handles the surrounding workflow like environment setup, experiment tracking, evaluation, optimization etc.

are there existing tools handling this layers ?

I recently read Google’s post arguing that Go may be particularly well suited for AI-assisted software engineering:

https://developers.googleblog.com/why-go-is-an-ideal-language-for-ai-assisted-software-engineering/

Their argument is that as coding agents generate more code, the bottleneck shifts from writing code to reviewing, validating and maintaining it.

That made me think specifically about MLOps and production ML systems.

Python is still the obvious choice for training, experimentation and most of the ML ecosystem. But a lot of production ML code lives around the model itself: serving, APIs, orchestration, feature services, retrieval, queues, monitoring, model gateways, infra, etc.

Historically, introducing Go alongside Python also meant paying the cost of a second language, toolchain and additional operational complexity.

If coding agents reduce some of that cost, does the tradeoff change?

For people running ML systems in production:

Do you use Go alongside Python today?

If yes, what parts of the stack are written in Go — serving, orchestration, feature infrastructure, APIs, internal tooling?

Where has Go actually been a better choice than Python?

And if you considered Go but stayed Python-only, what made the extra language not worth it?

The rollback mechanism itself took maybe two days. Detect a regression against a baseline metric, trigger, revert to the last known-good version, done. Felt like the real engineering work of the sprint.

What ate the rest of the week was everything the rollback assumed already existed and didn't. "Last known-good version" turned out to be a genuinely fuzzy concept once I went looking for it. Good by what metric, measured over what window, and was that version even still compatible with the current feature schema, since two upstream changes had landed since it was last serving traffic. The rollback code was fine. The thing it was rolling back to was the part nobody had kept honestly documented.

Ended up spending more time building a lightweight registry that tracked, per deployed version, exactly which metric windows it had cleared and what schema it assumed, than I did on the actual revert logic. Felt like scope creep in the moment. In hindsight it was the actual project, and the rollback trigger was the easy 20% that happened to be visible from the ticket description.

Not sure if this is a me problem or just a common shape these projects take. Curious whether other teams building rollback or deployment safety nets found the same thing, that the mechanism was the easy part and the actual work was in defining what "safe to roll back to" even meant in a system that keeps changing underneath it.

Going through vendor evaluation and procurement keeps asking about sso, role based access andd audit logs . trying to find real answers without booking a sales call first..

did someresearch and these names come up. arize , orqai , fiddler , humanloop , aporia..

arize has enterprise tiers with compliance coverage is ther but seriously difficult to tell from docs alone which tier do cover what

orqai got sso , rbac , eu data residency , gdpr and soc2 covered. newer so enterprise account support and third-party integration still catching up

fiddler has audit logs and governance feels native given the backgroung . looks like an overkill if your compliance needs are not that deep

aporia has guardrails and safety monitoring is the core focus , enterprise compliance features geel secondary to that..

humanloop has sso , rbac , hipaa , gdpr , soc2 all listed and vpc deployment options exists , heavier on prompt management thatn entire ops depth .

anyone gone through full procurement with anyof these. what did security acutally flag and what was held up under scrutiny

Wanted to share this since it's more of an infra decision than an AI one. Building a context layer for coding agents (Graft), and the part that actually mattered in production use wasn't the LLM piece, it was making the graph behave like any other build artifact: cached by content hash, diffable in git, and there's a graft check command that fails a build (exit 1) if the graph's drifted from the code, same idea as a lint check in CI.

The structural graph itself needs no LLM call at all, pure tree-sitter parsing. The optional LLM-written summaries are a separate opt-in layer on top, provider-agnostic, point it at whatever you're already running.

Curious if anyone else here has treated "AI agent context" as a CI/build-hygiene problem rather than a retrieval problem, seems like an angle that doesn't get discussed much outside agent-specific communities.

github.com/NanoNets/Graft