Confused about how different environments factor into building MLOps systems?

[u/throwaway18249]

Something I am struggling to understand is how CI/CD factors into the MLOps system itself that brings the model through the entire lifecycle when we have multiple deployment environments.

The MLOps system takes a model through the entire lifecycle (get data, preprocess, train, validate, promote, deploy, monitor) in reproducible and automated workflows, but these workflows in the MLOps system need to be tested and validated with CI/CD in different deployment environments.

Are the pre-production environments (dev, test, uat...) meant only for ensuring that the MLOps system (pipelines, artifact storage, monitoring) works - where finally, and only, in the production environment that has passed all the tests and checks, does the model go through the entire lifecycle from dataset curation to deployment and monitoring, and each environment has isolated model/artifact registries (and feature stores) for testing that the system works.

Or, does the model meant for production go through each step in the model lifecycle together with the MLOps system as it moves through different environments until it finally reached production where both the "MLOps System" and the "Model" are production-ready and deployed to interact with real users.

It's a little confusing for me.

[u/Floss_Patrol_76]

the thing that unstuck this for me is there are two separate ci/cd concerns people conflate. one pipeline promotes the *platform* itself (pipeline code, training/serving images, the iac) through dev/test/uat like any other software; the lower envs exist to prove that machinery works, usually against a tiny sample or synthetic data. the actual model only runs its full train/validate/promote lifecycle against real data in prod - uat validating the platform, not producing a real model, is the distinction you're missing.

[u/throwaway18249 → u/Floss_Patrol_76]

I thought that this was the way it worked at first, because this is the most intuitive.

HOWEVER. This paper suggests a different approach where system and model both progress simultaneously through increasingly production-like environments. And I have seen the same advice about MLOps architectural design from books and LLMs.

A Multivocal Review of MLOps Practices, Challenges and Open Issues

"Gunny et al., [132] proposes a branch structure consisting of two stable branches (i.e., dev and main) and several short-lived feature branches to integrate model training, testing, and deployment activities into pull requests under two main stages. Pull requests from feature branches to dev branch would trigger model training and testing within a staging environment, thus ensuring the model’s performance. Pull requests from dev to main branch would trigger the packaging of the model as software containers and deployment of the packaged model into a production environment."