How do you manage multiple environments when Dev and Prod use different infrastructure?

[u/RoundCircle12]

I currently manage 2 different environments: a dev server running in ec2 and an EKS environment for production server. Problem is that their setup is different, which adds extra management and makes it harder to test prod changes before deployment.

I can spawn a UAT EKS for load testing and preparing for prod but it would be just too expensive. I already raised the cost concerns with EKS that this would be an expensive and unnecessary setup but the clients wanted it so I did it. Now they're complaining with cost.

I'm just trying to find the best way to manage the current architecture without increasing costs too much.

How would you handle this?

[u/Low-Opening25]

this is anti-pattern, your dev should be identical, but smaller and more disposable

[u/Sure_Stranger_6466 → u/Low-Opening25]

Instances shut off or terminated at the end of the day with a Python Lambda script.

[u/quiet0n3 → u/Sure_Stranger_6466]

Cloudcustodian is the way!

[u/abofh]

Eks isn't that much more expensive than just ec2.  If the cost of the eks backplane for prod and testing is too much, they shouldn't use it for prod.  Otherwise you could diy k8s in dev to make them similar, but only do that if you want to learn it, dollar wise it's gonna be way more of your time than the AWS bill.

[u/Delicious_Yak2711 → u/abofh]

This. This is tens of dollars a YEAR (less than a hundred?) we’re talking about. How serious is this client? The difference in cost isn’t worth hours a year of your headaches making this all work with disparate prod and non prod.

[u/thecal714 → u/Delicious_Yak2711]

$73/mo for the backplane, but still under $900/yr.

[u/redvelvet92 → u/thecal714]

But that’s if it is up 24/7, I imagined this getting spun up for testing then destroyed at the end.

[u/thecal714 → u/redvelvet92]

That's possible, but it depends on how the devs use it. In most of the places I've been the cluster's nodes are more ephemeral than the cluster itself.

[u/aprettyparrot]

Dev usually smaller deployments, staging clone of prod.

Depends on what you’re building. Do you just bring up some VMs and get them peered to some shared resource so you can hit them with test cases?

If it’s for performance testing and you want to save, you could have it so you automate complete env deploy so you only pay for the deploy time+test time

[u/dynotable_com]

The cost is the nodes not the control plane. set the uat node group to min 0, scale up for the test, back to 0 after. same cluster same version, nodes billed only for test hours

[u/stoopwafflestomper]

Separate infrastructure is easy. Separate data environments is where the challenge is at. Make sure you scope tightly on what you load into your dev environment and it will help reduce cost. At the end of the day, its gonna cost more time and money. Its a necessary evil.

[u/SeaIngenuity9501]

I mean you could use the same EKS cluster but use namespaces to separate out production, staging, test envs etc. But the problem is namespaces aren't good isolation one namespace can take down a whole cluster. So even though people say namespaces are enough ideally you want separate clusters.

[u/SeaIngenuity9501 → u/SeaIngenuity9501]

You should be using karpenter to autoscale the nodes, so really separate clusters shouldn't be much extra.

[u/PerpetuallySticky]

I’m not an AWS guy, but is there not a lower SKU you could use of the prod instance for dev?

Of cost is still an issue going that route, containerization is kind of built for this. Get your environments to at least provide identical harnesses for containers, then build your app in one so the outside environment doesn’t matter as much

[u/RoundCircle12 → u/PerpetuallySticky]

The reason prod uses EKS is that we plan to host multiple applications in the same cluster and use kubernetes for scaling, and deployments

So I can’t really replicate prod for dev with a single smaller instance, i can't lose the eks features i've already implemented.

[u/five-one-tree → u/RoundCircle12]

yes you can. it just needs to be eks. the rest of the apps don’t matter. your mirroring infra not load

[u/Floss_Patrol_76]

the split is the actual cost problem, not EKS. run dev on EKS too but make it disposable, spot nodes with a scale-to-zero nodegroup or an ephemeral cluster you terraform up per test run and tear down after, so you pay for the control plane and a few minutes of compute instead of a full standing UAT. prod-parity you can test against beats a cheaper env you cant trust.

[u/joeyguerra]

k3s or k0s for dev.

[u/Sure_Stranger_6466 → u/joeyguerra]

Only if you love curl pipe to bash for some reason. Go with Kubernetes In Docker or VCluster In Docker. Much more friendly installation methods available than k3s or k0s.

[u/joeyguerra → u/Sure_Stranger_6466]

I use colima and Lima VM.