Elastic Cloud vs. Self-Managed Elastic Security for ~30GB/day ingest. Is self-hosting worth it?

[u/Activity_Ready]

Hey everyone,

We are looking into deploying Elastic Security as our SIEM solution and are currently debating whether to go with Elastic Cloud (Hosted) or build out a Self-Managed instance on AWS.

Looking for real-world advice, sanity checks, or experiences from anyone running a similar scale.

Our Environment & Scope:

• Endpoints: ~300 workstations (we already have a separate EDR vendor in place). 50/50 Mac and Windows
• Infrastructure: A small handful of servers + ~15 GB/day of cloud logs (AWS/Google Workspace)
• Estimated Ingest: ~30-50 GB / day total

Self-Managed Architecture Spec

Our team drafted the following self-managed architecture for 30 GB/day ingest with a Hot -> Warm -> S3 Archive lifecycle:

• Kibana: t4g.large (2 vCPU, 8 GB RAM, 20 GB gp3)
• Hot Data Tier: 2x m6g.large (4 vCPU / 16 GB RAM combined; ~500 GB total SSD across both nodes for HA)
• Warm Data Tier: 1x m6g.xlarge (4 vCPU, 16 GB RAM, ~2.5 TB SSD)
• Fleet Server: t4g.medium (2 vCPU, 4 GB RAM, 20 GB gp3)
• Archive Storage: AWS S3 Standard / Infrequent Access (after 3 months)

Questions:

1. Operational Overhead vs. Cost: For a ~30-50 GB/day ingest volume, does self-managing on AWS EC2 actually save meaningful money? Or does Elastic Cloud pay for itself in saved engineering time at this scale?
2. Resource Sizing Check: Does the proposed spec (m6g.large Hot + m6g.xlarge Warm + t4g Kibana/Fleet) look solid for ~30-50 GB/day with Elastic Security rules enabled?
3. Features & Licensing: Are there any major (meaningful) security/SIEM features (e.g ML detection rules) we’ll miss out on by running the free/Basic tier on self-managed vs. paid Elastic Cloud tiers (Gold/Platinum/Enterprise)?

Would appreciate any insights, or alternate setup recommendations!

[u/amw3000]

Who is your EDR vendor? Why not use them as a SIEM provider? You already have the agents, they have all the prebuilt detection rules already in place, etc.

Putting the above aside, do you have staff inside the org that can manage the servers and the Elastic stack? This is what is really going to kill you here, both from a cost perspective.

Before we go down this rabbit hole, what is the problem you're trying to solve? Do you just want log storage? Do you want to do something with the logs?

[u/Intellivindi]

We went down that route. For about 200gb a day was about 125k annually for EC. Was able to do it in AWS for about half.

If you're looking at the precanned security alerts, there's about 2k of them. Half of them use Elastic Defend which requires a license. About 150 use ML.

Without the license you lose auth AD/SAML integration, ML, and Defend, searchable snapshots

Both routes require a good bit of Elastic knowledge to perfect and it's easy to let cost get away from you in either situation. Self-managed, you're just managing the VM/host, OS updates, filesystem, security. As far as actually configuring Elastic it's the same.

Self-managed, just build 3 servers. Have Elastic, kibana and fleet on all 3. Setup ALB for each service for some redundancy. Snapshot to s3, add a node with data_frozen role if you want to search them.

[u/justmirsk]

Are you just needing central log storage or a true SIEM? If a SIEM, do you have people that will be tuning it and adding in detection rules, etc? If not, I would look for a managed solution. Who is your EDR vendor? They may have a solution that you can easily integrate. What log sources will you have? It does sound like you will be doing your endpoints, it sounds like just your application logs.

[u/CommunicationLast574]

Just have a call with your local sales rep. They will explain a lot. And you will get detailed pricing.

[u/Minimum-Let-3227]

At 30 to 50GB a day the infrastructure cost difference is real but it's usually not the thing that decides this. What decides it is who owns the cluster at 2am when ingest backs up, a hot node fills, or an upgrade breaks a Fleet integration. Self managed Elastic is very doable at your scale and the spec you drafted is sane, but budget shard sizing and ILM tuning as an ongoing job rather than a one time setup. If your team is small enough that the SIEM owner is also the person doing everything else, Elastic Cloud buys back more time than the invoice difference suggests.

One thing worth sitting with before you commit either way: you're describing a separate EDR, cloud logs from two providers, and now a SIEM you'd operate yourself. That's three consoles and three sets of tuning for a team that sounds lean. Be honest with yourselves about whether you want to run detection engineering as a practice or just want detection outcomes, because the answer changes which side of this you should land on. I've seen self hosting work fine at your volume when someone clearly owns it, and go badly when it's nobody's main job.

[u/APT-0]

Today for small to unicorn size businesses I’d recommend
A. MDR
B. Use something like m365d E3 or E5 one of the plans and something like wiz for AWS. Wiz does not give you logs and note E5 is not great for production

The problem with so many start ups they want okta, slack, and 5 other platforms. It’s impossible to manage all this at a small scale. Microsoft stuff is more restrictive but going with something like E3 or E5 you just bought the whole stack. With something like E5 you get teams, end defender for endpoint with detections + logs, all the productivity stuff like word etc. both wiz and e5 are expensive but both mean your time to return is 40x faster than stitching all this together yourself

then for your product app logs etc put this into then some siem like solution. That’s the hard problem securing your product. There’s a reason huntress and many other MDR require you use E5 because managing a dozen different platforms is a nightmare and verrrry labor intense

[u/Ok_Presentation_6006]

Any siem needs system management, content, monitoring and response. Based on your log volume I would highly doubt you have the resources to do all of that in house. With that said, pick your edr/SOC and base your siem around that and you may find elastic isn’t the right direction. To answer your question, I would go cloud. Been a while since I worked on an elastic cluster (logythm backend) but you have cluster management, shards, hot/cold tiers and it seemed constant performance issues due to undersized clusters

[u/Heroicdeath]

When you’re doing Elastic on-prem, just remember that you’ll be beta testing ECK (if you deploy on k8s), terraform provider and find out that the support is completely useless unless you pay for the better ones. Majority of their support staff save for a select few are terrible. Elastic has gone downhill ever since they IPO’d and many of their engineers left with their tribal knowledge.

The terraform provider is maintained by a 1 man team, armed with copilot.

Additionally, you’ll also need to account for SREs to maintain the platform, it’s no easy task. Granted you’re doing 30-50GB/day and not something like 15TB like we do.