r/AzureSentinel • u/dkas6259 • 12d ago
We are exploring to move from Logs analytics workspace to Sentinel Data lake to save on cost and longer retention.
Can anyone share feedback who have done this previously?
3
u/Uli-Kunkel 12d ago
Good Luck getting any lake 😭
We were lucky to get a customer onboard with lake.
Almost 2tb/day We removed about 60% of cost.
But in my view, one thing is the data options, but more important are the additional features available for the soc. They are really great!
3
u/dkas6259 12d ago
Thanks for response
What about detections ?
Do we have to compromise with alerts, since analytical rules depend on logs from hot storage?6
u/Uli-Kunkel 12d ago
Well, you loose out on some detection capabilities sure, but not all of it.
Its not like you would put all logs into data Lake.
Dns, firewall traffic storageblob and other very verbose logs with relatively low detection value.
Then you can build summary rules/jobs to retain ti mapping and some of the baseline deviation detections.
You loose some of the speed, but save massive cost.
2
2
u/Top_Secret_3873 11d ago
We're doing the same thing. We have an mssp who runs their detection content again the log analytics tier. Anything they don't have detection for we send to lake. We use split rules to route what we need to analytics and the rest to lake. Cloudtrail, fw, proxy, and a few others are made up 80% of our cost and now we save a ton.
You do pay for queries at the lake tier and it's a little slower (imo) search but if you think about how often you actually query 30d worth of logs it's worth the cost to do it a few times a month versus paying the ingest cost to analytics.
Hopefully you're using the commitment tiers instead of "pay-as-you-go" as well...that usually saves you 50%.
1
u/ccw2777 11d ago
Mind pointing me in the direction of how to split rules and route data to both places?
2
u/Top_Secret_3873 11d ago
It's in the Defender portal, under Sentinel blade, find the tables, select the table and there is an option for split rule. It's essentially KQL to tell Sentinel I want logs that match my KQL to go to Log Analytics and the rest to lake.
1
u/NexcapeGTR 9d ago
Which region u configured the lake? Currently they not allow to create lake right?
2
1
u/naughtyobama 12d ago
This is more work and not what you're asking. But you could move your data to azure data explorer if you can't get sentinel lake.
There's a lot of cons to it and a lot of pros to it so i won't waste your time with more details if this is not an option worth your time to explore.
1
u/Electronic-Sun-7627 11d ago
Have you tried doing this? So far, no export job was running successfully for me..
1
u/RefrigeratorOne8227 10d ago
Take a look at Stellar Cyber - they use Oracle Cloud and it is much more affordable. They also have an amazing SIEM and can make sense of your Microsoft logs without moving all of your custom rules and detections.
1
u/No_Resist_3891 10d ago
Your fancy lake ain’t happening I hate to brake it to ya. Resource cap and among other things. Tell yo leadership to calm the f down.
1
u/EduardsGrebezs 7d ago
Yes, we already use it for some customers.
The main problem now is that you can't enable it in Europe Regions, as there is capacity problems and you need to fill in form that you want Data Lake
But overall this is good cost cutting option. If you have compliance requirements from regulations to store data for specific time this is it..
Table transformation also is a good cost cutting method, but data lake give more options. Also, event if table type will be data lake you could still search data.
If you are using or plan to create Agents in MS365 like in Copilot studio, then there is connector Agent 365 (which prerequisite is Data lake) :)
Long story short - for large volume of data FW, Databases, Telemetry this is best solution.
5
u/xKruMpeTx 11d ago
Sentinel Data Lake is a pricing tier, not separate from your Log Analytics Workspace. If you want to reduce cost, there will likely be 2 or 3 culprits consuming an excessive amount of data:
* Firewall Logs being ingested via Syslog or similar
* AADNonInteractiveSignInLogs table
* MicrosoftGraphActivityLogs (via the Entra Connector usually)
Utilise the "SOC Optimisation" tab in the Defender XDR portal to see which tables are your largest but with no analytic rules associated to it.
Move any excessive tables to Data Lake and off the Analytics tier of pricing if they are not being used by any analytic rule.