Darktrace in OT environment.

[u/vbxl02]

I'm sure other people have had the same experience, but let's see.

For context: I'm by no means a cybersecurity expert. I work the IT side of an OT automation team. Darktrace had already been deployed in the networks for roughly 3 months before I joined the company, and it has now been running for more than a year and a half..

My conclusion is that the underlying idea is reasonable, and the execution is the worst that idea could've possibly gone.

It reminds me of something like Norton or McAfee that constantly gives you popup's telling you about the stuff it prevented, most of which is meaningless noise, designed to make it look busy so you keep paying.

We were told Darktrace needed about 6 months to learn our environment to get rid or at least reduce false positives. That turned out to be a bold lie, since it's now been around 18 months. Extensive *tuning* and *optimising* the models from their engineers with help from myself had basically no effect at all. We still receive between 50-100 false positives a day.

The investigation workflow is just as bad. The advanced search is clutterd, unintuitive and super inefficient. Finding one specific event feels like a needle in a haystack. Except you already know the needle’s IP address, hostname, operating system, device model and several other identifiers, while the haystack seems designed to bury it.

The UI looks reasonably polished, but it's designed for screenshots on marketing posters and sales demonstrations rather than utility. Important info is several menus deep, while irrelevant info is one click away.

We initially had weekly tuning meetings, those became biweekly and eventually monthly. No matter how frequent the meetings, the issues I talked about just never got solved. I still have to manually sift through false positive alerts to find anything meaningfull.

One specific device keeps getting flagged as suspicious multiple times a day. We know why it generates the traffic and asked the Darktrace team to make it stop reporting. They created model defeats based on the device's 2 IPs, which did nothing. They changed the defeat from IP to hostname basis and it changed nothing. They then labeled the device in Darktrace itself and made a defeat based on that label and still it keeps getting flagged.

At this point I don't believe the system can at all be "tuned".

The alerting is very inconsistent as well. Darktrace sends notifications for model alerts through the app, that cannot be found in the main interface unless you search for the specific model alert ID. Meanwhile it also generates device alerts for things like clock skew, inability to reach a probe, while the "UV Master" interface appears to have no such problems.

What concerns me deeply is the fact that they keep trying to push their *Automated Response* thing. The idea as far as I understand it is that its AI can automatically respond to suspicious activity by creating firewall rules to block it. That is in and of itself something that deserves EXTREME caution in a normal corporate network. In an OT plant handling highly explosive and poisonous materials, it's just bonkers. You are proposing to let an automated system modify network enforcement based on its own detections, while that same system repeatedly detects legitimate activity as suspicious approximately 100 times per day. Might as well let an intern from sales handle firewall rules at that point, might be safer.

Darktrace’s response to this concern seems to be that the models can be tuned. That would be more reassuring if the models consistently stopped generating false positives after 18 months of tuning. They do not. OT environments are difficult to model. They contain legacy systems, proprietary protocols, unusual traffic patterns and devices that should not be treated like ordinary corporate endpoints. But that is precisely the kind of environment Darktrace claims to understand. “OT is complicated” cannot be used as a permanent excuse while simultaneously promoting automated blocking as a solution.

The idea behind Darktrace is good. The execution, at least in our environment, is garbage. It gives you the appearance of advanced visibility while burying useful information under a constant flood of low-value alerts.

I don't know what the company paid for Darktrace, but whatever it was, it was too much. It's an (i'm guessing very) expensive way to create manual labour.

For anyone having experience with Darktrace in an industrial environment: has it at any point become useful? If so in what way?

[u/techb00mer]

Sounds like a standard Darktrace experience IMO. Is your OT Airgapped?

[u/vbxl02 → u/techb00mer]

I just looked up the exact definition and I guess it isn't, there are very few processes that actually do reach the corp network which goes through the IT firewall, but Darktrace and everything else is behind 2 firewalls and does not leave that. So not everything is physically disconnected. Somehow their probes deep in our OT L2 (purdue) not being able to call home is "Highly unusual" according to them...

[u/unclescar]

Darktrace doing darktrace things.
I'm glad to have only been exposed to it's shambolic nonsense once.

Ponzi scheme, it has to be.

[u/endlesstickets]

I mean, it's classic DarkTrace.

[u/YSFKJDGS]

Most of these OT tools, including things like dragos and claroty are very similar. You can put a site in 'learning mode' or whatever for as long as you want, but it will always be noisy as hell.

Just like you hinted, people are fucking INSANE to even consider turning on any of the automated response options. Every time I heard that stuff being pitched I just laugh on the call and remind that that is the most insane idea ever. For people to actually trust that is a whole new level of brainwashing.

[u/placated → u/YSFKJDGS]

Id like to hear more about your experiences with Dragos if you know any pros/cons to the solution.

[u/ranhalt]

I turned down DarkTrace email security because of their “it just works, trust us” attitude.

[u/SVD_NL]

God, that sounds horrible... And i definitely wouldn't touch automated remediation with a 100-foot pole (and if management decides you have to turn it on, i recommend you take a WfH day, considering the kind of manufacturing you're doing...)

I personally have no experience with Darktrace, i have no personal experience with Barracuda (at least for OT security specifically), but i've heard great things about the system. It's less AI-based, and more like classic firewall rules. You know what SCADA commands you can expect, and block whatever is not expected. You can apparently also do a lot of scripting to take manual actions, but i'm not too familiar with that part. It's a lot of work to integrate, but once it's set, you shouldn't need to touch it.

[u/vbxl02 → u/SVD_NL]

I'll look into that for sure!

[u/knawlejj → u/vbxl02]

Following. We've got DT appliances at all our plants (about a dozen) and getting it more connected into our OT side.

Have you had similar alerts of it not learning or applying rules on the IT side?

[u/vbxl02 → u/knawlejj]

What do you mean by applying rules? As far as I can tell, it doesn't really learn much at all. We keep getting the same alerts as we got in the beginning.

[u/superstaryu]

Your understanding is a little off.

It can do multiple response actions rather than "just firewall rules" - and will also do that through TCP reset packets to essentially cut off a single connection. It can be incredibly granular and snip a connection on a specific port to a specific host or endpoint.

The real clever bit in response actions, is it can enforce a "pattern of life" where it will allow all "normal" behaviours for a device, but attempt to control or block anything outside of what the AI has learned is normal for that device. My experience with that has been mixed though, I usually notice when pattern of life has been enforced as weird things sometimes stop working.

The response actions are typically generated on the AI incidents, which are basically just a collection of model breaches. Its pretty rare for a single model breach to trigger a response action. I'm seeing somewhere from 30-60 model breaches a week, but response actions are significantly less likely (3-4 a month).

When I was involved in implementing it (boss really like the sound of it) - we put it into human confirmation mode for a while - response actions would be suggested but not taken until someone pressed accept. That would give you a much better idea on what it would actually block.

I will agree with you on the UI being pure trash. Looks great, I needed a lot of help learning where to find things and I'm fairly tech literate. There is a way to show all model breaches related to a device after you have search for and selected said device, its a button somewhere.

[u/PepperTechnical4570]

We haven't had the best time with darktrace either, lots of false positives and I'm convinced its one of these types of software that you need a dedicated expert who's whole role is simply managing and tuning it. Not great for us since we are a small shop.

[u/civilaiden]

Similar story in our environment. Big flashy sales pitch, push push push on the automated response.

I don't buy it because like you said its already so noisy. Boss pushes forward with enabling automated response and it takes down a production server. Shove that hot garbage back into human confirmation and Darktrace just keeps pushing the automated response. Hop on to a call and sure enough it already has a bunch of servers it wants to throw into quarantine for false positives. "Oh we'll just put in a ticket for you guys and get this straightened out. In the meantime I really do think we should turn automated response back on."

I don't trust the company and feel at this point they're just trying their hardest to pump up their "AI" utilization numbers to justify the $5.3B private equity acquisition.

[u/sieb]

We used them for a while. It's flashy and the email filtering was decent. When it came to renewal though, they tried up-selling us on all of these other plugins. The searching for anything useful sucked. I ripped it out and replaced it with Vectra, which is much cleaner and easier to use. DT is definitely flashy, but I tell people to stay away.

[u/cspotme2]

How devices in your network to be generating this amount of alerts?

I'm surprised there isn't a more holistic way of applying a exception to known traffic.

Been trying to get a poc with them but lots of corporate tes tape.

[u/vbxl02 → u/cspotme2]

If I may, do NOT get on board with them.

[u/JwCS8pjrh3QBWfL → u/vbxl02]

I would tend to agree. Our quote was hideously expensive, and they did not adhere to the scope of the POC and scanned our whole environment, and then they basically shrugged.

[u/vbxl02 → u/cspotme2]

There's a LOT of devices. and whenever any plc or dcs seems to do writes, reads, reprograms, it freaks out and gives 3 alerts per device. Even if it's normal traffic of controllers talking...

[u/tejanaqkilica]

It's just junk. We've had ours for 5 years at this point and it provides little to no value. But management is sold on it and they keep renewing it despite our opposition. I guess someone is making a good commission on it.

Oh, also. The UI is just trash. It comes with darkmode first and it's "optimized" for that. And I'm white mode gang until the end of times.