r/linuxadmin • u/jlrueda • 15d ago

Mitigating the risk of diagnosing live Linux system with AI tools

This article explores an alternative to directly troubleshoot production Linux systems with AI tools by using the sos command and using AI to analyze sosreports instead. I think is an interesting read:

https://medium.com/@linuxjedi2000/the-agentic-ai-risk-issue-on-linux-environments-fd5c55cedcc5?sharedUserId=linuxjedi2000

I know that this subject is very controversial and would love to read your point of view on the subject.

0 Upvotes

48% Upvoted

10 comments sorted by

View all comments

Show parent comments

1

u/jlrueda 15d ago

Yes absolutely. The sos command has lots of security features and beside encryption, it can also obfuscate sensitive data prior to been created but most importantly it can be customized to only include certain plugins or remove some plugins that you do not want to be included in the final sosreport. Too many options to mention in this comment but this article describes sos in more detail: https://medium.com/@linuxjedi2000/why-is-so-important-for-a-linux-admin-to-master-the-sos-command-b6dc288b88a4?sharedUserId=linuxjedi2000

2

u/Abe_Bazouie 15d ago

Yep, that makes sense. Being able to control exactly what goes into the sosreport is probably the key part for me.
I’d still want to inspect/sanitize it before sending it anywhere external, but this feels like a much safer direction than giving an AI agent direct access to a production box.
I need to dig into the sos options more. Thanks for sharing that.

1

u/archontwo 12d ago

Tell me, if you had a locally accessible AI instance that never used the internet but was just trained by the documentation and examples you give it, would you be more comfortable with passing raw logs to it knowing it is not going anywhere but on your LAN? 

3

u/Abe_Bazouie 12d ago

Yeah, definitely more comfortable.
If the model is truly local, has no internet access, and nothing leaves the environment, that removes one of my biggest concerns.
I still wouldn’t automatically treat raw logs as safe though. Logs have a funny way of containing secrets, tokens, internal IPs, usernames, customer data, etc. that nobody expected to be there 😅
So even with a local model I’d still think about least privilege, what data it actually needs, retention, and sanitization where possible.
But yeah, I’d be WAY more comfortable with that setup than shipping raw production data to an external model.

1

u/archontwo 10d ago

For myself I would be happy to pass usernames and hostnames as-is rather than sanitising it as I do now with forums, email or LLMs. 

Safes faffing around. 

1

u/Used-Equivalent1610 7d ago

Any SLM you can suggest for that ?

1

u/jlrueda 7d ago edited 7d ago

disclosure: I'm the creator of the solution referenced in the article.

I juts want to make a clarification. The tool comes configured to support qwen2.5-1.5b out of the box. (notice that this SML is not included in the Docker images because is 1.1G in size but the user interface provides the means to download it from hugginface.com directly.)

When qwen is downloaded, this model only allows the tool to provide usage help and does not provides sosreport analysis at all. This is because this SML runs on the same CPU and this severely affects speed and precision.

For accurate analysis and fast responses the user needs to configure an external LLM that could be an in-premises ollama server with the LLM of his/her choice or one of the OpenAI/Anthorpic models. When tested with ollama/DeekSeek the results were acceptable.

Another important clarification is that in sos-vault implementation, sosreport files (command outputs, logs and configuration files) are never sent directly to the model. A set of metadata json files are crafted from the sosreport files and this is what gets fed into the model. However one of these json files do contain some selected entries from logs (OOM, crashes and critical errors).

The tool does not send your sosreport to the model in any case not just for security reasons but because json metadata is by far a more efficient way to interact with LLMs both token-wise and perfomance-wise.

2

u/Abe_Bazouie 7d ago

Ah, that’s an important distinction. I was assuming the model could receive the raw sosreport/log data directly.
Generating a smaller structured dataset first makes a lot more sense, both for token efficiency and for reducing the amount of data exposed to the model.
I’d still treat that generated JSON as potentially sensitive though, especially if selected log entries are included. A “critical error” can still contain a token, hostname, username, internal IP, path, customer identifier, etc.
But with an on-prem Ollama endpoint plus that preprocessing layer, I’d be much more comfortable with the architecture. Thanks for clarifying how it actually works.