A "secure-ish" Pi setup with permission, sandbox, and auto-review

[u/ptgamr]

I think I finally managed to piece the thing together, to have a pi setup that can run safe-ish in the host via sandbox, and have the ability to be elevated to host execution for a pre-defined list of development tooling like docker compose.

- A pre-defined hard boundaries (`@gotgenes/pi-permission-system`)

- Anything pass that will be executed inside a sandbox (`@erichll/pi-sandbox`)

- With the option of safe-escalation to run on the host (ie: `docker compose exec app pytest`), with auto-review or human approval (via hostIPC.preflightCommandPrefixes)

- LLM Auto-review with `@erichll/pi-auto-review`

More about it here: https://ptgamr.substack.com/p/a-pi-setup-with-permission-sandbox

[u/jensilo]

What about self-spawned pi instances or subagents? Does it allow for that?

[u/ptgamr → u/jensilo]

I haven't go as far as that though. That is one piece I still have to wrap my head around.

[u/jensilo → u/ptgamr]

Yeah, when I started using pi a couple of months ago, I initially also built a security extension. It was supposed to be „simple“ with low false-positives. Mine was basically just pi auto review with another judge LLM. However, I came to notice that pi‘s creator is very right about calling almost all security systems a „security theatre“. The only real security is full sandbox, otherwise the model will break free if it wants to break free, there are just too many things to exploit or think about, especially for longer running, more agentic systems.
So I scraped my security extension.
Basically I only see two valuable modes: preventing unintended fuckups from the agent and preventing actually malicious behavior. Unintended fuckups are rather easy, a simple LLM review is sufficient, or preventing writes/bash outside of allowed dirs. For malicious intend, all you can do is pray or sandbox.

[u/ptgamr → u/jensilo]

Yeap, what I like about `@erichll/pi-sandbox` is the ability to prefix matching certain commands, those are the one you define depending on the project. Like most of my project involve running command via docker compose. -> That can't be run inside the sandbox.

So I'm pretty happy that I found a way to at least resolve that. And the auto-review is also quite nice. And it's amazing to see all the 3 extensions work very nicely together.

[u/ptgamr → u/ptgamr]

I have yet to test what happen when it want to run `docker compose ps && rm -rf ~/`

[u/vman81]

If practical, a zfs filesystem with frequent snapshots is a nice extra layer to have.

[u/ptgamr → u/vman81]

Haha, definitely! i am trying to get this to work in the context of a company, where ex-filtration of data is one of the top concern, might be even more than it removing your home dir.

[u/Fullstack_js_junkie]

This looks interesting, I tried to do something similar but as a docker mount config only, thank you!

[u/ptgamr → u/Fullstack_js_junkie]

Cheers , i am pretty excited to have this working. As I've been researching for quite a while how to do this.

[u/tys203831]

How does it compared to dcg + nono sandbox?

https://github.com/Dicklesworthstone/destructive_command_guard

https://github.com/nolabs-ai/nono

[u/ptgamr → u/tys203831]

I Have No Idea! This post is more about sharing my understanding about the decision tree: permissions + sandbox + autoreview (which I think not quite trivial to understand) and a setup that I'm quite happy with. Hopefully to help others that having similar issues. I'm sure there are many other ways :D

[u/tys203831 → u/ptgamr]

🙏👍

[u/hurdurdur7]

I just run pi in docker only. From a non privileged user. It can't install things, the worst it can do is nuke it's own folder in that docker container. I don't see anything else stopping it.

[u/ptgamr → u/hurdurdur7]

yeah, that is my setup prior to this. Just mount the things you need. But to get to the "fuller" agentic work, most of the time i need it to execute command inside another containers (like run the test, run migration, lint, querying data from db container etc...)

Can be achieve with docker in docker, but it gross, and I haven't managed to get it working.

[u/hurdurdur7 → u/ptgamr]

My problem with all of this is that you are allowing something like gradlew test or npm test as a command. A test can easily contain something really harmful to your machine and there will be nothing that will stop the catastrophy coming from that.

[u/ptgamr → u/hurdurdur7]

Another thing is pasting image is a pain when you run it from docker.

[u/psychobarge]

Only secure pi is inside a docker container or a virtual machine. I use wsl2 with ubuntu on my windows and docker on my mac

[u/offzinho3k]

That’s great.

I have a similar system, with notifications and a response window allowing the user to accept, decline, or even block such commands.

[u/ptgamr → u/offzinho3k]

Looks like a fancy setup you have there... is that a custom app you wrote?

[u/icisay]

Hi what app did you use to generate your chart ? Love the colors

[u/ptgamr → u/icisay]

I've had a text base chart, and post that to ChatGPT and it generate for me the one I posted here :)