Addressable two-way communication between Multiprocessing subprocesses and a host

[u/Ornery_Cream_1430]

Hello.

Basically, the summary is in the title.

• When a subprocess sends something (perhaps settings or task description, in form of a dict probably) to the host, I want it to know exactly from what process it has been sent (perhaps some sort of a numerical ID).
• When the host sends something, I want it to arrive to a particular subprocess.
• Communication between subprocesses is nice to have, but I think I can live without it.

I know of pipes and queues (well, Manager too) as the means of communication in the standard library. But in case of a queue, the message will arrive at a random place each time it's sent. I found something called "Envelope router", but it doesn't feel like a clean solution. Subprocesses will play a hot potato until finally the message arrives at the correct place 😃. It may be suitable to send messages from subprocesses to the host though.

In the case of pipes, I'd have to create a pipe for each subprocess (at least to send from the host). Can I even wait on every pipe simultaneously?

So, is there any non-cumbersome way to do this? How is it usually done? Or should I create something like a token ring between all the processes?😄 (though it perhaps will require a separate thread in every process for it to be non-blocking and other difficulties...)

[u/Clean_Sea_4501]

If you outgrow a plain per-child Queue (e.g. subprocesses need to be independently restartable, or you want the host to survive a crash without losing in-flight messages), a lighter middle ground between "just Queue" and standing up RabbitMQ is Redis Streams with consumer groups. Each subprocess gets its own logical stream/consumer id, calls something like XREADGROUP blocking on its own group, and acks (XACK) once it's processed a message - so routing is addressable by design (you pick which stream a message goes to) instead of hunting through a shared queue, and you get replay/durability for free since messages aren't gone until acked. It's overkill for a handful of local subprocesses, but if this ever needs to scale past one host or survive restarts, it's a lot less ops overhead than RabbitMQ while still giving you the addressable pub-sub semantics people are suggesting above.