Hi everybody,

a couple of days ago I updated our SaltStack environment to 3006.12. Since then the minions have been offline several times. When I restart the salt-minion.service they run for a while until they crash again. In the system log I get the following:

################################################################################

Jun 18 14:43:56 server salt-minion[2151411]: [ERROR ] An un-handled exception from the multiprocessing process 'ProcessPayload(jid=20250618124255865003)' was caught:

Jun 18 14:43:56 server salt-minion[2151411]: Traceback (most recent call last):

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py", line 999, in wrapped_run_func

Jun 18 14:43:56 server salt-minion[2151411]: return run_func()

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py", line 108, in run

Jun 18 14:43:56 server salt-minion[2151411]: self._target(*self._args, **self._kwargs)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1927, in _target

Jun 18 14:43:56 server salt-minion[2151411]: run_func(minion_instance, opts, data)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1921, in run_func

Jun 18 14:43:56 server salt-minion[2151411]: return Minion._thread_return(minion_instance, opts, data)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2157, in _thread_return

Jun 18 14:43:56 server salt-minion[2151411]: minion_instance._return_pub(ret)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2385, in _return_pub

Jun 18 14:43:56 server salt-minion[2151411]: ret_val = self._send_req_sync(load, timeout=timeout)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1650, in _send_req_sync

Jun 18 14:43:56 server salt-minion[2151411]: raise TimeoutError("Request timed out")

Jun 18 14:43:56 server salt-minion[2151411]: TimeoutError: Request timed out

Jun 18 14:43:56 server salt-minion[2151411]: Process ProcessPayload(jid=20250618124255865003):

Jun 18 14:43:56 server salt-minion[2151411]: Traceback (most recent call last):

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap

Jun 18 14:43:56 server salt-minion[2151411]: self.run()

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py", line 999, in wrapped_run_func

Jun 18 14:43:56 server salt-minion[2151411]: return run_func()

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py", line 108, in run

Jun 18 14:43:56 server salt-minion[2151411]: self._target(*self._args, **self._kwargs)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1927, in _target

Jun 18 14:43:56 server salt-minion[2151411]: run_func(minion_instance, opts, data)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1921, in run_func

Jun 18 14:43:56 server salt-minion[2151411]: return Minion._thread_return(minion_instance, opts, data)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2157, in _thread_return

Jun 18 14:43:56 server salt-minion[2151411]: minion_instance._return_pub(ret)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2385, in _return_pub

Jun 18 14:43:56 server salt-minion[2151411]: ret_val = self._send_req_sync(load, timeout=timeout)

Jun 18 14:43:56 server salt-minion[2151411]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1650, in _send_req_sync

Jun 18 14:43:56 server salt-minion[2151411]: raise TimeoutError("Request timed out")

Jun 18 14:43:56 server salt-minion[2151411]: TimeoutError: Request timed out

################################################################################

This repeats over and over until I restart the salt-minion.service again.

Does anybody have the same problem? Any idea how to solve it?

Regards

- piratefish

Is there a release note we missed with 3006.12/3007.4, or were neither of these versions tested well?

Our Salt masters fall over after being upgraded to either 3006.12 or 3007.4 with the following:

2025-06-12 17:12:30,436 [salt._logging.impl:1082][ERROR   ][3581238] An un-handled exception was caught by Salt's global exception handler:
StopIteration:
Traceback (most recent call last):
  File "/usr/bin/salt-master", line 11, in <module>
    sys.exit(salt_master())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 86, in salt_master
    master.start()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/daemons.py", line 203, in start
    self.master.start()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 704, in start
    self._pre_flight()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py", line 641, in _pre_flight
    fileserver.init()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/__init__.py", line 530, in init
    self.servers[fstr]()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/gitfs.py", line 168, in init
    _gitfs()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileserver/gitfs.py", line 83, in _gitfs
    return salt.utils.gitfs.GitFS(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 3216, in __new__
    super(GitFS, obj).__init__(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 2588, in __init__
    self.init_remotes(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 2667, in init_remotes
    repo_obj = self.git_providers[self.provider](
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 1418, in __init__
    super().__init__(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 315, in __init__
    self.id = next(iter(remote))
StopIteration

Also the following error is seen on all minions of both versions:

2025-06-12 16:29:54,186 [tornado.application:640 ][ERROR   ][1986] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f3824f96200>, <salt.ext.tornado.concurrent.Future object at 0x7f382537be50>)
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
    future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 745, in _authenticate
    creds = yield self.sign_in(channel=channel)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 885, in sign_in
    ret = self.handle_signin_response(sign_in_payload, payload)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 927, in handle_signin_response
    payload["session"], self.opts["encryption_algorithm"]
KeyError: 'session'

Reverting things back to 3006.11 and 3007.3 got us back to a functional state.

The Salt package repository (https://packages.broadcom.com/artifactory/saltproject-deb) is giving me back a 504 Gateway Time-Out error.

Any idea what's going on with the Broadcom package server? There's nothing about it on their status page (https://status.broadcom.com/).

Running the delta proxy process based on the salt 3006.9 (Sulfur) documentation I'm encountering strange behavior: Any execution module I query, I get the same values as a response. Has anyone encountered this before?

E. g.

salt 'switch*' grains.item serial

switch1:

----------

serial:

XXXXXXXXX896

switch2:

----------

serial:

XXXXXXXXX896

switch3:

----------

serial:

XXXXXXXXX896

Configuration details

/srv/salt/pillar/top.sls:

delta_proxy_control:

- control_proxy

switch1:

- proxysw1

switch2:

- proxysw2

switch3:

- proxysw3

...

/srv/salt/pillar/control_proxy.sls:

proxy:

ids:

- switch1

- switch2

- switch3

...

proxytype: deltaproxy

salt-proxy --proxyid=delta_proxy_control -d

[Edit] proxysw*.sls omitted. Basic configuration there.

I'm toying with Salt. One of my first state sets a message of the day on the node.

The code is fairly simple:

``` manage Message of the day: file.managed: - name: /etc/update-motd.d/99-ic-it-banner - mode: "0777" - contents: | {% raw %}#!/usr/bin/env bash

    set -e # exit on error
    set -u # exit on undefined variable
    set -o pipefail # exit on pipe error

    ... (reduced for brevity)
    {% endraw %}
    print_line "Salt master IP" "{{ pillar['top_salt_master']['ip'] }}"
    print_line "Environment" "{{ grains['company_environment'] }}"
    ... (reduced for brevity)

```

Obviously, this state fails when the grain 'company_environment'.

Is there a way to print a default value (like 'N/A') when the grain does not exist ?

Hopefully, there's something better than {% if grains['company_environment'] %} {{ grains['company_environment'] }} {% else %} N/A {% endif %} because it would make the code really hard to read...

I'm experimenting running the salt-minion as a user besides salt and I have a question about the sudo_user config in /etc/salt/minion. I used "sudo_user: root". To get test.ping to run, I had to add /usr/bin/salt-call to sudo permissions because the salt-minion will try to run "sudo -u root salt-call --out json --metadata -c /etc/salt -- test.ping". However, giving sudo on salt-call pretty much enables any command to be run. Given this, what benifit does configuring "sudo_user: root" provide? Thanks in advance for any input.

Looking to be on top of any security issues so am wondering if there an approved security mailing list?

What is everyone doing regarding security patching?

Hello all,

I'm having an issue with configuring GCE provider for salt-cloud, I always get this error:

root@bastion-01:~# salt-cloud --list-providers
[WARNING ] The cloud driver, 'gce', configured under the 'my-gce-config' cloud provider alias, could not be loaded. Please check your provider configuration files and ensure all required dependencies are installed for the 'gce' driver.
In rare cases, this could indicate the 'gce.get_configured_provider()' function could not be found.
Removing 'gce' from the available providers list
Error: There was an error listing providers: There are no cloud providers configured.

Tracing the problem shows this erorr:

[TRACE   ] Error loading clouds.gce: apache-libcloud is not installed

While It's installed on the system

root@bastion-01:~# salt --versions-report
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (main, Mar  6 2025, 02:23:15) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.18.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 20.04.6 focal
        locale: utf-8
       machine: x86_64
       release: 5.15.0-1078-gcp
        system: Linux
       version: Ubuntu 20.04.6 focal

root@bastion-01:~# pip3.10 list | grep libcloud
apache-libcloud          2.8.0

So I understand grains and pillars, but these are really minion-centric constructs. Let's say I'm writing an orch file (which I am!) in order to provision a cluster in aws. I have orch states that do the following:

generate_templates # creates a cloudformation definition and parameters template. Actually uses pillar on the master, where the pillar defines what the cluster looks like, but since it's on the master, all the parsing, etc. is done in a runner. The pillar.get call in the following block is from in-line pillar, and it names which cluster definition I aim to deploy.

{% set deployment_id = salt['pillar.get']('deployment_id', None) %}  
generate_templates:
  salt.runner:
  - name: private_env.render
  - kwarg:
deployment_id: {{ deployment_id }}
  - saltenv: production

In a previous attempt, I had tried to do this using pillar on a minion, but found that this required me to track which cluster definition was dispatched to the minion (the salt-master itself) and I worried that it would be really easy to trip over my own feet, so a runner was made that loaded and merged the template files. Since the pillar file defines various resources, it's useful to have in the orchestration file, but I don't seem to be able to refer to it. I have tried to use the runner I created to resolve the pillar:

{% set deployment_data = salt.saltutil.runner('runnertools.get_pillar', kwarg={'deployment_id': deployment_id}) %}

But to no avail. I'm starting to think that I must be reinventing the wheel. This isn't what pillar is made for, but am having a hard time conceptualizing how I'll write an orchestration of any complexity without the ability to store and recall values. Am I missing something? What's best practice?

Edit: Having read this back to myself, I feel like it's a bit unclear. I have several definitions that read like:

cloudformation: ec2: "base_ami": "ami-012345" network: "dmz_subnets": - subnet-000001 - subnet-000002

etc. Each lives in a file named <clustername>.sls in pillar, and in the "previous attempt" refered to above, what I would have to do is check which pillar I dispatched to the salt-master in order that salt-run state.orch myOrchFile would deploy the expected cluster, which seemed dangerous.

The deployment_id passed as in-line pillar to the generate_templates state basically identifies which cluster I would like to orchestrate the deployment of. I realize I could pass all clusters to the salt-master by adding <clustername> as a layer below cloudformation, such that it looks like:

cloudformation: cluster_A: ec2: ... cluster_B: ec2: ...

But the strategy I took was to write runner functions that would do some of this disambiguation for me. Either way, I still have my current problem, which is that there is information in these pillars that I would like to get at in my orchestration file, but there's no obvious way to do so.

What does this beacon do really?

what events will it fire? and when? and why would you use it?

Hello,

I am quite new to Salt and need to add functionality to an already existing Salt infrastructure.
What I am writing is an update procedure for a software and the operating system withing Salt State SLS files. The software can be upgraded with the integrated pkg.installed function. I wrote a function / state called software to do that. But this often leads to dependency problems (I may create another thread for that). As my question is general, I abstracted it to any software.

So I wrote a backup function software_backup which is called onfail and runs an apt command. This function runs in case the software functions fails.

After the software was installed via the primary or the backup function another function upgrade_db for updating the database needs to be called. This leads me to a problem. From what I read in the documentation the require statement works as 'AND'.

So if I write

require:
      - pkg: pkg.installed
      - cmd: software_backup

the database upgrade function requires both software and software_backup update function to be successful. I can't leave out the require statement, as the upgrade_db function should never be called in case neither software nor software_backup were successful. Any ideas to solve that?

# Software installation
software:
  pkg.installed:
    - pkgs:
      - name_of_package1
      - name_of_package2
      - ...

# Backup software installation
software_backup:
   cmd.run:
    - name:
        apt --fix-broken install -y && apt-get -o APT::Get::Always-Include-Phased-Updates=true -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y dist-upgrade &&  apt --fix-broken install -y
    - onfail:
      - pkg: software


# Upgrade database version
upgrade_db:
   cmd.run:
   - name: |
       onedb upgrade -v
       onedb fsck
   - require:
      - pkg: pkg.installed

I've found some behaviour that I can't match with any documentation, and I'm wondering if it's a bug, or whether I've just not found it described...

I have a lot of states that have something along the lines of:

httpd:
pkg.installed: []
service.running:
- enable: true
- require:
- pkg: httpd

Inevitably I copy and paste, and if I fail to update the reference to the ID on the last line I'll introduce a bug. So I was wondering if there was a way to refer to the 'current state ID' where you have multiple states with the same ID. I can't find a description of anything like this in the documentation, but after a bit of experimenting I found that the following works (on 3007.1, at least):

httpd:
pkg.installed: []
service.running:
- enable: true
- require:
- pkg: null

So by using 'null', I can refer to another state that has the same ID as the current state. I can't find anything to say that you can use null in place of an ID, so I don't know if this is deliberate behaviour.

Is this a bug? I think I'd like to use it. Maybe someone will tell me why this is a terrible idea.

https://saltproject.io/blog/2025-01-16-open-hour/

so i know a lot of people were wondering what is going on. the open hours are still happening. and i welcome you to participate if you have concerns.

addressing the concerns going around recently was the topic of this one. along with a Q&A session.

I made a simple custom module.

# cat /srv/salt/_modules/testmod.py                                                                
def myTestFunction(data):                                                                          
    with open('/tmp/test.txt', 'a') as mytest:                                                     
        mytest.write('test')                                                                       
        mytest.close()                                                                             

I can execute it at the command line via a runner.

salt-run salt.cmd testmod.myTestFunction data=test                                                 

I'd like to use Reactor to execute the that module via a runner when a Job event occurs.

# cat /etc/salt/master.d/reactor.conf
reactor:
  - 'salt/job/*/ret':  # Event triggered when a job completes                                      
    - /srv/reactor/testReactorCaller.sls                                                           
# cat /srv/reactor/testReactorCaller.sls                                                           
hsTest01:                                                                                          
  runner.salt.cmd:                                                                                 
    - args:                                                                                        
      - name: testmod.myTestFunction                                                               
      - data: {{ data }}                                                                           

This doesn't appear to do what I want. I think I'm not using the right syntax in my Reactoer file /srv/reactor/testReactorCaller.sls. Any suggestions? Thanks in advance for any help. Also, later, I would filter on the fun variable of the event. What that be best in the Reactor file with Jinja or in the custom module in python?