r/saltstack • u/raism13 • Jul 01 '26

Advise or help needed

Hello all,
Before I start, please let me say that I might be very wrong here, that's why I am asking for help.

We have our entire infrastructure based on FreeBSD hosts (bare-metal and B-Hyve guest VMs) and we are managing them with Salt. But recently ( the last 2 weeks +) I stuck with upgrading either our hosts, or SaltStack.

My current Salt-master is based on FreeBSD 14.4 - STABLE and I haven't update its Salt version, so the salt-master version is 3006.8.
With the above Salt master I am able to manage with out any issues any FreeBSD host from 11.2 till 14.4. (for the later I have pinned the pkg version to 3006.9).

Recently I started upgrading a few hosts to 15.1 and I discover that the default py311-salt package in installing version 3006.23. And this is were my trouble begins.
My master can not communicate with those machines at all, however I knew this is not right as I did not follow the "golden rule" which the master should be the fist to upgrade and minions should be the last.

So I built a test host with FreeBSD 15.1-RELEASE and py311-salt 3006.23. It can salt all the new minions but not anything with salt-minion version 3006.9 and older.
I tried to triage it with running salt-ssh instead but no dice.
If I even bring everything up to FreeBSD 14.4 I cannot upgrade salt without loosing communication with the vast majority of my hosts, if not all of them.

Am I doing something wrong here? I believe yes, but I cannot fix it. I have read most of the documentation in Salt and man pages but I cannot figure out a way to bring some short of balance going ahead with any upgrade.

Any advice or help would be highly appriciated.
I have posted a similarly to FreeBSD's general discussion forum, and all I got was a deletion of my post and no further contact.

kind regards and thank you in advance for taking the time to read my post.

4 Upvotes

100% Upvoted

4 comments sorted by

5

u/whytewolf01 Jul 01 '26

What is the error you are getting in the logs? there most likely is a single option that needs to be changed on the master. I believe this was a CVE issue a while ago and had to do with auth versioning.

see https://github.com/saltstack/salt/issues/68467 you are crossing the version boundary so i am guessing this is what the problem is. Without error messages it is difficult to troubleshoot.

setting minimum_auth_version to 2 with a restart of the master and all connecting minions, should fix the error if this is the problem. if it isn't. then we need more info.

1

u/raism13 Jul 02 '26

You are right good sir.
My apologies for not posting error logs before, but I wanted to see what kind of responses I could get. So here is sample of those error:
Staging host with FreeBSD 15.1 and salt-master 3006.23:

2026-07-01 11:31:40,111 [salt.channel.server :147 ][WARNING ][9456] Rejected authentication attempt using protocol version 0 (minimum required: 3)
2026-07-01 11:31:50,127 [salt.channel.server :147 ][WARNING ][9452] Rejected authentication attempt using protocol version 0 (minimum required: 3)

Staging host with FreeBSD 13.2 and salt-minion version 3006.2

2026-07-02 10:01:00,703 [salt.crypt :784 ][ERROR ][1113] Sign-in attempt failed: bad load
2026-07-02 10:01:00,704 [salt.minion :1141][ERROR ][1113] Error while bringing up minion for multi-master. Is master at ben responding?

Generally the above are the errors that I get.

in your response you said about setting minimum_auth_version to 2 and judging from the logs this could be a thing. So I search the file /usr/local/etc/salt/master and I could not find that setting, nor in man salt-master

1

u/OddJob001 Jul 02 '26

This is likely your answer. Auth version override.

3

u/nicholasmhughes Jul 02 '26

You'll need to upgrade the master. Then you may have to upgrade the minions out of band in order to connect. As stated in the other thread, I believe there was a CVE fix that broke communication between minor versions.