infrastructure-notes/MDR Salt Upgrade.txt

MDR Salt Upgrade.txt
https://jira.mdr.defpoint.com/browse/MSOCI-1164 

Done when:

All salt minions are running same version (2018)
All server minions are pegged to specific version (that can be changed at upgrade time)
Remove yum locks for minion

Notes:

Packer installs 2019 repo (packer/scripts/add-saltstack-repo.sh & packer/scripts/provision-salt-minion.sh) , then os_modifications ( os_modifications.repo_update) overwrites the repo with 2018. This leaves the salt minion stuck at the 2019 version without being able to upgrade. 

#salt master (two salt repo files)

/etc/yum.repos.d/salt.repo (salt/fileroots/os_modifications/minion_upgrade.sls)

[salt-2018.3]
name=SaltStack 2018.3 Release Channel for Python 2 RHEL/Centos $releasever
baseurl=https://repo.saltstack.com/yum/redhat/7/$basearch/2018.3
failovermethod=priority
enabled=1
 
/etc/yum.repos.d/salt-2018.3.repo

[salt-2018.3]
name=SaltStack 2018.3 Release Channel for Python 2 RHEL/Centos $releasever
baseurl=https://repo.saltstack.com/yum/redhat/7/$basearch/2018.3
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/saltstack-signing-key, file:///etc/pki/rpm-gpg/centos7-signing-key
 

#reposerver.msoc.defpoint.local
/etc/yum.repos.d/salt.repo

[salt-2018.3]
name=SaltStack 2018.3 Release Channel for Python 2 RHEL/Centos $releasever
baseurl=https://repo.saltstack.com/yum/redhat/7/$basearch/2018.3
failovermethod=priority
enabled=1
gpgcheck=0
Two repo files in salt, both are 2018.3; one has proxy=none other doesn't.  the salt_rhel.repo is just for RHEL and the other is for CENTOS. 

salt/fileroots/os_modifications/files/salt.repo (salt/fileroots/os_modifications/repo_update.sls uses this file and it is actively pushed to CENTOS minions)

salt/fileroots/os_modifications/files/salt_rhel.repo  (salt/fileroots/os_modifications/repo_update.sls uses this file and it is actively pushed to RHEL minions)



/etc/yum.repos.d/salt-2018.3.repo ( not sure how this file is being pushed. possibly pushed from Chris fixing stuff )


STEPS
1. remove /etc/yum.repos.d/salt-2018.3.repo from test
1.2 remove yum versionlock in test (if there are any; None found)
1.3 yum clean all ; yum makecache fast
2. use git to update os_modifications/files/salt_rhel.repo file to 2019.2.2 ( match salt master)
2.1 use salt + repo to update minion to 2019.2.2
2.5 salt minion cmd.run 'rm -rf /etc/yum.repos.d/salt-2018.3.repo'
2.5.1 salt minion cmd.run 'ls /etc/yum.repos.d/salt*'
2.6 salt salt-master* state.sls os_modifications.repo_update
2.7 salt salt-master* cmd.run 'yum clean all ; yum makecache fast'
2.8 salt minion cmd.run 'yum update salt-minion -y' 
2.9 salt minion cmd.run 'yum remove salt-repo -y'
3. upgrade salt master to 2019.2.3 using repo files as a test
4. upgrade salt mininos to 2019.2.3 using repo files as a test
5. push to prod. 





PROBLEMS
bastion.msoc.defpoint.local
error: unpacking of archive failed on file /var/log/salt: cpio: lsetfilecon
mailrelay.msoc.defpoint.local
pillar broken


PROD

1. remove dup repos
1.1 remove /etc/yum.repos.d/salt-2018.3.repo from environment (looks like it was installed with a RPM) 
1.1.1 salt minion cmd.run 'yum remove salt-repo -y' (does not remove the proper salt.repo file)
1.1.2 salt minion cmd.run 'rm -rf /etc/yum.repos.d/salt-2018.3.repo'   (just to make sure)
1.2 remove yum versionlock
 yum versionlock list
1.2.1 salt minion cmd.run 'yum versionlock delete salt-minion'
1.2.2 salt minion cmd.run 'yum versionlock delete salt'
1.2.3 salt minion cmd.run 'yum versionlock delete salt-master'
2. use salt + repo to update master/minion to 2019.2.2
2.1 use git to update os_modifications/files/salt_rhel.repo file to 2019.2.2 pin to minor release (match TEST)(https://repo.saltstack.com/yum/redhat/$releasever/$basearch/archive/2019.2.2)
2.2 Check for environment grain ( needed for repo_update state file. )
2.2.1 salt minion grains.item environment
2.6 salt salt-master* state.sls os_modifications.repo_update
2.7 salt salt-master* cmd.run 'yum clean all ; yum makecache fast'
2.7.5 salt minion cmd.run 'yum check-update | grep salt'
2.8 salt minion cmd.run 'yum update salt-minion -y' 
OR salt minion pkg.upgrade name=salt-minion
  salt minion pkg.upgrade name=salt-minion fromrepo=salt-2019.2.4
2.9 salt master cmd.run 'yum update salt-master -y'
3. ensure salt master and minions are at that minor version. 
3.1 salt * test.version
6. upgrade test and prod to 2019.2.3 via repo files to ensure upgrade process works properly. 
6.5 fix permissions on master to allow non-root users to be able to run ( or run highstate )
6.5.1 chmod 700 /etc/salt/master.d/
6.5.2 then restart master
7. never upgrade salt again. 

PROBLEMS
the pillar depends on a custom grain, the custom grain depends on specific python modules. the moose servers seem to have python module issues. 
these commands helped fix them. python yum VS. pip 
ImportError: cannot import name certs
pip list | grep requests
yum list installed | grep requests
sudo pip uninstall requests
sudo pip uninstall urllib3
sudo yum install python-urllib3
sudo yum install python-requests
pip install boto3 (this installs urllib3 via pip as a dependency!)
pip install boto


slsutil.renderer salt://os_modifications/repo_update.sls
if the grain is wrong on the salt master, but correct with salt-call restart the minion. 

salt moose* grains.item environment
cmd.run 'salt-call grains.get environment'
cmd.run 'salt-call -ldebug --local grains.get environment'
cmd.run 'salt-call -lerror --local grains.get environment'

boto3 issue
on indexers python3 is installed and pip points to python3 not python2
/usr/local/lib/python3.6/site-packages/pip

Salt root is setup with python3
salt moose-splunk-indexer-1* cmd.run 'pip install boto3'
salt 'moose*indexer*' cmd.run 'pip install boto3'

salt-call is different connecting to python2
/bin/bash: pip: command not found
salt 'moose*indexer*' cmd.run "salt-call cmd.run 'pip install boto3'"


resolution steps
Duane will remove /usr/local/bin/pip which is pointing to python3
pip should be at /use/bin/pip
yum --enablerepo=epel -y reinstall python2-pip

to proceed:
1. install boto3 via pip
2. salt '*.local' cmd.run 'pip install --upgrade urllib3'


Permissions issue? Run this command as root:
salt salt* state.sls salt_master.salt_posix_acl