UCS Performance Manager is another “wonderful” product of Cisco for monitoring your physical and virtual infrastructure.
Requirements:
- 8 vCPUs
- 40 GB of memory
The upgrade should be easy, but it’s not. My upgrade process was stuck due to the process zenmail blocked in running mode. Then to be able to upgrade I had to open 2 shell sessions:
- 1 with the upgrade process
- 1 with the root shell
During the process, I have got a quick look at /mnt/cdrom/update-zenoss.sh
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 |
#!/bin/bash # Path to all the resources on the CD. ISOMOUNT="/mnt/cdrom" MANIFEST_FILE=$ISOMOUNT/build/manifest.json ZENOSS_TYPE=$(python -c "import json; print json.load(open('$MANIFEST_FILE'))['type']") LOGFILE="/tmp/upgrade-$ZENOSS_TYPE-$(date +'%Y-%m%d-%H%M%S').log" > $LOGFILE exec > >(tee -a $LOGFILE) exec 2> >(tee -a $LOGFILE >&2) ## PREREQUISITES AND CONTEXT function red () { tput setaf 1 echo $@ tput sgr0 } function green () { tput setaf 2 echo $@ tput sgr0 } function failure_prompt () { red $@ >&2 red Upgrade log stored at $LOGFILE. read -n1 -r -p "Upgrade failed. Press any key to return." exit 1 } green Upgrade log stored at $LOGFILE. THINPOOL_DEVICE=$(source /etc/default/serviced; echo $SERVICED_DM_THINPOOLDEV) if [ "$THINPOOL_DEVICE" = "" ]; then red "Serviced appears to be using a loopback thinpool device." red "It is highly recommended that you migrate to an LVM device for improved performance." echo read -n1 -r -p "Press Escape to abort now, or any other key to continue." II echo if [ "$II" = $'\e' ]; then exit 1 fi fi # Setup variables from manifest file. DOCKER_IMAGES=$(python -c "import json; print ' '.join(json.load(open('$MANIFEST_FILE'))['images'].values())") RPM_IMPORT4=$(python -c "import json; print json.load(open('$MANIFEST_FILE'))['rpms']['import4']" 2>/dev/null) RPM_SERVICED=$(python -c "import json; print json.load(open('$MANIFEST_FILE'))['rpms']['serviced']") RPM_SVCDEF=$(python -c "import json; print json.load(open('$MANIFEST_FILE'))['rpms']['svcdef']") ZENOSS_IMAGE=$(python -c "import json; print json.load(open('$MANIFEST_FILE'))['images']['zenoss'][len('install-zenoss-'):-len('.run')]") ZENOSS_SHORT=$(python -c "import json; print json.load(open('$MANIFEST_FILE'))['images']['zenoss'][len('install-zenoss-'):].split(':')[0]") ZENOSS_VERSION=$(python -c "import json; print json.load(open('$MANIFEST_FILE'))['zenoss-version']") IS_MASTER=$(source /etc/default/serviced; echo $SERVICED_MASTER) if [ "$IS_MASTER" = "" ]; then IS_MASTER="1" fi # Update admin menu. \cp -r -f $ISOMOUNT/common/appliance/standard/* /root/appliance \cp -r -f $ISOMOUNT/common/appliance/$ZENOSS_TYPE/* /root/appliance ## UTILITY FUNCTIONS function stop_all_services () { serviced service stop $ZENOSS_TYPE 2>/dev/null RETRIES=60 STATUS="" until [ "$STATUS" = "Stopped" ] || [ "$RETRIES" -le 0 ]; do STATUS=$(serviced service status --show-fields 'Status' 2>/dev/null | grep -v "Stopped" | grep -v "Status" | grep -v ^[[:space:]]*$) if [ -z "$STATUS" ]; then STATUS="Stopped" fi sleep 5 ((RETRIES--)) done if [ "$RETRIES" -le 0 ]; then failure_prompt "Failed to stop all $ZENOSS_TYPE services." fi } function restart_serviced () { green "Restarting $ZENOSS_TYPE..." systemctl unmask serviced --quiet systemctl enable serviced --quiet systemctl restart serviced --quiet STATUS="" RETRIES=60 until (serviced healthcheck &> /dev/null) || [ "$RETRIES" -le 0 ]; do sleep 5 ((RETRIES--)) done if [ "$RETRIES" -le 0 ]; then failure_prompt "Failed to restart serviced." fi green "Restarted $ZENOSS_TYPE." } function create_thinpool_check() { if [ ! -f "$1" ]; then cat <<EOF > $1 #!/bin/bash function check_for_thinpools () { if [ ! -e /dev/mapper/docker-docker--pool ]; then exit 1 fi if [ ! -e /dev/mapper/serviced-serviced--pool ]; then exit 1 fi } while ( ! check_for_thinpools ); do sleep 5 done EOF chmod +x $1 fi } ## UPGRADE PROCEDURE green "Stopping $ZENOSS_TYPE services..." stop_all_services green "$ZENOSS_TYPE services stopped." green "Stopping serviced..." systemctl stop serviced --quiet STATUS="" RETRIES=60 until [ "$STATUS" = "inactive" ] || [ "$STATUS" = "failed" ] || [ "$RETRIES" -le 0 ]; do STATUS=$(systemctl show serviced 2>/dev/null | awk -F= '/ActiveState/ { print $2 }') sleep 5 ((RETRIES--)) done if [ "$RETRIES" -le 0 ]; then failure_prompt "Failed to stop serviced." fi systemctl mask serviced --quiet green "Serviced stopped." green "Updating system packages..." cat <<EOF > /etc/yum.repos.d/centos-update-mirror.repo [centos-update-mirror] name=Centos update mirror baseurl=file://$ISOMOUNT/centos-repo enabled=0 gpgcheck=0 EOF yum update -y --disablerepo=\* --enablerepo=centos-update-mirror rm -f /etc/yum.repos.d/centos-update-mirror.repo green "System packages updated." cd $ISOMOUNT/build # Replace old yum-mirror, if any, with new one. RHEL_VERSION=$(awk '{print $4}' /etc/redhat-release) YUM_MIRROR=$(python -c "import json; print json.load(open('$MANIFEST_FILE'))['yum-mirror-centos$RHEL_VERSION']") OLD_MIRROR=$(yum list --disablerepo=\* | awk '/^yum-mirror-centos/ { print $1}') if [ ! -z "$OLD_MIRROR" ]; then yum remove -y $OLD_MIRROR &> /dev/null fi yum remove -y yum-mirror-ucspm-unstable yum install -y $YUM_MIRROR 2>/dev/null yum clean all # Modify the serviced configs before it's upgraded and restarted. if ! grep -xq 'SERVICED_DOCKER_REGISTRY=.*' /etc/default/serviced; then echo 'SERVICED_DOCKER_REGISTRY=localhost:5000' >> /etc/default/serviced fi if [ "$THINPOOL_DEVICE" = "" ]; then echo 'SERVICED_ALLOW_LOOP_BACK=true' >> /etc/default/serviced fi green "Importing $ZENOSS_TYPE Docker images..." systemctl restart docker --quiet for IMAGE in $DOCKER_IMAGES; do ./$IMAGE -y &> /dev/null IMAGE_NV=${IMAGE%%.run} IMAGE_NV=${IMAGE_NV##install-} IFS=":" read -ra IMAGE_INFO <<< "$IMAGE_NV" IMAGE_NAME=$(echo ${IMAGE_INFO[0]} | tr '-' '.') IMAGE_VERSION=${IMAGE_INFO[1]} IMAGE_INSTALLED=$(docker images | grep $IMAGE_NAME | grep $IMAGE_VERSION) if [ -z "$IMAGE_INSTALLED" ]; then failure_prompt "Error importing $IMAGE." fi done green "Imported $ZENOSS_TYPE Docker images." # Updating serviced will also upgrade Docker. green "Upgrading serviced and dependencies..." yum install -y --disablerepo=\* --enablerepo=zenoss-mirror ${RPM_SERVICED%%.rpm} if [ $? -ne 0 ]; then failure_prompt "Failed to upgrade serviced binary." fi green "Upgraded serviced and dependencies." DOCKER_CONF="/etc/systemd/system/docker.service.d/docker.conf" if [ -f "$DOCKER_CONF" ]; then # Switch Docker DNS to support Docker 1.9 upgrade. DOCKER_DNS=$(ip addr show docker0 | grep inet | grep -v inet6 | awk '{print $2}' | awk -F / '{print $1}') DOCKER_BIP=$DOCKER_DNS/24 sed -i -e "s;--dns=.*;--dns=$DOCKER_DNS --bip=$DOCKER_BIP;" "$DOCKER_CONF" # Set Docker startup timeout TimeoutSec=$(source $DOCKER_CONF &>/dev/null; echo $TimeoutSec) if [ -z "$TimeoutSec" ]; then echo "TimeoutSec=300" >> "$DOCKER_CONF" elif [ "$TimeoutSec" -lt 300 ]; then sed -i -e "s/TimeoutSec=.*/TimeoutSec=300/" "$DOCKER_CONF" fi # Add devicemapper wait if not using thinpools if [ "$THINPOOL_DEVICE" != "" ]; then # Create thinpool check script THINPOOL_SCRIPT=/root/appliance/docker-thinpool-check.sh create_thinpool_check $THINPOOL_SCRIPT THINPOOL_PRE=$(grep $THINPOOL_SCRIPT $DOCKER_CONF) if [ -z "$THINPOOL_PRE" ]; then echo "ExecStartPre=$THINPOOL_SCRIPT" >> "$DOCKER_CONF" fi systemctl daemon-reload fi fi if [ "$IS_MASTER" = "1" ]; then # Serviced must be running for 'serviced docker sync' to succeed. restart_serviced serviced docker sync green "Installing new service definition..." yum install -y --disablerepo=\* --enablerepo=zenoss-mirror $RPM_IMPORT4 $RPM_SVCDEF 2> /dev/null green "New service definition installed." stop_all_services # If this is the master, we cannot proceed with the upgrade until the agents have been upgraded first. if [ "$IS_MASTER" = "1" ]; then OLD_AGENTS="" MY_INFO=$(serviced host list --show-fields=Addr,Release | grep $(ifconfig | awk '/inet / { print "-e" $2 }')) if [ -z "$MY_INFO" ]; then red "Could not determine this host's address and release. Cannot determine if all agents are updated." echo read -n1 -r -p "Press Enter to continue." else MY_IP=$(awk '{print $1}' <<< "$MY_INFO") MY_RELEASE=$(awk '{print $2}' <<< "$MY_INFO") AGENT_RELEASES=$(serviced host list --show-fields=Addr,Release | grep -v Addr | grep -v -F "$MY_IP") OLD_AGENTS=$(grep -v -F $MY_RELEASE <<< "$AGENT_RELEASES") fi if [ ! -z "$OLD_AGENTS" ]; then red "This host appears to be the master host." red "One or more agents do not appear to have been upgraded yet." serviced host list --show-fields=Name,Addr,Release | grep -v -F $MY_IP echo failure_prompt "Please upgrade agent hosts first before upgrading this host." fi fi # Pull the zenoss image, rsync to /root/5.Z.x green "Updating" $ZENOSS_TYPE "..." ZENOSS_5DOT=${ZENOSS_VERSION%?}x docker run -it --rm -v /root:/mnt/root zenoss/${ZENOSS_IMAGE} rsync -a /root/${ZENOSS_5DOT} /mnt/root sed -i -e "s/^SVC_USE zenoss\/${ZENOSS_SHORT}:5.* /SVC_USE zenoss\/${ZENOSS_IMAGE} /g" /root/${ZENOSS_5DOT}/upgrade-${ZENOSS_TYPE}.txt /root/${ZENOSS_5DOT}/upgrade-${ZENOSS_TYPE}-${ZENOSS_5DOT}.sh if [ $? -ne 0 ]; then red "Error upgrading $ZENOSS_TYPE." >&2 failure_prompt "You may need to remove one or more preupgrade snapshots before retrying this upgrade." fi green "Updated $ZENOSS_TYPE." stop_all_services fi systemctl stop serviced --quiet green "$ZENOSS_TYPE upgraded successfully." read -n1 -r -p "Press any key to reboot..." systemctl unmask serviced systemctl enable serviced green Upgrade log stored at $LOGFILE. reboot |
The script was hanging every time, it had to stop all services with serviced daemon.
After a quick look, the interesting line was: serviced service status --show-fields 'Status' 2>/dev/null | grep -v "Stopped" | grep -v "Status" | grep -v ^[[:space:]]*$
During 60 retries, the result was “Running” but to go further, it should be “Stopped”.
One service was still running every time: zenmail
To stop the zenmail process:
|
1 |
serviced service status |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
Name ServiceID Status HC Fail Uptime RAM Cur/Max/Avg Hostname InSync DockerID └─ucspm 4bpw93i5acfbpgofx7fhdcts5 Stopped 1G ├─Zenoss 1wttgp70u7rr1hp5hsekaf5ok 0 │ ├─Events 3fde50vxwmsv3iiaa9l7v6h6h 0 │ │ ├─zeneventd 2c1nj565tsud6xa6fmtar8ya9 Stopped 1G │ │ ├─zenactiond 65zdx83tz84wg6zp7rhkz304v Stopped 1G │ │ └─zeneventserver 74l937xhigujnvuvyo9x4165g Running X 13m39s 4G 1.2G / 1.6G / 653.5M MUT0257AV Y ab9b85560da0 │ ├─Metrics 59472xtm2k2vqvbc7k3vyluyj 0 │ │ ├─MetricShipper amkmzv3usnv94yp87btyzgh0j Stopped 256M │ │ ├─MetricConsumer ej4gfzjrsz8jbdfb7l31zefsj Stopped 1G │ │ └─CentralQuery erw5pa9iwjjxmmvscvnqzycur Stopped 1G │ ├─User Interface bi300eavwhrtfdd20h098ufpm 0 │ │ ├─zenjobs 3raxfni42bz0ag0s1j6b2i9m9 Stopped 1G │ │ ├─zenjserver 4cye93z8j2rvxv8oddn7bbn3y Stopped 1G │ │ ├─Zope d2keujmor79xfifa9kjlb93d7 Stopped 1G │ │ └─Zauth emz4rt91lsrwf8mbx30h155wi Stopped 1G │ └─Collection cmwtkr56kpfw82ollsjww4ugx 0 │ └─localhost agjty667yjci4rj530c4wir98 0 │ ├─zenhub 2g6ilamrg1q2qby6j5ry1vne5 Stopped 1G │ └─localhost 8ixkyatz9ozvd0zdxg8655i9w 0 │ ├─zenucsevents 1l5h6nxxqqg1ykjk4yokzog8w Stopped 1G │ ├─zenping 2yuckbqa6k4dquttb63citugb Stopped 1G │ ├─zenperfsnmp 36vqrbl3r5aeiupbwvbblr4ml Stopped 1G │ ├─zencommand 3nysn2vsmjkrbc41sl20n37o9 Stopped 1G │ ├─zenmail 6bnlfl5qzog18j4mrnxszllyx Stopped 1G │ ├─zminion 7fs2tbiw227voujzjljiqpxn7 Stopped 256M │ ├─zenvsphere ae8pgjgpr4ci9r7cb20nfgvqq Stopped 1G │ ├─MetricShipper bxl1uxunffbaqj5p41py43s02 Stopped 1G │ ├─zenpython cl48l5pn0jipt6jj61g51qe7a Stopped 1G │ ├─zenmodeler cso37jg9vvki7zp35fm5vyxdx Stopped 1G │ ├─collectorredis cw9y1ype76ufi3jn2o5371m7g Stopped 1G │ └─zenpropertymonitor dzn0u3g09qf04t3e1fcki3j8r Stopped 256M └─Infrastructure aopt5zq50uted7ptnd6avjvxe 0 ├─Imp4MariaDB 2grmsfeannpimx6pov02i3gt6 Stopped 256M ├─mariadb-model 46f3tnue52nmrbnm07n0xkzdv Running 13m41s 2G 537.2M / 1.9G / 1.8G MUT0257AV Y ed3ea6b27095 ├─memcached 4qa68pzradzugromxq7umdb4v Stopped 1G ├─HBase 5ume15fx1qxpjnldmhg4x7ct3 0 │ ├─ZooKeeper 15zzn6r3eysc48yv2byyfsppt Stopped 1G │ ├─HMaster 7uqk4psp2132b3ro1j090c4gl Stopped 1G │ └─RegionServer a9t5wbuv55ay7tflllb494wxm Stopped 1G ├─zencatalogservice 8idrd6dcauitv9hq08eyei8da Running 13m39s 1G 457.8M / 684.3M / 607.6M MUT0257AV Y 64e11a487276 ├─mariadb-events 8p5p25l8iqn3rz787h29blf5x Running 13m40s 2G 1.3G / 1.3G / 497.4M MUT0257AV Y 46f85bb87ec1 ├─RabbitMQ anzmn1arugpffve1ymt389ckn Running 13m40s 256M 120.9M / 202.3M / 181.7M MUT0257AV Y 1c6f5a165921 ├─opentsdb bnci6e2y1ctl48ol3pm3w8uav 0 │ ├─reader 706zlrn2rsspwzeo4foippngt Stopped 1G │ └─writer agwas47rorq9rs2wcdpugziq Stopped 1G ├─Imp4OpenTSDB dhjjbpd4vq7776q7zm4ubhatn Stopped 128M └─redis f3kbg6gtrkmfy7xb02365d5nv Running 13m38s 1G 26M / 28M / 25.1M MUT0257AV Y f3e651116e0c |
Select the serviceID and execute the following command:
serviced service stop 6bnlfl5qzog18j4mrnxszllyx
Then wait till the end of the upgrade process and enjoy your upgraded UCS Performance Manager (and thanks Cisco for not providing any kind of document for the upgrade …).