openstack/devstack
Code Issues Proposed changes

Fix rabbitmq retry for error checking

Browse Source 
I think this retry check has been broken since we introduced "set -e".
Unfortunately it seems the issue of rabbitmq not starting first-time
persists on centos 7 hosts occasionally, e.g. [1]:

---
 + rabbit_setuser stackrabbit secretrabbit
 + local user=stackrabbit pass=secretrabbit found= out=
 ++ sudo rabbitmqctl list_users
 Error: unable to connect to node 'rabbit@devstack-centos7-rax-iad-100675': nodedown

 DIAGNOSTICS
 ===========

 nodes in question: ['rabbit@devstack-centos7-rax-iad-100675']

 hosts, their running nodes and ports:
 - devstack-centos7-rax-iad-100675: [{rabbitmqctl29293,39511}]

 current node details:
 - node name: 'rabbitmqctl29293@devstack-centos7-rax-iad-100675'
 - home dir: /var/lib/rabbitmq
 - cookie hash: KieJnx1pnllKbHVihGcDqA==
---

Fix up this retry while we investigate [2]

[1] http://logs.openstack.org/64/141864/1/check//check-tempest-dsvm-centos7/4308f0c/logs/devstacklog.txt.gz
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1144100

Change-Id: I11fb3728e08adc1e0f7acca63e5a308d24dce78e
This commit is contained in:
Ian Wienand 2014-12-16 09:53:36 +11:00
parent 7f80280698
commit 64b56a53d8
1 changed files with 18 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
lib/rpc_backend
View File

@ -176,17 +176,31 @@ function restart_rpc_backend {
echo_summary "Starting RabbitMQ" echo_summary "Starting RabbitMQ"
# NOTE(bnemec): Retry initial rabbitmq configuration to deal with # NOTE(bnemec): Retry initial rabbitmq configuration to deal with
# the fact that sometimes it fails to start properly. # the fact that sometimes it fails to start properly.
# Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1059028 # Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1144100
local i local i
for i in `seq 10`; do for i in `seq 10`; do
local rc=0
[[ $i -eq "10" ]] && die $LINENO "Failed to set rabbitmq password"
if is_fedora || is_suse; then if is_fedora || is_suse; then
# service is not started by default # service is not started by default
restart_service rabbitmq-server restart_service rabbitmq-server
fi fi
rabbit_setuser "$RABBIT_USERID" "$RABBIT_PASSWORD"
rabbit_setuser "$RABBIT_USERID" "$RABBIT_PASSWORD" || rc=$?
if [ $rc -ne 0 ]; then
continue
fi
# change the rabbit password since the default is "guest" # change the rabbit password since the default is "guest"
sudo rabbitmqctl change_password $RABBIT_USERID $RABBIT_PASSWORD && break sudo rabbitmqctl change_password \
[[ $i -eq "10" ]] && die $LINENO "Failed to set rabbitmq password" $RABBIT_USERID $RABBIT_PASSWORD || rc=$?
if [ $rc -ne 0 ]; then
continue;
fi
break
done done
if is_service_enabled n-cell; then if is_service_enabled n-cell; then
# Add partitioned access for the child cell # Add partitioned access for the child cell