install ebtables locking workaround

ebtables is racing with itself when nova and libvirt attempt to create
rules at the same time in the nat table. ebtables now has an explicit
--concurrent flag, that all tools must opt into to prevent ebtables
from inherently being unsafe to run.

libvirt gained this support in 1.2.11, which is too new for our ubuntu
primary testing environment. Nova still hasn't added this support,
though even if it did, we'd run into the issue with libvirt.

We can do the most ghetto thing possible and create a wrapper for
ebtables that does explicit locking on it's own. It's pretty terrible,
but it should work. And it is the kind of work around that people
unable to upgrade libvirt will probably need to do.

This is an opt in value which we should set in the gate to True.

Related-Bug: #1501558

Change-Id: Ic6fa847eba34c21593b9df86a1c2c179534d0ba5

files/ebtables.workaround

@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# Copyright 2015 Hewlett-Packard Development Company, L.P.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+#
+#
+# This is a terrible, terrible, truly terrible work around for
+# environments that have libvirt < 1.2.11. ebtables requires that you
+# specifically tell it you would like to not race and get punched in
+# the face when 2 run at the same time with a --concurrent flag.
+
+flock -w 300 /var/lock/ebtables.nova /sbin/ebtables.real $@

lib/nova_plugins/functions-libvirt

@@ -31,6 +31,11 @@ function install_libvirt {
         fi
         install_package libvirt-bin libvirt-dev
         pip_install_gr libvirt-python
+        if [[ "$EBTABLES_RACE_FIX" == "True" ]]; then
+            # Work around for bug #1501558. We can remove this once we
+            # get to a version of Ubuntu that has new enough libvirt.
+            TOP_DIR=$TOP_DIR $TOP_DIR/tools/install_ebtables_workaround.sh
+        fi
         #pip_install_gr <there-si-no-guestfs-in-pypi>
     elif is_fedora || is_suse; then
         install_package kvm

stackrc

@@ -769,6 +769,16 @@ GIT_DEPTH=${GIT_DEPTH:-0}
 # Use native SSL for servers in ``SSL_ENABLED_SERVICES``
 USE_SSL=$(trueorfalse False USE_SSL)
 
+# ebtables is inherently racey. If you run it by two or more processes
+# simultaneously it will collide, badly, in the kernel and produce
+# failures or corruption of ebtables. The only way around it is for
+# all tools running ebtables to only ever do so with the --concurrent
+# flag. This requires libvirt >= 1.2.11.
+#
+# If you don't have this then the following work around will replace
+# ebtables with a wrapper script so that it is safe to run without
+# that flag.
+EBTABLES_RACE_FIX=$(trueorfalse False EBTABLES_RACE_FIX)
 
 # Following entries need to be last items in file
 

tools/install_ebtables_workaround.sh

@@ -0,0 +1,31 @@
+#!/bin/bash -eu
+#
+# Copyright 2015 Hewlett-Packard Development Company, L.P.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+#
+#
+# This replaces the ebtables on your system with a wrapper script that
+# does implicit locking. This is needed if libvirt < 1.2.11 on your platform.
+
+EBTABLES=/sbin/ebtables
+EBTABLESREAL=/sbin/ebtables.real
+FILES=$TOP_DIR/files
+
+if [[ -f "$EBTABLES" ]]; then
+    if file $EBTABLES | grep ELF; then
+        sudo mv $EBTABLES $EBTABLESREAL
+        sudo install -m 0755 $FILES/ebtables.workaround $EBTABLES
+        echo "Replaced ebtables with locking workaround"
+    fi
+fi