Enable dstat for in-gate functional and fullstack test runs

This may become handy when debugging gate failures like job or test case
or ovsdb timeouts, to see the load of the machine, as well as to
understand whether the machine was e.g. locked by its hypervisor during
test run.

Instead of using devstack as a library, we just enable the service in
local.conf [[localrc]], and then call ./stack.sh.

Now that we install some pypi packages into the system, and since
functional targets were using those system packages, post_gate_hook.sh
broke failing to locate subunit-2to1 script in the tox env directory. To
fix that, we disable sitepackages= for all functional tox targets. This
was enabled back in the times when we attempted to install neutron for
the functional job using devstack [1], and we no longer do it, so it
should be ok to stick to all packages from the tox venv.

[1] Icc38cacd69bc5843ccfcc60237a7102df6d8597f

Change-Id: I73857f63f512e26d595c61a20ec5bf22d2fb0da7

neutron/tests/contrib/gate_hook.sh

@@ -67,8 +67,17 @@ case $VENV in
     fi
     upgrade_ovs_if_necessary $compile_modules
 
+    # prepare base environment for ./stack.sh
+    load_conf_hook stack_base
+
+    # enable monitoring
+    load_conf_hook dstat
+
     # Make the workspace owned by the stack user
     sudo chown -R $STACK_USER:$STACK_USER $BASE
+
+    # deploy devstack as per local.conf
+    cd $DEVSTACK_PATH && sudo -H -u $GATE_STACK_USER ./stack.sh
     ;;
 
 "api"|"api-pecan"|"full-ovsfw"|"full-pecan"|"dsvm-scenario")

neutron/tests/contrib/hooks/dstat

@@ -0,0 +1,2 @@
+[[local|localrc]]
+enable_service dstat

neutron/tests/contrib/hooks/stack_base

@@ -0,0 +1,9 @@
+[[local|localrc]]
+# set password, otherwise devstack enters interactive mode and fails
+ADMIN_PASSWORD=secretadmin
+# don't use screen to start services (needed to disable colorization in
+# captured service logs)
+USE_SCREEN=False
+# start with an empty service list, otherwise devstack will configure several
+# 'default' services, including rabbitmq and mysql
+disable_all_services

tox.ini

@@ -52,7 +52,6 @@ deps =
 basepython = python2.7
 setenv = {[testenv:functional]setenv}
          {[testenv:dsvm]setenv}
-sitepackages=True
 deps =
   {[testenv:functional]deps}
 commands =
@@ -62,7 +61,6 @@ commands =
 [testenv:dsvm-functional-py35]
 basepython = python3.5
 setenv = {[testenv:dsvm-functional]setenv}
-sitepackages={[testenv:dsvm-functional]sitepackages}
 deps =
   {[testenv:dsvm-functional]deps}
 commands =
@@ -76,7 +74,6 @@ setenv = {[testenv]setenv}
          # workaround for DB teardown lock contention (bug/1541742)
          OS_TEST_TIMEOUT=600
          OS_TEST_PATH=./neutron/tests/fullstack
-sitepackages=True
 deps =
   {[testenv:functional]deps}