Use nodeenv for npm and yarn in tox

Add a wrapper installation script that will install npm and yarn into the
current virtualenv. If yarn is installed globally, the script is a
no-op.

If yarn is not installed globally, whenever tox thinks it needs to
create or re-create a virtualenv, install nodeenv then use it to
install yarn.

This removes the use of the zuul-tox-py35 job because it should now just work
properly with the normal job. It does not remove the job itself because
it's still used in tox-py35-on-zuul.

Change-Id: If360a3f0c6b3d74498f8aa063d8b1ae87daff101

.zuul.yaml

@@ -61,7 +61,7 @@
             vars:
               sphinx_python: python3
         - tox-pep8
-        - zuul-tox-py35:
+        - tox-py35:
             irrelevant-files:
               - zuul/cmd/migrate.py
               - playbooks/zuul-migrate/.*
@@ -97,7 +97,7 @@
             vars:
               sphinx_python: python3
         - tox-pep8
-        - zuul-tox-py35:
+        - tox-py35:
             irrelevant-files:
               - zuul/cmd/migrate.py
               - playbooks/zuul-migrate/.*

doc/source/developer/javascript.rst

@@ -18,6 +18,19 @@ The best thing would be to get familiar with the tools, there are a lot of
 good features available. But, if you don't want to know anything about the
 Javascript toolchains a few helpers have been provided.
 
+tox
+~~~
+
+If you do not have `yarn`_ installed, ``tox`` will use `nodeenv`_ to install
+node into the active python virtualenv, and then will install `yarn`_ into
+that virtualenv as well.
+
+npm + docker
+~~~~~~~~~~~~
+
+.. We should remove the build:docker command. If you have npm, you can
+   npm install yarn.
+
 If you have npm and docker installed and don't want to install newer nodejs
 or a bunch of javascript libraries, you can run:
 
@@ -25,6 +38,9 @@ or a bunch of javascript libraries, you can run:
 
   npm run build:docker
 
+docker
+~~~~~~
+
 If you have docker but do not have npm or nodejs installed, you can build
 the web app with:
 
@@ -221,3 +237,4 @@ our case we use it for both.
 .. _nodejs: https://nodejs.org/
 .. _webpack: https://webpack.js.org/
 .. _devtool: https://webpack.js.org/configuration/devtool/#devtool
+.. _nodeenv: https://pypi.python.org/pypi/nodeenv

tests/unit/test_web.py

@@ -46,9 +46,6 @@ class BaseTestWeb(ZuulTestCase):
     config_ini_data = {}
 
     def setUp(self):
-        self.assertTrue(
-            os.path.exists(zuul.web.STATIC_DIR),
-            "Static web assets are missing, be sure to run 'npm run build'")
         super(BaseTestWeb, self).setUp()
         self.executor_server.hold_jobs_in_build = True
         A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')

tools/pip.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# Copyright 2018 Red Hat, Inc.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+# This script checks if yarn is installed in the current path. If it is not,
+# it will use nodeenv to install node, npm and yarn.
+# Finally, it will install pip things.
+if [[ ! $(command -v yarn) ]]
+then
+    pip install nodeenv
+    # Initialize nodeenv and tell it to re-use the currently active virtualenv
+    nodeenv --python-virtualenv
+    # Use -g because inside of the virtualenv '-g' means 'install into the'
+    # virtualenv - as opposed to installing into the local node_modules.
+    # Avoid writing a package-lock.json file since we don't use it.
+    # Avoid writing yarn into package.json.
+    npm install -g --no-package-lock --no-save yarn
+fi
+if [[ ! -f zuul/web/static/status.html ]]
+then
+    yarn install
+    npm run build:dev
+fi
+pip install $*

tox.ini

@@ -10,11 +10,21 @@ setenv = VIRTUAL_ENV={envdir}
 passenv = ZUUL_TEST_ROOT OS_STDOUT_CAPTURE OS_STDERR_CAPTURE OS_LOG_CAPTURE OS_LOG_DEFAULTS NODEPOOL_ZK_HOST
 usedevelop = True
 install_command = pip install {opts} {packages}
+whitelist_externals = bash
 deps = -r{toxinidir}/requirements.txt
        -r{toxinidir}/test-requirements.txt
 commands =
   python setup.py test --slowest --testr-args='{posargs}'
 
+[nodeenv]
+install_command = bash tools/pip.sh {opts} {packages}
+
+[testenv:py35]
+install_command = {[nodeenv]install_command}
+
+[testenv:py36]
+install_command = {[nodeenv]install_command}
+
 [testenv:bindep]
 # Do not install any requirements. We want this to be fast and work even if
 # system dependencies are missing, since it's used to tell you what system
@@ -32,6 +42,7 @@ commands =
   mypy --ignore-missing-imports zuul
 
 [testenv:cover]
+install_command = {[nodeenv]install_command}
 commands =
   python setup.py test --coverage
 
@@ -45,12 +56,14 @@ commands = {posargs}
 [testenv:nodepool]
 setenv =
    OS_TEST_PATH = ./tests/nodepool
+install_command = {[nodeenv]install_command}
 commands = python setup.py test --slowest --testr-args='--concurrency=1 {posargs}'
 
 [testenv:remote]
 setenv =
    OS_TEST_PATH = ./tests/remote
 passenv = ZUUL_TEST_ROOT OS_STDOUT_CAPTURE OS_STDERR_CAPTURE OS_LOG_CAPTURE OS_LOG_DEFAULTS ZUUL_REMOTE_IPV4 ZUUL_SSH_KEY NODEPOOL_ZK_HOST
+install_command = {[nodeenv]install_command}
 commands = python setup.py test --slowest --testr-args='--concurrency=1 {posargs}'
 
 [flake8]