From 7fa388cae3a8dccadfcf1e90b925bc02c78759e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20Kundr=C3=A1t?= Date: Wed, 8 Apr 2020 14:34:33 +0200 Subject: [PATCH] Ensure we use recent enough virtualenv I installed Zuul 18.0 on a CentOS 7 with Python 3.6 from RH SCL repos which comes with virtualenv==15.1.0. This is effectively too old to provision the virtual environments for Ansible, likely because that version of virtualenv hardcodes PIP 9. During the installation, zuul-executor dies with this error: Installing collected packages: MarkupSafe, jinja2, PyYAML, pycparser, cffi, six, pynacl, cryptography, bcrypt, paramiko, ansible, XStatic-Bootstrap-SCSS, pyfakefs, XStatic-Patternfly-Bootstrap-Treeview, wcwidth, pyparsing, pyperclip, cmd2, pbr, stevedore, PrettyTable, cliff, XStatic, XStatic-Patternfly, wrapt, debtcollector, msgpack, pytz, netaddr, iso8601, netifaces, Babel, oslo.i18n, oslo.utils, oslo.serialization, itsdangerous, Werkzeug, click, Flask, SQLAlchemy, Flask-SQLAlchemy, XStatic-jQuery, Mako, python-editor, python-dateutil, alembic, Flask-Migrate, pygments, junit-xml, Flask-Script, Frozen-Flask, extras, linecache2, traceback2, argparse, unittest2, fixtures, python-mimeparse, testtools, python-subunit, XStatic-DataTables, ara, os-service-types, urllib3, chardet, idna, certifi, requests, keystoneauth1, jsonpointer, jsonpatch, appdirs, munch, decorator, jmespath, requestsexceptions, dogpile.cache, openstacksdk, dictdiffer, websocket-client, pyasn1, rsa, pyasn1-modules, cachetools, google-auth, oauthlib, requests-oauthlib, kubernetes, python-string-utils, ruamel.yaml.clib, ruamel.yaml, openshift, google-resumable-media, protobuf, googleapis-common-protos, google-api-core, google-cloud-core, google-cloud-storage Running setup.py install for PyYAML: started Running setup.py install for PyYAML: finished with status 'done' Running setup.py install for ansible: started Running setup.py install for ansible: finished with status 'done' Running setup.py install for XStatic-Patternfly-Bootstrap-Treeview: started Running setup.py install for XStatic-Patternfly-Bootstrap-Treeview: finished with status 'error' Complete output from command /var/lib/zuul/ansible-bin/2.6/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-24z_rrjk/XStatic-Patternfly-Bootstrap-Treeview/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6ie_p40b-record/install-record.txt --single-version-externally-managed --compile --install-headers /var/lib/zuul/ansible-bin/2.6/include/site/python3.6/XStatic-Patternfly-Bootstrap-Treeview: Traceback (most recent call last): File "", line 1, in File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/setuptools/__init__.py", line 10, in from setuptools.extern.six.moves import filter, filterfalse, map File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/setuptools/extern/__init__.py", line 1, in from pkg_resources.extern import VendorImporter File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3017, in @_call_aside File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside f(*args, **kwargs) File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3045, in _initialize_master_working_set dist.activate(replace=False) File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2577, in activate declare_namespace(pkg) File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2151, in declare_namespace _handle_ns(packageName, path_item) File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2091, in _handle_ns _rebuild_mod_path(path, packageName, module) File "/var/lib/zuul/ansible-bin/2.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2120, in _rebuild_mod_path orig_path.sort(key=position_in_sys_path) AttributeError: '_NamespacePath' object has no attribute 'sort' This points to https://github.com/pypa/pip/issues/4216, so the fix is apparently to make sure that there's new enough PIP + setuptools in the created virtual env, and these versions are determined by host's virtualenv module (thanks to webknjaz on IRC for explaining this to me). After a restart zuul-executor runs normally, but as a side effect not all required Python modules are installed for Ansible's usage. For me, this meant that (e.g.) `upload-logs-swift` won't work ebcause there's no "openstack" module. I don't know how to make zuul-executor error persistent, but let's at least make the life easier for the next poor sod installing this SW on a system where "virtualenv" is "too old" -- which is also the latest available Ubuntu if I'm checking their revision numbers correctly. Require virtualenv >= 20 while preserving the blacklist from commit 7c6503aba. Given that Zuul's own CI uses PIP to pull the latest virtualenv for the test suite, make sure that the users get what was tested. Change-Id: I2b53e532b63a9ae20e817d9599c46cccf613e425 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 2aded32369..693ec47d66 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,7 +2,7 @@ pbr>=1.1.0 # Temporary fix for https://gitlab.com/python-devs/importlib_resources/issues/83 importlib-resources==1.0.2 # Early virtualenv 20 had bad file location assumptions -virtualenv!=20.0.0,!=20.0.1 +virtualenv!=20.0.0,!=20.0.1,>20 github3.py>=1.1.0 PyYAML>=3.1.0