87 Commits

Author SHA1 Message Date
Victor Stinner
b9891eea1f Fix Python3 get_python_exec_prefix on Fedora 27
On Fedora 27, the default Pytho 3 install prefix changed from /usr
to /usr/local:
https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Closes-Bug: #1741901
Change-Id: Id40620efdf173189df053b5d380a801092933f83
2018-01-09 11:22:39 +01:00
Monty Taylor
e46f22db7f
Add workaround for openstacksdk in check_libs_from_git
python-openstacksdk does not match its pip name which is openstacksdk.
So setting python-openstacksdk in LIBS_FROM_GIT leads to devstack
thinking there is a problem.

Put in a workaround for now. It would be better to either:

  a) rename python-openstacksdk repo to openstacksdk
  b) rename the pip name for openstacksdk back to python-openstacksdk
  c) add general support in the various GIT hashes for a pip name

Change-Id: I57cf95763d54ad2060a4ce2af91c3ba18ca04db0
2017-12-03 10:21:26 -06:00
DamonLi
007f588f27 Convert to safe name in lib_installed_from_git
The 'pip list' command prints the "safe name" which converts _'s to
-'s amongst other things; e.g. glance_store becomes

  "glance-store             0.21.1.dev22      /opt/stack/glance_store"

Because people may use these more familiar "file system" names in
LIBS_FROM_GIT automatically convert names when checking if libraries
are installed.

Change-Id: I30524f80a341f38dfa794a8f629d859e85a4a448
2017-11-28 07:34:49 +11:00
rabi
b5fb7fd627 Fix lib_installed_from_git
In commit f0cd9a8b08d92524fc8e2c3f05d08cdebc638e2a we changed to
use column format, but it checks for zero length string and
check_libs_from_git fails.

Change-Id: I97b52b80efb33749647229a55147a08afa112dd2
2017-11-22 10:09:30 +05:30
Zuul
ec31a52560 Merge "Update lib_install_from_git to use column format" 2017-11-21 20:04:33 +00:00
Clark Boylan
065779517f Properly get pip version
The old code was strip()ing the version string instead of split()ing the
version string so we always got the first character of the version
string. This worked fine as long as the pip version was single digit but
as soon as it rolls over to '10.stuff' we will compare:

  pip version 1 (instead of 10) > 6

Which fails bceause 1 is less than six. Instaed we really do want to
compare 10 > 6 so use split on '.' instead.

Change-Id: Ic7d0c04d7fa77774ab2d70fb9d11f182becec553
2017-10-20 12:16:32 -07:00
Monty Taylor
f0cd9a8b08 Update lib_install_from_git to use column format
The pip list command supports a --format=columns option which outputs
things in space delimited columns. Switch to using that.

Change-Id: I5140a7d83bf567b1c3c67516112eb4c57074fa53
2017-10-19 15:35:38 +11:00
Zane Bitter
9e7ead9ac2 Calculate package directory correctly in pip_install
Strip the [<extras>] string from a <package_dir>[<extras>] argument
when looking for the package directory.  Explain what the heck is
going on.

Change-Id: I79beb5c3e9e7c35c91cdd0d5a1d91532bebc4b6d
Closes-Bug: #1721638
2017-10-06 22:19:05 +11:00
Ian Wienand
ae9c6ab759 Use "pip list" in check_libs_from_git
As described in the change, "pip freeze" has issues with the way
zuulv3 clones repos without a remote.  This is an attempt to use "pip
list" to check for local install

Change-Id: I33d25f86b6afcadb4b190a0f6c53311111c64521
2017-09-29 12:55:45 +10:00
Sampath Priyankara
87d2396d22 Don't uninstall pip packages if OFFLINE=True
lib/nova does a pip re-install of libvirt-python to rebuild the python
library incase the underlying libvirt version changed during package
installs.  In offline mode, the underlying version of libvirt can't
have changed; so we have the situation that we've removed the libvirt
python bindings but can't reinstall them (because we're offline).

This fixes that particular situation, but skipping uninstalls in
offline mode seems generically OK.

Change-Id: I2b75d45d94d82f87d996c7570c125d46f5f99f6a
Closes-Bug: #1708369
2017-08-14 16:03:41 +10:00
Takashi NATSUME
fa0077707e Update URLs
Some URLs are broken, so fix them.
The others are redirect to new URLs,
so replace them with new ones.

The config options of nova serial console proxy
have been gathered in nova/conf/serial_console.py.
So the description in doc/source/guides/nova.rst
is fixed.

Change-Id: Ifd81cc09969341fbf8f135a913fc6003b94e0acc
2017-07-22 09:12:59 +09:00
Clark Boylan
f266a2dc81 Install test-requirements with main install
To reduce the total number of invocations necessary for pip which isn't
the quickest thing ever (due to needing to evaluate constraints and deps
lists and what is currently installed) combine the main installation of
software with its test-requirements.txt file which should roughly halve
our pip invocations.

Change-Id: Ibcc3264136e66d34a879ad1c90a62e1bb6a84243
2017-06-12 14:57:59 -07:00
Brian Haley
954fd1b729 Use -y with 'pip uninstall'
'pip uninstall' will hang running stack.sh if it has to
prompt the user for input, use -y.

Change-Id: Ic94639e444b87fd3538463d5a51c01a0208a2ab2
Closes-bug: #1691172
2017-05-16 12:24:45 -04:00
Sean Dague
f28e7ef6ba uninstall libvirt-python and reinstall
libvirt-python compiles against the currently installed libvirt. If
you upgrade that, it needs to rebuild, however it won't change
versions, so pip install just noops. Force an uninstall / reinstall of
it every time to handle potential upgrades of libvirt.

Change-Id: If34541b34aa6d55eedaf6c603fd1fe92eb887308
2017-05-08 07:30:20 -04:00
Armando Migliaccio
bacfb94390 Enable openSUSE to work in Python 3.x environments
Add packages required to run devstack with USE_PYTHON3=True.

Change-Id: Iee43c9335bd82c10cfaeffb02d1d99290c34bb83
2017-03-20 22:33:56 -07:00
Jenkins
6523d6e097 Merge "Make declared variables global" 2017-03-08 13:31:57 +00:00
Sean Dague
afef8bf097 Make declared variables global
When variables use the 'declare' directive, it is by default a local
variable. While other variables have global scope.

For example:
   declare -A AN_ARRAY   # local in scope
   foo=1                 # global in scope

This causes errors to occur as some of the variables will be local only
and others will be global.

Update the code, as appropriate, so that variables using the 'declare'
directive also include the '-g' flag to have them also be global. Not
every instance of a declared variable has been updated.

Closes-Bug: #1669509
Co-Authored-By: John L. Villalovos <john.l.villalovos@intel.com>
Change-Id: I2180b68fe861ad19c6d4ec0df0f9f8a528347862
2017-03-07 22:07:29 -08:00
Jenkins
0e1e78117e Merge "install LIBS_FROM_GIT using python 2 and 3 where appropriate" 2017-02-28 16:37:49 +00:00
Jenkins
259c5e19a7 Merge "allow config to manage python3 use explicitly" 2017-02-28 16:37:20 +00:00
Sean Dague
f80e2cfee8 add install_devstack_tools
An initial install for devstack-tools, this will need to use all the
fun pip extra variables for installation, however the current
pip_install always prefers python2, and we only want to do python3
here.

Change-Id: I3dcdb35130f76fad81cb7b0d4001b7e96efbbd84
2017-01-18 15:42:53 -05:00
Doug Hellmann
a2eb89417f install LIBS_FROM_GIT using python 2 and 3 where appropriate
When installing a library from source and python 3 is enabled, first run
the installation process with python 2 enabled to ensure the library is
also installed under python 2 for any services not yet running under
3. The python 3 version is installed second so that command line tools
installed with libraries are installed under python 3 when python 3 is
enabled.

Change-Id: Ibb0f7a68d21081bf7652a0c1515080c0c54888ca
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-01-10 16:08:43 +00:00
Doug Hellmann
94129c7d02 allow config to manage python3 use explicitly
Add variables ENABLED_PYTHON3_PACKAGES and DISABLED_PYTHON3_PACKAGES to
work like ENABLED_SERVICES and DISABLED_SERVICES and to manage which
packages are installed using Python 3. Move the list of whitelisted
packages in pip_install to the default for ENABLED_PYTHON3_PACKAGES,
except swift which is not enabled by default for now.

Add enable_python3_package and disable_python3_package functions to make
editing the variables from local.conf easier.

Add python3_enabled_for and python3_disabled_for functions to check the
settings against packages being installed by pip.

Update pip_install to check if python3 is disabled for a service, then
see if it is explicitly enabled, and only then fall back to looking at
the classifiers in the packaging metadata.

Update pip_install messages to give more detail about why the choice
between python 2 and 3 is being made for a given package.

Change-Id: I69857d4e11f4767928614a3b637c894bcd03491f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-01-10 16:08:14 +00:00
Mehdi Abaakouk
52b1074615 cinder: configure dlm when this one is zookeeper
Previous this was set the zake, but that was revert to missing
dependencies issue and because zake is a test fixture and not somthing
to deploy.

This change configures the Cinder dlm with this one is zookeeper.
And it installs tooz and the extra dependencies needed for the
zookeeper driver.

To do it, this commit have to introduce a new method for package
installation: 'pip_install_gr_extras package extra1,extra2'.

Change-Id: Idca310c08e345db59840eb31434c6cb1f849fa70
2017-01-04 10:21:29 +01:00
Jenkins
72bfe17b6a Merge "Warn instead of die on undefined config names" 2017-01-03 20:07:35 +00:00
Davanum Srinivas
0c0d848b8c Add swift and uwsgi to py35 whitelist
Really close to getting swift and keystone under uwsgi working, so
let's white list them. Won't affect any existing jobs, so we should
be good.

Change-Id: I51d56d16a5b175bd45dee09edc0b2748d72a5d06
2017-01-03 08:52:28 -05:00
Davanum Srinivas
afa8a00c14 Switch to python 3.5
Use trueorfalse to normalize the values for USE_PYTHON3

Install 3.5 instead of 3.4 When USE_PYTHON3 is specified.
Also, since not many packages are classified correctly, fallback
to looking for just "Programming Language :: Python :: 3" and
log a message for the package to highlight the problem.

Also special case some services that are *almost* ready

Depends-On: Id48e1b328230fcdf97ed1cb4b97f4c3f9cf6eb8a
Depends-On: Ib7d9aa0e0b74a936002e0eea0b3af05102b06a62
Change-Id: I243ea4b76f0d5ef57a03b5b0798a05468ee6de9b
2016-12-23 13:15:03 -05:00
Sean Dague
22b63666de Warn instead of die on undefined config names
When using local.conf in multinode envs not everything is going to be
defined in all places. Eventually we probably want to make it so we
have a host role for these sections or something. But for now warn
instead of die when we can't find a config var.

Change-Id: I6959099373f035fbfe9e540a44e4c52b8e7c95c0
Closes-Bug: #2000824
2016-12-12 09:55:32 -05:00
YAMAMOTO Takashi
02f3f9a6bb meta-config: Fix consecutive same sections
The current coding fails to process local.conf like
the following.  Note: This example is taken from a
real use case. [1]

    [[post-config|$NEUTRON_CONF]]
    [qos]
    notification_drivers = midonet
    [[post-config|$NEUTRON_CONF]]

    [quotas]
    # x10 of default quotas (at the time of writing)
    quota_network=100
    quota_subnet=100
    quota_port=500
    quota_router=100
    quota_floatingip=500
    quota_security_group=100
    quota_security_group_rule=1000

[1] https://review.openstack.org/#/c/400627/

Closes-Bug: #1583214
Change-Id: Ie571b5fa5a33d9ed09f30ba7c7724b958ce17616
2016-11-26 00:49:55 +09:00
Lubosz "diltram" Kosnik
0a099763b0 Change python version to 3.5
On Ubuntu Xenial there is no way to install python3.4
Use value specified in PYTHON3_VERSION

Change-Id: Ibc69b1c8270bdd240c82cf2acfdfd0730ef0f182
2016-08-03 10:21:41 -05:00
Spyros Trigazis
88ccd47c88 Keep old behavior of setuptools for editable installs
In the 25.0.0 release [1] of setuptools during any install
operation the package in not overwritten. If a package is
installed from another requirement via pip and then it is
installed again from git, it is not updated causing
check_libs_from_git to fail.

[1] https://setuptools.readthedocs.io/en/latest/history.html#v25-0-0

Change-Id: Ibaa1d4157816ea649f4452756fbde25951347001
Closes-Bug: #1605998
2016-07-25 14:31:51 +02:00
Sean Dague
bb35715cfe add local.conf modifying functions
This adds a set of local.conf modifying functions which make it easier
for consuming projects like devstack-gate to programatically add
elements to local.conf structured files.

Change-Id: I3427968c2bd43aba12b3619acc27f73c74f0dabb
Co-Authored-By: fumihiko kakuma <kakuma@valinux.co.jp>
2016-07-01 12:50:18 +00:00
Marc Koderer
46f8cb7f20 Add option to download all libs from git
For client debugging that invokes multiple libs it can be useful
to have all libs directly in git and not listing all of them
in LIBS_FROM_GIT.

TrivialFix

Change-Id: Ie631cc4045231ebbe8177d2d113e47e4bf83f61c
2016-05-17 08:57:39 +02:00
YAMAMOTO Takashi
c8c1c615f2 Normalize path for upper-constraints
It seems pip distinguishes paths with .. or extra / for constraints.
For example, the following directories are considered different.

    /path/to/dir
    /path/to//dir
    /path/to/dir/subdir/..

This commit tries to normalize the given directory name to avoid
"Could not satisfy constraints for 'xxxx': installation from path
or url cannot be constrained to a version" error due to directory
name mismatch.

Reference: https://github.com/pypa/pip/pull/3582
Closes-Bug: #1542545
Change-Id: Iae9d58c27d3b10bca16e4a471507c4d5c16439a0
2016-03-22 21:23:23 +09:00
Ian Wienand
7ae97298c5 Minor fixes for bashate trunk
We don't have a new bashate release yet, but this fixes some minor
issues when used with bashate trunk.

The only two things triggered are stricter checking of assignment in
local declartions (catches quotes) and one instance of evaluating
arithmetic result in tools/xen/functions.

Therefore, hopefully, this change has no effect!

Change-Id: I3fd5d55236e90c2f2c25243af87adaf0c1b763e2
2016-02-16 14:50:53 +11:00
Jenkins
b615691576 Merge "XenAPI: add support to use local.conf" 2016-01-07 04:51:19 +00:00
Jenkins
c615993fe1 Merge "Support extra dependencies when setup_develop" 2016-01-06 20:17:12 +00:00
Jenkins
8ba3cf0132 Merge "Preserve result returned by apt-get and pip" 2016-01-05 18:56:27 +00:00
Brant Knudson
0842b8147f Support extra dependencies when setup_develop
Recent pip supports using [extras] to install extra dependencies
from the project setup.cfg. Add support so that projects can take
advantage of it.

For example, if devstack is configured to use ldap, install the
extra ldap dependencies using:

 setup_develop $KEYSTONE_DIR ldap

Partial-Bug: 1479962
Change-Id: Ic13d95b99aaa4d3854b2723343e90f5de6b98aa2
2016-01-05 07:41:19 -05:00
Federico Ressi
e208d060e1 Preserve result returned by apt-get and pip
Change I855ffe9c7a75e9943106af0f70cf715c34ae25c5 and
I368fec44858bd97fc6a314fb20eed2b10932cbb1 added timing
infrastructure which hides the return value of
the main commands.  Restore the prior behaviour.

Change-Id: I4a536abefa7824ac6d42e7bbbf1079d9cc055984
Closes-Bug: #1518545
2016-01-05 08:21:05 +00:00
vsaienko
135bd484a9 Introduce iniget_sections function
This function returns list of sections from specified INI file.
It is needed for I7938f26015d953efdf720774b5ad2cee4068ac90.
'iniget_sections' is needed for automatic node enrollment to
Ironic from INI syntax file.

Change-Id: I52a69ad3bc8fc528e7f20e6b03a98f327acaaa74
Implements: blueprint ironic-ml2-integration
2015-12-29 09:42:35 +00:00
Huan Xie
cc6af3fc94 XenAPI: add support to use local.conf
XenServer install with devstack doesn't support local.conf, this fix
is to add support for using local.conf and backward-compatibility of
localrc

Change-Id: Ie494e01f8f1ecb8720e14392ef3f12d5a5a01dcd
Closes-Bug: #1528520
2015-12-28 08:56:14 +00:00
Jenkins
1f7631dce6 Merge "Enable optional Python 3 support" 2015-12-09 18:03:58 +00:00
Jenkins
563c68f3cd Merge "Fix path setup in add_sudo_secure_path" 2015-12-06 13:34:19 +00:00
Doug Hellmann
ddc3839bdc Enable optional Python 3 support
Add USE_PYTHON3 and PYTHON3_VERSION variables to allow services to use
python 3 if they indicate support in their python package metadata.

Tested in Heat here -> I837c2fba682ab430d50e9f43913f2fed20325a7a.
Project config change to add a dedicated job to Heat is here -> I0837e62d6ccc66397a5e409f0961edd4be31f467

Change-Id: I079e18b58b214bf8362945c253d6d894ca8b1a6b
2015-12-01 14:52:35 -05:00
Dmitry Guryanov
e0ac37c257 Fix path setup in add_sudo_secure_path
There are two bugs in add_sudo_secure_path.

Firstly we don't properly check if the file exists, so always append
the new line.  This will overwrite any existing changes.

Secondly the logic for checking if the path exists is inverted, so we
miss adding paths when we should.  This particularly causes failures
when installing with virtualenv's since the paths are inside the
virtualenv, rather than the standard system locations.

Change-Id: I646fe0c68958470d464fe4f3d81d5c17dd6f2ab6
Closes-bug: #1521241
2015-12-01 13:43:01 +11:00
Atsushi SAKAI
5509ed5797 Fix comment typos in inc/ and tests/ directories
valu3 => value3
enviromnet => environment
direcotry => directory
virualenv => virtualenv
editiable => editable
envirnment => environment

Change-Id: I97fb2d44a37b16d02d4fbdb08bfa33414349f651
2015-11-30 20:25:17 +09:00
Ian Wienand
523f488036 Namespace XTRACE commands
I noticed this when debugging some grenade issues failures.

An include of grenade/functions stores the current value of XTRACE
(on) and disables xtrace for the rest of the import.

We then include devstack's "functions" library, which now overwrites
the stored value of XTRACE the current state; i.e. disabled.

When it finishes it restores the prior state (disabled), and then
grenade restores the same value of XTRACE (disabled).

The result is that xtrace is incorrectly disabled until the next time
it just happens to be turned on.

The solution is to name-space the store of the current-value of xtrace
so when we finish sourcing a file, we always restore the tracing value
to what it was when we entered.

Some files had already discovered this.  In general there is
inconsistency around the setting of the variable, and a lot of obvious
copy-paste.  This brings consistency across all files by using
_XTRACE_* prefixes for the sotre/restore of tracing values.

Change-Id: Iba7739eada5711d9c269cb4127fa712e9f961695
2015-11-27 15:36:04 +11:00
Jenkins
73de4a42d9 Merge "Provide an error message on bogus config file spec" 2015-11-04 15:39:03 +00:00
Thomas Morin
85f42f698c Provide an error message on bogus config file spec
If local.conf specifies a config file addition in the following way...

[[post-config|$MY_CONF_FILE]]
[xyz]
foo=bar

...and $MY_CONF_FILE points to a file whose directory is not writable by
the user running the script, then stack.sh aborts with the following
obscure message:

  2015-09-01 08:20:08.113 | touch: setting times of '/': Permission denied

This patch modifies inc/meta-config to provide a useful error message,
such as:

  2015-09-01 08:20:08.114 | could not create config file / ($MY_CONF_FILE)

This patch also modifies inc/meta-config so that it provides an error
message if $MY_CONF_FILE is empty (instead of silently ignoring this local.conf
statement):

  2015-09-01 09:38:53.406 | bogus config file specification: $MY_CONF_FILE
  is undefined

Change-Id: I9b78407420318548561012a8672762bc7fdd6db6
Closes-Bug: 1490881
2015-11-02 15:35:01 +01:00
Jenkins
d111182f21 Merge "Don't mix declaration and set of locals" 2015-10-14 02:43:52 +00:00