Add install-siblings command
Extract the logic for installing sibling packages from zuul-jobs and encapsulate it in a CLI utility.
This commit is contained in:
Monty Taylor
parent
a380c6aa87
commit
12d5941f35
10
README.rst
10
README.rst
|
|
@ -22,6 +22,14 @@ support for such actions.
|
|||
Features
|
||||
--------
|
||||
|
||||
* TODO
|
||||
Each utility is implemented as a subcommand on the ``pbrx`` command.
|
||||
|
||||
install-siblings
|
||||
Updates an installation with local from-source versions of dependencies.
|
||||
For any dependency that the normal installation installed from pip/PyPI,
|
||||
``install-siblings`` will look for an adjacent git repository that provides
|
||||
the same package. If one exists, the source version will be installed to
|
||||
replace the released version. This is done in such a way that any given
|
||||
``constraints`` will be honored and not get messed up by transitive depends.
|
||||
|
||||
.. _pbr: https://docs.openstack.org/pbr/latest/
|
||||
|
|
|
|||
|
|
@ -3,3 +3,7 @@ Users guide
|
|||
===========
|
||||
|
||||
Users guide of pbrx.
|
||||
|
||||
.. toctree::
|
||||
|
||||
siblings
|
||||
|
|
|
|||
|
|
@ -0,0 +1,92 @@
|
|||
================================
|
||||
Installation of Sibling Packages
|
||||
================================
|
||||
|
||||
There are times, both in automated testing, and in local development, where
|
||||
one wants to install versions of a project from git that are referenced in
|
||||
a requirements file, or that have somehow already been installed into a given
|
||||
environment.
|
||||
|
||||
This can become quite complicated if a constraints file is involved, as the
|
||||
git versions don't match the versions in the constraints file. But if a
|
||||
constraints file is in play, it should also be used for the installation of
|
||||
the git versions of the additional projects so that their transitive depends
|
||||
may be properly constrained.
|
||||
|
||||
To help with this, `pbrx` provides the ``install-siblings`` command. It takes
|
||||
a list of paths to git repos to attempt to install, as well as an optional
|
||||
constraints file.
|
||||
|
||||
It will only install a git repositoriy if there is already a corresponding
|
||||
version of the package installed. This way it is safe to have other repos
|
||||
wind up in the package list, such as if a Zuul job had a Depends-On including
|
||||
one or more additional packages that were being put in place for other
|
||||
purposes.
|
||||
|
||||
``pbrx siblings`` expects to be run in root source dir of the primary project.
|
||||
Sibling projects may be given as relative or absolute paths.
|
||||
|
||||
For example, assume the following directory structure:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
$ tree -ld -L 3
|
||||
├── git.openstack.org
|
||||
│ ├── openstack
|
||||
│ │ ├── keystoneauth
|
||||
│ │ ├── python-openstackclient
|
||||
│ │ ├── python-openstacksdk
|
||||
│ │ ├── requirements
|
||||
|
||||
The user is in the ``git.openstack.org/openstack/python-openstackclient`` and
|
||||
has installed the code into a virtualenv called ``venv``.
|
||||
``python-openstackclient`` has the following requirements:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
keystoneauth1>=3.3.0 # Apache-2.0
|
||||
openstacksdk>=0.9.19 # Apache-2.0
|
||||
|
||||
And in the ``git.openstack.org/openstack/requirements`` directory is a file
|
||||
called ``upper-constraints.txt`` which contains:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
keystoneauth1===3.4.0
|
||||
openstacksdk===0.11.3
|
||||
requests===2.18.4
|
||||
|
||||
The command:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
$ venv/bin/pbrx install-siblings ../keystoneauth
|
||||
|
||||
would result in an installation of the contents of ``../keystoneauth``, since
|
||||
``keystoneauth1`` is already installed and the package name in the
|
||||
``git.openstack.org/openstack/keystoneauth`` directory is ``keystoneauth1``.
|
||||
No constraints are given, so any transitive dependencies that are
|
||||
in ``git.openstack.org/openstack/keystoneauth`` will be potentially installed
|
||||
unconstrained.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
$ venv/bin/pbrx install-siblings -c ../requirements/upper-constraints.txt ../keystoneauth
|
||||
|
||||
Will also update ``keystoneauth1``, but will apply constraints properly to
|
||||
any transitive depends.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
$ venv/bin/pbrx install-siblings -c ../requirements/upper-constraints.txt ../keystoneauth ../python-openstacksdk
|
||||
|
||||
will install both ``keystoneauth1`` and ``openstacksdk``.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
$ venv/bin/pbrx install-siblings -c ../requirements/upper-constraints.txt ../keystoneauth ../python-openstacksdk ../requirements
|
||||
|
||||
will also install both ``keystoneauth1`` and ``openstacksdk``. Even though
|
||||
``git.openstack.org/openstack/requirements`` is itself a python package, since
|
||||
it is not one of the ``python-openstackclient`` dependencies, it will be
|
||||
skipped.
|
||||
|
|
@ -15,4 +15,4 @@
|
|||
import pbr.version
|
||||
|
||||
|
||||
__version__ = pbr.version.VersionInfo('pbrx').version_string()
|
||||
__version__ = pbr.version.VersionInfo("pbrx").version_string()
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ try:
|
|||
except ImportError:
|
||||
yaml = None
|
||||
|
||||
from pbrx import siblings
|
||||
import pbr.version
|
||||
|
||||
log = logging.getLogger("pbrx")
|
||||
|
|
@ -82,6 +83,20 @@ def main():
|
|||
title="commands", description="valid commands", help="additional help"
|
||||
)
|
||||
|
||||
cmd_siblings = subparsers.add_parser(
|
||||
"install-siblings", help="install sibling packages"
|
||||
)
|
||||
cmd_siblings.set_defaults(func=siblings.main)
|
||||
cmd_siblings.add_argument(
|
||||
"-c,--constraints",
|
||||
dest="constraints",
|
||||
help="Path to constraints file",
|
||||
required=False,
|
||||
)
|
||||
cmd_siblings.add_argument(
|
||||
"projects", nargs="*", help="List of project src dirs to process"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
setup_logging(args.log_config, args.debug)
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,260 @@
|
|||
# Copyright (c) 2018 Red Hat
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
try:
|
||||
import configparser
|
||||
except ImportError:
|
||||
import ConfigParser as configparser
|
||||
|
||||
import logging
|
||||
import os
|
||||
import pkg_resources
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
log = logging.getLogger("pbrx")
|
||||
|
||||
|
||||
def get_package_name(setup_cfg):
|
||||
"""Get package name from a setup.cfg file."""
|
||||
try:
|
||||
c = configparser.ConfigParser()
|
||||
c.read(setup_cfg)
|
||||
return c.get("metadata", "name")
|
||||
|
||||
except Exception:
|
||||
log.debug("No name in %s", setup_cfg)
|
||||
return None
|
||||
|
||||
|
||||
def get_requires_file(dist):
|
||||
"""Get the path to the egg-info requires.txt file for a given dist."""
|
||||
return os.path.join(
|
||||
os.path.join(dist.location, dist.project_name + ".egg-info"),
|
||||
"requires.txt",
|
||||
)
|
||||
|
||||
|
||||
def get_installed_packages():
|
||||
"""Get the correct names of the currently installed packages."""
|
||||
return [f.project_name for f in pkg_resources.working_set]
|
||||
|
||||
|
||||
def pip_command(*args):
|
||||
"""Execute a pip command in the current python."""
|
||||
pip_args = [sys.executable, "-m", "pip"] + list(args)
|
||||
log.debug("Executing %s", " ".join(pip_args))
|
||||
output = subprocess.check_output(pip_args, stderr=subprocess.STDOUT)
|
||||
for line in output.decode("utf-8").split("\n"):
|
||||
log.debug(line)
|
||||
return output
|
||||
|
||||
|
||||
class Siblings(object):
|
||||
|
||||
def __init__(self, name, projects, constraints):
|
||||
self.name = name
|
||||
self.projects = projects
|
||||
self.constraints = constraints
|
||||
self.packages = {}
|
||||
self.get_siblings()
|
||||
log.info(
|
||||
"Sibling Processing for %s from %s",
|
||||
self.name,
|
||||
os.path.abspath(os.path.curdir),
|
||||
)
|
||||
|
||||
def get_siblings(self):
|
||||
"""Finds all python packages that are there.
|
||||
|
||||
From the list of provided source dirs, find all of the ones that are
|
||||
python projects and return a mapping of their package name to their
|
||||
src_dir.
|
||||
|
||||
We ignore source dirs that are not python packages so that this can
|
||||
be used with the list of all dependencies from a Zuul job. In the
|
||||
future we might want to add a flag that causes that to be an error
|
||||
for local execution.
|
||||
"""
|
||||
self.packages = {}
|
||||
|
||||
for root in self.projects:
|
||||
root = os.path.abspath(root)
|
||||
name = None
|
||||
setup_cfg = os.path.join(root, "setup.cfg")
|
||||
found_python = False
|
||||
if os.path.exists(setup_cfg):
|
||||
found_python = True
|
||||
name = get_package_name(setup_cfg)
|
||||
self.packages[name] = root
|
||||
if not name and os.path.exists(os.path.join(root, "setup.py")):
|
||||
found_python = True
|
||||
# It's a python package but doesn't use pbr, so we need to run
|
||||
# python setup.py --name to get setup.py to tell us what the
|
||||
# package name is.
|
||||
name = subprocess.check_output(
|
||||
[sys.executable, "setup.py", "--name"],
|
||||
cwd=root,
|
||||
stderr=subprocess.STDOUT,
|
||||
)
|
||||
if name:
|
||||
name = name.strip()
|
||||
self.packages[name] = root
|
||||
if found_python and not name:
|
||||
log.info("Could not find package name for %s", root)
|
||||
else:
|
||||
log.info("Sibling %s at %s", name, root)
|
||||
|
||||
def write_new_constraints_file(self):
|
||||
"""Write a temporary constraints file excluding siblings.
|
||||
|
||||
The git versions of the siblings are not going to match the values
|
||||
in the constraints file, so write a copy of the constraints file
|
||||
that doesn't have them in it, then use that when installing them.
|
||||
"""
|
||||
constraints_file = tempfile.NamedTemporaryFile(delete=False)
|
||||
existing_constraints = open(self.constraints, "r")
|
||||
for line in existing_constraints.read().split("\n"):
|
||||
package_name = line.split("===")[0]
|
||||
if package_name in self.packages:
|
||||
continue
|
||||
|
||||
constraints_file.write(line.encode("utf-8"))
|
||||
constraints_file.write(b"\n")
|
||||
constraints_file.close()
|
||||
return constraints_file
|
||||
|
||||
def find_sibling_packages(self):
|
||||
for package_name in get_installed_packages():
|
||||
log.debug("Found %s python package installed", package_name)
|
||||
if package_name == self.name:
|
||||
# We don't need to re-process ourself. We've filtered
|
||||
# ourselves from the source dir list, but let's be sure
|
||||
# nothing is weird.
|
||||
log.debug("Skipping %s because it's us", package_name)
|
||||
continue
|
||||
|
||||
if package_name in self.packages:
|
||||
log.debug(
|
||||
"Package %s on system in %s",
|
||||
package_name,
|
||||
self.packages[package_name],
|
||||
)
|
||||
|
||||
log.info("Uninstalling %s", package_name)
|
||||
pip_command("uninstall", "-y", package_name)
|
||||
yield package_name
|
||||
|
||||
def clean_depends(self, installed_siblings):
|
||||
"""Overwrite the egg-info requires.txt file for siblings.
|
||||
|
||||
When we install siblings for a package, we're explicitly saying
|
||||
we want a local git repository. In some cases, the listed requirement
|
||||
from the driving project clashes with what the new project reports
|
||||
itself to be. We know we want the new project, so remove the version
|
||||
specification from the requires.txt file in the main project's
|
||||
egg-info dir.
|
||||
"""
|
||||
dist = None
|
||||
for found_dist in pkg_resources.working_set:
|
||||
if found_dist.project_name == self.name:
|
||||
dist = found_dist
|
||||
break
|
||||
|
||||
if not dist:
|
||||
log.debug(
|
||||
"main project is not installed, skipping requires clean"
|
||||
)
|
||||
return
|
||||
|
||||
requires_file = get_requires_file(dist)
|
||||
if not os.path.exists(requires_file):
|
||||
log.debug("%s file for main project not found", requires_file)
|
||||
return
|
||||
|
||||
new_requires_file = tempfile.NamedTemporaryFile(delete=False)
|
||||
with open(requires_file, "r") as main_requires:
|
||||
for line in main_requires.readlines():
|
||||
found = False
|
||||
for name in installed_siblings:
|
||||
if line.startswith(name):
|
||||
log.debug(
|
||||
"Replacing %s with %s in requires.txt",
|
||||
line.strip(),
|
||||
name,
|
||||
)
|
||||
new_requires_file.write(name.encode("utf-8"))
|
||||
new_requires_file.write(b"\n")
|
||||
found = True
|
||||
break
|
||||
|
||||
if not found:
|
||||
new_requires_file.write(line.encode("utf-8"))
|
||||
os.rename(new_requires_file.name, requires_file)
|
||||
|
||||
def process(self):
|
||||
"""Find and install the given sibling projects."""
|
||||
installed_siblings = []
|
||||
package_args = []
|
||||
for sibling_package in self.find_sibling_packages():
|
||||
log.info(
|
||||
"Installing %s from %s",
|
||||
sibling_package,
|
||||
self.packages[sibling_package],
|
||||
)
|
||||
package_args.append("-e")
|
||||
package_args.append(self.packages[sibling_package])
|
||||
installed_siblings.append(sibling_package)
|
||||
if not package_args:
|
||||
log.info("Found no sibling packages, nothing to do.")
|
||||
return
|
||||
|
||||
args = ["install"]
|
||||
|
||||
if self.constraints:
|
||||
constraints_file = self.write_new_constraints_file()
|
||||
args.extend(["-c", constraints_file.name])
|
||||
args.extend(package_args)
|
||||
|
||||
try:
|
||||
pip_command(*args)
|
||||
finally:
|
||||
os.unlink(constraints_file.name)
|
||||
|
||||
self.clean_depends(installed_siblings)
|
||||
|
||||
|
||||
def main(args):
|
||||
if not os.path.exists("setup.cfg"):
|
||||
log.info("No setup.cfg found, no action needed")
|
||||
return 0
|
||||
|
||||
if not args.projects:
|
||||
log.info("No sibling projects given, no action needed.")
|
||||
return 0
|
||||
|
||||
if args.constraints and not os.path.exists(args.constraints):
|
||||
log.info("Constraints file %s was not found", args.constraints)
|
||||
return 1
|
||||
|
||||
# Who are we?
|
||||
package_name = get_package_name("setup.cfg")
|
||||
if not package_name:
|
||||
log.info("No name in main setup.cfg, skipping siblings")
|
||||
return 0
|
||||
|
||||
siblings = Siblings(package_name, args.projects, args.constraints)
|
||||
siblings.process()
|
||||
return 0
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
features:
|
||||
- |
|
||||
Added ``install-siblings`` command to install relevant packages from
|
||||
nearby git repositories.
|
||||
some details.
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
# Quick test script used during development. This should obviously be replaced
|
||||
# with actual tests.
|
||||
|
||||
# However, if you have repos locally in golang organization, and you have:
|
||||
# git.openstack.org/openstack-dev/pbr
|
||||
# git.openstack.org/openstack/keystoneauth
|
||||
# git.openstack.org/openstack/openstacksdk
|
||||
# git.openstack.org/openstack/pbrx
|
||||
# git.openstack.org/openstack/python-openstackclient
|
||||
# git.openstack.org/openstack/requirements
|
||||
# github.com/requests/requests
|
||||
#
|
||||
# You should be able to run this script in the openstack/python-openstackclient
|
||||
# directory and verify it does the right things.
|
||||
# openstack/python-openstackclient was selected because it exhibits a complex
|
||||
# Entrypoints issue.
|
||||
set -e
|
||||
|
||||
BASE=../../..
|
||||
OPENSTACK=$BASE/git.openstack.org/openstack
|
||||
for interp in python2 python3 ; do
|
||||
venv=venv-$interp
|
||||
rm -rf $venv
|
||||
virtualenv --python=$interp venv-$interp
|
||||
# Install python-openstackclient's requirements with constraings
|
||||
$venv/bin/pip install -c $OPENSTACK/requirements/upper-constraints.txt -r requirements.txt
|
||||
# Install python-openstackclient itself
|
||||
$venv/bin/pip install --no-deps -e .
|
||||
# Install pbrx with this patch
|
||||
$venv/bin/pip install -e $BASE/git.openstack.org/openstack/pbrx/
|
||||
# Run siblings
|
||||
$venv/bin/pbrx --debug install-siblings -c $OPENSTACK/requirements/upper-constraints.txt $OPENSTACK/keystoneauth $OPENSTACK/openstack/python-openstacksdk $BASE/github.com/requests/requests $BASE/git.openstack.org/openstack-dev/pbr/
|
||||
# openstack help should not break
|
||||
$venv/bin/openstack help
|
||||
done
|
||||
Loading…
Reference in New Issue