This updates the script to automatically update puppet constraints so
that Puppetfile_unit is updated as well as Puppetfile. We introduced
this separate file[1] a while ago but have not been updated it
properly. Updating it periodically helps us catch up with any update
in the dependent modules more timely.
[1] 047ea125ddefa8151d6592d4772560a1b23a13dc
Depends-on: https://review.opendev.org/875301
Change-Id: I4d89a2985781f14d9ff7f76f412f0b8ccc4c6ddb
Since the PTI switched the supported python versions for the current
cycle from 3.8+3.9 to 3.9+3.10, we want to update the constraints
generation accordingly. Since we don't have a stable platform where we
can run both py3.9 and py3.10 in parallel, generate constraints only for
py3.10 for now.
Change-Id: I4a4d9cb6e292bb693ae424be0928501fed81113f
Previously we had this error:
No key/value pairs provided, at least one is required for this action to succeed
Which appears to be due to the default(omit) condition when setting
ensure_pip_virtualenv_command in focal proposal jobs. The issue is
default(omit) doesn't do what you think it does when defining a
variable. It is really only useful for defining module parameters that
may be omitted.
Instead we need to call ensure-tox two different ways depending on
whether or not we want to override the defaults for
ensure_pip_virtualenv_command. One method for Bionic and another for
Focal keyed off of whether or not _venv_command is defined.
Change-Id: I0cbca64f4a31c8b4eacb5e1c50f2e9fb289ce18e
This fixes a typo in I683a61b1bae1a809caf724aee87a21af2e18fb0c -- it
should be "ansible_distribution_release"
Change-Id: I074f089371054856a8b96bd6e650496a0bd53685
I think we have quite a tangle of fixes on fixes in the translation
jobs, and I hope this refactor makes things a bit better.
The fundamental problem seems to be that python3.6 can't work with
master requirements.txt. But we can not move some jobs off bionic
(python 3.6) because of java version contsraints.
The first thing that happened was these jobs got "ensure-python" with
"python_version: 3.8" set to bring in python3.8 on Bionic. This role
was not bringing in python3.8-venv, so we had no way to create 3.8
virtual environments. Thus ensure-pip ended up getting modified to
drag this package in (reverted with notes in
372f3af706c796331578ef81c2680e8c4c03c3b6). This has a fairly easy
solution; let's just make ensure-python bring in the venv packages for
the udpated python -- see dependency
Ie3c03fea82bcec80a897f0905c15f35405a50396.
The git-review install here uses 'ensure_pip_virtualenv_command' to
install git-review in a virtualenv. git-review still supports bionic,
and does not need to be installed in any speical way. However, by
overriding the 'ensure_pip_virtualenv_command' for the entire job,
this unrelated install now grew a dependency on python3.8 and created
more confusion over why it was failing.
I believe what we actually want to do on Bionic is just run tox under
python3.8. So this pulls out all the job-level defintions and
encapsulates them in a single block scoped to run on bionic. It will
pull in python3.8 packages, and then install tox under python3.8 with
an override of the virtulenv command just for that step. The jobs
just maintain their node definition and note on why it is required.
The long-term future of these jobs is obviously a question; but I
believe this keeps them running on bionic in as sane a way as possible
for now.
The second part of this is the python3.9 interpreter required on
Focal. Setting "python_version: 3.9" in the job definition is a bit
confusing, because the job actually requires both Python 3.8 AND 3.9.
This variable is passed to ensure-python, and the naming is a bit
unfortunate because it is so broad (but we can't change that without
zuul-jobs updates). Here I've pulled out the Python 3.9 install into
a separate focal-gated section in the pre.yaml that explains what is
going on (note that Python 3.8 is just the base python on focal, so
needs no extra installation). I've also updated the description on
the job node with this info. There's no need to install the tox
environment any differently on focal; it just uses the base system
python (3.8) which is fine.
Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/850957
Change-Id: I547e5eb7faabcd9d3983ae85f3291b1a740bc77c
Currently both python3.8 and python3.9 venv package
is being installed, but since python3.9-venv is not
available in bionic, python3.8-venv installation also
fails as mentioned in [1].
This patch moves ensure_pip_from_upstream_interpreters
var from proposal pre playbook to job definition and
override it in translation jobs to attempt installation
only for python3.8-venv.
Also switch back to "python3.8 -m venv" virtualenv_cmd
as with this patch python3.8-venv will get installed.
[1] https://review.opendev.org/c/zuul/zuul-jobs/+/846201
[2] https://review.opendev.org/c/openstack/project-config/+/846390
Change-Id: I9d9a423e3147d3f04c1951a796636e5520202a6d
Install git-review task relies on ensure_pip_virtualenv_command
variable but with [1] this is set to python3.8 in translation
update jobs but it get's installed later by ensure-python role,
let's run it earlier to fix the issue.
[1] https://review.opendev.org/c/openstack/project-config/+/845907
Change-Id: Iecbdcb50063cb461b3a09ddfa205d071c7d77b75
The previous attempts at fixing the job were not successful yet, it
seems the variable in the pre playbook isn't overridden by a job
variable. Since the other uses of this job are running with ubuntu-focal
anyway, set the list of available python interpreters in the playbook
directly and remove the modifications from the requirements specific job
again. Also pin the nodeset for this job to guard against changes in the
base job, which will likely require another change to the interpreter
list.
[0] I9735f34639777c928fbe06febc16dd24667b1405
[1] I52cc316ce9756c4411c18c8339d44c457cb59897
Change-Id: Iddd61e1be766f2bb6da4bf62766829247fa143ce
3.10 will come later when there is an image that supports all three
Change-Id: I663de65a971337fce65a51f0e5a7f959b8da2c03
Signed-off-by: Matthew Thode <mthode@mthode.org>
If00afb50c37b45aefbb45070da4efef3e43e62b2 updated the requirements
update job to use tox as part of the update checking script. We have
ensure-tox as one of the roles in the pre playbook for this job, but it
is not set to be made available globally, which results in the script
failing with being unable to find tox.
This updates the role inclusion to set ensure_global_symlinks: true so
tox will be available in the script.
Change-Id: I5d3ed4ce62cd8df604802fedf522d644d5664698
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This script creates a virtualenv to run a requirements script to update
constraints. virtualenv is not globally available anymore. Rather than
trying to call it directly, this now uses tox to set up the virtual
environment.
Change-Id: If00afb50c37b45aefbb45070da4efef3e43e62b2
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Pip is used by this task, make sure ensure-pip is used so it is
present.
While here, install git-review to a venv and symlink it globally to
keep things separate.
Change-Id: If6b1abd2545c68f727bddc5317797236f451b67c
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Signed-off-by: Ian Wienand <iwienand@redhat.com>
We were installing git-review under python2 in order to propose updates
to Gerrit. git-review happily runs under python3 so update the
installation to avoid python2.
Change-Id: I1671345d71cd60f6508a2c067c4e308680234afe
With change I15d451f6a832b2e698b28e7351a36ef7aea92abe
generate-constraints needs python 3, setup the venv using python 3.
Change-Id: I3dba35267dc6f344ad27910c7dc18c1a33abf4ba
after requirements branches we will remove py27 and py37
Change-Id: I5fde605634108dd628df546a2318731e1832b6b3
Signed-off-by: Matthew Thode <mthode@mthode.org>
- Use zuul_return fake module to make ansible lint happy, this allows
to remove Zuul import.
- While ansible-lint 4.2.0 is now able to detect playbooks/roles, this
is broken, so don't use it
- move its config out of tox.ini so it can be used by any tools, or
without tox
Change-Id: Ie8935f47db855647e19ae091044e5ac1871f1551
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Co-Authored-By: Andreas Jaeger <aj@suse.com>
We have started to maintain some of the items inside doc/* to be
sync'd such as requirements to build docs, this patch adds it.
Change-Id: I73c32ea17a9fb0f6d0ba1887f8618aa508ba63c5
This prevents installing unconstrainted versions of libraries for
python3.7. When it was unmapped there was no constraint for it for
certian libraries (some libraries had diferent versions for each version
of python).
Change-Id: I0a5ecc715142756a9ec37fca559fad691651430c
Follow OpenDev changes:
* Use gerrit.opendev.org instead of gerrit.openstack.org
* Use opendev.org insted of git.openstack.org
* Follow move of openstack-infra/project-config to
openstack/project-config
Change-Id: Ic27ea41407abc0bdec29768e5ee5dca33f8adeda
We sometimes [1] need to use a commit hash as a pin,
so adapt the script to work with commit hash pins as well.
[1] https://review.openstack.org/#/c/586946/
Change-Id: I8867b3d41dc6ceef084c6d978cc405b9f6710375
Currently we're running propose-updates forrequirements-constraints
under python 3.5 (on xenial). We take the results of freezing the
constraints fro 3.5 and emit them as 3.4 constraints also. In the near
term[1] we'd like to use the same technique to generate 3.6 constraints.
With the inclusion of I8784509bc162eb6f2e80261bc2d81dbe63ce7989, and
knowing that we no longer run on trusty node, we can now clone the 3.5
constraints freeze to both 3.4 and 3.6. However given that soon we'll
be moving from xenial to bionic this introduces debt that will need to
be fixed after the August release.
Instead just move propose-updates for requirements-constraints job to
bionic now and start cloning 3.6 to 3.5 and 3.4.
Several jobs use propose-updates as a parent so I've made this change
specifically where we call the propose-updates job specifically for
requirements-constraints
Note: This may feel very familiar if you reviewed
I541b64afaee3bf466ff1a461fd2cb472657c4986, The aim is the same but I
mis-understood what propose-update-constraints was doing and it wasn't
the right job and was reverted in I75280b23e2e60800af45dc2343c82008ec9971a9
[1] See re-design discussion:
http://lists.openstack.org/pipermail/openstack-dev/2018-July/132152.html
Change-Id: I1874d4f20a6d69a881f0d9407e159c35ab21542d
This reverts commit c1de884d33.
propose-updates-constraints != propose-updates. The later does what I want, the former is used by the release-team after releasing a new library.
#eggonface
Change-Id: I75280b23e2e60800af45dc2343c82008ec9971a9
Currently we're running propose-update-constraints under python 3.5 (on
xenial). We take the results of freezing the constraints fro 3.5 and
emit them as 3.4 constraints also. In the near term[1] we'd like to use
the same technique to generate 3.6 constraints also.
With the inclusion of I8784509bc162eb6f2e80261bc2d81dbe63ce7989, and
knowing that we no longer run on trusty node, we can now clone the 3.5
constraints freeze to both 3.4 and 3.6. However given that soon we'll
be moving from xenial to bionic this introduces debt that will need to
be fixed after the August release.
Instead just move propose-update-constraints job to bionic now and start
cloning 3.6 to 3.5 and 3.4.
[1] See re-design discussion:
http://lists.openstack.org/pipermail/openstack-dev/2018-July/132152.html
Change-Id: I541b64afaee3bf466ff1a461fd2cb472657c4986
We have an emerging need to synchronise a common set of tasks across
roles. While it may sometimes make sense to create a role for this
sort of thing, the overhead of doing so would be greater than simply
copying the file between roles. We're also finding include_role to
be somewhat buggy, especially when used from inside a role.
This patch adds the ability to copy a set of files from the source [1]
to the same path (without the 'sync' prefix) in the destination roles,
abiding by the same rules as all other synchronised files.
[1] openstack/openstack-ansible/tests/sync
Relates-to: https://review.openstack.org/579527
Change-Id: I598ff6aae50e6570d5f262c830c4be10f1cc0b75
This reverts commit 022703d084.
We actually still want to sync this file for stable branches, so we revert
this commit.
Change-Id: I63ac677b620cd627b6a4c8dfa6c8e978d4ec6f61
We do not automatically propose updates to projects' requirements, so
we don't need this branch of logic any more.
Depends-On: https://review.openstack.org/555402/
Change-Id: I511ead56a6a6efecde25ce6b118256aa0560023a
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The tests/tests-repo-clone.sh file is being phased out from the OSA
repositories so the proposal bot should not sync that anymore.
Change-Id: Ia923fe60a80110ea09a350670c54a4011a13b089
To not install these on the images, create a new scripts directory to
hold them.
Create new copy-release-tools-scripts role to copy the files over,
we do not need using legacy-copy-project-config-scripts anymore since
everything is copied elsewhere.
Add the role to all playbooks that use release-tools scripts.
Change-Id: I36c458e297fec58023799f625739b341c4d403cf
Move proposal scripts from jenkins/scripts to the proposal directory
so that we can install them only for proposal scripts and do not need
to bake them into nodes.
Change-Id: I356906e4a43597cc913bfa539cac7c2c6dfce117
Create new role for copying common scripts, this includes for now only
common.sh. Use this for proposal and translation jobs.
In the past we duplicated the file, this allows to have one definition
only.
Change-Id: I72c14341f511fdaa2c67a3758387afc9445869cc
All of these invocations of the bindep role are passing what amount to
the defaults. In zuul-jobs, doing bindep_dir: "{{ zuul_work_dir }}" is
important because it allows in-repo uses of those jobs to override how
they work for things like triggering from one repo but running on
another.
The publication jobs in project-config do not need that level of
flexibility, so just remove params and let defaults take over.
As a follow up, we could also add defaults for zuul_work_dir - and
honestly for bindep_dir - to the base job. That way we can always count
on them being set to a sane default value, and they're still
overrideable in a job.
Change-Id: I5bcda84aefa4726d0a904e8bc9698c21ca07f9a5
Instead of scripting the git configuration, use tasks in pre playbooks
for the jobs. We use different settings for tag-releases and
propose-constraint-updates.
Change-Id: Id3713d89a8b2af821de41a146954be551337cf75
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We need gettext on the systems for the translations jobs, this should be
part of the bindep setup. Since we invoke tox, add bindep and tox setup
in pre-run.
Also, some proposal jobs need pyyaml, so add python-yaml to bindep.txt -
and use bindep for all proposal jobs.
Change-Id: I3c0e4741e1c0186b78859841db1ca5c791d4806b
ZUUL_REFNAME is not set in periodic pipelines.
Provide optionally branch and use that one instead if REFNAME is not set
to mimic how the script was used in v2.
Change-Id: I3a514b5ffd9d0797c1bd43a0762bd5709630b1cd
We don't need a legacy job for this, we have a propose-updates job that
can do any of the tagets the script knows about.
We do need to add git-review to the remote host though, so do that.
Change-Id: I3cf10120ab0e9aff65c35468d3a7bb9d42725712
So that we can land a changes to project-config after v2 has been turned
off, shift project-config to v3 and remove from v2.
Change-Id: I67fc88a86d4e8b094cab88929bb6b8c47198964a
Depends-On: I7cfb90df4eb49defce7aa41453918347a2c6742e
We'll need it for more than just proposal jobs, but it's clearly very
project-config specific, so keep it here.
While refactoring that, consume add-sshkey from zuul-jobs.
Depends-On: I569b93ba0e775896b5a4f59b544fe32d1e2dce9b
Change-Id: I6b16acdcf55bcc6ff28ad496c044fe7db4599cd8
By adding a run here, we can easily make jobs either as children or
variants since the only thing needed is the argument.
Change-Id: I6d19b8d9fb96210b8b92103c095fd8c8b9c8309b
After we release to PyPI, we want to send out update-constraints.
We don't need to rewrite all the scripts involved with this for now, so
update them to call each other using relative not absolute paths.
Then, copy the scripts dir from the trusted checkout on the executor
to the homedir on the remote host so that we can execute the appropriate
script. Local paths are relative to the location of the playbook.
Change-Id: I080f71fefe79f7faa581fae91eac74384cef4050