Move the information about the series status into a data file and use
that to generate the table on the main page. This is the first step
toward adding deliverable-specific series status information to the
output, since we need a place to put the default values.
Because we're adding a yaml file under the deliverables directory that
is not actually a deliverable file, we have to update the validation
tool to ignore it.
Story: #2001852
Task: #14347
Change-Id: I99bd94a323b53c0dfc2cb648268e51a30321cd46
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Provide some wrappers around subprocess functions to capture stderr
and stdout and divert it to the logs so that the test output is less
cluttered with the output of various git commands.
Change-Id: If36ef013aca498e3a0a9cc3a2b78d666775439ab
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Compute the dependency set for the python code being released and
report when the old minimum version no longer falls within the
specified range. For releases from master treat the message as an
error. For other branches treat the message as a warning.
Update clone_repo() to return the location where the clone was written
as a convenience to the caller.
Extract the logic for determining if a version is using
pre-versioning (alpha, beta, etc.) so it can be reused.
Change-Id: I22a2f6df7f3502e4fcbf2d61ef5fee849ab15529
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
When the branch for the series does not exist, look on 'origin/master'
as well as 'master' because depending on how the repo is cloned we
might not have a branch called 'master'.
Change-Id: I6abc0b0b3a1657211a01b80547602bbbe83f065e
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Turn on logging output, to the console, and add some log statements in
gitutils.
Change-Id: I63c80c9b2d6674d64dbc8b341a36ad51248aa19c
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We have logic for dealing with cloning errors repeated in several
places. This patch combines it into one function that can be called
more easily without having to trap exceptions.
Change-Id: Ib5f6c3f801481b28513ce8fab0f5624c02e60d97
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Under zuulv3 the behavior has changed significantly to now require
specifying which repositories need to be cloned for each job, and
zuul-cloner only copies the repositories that are already
available. As soon as zuulv3 is released, our jobs relying on the old
fallback behavior of checking out from a cache when present but using
the upstream server in other situations will fail.
The new script tools/clone_repo.sh handles the cases we have in the
tools and jobs within this repository, and should also be usable from
the tag-releases job.
zuul is removed from the list of requirements so it is no longer
installed when jobs run under tox.
list_unreleased_changes.sh is updated to drop the use of the
virtualenv, which was only present for zuul-cloner.
The other updates in the patch are to change the tools to use the new
script.
Change-Id: Ic559d27881a9a89fb0dcb5295a4d7ed7c578112f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
In order for list-changes to tag the repo we need to ensure the git
user identity is set. We do that conditionally to avoid overwriting
local user settings when running the command outside of CI.
Change-Id: I6ac6be1ad1907f19e9ba8850d80460f84d34446c
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Import the release-notes command from the release-tools repository and
integrate it with the list-changes command so that we see the release
notes for an upcoming release. This lets us review the release notes
output for warnings (like not having a documentation page linked) and
errors (such as reno failing to work).
Change-Id: I710606fe44601fe5414bfc82f25894319f16e558
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Set up the logging module to print to stdout so we can capture info
about processing in the console log. This patch changes the existing
error and warning print messages, but logging will be more important
when we integrate the reno for release notes printing later.
Change-Id: Ibf94d8527b879b0075516b6bdb26b7f5f266b021
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The list changes output of I5485b7ea35be76d7c980e7b222b0c3320770493c
looks 'messy' as some branches do not resolve to a tag. A shell based
example of what the current code does:
[tony@thor networking-bgpvpn]$ for branch in $(git branch -a | grep -v -- -\> | tr '[*]' '[ ]') ; do printf "%-30s %s\n" $branch $(git describe --abbrev=0 $branch); done
master 7.0.0.0rc1
fatal: No tags can describe '8ff78c308a004cdc948c804190d1178d7263781f'.
Try --always, or create some tags.
remotes/gerrit/backport/juno
fatal: No tags can describe '06aa314936eb9a43d358ed75b2e8fef2dadb59d6'.
Try --always, or create some tags.
remotes/gerrit/backport/kilo
remotes/gerrit/master 7.0.0.0rc1
remotes/gerrit/stable/newton 5.0.0
remotes/gerrit/stable/ocata 6.0.0
remotes/gerrit/stable/pike 7.0.0
fatal: No tags can describe '8ff78c308a004cdc948c804190d1178d7263781f'.
Try --always, or create some tags.
remotes/origin/backport/juno
fatal: No tags can describe '06aa314936eb9a43d358ed75b2e8fef2dadb59d6'.
Try --always, or create some tags.
remotes/origin/backport/kilo
remotes/origin/master 7.0.0.0rc1
remotes/origin/stable/newton 5.0.0
remotes/origin/stable/ocata 6.0.0
remotes/origin/stable/pike 7.0.0
Adding --always:
[tony@thor networking-bgpvpn]$ for branch in $(git branch -a | grep -v -- -\> | tr '[*]' '[ ]') ; do printf "%-30s %s\n" $branch $(git describe --always --abbrev=0 $branch); done
master 7.0.0.0rc1
remotes/gerrit/backport/juno 8ff78c308a004cdc948c804190d1178d7263781f
remotes/gerrit/backport/kilo 06aa314936eb9a43d358ed75b2e8fef2dadb59d6
remotes/gerrit/master 7.0.0.0rc1
remotes/gerrit/stable/newton 5.0.0
remotes/gerrit/stable/ocata 6.0.0
remotes/gerrit/stable/pike 7.0.0
remotes/origin/backport/juno 8ff78c308a004cdc948c804190d1178d7263781f
remotes/origin/backport/kilo 06aa314936eb9a43d358ed75b2e8fef2dadb59d6
remotes/origin/master 7.0.0.0rc1
remotes/origin/stable/newton 5.0.0
remotes/origin/stable/ocata 6.0.0
remotes/origin/stable/pike 7.0.0
Change-Id: Ic59797ea2c7e2278e8b25f45511009620e6f103a
When we delete stable branches a commit may not appear to be on a
named branch, so validation for old releases may fail even though the
commit is still in the repository. This change uses 'git show' instead
of 'git branch --contains' to separate the tests for existence and
branch membership.
Change-Id: I58213b29bb37c7c428017204a4bdd5235a5816ca
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Use the existing heuristic to try to check out the right branch, but
then look at the available branches and if we find a stable branch
with the right series name check that out explicitly and use that name
for the branch so that the gerrit queries include the right
patches. This allows us to see patches on stable branches during the
release candidate period, for example.
Change-Id: Ie7a389acb3d912767a8e35dd0dd86053feb7b3d0
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
All patches pushed to gerrit are mirrored and then visible through cgit,
so using that web service to check for a "valid" patch does not do what
we thought. This patch changes the logic to look for any named branch to
contain the patch, meaning it has been merged into a branch already
somewhere.
In the course of working on the tests, I also discovered that the one
verifying ancestry of patches being tagged as part of a series was using
the stable/mitaka branch in automaton, which is no EOL. I rewrote the
test to try to be more future-proof.
Change-Id: I2a1c797de4d61649dd047aca28a241c8901e18f3
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The rules for stable branch releases need to be relaxed a bit to account
for projects that are late to create their branches and only do so after
the default branch has been updated. This change makes the rules more
flexible so that it is possible to release for a stable branch on master
as long as that stable branch does not exist. The primary use for this
is to release and then create the stable branch in the same patch.
Change-Id: I2d8f9abcf8ffa34e29eb7021830fd635e6e0f85f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Update the check_output() calls used by the validate command so the
results are decoded before being treated as strings.
Change-Id: I099ff997f69743d2958fc5581292aacebd7f5d37
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We had at one case where the release validation logic failed to catch a
release from master after the stable/ocata branch was created for
mox3. This script looks at the other "late" releases, created after a
branch, and verifies that they are all on the correct branch.
Change-Id: Ie148a34efa029e6715db099f7e5c9575ed94238d
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Strip the * indicating which branch is current and drop the lines that
are not named branches.
Change-Id: Ie97b8c6139cf873fc083c3205e7495b65ba49994
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
When cloning or updating the local copy of the repository, set the
working copy to a specific SHA so the validation steps can look at the
files on the filesystem.
Change-Id: I913b4608eae18c6e85982a5dff3b5388ad80eda6
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
In the period between the time when we start creating stable branches
and we update the default series setting it is possible to release from
the wrong branch. Update the check_branch_sha() logic to look for an
existing stable branch and use that, and only default to master if the
series matches and there is no stable branch yet.
Change-Id: I80b0f39c21b55fb8037c1309392f9deaecef20e7
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We need a tool to verify that the release artifacts exist on the
tarballs site. The missing-releases command was already looking for
missing tags, so have it look for the tarballs, too.
This change refactors the URL generation code into a separate module so
it can be used by missing-releases and the sphinx extension. It also
creates a reusable function for testing a link, and has the gitutils
module functions responsible for looking for tags, branches, etc. use
it.
Change the default behavior to only look at the most recent release for
each deliverable, and add an option to go back to the previous behavior
of scanning all of them.
Change-Id: Ic7345466ccd83cf2d8d9d6d019107d6fbba171cc
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
For the first tag in the series, we skipped the descendant
check, which means that an existing SHA from a wrong branch
would pass checks. This change adds a branch match check to
cover that specific case and emit an error.
This change removes a warning which was emitted for first
releases, so we adjust the expected warning counts for some
impacted tests.
Change-Id: I741449a11ea68caeadd5be4aef8d0130efb5f5ec
Add a tool for reviewing the diff-start values for a final release.
Change-Id: I0d51f766d1cf899bd5c210060b0661a08246669b
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Use a separate URL for testing a tag from testing a SHA, since sometimes
the latter does not work for tags.
Change-Id: I01e71df99b0ab81f171f9e69e5190164e20f5106
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The old implementation of list-changes assumed that the deliverable file
included all releases and that they were in order. That assumption does
not hold for independent projects, where the history might not be there
and new entries might be from different branches than the previous
entries, all in the same file. This change uses git to determine the
previous tag for each repository, then shows the right diffs based on
that instead of the assumed tag from the deliverable file.
Change-Id: I122a6ec1792a07cdb23f09a880110dff2446555e
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
In the CI system we may clone from a repo that doesn't have everything,
such as tags, so after zuul-cloner is done force an update to pull in
the tags.
Change-Id: Ic4b5a217ed142ff83555a0fececb424a1a400fc9
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Before the fix, sha values were always extracted from master head, even
when series has a stable branch.
Change-Id: I9810660df7a92c021cd9c154bb3f24d8d828bfd7
The validation command in particular ends up cloning the same repo
repeatedly. Skip the extra work by checking to see if it already exists.
Change-Id: If56f03be03e37116edb8ee9443c9ab286c9bbdd9
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Let the validation and list-changes code use the zuul cache directory,
if ZUUL_CACHE_DIR is set or if /opt/git is present (the default for the
CI systems).
Change-Id: Ia6cf24d1361c1112c37e5aeec05546b19798bbbd
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The ancestor check depends on finding some output. If we get an error
message, that shouldn't count.
Change-Id: If5c079249d7b32ac6a7418f853a7db06082593bc
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
"tox -e pep8" is showing these new errors
./openstack_releases/gitutils.py:52:9: W503 line break before binary operator
./openstack_releases/governance.py:77:21: W503 line break before binary operator
./openstack_releases/cmds/list_changes.py:152:21: W503 line break before binary operator
./openstack_releases/cmds/validate.py:36:1: E402 module level import not at top of file
./openstack_releases/cmds/validate.py:37:1: E402 module level import not at top of file
./openstack_releases/cmds/validate.py:38:1: E402 module level import not at top of file
Change-Id: I4c460928af038fc93ca1057953408f8cc7cac24e
The git describe command reports errors if there are no tags to be used
to get a description. Trap the errors in places where it is reasonable
to encounter that situation.
Change-Id: I436af248b1740c493b14178cbbe59c8b0474aa94
If we can't figure out the previous version from looking in the YAML
file, use git describe to find it.
Show the version number being released in the header of the list-changes
output so we can verify that it matches the expected value after seeing
what sorts of changes are included.
Change-Id: I8f65400900ee0779fd8d1736bf319e638ebbeca0
Look for the SHA being assigned a new version in the descendants of the
previous version to ensure it is merged properly.
Change-Id: I82fcb9295bc5eea0c404db898272329dcc515b7b
Let sha_for_tag assume that the repository already exists, and create a
clone_repo function to be used to enforce that.
Change-Id: Iab9b54cf7a82b70aeebc46a09cab92ccfbe76f4c
Update the validate command to compare the actual SHA of a tag with the
one in the file to ensure they match.
Change-Id: I5c04667cbb43cda548c3cc05a47d5a5830b32951
The merge commits created by zuul do not always list all of the files
being modified, so use git diff instead of git show.
Also force python to use unbuffered I/O so the output comes out in a
useful order instead of mixed up with the subprocess commands.
Change-Id: I5e0e4602e463c05e488537192be8ba59c08644dc
Move some of the validation code into a real python package so we can
more easily reuse it.
Make the validation script reprint all errors, for cases where it has
checked a lot of files and the original messages might have been mixed
in with other output.
Fix the launchpad project name for tooz so the validation script will
pass.
Change-Id: I17534d460dd9ed19d10e48a0ef5d28dac3ab0fe7
