With the new release identification / naming schema [1] (like:
2023.1 Antelope) new stable branch naming was introduced (like:
stable/2023.1). This results a false warning message now when
validating bobcat releases [2], as the validator searches for
stable/antelope instead of stable/2023.1 stable branch.
This patch fixes the validate_series_open validator to use release_id
for the branch name, where that exists. Besides, the patch also
refactors the release-id fetching and ensures that the release_id is
always returened as string and the series name is used if there is no
release_id for the given series.
[1] https://governance.openstack.org/tc/reference/release-naming.html
[2] validate_series_open: There is no stable/antelope branch defined in deliverables/antelope/<x>.yaml. Is the bobcat series open?
Change-Id: I71eae27f124473699c22dac0b0eb2b0b1d44b4da
With the new release identification / naming schema [1] (like:
2023.1 Antelope) new stable branch naming was introduced (like:
stable/2023.1). This was not handled in check_branch_sha method
in validate.py, as the method still searches stable/$series.
This patch fixes this by reading the 'release-id' field from
series_status.yaml if present and uses it as stable/<release-id>
for the branch search.
[1] https://governance.openstack.org/tc/reference/release-naming.html
Change-Id: I014b1d6dc561be4db0fc8faa85fb3133e851acfc
When the latest branch cut is from the very same hash as the previous
branch was cut from, then the rev-list gives an empty list. Originally
the last item from the list should have been taken. Since the list is
empty the splitlines()[-1] gives "IndexError: list index out of range".
This patch fixes this case with returning None early.
Change-Id: If5ea4a356b0c7620e2de1d2ab77951d8b76bb5e2
We recently observed that this check was skipped. Indeed it was
possible to propose a bugfix version on the first release of a series
without facing validation errors [1].
The described situation was allowed by checks that we introduced
during Wallaby [2]. Those was designed to ensure that we can
build sdist from the proposed tag. Indeed, they temporarly created
the new git tag to build sdist from it. However we never delete this
tag after our tests so this temporary tag is see as an existing tag
and so that led us to the observed bug.
Our validation check that the first version number of series isn't a
bugfix after it tried to build the sdist from the new tag, so, without
this patch the tag newly created by the sdist check remains available
in the repo, and so a couple of next checks are skipped, including the
validation of the version number, because these checks are decorated
to ignore existing tags.
Indeed the function decorated with `skip_existing_tags` detect this
temporary tag as an existing tag, so, the decorated functions are
skipped.
We should notice that it surely impacted other checks in the same manner.
Indeed, all validation functions decorated with this function
`skip_existing_tags` are impacted by this bug.
Hopefully it didn't have too much side effects.
These changes delete the temporary tag after usage to ensure to restore
the initial environment and to stop skipping next checks.
[1] https://review.opendev.org/c/openstack/releases/+/795836
[2] 80652b5232
Change-Id: I5d9024528d08487a7582e0a0e73576b22708a6cf
This updates the version of hacking we are using for our linting and
addresses various issues that the latest version flags.
Change-Id: I95ed73411e96451bc447e1b5858b0466fb8f10a9
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Rewrite list_unreleased_changes as python format and add new features.
By default it will behave as the previous version of this command (the
shell script).
Few new feature have been added in these changes to allow us to more
easily handle outputs in scripts.
Features added:
- allow user to retrieve results in json format
- allow user to retrieve results in yaml format
- allow user to ignore project not yet released
The shell script entry-point (tools) is still there but it will call
the python command in a venv instead of directly implement features.
Also the python version allow us to more surround this tools with unit tests.
Change-Id: Iaf86ecb1589c40102acb621b23ea12d71ed453bb
This does a new release with release-test to see if everything is
covered to allow branches to have a bugfix/* prefix.
Change-Id: I7869cf412524d7352249f02f7449b2fddf401d2c
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Shamelessly steal^Wcopy some of the code from interactive-release so
that after validating the release we list the changes that will be
released. This gives us the ability to decide that the release
contains no functional changes and elect to not create that release.
NOTE: I chose not to use interactive-release as it seems not to
correctly handle first releases in a series (because it doesn't load all
release history. It also doesn't use some of the new features (like
series_status). Adding --interactive to new-release gets us a long way
to deprecating interactive-release but we aren't quire there yet.
Change-Id: I25bbb4d7df9ae618500dd37f4b0cbc32c0bbd153
Adding liaisons to deliverables where there is a liasion listed here:
https://wiki.openstack.org/wiki/CrossProjectLiaisons#Release_management
Also creates a data dir to put the non deliverable files that were in the
deliverables dir elsewhere and updates filepaths to reflect the change.
Story: 2005702
Task: 31024
Change-Id: Idc5f4b29dd375465b21d678fa8503cbd8d6d3eb6
Updates to use opendev.org or published documentation.
Change-Id: I0bd488b25d1259bce3f723a6c58283fdf670e721
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Before this change we always had a start revision value because
get_latest_tag() returned a SHA if it didn't have a real tag. The way
that SHA is determined leads to not including all changes in the
report about the release, so we don't want to use the value. This
change allows get_latest_tag() to return None and in that case use an
empty starting point for the diff range when computing the release
notes contents.
Change-Id: I7403b0ab2264ef7828c012a943cf093da2951d68
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
When the current code needs the repo checked out to a specific commit it
calls clone_repo. The clone_repo call makes sure the repo has been
cloned locally, performs a fetch to get all tags and prunes any
references no longer on the remote, then finally checks out the
requested ref.
The validation logic starts with cloning all necessary repos and
performing these actions, so after that point, we should be able to just
checkout the ref that we need for the given check. This will greatly
improve the execution time of the validate job for repos that have had a
large number of releases.
Change-Id: Ie444c8ef8e91832c38ee69307382ddb5fb8e1b08
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The wrong variable was being used to check for the existing
branch. Add some debug messages to show what we do find and fix that
variable so the right branch name is checked.
Change-Id: I758f6aa3fd0b983af1b83a922b40c3181ae48014
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Move the information about the series status into a data file and use
that to generate the table on the main page. This is the first step
toward adding deliverable-specific series status information to the
output, since we need a place to put the default values.
Because we're adding a yaml file under the deliverables directory that
is not actually a deliverable file, we have to update the validation
tool to ignore it.
Story: #2001852
Task: #14347
Change-Id: I99bd94a323b53c0dfc2cb648268e51a30321cd46
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Provide some wrappers around subprocess functions to capture stderr
and stdout and divert it to the logs so that the test output is less
cluttered with the output of various git commands.
Change-Id: If36ef013aca498e3a0a9cc3a2b78d666775439ab
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Compute the dependency set for the python code being released and
report when the old minimum version no longer falls within the
specified range. For releases from master treat the message as an
error. For other branches treat the message as a warning.
Update clone_repo() to return the location where the clone was written
as a convenience to the caller.
Extract the logic for determining if a version is using
pre-versioning (alpha, beta, etc.) so it can be reused.
Change-Id: I22a2f6df7f3502e4fcbf2d61ef5fee849ab15529
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
When the branch for the series does not exist, look on 'origin/master'
as well as 'master' because depending on how the repo is cloned we
might not have a branch called 'master'.
Change-Id: I6abc0b0b3a1657211a01b80547602bbbe83f065e
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Turn on logging output, to the console, and add some log statements in
gitutils.
Change-Id: I63c80c9b2d6674d64dbc8b341a36ad51248aa19c
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We have logic for dealing with cloning errors repeated in several
places. This patch combines it into one function that can be called
more easily without having to trap exceptions.
Change-Id: Ib5f6c3f801481b28513ce8fab0f5624c02e60d97
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Under zuulv3 the behavior has changed significantly to now require
specifying which repositories need to be cloned for each job, and
zuul-cloner only copies the repositories that are already
available. As soon as zuulv3 is released, our jobs relying on the old
fallback behavior of checking out from a cache when present but using
the upstream server in other situations will fail.
The new script tools/clone_repo.sh handles the cases we have in the
tools and jobs within this repository, and should also be usable from
the tag-releases job.
zuul is removed from the list of requirements so it is no longer
installed when jobs run under tox.
list_unreleased_changes.sh is updated to drop the use of the
virtualenv, which was only present for zuul-cloner.
The other updates in the patch are to change the tools to use the new
script.
Change-Id: Ic559d27881a9a89fb0dcb5295a4d7ed7c578112f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
In order for list-changes to tag the repo we need to ensure the git
user identity is set. We do that conditionally to avoid overwriting
local user settings when running the command outside of CI.
Change-Id: I6ac6be1ad1907f19e9ba8850d80460f84d34446c
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Import the release-notes command from the release-tools repository and
integrate it with the list-changes command so that we see the release
notes for an upcoming release. This lets us review the release notes
output for warnings (like not having a documentation page linked) and
errors (such as reno failing to work).
Change-Id: I710606fe44601fe5414bfc82f25894319f16e558
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Set up the logging module to print to stdout so we can capture info
about processing in the console log. This patch changes the existing
error and warning print messages, but logging will be more important
when we integrate the reno for release notes printing later.
Change-Id: Ibf94d8527b879b0075516b6bdb26b7f5f266b021
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The list changes output of I5485b7ea35be76d7c980e7b222b0c3320770493c
looks 'messy' as some branches do not resolve to a tag. A shell based
example of what the current code does:
[tony@thor networking-bgpvpn]$ for branch in $(git branch -a | grep -v -- -\> | tr '[*]' '[ ]') ; do printf "%-30s %s\n" $branch $(git describe --abbrev=0 $branch); done
master 7.0.0.0rc1
fatal: No tags can describe '8ff78c308a004cdc948c804190d1178d7263781f'.
Try --always, or create some tags.
remotes/gerrit/backport/juno
fatal: No tags can describe '06aa314936eb9a43d358ed75b2e8fef2dadb59d6'.
Try --always, or create some tags.
remotes/gerrit/backport/kilo
remotes/gerrit/master 7.0.0.0rc1
remotes/gerrit/stable/newton 5.0.0
remotes/gerrit/stable/ocata 6.0.0
remotes/gerrit/stable/pike 7.0.0
fatal: No tags can describe '8ff78c308a004cdc948c804190d1178d7263781f'.
Try --always, or create some tags.
remotes/origin/backport/juno
fatal: No tags can describe '06aa314936eb9a43d358ed75b2e8fef2dadb59d6'.
Try --always, or create some tags.
remotes/origin/backport/kilo
remotes/origin/master 7.0.0.0rc1
remotes/origin/stable/newton 5.0.0
remotes/origin/stable/ocata 6.0.0
remotes/origin/stable/pike 7.0.0
Adding --always:
[tony@thor networking-bgpvpn]$ for branch in $(git branch -a | grep -v -- -\> | tr '[*]' '[ ]') ; do printf "%-30s %s\n" $branch $(git describe --always --abbrev=0 $branch); done
master 7.0.0.0rc1
remotes/gerrit/backport/juno 8ff78c308a004cdc948c804190d1178d7263781f
remotes/gerrit/backport/kilo 06aa314936eb9a43d358ed75b2e8fef2dadb59d6
remotes/gerrit/master 7.0.0.0rc1
remotes/gerrit/stable/newton 5.0.0
remotes/gerrit/stable/ocata 6.0.0
remotes/gerrit/stable/pike 7.0.0
remotes/origin/backport/juno 8ff78c308a004cdc948c804190d1178d7263781f
remotes/origin/backport/kilo 06aa314936eb9a43d358ed75b2e8fef2dadb59d6
remotes/origin/master 7.0.0.0rc1
remotes/origin/stable/newton 5.0.0
remotes/origin/stable/ocata 6.0.0
remotes/origin/stable/pike 7.0.0
Change-Id: Ic59797ea2c7e2278e8b25f45511009620e6f103a
When we delete stable branches a commit may not appear to be on a
named branch, so validation for old releases may fail even though the
commit is still in the repository. This change uses 'git show' instead
of 'git branch --contains' to separate the tests for existence and
branch membership.
Change-Id: I58213b29bb37c7c428017204a4bdd5235a5816ca
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Use the existing heuristic to try to check out the right branch, but
then look at the available branches and if we find a stable branch
with the right series name check that out explicitly and use that name
for the branch so that the gerrit queries include the right
patches. This allows us to see patches on stable branches during the
release candidate period, for example.
Change-Id: Ie7a389acb3d912767a8e35dd0dd86053feb7b3d0
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
All patches pushed to gerrit are mirrored and then visible through cgit,
so using that web service to check for a "valid" patch does not do what
we thought. This patch changes the logic to look for any named branch to
contain the patch, meaning it has been merged into a branch already
somewhere.
In the course of working on the tests, I also discovered that the one
verifying ancestry of patches being tagged as part of a series was using
the stable/mitaka branch in automaton, which is no EOL. I rewrote the
test to try to be more future-proof.
Change-Id: I2a1c797de4d61649dd047aca28a241c8901e18f3
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The rules for stable branch releases need to be relaxed a bit to account
for projects that are late to create their branches and only do so after
the default branch has been updated. This change makes the rules more
flexible so that it is possible to release for a stable branch on master
as long as that stable branch does not exist. The primary use for this
is to release and then create the stable branch in the same patch.
Change-Id: I2d8f9abcf8ffa34e29eb7021830fd635e6e0f85f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Update the check_output() calls used by the validate command so the
results are decoded before being treated as strings.
Change-Id: I099ff997f69743d2958fc5581292aacebd7f5d37
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We had at one case where the release validation logic failed to catch a
release from master after the stable/ocata branch was created for
mox3. This script looks at the other "late" releases, created after a
branch, and verifies that they are all on the correct branch.
Change-Id: Ie148a34efa029e6715db099f7e5c9575ed94238d
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Strip the * indicating which branch is current and drop the lines that
are not named branches.
Change-Id: Ie97b8c6139cf873fc083c3205e7495b65ba49994
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
When cloning or updating the local copy of the repository, set the
working copy to a specific SHA so the validation steps can look at the
files on the filesystem.
Change-Id: I913b4608eae18c6e85982a5dff3b5388ad80eda6
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
In the period between the time when we start creating stable branches
and we update the default series setting it is possible to release from
the wrong branch. Update the check_branch_sha() logic to look for an
existing stable branch and use that, and only default to master if the
series matches and there is no stable branch yet.
Change-Id: I80b0f39c21b55fb8037c1309392f9deaecef20e7
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We need a tool to verify that the release artifacts exist on the
tarballs site. The missing-releases command was already looking for
missing tags, so have it look for the tarballs, too.
This change refactors the URL generation code into a separate module so
it can be used by missing-releases and the sphinx extension. It also
creates a reusable function for testing a link, and has the gitutils
module functions responsible for looking for tags, branches, etc. use
it.
Change the default behavior to only look at the most recent release for
each deliverable, and add an option to go back to the previous behavior
of scanning all of them.
Change-Id: Ic7345466ccd83cf2d8d9d6d019107d6fbba171cc
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
For the first tag in the series, we skipped the descendant
check, which means that an existing SHA from a wrong branch
would pass checks. This change adds a branch match check to
cover that specific case and emit an error.
This change removes a warning which was emitted for first
releases, so we adjust the expected warning counts for some
impacted tests.
Change-Id: I741449a11ea68caeadd5be4aef8d0130efb5f5ec
Add a tool for reviewing the diff-start values for a final release.
Change-Id: I0d51f766d1cf899bd5c210060b0661a08246669b
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Use a separate URL for testing a tag from testing a SHA, since sometimes
the latter does not work for tags.
Change-Id: I01e71df99b0ab81f171f9e69e5190164e20f5106
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The old implementation of list-changes assumed that the deliverable file
included all releases and that they were in order. That assumption does
not hold for independent projects, where the history might not be there
and new entries might be from different branches than the previous
entries, all in the same file. This change uses git to determine the
previous tag for each repository, then shows the right diffs based on
that instead of the assumed tag from the deliverable file.
Change-Id: I122a6ec1792a07cdb23f09a880110dff2446555e
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
In the CI system we may clone from a repo that doesn't have everything,
such as tags, so after zuul-cloner is done force an update to pull in
the tags.
Change-Id: Ic4b5a217ed142ff83555a0fececb424a1a400fc9
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Before the fix, sha values were always extracted from master head, even
when series has a stable branch.
Change-Id: I9810660df7a92c021cd9c154bb3f24d8d828bfd7
The validation command in particular ends up cloning the same repo
repeatedly. Skip the extra work by checking to see if it already exists.
Change-Id: If56f03be03e37116edb8ee9443c9ab286c9bbdd9
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
