From 91577f551a77e1c65d9866246fbbe8ab66276f9c Mon Sep 17 00:00:00 2001 From: Doug Hellmann Date: Tue, 21 Jun 2016 15:38:52 -0400 Subject: [PATCH] move the release tools that need to run on secure nodes Move the scripts that need to run on secure nodes to automatically tag releases from release-tools to project-config Change-Id: Ia451d75998e50c3e82536c6cad72e851e1c8f18e Depends-On: I66e15cd0fa270e9b7a5cb3849cb8d14fa4c617b1 Signed-off-by: Doug Hellmann --- README.rst | 78 ++++++++--------------- launchpad_add_comment.py | 78 ----------------------- release.sh | 129 --------------------------------------- release_from_yaml.sh | 71 --------------------- 4 files changed, 27 insertions(+), 329 deletions(-) delete mode 100755 launchpad_add_comment.py delete mode 100755 release.sh delete mode 100755 release_from_yaml.sh diff --git a/README.rst b/README.rst index 769023e..61b1a48 100644 --- a/README.rst +++ b/README.rst @@ -26,12 +26,12 @@ if you start a release and run into trouble with launchpad auth. When a release request is ready to be approved, follow these steps: -1. The release team member taking responsibility for the +#. The release team member taking responsibility for the release should approve the change in ``openstack/releases``. Release requests should not be approved until we are actually ready to cut the release. -2. After the release request merges, check out or update a local copy +#. After the release request merges, check out or update a local copy of ``openstack/releases`` to get the new version of the file under the ``deliverables`` directory. Make sure you check out the releases repository to the commit with the new release request you @@ -39,22 +39,38 @@ When a release request is ready to be approved, follow these steps: time. The release tools only look at the most recent commit to detect which deliverable files have changed. -3. In a local copy of this - ``openstack-infra/release-tools`` repository, run - ``release_from_yaml.sh``, giving the path to the +#. Check out or update a local copy of + ``openstack-infra/project-config`` repository, which contains the + tools for tagging a release. + +#. Change directories to + ``project-config/jenkins/scripts/release-tools``. + +#. Create or update a virtualenv and install all of the dependencies + for the release tools using + ``openstack-infra/project-config/jenkins/scripts/release-tools/requirements.txt`` + as the basis. + + For example:: + + $ virtualenv .venv + $ source .venv/bin/activate + $ pip install -U -r requirements.txt + +#. Run ``release_from_yaml.sh``, giving the path to the ``openstack/releases`` repository. For example:: $ ./release_from_yaml.sh ~/repos/openstack/releases -4. As the release script runs, it will prompt you for your GPG key +#. As the release script runs, it will prompt you for your GPG key passphrase before adding the tag. This gives you a last chance to review the proposed tag before proceeding. After the tag is created locally and pushed up to the remote server, the script will push comments to closed Launchpad bugs since the previous tag. -5. Announce the release. +#. Announce the release. 1. Milestones are manually announced once all projects are done (usually at the closing of the milestone window), using an email @@ -84,51 +100,11 @@ Top-level scripts The top-level scripts call the various base tools to get their work done. +.. note:: -release_from_yaml.sh --------------------- - -This script takes YAML files describing deliverables to release (like those -living in ``openstack/releases``) and calls the release.sh script (see below) -to apply the corresponding tags. It will create a tag for the last release -mentioned in the file(s). You can point it to specific YAML files, or to a -local git repository (in which case it will look at the files modified in the -most recent commit). - -Examples: - -:: - - ./release_from_yaml.sh ../openstack-releases deliverables/mitaka/nova.yaml - -Call release.sh for all repositories mentioned in the last release added -to ../openstack-releases/deliverables/mitaka/nova.yaml - -:: - - ./release_from_yaml.sh ../openstack-releases - -Look into the git repository at ../openstack-releases for deliverable YAML -files modified at the last commit, and call release.sh for all repositories -mentioned on the last release in each such file. - - -release.sh ----------- - -This script creates a tag on a given repository SHA and pushes it to Gerrit. -Additionally it will add a message on Launchpad bugs that are mentioned as -"closed" in git commit messages since the last tag on the same series. - -Example: - -:: - - ./release.sh openstack/oslo.rootwrap mitaka 3.0.3 gerrit/master - -Apply a 3.0.3 tag (associated to the mitaka series) to the gerrit master -HEAD of the openstack/oslo.rootwrap reporitory, and add a comment for each -closed bug mentioned in commit messages since the previous mitaka tag (3.0.2). + The scripts involved in tagging releases have moved from this + repository to + ``openstack-infra/project-config/jenkins/scripts/release-tools``. announce.sh ----------- diff --git a/launchpad_add_comment.py b/launchpad_add_comment.py deleted file mode 100755 index efd8a7f..0000000 --- a/launchpad_add_comment.py +++ /dev/null @@ -1,78 +0,0 @@ -#!/usr/bin/env python -# -# Add a comment on a number of Launchpad bugs -# -# Copyright 2015 Thierry Carrez -# All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. - -from __future__ import print_function - -import argparse -import os - -import launchpadlib.launchpad -import lazr.restfulclient.errors - - -def main(): - # Parameters - parser = argparse.ArgumentParser(description="Add comment on bugs") - parser.add_argument('--subject', help='The comment subject', - default='Comment added by add_comment') - parser.add_argument('--content', help='The comment content', - default='Comment added by add_comment') - lp_grp = parser.add_argument_group('launchpad') - lp_grp.add_argument( - "--test", - action='store_const', - dest='lp_service_root', - const='staging', - default='production', - help='Use LP staging server to test', - ) - lp_grp.add_argument( - '--credentials-file', '-c', - dest='lp_creds_file', - default=os.environ.get('LP_CREDS_FILE'), - help=('plain-text credentials file, ' - 'defaults to value of $LP_CREDS_FILE'), - ) - parser.add_argument('bugs', type=int, nargs='+', - help='Bugs to add comment to') - args = parser.parse_args() - - # Connect to Launchpad - print("Connecting to Launchpad...") - launchpad = launchpadlib.launchpad.Launchpad.login_with( - application_name='openstack-releasing', - service_root=args.lp_service_root, - credentials_file=args.lp_creds_file, - ) - - # Add comment - for bugid in args.bugs: - print("Adding comment to #%d..." % bugid, end='') - try: - bug = launchpad.bugs[bugid] - bug.newMessage(subject=args.subject, content=args.content) - print (" done.") - except lazr.restfulclient.errors.ServerError as e: - print(" TIMEOUT during save !") - except Exception as e: - print(" ERROR during save ! (%s)" % e) - - -if __name__ == '__main__': - main() diff --git a/release.sh b/release.sh deleted file mode 100755 index 1aef31e..0000000 --- a/release.sh +++ /dev/null @@ -1,129 +0,0 @@ -#!/bin/bash -# -# Script to release a project in one shot, including the git tag and -# launchpad updates. -# -# Copyright 2015 Thierry Carrez -# All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. - -set -e - -TOOLSDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -source $TOOLSDIR/functions - -function usage { - echo "Usage: release.sh [-a] repository series version SHA announce include_pypi first-full-release extra-metadata" - echo - echo "Example: release.sh openstack/oslo.rootwrap mitaka 3.0.3 gerrit/master openstack-dev@lists.openstack.org yes no 'meta:release:Workflow+1: Doug Hellmann '" -} - -if [ $# -lt 5 ]; then - usage - exit 2 -fi - -REPO=$1 -SERIES=$2 -VERSION=$3 -SHA=$4 -ANNOUNCE=$5 -INCLUDE_PYPI=${6:-no} -FIRST_FULL=${7:-no} -EXTRA_METADATA="$8" - -SHORTNAME=`basename $REPO` - -pre_release_pat='\.[[:digit:]]+[ab][[:digit:]]+' -rc_release_pat='\.[[:digit:]]+rc[[:digit:]]+' -if [[ $VERSION =~ $pre_release_pat ]]; then - RELEASETYPE="development milestone" -elif [[ $VERSION =~ $rc_release_pat ]]; then - RELEASETYPE="release candidate" -else - RELEASETYPE="release" -fi - -setup_temp_space release-tag-$SHORTNAME -clone_repo $REPO -REPODIR="$(cd $REPO && pwd)" -cd $REPODIR -TARGETSHA=`git log -1 $SHA --format='%H'` - -# Determine the most recent tag before we add the new one. -PREVIOUS=$(get_last_tag $TARGETSHA) - -title "Tagging $TARGETSHA as $VERSION" -if git show-ref "$VERSION"; then - echo "$REPO already has a version $VERSION tag" - # Reset the notion of "previous" to the version associated with - # the parent of the commit being tagged, since the tag we're - # applying already exists. - PREVIOUS=$(get_last_tag ${TARGETSHA}^1) -else - # WARNING(dhellmann): announce.sh expects to be able to parse this - # commit message, so if you change the format you may have to - # update announce.sh as well. - TAGMSG="$SHORTNAME $VERSION $RELEASETYPE - -meta:version: $VERSION -meta:series: $SERIES -meta:release-type: $RELEASETYPE -meta:announce: $ANNOUNCE -meta:pypi: $INCLUDE_PYPI -meta:first: $FIRST_FULL -$EXTRA_METADATA -" - echo "Tag message is '$TAGMSG'" - git tag -m "$TAGMSG" -s "$VERSION" $TARGETSHA - git push gerrit $VERSION -fi - -# We don't want to die just because we can't update some bug reports, -# so ignore failures. -set +e - -title "Adding comments to fixed bugs" -BUGS=$(git log $PREVIOUS..$VERSION | egrep -i "Closes(.| )Bug:" | egrep -o "[0-9]+") -if [[ -z "$BUGS" ]]; then - echo "No bugs found $PREVIOUS .. $VERSION" -else - $TOOLSDIR/launchpad_add_comment.py \ - --subject="Fix included in $REPO $VERSION" \ - --content="This issue was fixed in the $REPO $VERSION $RELEASETYPE." \ - $BUGS -fi - -# Try to propose a constraints update for libraries. -if [[ $INCLUDE_PYPI == "yes" ]]; then - title "Proposing constraints update" - dist_name=$(python setup.py --name) - if [[ -z "$dist_name" ]]; then - echo "Could not determine the name of the constraint to update" - else - cd $MYTMPDIR - clone_repo openstack/requirements stable/$SERIES - cd openstack/requirements - git checkout -b "$dist_name-$VERSION" - sed -e "s/^${dist_name}=.*/$dist_name===$VERSION/" --in-place upper-constraints.txt - git commit -a -m "update constraint for $dist_name to new release $VERSION - -$TAGMSG -" - git show - git review -t 'new-release' - fi -fi - -exit 0 diff --git a/release_from_yaml.sh b/release_from_yaml.sh deleted file mode 100755 index d2d5b41..0000000 --- a/release_from_yaml.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/bin/bash -# -# Script to release projects based on changes to the deliverables -# files in the openstack/releases repository. -# -# All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. - -set -e - -TOOLSDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -source $TOOLSDIR/functions - -function usage { - echo "Usage: release_from_yaml.sh releases_repository [deliverable_files]" - echo - echo "Example: release_from_yaml.sh ~/repos/openstack/releases" - echo "Example: release_from_yaml.sh ~/repos/openstack/releases" - echo "Example: release_from_yaml.sh ~/repos/openstack/releases deliverables/mitaka/oslo.config.yaml" -} - -if [ $# -lt 1 ]; then - echo "ERROR: No releases_repository specified" - echo - usage - exit 1 -fi - -RELEASES_REPO="$1" -shift -DELIVERABLES="$@" - -# Configure git to pull the notes where gerrit stores review history -# like who approved a patch. -cd $RELEASES_REPO -if ! git config --get remote.origin.fetch | grep -q refs/notes/review; then - git config --add remote.origin.fetch refs/notes/review:refs/notes/review - git config --add core.notesRef refs/notes/review - git remote update origin -fi - -# Look for metadata about the release instructions to include in the -# tag message. -if git show --no-patch --pretty=format:%P | grep -q ' '; then - # multiple parents, look at the submitted patch instead of the - # merge commit - parent=$(git show --no-patch --pretty=format:%P | cut -f2 -d' ') -else - # single parent, look at the current patch - parent='' -fi -RELEASE_META=$(git show --format=full --show-notes=review $parent | egrep -i '(Author|Commit:|Code-Review|Workflow|Change-Id)' | sed -e 's/^ //g' -e 's/^/meta:release:/g') - -$TOOLSDIR/list_deliverable_changes.py -r $RELEASES_REPO $DELIVERABLES \ -| while read deliverable series version repo hash announce_to pypi first_full; do - title "$repo $series $version $hash $announce_to" - $TOOLSDIR/release.sh $repo $series $version $hash $announce_to $pypi $first_full "$RELEASE_META" -done - -exit 0