Browse Source

Adds tripleo-get-hash module get tripleo-ci hash info from tag

This is cherrypicked from [1] where it originally merged, except
minor update for setup.cfg to point to tripleo-repos instead of
tripleo-ci and tox.ini to include tripleo-get-hash in the tox
python test discovery and execution.

We want to replace the current ansible role and bash scripts
that fetch the hash for us in tripleo CI jobs.
This tripleo-get-hash module will be packaged to pypi, then used in
an ansible python module to replace the current ansible invocations
of get-hash.

[1] https://review.opendev.org/c/openstack/tripleo-ci/+/784392/

Change-Id: I256175f55a783fe5f4e787bcb0af76bbf09cc465
changes/35/789535/5
Marios Andreou 3 months ago
parent
commit
7101e4cdbc
17 changed files with 1104 additions and 1 deletions
  1. +5
    -1
      tox.ini
  2. +13
    -0
      tripleo-get-hash/LICENSE
  3. +49
    -0
      tripleo-get-hash/README.md
  4. +46
    -0
      tripleo-get-hash/config.yaml
  5. +2
    -0
      tripleo-get-hash/requirements.txt
  6. +33
    -0
      tripleo-get-hash/setup.cfg
  7. +19
    -0
      tripleo-get-hash/setup.py
  8. +1
    -0
      tripleo-get-hash/test-requirements.txt
  9. +0
    -0
      tripleo-get-hash/test/__init__.py
  10. +112
    -0
      tripleo-get-hash/test/fakes.py
  11. +206
    -0
      tripleo-get-hash/test/test_tripleo_get_hash.py
  12. +206
    -0
      tripleo-get-hash/test/test_tripleo_get_hash_info.py
  13. +0
    -0
      tripleo-get-hash/tripleo_get_hash/__init__.py
  14. +113
    -0
      tripleo-get-hash/tripleo_get_hash/__main__.py
  15. +33
    -0
      tripleo-get-hash/tripleo_get_hash/constants.py
  16. +48
    -0
      tripleo-get-hash/tripleo_get_hash/exceptions.py
  17. +218
    -0
      tripleo-get-hash/tripleo_get_hash/tripleo_hash_info.py

+ 5
- 1
tox.ini View File

@ -11,7 +11,11 @@ deps =
-c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
-r{toxinidir}/test-requirements.txt
-r{toxinidir}/requirements.txt
commands = stestr run --slowest {posargs}
-r{toxinidir}/tripleo-get-hash/requirements.txt
-r{toxinidir}/tripleo-get-hash/test-requirements.txt
commands =
stestr run --slowest {posargs}
stestr run --combine --slowest {posargs} --test-path ./tripleo-get-hash/test --top-dir ./tripleo-get-hash
[testenv:venv]
commands = {posargs}


+ 13
- 0
tripleo-get-hash/LICENSE View File

@ -0,0 +1,13 @@
Copyright 2021 Red Hat, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.

+ 49
- 0
tripleo-get-hash/README.md View File

@ -0,0 +1,49 @@
# tripleo-get-hash
## What is tripleo-get-hash
This utility is meant for use by TripleO deployments, particularly in zuul
continuous integration jobs. Given an RDO named tag, such as 'current-tripleo'
or 'tripleo-ci-testing' [1] it will return the hash information, including
the commit, distro and full hashes where available.
It includes a simple command line interface. If you clone the source you can
try it out of the box without installation invoking it as a module:
```
python -m tripleo_get_hash # by default centos8, master, current-tripleo.
python -m tripleo_get_hash --component tripleo --release victoria --os-version centos8
python -m tripleo_get_hash --release master --os-version centos7
python -m tripleo_get_hash --release train # by default centos8
python -m tripleo_get_hash --os-version rhel8 --release osp16-2 --dlrn-url http://osp-trunk.hosted.upshift.rdu2.redhat.com
python -m tripleo_get_hash --help
```
## Quick start
```
python setup.py install
```
The tripleo-get-hash utility uses a yaml configuration file named 'config.yaml'.
If you install this utility using setup.py as above, the configuration file
is placed in /etc:
```
/etc/tripleo_get_hash/config.yaml
```
Alternatively if you are running from a checked out version of the repo and
invoking as a module (see examples above) the config.yaml in the repo checkout
is used instead.
After installation you can invoke tripleo-get-hash in /usr/local/bin/:
```
tripleo-get-hash --help
```
By default this queries the delorean server at "https://trunk.rdoproject.org",
with this URL specified in config.yaml. To use a different delorean server you
can either update config.yaml or use the --dlrn-url parameter to the cli. If
instead you are instantiating TripleOHashInfo objects in code, you can create
the objects passing an existing 'config' dictionary. Note this has to contain
all of constants.CONFIG_KEYS to avoid explosions.
[1] https://docs.openstack.org/tripleo-docs/latest/ci/stages-overview.html#rdo-dlrn-promotion-criteria

+ 46
- 0
tripleo-get-hash/config.yaml View File

@ -0,0 +1,46 @@
# This file is installed to the path in [options.data_files] of the project
# setup.cfg file. It *must* contain all the keys specified in constants
# CONFIG_KEYS or there will be explosions.
dlrn_url: 'https://trunk.rdoproject.org'
tripleo_releases:
- master
- wallaby
- victoria
- ussuri
- train
- osp16-2
- osp17
tripleo_ci_components:
- baremetal
- cinder
- clients
- cloudops
- common
- compute
- glance
- manila
- network
- octavia
- security
- swift
- tempest
- tripleo
- ui
- validation
rdo_named_tags:
- current
- consistent
- component-ci-testing
- promoted-components
- tripleo-ci-testing
- current-tripleo
- current-tripleo-rdo
os_versions:
- centos7
- centos8
- rhel8

+ 2
- 0
tripleo-get-hash/requirements.txt View File

@ -0,0 +1,2 @@
PyYAML
requests

+ 33
- 0
tripleo-get-hash/setup.cfg View File

@ -0,0 +1,33 @@
[metadata]
name = tripleo-get-hash
author = Marios Andreou
author_email = marios@redhat.com
description = Get the tripleo build hash for a known RDO named tag.
long_description = file: README.md LICENSE
long_description_content_type = text/markdown
url = https://opendev.org/openstack/tripleo-repos/tripleo-get-hash/
project_urls =
Bug Tracker = https://launchpad.net/tripleo
license_file = LICENSE
license = Apache-2.0
classifiers =
License :: OSI Approved :: Apache License, Version 2.0
Programming Language :: Python
[options]
package_dir =
= .
packages = find:
python_requires = >=3.6
install_requires =
pyyaml
requests
tests_require =
requests_mock
[options.entry_points]
console_scripts =
tripleo-get-hash = tripleo_get_hash.__main__:cli_entrypoint
[options.data_files]
/etc/tripleo_get_hash/ = config.yaml

+ 19
- 0
tripleo-get-hash/setup.py View File

@ -0,0 +1,19 @@
# Copyright 2021 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
import setuptools
setuptools.setup(setup_requires=['pbr'], pbr=True)

+ 1
- 0
tripleo-get-hash/test-requirements.txt View File

@ -0,0 +1 @@
requests_mock

+ 0
- 0
tripleo-get-hash/test/__init__.py View File


+ 112
- 0
tripleo-get-hash/test/fakes.py View File

@ -0,0 +1,112 @@
# Copyright 2021 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
TEST_COMMIT_YAML_COMPONENT = """
commits:
- artifacts: repos/component/common/47/6a/476a52df13202a44336c8b01419f8b73b93d93eb_1f5a41f3/openstack-tacker-4.1.0-0.20210325043415.476a52d.el8.src.rpm,repos/component/common/47/6a/476a52df13202a44336c8b01419f8b73b93d93eb_1f5a41f3/python3-tacker-doc-4.1.0-0.20210325043415.476a52d.el8.noarch.rpm,repos/component/common/47/6a/476a52df13202a44336c8b01419f8b73b93d93eb_1f5a41f3/python3-tacker-tests-4.1.0-0.20210325043415.476a52d.el8.noarch.rpm,repos/component/common/47/6a/476a52df13202a44336c8b01419f8b73b93d93eb_1f5a41f3/openstack-tacker-common-4.1.0-0.20210325043415.476a52d.el8.noarch.rpm,repos/component/common/47/6a/476a52df13202a44336c8b01419f8b73b93d93eb_1f5a41f3/python3-tacker-4.1.0-0.20210325043415.476a52d.el8.noarch.rpm,repos/component/common/47/6a/476a52df13202a44336c8b01419f8b73b93d93eb_1f5a41f3/openstack-tacker-4.1.0-0.20210325043415.476a52d.el8.noarch.rpm
civotes: '[]'
commit_branch: master
commit_hash: 476a52df13202a44336c8b01419f8b73b93d93eb
component: common
distgit_dir: /home/centos8-master-uc/data/openstack-tacker_distro/
distro_hash: 1f5a41f31db8e3eb51caa9c0e201ab0583747be8
dt_build: '1616646776'
dt_commit: '1616646661.0'
dt_distro: '1616411951'
dt_extended: '0'
extended_hash: None
flags: '0'
id: '21047'
notes: OK
project_name: openstack-tacker
promotions: '[]'
repo_dir: /home/centos8-master-uc/data/openstack-tacker
status: SUCCESS
type: rpm
""" # noqa
TEST_COMMIT_YAML_CENTOS_7 = """
commits:
- artifacts: repos/b5/ef/b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465/openstack-tripleo-heat-templates-12.1.1-0.20200227052810.b5ef03c.el7.src.rpm,repos/b5/ef/b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465/openstack-tripleo-heat-templates-12.1.1-0.20200227052810.b5ef03c.el7.noarch.rpm
commit_branch: master
commit_hash: b5ef03c9c939db551b03e9490edc6981ff582035
component: None
distgit_dir: /home/centos-master-uc/data/openstack-tripleo-heat-templates_distro/
distro_hash: 76ebc4655502820b7677579349fd500eeca292e6
dt_build: '1582781227'
dt_commit: '1582780705.0'
dt_distro: '1580409403'
dt_extended: '0'
extended_hash: None
flags: '0'
id: '86894'
notes: OK
project_name: openstack-tripleo-heat-templates
repo_dir: /home/centos-master-uc/data/openstack-tripleo-heat-templates
status: SUCCESS
type: rpm
""" # noqa
TEST_REPO_MD5 = 'a96366960d5f9b08f78075b7560514e7'
BAD_CONFIG_FILE = """
awoo: 'foo'
"""
CONFIG_FILE = """
dlrn_url: 'https://trunk.rdoproject.org'
tripleo_releases:
- master
- wallaby
- victoria
- ussuri
- train
- osp16-2
- osp17
tripleo_ci_components:
- baremetal
- cinder
- clients
- cloudops
- common
- compute
- glance
- manila
- network
- octavia
- security
- swift
- tempest
- tripleo
- ui
- validation
rdo_named_tags:
- current
- consistent
- component-ci-testing
- tripleo-ci-testing
- current-tripleo
- current-tripleo-rdo
os_versions:
- centos7
- centos8
- rhel8
"""

+ 206
- 0
tripleo-get-hash/test/test_tripleo_get_hash.py View File

@ -0,0 +1,206 @@
# Copyright 2021 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
import requests_mock
import sys
import unittest
from unittest import mock
from unittest.mock import mock_open
import yaml
import tripleo_get_hash.exceptions as exc
import tripleo_get_hash.__main__ as tgh
import test.fakes as test_fakes
@mock.patch(
'builtins.open', new_callable=mock_open, read_data=test_fakes.CONFIG_FILE
)
class TestGetHash(unittest.TestCase):
"""In this class we test the CLI invocations for this module.
The builtin 'open' function is mocked at a
class level so we can mock the config.yaml with the contents of the
fakes.CONFIG_FILE
"""
def test_centos_8_current_tripleo_stable(self, mock_config):
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos8-victoria/current-tripleo/delorean.repo.md5', # noqa
text=test_fakes.TEST_REPO_MD5,
)
args = ['--os-version', 'centos8', '--release', 'victoria']
sys.argv[1:] = args
main_res = tgh.main()
self.assertEqual(main_res.full_hash, test_fakes.TEST_REPO_MD5)
self.assertEqual(
'https://trunk.rdoproject.org/centos8-victoria/current-tripleo/delorean.repo.md5', # noqa
main_res.dlrn_url,
)
self.assertEqual('centos8', main_res.os_version)
self.assertEqual('victoria', main_res.release)
def test_verbose_logging_on(self, mock_config):
args = ['--verbose']
debug_msgs = []
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5', # noqa
text=test_fakes.TEST_REPO_MD5,
)
with self.assertLogs() as captured:
sys.argv[1:] = args
tgh.main()
debug_msgs = [
record.message
for record in captured.records
if record.levelname == 'DEBUG'
]
self.assertIn('Logging level set to DEBUG', debug_msgs)
def test_verbose_logging_off(self, mock_config):
debug_msgs = []
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5', # noqa
text=test_fakes.TEST_REPO_MD5,
)
args = ['--tag', 'current-tripleo', '--os-version', 'centos8']
with self.assertLogs() as captured:
sys.argv[1:] = args
tgh.main()
debug_msgs = [
record.message
for record in captured.records
if record.levelname == 'DEBUG'
]
self.assertEqual(debug_msgs, [])
def test_invalid_unknown_components(self, mock_config):
args = ['--component', 'nosuchcomponent']
sys.argv[1:] = args
self.assertRaises(SystemExit, lambda: tgh.main())
def test_valid_tripleo_ci_components(self, mock_config):
config_file = open("fake_config_file") # open is mocked at class level
config_yaml = yaml.safe_load(config_file.read())
config_file.close()
# interate for each of config components
for component in config_yaml['tripleo_ci_components']:
with requests_mock.Mocker() as req_mock:
req_mock.get(
"https://trunk.rdoproject.org/centos8-master/component"
"/{}/current-tripleo/commit.yaml".format(
component
),
text=test_fakes.TEST_COMMIT_YAML_COMPONENT,
)
args = ['--component', "{}".format(component)]
sys.argv[1:] = args
main_res = tgh.main()
self.assertEqual(
"https://trunk.rdoproject.org/centos8-master/component"
"/{}/current-tripleo/commit.yaml".format(
component
),
main_res.dlrn_url,
)
self.assertEqual("{}".format(component), main_res.component)
def test_invalid_component_centos7(self, mock_config):
args = ['--os-version', 'centos7', '--component', 'tripleo']
sys.argv[1:] = args
self.assertRaises(exc.TripleOHashInvalidParameter, lambda: tgh.main())
def test_invalid_os_version(self, mock_config):
args = ['--os-version', 'rhelos99', '--component', 'tripleo']
sys.argv[1:] = args
self.assertRaises(SystemExit, lambda: tgh.main())
def test_invalid_unknown_tag(self, mock_config):
args = ['--tag', 'nosuchtag']
sys.argv[1:] = args
self.assertRaises(SystemExit, lambda: tgh.main())
def test_valid_rdo_named_tags(self, mock_config):
config_file = open("fake_config_file") # open is mocked at class level
config_yaml = yaml.safe_load(config_file.read())
config_file.close()
# iterate for each of config named tags
for tag in config_yaml['rdo_named_tags']:
with requests_mock.Mocker() as req_mock:
req_mock.get(
"https://trunk.rdoproject.org/centos8-master"
"/{}/delorean.repo.md5".format(
tag
),
text=test_fakes.TEST_REPO_MD5,
)
args = ['--tag', "{}".format(tag)]
sys.argv[1:] = args
main_res = tgh.main()
self.assertEqual(
"https://trunk.rdoproject.org/centos8-master"
"/{}/delorean.repo.md5".format(
tag
),
main_res.dlrn_url,
)
self.assertEqual(tag, main_res.tag)
def test_override_dlrn_url(self, mock_config):
with requests_mock.Mocker() as req_mock:
req_mock.get(
"https://awoo.com/awoo/centos8-master/current-tripleo"
"/delorean.repo.md5",
text=test_fakes.TEST_REPO_MD5,
)
args = ['--dlrn-url', 'https://awoo.com/awoo']
sys.argv[1:] = args
main_res = tgh.main()
self.assertEqual(
"https://awoo.com/awoo/centos8-master/current-tripleo"
"/delorean.repo.md5",
main_res.dlrn_url,
)
def test_override_os_version_release_rhel8(self, mock_config):
with requests_mock.Mocker() as req_mock:
req_mock.get(
"https://awoo.com/awoo/rhel8-osp16-2/current-tripleo"
"/delorean.repo.md5", text=test_fakes.TEST_REPO_MD5,
)
args = [
'--dlrn-url',
'https://awoo.com/awoo',
'--os-version',
'rhel8',
'--release',
'osp16-2',
]
sys.argv[1:] = args
main_res = tgh.main()
self.assertEqual('rhel8', main_res.os_version)
self.assertEqual('osp16-2', main_res.release)
self.assertEqual(
"https://awoo.com/awoo/rhel8-osp16-2/current-tripleo"
"/delorean.repo.md5", main_res.dlrn_url,
)
if __name__ == '__main__':
unittest.main()

+ 206
- 0
tripleo-get-hash/test/test_tripleo_get_hash_info.py View File

@ -0,0 +1,206 @@
# Copyright 2021 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
import unittest
import tripleo_get_hash.tripleo_hash_info as thi
import tripleo_get_hash.exceptions as exc
import test.fakes as test_fakes
import requests_mock
from unittest import mock
from unittest.mock import mock_open
@mock.patch(
'builtins.open', new_callable=mock_open, read_data=test_fakes.CONFIG_FILE
)
class TestGetHashInfo(unittest.TestCase):
"""In this class we test the functions and instantiation of the
TripleOHashInfo class. The builtin 'open' function is mocked at a
class level so we can mock the config.yaml with the contents of the
fakes.CONFIG_FILE
"""
def test_hashes_from_commit_yaml(self, mock_config):
sample_commit_yaml = test_fakes.TEST_COMMIT_YAML_COMPONENT
expected_result = (
'476a52df13202a44336c8b01419f8b73b93d93eb_1f5a41f3',
'476a52df13202a44336c8b01419f8b73b93d93eb',
'1f5a41f31db8e3eb51caa9c0e201ab0583747be8',
'None',
)
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos8-master/component/common/current-tripleo/commit.yaml', # noqa
text=test_fakes.TEST_COMMIT_YAML_COMPONENT,
)
mock_hash_info = thi.TripleOHashInfo(
'centos8', 'master', 'common', 'current-tripleo'
)
actual_result = mock_hash_info._hashes_from_commit_yaml(
sample_commit_yaml
)
self.assertEqual(expected_result, actual_result)
def test_resolve_repo_url_component_commit_yaml(self, mock_config):
with requests_mock.Mocker() as req_mock:
# test component url
req_mock.get(
'https://trunk.rdoproject.org/centos8-master/component/common/current-tripleo/commit.yaml', # noqa
text=test_fakes.TEST_COMMIT_YAML_COMPONENT,
)
c8_component_hash_info = thi.TripleOHashInfo(
'centos8', 'master', 'common', 'current-tripleo'
)
repo_url = c8_component_hash_info._resolve_repo_url("https://woo")
self.assertEqual(
repo_url,
'https://woo/centos8-master/component/common/current-tripleo/commit.yaml', # noqa
)
def test_resolve_repo_url_centos8_repo_md5(self, mock_config):
with requests_mock.Mocker() as req_mock:
# test vanilla centos8 url
req_mock.get(
'https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5', # noqa
text=test_fakes.TEST_REPO_MD5,
)
c8_hash_info = thi.TripleOHashInfo(
'centos8', 'master', None, 'current-tripleo'
)
repo_url = c8_hash_info._resolve_repo_url("https://woo")
self.assertEqual(
repo_url, 'https://woo/centos8-master/current-tripleo/delorean.repo.md5' # noqa
)
def test_resolve_repo_url_centos7_commit_yaml(self, mock_config):
with requests_mock.Mocker() as req_mock:
# test centos7 url
req_mock.get(
'https://trunk.rdoproject.org/centos7-master/current-tripleo/commit.yaml', # noqa
text=test_fakes.TEST_COMMIT_YAML_CENTOS_7,
)
c7_hash_info = thi.TripleOHashInfo(
'centos7', 'master', None, 'current-tripleo'
)
repo_url = c7_hash_info._resolve_repo_url("https://woo")
self.assertEqual(
repo_url, 'https://woo/centos7-master/current-tripleo/commit.yaml' # noqa
)
def test_get_tripleo_hash_info_centos8_md5(self, mock_config):
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5', # noqa
text=test_fakes.TEST_REPO_MD5,
)
created_hash_info = thi.TripleOHashInfo(
'centos8', 'master', None, 'current-tripleo'
)
self.assertIsInstance(created_hash_info, thi.TripleOHashInfo)
self.assertEqual(
created_hash_info.full_hash, test_fakes.TEST_REPO_MD5
)
self.assertEqual(created_hash_info.tag, 'current-tripleo')
self.assertEqual(created_hash_info.os_version, 'centos8')
self.assertEqual(created_hash_info.release, 'master')
def test_get_tripleo_hash_info_component(self, mock_config):
expected_commit_hash = '476a52df13202a44336c8b01419f8b73b93d93eb'
expected_distro_hash = '1f5a41f31db8e3eb51caa9c0e201ab0583747be8'
expected_full_hash = '476a52df13202a44336c8b01419f8b73b93d93eb_1f5a41f3' # noqa
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos8-victoria/component/common/tripleo-ci-testing/commit.yaml', # noqa
text=test_fakes.TEST_COMMIT_YAML_COMPONENT,
)
created_hash_info = thi.TripleOHashInfo(
'centos8', 'victoria', 'common', 'tripleo-ci-testing'
)
self.assertIsInstance(created_hash_info, thi.TripleOHashInfo)
self.assertEqual(created_hash_info.full_hash, expected_full_hash)
self.assertEqual(
created_hash_info.distro_hash, expected_distro_hash
)
self.assertEqual(
created_hash_info.commit_hash, expected_commit_hash
)
self.assertEqual(created_hash_info.component, 'common')
self.assertEqual(created_hash_info.tag, 'tripleo-ci-testing')
self.assertEqual(created_hash_info.release, 'victoria')
def test_get_tripleo_hash_info_centos7_commit_yaml(self, mock_config):
expected_commit_hash = 'b5ef03c9c939db551b03e9490edc6981ff582035'
expected_distro_hash = '76ebc4655502820b7677579349fd500eeca292e6'
expected_full_hash = 'b5ef03c9c939db551b03e9490edc6981ff582035_76ebc465' # noqa
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/commit.yaml', # noqa
text=test_fakes.TEST_COMMIT_YAML_CENTOS_7,
)
created_hash_info = thi.TripleOHashInfo(
'centos7', 'master', None, 'tripleo-ci-testing'
)
self.assertIsInstance(created_hash_info, thi.TripleOHashInfo)
self.assertEqual(created_hash_info.full_hash, expected_full_hash)
self.assertEqual(
created_hash_info.distro_hash, expected_distro_hash
)
self.assertEqual(
created_hash_info.commit_hash, expected_commit_hash
)
self.assertEqual(created_hash_info.os_version, 'centos7')
def test_bad_config_file(self, mock_config):
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/commit.yaml', # noqa
text=test_fakes.TEST_COMMIT_YAML_CENTOS_7,
)
with mock.patch(
'builtins.open',
new_callable=mock_open,
read_data=test_fakes.BAD_CONFIG_FILE,
):
self.assertRaises(
exc.TripleOHashInvalidConfig,
thi.TripleOHashInfo,
'centos7',
'master',
None,
'tripleo-ci-testing',
)
def test_missing_config_file(self, mock_config):
with requests_mock.Mocker() as req_mock:
req_mock.get(
'https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/commit.yaml', # noqa
text=test_fakes.TEST_COMMIT_YAML_CENTOS_7,
)
with mock.patch('os.path.isfile') as mock_is_file:
mock_is_file.return_value = False
self.assertRaises(
exc.TripleOHashMissingConfig,
thi.TripleOHashInfo,
'centos7',
'master',
None,
'tripleo-ci-testing',
)

+ 0
- 0
tripleo-get-hash/tripleo_get_hash/__init__.py View File


+ 113
- 0
tripleo-get-hash/tripleo_get_hash/__main__.py View File

@ -0,0 +1,113 @@
# Copyright 2021 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
import argparse
import logging
import sys
from tripleo_get_hash.tripleo_hash_info import TripleOHashInfo
import tripleo_get_hash.exceptions as exc
def _validate_args(parsed_args):
if parsed_args.os_version == 'centos7' and (
parsed_args.component is not None
):
raise exc.TripleOHashInvalidParameter(
'Cannot specify component for centos 7'
)
def main():
TripleOHashInfo.load_logging()
config = TripleOHashInfo.load_config()
parser = argparse.ArgumentParser(description='tripleo-get-hash.py')
parser.add_argument(
'--component',
help=('Use this to specify a component '
'This is NOT valid for Centos 7.'),
choices=config['tripleo_ci_components'],
)
parser.add_argument(
'--dlrn-url',
help=(
'The URL for the delorean server to use. Defaults to '
'https://trunk.rdoproject.org'
),
)
parser.add_argument(
'--os-version',
default='centos8',
choices=config['os_versions'],
help=('The operating system and version to fetch the build tag for'),
)
parser.add_argument(
'--tag',
default='current-tripleo',
choices=config['rdo_named_tags'],
help=('The known tag to retrieve the hash_info for'),
)
parser.add_argument(
'--release',
default='master',
help=('The release of OpenStack you want the hash info for. '
'Default master'),
choices=config['tripleo_releases'],
)
parser.add_argument(
'--verbose',
action='store_true',
help=('Enable verbose log level for debugging'),
)
args = parser.parse_args()
if args.verbose:
logging.getLogger().setLevel(logging.DEBUG)
logging.debug("Logging level set to DEBUG")
_validate_args(args)
if args.dlrn_url is not None:
logging.debug(
"Overriding configuration dlrn_url. Original value {}. "
"New value {}".format(config['dlrn_url'], args.dlrn_url)
)
config['dlrn_url'] = args.dlrn_url
logging.debug(
"Proceeding with the following configuration: {}".format(config)
)
tripleo_hash_info = TripleOHashInfo(
args.os_version,
args.release,
args.component,
args.tag,
config,
)
tripleo_hash_info.pretty_print()
return tripleo_hash_info
def cli_entrypoint():
try:
main()
sys.exit(0)
except KeyboardInterrupt:
logging.info("Exiting on user interrupt")
raise
if __name__ == "__main__":
main()

+ 33
- 0
tripleo-get-hash/tripleo_get_hash/constants.py View File

@ -0,0 +1,33 @@
# Copyright 2021 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
"""
These are the keys we expect to find in a well-formed config.yaml
If any keys are missing from the configuration hash resolution doesn't proceed.
"""
CONFIG_KEYS = [
'dlrn_url',
'tripleo_releases',
'tripleo_ci_components',
'rdo_named_tags',
'os_versions',
]
"""
This is the path that we expect to find the system installed config.yaml.
The path is specified in [options.data_files] of the project setup.cfg.
"""
CONFIG_PATH = '/etc/tripleo_get_hash/config.yaml'

+ 48
- 0
tripleo-get-hash/tripleo_get_hash/exceptions.py View File

@ -0,0 +1,48 @@
# Copyright 2021 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
class Base(Exception):
"""Base Exception"""
class TripleOHashMissingConfig(Base):
"""Missing configuration file for TripleOHashInfo. This is thrown
when there is no config.yaml in constants.CONFIG_PATH or the local
directory assuming execution from a source checkout.
"""
def __init__(self, error_msg):
super(TripleOHashMissingConfig, self).__init__(error_msg)
class TripleOHashInvalidConfig(Base):
"""Invalid configuration file for TripleOHashInfo. This is used when
any of they keys in constants.CONFIG_KEYS is not found in config.yaml.
"""
def __init__(self, error_msg):
super(TripleOHashInvalidConfig, self).__init__(error_msg)
class TripleOHashInvalidParameter(Base):
"""Invalid parameters passed for TripleOHashInfo. This is thrown when
the user passed invalid combination ofparameters parameters to the cli
entrypoint, for example specifying --component with centos7.
"""
def __init__(self, error_msg):
super(TripleOHashInvalidParameter, self).__init__(error_msg)

+ 218
- 0
tripleo-get-hash/tripleo_get_hash/tripleo_hash_info.py View File

@ -0,0 +1,218 @@
# Copyright 2021 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
import logging
import sys
import yaml
import os
import requests
import tripleo_get_hash.constants as const
import tripleo_get_hash.exceptions as exc
class TripleOHashInfo:
"""
Objects of type TripleOHashInfo contain the attributes required to
represent a particular delorean build hash. This includes the full, commit,
distro and extended hashes (where applicable), as well as the release,
OS name and version, component name (if applicable), named tag
(current-tripleo, tripleo-ci-testing etc) as well as the URL to the
delorean server that provided the information used to build each object
instance.
"""
@classmethod
def load_logging(cls):
"""
This is a class method since we call it from the CLI entrypoint
before the TripleOHashInfo object is created. Default is to add
logging.INFO level logging.
"""
logger = logging.getLogger()
# Only add logger once to avoid duplicated streams in tests
if not logger.handlers:
stdout_handlers = [
_handler
for _handler in logger.handlers
if
(
hasattr(_handler, 'stream') and 'stdout' in
_handler.stream.name
)
]
if stdout_handlers == []:
formatter = logging.Formatter(
(
"%(asctime)s - tripleo-get-hash - %(levelname)s - "
"%(message)s"
)
)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
@classmethod
def load_config(cls):
"""
This is a class method since we call it from the CLI entrypoint
before the TripleOHashInfo object is created. The method will first
try to use constants.CONFIG_PATH. If that is missing it tries to use
a local config.yaml for example for invocations from a source checkout
directory. If the file is not found TripleOHashMissingConfig is raised.
If any of the contants.CONFIG_KEYS is missing from config.yaml then
TripleOHashInvalidConfig is raised. Returns a dictionary containing
the key->value for all the keys in constants.CONFIG_KEYS.
:raises TripleOHashMissingConfig for missing config.yaml
:raises TripleOHashInvalidConfig for missing keys in config.yaml
:returns a config dictionary with the keys in constants.CONFIG_KEYS
"""
def _check_read_file(filepath):
if os.path.isfile(filepath) and os.access(filepath, os.R_OK):
return True
return False
result_config = {}
config_path = ''
# if this isn't installed and running from a source checkout then
# try to use local ../config.yaml
local_config = os.path.join(
os.path.split(os.path.split(os.path.abspath(__file__))[0])[0],
'config.yaml'
)
# If we can read /etc/tripleo_get_hash/config.yaml then use that
if _check_read_file(const.CONFIG_PATH):
config_path = const.CONFIG_PATH
elif _check_read_file(local_config):
config_path = local_config
else:
raise exc.TripleOHashMissingConfig(
"Configuration file not found at {} or {}".format(
const.CONFIG_PATH, local_config
)
)
logging.info("Using config file at {}".format(config_path))
with open(config_path, 'r') as config_yaml:
conf_yaml = yaml.safe_load(config_yaml)
for k in const.CONFIG_KEYS:
if k not in conf_yaml:
error_str = (
"Malformed config file - missing {}. Expected all"
"of these configuration items: {}"
).format(
k, ", ".join(const.CONFIG_KEYS)
)
logging.error(error_str)
raise exc.TripleOHashInvalidConfig(error_str)
loaded_value = conf_yaml[k]
result_config[k] = loaded_value
return result_config
def __init__(self, os_version, release, component, tag, config=None):
"""Create a new TripleOHashInfo object
:param os_version: The OS and version e.g. centos8
:param release: The OpenStack release e.g. wallaby
:param component: The tripleo-ci component e.g. 'common' or None
:param tag: The Delorean server named tag e.g. current-tripleo
:param config: Use an existing config dictionary and don't load it
"""
if config is None:
config = TripleOHashInfo.load_config()
self.os_version = os_version
self.release = release
self.component = component
self.tag = tag
repo_url = self._resolve_repo_url(config['dlrn_url'])
self.dlrn_url = repo_url
repo_url_response = requests.get(repo_url).text
if repo_url.endswith('commit.yaml'):
from_commit_yaml = self._hashes_from_commit_yaml(repo_url_response)
self.full_hash = from_commit_yaml[0]
self.commit_hash = from_commit_yaml[1]
self.distro_hash = from_commit_yaml[2]
self.extended_hash = from_commit_yaml[3]
else:
self.full_hash = repo_url_response
self.commit_hash = None
self.distro_hash = None
self.extended_hash = None
def _resolve_repo_url(self, dlrn_url):
"""Resolve the delorean server URL given the various attributes of
this TripleOHashInfo object. The only passed parameter is the
dlrn_url. There are three main cases:
* centos8/rhel8 component https://trunk.rdoproject.org/centos8/component/common/current-tripleo/commit.yaml
* centos7 https://trunk.rdoproject.org/centos7/current-tripleo/commit.yaml
* centos8/rhel8 non component https://trunk.rdoproject.org/centos8/current-tripleo/delorean.repo.md5
Returns a string which is the full URL to the required item (i.e.
commit.yaml or repo.md5 depending on the case).
:param dlrn_url: The base url for the delorean server
:returns string URL to required commit.yaml or repo.md5
""" # noqa
repo_url = ''
if 'centos7' in self.os_version:
repo_url = "%s/%s-%s/%s/commit.yaml" % (
dlrn_url,
self.os_version,
self.release,
self.tag,
)
elif self.component is not None:
repo_url = "%s/%s-%s/component/%s/%s/commit.yaml" % (
dlrn_url,
self.os_version,
self.release,
self.component,
self.tag,
)
else:
repo_url = "%s/%s-%s/%s/delorean.repo.md5" % (
dlrn_url,
self.os_version,
self.release,
self.tag,
)
logging.debug("repo_url is {}".format(repo_url))
return repo_url
def _hashes_from_commit_yaml(self, delorean_result):
"""This function is used when a commit.yaml file is returned
by _resolve_repo_url. Returns a tuple containing the various
extracted hashes: full, commit, distro and extended
:returns tuple of strings full, commit, distro, extended hashes
"""
parsed_yaml = yaml.safe_load(delorean_result)
commit = parsed_yaml['commits'][0]['commit_hash']
distro = parsed_yaml['commits'][0]['distro_hash']
full = "%s_%s" % (commit, distro[0:8])
extended = parsed_yaml['commits'][0]['extended_hash']
logging.debug(
"delorean commit.yaml results {}".format(parsed_yaml['commits'][0])
)
return full, commit, distro, extended
def pretty_print(self):
attrs = vars(self)
print(',\n'.join('%s: %s' % item for item in attrs.items()))

Loading…
Cancel
Save