29 Commits

Author SHA1 Message Date
Jeremy Stanley
cfb735ca38 Fix PBR name check with new tox
Changes to tox in the past year have rendered the PBR-specific
tweaks in get_sdist_name invalid, since calling tox from within tox
now creates testenv directories relative to the parent tox's working
directory rather than the child's. While we're here, turn on more
verbose logging in the tox invocation, and fix the subprocess
routine to not discard the command's output.

Change-Id: I35fb5dde9ec400397ed21035d57c3efb872d8e58
2023-11-03 17:19:12 +00:00
Zuul
7fc4981f74 Merge "Remove PBR-specific validation code" 2022-08-11 13:53:23 +00:00
Thierry Carrez
6075722275 Do not validate PyPI uploaders for new packages
The validate_pypi_permissions test checks that if the package exists on
PyPI, we have permissions to upload it. In order to validate that, it
first checks that the package exists by retrieving JSON information from
PyPI, and assuming that anything returned means the package exists.

However currently PyPI returns {'message': 'Not Found'} when queried for
JSON information about a non-existant package. This makes the test
assume that the package exists but we don't have permissions to upload
it, therefore failing the test.

This fix makes sure an empty dict is returned when an inexistant package
is queried, which makes the test succeed with the following message:

'no %s project data on pypi, assuming it will be created by release'

Change-Id: I27eb82109ab93fe33c339f82ec519feeb7228a1f
2022-08-09 14:34:27 +02:00
Jeremy Stanley
b10e4a0d7d Remove PBR-specific validation code
With the recent switch to using the build utility, this should no
longer be necessary. Even more recent changes in the Python
packaging toolchain seem to have made it stop working anyway, so
remove it and treat PBR's builds like any other project.

Change-Id: I6f9a4546fb6a7b2a4836d6aeea0713f7a7da3002
2022-08-05 15:35:45 +00:00
Iury Gregory Melo Ferreira
83571291e5 Fix typo in build wheel command
A missing comma was causing the strings to be concatenated,
resulting in a wrong command that caused the validate job
to fail.

Change-Id: I7c5dab48db725608c7b1cfa4e8e8034d8225b165
2022-05-06 11:54:57 -03:00
Előd Illés
97baab754f Replace old sdist and wheel build command in validate
The new way of packaging a python project is via build package. This
patch replaces the old setup.py style sdist and wheel build.

Change-Id: Ia351d099aa3b2491e3a4d255d87bb7c6bd691cac
2022-05-02 20:26:47 +02:00
Thierry Carrez
efc876585e Avoid PyPI ratelimit on roles retrieval retry
When validating PyPI settings, we retrieve package roles. If that
retrieval is not successful on the first try, we immediately retry
with a canonicalized version of the package. This sometimes hits
PyPI ratelimits. This patch introduces a one second sleep to avoid
that error.

Change-Id: Ifd20200159a2f136827a1987606b650a6db5588f
2021-03-31 12:36:13 +02:00
Sean McGinnis
f402cde902
Pass full environ copy to sdist subprocess
In order to tell pbr not to build release notes and create CHANGELOG and
AUTHORS files, we need to pass flags in as environment variables. We
can't just pass these flags though, as that will exclude the rest of the
normal environment set in. So copy the current environment, append our
updates, and pass that in explicitly to the subprocess call.

Change-Id: Ia47e73b51b3216572d51399338e95bd4269a8d1d
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-05-11 17:09:19 -05:00
Sean McGinnis
c75a909d1f
Drop PBR environment flags from sdist build
Change-Id: Ia36d9eab7d9f3e29d8804b592210ab6fb5e06148
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-05-11 10:26:34 -05:00
Sean McGinnis
854d9e270b
Speed up sdist build
We have a couple validations that need to build an sdist now. But once
it has been built in our temporary pristine clone, we don't need to
build it again. Because we create a clean clone, we also don't need to
worry about any stale dist builds. This adds a check to the sdist build
logic to recognize if an sdist has already been built and skip it.

There are also a few extra steps in the build process that are not
needed for our purposes. The biggest is the new(ish) pbr behavior of
building all release notes in the git tree if reno is present. This adds
an environment variable flag to tell pbr not to do that. While we're at
it, we don't need AUTHORS or CHANGELOG created either, so set the flags
telling pbr to skip those as well.

Change-Id: I15be9a12ff6684729980c40afae9625a8b8e550a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-05-10 17:42:35 -05:00
Sean McGinnis
44d6736a5e
Be explicit about python3 executable
Per best practices, explicitly use the python3 executable rather than
assuming the platform with have a "python" executable that maps to
python3.

https://www.python.org/dev/peps/pep-0394/

Change-Id: I39b8a1013a891f4570f374d8faa3cfa2ecaf3347
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2019-10-16 09:45:29 -05:00
Sean McGinnis
5359acf788
Use recommended twine check for README validation
The use of ``python setup.py check -r -s`` is actually deprecated and
the recommended method for validating a README for PyPi upload is to
build the sdist and wheel and run ``twine check $DIST_PATH``. This
switches our validation check to use the supported method.

Change-Id: I321b8cf6f1e2116d2fe23059d5f13aa6f2ae8b4c
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2019-04-05 10:56:16 -05:00
Doug Hellmann
b952446ba2 fix URL in get_pypi_info()
The JSON API path is wrong.

See https://warehouse.pypa.io/api-reference/json/

Change-Id: I1574e568e6c3eeddc3619591b6335c314bf92c39
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2019-03-07 18:33:24 -05:00
Doug Hellmann
81047e0096 add validation for sdist building
Try to build the sdist and wheel for python packages.

Change-Id: I6cf0ff8e7771367bce17061c9b21b7740516d31d
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-08-09 15:15:31 -04:00
Doug Hellmann
dcb0bbaa27 add a validation step to check README.rst parsing for PyPI
PyPI is now more strict about requiring README.rst files to be
parseable. Failures to parse the readme block uploading new versions
of packages. To detect these errors in advance, we should validate the
README formatting as part of validating a release.

Change-Id: I416befcf3e47818680018394fa63de8983351587
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-05-29 14:42:26 -04:00
Thierry Carrez
7e8f409ebc Fix XMLRPC URL after Pypi URL update
A seemingly-trivial change updated the URL for the
XMLRPC Pypi service from https://pypi.python.org/pypi
to https://pypi.org/project, while the XMLRPC service
lives at https://pypi.org/pypi. This resulted in
systematic validation failures.

Change-Id: I97218093a9346a7d84f58136764cce53930ddacb
2018-04-25 14:26:26 +02:00
Tovin Seven
0290a15965 Trivial: Update pypi url to new url
Pypi url changed from [1] to [2]

[1] https://pypi.python.org/pypi/<package>
[2] https://pypi.org/project/<package>

Change-Id: Iffc845fad9dab3c9af8c7b78bfa9fb8ab6fe0153
2018-04-20 17:29:00 +07:00
Doug Hellmann
4ad924d16f more robustly handle error when we can't determine sdist name
Instead of passing None to the service as the sdist name, report that
situation as an error.

Change-Id: Ib6380d997db780ae324a4df61fdfa7f17e4507c3
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-27 14:59:03 -04:00
Doug Hellmann
d6951955bd update list-deliverables to use object model classes
Change-Id: I93d0125d895c875f3e3a5e1579ec63b70ab16cae
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-08 13:18:31 -05:00
Doug Hellmann
9b8f290117 use packaging to determine the canonical name for projects on PyPI
Rather than applying the rules ourselves, use the packaging project
utility function to prepare the canonical name.

Change-Id: I6310585ff14c376e954b0edc9def67cd65a5d550
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-07 10:49:50 -05:00
Doug Hellmann
40a3e39d38 be consistent with use of log levels and print
Change-Id: I0bd5b6f871cc59e6fb5f29da9daa175524541132
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-07 10:49:49 -05:00
Doug Hellmann
1bb935477a add validation step for pypi permissions
Projects that are going to try to upload releases to PyPI need to have
given openstackci permission to do that.

Change-Id: I8e5e60ea6bf2842e834d2352134a11b1f9fe02b9
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-02-08 17:07:42 -05:00
Doug Hellmann
59a998da02 update missing-releases to handle non-python projects better
Skip the PyPI checks for things that are not python modules.

Change-Id: I15b0e42c961f7fae600f88ffe23edc22d1f98922
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-02-08 15:22:02 -05:00
Doug Hellmann
bfd96f4fc0 wrap subprocess to capture output in tests
Provide some wrappers around subprocess functions to capture stderr
and stdout and divert it to the logs so that the test output is less
cluttered with the output of various git commands.

Change-Id: If36ef013aca498e3a0a9cc3a2b78d666775439ab
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-02-08 05:14:13 -06:00
Doug Hellmann
6c7a19708b extend missing-releases to look at pypi
Extend the missing-releases command to look at PyPI for uploads there.

Change-Id: I5693686bbe46a4ec90944bbf2fa81e1c5c91749e
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-01-29 10:07:01 -05:00
Doug Hellmann
6dbce7dd42 make validate work with python 3
Update the check_output() calls used by the validate command so the
results are decoded before being treated as strings.

Change-Id: I099ff997f69743d2958fc5581292aacebd7f5d37
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-03-03 15:00:04 -05:00
Doug Hellmann
270492fc5a be more robust when getting the sdist name
Only look at the last line of output, in case the project relies on an
old pbr that spews extra information to stdout instead of stderr.

Comment out the 0.6.0 release because getting its name triggers an error
related to testrepository.

Change-Id: I9eb2bd7ad17130115fb189ef5296cff18af6edb4
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-03-02 13:48:21 -05:00
Doug Hellmann
d1566749d6 use tox env when asking pbr for its name
It seems pbr needs to be installed in order to get its name, so use tox
instead of the system python.

Change-Id: I8809978b23430daacfc6a48d1f4bc84cce24c003
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-02-08 07:53:33 -05:00
Doug Hellmann
32e9f4d7cc add validation rule to check tarball-base
We don't want to take the time to check out all of the source code for
all projects during the documentation build, so add a validation step to
check that the deliverable being modified has the tarball-base value set
properly, if it needs to be overridden from the default.

Change-Id: Id39bdd1e0a9fe771b78a053ba73803c3c095f135
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2016-12-16 16:26:28 -05:00