The tarball builds were using the wheelhouse paths
for find_links, that should have been using the
pip download cache path.
The pre-build upgrade now upgrades distribute, pbr,
and setuptools to handle the ;environment_factor
style requirements.
Change-Id: I68d3e090779d9b73613e3aab7fe9c7ee9513e45d
The http://ci.openstack.org/ documentation site has been deprecated,
replaced by redirects to corresponding paths within
http://docs.openstack.org/infra/ where other Project Infrastructure
documentation already resides.
Change-Id: I30e4029bf72f0bcafcb41e8c6a748ea7efb89d52
There's a check to make sure that we only run
mirroring against interesting branches by filtering
out any that aren't HEAD or don't start with
remotes/origin, but if we specify a branch on
the command line, we want that branch no matter what.
This is primarily so we can run mirroring tests
on proposed changes to openstack/requirements.
Change-Id: I19cd2b032b6219aed972abb95a180e3d402d529e
Replace URLs for workflow documentation to appropriate parts of the
OpenStack Project Infrastructure Manual. Pulled in the final form used
in the gear repo.
Change-Id: Ib4d23d0db1a91d51176105ee38a30073a789374a
Wheels get built for the OS that pypi-mirror is running
on, but the non arch specific wheels weren't going into
the main tarball mirror, which they should be able to.
This change puts the -any arch wheels into the main tarball
mirror. The method rename (_write_tarball_mirror to
_write_main_mirror) is to reflect that the mirror doesn't
just contain tarballs.
It also fixes an issue with the wheelhouse location.
Previously, the same wheelhouse location was used for
all mirror targets, but this could have easily led to
unwanted versions leaking.
Change-Id: I99223bd54c4512b142753d2b97f99275db82361f
pip 1.5.1 removed options that pypi-mirror needed for
reliable downloading, and pip 1.5.0 has problems with
some proxies, so this change updates the tmp virtualenvs
to use pip>6, uses the download only option, and
parses the package names from the tarball filenames.
Also, don't use the cached downloads as an extra search
path when preparing the build virtualenvs, as older
distributions will pick up newer versions of packages
that the system can't yet install, until the pip
and virtualenv has been upgraded inside the build
virtualenv.
Change-Id: I645938aa790ba5293a8c78d3b410c3fe339f383c
The new release of pip does not allow you to specify
a dowload cache location - it is deprecated. This means
that the tarball mirror cannot be created because it
relies on the download cache being in the configured
location. This change pins the versin of pip to 1.5.6
so the download cache location still works.
Change-Id: Ib4d058f27de3c37e0c58c50516e6af2b3f8188a5
Adding a '-r' parameter which allows for specifying one or more
requirements files instead of the built-in 'global-requirements.txt'
file.
This is to enable building an internal pypi mirror using additional
internal requirements files.
Change-Id: I506a9beddf773d6290639ed4d01588339ad3f99c
* pypi_mirror/cmd/run_mirror.py: Stop whitelisting netifaces,
psutil, pysendfile and pytidylib for insecure/external hosting
off-PyPI. They all now host versions on PyPI which meet the listed
requirements in openstack/requirements:global-requirements.txt for
currently supported branches.
Change-Id: I640a85435de7bb300aaaa347cb97b1fad7543e17
The README is very generic; every time I follow it, I end up with a repo
that doesn't work. The pip element's readme is much more useful for
setting up a mirror that works with DIB and TripleO.
Rather than replicate content from the element here, I've just added a
pointer to the element's README.
Change-Id: Id6d0f8a75d3fc91a72c4dbb9b9628d4c80dbfb9d
Currently the configuration syntax is only documented in the run_mirror
source. Adding enough information to the RAEDME for users to get up and
running with a simple configuration is much more user friendly.
Change-Id: I54d978ac127dfe37348167f3a2b08cc09119619b
When run-mirror downloads a package whose download files have a
different base name to the download directory, run-mirror ends up
writing the wrong directory name, which causes 404's on the mirror.
In this patch we use the actual upstream parent dir name exactly.
Change-Id: I90ca0e280dc59d26e5cfc70c8c02feeb734d1ae9
Closes-Bug: #1294522
Virtualenv has recently been upgraded to 1.11, which comes with pip 1.5. In
this version of pip, if 'pip install' can find a wheel, it downloads it instead
of the tar.gz. This is an issue since the version of pip used in the gates can
still be too old to work with wheels by default.
Use '--no-use-wheel' in the 'pip install' command to fix this.
Closes-Bug: #1276242
Change-Id: I00a32b3abd3b08703f66834ae30e6c4b0cf7bd7d
* pypi_mirror/cmd/run_mirror.py: Use the --allow-external and
--allow-insecure options so that pip 1.5 and later will assent to
retrieve the netifaces, psutil, pysendfile and pytidylib packages
even though they're not hosted on PyPI. The --allow-insecure option
is aliased to a clearer --allow-unverified wording in 1.5, but the
old form is being used to avoid breaking users of 1.4.x and will be
valid at least through 1.6.x according to comments in the pip
source.
Change-Id: I43d276fd6d294f74fae6d7ef36ac38d27a53f14e
OpenStack LLC isn't a thing, all of its stuff was transferred to
the OpenStack Foundation. Also, this is not an executable script,
so there is no need to have the shebang line.
Change-Id: I1b31a275e14ebc11a61b02788ffe5c0945f63bf0
Add an option to export the list of packages that got installed
to the mirror so that we can use the list to compare
the difference between the requirements (packages to be installed)
and the mirror (what actually got installed)
Change-Id: Ifda23d78ea4f324a145fc0279f4c80dc25570f73
Related-Bug: #1172417
On Centos, the lsb_release returns the text on one line. To solve that,
we need to replace not just newlines, but also spaces in the output
to match what lsb_release -i -r -s | xargs | tr ' ' '-' will do.
Change-Id: I5e4ecdb918d1f9e3c225974f89d7ceb4f1abc4f1
Clients of the mirror may need to fall back on non-wheel files,
and if an upstream project has published wheels, then the mirror
will be missing those in the non-wheel portion. Additionally,
use-wheel given to the wheel command will cause transitive depends
to be skipped in the local wheelhouse.
The only thing that should be making/injecting wheels is the wheel
creation portion of things.
Change-Id: I5e3c4754097249e52f013f6122ff86f01568243c
It turns out that upgrading setuptools and distribute in the venv
we use to build the mirror causes MySQL-python to error out. However,
as of virtualenv 1.10.1, there's a new enough version of pip (1.4)
so it's actually not necessary for us to explicitly upgrade.
Change-Id: Ib08064ad1ce674323cd5dec87b7a9911c723d12a
mysql-python is crapping the fantastic bong because of a continued
reliance on distribute. Fine. We'll just make sure we have latest
no-op package in our virtualenv so that epic fail stops killing bunnies.
Also, when we're running the wheel command in our venvs, we kinda need
wheel to be installed in them.
Change-Id: I004ca4efdbe5997b4f5f56e71a330ed7354d5cd2
* setup.cfg: The files section was missing, causing the sdist
package to lack the Python module being packaged. Fix it!
Change-Id: I0726708caf949f72e4fc244d746583e826a9bf9d
There are several places in the code where out is collected but not
used. These make the code harder to reason about. Remove them and
only actually get the output if we care.
Change-Id: I0280df56538fc7a56ee2d2bb47f05908270a3045
The mirror should build wheels, because they're awesome. However, because
the wheel format does not do fine-grained enough selection on
things like the high likelihood that Ubuntu precise and Ubuntu raring
might have different underlying C libraries for python 2.7, we need to
put our built wheels into a structure that allows them to be used
safely.
lsb_release -r -i is used because Centos lists lsb_release -c as "Final"
which is not particularly helpful. This will wind up with identifying
strings like Ubuntu-12.04 and Centos-6.4 - which should be good enough
for what we need. A call to a shell is used so that the same logic can
be easily duplicated in select-mirror, which is shell, to produce the
appropriate pip.conf file.
Change-Id: Icf360c921dbc357dbeb43386c6e8f51bd5f993ff
When creating git repositories, we want to recursively chown
them to the cgit user so the cgit user for replication can
write to them.
Change-Id: Ib349b4af32090d0fbda933d51ef514cae7033499
Git repos need to be created locally. Ensure that we create the
necessary repos when creating the config file for them.
Change-Id: I0b0b8e183049dd9f825224f1657cb88a64edf37b
Since we've decided to split puppet-openstack launchpad page into
several sub-projects, this patch update the configuration to
support both blueprints & bugs for all current puppet modules
in StackForge.
Change-Id: Iaef1de681507e9de9d80d6fecd2e5e56ca960a1a
global-requirements is a fully self-contained file, containing both
test and normal requirements. In the situation where the target repo
has a global-requirements.txt file, it should be treated as the only
file that matters- otherwise the mirror builder will hit double include
issues.
Change-Id: I01fd1e0d7f0923456f35a0f931159850cb52e3e7
