Commit Graph

24 Commits

Author SHA1 Message Date
danpawlik
ce73ffedaa Added updates dir for Fedora 31 release
Patch set [1] does not provide updates directory for
Fedora 31 which is required to run CI jobs.

[1] https://review.opendev.org/#/c/712680/

Change-Id: I01b9b50660f3c7a3b4eb05cc8a8c0847f47dc7ee
2020-03-16 09:08:41 +01:00
danpawlik
b5bb9790b4 Added new Fedora release to mirroring
Some jobs are using Fedora 29 release, which is deprecated.
Create new mirror with release 31 and wait for changing job
rules.

Change-Id: I418f4c815d337c2b1edd0df1aa065536d992cb62
2020-03-12 14:46:31 +01:00
Ian Wienand
97c4735129 Move afsmon to mirror-update.opendev.org
This migrates the afsmon script from puppet deploying on
mirror-update.openstack.org to ansible deploying on
mirror-update.opendev.org.

There is nothing particularly special and this just a straight install
with some minor dependencies.  Since we have log publishing running on
the opendev.org server, we publish the update logs alongside the
others.

Change-Id: Ifa3b4d59f8d0fc23a4492e50348bab30766d5779
2020-02-12 14:38:48 +11:00
Ian Wienand
fbb9790d49 Allow for periodic afs releases from mirror-update
This is a migration of the current periodic "vos release" script to
mirror-update.opendev.org.

The current script is deployed by puppet and run by a cron job on
afsdb01.dfw.openstack.org.

My initial motivation for this was wanting to better track our release
of these various volumes.  With tarballs and releases moving to AFS
publishing, we are going to want to track the release process more
carefully.

Initially, I wanted to send timing statistics to graphite so we could
build a dashboard and track the release times of all volumes.  Because
this requires an additional libraries and since we are deprecating
puppet, further development there is unappealing and it would better
live in ansible.

Since I6c96f89c6f113362e6085febca70d58176f678e7 we have the ability to
call "vos release" with "-localauth" permissions via ssh on
mirror-update; this avoids various timeout issues (see the changelog
comment there for more details).  So we do not need to run this script
directly on the afsdb server.

We are alreadying publishing mirror update logs from mirror-update,
and it would be good to also publish these release logs so anyone can
see if there are problems.

All this points to mirror-update.opendev.org being a good future home
for this script.

The script has been refactored some to

 - have a no-op mode
 - send timing stats for each volume release
 - call "vos release" via the ssh mecahnism we created
 - use an advisory lock to avoid running over itself

It runs from a virtualenv and it's logs are published via the same
mechanism as the mirror logs (slightly misnamed now).

Note this script is currently a no-op to test the deployment, running
and log publishing.  A follow-up will disable the old job and make
this active.

Change-Id: I62ae941e70c7d58e00bc663a50d52e79dfa5a684
2020-02-11 08:52:01 +11:00
Ian Wienand
f62a75fae9 centos mirror: drop -p from rsync
This appears to be the same thing we saw in Fedora with
Id24196791f80cd99fe8a330fb2c7c6d893fc9995, somehow upstream
directories have started acquiring a setgid bit, which breaks AFS
mirroring

 rsync: failed to set permissions on "/afs/.openstack.org/mirror/centos/8/AppStream/aarch64/os": Permission denied (13)

and when we look

 chmod("AppStream/aarch64/os/Packages", 02755) = -1 EACCES (Permission denied)

Drop the "-p" so we don't try and replicate these permissions.

Change-Id: Ib5db052cdd23e39aecbeead15cf08d4bd7fcab38
2020-01-30 18:26:10 +11:00
Ian Wienand
98ab53ee42 rsync mirrors: use localauth vos release
We switched Fedora to do vos release via ssh with localauth in
I56ecdb2511597197deeeadf51f50da7e02f56954 and it has been working.

Switch the rest of the update scripts.  There is an increasing amount
of common code, start a common functions.sh script where we can put
this.

Change-Id: I4ba6d64a84bb66e8686901b16010352de942f303
2019-11-26 13:40:15 +00:00
Andreas Jaeger
e474f5ce73 Switch openSUSE mirror
The former mirror has outdated content, use a mirror from
http://download.opensuse.org/update/leap/15.1/oss/repodata/repomd.xml.mirrorlist
that is in the US with prio 100 (highest) and current content.

Change-Id: I5649584d01f0370fb09217b00038992a69cf964c
2019-11-25 09:07:25 +01:00
Ian Wienand
3f9c737483 Fedora mirror update: use localauth release
Use the new vos_release user on the remote host to release the volume
via localauth, to avoid any timeouts.

Change-Id: I56ecdb2511597197deeeadf51f50da7e02f56954
Depends-On: https://review.opendev.org/#/c/695554/
2019-11-22 01:13:50 +00:00
Andreas Jaeger
d2095d96b7 Remove mirroring of openSUSE 15.0
openSUSE 15.0 has been removed from infra, remove it from mirrors as
well.

Change-Id: I0ce4a2cf27963c0a63c86b6705c16e379eafc627
2019-11-13 20:33:54 +01:00
Ian Wienand
fd7ffbcab4 Fedora mirror - remove 28
Add a purge phase to remove old releases; remove Fedora 28 and stop
mirroring.  Update the atomic list while we're there.

Fedora 28 was removed with Ic0b4b065a217dcfaa8c230cda53114793e93b803

Change-Id: If713844ac90ea37e8c4db30108c45d7a59832776
2019-10-28 13:10:17 +11:00
Andreas Jaeger
757aebbfc3 Stop openSUSE 42.3 mirroring
openSUSE 42.3 is removed from Infra, remove it from mirroring as well.

Change-Id: I60c77fd48e55edbacd589c2f94de977a034eb9c9
2019-10-16 11:06:21 +02:00
Ian Wienand
1ef7315ae4 mirror-update: Add EPEL 8 mirroring
Change-Id: I293fe01806743e3ec45877bdadd3e5b0fb851aee
2019-10-02 14:30:41 +10:00
Ian Wienand
d59e654936 Add CentOS 8 mirror
This adds mirroring of CentOS 8.  It is somewhat simpler because the
architecture we're interested in are in the base repos, no need for
altarch.

The current mirror doesn't have a 8/ directory; possibly they require
their own mirroring filter updates?  Use an up-to-date mirror for 8
(we can switch 7 too, but leaving alone for now).

Additionally, the altarch mirror we are using appears to have gone
offline for at least a few days.  Switch to another one that is in
Texas, which should be close-ish to the DFW servers.

Change-Id: I33d95fa6b2df23fbfdb6745a3079761e228f677b
2019-09-25 06:28:23 +00:00
Itxaka
1087cbf433
Revert to rit.edu mirror for Suse
Looks like leaseweb, kernel.org and others are not properly
syncing the Suse mirrors as they are out of date for Leap15.0 and
missing files (deltainfo.xml.gz) which causes job breakage for
Leap15.0 based jobs.

Revert this to the original mirror from a year ago which is updated
and not broken

Change-Id: Id7184ee973bbabfec3f601fc9200ffac17322558
2019-09-23 16:25:57 +02:00
Radosław Piliszek
dc2510aac7 Mirror OBS ha-clustering repo for CentOS 7
Kolla uses this to build hacluster images.
Direct usage is causing timeouts recently.

I changed the formatting to make it more readable
and slapped a comment note for maintainers.

Change-Id: I68d7155718c0ae0744198ca96aca1a207bab7ed6
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-09-17 10:25:57 +02:00
Ian Wienand
3dcfe61d2f fedora mirror update : add sleep
As described inline, this should make our mirror pulses more robust
against timeouts.

This is probably ripe for turning into more of a library situation for
all the other "vos release" calls too.  But one thing at a time ... I
think we test with this for a while to see if stability returns.

Change-Id: I041a290053e4e8ceba80785598a5945e5adcf6f1
2019-09-11 13:15:29 +10:00
Ian Wienand
c9bea2cfc4 fedora-mirror: Use rsync itemize output
We are debugging why the fedora mirror has started to take 8+ hours to
release.  There is suspicion that rsync is updating the metadata of a
lot of files, causing the incremental update to be more of a full
release.

If you read the man page carefully:

 At the single -v level of verbosity, this does not mention when a
 file gets its attributes changed.  If you ask for an itemized list of
 changed attributes (either --itemize-changes or adding "%i" to the
 --out-format setting), the output (on the client) increases to
 mention all items that are changed in any way

So we need "-i" to see these updates

Change-Id: Ia9cff1e126404bc5a3346e758a15314834aef409
2019-08-30 13:29:54 +10:00
Ian Wienand
b617a08989 fedora-mirror-update: update logging
We're having trouble with this volume; this makes the logging more
helpful for humans trying to establish the order of things happening.

It turns down the tracing, puts timestamps on the logs messages, makes
the messages more descriptive, and makes sure to timestamp the output
of the vos commands so we can see how long each step took.

Change-Id: Ia832b7ef86350efcbbe7d3b957e820a8a39df3e1
2019-08-27 14:09:49 +10:00
Clark Boylan
a248dbae63 Add more fedora-atomic mirror exclusions
We don't need things like efi configs or pxeboot configs or ISOs.
Exclude these items to further reduce the size of the fedora mirror.

Change-Id: I93003b2f48d79dae627026e2c7af748ea9a9e34d
2019-08-19 15:53:19 -07:00
Clark Boylan
3ff4bed27c Trim fedora mirror
The fedora mirror is our largest mirror (850GB about twice as big as the
next mirror). Much of this size is due to the fedora atomic images we
mirror.

On further investigation I notice that we are mirroring ppc images (for
which we do not have cpus to run them on), image for fedora 25 and 36
which are quite EOL'd, and our exclusion of the raw.xz and vagrant
images is failing.

Update the rsync excludes to ensure we don't mirror any of these images
we don't need.

Change-Id: I86856cb4e51b0e687aac45a1f014f87c5141318f
2019-08-02 14:35:21 -07:00
Clark Boylan
f686ec39f5 Switch fedora mirroring to pubmirror2.math.uh.edu
pubmirror1.math.uh.edu is currently offline and listed as an altonly.
pubmirror2 seems to work fine so switch to it.

Change-Id: I2562f8686146d17d4fad3997b9be22361fa05fca
2019-08-02 14:27:00 -07:00
Kien Nguyen
9b2862d5f9 Allow to rsync Centos Software Collections repo
Remove Centos sclo from exclude list.

Change-Id: I8aae878d4dbb40b537913c66912320a91a4ac99b
Needed-By: https://review.opendev.org/#/c/671178/
2019-07-18 17:06:11 +07:00
Ian Wienand
959f0301e7 mirror-update: export mirroring logs
This adds a periodic job to copy logs to a mirror volume, and export
it via the usual mirror http.

I have precreated the log volume; just as a R/W volume because this is
expected to be very low volume access.

Change-Id: I67870f6d439af2d2a63a5048ef52cecff3e75275
2019-07-04 09:11:29 +10:00
Ian Wienand
b85282c046 Move rsync mirror updates to new opendev.org mirror-update host
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).

Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.

Most magic is included in the scripts, so there is not much more to do
than copy them.  The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).

Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously.  This will also require a manual
clean to remove the cron jobs as a once-off when merging.

The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories.  They are left as future work for now.

Testing is added to ensure dependencies and scripts are all in place.

Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
2019-07-02 16:42:33 +10:00