I was a little too hasty in I76acbd08acda20c88ff9fd9148e3918b78d6c6c6
removing the scripts/ directory. It has broken the puppeting of old
hosts.
Restore the directory with a blank file explaining the situation.
Also, we don't need to copy this in the nodepool elements; remove that.
Change-Id: I8b82950237ef69c4941866900cac9bda42f58ca2
We now only have afs-slug.sh in jenkins/scripts/ and it is only
used by nodepool/scripts/configure_mirror.sh. Move the file into the
nodepool/scripts directory and adjust paths.
Remove copying of jenkins/scripts from nodepool, the directory does not
exist anymore.
Remove 89-jenkins-scripts, we don't need it anymore.
Depends-On: https://review.openstack.org/548393
Change-Id: I13cac791d70a2e5bcb0de1ba0adcd79e1b0aee9f
We no longer have the ability to run ready-scripts in nodepool, so
let's delete / clean them up.
Change-Id: I2551037dcfba2362d23401536295eb8f8f5d3e0d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We likely want to change all our variables to have trailing slashes,
but need to confirm other clients will be fine first. I've already
tested docker, so make the change.
Change-Id: I2c3fcb0b99b45b04068ef9be6a2887adf1f0ebab
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Both gem install and bundle install will use the
NODEPOOL_RUGYGEMS_PROXY variable. However, each client has a different
configuration file.
Change-Id: I329d871d6dff5b8df0b3066363c52b56f1f6fc7f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Now that we have a working reverse proxy cache for api.rubygems.org,
add the .gemrc file and start using it.
Change-Id: I6d8cd1d1353fc28469aceb16d487cd9b39eda73b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This is a partial revert and update to our NPM mirror settings. We now
have a reverse proxy change enabled on our mirrors. Update .npmrc to
use them now.
This reverts commit 0b00bd6abd3478f9b904e5561338f7216d0e2911.
Change-Id: Ic73c72627811d2f92ef08478edbd2282d23feb4f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
In I8336aeb85689cfaac8a794a5c3f5c62273d8ded0 a reverse proxy
for images.linuxcontainers.org was added to the infra services.
This adds a record in the mirror host env vars so that it's
easy to know which host name to use when executing lxc-create.
eg: lxc-create --template download --name test1 -- --server ${NODEPOOL_LXC_IMAGE_PROXY} --dist ubuntu --release xenial --arch amd64
Change-Id: I457fb98c0aa4e4b4a77acbc77a927a94643b5d65
All jobs to reference buildlogs.centos from /etc/ci/mirror_info.sh.
Change-Id: Ic0e8008261f47ac9496c2a3d8696c81475556bd3
Depends-On: I9d9d51971e52e8dc594d9d46bafc60147ad3c96b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We now proxy tarballs.o.o via apache reverse proxy, add a variable to
make it easier for jobs to use it.
Change-Id: I6d8084c0d0443d9e499215acd06d24484d22f91a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The configure mirror script is trying to run utilities like ip and
restorecon which at least on some distros are hidden away in /usr/sbin
and not where nodepool would find them otherwise. Explicitly add /sbin,
/usr/sbin, /bin, /usr/bin, and /usr/local/bin to the end of PATH to list
all of the various places we might find things. With this in place we
should be able to find ip and restorecon whereever they are.
Change-Id: I800cd69e47b4b099b453e47f1b579ab015ed628f
When we modify the forwarding.conf in our ready script we have to
restore the selinux contexts if on centos and fedora. If we don't the
unbound services fails to read the file (despite normal file system
permissions being fine). Address this by restorecon'ing the file if
restorecon is present (implying this is an selinux enabled host).
Change-Id: Ib8a64b0ba9ee4144a6123d7b8f7c1c8fc9501112
We need sudo permissions to write the forwarding.conf file.
Change-Id: I788915fccc5ac92b57b6a0815b90c40e32948bed
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
If we have ipv6 networking that is routable then only use ipv6 name
resolvers to avoid traversing potential NAT with ipv4. If we don't have
ipv6 available and routable then only use ipv4 resolvers to avoid
timeouts and failovers.
Change-Id: I885418e07265002b09592c3cf3aff9712c6e001f
Rather than a sophisticated sed explicitely set the repository
mirros similarly like for all other distributions.
Change-Id: I846cd43b1f9c5949b656738959b74d60c21c98ef
It seems this optimization is no longer needed. Once we address:
https://bugs.launchpad.net/testrepository/+bug/1416512
We could consider adding it back. However, this also means we stop
hitting logstash.openstack.org each time we build an image. So, we
should maybe consider caching that data some place over having our
image builds do it.
Change-Id: I16b56a13028c0f228d21dde423db3f70f07f4e1a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The NPM mirror has unfortunately become unuseable as NPM itself has
grown to many terabytes in size and we haven't been able to keep up
downstream. Stop using the mirror by default as its current state is
causing some jobs to fail.
Change-Id: Iab9ece625ca71bb02c9c9b77ca7be9c89bcc47b9
It is possible for our APT lists on disk to be out-of-date. Be sure to
update them, so we don't leave jobs in a broken state.
Change-Id: I019c1832fcaa0a6f162f71e9a8168e2b202f9a79
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We'll we don't populate files on disk, give a way for projects to
source them first.
Change-Id: I58380647ad16cdffe5ced78c632caf2b05ddc5f3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Provide a simple way for projects to source nodepool mirror
variables. We construct the new file in such a way that it doesn't
depend on nodepool, but existing nodepool scripts use it just as
downstream consumers would so that we are sure that everything lines up
without double accounting.
Change-Id: I2642659406e72090cffe5163783290e22d6ef3c2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
I think we want to go with: https://review.openstack.org/#/c/339261/
This reverts commit 5151d13338fc78f2a8ffde8abdd98d45dbc8d530.
Change-Id: I2e26ce98d82c18009e237c6ae9042411e43f8959
In various places we have wanted to have just the mirror host location
and not specific mirror config like for pip or apt or yum. For example
devstack-gate attempts to use this information to do simple networking
ping tests, dib tests construct yum repo locations on ubuntu and apt
repo locations on centos, and so on. Have a clear file that is set to
the root mirror location if a mirror is in use to make this simpler.
Change-Id: I306467f1870e5ee823b8b5c98d81917e607348ef
Until now, this wasn't an issue. However it seems fedora 25 started
looking for a drpm file today:
[MIRROR] pcre-8.40-5.fc25_8.40-6.fc25.x86_64.drpm: Status code: 404 for
http://mirror.iad.rax.openstack.org/fedora/updates/25/x86_64/drpms/pcre-8.40-5.fc25_8.40-6.fc25.x86_64.drpm
Because we don't mirror drpms (to save space) our fedora-25 nodes are
failing. So, just disable drpms for our mirrors.
Change-Id: I35b2f3f638f4551d5f169d1d6e081f29377ccb85
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Because we are using the 'zuul' user to SSH into our zuul workers, we
need to add it to configure_mirror.sh and still support the jenkins
user.
Change-Id: I79eff670f56d06c49a16e1330a73889e7b8482f1
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Now that we have migrated all our jobs to fedora-25, we can remove
fedora-24 from project-config.
Change-Id: I3a58c785c12f779060a6dca5cd40ddcc46429971
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Now that fedora-25 is online, we can start using the new AFS mirror for
fedora. Keep in mind, we didn't mirror fedora-24, since fedora-25 will
be replacing it.
Once fedora-24 has been removed from nodepool.yaml, we can remove our
conditional check.
Change-Id: I1bd510c6a568b71b604c27d6cae16d2654063d0c
Depends-On: I762e55fb7fe0c5ebfb7810545ddb4d5ccfedf2ff
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We incorrectly setup our mirrors for UCA. Now we support 3 openstack
releases.
Change-Id: Ie3a05398d9fdc7a7f7c0c74366a7c4dd70200a2b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This review cleanup scripts that were exclusively used by
snapshot scripts for nodepool which were removed in:
https://review.openstack.org/#/c/331494/
Change-Id: I37900a5d698a43bc0d510100289a5a7ec527021d
Update our script to provide additional debug information, such as the
builddate of our DIB images.
Change-Id: Ie399b1ac9cd6c6ea4372378ae7b7bf930fac16a3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
A cosmetic change to reduce the debug output on DNS failures. First
check for valid DNS, then proceed with configuration.
Change-Id: I3370a155bf6b49088c398dc9d4d572b6210b84b2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
After noticing ubuntu-xenial launch failures in osic-cloud1, it looks
like our unbound service is taking up to 1 minute to start properly.
So wait 30 seconds for host to timeout, and try 10 times. This gives
us a 300 second timeout window to configure DNS properly.
Change-Id: Id0432c91cc853fb4ecab43da991948c2e9d84b7d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>