Spec to retire static.openstack.org
Change-Id: Ic5557750ee6c52def01c8d362b8d9e7563cc0f8a
This commit is contained in:
Ian Wienand
parent
73f5500b39
commit
1533f03a30
@ -44,6 +44,7 @@ permits.
|
|||||||
specs/translation_check_site
|
specs/translation_check_site
|
||||||
specs/wiki_modernization
|
specs/wiki_modernization
|
||||||
specs/letsencrypt
|
specs/letsencrypt
|
||||||
|
specs/retire-static
|
||||||
|
|
||||||
Help Wanted
|
Help Wanted
|
||||||
===========
|
===========
|
||||||
|
|||||||
378
specs/retire-static.rst
Normal file
378
specs/retire-static.rst
Normal file
@ -0,0 +1,378 @@
|
|||||||
|
::
|
||||||
|
|
||||||
|
Copyright 2019 Red Hat Inc.
|
||||||
|
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0
|
||||||
|
Unported License.
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
..
|
||||||
|
This template should be in ReSTructured text. Please do not delete
|
||||||
|
any of the sections in this template. If you have nothing to say
|
||||||
|
for a whole section, just write: "None". For help with syntax, see
|
||||||
|
http://sphinx-doc.org/rest.html To test out your formatting, see
|
||||||
|
http://www.tele3.cz/jbar/rest/rest.html
|
||||||
|
|
||||||
|
===========================
|
||||||
|
Retire static.openstack.org
|
||||||
|
===========================
|
||||||
|
|
||||||
|
Include the URL of your StoryBoard story:
|
||||||
|
|
||||||
|
https://storyboard.openstack.org/#!/story/2006598
|
||||||
|
|
||||||
|
Move the services provided by ``static.openstack.org`` into less
|
||||||
|
centralised approaches more consistent with modern deployment trends.
|
||||||
|
|
||||||
|
Problem Description
|
||||||
|
===================
|
||||||
|
|
||||||
|
The ``static.openstack.org`` host is a monolithic server providing
|
||||||
|
various hosting services via a large amount of volume-attached
|
||||||
|
storage.
|
||||||
|
|
||||||
|
The immediate problem is it currently running Ubuntu Trusty which is
|
||||||
|
reaching the end of its supported life.
|
||||||
|
|
||||||
|
The secondary problems are twofold:
|
||||||
|
|
||||||
|
Firstly, we would like to move the various publishing and hosting
|
||||||
|
operations from centralised volumes on a single server to our AFS
|
||||||
|
distributed file-system.
|
||||||
|
|
||||||
|
Secondly, we would like to make the hosting portion more OpenDev
|
||||||
|
compatible; this means avoiding working on legacy deployment methods
|
||||||
|
(i.e. puppet) and integrating with our general idea of a "whitebox"
|
||||||
|
service that can be used by many different projects.
|
||||||
|
|
||||||
|
Thus we propose breaking up the services it offers to utlise more
|
||||||
|
modern infrastructure alternatives and retiring the host.
|
||||||
|
|
||||||
|
Proposed Change
|
||||||
|
===============
|
||||||
|
|
||||||
|
We can break the services down
|
||||||
|
|
||||||
|
Log storage
|
||||||
|
Legacy log storage (~14tb)
|
||||||
|
|
||||||
|
Redirects
|
||||||
|
Apache service redirects a number of legacy URLs to new locations
|
||||||
|
|
||||||
|
Static site serving
|
||||||
|
100gb attached partition holding various static sites (i.e. plain
|
||||||
|
HTML publishing, no middleware, etc)
|
||||||
|
|
||||||
|
Tarball
|
||||||
|
512gb partition which holds and publishes release tarballs for all
|
||||||
|
projects.
|
||||||
|
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
``apt-get dist-ugprade`` the host to a more recent distribution, fix
|
||||||
|
any puppet issues and ignore it until next time it needs updating.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
TBD
|
||||||
|
|
||||||
|
Gerrit Topic
|
||||||
|
------------
|
||||||
|
|
||||||
|
Use Gerrit topic "static-services" all patches related to this spec.
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
git-review -t static-services
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
Log storage
|
||||||
|
~~~~~~~~~~~
|
||||||
|
|
||||||
|
OpenDev CI logs have been moved to various object-storage backends
|
||||||
|
provided by donors. The existing logs will age out per our existing
|
||||||
|
old-log cleanup jobs.
|
||||||
|
|
||||||
|
Since logs were always ephemeral there should be no issues with old
|
||||||
|
links. For clarity we will remove (rather than redirect) the
|
||||||
|
``logs.openstack.org`` DNS entry so there is no confusion that logs
|
||||||
|
might still live there.
|
||||||
|
|
||||||
|
Workitems:
|
||||||
|
|
||||||
|
* remove ``logs.openstack.org`` DNS entries after old logs entries
|
||||||
|
have cleared out
|
||||||
|
|
||||||
|
Legacy redirects
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The following do straight redirects from their config hostnames to
|
||||||
|
``docs.openstack.org``
|
||||||
|
|
||||||
|
* 50-cinder.openstack.org.conf
|
||||||
|
* 50-devstack.org.conf
|
||||||
|
* 50-glance.openstack.org.conf
|
||||||
|
* 50-horizon.openstack.org.conf
|
||||||
|
* 50-keystone.openstack.org.conf
|
||||||
|
* 50-nova.openstack.org.conf
|
||||||
|
* 50-swift.openstack.org.conf
|
||||||
|
|
||||||
|
The following have slightly different semantics
|
||||||
|
|
||||||
|
* 50-ci.openstack.org.conf
|
||||||
|
|
||||||
|
* ``/nodepool``, ``/shade``, ``/zuul``, etc all to docs; see
|
||||||
|
https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/ci.vhost.erb
|
||||||
|
|
||||||
|
* 50-qa.openstack.org.conf
|
||||||
|
|
||||||
|
* currently redirects to broken link
|
||||||
|
https://docs.openstack.org/developer/qa
|
||||||
|
|
||||||
|
The following redirects to ``openstack.org``
|
||||||
|
|
||||||
|
* 50-summit.openstack.org.conf
|
||||||
|
|
||||||
|
Clearly there is a need for a generic ability to redirect various URLs
|
||||||
|
as things change over time.
|
||||||
|
|
||||||
|
We will use a single containerised ``haproxy`` instance to handle
|
||||||
|
redirects for the OpenDev project. Although initially it will simply
|
||||||
|
be handling 302 redirects, it is imagined that future services can use
|
||||||
|
it for it's availability or load-balancing services as well. Note
|
||||||
|
that ``gitea`` services also have their own load-balancer; although it
|
||||||
|
reuses all the deployment mechanisms, the production service is kept
|
||||||
|
separately to maintain isolation been probably the most important
|
||||||
|
service (code) and more informational services.
|
||||||
|
|
||||||
|
Proof-of-concept reviews are provided at:
|
||||||
|
|
||||||
|
* https://review.opendev.org/677903 : make haproxy role more generic
|
||||||
|
* https://review.opendev.org/678159 : add a service load balancer
|
||||||
|
|
||||||
|
The work items consist of:
|
||||||
|
|
||||||
|
* approval of the above reviews
|
||||||
|
* starting the production host
|
||||||
|
* iterating the extant DNS records and pointing them to the new
|
||||||
|
load-balancer
|
||||||
|
|
||||||
|
OpenDev infrastructure migration
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
We wish to provide new services only using our latest deployment
|
||||||
|
methods, to avoid introducing even more legacy services and to provide
|
||||||
|
a basis for the migration process to OpenDev services.
|
||||||
|
|
||||||
|
Although ``files02.openstack.org`` has an existing role as a webserver
|
||||||
|
serving content from the ``/openstack.org`` AFS mount, it is
|
||||||
|
configured using legacy puppet. Thus a new server will be provisioned
|
||||||
|
using our Ansible environment, rather than adding more hosts to legacy
|
||||||
|
configuration.
|
||||||
|
|
||||||
|
This server should be a "whitebox" server that is capable of serving a
|
||||||
|
range of domains that OpenDev would like to serve. However, it's role
|
||||||
|
will only to be to serve static directories on AFS volumes. After
|
||||||
|
this process, there will be numerous examples of SSL certificate
|
||||||
|
generation, vhost configuration, AFS volume setup and publishing jobs
|
||||||
|
for any other projects to copy and implement.
|
||||||
|
|
||||||
|
Initially this server needs to serve https sites for the replacement
|
||||||
|
services; namely
|
||||||
|
|
||||||
|
* governance.openstack.org
|
||||||
|
* specs.openstack.org
|
||||||
|
* security.openstack.org
|
||||||
|
* service-types.openstack.org
|
||||||
|
* releases.openstack.org
|
||||||
|
* tarballs.openstack.org
|
||||||
|
|
||||||
|
Currently, SSL certificates are manually provisioned and entered into
|
||||||
|
puppet secret data, where they are deployed to the host. We wish to
|
||||||
|
use automatically renewing letsencrypt certificates per our other
|
||||||
|
infrastructure, utilising our DNS based authentication. However,
|
||||||
|
since ``openstack.org`` remains administered by external teams in
|
||||||
|
RAX's propietary environment, we will make an exception and setup DNS
|
||||||
|
validation records manually for these legacy sites until a full
|
||||||
|
migration of ``openstack.org`` to OpenDev infrastructure is possible.
|
||||||
|
Other domains will use OpenDev nameservers, which support automated
|
||||||
|
DNS validation renewals.
|
||||||
|
|
||||||
|
We will have the new server provisioned and ready before we begin the
|
||||||
|
steps of migrating publishing locations. This means we can debug any
|
||||||
|
setup issues outside production, and effects a zero-downtime cutover
|
||||||
|
when the sites are ready.
|
||||||
|
|
||||||
|
Workitems are as follows:
|
||||||
|
|
||||||
|
* Write roles and tests to provision a new ``static01.opendev.org``
|
||||||
|
server which will be limited to running Apache and serving AFS
|
||||||
|
directories.
|
||||||
|
* Create the server
|
||||||
|
* Create CNAME ``static.opendev.org`` which will be the main service
|
||||||
|
hostname, to provide for easier server replacement or other updates
|
||||||
|
in the future.
|
||||||
|
* Pre-provision https certificates for the above listed services
|
||||||
|
|
||||||
|
* Using the RAX web interface for name services and the openstack
|
||||||
|
infra permissions, setup
|
||||||
|
``_acme-challenge.<service>.openstack.org`` records as a CNAME to
|
||||||
|
``acme.opendev.org``.
|
||||||
|
* Each site should have a separate certificate provisioned. The
|
||||||
|
configuration would be something like
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
letsencrypt_certs:
|
||||||
|
governance-openstack-org:
|
||||||
|
- governance.openstack.org
|
||||||
|
specs.openstack.org:
|
||||||
|
- specs.openstack.org
|
||||||
|
and.so.on.
|
||||||
|
* Debug any failures; however the theory is (taking one example):
|
||||||
|
the existing letsencrypt roles should request a certificate for
|
||||||
|
``governance.openstack.org`` on ``static01.opendev.org`` and
|
||||||
|
receive the authentication key, which is placed in a TXT record in
|
||||||
|
``acme.opendev.org``. The certificate creation will will trigger
|
||||||
|
a lookup of ``_acme-challenge.governance.openstack.org`` which
|
||||||
|
will be a CNAME to ``acme.opendev.org``, which contains the
|
||||||
|
correct TXT record. The certificate is issued on
|
||||||
|
``static01.opendev.org``.
|
||||||
|
|
||||||
|
* Preconfigure the vhost configuration for the above sites (using
|
||||||
|
prior provisioned keys for SSL)
|
||||||
|
* Confirm correct operation of the sites with dummy content.
|
||||||
|
|
||||||
|
Static hosting
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
A number of jobs publish directly to ``/srv/static`` on the server.
|
||||||
|
These are then served by Apache as static websites.
|
||||||
|
|
||||||
|
In general, we want these jobs to publish to our AFS volumes. By
|
||||||
|
publishing to AFS we remove the central point of failure of a single
|
||||||
|
server and it's attached disks (mitigated by multiple AFS servers and
|
||||||
|
replicas).
|
||||||
|
|
||||||
|
The AFS volumes are then served by ``static01.opendev.org`` which has
|
||||||
|
a dedicated role as an AFS to HTTP bridge.
|
||||||
|
|
||||||
|
The sites in question are:
|
||||||
|
|
||||||
|
* 50-governance.openstack.org.conf
|
||||||
|
* https://governance.openstack.org
|
||||||
|
* main source -> https://opendev.org/openstack/governance-website
|
||||||
|
* published via https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L2298
|
||||||
|
* aliases ``/srv/static/<election|sigs|tc|uc>``
|
||||||
|
|
||||||
|
* 50-security.openstack.org.conf
|
||||||
|
* https://security.openstack.org
|
||||||
|
* single repo source -> https://opendev.org/openstack/ossa
|
||||||
|
* deployed by publish-security job -> https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L739
|
||||||
|
|
||||||
|
* 50-service-types.openstack.org.conf
|
||||||
|
* https://service-types.openstack.org
|
||||||
|
* single repo -> https://opendev.org/openstack/service-types-authority
|
||||||
|
* https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L551
|
||||||
|
|
||||||
|
* 50-specs.openstack.org.conf
|
||||||
|
* https://specs.openstack.org
|
||||||
|
* various spec repos; published by ``openstack-spec-jobs`` to subdirectories
|
||||||
|
|
||||||
|
* 50-releases.openstack.org.conf
|
||||||
|
* https://releases.openstack.org
|
||||||
|
* generated by -> https://opendev.org/openstack/releases/
|
||||||
|
* note generates .htaccess with contsraints links, used widely in pip
|
||||||
|
* publish-tox-jobs-static : https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L685
|
||||||
|
|
||||||
|
* 50-tarballs.openstack.org.conf
|
||||||
|
* https://tarballs.openstack.org
|
||||||
|
* every project's release jobs
|
||||||
|
|
||||||
|
The extant AFS layout has volumes for each project. Thus we will
|
||||||
|
continue this theme and an admin will create one volume for each of
|
||||||
|
the above static sites; e.g.
|
||||||
|
|
||||||
|
* /afs/openstack.org/project/governance.openstack.org (~200mb)
|
||||||
|
* /afs/openstack.org/project/security.openstack.org (100mb)
|
||||||
|
* /afs/openstack.org/project/service-types.openstack.org (520k)
|
||||||
|
* /afs/openstack.org/project/specs.openstack.org (current 706mb)
|
||||||
|
* /afs/openstack.org/project/releases.openstack.org (current 57mb)
|
||||||
|
* /afs/openstack.org/project/tarballs.openstack.org (current 134gb)
|
||||||
|
|
||||||
|
The work items are as follows
|
||||||
|
|
||||||
|
* Create the volumes for each site as described above
|
||||||
|
* Migrate the extant data to the new volumes. It is impractical to
|
||||||
|
recreate all the sites as it would require triggering many often
|
||||||
|
infrequently updated repos.
|
||||||
|
* Publishing jobs will be updated to use AFS publishing to these new
|
||||||
|
locations. During transition period, we can publish to both
|
||||||
|
locations.
|
||||||
|
* Update the site configuration on ``static01.opendev.org`` to serve
|
||||||
|
the site from the new location
|
||||||
|
* We should be able to fully test the new sites at this point with
|
||||||
|
manual host entries. Ensure:
|
||||||
|
* https certificates working correctly
|
||||||
|
* old links remain consistent
|
||||||
|
|
||||||
|
* For each site, move to production by updating the CNAME entries in
|
||||||
|
the ``openstack.org`` domain for the main server to point to
|
||||||
|
``static.opendev.org`` (note, not the server directly,
|
||||||
|
i.e. ``static01.opendev.org``, to give us flexibility in managing
|
||||||
|
the backend service with server replacements or load-balancing in
|
||||||
|
the future). Per prior testing, this should be transparent.
|
||||||
|
* Old publishing jobs removed
|
||||||
|
|
||||||
|
Repositories
|
||||||
|
------------
|
||||||
|
|
||||||
|
Unlikley to require new repositories
|
||||||
|
|
||||||
|
Servers
|
||||||
|
-------
|
||||||
|
|
||||||
|
* a new http server for serving AFS content
|
||||||
|
* A load-balancer server is suggested to host the haproxy container
|
||||||
|
|
||||||
|
DNS Entries
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Quite a few DNS entries will need to be updated as described
|
||||||
|
|
||||||
|
Documentation
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Developers should largely not care where the results are published.
|
||||||
|
|
||||||
|
Small doc updates for any new services.
|
||||||
|
|
||||||
|
A guide to setting up jobs, host configuration, etc. for publishing
|
||||||
|
static data for other projects may be useful.
|
||||||
|
|
||||||
|
Security
|
||||||
|
--------
|
||||||
|
|
||||||
|
N/A
|
||||||
|
|
||||||
|
Testing
|
||||||
|
-------
|
||||||
|
|
||||||
|
Since all updates are replacements, we can confirm that the new sites
|
||||||
|
are operational before putting them into production. Any DNS switches
|
||||||
|
can be essentially zero impact.
|
||||||
|
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
N/A at this time
|
||||||
Loading…
Reference in New Issue
Block a user