.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode .. This template should be in ReSTructured text. The filename in the git repository should match the launchpad URL, for example a URL of https://blueprints.launchpad.net/kolla/+spec/awesome-thing should be named awesome-thing.rst . Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None For help with syntax, see http://www.sphinx-doc.org/en/stable/rest.html To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html ================== CentOS 8 Migration ================== https://blueprints.launchpad.net/kolla/+spec/centos-rhel-8 https://blueprints.launchpad.net/kolla-ansible/+spec/centos-rhel-8 CentOS 8 is here, and RDO support for CentOS 7 is going away. Problem description =================== This spec is largely focussed on the issue of migrating running systems from CentOS 7 to CentOS 8. We will not in general look at the details of porting Kolla to CentOS 8 here. Kolla images ------------ Train is the last release of OpenStack to support Python 2. Since CentOS 7 provides only limited support for Python 3, RDO plans to support only CentOS 8 for the Ussuri cycle. While RDO Train initially only provided Python 2 based RPMs for CentOS 7, it is expected that Python 3 based Train RPMs will be made available for CentOS 8. We should therefore support building Train images with either CentOS 7 or CentOS 8 as a base. Both sets of images should be published to Dockerhub. The image naming scheme will need to take account of the OS version, which it currently does not. Kolla Ansible ------------- There is no supported upgrade path between CentOS 7 and 8 hosts - a full reinstall is necessary. This provides us with one release (Train) that supports both CentOS 7 and 8, with which we could perform a rolling reinstallation of hosts. The cloud should continue to operate in this mixed 7/8 mode during the migration. The following diagram shows an example with 6 hosts h1 to h6 being migrated in batches of two:: h1 h2 h3 h4 h5 h6 ------------------- t0 | c7 c7 c7 c7 c7 c7 t1 | c8 c8 c7 c7 c7 c7 t2 | c8 c8 c8 c8 c7 c7 t3 | c8 c8 c8 c8 c8 c8 Proposed change =============== Kolla images - Ussuri --------------------- Support for building CentOS 8 based container images will be added to Kolla during the Ussuri cycle. Initially, this will exist in parallel with CentOS 7 support, and be controlled via the ``distro-package-manager`` configuration option, which takes its default value from the base image tag (``7.*`` for ``yum``, ``8.*`` for ``dnf``). New build and publishing CI jobs will be added for CentOS 8. We will use an alternative Docker image tag to tag CentOS 8 images on Dockerhub: ``master-centos8``. An example image name is ``kolla/centos-binary-base:master-centos8``. Once this work is complete, support for CentOS 7 will be removed. CentOS 8 images will then be tagged as ``master`` (or ``ussuri`` after branching). This two-phase approach allows for a clean backport to Train. A downside is that ``master-centos8`` tags will continue to exist in Dockerhub after the transition, which could be confusing for users. Kolla images - Train -------------------- The CentOS 8 support developed for Ussuri will be backported to the stable/train branch. Since these changes are significant, we should advertise this clearly to users who might not be expecting it on a stable branch. Train images will be built and published for CentOS 7 and 8, with CentOS 8 images using a tag of ``train-centos8``. Kolla Ansible - Ussuri ---------------------- Support for deploying CentOS 8 based container images will be added to Kolla Ansible during the Ussuri cycle. Initially, this will exist in parallel with CentOS 7 support, and be controlled via Ansible host facts about the host OS distribution. We will deploy CentOS 7 containers on CentOS 7 hosts, and CentOS 8 containers on CentOS 8 hosts to avoid kernel and userspace incompatibility issues. This also aligns with the `migration plan `__ proposed by Tripleo. During the migration, we may have a mix of CentOS 7 hosts running CentOS 7 containers, and CentOS 8 hosts running CentOS 8 containers. The image tag used will need to depend on the host OS. We have a hierarchy of image tag variables, for example: the ``nova-compute`` container uses a tag defined by ``nova_compute_tag``, which defaults to ``nova_tag``, which defaults to ``openstack_release``. To preserve the usage of ``openstack_release``, a new intermediate variable will be introduced. In this new scheme, the ``nova-compute`` container uses a tag defined by ``nova_compute_tag``, which defaults to ``nova_tag``, which defaults to ``openstack_tag``, which defaults to ``openstack_release ~ '-centos8'`` on CentOS 8, or ``openstack_release`` otherwise. Providing this suffix as a variable would help users who are using more fine-grained image tags (e.g. ``nova_compute_tag``). Once this work is complete, support for CentOS 7 will be removed. The intermediate variable ``openstack_tag`` will remain, but will always default to ``openstack_release``. Kolla Ansible - Train --------------------- The CentOS 8 support developed for Ussuri will be backported to the stable/train branch. Since these changes are significant, we should advertise this clearly to users who might not be expecting it on a stable branch. Kolla Ansible - Rolling Migration --------------------------------- In order to keep the cloud functional during the migration, the changes will be rolled through in batches. The size and scope of the batches are to be determined by the operator, but we should provide some guidance on selecting batches safely to avoid losing quorum of clustered services or reducing service availability below some minimum threshold. Provisioning of the new OS is out of the scope of this discussion, but could be handled by Kayobe or another tool. We should treat each batch as a removal of a set of hosts followed by an addition of new set of hosts. This is a good opportunity for us to tidy up these procedures and fill in gaps where necessary. To remove a batch of hosts: #. Migrate running OpenStack resources from the hosts to other hosts which are not in the batch. This includes compute instances (VMs), storage (Swift disks), network services (DHCP & L3 agents etc.). Ideally this would be automated, but should at least be discussed in documentation #. Cleanly disable (if necessary) and shutdown running OpenStack services #. Shutdown hosts The hosts should then be provisioned with CentOS 8 - as mentioned this is out of scope. Then to add a batch of hosts: #. Provision CentOS 8 to hosts (out of scope) #. Bootstrap hosts #. Deploy services #. Validate new hosts We may wish to adopt existing data on these hosts, e.g. for Swift a full sync from empty for each node would take a long time. For systems where Docker volumes are stored on a separate disk or partition, it may be possible to preserve these. There may be some corner cases where this does not work well, and we should check for these and advise in documentation. For the majority of these tasks it should be possible to use the ``--limit`` argument to speed up the process, but this should be investigated and verified. Stable upgrades --------------- Where possible, we should aim to use the same versions of packages on CentOS 7 and 8 for Train. Upgrades should wait until Ussuri. In some cases this may not be possible, and we should make it clear in release notes where services will be upgraded in Train. Given that a rolled migration may take a significant amount of time to complete, we should consider that services may be partially upgraded. In cases where this may cause a problem, it should be called out in the documentation, with any available mitigation (e.g. disable during migration). Feature removal --------------- Due to differences between CentOS 7 and 8, it may not be possible or practical to support all existing features. We should bear in mind that this will apply to the Train release, and continue to support these features on CentOS 7. Ceph support ^^^^^^^^^^^^ Migrating a Ceph cluster deployed by Kolla Ansible to CentOS 8 would represent a significant challenge. Ceph deployment support has been deprecated since the Stein release, and this is a good point to remove that support. SCSI target daemon ^^^^^^^^^^^^^^^^^^ For CentOS 7, the SCSI target daemon (``tgtd``) is provided by EPEL. For CentOS 8, this is no longer available, probably due to a preference for the kernel SCSI target, `LIO `__. Due to the requirement to reinstall, this migration provides a good opportunity to stop supporting ``tgtd`` for CentOS. This will affect some Cinder volume drivers, including the LVM backend. ``tgtd`` will still be supported on Debian/Ubuntu. Note that no changes are required for the iSCSI initiator. Security impact --------------- There should be no security impact due to this change. Performance Impact ------------------ There should be no performance impact due to this change. Alternatives ------------ CentOS 7 to 8 upgrade ^^^^^^^^^^^^^^^^^^^^^ There are various documented procedures for upgrading from CentOS 7 to 8 without a reinstall. These typically come with various disclaimers, and are not recommended for production environments. Image naming ^^^^^^^^^^^^ Rather than using a different tag to mark CentOS 8 images, it would be possible to use a base distro of ``centos8``. e.g. ``kolla/centos8-binary-base:master``. The reason for not doing this is that there is a reasonable amount of logic both in Kolla and Kolla Ansible that uses the ``base_distro`` which would need to be updated. Don't publish master-centos8 tag ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ During the transition period, we could manage with always building images for CentOS 8 on master, thus avoiding the need for a temporary ``master-centos8`` tag. Implementation ============== Assignee(s) ----------- Primary assignees: Marcin Juszkiewicz (hrw) Mark Goddard (mgoddard) Michal Nasiadka (mnasiadka) Radoslaw Piliszek (yoctozepto) Work Items ---------- Kolla ^^^^^ * Support building CentOS 8 images in Ussuri * Build and publish both CentOS 7 and 8 images in CI * Backport CentOS 8 support to Train * Remove CentOS 7 image support in Ussuri * Update documentation (image tags etc.) Kolla Ansible ^^^^^^^^^^^^^ * Support deploying CentOS 8 images in Ussuri * Test (separately) deployment to both CentOS 7 and 8 in CI * Backport CentOS 8 support to Train * Test migration from CentOS 7 to 8 on Train in CI * Test upgrade from Train to Ussuri on CentOS 8 * Remove CentOS 7 support in Ussuri * Update documentation (migration) Testing ======= This will be tested by parallel build and deploy jobs for CentOS 7 and 8, in addition to a job testing migration from CentOS 7 to 8. Significant manual testing will be required to explore edge cases associated with the migration. Documentation Impact ==================== * Building CentOS 8 images * Deploying to CentOS 8 * Image tags for CentOS 8 * Migration procedure covered in detail References ========== * `CentOS 8 release notes `__