From 780a4c4eadef04fb902541fdff47be69ae6424b6 Mon Sep 17 00:00:00 2001 From: Dmitrii Shcherbakov Date: Mon, 25 May 2020 21:51:06 +0000 Subject: [PATCH] Use focal/core20/Ussuri/OVN & enable confinement Major changes: * Plumbing necessary for strict confinement with the microstack-support interface https://github.com/snapcore/snapd/pull/8926 * Until the interface is merged, devmode will be used and kernel modules will be loaded via an auxiliary service. * upgraded OpenStack components to Focal (20.04) and OpenStack Ussuri; * reworked the old patches; * added the Placement service since it is now separate; * addressed various build issues due to changes in snapcraft and built dependencies: * e.g. libvirt requires the build directory to be separate from the source directory) and LP: #1882255; * LP: #1882535 and https://github.com/pypa/pip/issues/8414 * LP: #1882839 * LP: #1885294 * https://storyboard.openstack.org/#!/story/2007806 * LP: #1864589 * LP: #1777121 * LP: #1881590 * ML2/OVS replated with ML2/OVN; * dnsmasq is not used anymore; * neutron l3 and DHCP agents are not used anymore; * Linux network namespaces are only used for neutron-ovn-metadata-agent. * ML2 DNS support is done via native OVN mechanisms; * OVN-related database services (southbound and northbound dbs); * OVN-related control plane services (ovn-controller, ovn-northd); * core20 base support (bionic hosts are supported); * the removal procedure now relies on the "remove" hook since `snap remove` cannot be used from the confined environment anymore; * prerequisites to enabling AppArmor confinement for QEMU processes created by the confined libvirtd. * Added the Spice html5 console proxy service to enable clients to retrieve and use it via `microstack.openstack console url show --spice `. * Added missing Cinder templates and DB migrations for the Cinder DB. * Added experimental support for a loop device-based LVM backend for Cinder. Due to LP: #1892895 this is not recommended to be used in production except for tempest testing with an applied workaround; * includes iscsid and iscsi-tcp kernel module loading; * includes LIO and loading of relevant kernel modules; * An LVM PV is created on top of a loop device with a backing file present in $SNAP_COMMON/cinder-lvm.img; * A VG is created on top of the PV; * LVs are created by Cinder and exported via LIO over iscsi to iscsid which hot-plugs new SCSI devices. Those SCSI devices are then propagated by Nova to libvirt and QEMU during volume attachment; * Added post-deployment testing via rally and tempest (via the microstack-test snap). A set of tests included into Refstack 2018.02 is executed (except for object storage tests due to the lack of object storage support). Change-Id: Ic70770095860a57d5e0a55a8a9451f9db6be7448 --- ...patch => 0001-Use-SNAP_COMMON-paths.patch} | 36 +- patches/neutron/dnsmasq | 4 - .../0001-drop-setuid-setgid-initgroups.patch | 77 ++ patches/nginx/drop-nginx-setgroups.patch | 57 - ...nap-specific-abstract-socket-address.patch | 56 + snap-overlay/bin/iscsid-start | 19 + snap-overlay/bin/load-modules | 5 + snap-overlay/bin/set-default-config | 17 +- snap-overlay/bin/setup-br-ex | 21 +- snap-overlay/bin/target-start | 4 + snap-overlay/bin/target-stop | 5 + snap-overlay/etc/neutron/dhcp_agent.ini | 5 - snap-overlay/etc/neutron/l3_agent.ini | 2 - snap-overlay/etc/neutron/metadata_agent.ini | 3 - snap-overlay/etc/neutron/neutron.conf | 8 +- .../etc/neutron/plugins/ml2/ml2_conf.ini | 19 +- .../neutron/plugins/ml2/openvswitch_agent.ini | 11 - snap-overlay/etc/placement/placement.conf | 3 + .../themes/ubuntu/_styles.scss | 3 +- .../themes/ubuntu/_variables.scss | 0 .../themes/ubuntu/bootstrap/_styles.scss | 0 .../themes/ubuntu/bootstrap/_variables.scss | 0 .../ubuntu/bootstrap/components/_navbar.scss | 0 .../ubuntu/bootstrap/components/_navs.scss | 0 .../themes/ubuntu/horizon/_styles.scss | 0 .../themes/ubuntu/horizon/_variables.scss | 0 .../components/_context_selection.scss | 0 .../horizon/components/_pie_charts.scss | 0 .../ubuntu/horizon/components/_sidebar.scss | 0 .../ubuntu/img/image-background-pattern.png | Bin 92706 -> 92702 bytes .../themes/ubuntu/img/logo-ubuntu-orange.svg | 0 .../themes/ubuntu/img/logo-ubuntu-white.svg | 0 snap-overlay/snap-openstack.yaml | 106 +- .../templates/cinder-rootwrap.conf.j2 | 6 + snap-overlay/templates/cinder-snap.conf.j2 | 17 + .../templates/cinder.database.conf.j2 | 2 + .../templates/cinder.keystone.conf.j2 | 13 + .../templates/cinder.rabbitmq.conf.j2 | 2 + snap-overlay/templates/iscsid.conf.j2 | 22 + snap-overlay/templates/microstack.json.j2 | 26 + snap-overlay/templates/neutron-snap.conf.j2 | 12 + .../neutron_ovn_metadata_agent.ini.j2 | 13 + snap-overlay/templates/nginx.conf.j2 | 2 +- snap-overlay/templates/nova-snap.conf.j2 | 14 + .../templates/nova.conf.d.console.conf.j2 | 18 + .../templates/nova.conf.d.placement.conf.j2 | 11 + snap-overlay/templates/nrpe.cfg.j2 | 8 +- ...cement-api.ini.j2 => placement-api.ini.j2} | 3 +- .../templates/placement-nginx.conf.j2 | 10 + snap-overlay/templates/placement-snap.conf.j2 | 9 + .../placement.conf.d.database.conf.j2 | 2 + ...f.j2 => placement.conf.d.keystone.conf.j2} | 16 +- .../placement.conf.d.placement-api.conf.j2 | 2 + snap-overlay/templates/targets.conf.j2 | 2 + snap-wrappers/ovn/ovn-wrapper | 25 + snap-wrappers/ovs/ovs-wrapper | 16 +- snap-wrappers/rabbitmq/erl | 2 +- .../rabbitmq/rabbitmq-script-wrapper | 4 + snap/hooks/install | 65 ++ snap/hooks/post-refresh | 6 + snap/hooks/remove | 79 ++ snapcraft.yaml | 1012 +++++++++++++---- tests/basic-test.sh | 31 +- tests/framework.py | 32 +- tests/test_basic.py | 39 +- tests/test_cluster.py | 2 +- tests/test_control.py | 2 +- tools/cluster/cluster/client.py | 2 + tools/init/init/main.py | 38 +- tools/init/init/questions/__init__.py | 337 ++++-- tools/init/init/questions/uninstall.py | 7 +- tools/init/init/shell.py | 20 + tools/init/requirements.txt | 5 +- tools/init/setup.py | 1 - tools/init/tests/test_question.py | 2 +- tools/lxd_build.sh | 8 +- tools/make-a-microstack.sh | 44 +- tools/setup-lvm-loopdev/setup-lvm-loopdev | 28 + 78 files changed, 1932 insertions(+), 546 deletions(-) rename patches/horizon/{0001-Added-SNAP_COMMON-pathing.patch => 0001-Use-SNAP_COMMON-paths.patch} (58%) delete mode 100644 patches/neutron/dnsmasq create mode 100644 patches/nginx/0001-drop-setuid-setgid-initgroups.patch delete mode 100644 patches/nginx/drop-nginx-setgroups.patch create mode 100644 patches/open-iscsi/0001-Use-a-snap-specific-abstract-socket-address.patch create mode 100755 snap-overlay/bin/iscsid-start create mode 100755 snap-overlay/bin/load-modules create mode 100755 snap-overlay/bin/target-start create mode 100755 snap-overlay/bin/target-stop delete mode 100644 snap-overlay/etc/neutron/dhcp_agent.ini delete mode 100644 snap-overlay/etc/neutron/l3_agent.ini delete mode 100644 snap-overlay/etc/neutron/metadata_agent.ini delete mode 100644 snap-overlay/etc/neutron/plugins/ml2/openvswitch_agent.ini create mode 100644 snap-overlay/etc/placement/placement.conf rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/_styles.scss (96%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/_variables.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_styles.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_variables.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navbar.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navs.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/horizon/_styles.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/horizon/_variables.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_context_selection.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_pie_charts.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_sidebar.scss (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/img/image-background-pattern.png (98%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-orange.svg (100%) rename snap-overlay/lib/{python3.6 => python3.8}/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-white.svg (100%) create mode 100644 snap-overlay/templates/cinder-rootwrap.conf.j2 create mode 100644 snap-overlay/templates/cinder.database.conf.j2 create mode 100644 snap-overlay/templates/cinder.keystone.conf.j2 create mode 100644 snap-overlay/templates/cinder.rabbitmq.conf.j2 create mode 100644 snap-overlay/templates/iscsid.conf.j2 create mode 100644 snap-overlay/templates/microstack.json.j2 create mode 100644 snap-overlay/templates/neutron_ovn_metadata_agent.ini.j2 create mode 100644 snap-overlay/templates/nova.conf.d.console.conf.j2 create mode 100644 snap-overlay/templates/nova.conf.d.placement.conf.j2 rename snap-overlay/templates/{nova-placement-api.ini.j2 => placement-api.ini.j2} (75%) create mode 100644 snap-overlay/templates/placement-nginx.conf.j2 create mode 100644 snap-overlay/templates/placement-snap.conf.j2 create mode 100644 snap-overlay/templates/placement.conf.d.database.conf.j2 rename snap-overlay/templates/{nova.conf.d.nova-placement.conf.j2 => placement.conf.d.keystone.conf.j2} (58%) create mode 100644 snap-overlay/templates/placement.conf.d.placement-api.conf.j2 create mode 100644 snap-overlay/templates/targets.conf.j2 create mode 100755 snap-wrappers/ovn/ovn-wrapper create mode 100755 snap/hooks/remove mode change 100644 => 100755 tools/cluster/cluster/client.py create mode 100755 tools/setup-lvm-loopdev/setup-lvm-loopdev diff --git a/patches/horizon/0001-Added-SNAP_COMMON-pathing.patch b/patches/horizon/0001-Use-SNAP_COMMON-paths.patch similarity index 58% rename from patches/horizon/0001-Added-SNAP_COMMON-pathing.patch rename to patches/horizon/0001-Use-SNAP_COMMON-paths.patch index a398646..ba41854 100644 --- a/patches/horizon/0001-Added-SNAP_COMMON-pathing.patch +++ b/patches/horizon/0001-Use-SNAP_COMMON-paths.patch @@ -1,18 +1,18 @@ -From 4d90b94a0a4ce3e7e69507c2c25a6981336c66a1 Mon Sep 17 00:00:00 2001 -From: Pete Vander Giessen -Date: Thu, 19 Sep 2019 13:18:50 +0000 -Subject: [PATCH] Added SNAP_COMMON pathing +From 36c45710c8cc3bbdf86fe2513a07a0d5f0a5c3f9 Mon Sep 17 00:00:00 2001 +From: Dmitrii Shcherbakov +Date: Mon, 8 Jun 2020 13:56:20 +0000 +Subject: [PATCH] Use SNAP_COMMON paths --- - lib/python3.6/site-packages/openstack_dashboard/local/local_settings.py | 4 +++- + openstack_dashboard/local/local_settings.py | 4 +++- openstack_dashboard/settings.py | 6 ++++-- 2 files changed, 7 insertions(+), 3 deletions(-) -diff --git a/lib/python3.6/site-packages/openstack_dashboard/local/local_settings.py b/lib/python3.6/site-packages/openstack_dashboard/local/local_settings.py -index 5f1ab10cc..cef4e9485 100644 ---- a/lib/python3.6/site-packages/openstack_dashboard/local/local_settings.py -+++ b/lib/python3.6/site-packages/openstack_dashboard/local/local_settings.py -@@ -10,6 +10,8 @@ from openstack_dashboard.settings import HORIZON_CONFIG +diff --git a/lib/python3.8/site-packages/openstack_dashboard/local/local_settings.py b/lib/python3.8/site-packages/openstack_dashboard/local/local_settings.py +index 2b084bf24..aad403c04 100644 +--- a/lib/python3.8/site-packages/openstack_dashboard/local/local_settings.py ++++ b/lib/python3.8/site-packages/openstack_dashboard/local/local_settings.py +@@ -22,6 +22,8 @@ from openstack_dashboard.settings import HORIZON_CONFIG DEBUG = True @@ -21,7 +21,7 @@ index 5f1ab10cc..cef4e9485 100644 # This setting controls whether or not compression is enabled. Disabling # compression makes Horizon considerably slower, but makes it much easier # to debug JS and CSS changes -@@ -62,7 +64,7 @@ DEBUG = True +@@ -74,7 +76,7 @@ DEBUG = True # including on the login form. #HORIZON_CONFIG["disable_password_reveal"] = False @@ -30,11 +30,11 @@ index 5f1ab10cc..cef4e9485 100644 # Set custom secret key: # You can either set it to a specific value or you can let horizon generate a -diff --git a/openstack_dashboard/settings.py b/openstack_dashboard/settings.py -index 02cd17ef3..69380f460 100644 ---- a/lib/python3.6/site-packages/openstack_dashboard/settings.py -+++ b/lib/python3.6/site-packages/openstack_dashboard/settings.py -@@ -55,6 +55,8 @@ if ROOT_PATH not in sys.path: +diff --git a/lib/python3.8/site-packages/openstack_dashboard/settings.py b/lib/python3.8/site-packages/openstack_dashboard/settings.py +index 81b8e45e1..5909bc8a8 100644 +--- a/lib/python3.8/site-packages/openstack_dashboard/settings.py ++++ b/lib/python3.8/site-packages/openstack_dashboard/settings.py +@@ -50,6 +50,8 @@ if ROOT_PATH not in sys.path: DEBUG = False @@ -43,7 +43,7 @@ index 02cd17ef3..69380f460 100644 ROOT_URLCONF = 'openstack_dashboard.urls' HORIZON_CONFIG = { -@@ -216,7 +218,7 @@ USE_TZ = True +@@ -211,7 +213,7 @@ USE_TZ = True DEFAULT_EXCEPTION_REPORTER_FILTER = 'horizon.exceptions.HorizonReporterFilter' SECRET_KEY = None @@ -52,7 +52,7 @@ index 02cd17ef3..69380f460 100644 ADD_INSTALLED_APPS = [] -@@ -265,7 +267,7 @@ else: +@@ -260,7 +262,7 @@ else: ) # allow to drop settings snippets into a local_settings_dir diff --git a/patches/neutron/dnsmasq b/patches/neutron/dnsmasq deleted file mode 100644 index 9923e9a..0000000 --- a/patches/neutron/dnsmasq +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/bash -# Wrapper for dnsmasq - -exec $SNAP/usr/sbin/dnsmasq-orig -u snap_daemon -g snap_daemon $@ diff --git a/patches/nginx/0001-drop-setuid-setgid-initgroups.patch b/patches/nginx/0001-drop-setuid-setgid-initgroups.patch new file mode 100644 index 0000000..131b8e2 --- /dev/null +++ b/patches/nginx/0001-drop-setuid-setgid-initgroups.patch @@ -0,0 +1,77 @@ +From a8df30a8a837c223945a13fe4cd9418084d8ed21 Mon Sep 17 00:00:00 2001 +From: Dmitrii Shcherbakov +Date: Wed, 10 Jun 2020 20:14:32 +0000 +Subject: [PATCH] drop setuid/setgid/initgroups + +--- + src/os/unix/ngx_process_cycle.c | 54 --------------------------------- + 1 file changed, 54 deletions(-) + +diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c +index 5817a2c2..305c6823 100644 +--- a/src/os/unix/ngx_process_cycle.c ++++ b/src/os/unix/ngx_process_cycle.c +@@ -825,60 +825,6 @@ ngx_worker_process_init(ngx_cycle_t *cycle, ngx_int_t worker) + } + } + +- if (geteuid() == 0) { +- if (setgid(ccf->group) == -1) { +- ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, +- "setgid(%d) failed", ccf->group); +- /* fatal */ +- exit(2); +- } +- +- if (initgroups(ccf->username, ccf->group) == -1) { +- ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, +- "initgroups(%s, %d) failed", +- ccf->username, ccf->group); +- } +- +-#if (NGX_HAVE_PR_SET_KEEPCAPS && NGX_HAVE_CAPABILITIES) +- if (ccf->transparent && ccf->user) { +- if (prctl(PR_SET_KEEPCAPS, 1, 0, 0, 0) == -1) { +- ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, +- "prctl(PR_SET_KEEPCAPS, 1) failed"); +- /* fatal */ +- exit(2); +- } +- } +-#endif +- +- if (setuid(ccf->user) == -1) { +- ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, +- "setuid(%d) failed", ccf->user); +- /* fatal */ +- exit(2); +- } +- +-#if (NGX_HAVE_CAPABILITIES) +- if (ccf->transparent && ccf->user) { +- struct __user_cap_data_struct data; +- struct __user_cap_header_struct header; +- +- ngx_memzero(&header, sizeof(struct __user_cap_header_struct)); +- ngx_memzero(&data, sizeof(struct __user_cap_data_struct)); +- +- header.version = _LINUX_CAPABILITY_VERSION_1; +- data.effective = CAP_TO_MASK(CAP_NET_RAW); +- data.permitted = data.effective; +- +- if (syscall(SYS_capset, &header, &data) == -1) { +- ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, +- "capset() failed"); +- /* fatal */ +- exit(2); +- } +- } +-#endif +- } +- + if (worker >= 0) { + cpu_affinity = ngx_get_cpu_affinity(worker); + +-- +2.17.1 + diff --git a/patches/nginx/drop-nginx-setgroups.patch b/patches/nginx/drop-nginx-setgroups.patch deleted file mode 100644 index 2bcc0eb..0000000 --- a/patches/nginx/drop-nginx-setgroups.patch +++ /dev/null @@ -1,57 +0,0 @@ -Description: Drop code where nginx drops privileges for worker - processes. While setuid is covered by the browser-support plug, - setgroups isn't covered by any plugs. This code isn't required - because in strict mode we run worker processes as root:root. - The seccomp violation follows: - = Seccomp = - Time: Jun 16 01:13:15 - Log: auid=4294967295 uid=0 gid=0 ses=4294967295 pid=6087 comm="nginx" - exe="/snap/keystone/x1/usr/sbin/nginx" sig=31 arch=c000003e - 116(setgroups) compat=0 ip=0x7f40e288af09 code=0x0 - Syscall: setgroups - Suggestion: - * adjust program to not use 'setgroups' until per-snap user/groups - are supported (https://launchpad.net/bugs/1446748) -Author: Corey Bryant -Forwarded: no - ---- - src/os/unix/ngx_process_cycle.c | 22 ---------------------- - 1 file changed, 22 deletions(-) - -diff --git a/src/os/unix/ngx_process_cycle.c b/src/os/unix/ngx_process_cycle.c -index 1710ea8..c428673 100644 ---- a/src/os/unix/ngx_process_cycle.c -+++ b/src/os/unix/ngx_process_cycle.c -@@ -824,28 +824,6 @@ ngx_worker_process_init(ngx_cycle_t *cycle, ngx_int_t worker) - } - } - -- if (geteuid() == 0) { -- if (setgid(ccf->group) == -1) { -- ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, -- "setgid(%d) failed", ccf->group); -- /* fatal */ -- exit(2); -- } -- -- if (initgroups(ccf->username, ccf->group) == -1) { -- ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, -- "initgroups(%s, %d) failed", -- ccf->username, ccf->group); -- } -- -- if (setuid(ccf->user) == -1) { -- ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno, -- "setuid(%d) failed", ccf->user); -- /* fatal */ -- exit(2); -- } -- } -- - if (worker >= 0) { - cpu_affinity = ngx_get_cpu_affinity(worker); - --- -2.7.4 - diff --git a/patches/open-iscsi/0001-Use-a-snap-specific-abstract-socket-address.patch b/patches/open-iscsi/0001-Use-a-snap-specific-abstract-socket-address.patch new file mode 100644 index 0000000..68fb5c7 --- /dev/null +++ b/patches/open-iscsi/0001-Use-a-snap-specific-abstract-socket-address.patch @@ -0,0 +1,56 @@ +From 84e8c808d146ef7d4a716bf951875f85fd7020c9 Mon Sep 17 00:00:00 2001 +From: Dmitrii Shcherbakov +Date: Tue, 18 Aug 2020 19:07:37 +0000 +Subject: [PATCH] Use a snap-specific abstract socket address + +* open-iscsi is included into Ubuntu cloud images and, as a result, +sockets with names hard-coded in the source get created and owned by +systemd at the host level; +* iscsid checks for the LISTEN_FDS environment variable to determine +whether systemd passes the necessary socket file descriptors to it - +this does not happen since iscsid.socket service name differs from the +actual service name: snap.microstack.iscsid.service; +* snapd's support for the systemd socket activation feature is present +but abstract socket names are restricted to be prefixed with +snap.. - this means that open-scsi needs to +be patched since the abstract domain socket name isn't configurable and +is hard-coded at the compile time. + +This patch alters the hard-coded abstract socket names in order to use +systemd socket activation via the means supported by snapd and to avoid +conflicts with an iscsid instance that might be used at the host where +this snap is installed. +--- + usr/mgmt_ipc.h | 2 +- + usr/uip_mgmt_ipc.h | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +diff --git a/usr/mgmt_ipc.h b/usr/mgmt_ipc.h +index 55972ed..aa66419 100644 +--- a/usr/mgmt_ipc.h ++++ b/usr/mgmt_ipc.h +@@ -23,7 +23,7 @@ + #include "iscsi_if.h" + #include "config.h" + +-#define ISCSIADM_NAMESPACE "ISCSIADM_ABSTRACT_NAMESPACE" ++#define ISCSIADM_NAMESPACE "snap.microstack.ISCSIADM_ABSTRACT_NAMESPACE" + #define PEERUSER_MAX 64 + + typedef enum iscsiadm_cmd { +diff --git a/usr/uip_mgmt_ipc.h b/usr/uip_mgmt_ipc.h +index 916113d..484e9f5 100644 +--- a/usr/uip_mgmt_ipc.h ++++ b/usr/uip_mgmt_ipc.h +@@ -24,7 +24,7 @@ + #include "initiator.h" + #include "transport.h" + +-#define ISCSID_UIP_NAMESPACE "ISCSID_UIP_ABSTRACT_NAMESPACE" ++#define ISCSID_UIP_NAMESPACE "snap.microstack.ISCSID_UIP_ABSTRACT_NAMESPACE" + + typedef enum iscsid_uip_cmd { + ISCSID_UIP_IPC_UNKNOWN = 0, +-- +2.17.1 + diff --git a/snap-overlay/bin/iscsid-start b/snap-overlay/bin/iscsid-start new file mode 100755 index 0000000..3c49567 --- /dev/null +++ b/snap-overlay/bin/iscsid-start @@ -0,0 +1,19 @@ +#!/bin/sh + +mkdir -p $SNAP_COMMON/etc/iscsi/ + +INAME_FILE=$SNAP_COMMON/etc/iscsi/initiatorname.iscsi + +if ! [ -f $INAME_FILE ]; then + # Generate a unique InitiatorName and save it + INAME=`iscsi-iname -p iqn.1993-08.org.debian:01` + echo "## DO NOT EDIT OR REMOVE THIS FILE!" > $INAME_FILE + echo "## If you remove this file, the iSCSI daemon will not start." >> $INAME_FILE + echo "## If you change the InitiatorName, existing access control lists" >> $INAME_FILE + echo "## may reject this initiator. The InitiatorName must be unique">> $INAME_FILE + echo "## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames." >> $INAME_FILE + printf "InitiatorName=$INAME\n" >> $INAME_FILE + chmod 600 $INAME_FILE +fi + +exec $SNAP/sbin/iscsid -p $SNAP_COMMON/var/run/iscsid.pid --initiatorname=$INAME_FILE --config=$SNAP_COMMON/etc/iscsi/iscsid.conf diff --git a/snap-overlay/bin/load-modules b/snap-overlay/bin/load-modules new file mode 100755 index 0000000..d6eb64a --- /dev/null +++ b/snap-overlay/bin/load-modules @@ -0,0 +1,5 @@ +#!/bin/bash + +set -ex + +modprobe -a vhost vhost-net vhost-scsi vhost-vsock pci-stub vfio nbd dm-mod dm-thin-pool dm-snapshot iscsi-tcp target-core-mod diff --git a/snap-overlay/bin/set-default-config b/snap-overlay/bin/set-default-config index f66e909..7d310cc 100755 --- a/snap-overlay/bin/set-default-config +++ b/snap-overlay/bin/set-default-config @@ -13,9 +13,11 @@ snapctl set \ # Networking related settings. snapctl set \ - config.network.dns=1.1.1.1 \ + config.network.dns-servers=1.1.1.1 \ + config.network.dns-domain=microstack.example. \ config.network.ext-gateway=10.20.20.1 \ config.network.control-ip=10.20.20.1 \ + config.network.node-fqdn=`hostname -f` \ config.network.compute-ip=10.20.20.1 \ config.network.ext-cidr=10.20.20.1/24 \ config.network.security-rules=true \ @@ -30,11 +32,19 @@ snapctl set \ config.credentials.os-password=keystone \ config.credentials.key-pair="/home/{USER}/snap/{SNAP_NAME}/common/.ssh/id_microstack" \ config.credentials.nova-password=nova \ + config.credentials.cinder-password=cinder \ config.credentials.neutron-password=neutron \ config.credentials.placement-password=placement \ config.credentials.glance-password=glance \ ; +# Cinder volume backend config. +snapctl set \ + config.cinder.setup-loop-based-cinder-lvm-backend=false \ + config.cinder.loop-device-file-size=32G \ + config.cinder.lvm-backend-volume-group=cinder-volumes \ + ; + # Host optimizations and fixes. snapctl set \ config.host.ip-forwarding=false \ @@ -45,12 +55,13 @@ snapctl set \ snapctl set \ config.services.control-plane=true \ config.services.hypervisor=true \ + config.services.spice-console=true \ ; # Clustering roles snapctl set \ - cluster.role=control \ - cluster.password=null \ + config.cluster.role=control \ + config.cluster.password=null \ ; # Uninstall stuff diff --git a/snap-overlay/bin/setup-br-ex b/snap-overlay/bin/setup-br-ex index 710816c..264595f 100755 --- a/snap-overlay/bin/setup-br-ex +++ b/snap-overlay/bin/setup-br-ex @@ -10,15 +10,30 @@ set -ex extcidr=$(snapctl get config.network.ext-cidr) +controlip=$(snapctl get config.network.control-ip) -# Create external integration bridge -ovs-vsctl --retry --may-exist add-br br-ex +# NOTE(dmitriis): this needs to be reworked to allow for OVN + direct exit of traffic to +# the provider network from a compute node. + +# Create an external bridge in the system datapath. +ovs-vsctl --retry --may-exist add-br br-ex -- set bridge br-ex datapath_type=system protocols=OpenFlow13,OpenFlow15 +ovs-vsctl set open . external-ids:ovn-bridge-mappings=physnet1:br-ex +ovs-vsctl set open . external-ids:ovn-cms-options="enable-chassis-as-gw" + +# Configure the settings used by self-configuration of ovn-controller. +ovs-vsctl set open . external-ids:ovn-encap-type=geneve -- set open . external-ids:ovn-encap-ip=$controlip +# Leave SB database connection details for ovn-controller to pick up. +ovs-vsctl set open . external-ids:ovn-remote='unix:/var/snap/microstack/common/run/ovn/ovnsb_db.sock' + +# NOTE: system-id is a randomly-generated UUID (see the --system-id=random option for ovs-ctl) +# As it is generated automatically, we do not set it here. +# It can be retrieved by looking at `ovs-vsctl get open_vswitch . external-ids`. # Configure br-ex ip address add $extcidr dev br-ex || : ip link set br-ex up || : -sudo iptables -w -t nat -A POSTROUTING -s $extcidr ! \ +iptables-legacy -w -t nat -A POSTROUTING -s $extcidr ! \ -d $extcidr -j MASQUERADE || : exit 0 diff --git a/snap-overlay/bin/target-start b/snap-overlay/bin/target-start new file mode 100755 index 0000000..823cbe7 --- /dev/null +++ b/snap-overlay/bin/target-start @@ -0,0 +1,4 @@ +#!/bin/sh + +# This wrapper is needed due to https://bugs.launchpad.net/snapd/+bug/1882839 +$SNAP/usr/bin/targetcli restoreconfig clear_existing=True diff --git a/snap-overlay/bin/target-stop b/snap-overlay/bin/target-stop new file mode 100755 index 0000000..97f78d4 --- /dev/null +++ b/snap-overlay/bin/target-stop @@ -0,0 +1,5 @@ +#!/bin/sh + +$SNAP/usr/bin/targetcli saveconfig +# This wrapper is needed due to https://bugs.launchpad.net/snapd/+bug/1882839 +$SNAP/usr/bin/targetcli clearconfig confirm=True diff --git a/snap-overlay/etc/neutron/dhcp_agent.ini b/snap-overlay/etc/neutron/dhcp_agent.ini deleted file mode 100644 index 8425be9..0000000 --- a/snap-overlay/etc/neutron/dhcp_agent.ini +++ /dev/null @@ -1,5 +0,0 @@ -[DEFAULT] -interface_driver = openvswitch -dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq -enable_isolated_metadata = True -dnsmasq_dns_servers = 1.1.1.1 diff --git a/snap-overlay/etc/neutron/l3_agent.ini b/snap-overlay/etc/neutron/l3_agent.ini deleted file mode 100644 index 015179f..0000000 --- a/snap-overlay/etc/neutron/l3_agent.ini +++ /dev/null @@ -1,2 +0,0 @@ -[DEFAULT] -interface_driver = openvswitch diff --git a/snap-overlay/etc/neutron/metadata_agent.ini b/snap-overlay/etc/neutron/metadata_agent.ini deleted file mode 100644 index 4cfdaf9..0000000 --- a/snap-overlay/etc/neutron/metadata_agent.ini +++ /dev/null @@ -1,3 +0,0 @@ -[DEFAULT] -nova_metadata_ip = 10.20.20.1 -metadata_proxy_shared_secret = supersecret diff --git a/snap-overlay/etc/neutron/neutron.conf b/snap-overlay/etc/neutron/neutron.conf index e29fbc7..fd17b4e 100644 --- a/snap-overlay/etc/neutron/neutron.conf +++ b/snap-overlay/etc/neutron/neutron.conf @@ -1,4 +1,10 @@ [DEFAULT] core_plugin = ml2 -service_plugins = router +service_plugins = ovn-router allow_overlapping_ips = True + +# Disable auto-scheduling of networks to DHCP agents since they are not used with OVN. +network_auto_schedule = False + +[ovn] +ovn_metadata_enabled = True diff --git a/snap-overlay/etc/neutron/plugins/ml2/ml2_conf.ini b/snap-overlay/etc/neutron/plugins/ml2/ml2_conf.ini index f86ee82..2c4de5c 100644 --- a/snap-overlay/etc/neutron/plugins/ml2/ml2_conf.ini +++ b/snap-overlay/etc/neutron/plugins/ml2/ml2_conf.ini @@ -1,13 +1,18 @@ [ml2] -mechanism_drivers = openvswitch -extension_drivers = port_security,trunk,qos -tenant_network_types = geneve,gre,vxlan +mechanism_drivers = ovn +extension_drivers = port_security,qos +tenant_network_types = geneve +overlay_ip_version = 4 +external_network_type = flat [ml2_type_geneve] vni_ranges = 1:65535 +max_header_size = 40 -[ml2_type_gre] -tunnel_id_ranges = 1:65535 +[ml2_type_flat] +flat_networks = * -[ml2_type_vxlan] -vni_ranges = 1:65535 +[ovn] +# TODO(dmitriis): replace the common path with a template. +ovn_nb_connection = unix:/var/snap/microstack/common/run/ovn/ovnnb_db.sock +ovn_sb_connection = unix:/var/snap/microstack/common/run/ovn/ovnsb_db.sock diff --git a/snap-overlay/etc/neutron/plugins/ml2/openvswitch_agent.ini b/snap-overlay/etc/neutron/plugins/ml2/openvswitch_agent.ini deleted file mode 100644 index 1740770..0000000 --- a/snap-overlay/etc/neutron/plugins/ml2/openvswitch_agent.ini +++ /dev/null @@ -1,11 +0,0 @@ -# Snap provided defaults for neutron-openvswitch-agent -[securitygroup] -enable_security_group = True -firewall_driver = openvswitch - -[AGENT] -tunnel_types = geneve,vxlan,gre - -[ovs] -local_ip = 127.0.0.1 -bridge_mappings = physnet1:br-ex diff --git a/snap-overlay/etc/placement/placement.conf b/snap-overlay/etc/placement/placement.conf new file mode 100644 index 0000000..ea4f7f4 --- /dev/null +++ b/snap-overlay/etc/placement/placement.conf @@ -0,0 +1,3 @@ +# Snap distribution defaults - do not change, override in $SNAP_COMMON/etc/cinder.conf.d +[database] +max_retries = -1 diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/_styles.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/_styles.scss similarity index 96% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/_styles.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/_styles.scss index cca943b..1857560 100644 --- a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/_styles.scss +++ b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/_styles.scss @@ -116,7 +116,8 @@ table { /* Login splash screen */ #splash { background: url("/static/themes/ubuntu/img/image-background-pattern.png"); - background: linear-gradient(to right, rgba(100, 100, 100, 0.2), transparent), url("/static/themes/ubuntu/img/image-background-pattern.png"); + position: absolute; + width: 100vw; .login { background-color: $white; diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/_variables.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/_variables.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/_variables.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/_variables.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_styles.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_styles.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_styles.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_styles.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_variables.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_variables.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_variables.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/_variables.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navbar.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navbar.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navbar.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navbar.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navs.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navs.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navs.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/bootstrap/components/_navs.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/_styles.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/_styles.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/_styles.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/_styles.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/_variables.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/_variables.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/_variables.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/_variables.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_context_selection.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_context_selection.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_context_selection.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_context_selection.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_pie_charts.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_pie_charts.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_pie_charts.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_pie_charts.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_sidebar.scss b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_sidebar.scss similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_sidebar.scss rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/horizon/components/_sidebar.scss diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/img/image-background-pattern.png b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/img/image-background-pattern.png similarity index 98% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/img/image-background-pattern.png rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/img/image-background-pattern.png index f19c7b906f7a216d60a394e3f3cdd20d01a43d6c..2abea65ab6e00fbc486ae1acc354aa649ccf0e55 100644 GIT binary patch delta 30 mcmZ2~K))(r}blNA{wCs#A-O}&t)(r}bjKz}`7$bpXC8HjYT+V3CSls-Gar+}iMw<)(1V;?_ diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-orange.svg b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-orange.svg similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-orange.svg rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-orange.svg diff --git a/snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-white.svg b/snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-white.svg similarity index 100% rename from snap-overlay/lib/python3.6/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-white.svg rename to snap-overlay/lib/python3.8/site-packages/openstack_dashboard/themes/ubuntu/img/logo-ubuntu-white.svg diff --git a/snap-overlay/snap-openstack.yaml b/snap-overlay/snap-openstack.yaml index 3037601..4dcb465 100644 --- a/snap-overlay/snap-openstack.yaml +++ b/snap-overlay/snap-openstack.yaml @@ -10,6 +10,7 @@ setup: - "{snap_common}/etc/nginx/sites-enabled" - "{snap_common}/etc/nginx/snap/sites-enabled" - "{snap_common}/etc/glance/glance.conf.d" + - "{snap_common}/etc/placement/placement.conf.d" - "{snap_common}/etc/horizon/horizon.conf.d" - "{snap_common}/etc/horizon/local_settings.d" - "{snap_common}/var/horizon/static" @@ -17,6 +18,7 @@ setup: - "{snap_common}/etc/cinder/uwsgi/snap" - "{snap_common}/etc/nova/uwsgi/snap" - "{snap_common}/etc/horizon/uwsgi/snap" + - "{snap_common}/etc/placement/uwsgi/snap" - "{snap_common}/etc/rabbitmq" - "{snap_common}/fernet-keys" - "{snap_common}/lib" @@ -25,6 +27,9 @@ setup: - "{snap_common}/log" - "{snap_common}/run" - "{snap_common}/lib/instances" + - "{snap_common}/etc/apparmor.d/libvirt" + - "{snap_common}/etc/iscsi" + - "{snap_common}/etc/target" templates: keystone-nginx.conf.j2: "{snap_common}/etc/nginx/snap/sites-enabled/keystone.conf" keystone-snap.conf.j2: "{snap_common}/etc/keystone/keystone.conf.d/keystone-snap.conf" @@ -33,28 +38,40 @@ setup: nova-snap.conf.j2: "{snap_common}/etc/nova/nova.conf.d/nova-snap.conf" nova-nginx.conf.j2: "{snap_common}/etc/nginx/snap/sites-enabled/nova.conf" glance-snap.conf.j2: "{snap_common}/etc/glance/glance.conf.d/glance-snap.conf" + placement-nginx.conf.j2: "{snap_common}/etc/nginx/snap/sites-enabled/placement.conf" + placement-snap.conf.j2: "{snap_common}/etc/placement/placement.conf.d/placement-snap.conf" cinder-nginx.conf.j2: "{snap_common}/etc/nginx/snap/sites-enabled/cinder.conf" cinder-snap.conf.j2: "{snap_common}/etc/cinder/cinder.conf.d/cinder-snap.conf" + cinder.database.conf.j2: "{snap_common}/etc/cinder/cinder.conf.d/database.conf" + cinder.rabbitmq.conf.j2: "{snap_common}/etc/cinder/cinder.conf.d/rabbitmq.conf" + cinder.keystone.conf.j2: "{snap_common}/etc/cinder/cinder.conf.d/keystone.conf" + cinder-rootwrap.conf.j2: "{snap_common}/etc/cinder/rootwrap.conf" horizon-snap.conf.j2: "{snap_common}/etc/horizon/horizon.conf.d/horizon-snap.conf" horizon-nginx.conf.j2: "{snap_common}/etc/nginx/snap/sites-enabled/horizon.conf" 05_snap_tweaks.j2: "{snap_common}/etc/horizon/local_settings.d/_05_snap_tweaks.py" libvirtd.conf.j2: "{snap_common}/libvirt/libvirtd.conf" virtlogd.conf.j2: "{snap_common}/libvirt/virtlogd.conf" microstack.rc.j2: "{snap_common}/etc/microstack.rc" + microstack.json.j2: "{snap_common}/etc/microstack.json" glance.conf.d.keystone.conf.j2: "{snap_common}/etc/glance/glance.conf.d/keystone.conf" + placement.conf.d.keystone.conf.j2: "{snap_common}/etc/placement/placement.conf.d/keystone.conf" nova.conf.d.keystone.conf.j2: "{snap_common}/etc/nova/nova.conf.d/keystone.conf" nova.conf.d.database.conf.j2: "{snap_common}/etc/nova/nova.conf.d/database.conf" nova.conf.d.rabbitmq.conf.j2: "{snap_common}/etc/nova/nova.conf.d/rabbitmq.conf" - nova.conf.d.nova-placement.conf.j2: "{snap_common}/etc/nova/nova.conf.d/nova-placement.conf" nova.conf.d.glance.conf.j2: "{snap_common}/etc/nova/nova.conf.d/glance.conf" nova.conf.d.neutron.conf.j2: "{snap_common}/etc/nova/nova.conf.d/neutron.conf" + nova.conf.d.placement.conf.j2: "{snap_common}/etc/nova/nova.conf.d/placement.conf" + nova.conf.d.console.conf.j2: "{snap_common}/etc/nova/nova.conf.d/console.conf" keystone.database.conf.j2: "{snap_common}/etc/keystone/keystone.conf.d/database.conf" glance.database.conf.j2: "{snap_common}/etc/glance/glance.conf.d/database.conf" + placement.conf.d.database.conf.j2: "{snap_common}/etc/placement/placement.conf.d/database.conf" neutron.keystone.conf.j2: "{snap_common}/etc/neutron/neutron.conf.d/keystone.conf" neutron.nova.conf.j2: "{snap_common}/etc/neutron/neutron.conf.d/nova.conf" neutron.database.conf.j2: "{snap_common}/etc/neutron/neutron.conf.d/database.conf" neutron.conf.d.rabbitmq.conf.j2: "{snap_common}/etc/neutron/neutron.conf.d/rabbitmq.conf" + neutron_ovn_metadata_agent.ini.j2: "{snap_common}/etc/neutron/neutron_ovn_metadata_agent.ini" rabbitmq.conf.j2: "{snap_common}/etc/rabbitmq/rabbitmq.config" + iscsid.conf.j2: "{snap_common}/etc/iscsi/iscsid.conf" # LMA stack templates telegraf.conf.j2: "{snap_common}/etc/telegraf/telegraf-microstack.conf" @@ -63,17 +80,22 @@ setup: chmod: "{snap_common}/instances": 0755 "{snap_common}/etc/microstack.rc": 0644 + "{snap_common}/etc/microstack.json": 0644 snap-config-keys: ospassword: 'config.credentials.os-password' nova_password: 'config.credentials.nova-password' + cinder_password: 'config.credentials.cinder-password' neutron_password: 'config.credentials.neutron-password' placement_password: 'config.credentials.placement-password' glance_password: 'config.credentials.glance-password' + placement_password: 'config.credentials.placement-password' control_ip: 'config.network.control-ip' + node_fqdn: 'config.network.node-fqdn' compute_ip: 'config.network.compute-ip' extgateway: 'config.network.ext-gateway' extcidr: 'config.network.ext-cidr' - dns: 'config.network.dns' + dns_servers: 'config.network.dns-servers' + dns_domain: 'config.network.dns-domain' dashboard_allowed_hosts: 'config.network.dashboard-allowed-hosts' dashboard_port: 'config.network.ports.dashboard' mysql_port: 'config.network.ports.mysql' @@ -83,6 +105,10 @@ setup: monitoring_tag: 'config.monitoring.tag' monitoring_ipmi: 'config.monitoring.ipmi' alerting_tag: 'config.alerting.tag' + ovn_nb_connection: 'config.network.ovn-nb-connection' + ovn_sb_connection: 'config.network.ovn-sb-connection' + setup_loop_based_cinder_lvm_backend: 'config.cinder.setup-loop-based-cinder-lvm-backend' + lvm_backend_volume_group: 'config.cinder.lvm-backend-volume-group' entry_points: keystone-manage: binary: "{snap}/bin/keystone-manage" @@ -116,19 +142,6 @@ entry_points: - "{snap_common}/etc/nova/nova.conf" config-dirs: - "{snap_common}/etc/nova/nova.conf.d" - nova-uwsgi: - type: uwsgi - uwsgi-dir: "{snap_common}/etc/nova/uwsgi/snap" - uwsgi-dir-override: "{snap_common}/etc/nova/uwsgi" - config-files: - - "{snap}/etc/nova/nova.conf" - config-files-override: - - "{snap_common}/etc/nova/nova.conf" - config-dirs: - - "{snap_common}/etc/nova/nova.conf.d" - templates: - nova-placement-api.ini.j2: - "{snap_common}/etc/nova/uwsgi/snap/nova-placement-api.ini" nova-conductor: binary: "{snap}/bin/nova-conductor" config-files: @@ -169,6 +182,17 @@ entry_points: - "{snap_common}/etc/nova/nova.conf" config-dirs: - "{snap_common}/etc/nova/nova.conf.d" + nova-spicehtml5proxy: + binary: "{snap}/bin/nova-spicehtml5proxy" + config-files: + - "{snap}/etc/nova/nova.conf" + config-files-override: + - "{snap_common}/etc/nova/nova.conf" + config-dirs: + - "{snap_common}/etc/nova/nova.conf.d" + templates: + nova.conf.d.console.conf.j2: + "{snap_common}/etc/nova/nova.conf.d/console.conf" neutron-db-manage: binary: "{snap}/bin/neutron-db-manage" config-files: @@ -215,36 +239,19 @@ entry_points: - "{snap_common}/etc/neutron/neutron.conf" config-dirs: - "{snap_common}/etc/neutron/neutron.conf.d" - neutron-l3-agent: - binary: "{snap}/bin/neutron-l3-agent" + neutron-ovn-metadata-agent: + binary: "{snap}/bin/neutron-ovn-metadata-agent" config-files: - "{snap}/etc/neutron/neutron.conf" - - "{snap}/etc/neutron/l3_agent.ini" + - "{snap}/etc/neutron/neutron_ovn_metadata_agent.ini" config-files-override: - "{snap_common}/etc/neutron/neutron.conf" - - "{snap_common}/etc/neutron/l3_agent.ini" - config-dirs: - - "{snap_common}/etc/neutron/neutron.conf.d" - neutron-dhcp-agent: - binary: "{snap}/bin/neutron-dhcp-agent" - config-files: - - "{snap}/etc/neutron/neutron.conf" - - "{snap}/etc/neutron/dhcp_agent.ini" - config-files-override: - - "{snap_common}/etc/neutron/neutron.conf" - - "{snap_common}/etc/neutron/dhcp_agent.ini" - config-dirs: - - "{snap_common}/etc/neutron/neutron.conf.d" - neutron-metadata-agent: - binary: "{snap}/bin/neutron-metadata-agent" - config-files: - - "{snap}/etc/neutron/neutron.conf" - - "{snap}/etc/neutron/metadata_agent.ini" - config-files-override: - - "{snap_common}/etc/neutron/neutron.conf" - - "{snap_common}/etc/neutron/metadata_agent.ini" + - "{snap_common}/etc/neutron/neutron_ovn_metadata_agent.ini" config-dirs: - "{snap_common}/etc/neutron/neutron.conf.d" + templates: + neutron_ovn_metadata_agent.ini.j2: + "{snap_common}/etc/neutron/neutron_ovn_metadata_agent.ini" glance-manage: binary: "{snap}/bin/glance-manage" config-files: @@ -269,6 +276,27 @@ entry_points: - "{snap_common}/etc/glance/glance-api.conf" config-dirs: - "{snap_common}/etc/glance/glance.conf.d" + placement-uwsgi: + type: uwsgi + uwsgi-dir: "{snap_common}/etc/placement/uwsgi/snap" + uwsgi-dir-override: "{snap_common}/etc/placement/uwsgi" + config-files: + - "{snap}/etc/placement/placement.conf" + config-files-override: + - "{snap_common}/etc/placement/placement.conf" + config-dirs: + - "{snap_common}/etc/placement/placement.conf.d" + templates: + placement-api.ini.j2: + "{snap_common}/etc/placement/uwsgi/snap/placement-api.ini" + placement-manage: + binary: "{snap}/bin/placement-manage" + config-files: + - "{snap}/etc/placement/placement.conf" + config-files-override: + - "{snap_common}/etc/placement/placement.conf" + config-dirs: + - "{snap_common}/etc/placement/placement.conf.d" cinder-backup: binary: "{snap}/bin/cinder-backup" config-files: diff --git a/snap-overlay/templates/cinder-rootwrap.conf.j2 b/snap-overlay/templates/cinder-rootwrap.conf.j2 new file mode 100644 index 0000000..782e70c --- /dev/null +++ b/snap-overlay/templates/cinder-rootwrap.conf.j2 @@ -0,0 +1,6 @@ +[DEFAULT] +filters_path={{ snap }}/etc/cinder/rootwrap.d + +use_syslog=False +syslog_log_facility=syslog +syslog_log_level=ERROR diff --git a/snap-overlay/templates/cinder-snap.conf.j2 b/snap-overlay/templates/cinder-snap.conf.j2 index 0b75ba5..b0f09d5 100644 --- a/snap-overlay/templates/cinder-snap.conf.j2 +++ b/snap-overlay/templates/cinder-snap.conf.j2 @@ -2,9 +2,26 @@ # Set state path to writable directory state_path = {{ snap_common }}/lib +resource_query_filters_file = {{ snap }}/etc/cinder/resource_filters.json + # Set volume configuration file storage directory volumes_dir = {{ snap_common }}/lib/volumes +my_ip = {{ compute_ip }} + +rootwrap_config = {{ snap_common }}/etc/cinder/rootwrap.conf + +enabled_backends = {% if setup_loop_based_cinder_lvm_backend %}lvm-loop-based-backend{% endif %} + +{% if setup_loop_based_cinder_lvm_backend %} +[lvm-loop-based-backend] +volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver +iscsi_protocol = iscsi +target_helper = lioadm +volume_group = {{ lvm_backend_volume_group }} +volume_backend_name=lvm-loop-based +{% endif %} + [oslo_concurrency] # Oslo Concurrency lock path lock_path = {{ snap_common }}/lock diff --git a/snap-overlay/templates/cinder.database.conf.j2 b/snap-overlay/templates/cinder.database.conf.j2 new file mode 100644 index 0000000..84a8c89 --- /dev/null +++ b/snap-overlay/templates/cinder.database.conf.j2 @@ -0,0 +1,2 @@ +[database] +connection = mysql+pymysql://cinder:cinder@{{ control_ip }}:{{ mysql_port }}/cinder diff --git a/snap-overlay/templates/cinder.keystone.conf.j2 b/snap-overlay/templates/cinder.keystone.conf.j2 new file mode 100644 index 0000000..ccc3fb1 --- /dev/null +++ b/snap-overlay/templates/cinder.keystone.conf.j2 @@ -0,0 +1,13 @@ +[DEFAULT] +auth_strategy = keystone + +[keystone_authtoken] +auth_uri = http://{{ control_ip }}:5000 +auth_url = http://{{ control_ip }}:5000 +memcached_servers = {{ control_ip }}:11211 +auth_type = password +project_domain_name = default +user_domain_name = default +project_name = service +username = cinder +password = {{ cinder_password }} diff --git a/snap-overlay/templates/cinder.rabbitmq.conf.j2 b/snap-overlay/templates/cinder.rabbitmq.conf.j2 new file mode 100644 index 0000000..ef149bc --- /dev/null +++ b/snap-overlay/templates/cinder.rabbitmq.conf.j2 @@ -0,0 +1,2 @@ +[DEFAULT] +transport_url = rabbit://openstack:rabbitmq@{{ control_ip }}:{{ rabbit_port }} diff --git a/snap-overlay/templates/iscsid.conf.j2 b/snap-overlay/templates/iscsid.conf.j2 new file mode 100644 index 0000000..387682d --- /dev/null +++ b/snap-overlay/templates/iscsid.conf.j2 @@ -0,0 +1,22 @@ +iscsid.startup = {{ snap }}/sbin/iscsid +node.startup = manual +node.leading_login = No +node.session.timeo.replacement_timeout = 120 +node.conn[0].timeo.login_timeout = 15 +node.conn[0].timeo.logout_timeout = 15 +node.conn[0].timeo.noop_out_interval = 5 +node.conn[0].timeo.noop_out_timeout = 5 +node.session.err_timeo.abort_timeout = 15 +node.session.err_timeo.lu_reset_timeout = 30 +node.session.err_timeo.tgt_reset_timeout = 30 +node.session.cmds_max = 128 +node.session.queue_depth = 32 +node.session.xmit_thread_priority = -20 +node.session.iscsi.InitialR2T = No +node.session.iscsi.ImmediateData = Yes +node.session.iscsi.FirstBurstLength = 262144 +node.session.iscsi.MaxBurstLength = 16776192 +node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144 +node.conn[0].iscsi.MaxXmitDataSegmentLength = 0 +discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768 +node.session.nr_sessions = 1 diff --git a/snap-overlay/templates/microstack.json.j2 b/snap-overlay/templates/microstack.json.j2 new file mode 100644 index 0000000..e0a8f8a --- /dev/null +++ b/snap-overlay/templates/microstack.json.j2 @@ -0,0 +1,26 @@ +{ + "openstack": { + "admin": { + "password": "{{ ospassword }}", + "project_domain_name": "default", + "project_name": "admin", + "user_domain_name": "default", + "username": "admin" + }, + "api_info": { + "keystone": { + "service_type": "identityv3", + "version": 3 + } + }, + "auth_url": "http://{{ control_ip }}:5000", + "endpoint_type": null, + "https_cacert": "", + "https_cert": "", + "https_insecure": false, + "https_key": "", + "profiler_conn_str": null, + "profiler_hmac_key": null, + "region_name": "" + } +} diff --git a/snap-overlay/templates/neutron-snap.conf.j2 b/snap-overlay/templates/neutron-snap.conf.j2 index 196f315..2d089db 100644 --- a/snap-overlay/templates/neutron-snap.conf.j2 +++ b/snap-overlay/templates/neutron-snap.conf.j2 @@ -4,6 +4,18 @@ state_path = {{ snap_common }}/lib # Log to systemd journal use_journal = True +{% if dns_domain %} +dns_domain = {{ dns_domain }} +{% endif %} + [oslo_concurrency] # Oslo Concurrency lock path lock_path = {{ snap_common }}/lock + +[ovn] +{% if dns_servers %} +dns_servers= {{ dns_servers }} +{% endif %} + +# TODO(dmitriis): enable once external bridge IP addressing for compute nodes is figured out. +# enable_distributed_floating_ip = True diff --git a/snap-overlay/templates/neutron_ovn_metadata_agent.ini.j2 b/snap-overlay/templates/neutron_ovn_metadata_agent.ini.j2 new file mode 100644 index 0000000..2f0f674 --- /dev/null +++ b/snap-overlay/templates/neutron_ovn_metadata_agent.ini.j2 @@ -0,0 +1,13 @@ +[DEFAULT] +metadata_proxy_shared_secret = supersecret + +[ovs] +ovsdb_connection = unix:{{ snap_common }}/run/openvswitch/db.sock + +[ovn] +{% if ovn_nb_connection %} +ovn_nb_connection = {{ ovn_nb_connection }} +{% endif %} +{% if ovn_sb_connection %} +ovn_sb_connection = {{ ovn_sb_connection }} +{% endif %} diff --git a/snap-overlay/templates/nginx.conf.j2 b/snap-overlay/templates/nginx.conf.j2 index 3cb59b1..a62b8c5 100644 --- a/snap-overlay/templates/nginx.conf.j2 +++ b/snap-overlay/templates/nginx.conf.j2 @@ -1,4 +1,4 @@ -user root root; +user snap_daemon snap_daemon; worker_processes auto; pid {{ snap_common }}/run/nginx.pid; diff --git a/snap-overlay/templates/nova-snap.conf.j2 b/snap-overlay/templates/nova-snap.conf.j2 index 196f315..bacb966 100644 --- a/snap-overlay/templates/nova-snap.conf.j2 +++ b/snap-overlay/templates/nova-snap.conf.j2 @@ -4,6 +4,20 @@ state_path = {{ snap_common }}/lib # Log to systemd journal use_journal = True +# Set a hostname to be an FQDN to avoid issues with port binding for +# which a hostname of a Nova node must match a hostname of an OVN chassis. +host = {{ node_fqdn }} + [oslo_concurrency] # Oslo Concurrency lock path lock_path = {{ snap_common }}/lock + +[os_vif_ovs] +# Nova relies on os-vif for openvswitch interface plugging and needs a connection to +# OVSDB. This is done via a TCP connection to localhost by default so we override this to +# use a unix socket instead. +# See os-vif/releasenotes/notes/add-ovsdb-native-322fffb49c91503d.yaml +ovsdb_connection = unix:{{ snap_common }}/run/openvswitch/db.sock + +[cinder] +os_region_name = microstack diff --git a/snap-overlay/templates/nova.conf.d.console.conf.j2 b/snap-overlay/templates/nova.conf.d.console.conf.j2 new file mode 100644 index 0000000..3f58232 --- /dev/null +++ b/snap-overlay/templates/nova.conf.d.console.conf.j2 @@ -0,0 +1,18 @@ +[DEFAULT] +web = {{ snap }}/usr/share/spice-html5 + +[vnc] +enabled = False + +[spice] +# Proxy configuration (controller only). +html5proxy_host = 0.0.0.0 +html5proxy_port = 6082 + +enabled = True +agent_enabled = True +html5proxy_base_url = http://{{ control_ip }}:6082/spice_auto.html +server_listen = {{ compute_ip }} +server_proxyclient_address = {{ compute_ip }} +keymap = en-us + diff --git a/snap-overlay/templates/nova.conf.d.placement.conf.j2 b/snap-overlay/templates/nova.conf.d.placement.conf.j2 new file mode 100644 index 0000000..7f83297 --- /dev/null +++ b/snap-overlay/templates/nova.conf.d.placement.conf.j2 @@ -0,0 +1,11 @@ +[placement] +auth_uri = http://{{ control_ip }}:5000 +auth_url = http://{{ control_ip }}:5000 +memcached_servers = {{ control_ip }}:11211 +auth_type = password +project_domain_name = default +user_domain_name = default +project_name = service +username = nova +password = {{ nova_password }} +os_region_name = RegionOne diff --git a/snap-overlay/templates/nrpe.cfg.j2 b/snap-overlay/templates/nrpe.cfg.j2 index abc002f..09cc3fc 100644 --- a/snap-overlay/templates/nrpe.cfg.j2 +++ b/snap-overlay/templates/nrpe.cfg.j2 @@ -7,8 +7,8 @@ server_port=5666 #allowed_hosts=0.0.0.0/0 #allowed_hosts=10.0.0.0/8,127.0.0.1 -nrpe_user=root -nrpe_group=root +nrpe_user=snap_daemon +nrpe_group=snap_daemon dont_blame_nrpe=0 debug=0 pid_file={{ snap_common }}/run/nrpe.pid @@ -32,10 +32,6 @@ command[check_libvirtd]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd. command[check_memcached]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.memcached command[check_mysqld]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.mysqld command[check_neutron_api]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.neutron-api -command[check_neutron_dhcp_agent]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.neutron-dhcp-agent -command[check_neutron_l3_agent]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.neutron-l3-agent -command[check_neutron_metadata_agent]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.neutron-metadata-agent -command[check_neutron_openvswitch_agent]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.neutron-openvswitch-agent command[check_nginx]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.nginx command[check_nova_api]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.nova-api command[check_nova_api_metadata]=python3 {{ snap }}/usr/lib/nagios/plugins/check_systemd.py snap.microstack.nova-api-metadata diff --git a/snap-overlay/templates/nova-placement-api.ini.j2 b/snap-overlay/templates/placement-api.ini.j2 similarity index 75% rename from snap-overlay/templates/nova-placement-api.ini.j2 rename to snap-overlay/templates/placement-api.ini.j2 index 320fc43..9df593d 100644 --- a/snap-overlay/templates/nova-placement-api.ini.j2 +++ b/snap-overlay/templates/placement-api.ini.j2 @@ -1,5 +1,5 @@ [uwsgi] -wsgi-file = {{ snap }}/bin/nova-placement-api +wsgi-file = {{ snap }}/bin/placement-api uwsgi-socket = {{ snap_common }}/run/placement-api.sock buffer-size = 65535 master = true @@ -8,3 +8,4 @@ processes = 4 thunder-lock = true lazy-apps = true home = {{ snap }}/usr +pyargv = {{ pyargv }} diff --git a/snap-overlay/templates/placement-nginx.conf.j2 b/snap-overlay/templates/placement-nginx.conf.j2 new file mode 100644 index 0000000..c60dc9d --- /dev/null +++ b/snap-overlay/templates/placement-nginx.conf.j2 @@ -0,0 +1,10 @@ +server { + listen 8778; + error_log syslog:server=unix:/dev/log; + access_log syslog:server=unix:/dev/log; + location / { + include {{ snap }}/usr/conf/uwsgi_params; + uwsgi_param SCRIPT_NAME ''; + uwsgi_pass unix://{{ snap_common }}/run/placement-api.sock; + } +} diff --git a/snap-overlay/templates/placement-snap.conf.j2 b/snap-overlay/templates/placement-snap.conf.j2 new file mode 100644 index 0000000..196f315 --- /dev/null +++ b/snap-overlay/templates/placement-snap.conf.j2 @@ -0,0 +1,9 @@ +[DEFAULT] +# Set state path to writable directory +state_path = {{ snap_common }}/lib +# Log to systemd journal +use_journal = True + +[oslo_concurrency] +# Oslo Concurrency lock path +lock_path = {{ snap_common }}/lock diff --git a/snap-overlay/templates/placement.conf.d.database.conf.j2 b/snap-overlay/templates/placement.conf.d.database.conf.j2 new file mode 100644 index 0000000..4100ba1 --- /dev/null +++ b/snap-overlay/templates/placement.conf.d.database.conf.j2 @@ -0,0 +1,2 @@ +[placement_database] +connection = mysql+pymysql://placement:placement@{{ control_ip }}:{{ mysql_port }}/placement diff --git a/snap-overlay/templates/nova.conf.d.nova-placement.conf.j2 b/snap-overlay/templates/placement.conf.d.keystone.conf.j2 similarity index 58% rename from snap-overlay/templates/nova.conf.d.nova-placement.conf.j2 rename to snap-overlay/templates/placement.conf.d.keystone.conf.j2 index c6c33c3..4dbc846 100644 --- a/snap-overlay/templates/nova.conf.d.nova-placement.conf.j2 +++ b/snap-overlay/templates/placement.conf.d.keystone.conf.j2 @@ -1,9 +1,13 @@ -[placement] -os_region_name = microstack -project_domain_name = default -project_name = service -auth_type = password -user_domain_name = default +[keystone_authtoken] +auth_uri = http://{{ control_ip }}:5000 auth_url = http://{{ control_ip }}:5000 +memcached_servers = {{ control_ip }}:11211 +auth_type = password +project_domain_name = default +user_domain_name = default +project_name = service username = placement password = {{ placement_password }} + +[paste_deploy] +flavor = keystone diff --git a/snap-overlay/templates/placement.conf.d.placement-api.conf.j2 b/snap-overlay/templates/placement.conf.d.placement-api.conf.j2 new file mode 100644 index 0000000..1def0f0 --- /dev/null +++ b/snap-overlay/templates/placement.conf.d.placement-api.conf.j2 @@ -0,0 +1,2 @@ +[placement] +randomize_allocation_candidates = true diff --git a/snap-overlay/templates/targets.conf.j2 b/snap-overlay/templates/targets.conf.j2 new file mode 100644 index 0000000..688c5b3 --- /dev/null +++ b/snap-overlay/templates/targets.conf.j2 @@ -0,0 +1,2 @@ +include {{ snap_common }}/lib/volumes/* +default-driver iscsi diff --git a/snap-wrappers/ovn/ovn-wrapper b/snap-wrappers/ovn/ovn-wrapper new file mode 100755 index 0000000..1dd9ed7 --- /dev/null +++ b/snap-wrappers/ovn/ovn-wrapper @@ -0,0 +1,25 @@ +#!/bin/bash + +set -e + +export OVN_LOGDIR=${SNAP_COMMON}/log/ovn +export OVN_RUNDIR=${SNAP_COMMON}/run/ovn +export OVN_SYSCONFDIR=${SNAP_COMMON}/etc +export OVN_PKGDATADIR=${SNAP}/usr/local/share/ovn +export OVN_BINDIR=${SNAP}/bin +export OVN_SBINDIR=${SNAP}/sbin + + +mkdir -p ${OVN_LOGDIR} +mkdir -p ${OVN_RUNDIR} +mkdir -p ${OVN_SYSCONFDIR}/ovn + +if [ `basename $1` = 'ovn-ctl' -a `snapctl get config.clustered` == 'true' ] +then + # TODO: replace this with a secure alternative once TLS is supported. + # Create an SB TCP socket to be used by remote ovn-controller and neutron-ovn-metadata + # agents. + exec $@ --db-sb-create-insecure-remote=yes +else + exec $@ +fi diff --git a/snap-wrappers/ovs/ovs-wrapper b/snap-wrappers/ovs/ovs-wrapper index 0c276e5..934019c 100755 --- a/snap-wrappers/ovs/ovs-wrapper +++ b/snap-wrappers/ovs/ovs-wrapper @@ -5,12 +5,16 @@ set -e export OVS_LOGDIR=${SNAP_COMMON}/log/openvswitch export OVS_RUNDIR=${SNAP_COMMON}/run/openvswitch export OVS_SYSCONFDIR=${SNAP_COMMON}/etc -export OVS_PKGDATADIR=${SNAP}/share/openvswitch +export OVS_PKGDATADIR=${SNAP}/usr/local/share/openvswitch export OVS_BINDIR=${SNAP}/bin export OVS_SBINDIR=${SNAP}/sbin -mkdir -p ${OVS_LOGDIR} -mkdir -p ${OVS_RUNDIR} -mkdir -p ${OVS_SYSCONFDIR}/openvswitch - -exec $@ +if [ `basename $1` = 'ovs-ctl' ] +then + mkdir -p ${OVS_LOGDIR} + mkdir -p ${OVS_RUNDIR} + mkdir -p ${OVS_SYSCONFDIR}/openvswitch + exec $@ --system-id=random +else + exec $@ +fi diff --git a/snap-wrappers/rabbitmq/erl b/snap-wrappers/rabbitmq/erl index 5c631e9..12bd6df 100755 --- a/snap-wrappers/rabbitmq/erl +++ b/snap-wrappers/rabbitmq/erl @@ -19,7 +19,7 @@ # %CopyrightEnd% # ROOTDIR=$SNAP/usr/lib/erlang -BINDIR=$ROOTDIR/erts-9.2/bin +BINDIR=$ROOTDIR/erts-10.6.4/bin EMU=beam PROGNAME=`echo $0 | sed 's/.*\///'` export EMU diff --git a/snap-wrappers/rabbitmq/rabbitmq-script-wrapper b/snap-wrappers/rabbitmq/rabbitmq-script-wrapper index a3bd83d..14ed2a7 100755 --- a/snap-wrappers/rabbitmq/rabbitmq-script-wrapper +++ b/snap-wrappers/rabbitmq/rabbitmq-script-wrapper @@ -15,6 +15,10 @@ ## Copyright (c) 2007-2014 GoPivotal, Inc. All rights reserved. ## +# Make sure files created by rabbitmq (including the .erlang.cookie file which +# needs to be restricted to the user only) are created with strict permissions. +umask 077 + mkdir -p $SNAP_COMMON/lib/rabbitmq cd $SNAP_COMMON/lib/rabbitmq diff --git a/snap/hooks/install b/snap/hooks/install index a1bc84b..81f2724 100755 --- a/snap/hooks/install +++ b/snap/hooks/install @@ -1,10 +1,47 @@ #!/bin/bash set -ex + # Initialize config set-default-config +# TODO(dmitriis): disable other services and only enable them once the +# prerequisites are met instead of allowing snapd to start them and make them fail. +# snapd starts all non-disabled services by default which may lead to errors such as +# a module loading error in case of ovs-vswitchd. The sequence is as follows: +# 1. The snap is installed; +# 2. Non-disabled services are started; +# 3. Interfaces that do not have auto-connection enabled are manually connected by +# an operator (connecting openvswitch-support loads the openvswitch kernel module +# but auto-connection is not enabled for openvswitch-support). +snapctl stop --disable $SNAP_INSTANCE_NAME.ovsdb-server +snapctl stop --disable $SNAP_INSTANCE_NAME.ovn-ovsdb-server-sb +snapctl stop --disable $SNAP_INSTANCE_NAME.ovn-ovsdb-server-nb +snapctl stop --disable $SNAP_INSTANCE_NAME.ovs-vswitchd +snapctl stop --disable $SNAP_INSTANCE_NAME.ovn-northd +snapctl stop --disable $SNAP_INSTANCE_NAME.ovn-controller + +snapctl stop --disable $SNAP_INSTANCE_NAME.iscsid +snapctl stop --disable $SNAP_INSTANCE_NAME.target + +# No meaningful default backend is available yet. +snapctl stop --disable $SNAP_INSTANCE_NAME.cinder-backup +# Will only be enabled based on the answers during initialization. +snapctl stop --disable $SNAP_INSTANCE_NAME.setup-lvm-loopdev +# Will only be enabled if a backend is chosen to be configured by the user. +snapctl stop --disable $SNAP_INSTANCE_NAME.cinder-volume + + mkdir -p $SNAP_DATA/lib/libvirt/images +mkdir -p ${SNAP_COMMON}/log/libvirt/qemu +# NOTE(dmitriis): there is currently no way to make sure this directory gets +# recreated on reboot which would normally be done via systemd-tmpfiles. +mkdir -p /run/lock/snap.$SNAP_INSTANCE_NAME + +# Copy TEMPLATE.qemu into the common directory. Libvirt generates additional +# policy dynamically which is why its apparmor directory is writeable under $SNAP_COMMON. +# Also copy other abstractions that are used by this template. +rsync -rh $SNAP/etc/apparmor.d $SNAP_COMMON/etc # MySQL snapshot for speedy install # snapshot is a mysql data dir with @@ -24,4 +61,32 @@ done # Make a place for our horizon config overrides to live mkdir -p ${SNAP_COMMON}/etc/horizon/local_settings.d +# ----- OVN ----- +# Lay out directories used for OVN configuration and persistent data +for dir in etc/ovn var/lib/ovn var/log/ovn var/run/ovn; do + if [ ! -d $SNAP_COMMON/$dir ]; then + mkdir -p $SNAP_COMMON/$dir + fi +done + +# Prepare access to the hosting systems Open vSwitch instance +# NOTE end user must execute `snap connect ovn:openvswitch` for this to work +ln -s /var/run/openvswitch $SNAP_COMMON/var/run/openvswitch + +# The `ovn-ctl` script does not have enough knobs for useful tailoring of +# execution of the `ovn-northd` daemon. Instead it provides a file to pass +# arguments directly to the `ovn-northd` process. +# +# We fill the `args_northd` with necessary defaults and link to the file +# `ovn-ctl` looks for. +# +# For other daemons the corrensponding args_* file is used to pass arguments to +# `ovn-ctl`. +cat << EOF > $SNAP_COMMON/args_northd +--ovnnb-db=unix:$SNAP_COMMON/run/ovn/ovnnb_db.sock +--ovnsb-db=unix:$SNAP_COMMON/run/ovn/ovnsb_db.sock +EOF +ln -s $SNAP_COMMON/args_northd $SNAP_COMMON/etc/ovn/ovn-northd-db-params.conf +# ----- END OVN ----- + snap-openstack setup # Sets up templates for the first time. diff --git a/snap/hooks/post-refresh b/snap/hooks/post-refresh index 017d2c4..dc9d8b5 100755 --- a/snap/hooks/post-refresh +++ b/snap/hooks/post-refresh @@ -1,6 +1,9 @@ #!/bin/bash set -ex +# Refresh the TEMPLATE.qemu apparmor profile and abstractions. +rsync -rh $SNAP/etc/apparmor.d $SNAP_COMMON/etc + if [ -z "$(snapctl get config)" ]; then # [2019-11-15] Handle build 171 (beta) -> 182 @@ -30,4 +33,7 @@ if [ -z "$(snapctl get config.network.ports.rabbit)" ]; then snapctl set config.network.ports.rabbit=5672 fi +mkdir -p ${SNAP_COMMON}/log/libvirt/qemu +mkdir -p /run/lock/snap.$SNAP_INSTANCE_NAME + snap-openstack setup # Write any template changes. diff --git a/snap/hooks/remove b/snap/hooks/remove new file mode 100755 index 0000000..9a0c29b --- /dev/null +++ b/snap/hooks/remove @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 + +import os +import sys +import logging +from subprocess import check_call, check_output, call, run +from pyroute2 import netns + +logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) + + +if __name__ == '__main__': + # Work around the lack of modified LD_LIBRARY_PATH and PATH variables with + # snap-specific content. + snap_dir = os.environ['SNAP'] + snap_libs = (f'{snap_dir}/lib:{snap_dir}/usr/lib:' + f'{snap_dir}/lib/x86_64-linux-gnu:' + f'{snap_dir}/usr/lib/x86_64-linux-gnu') + os.environ['LD_LIBRARY_PATH'] = snap_libs + + check_call(['snapctl', 'start', 'microstack.ovsdb-server']) + check_call(['snapctl', 'start', 'microstack.ovs-vswitchd']) + + logging.info('Attempting to remove br-ex.') + check_call(['ovs-vsctl', '--if-exists', 'del-br', 'br-ex']) + + check_call(['snapctl', 'stop', 'microstack.ovsdb-server']) + check_call(['snapctl', 'stop', 'microstack.ovs-vswitchd']) + + for ns in netns.listnetns(): + if ns.startswith('ovnmeta-'): + logging.info(f'Removing the {ns} network namespace.') + netns.remove(ns) + + # Need to expose targets prior to starting iscsid in order to properly log out + # of iSCSI sessions. + check_call(['snapctl', 'start', 'microstack.target']) + check_call(['snapctl', 'start', 'microstack.iscsid']) + check_call(['sync']) + + # Assuming the QEMU processes have already been killed by snapd, + # log out of all targets prior to removing the snap to clean up + # the kernel state. + # TODO: be more selective about logging out since there may be sessions + # unrelated to MicroStack in the kernel. + # TODO: also clean up block devices by writing to + # /sys/class/block//device/delete since those do not get deleted on + # session logout. + logging.info('Attempting to remove iscsi sessions from the kernel.') + res = run(['iscsiadm', '-m', 'node', '-u']) + # ISCSI_ERR_NO_OBJS_FOUND + if res.returncode == 21: + logging.debug('No iscsi sessions were found.') + elif res.returncode == 0: + logging.debug('Successfully logged the existing iscsi sessions out.') + else: + # Albeit this is an error condition we cannot do much in the remove + # hook to fix this besides logging since snapd does not stop the + # snap removal on error in the remove hook. + logging.error('Unexpected error code received from iscsiadm: ' + f'{res.returncode}') + check_call(['snapctl', 'stop', 'microstack.iscsid']) + check_call(['snapctl', 'stop', 'microstack.target']) + + # File-backed LVM resource cleanup (if present). + loop_file = f'{os.environ["SNAP_COMMON"]}/cinder-lvm.img' + allocated_loop_dev = check_output( + f'losetup -j {loop_file} | cut -d ":" -f 1', shell=True + ).decode('utf-8').strip() + + if allocated_loop_dev: + cinder_lvm_vg = check_output([ + 'snapctl', 'get', 'config.cinder.lvm-backend-volume-group'] + ).strip() + if not call(['vgdisplay', cinder_lvm_vg]): + check_call(['vgremove', '-f', cinder_lvm_vg]) + if not call(['pvdisplay', allocated_loop_dev]): + check_call(['pvremove', '-f', allocated_loop_dev]) + check_call(['losetup', '-d', allocated_loop_dev]) diff --git a/snapcraft.yaml b/snapcraft.yaml index 14a5077..3735b13 100644 --- a/snapcraft.yaml +++ b/snapcraft.yaml @@ -1,6 +1,6 @@ name: microstack -version: stein -base: core18 +version: ussuri +base: core20 summary: OpenStack on your laptop. description: | Microstack gives you an easy way to develop and test OpenStack @@ -9,14 +9,20 @@ grade: devel confinement: strict environment: LC_ALL: C - PATH: $SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$PATH - OS_PLACEMENT_CONFIG_DIR: $SNAP/etc/nova/ + PATH: $SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$SNAP/usr/local/bin:$SNAP/usr/local/sbin:$PATH + # OS_PLACEMENT_CONFIG_DIR: $SNAP/etc/nova/ + # Standard library components must have priority in module name resolution: https://storyboard.openstack.org/#!/story/2007806 + PYTHONPATH: $PYTHONPATH:/usr/lib/python3.8:/usr/lib/python3/dist-packages:$SNAP/usr/lib/python3.8:$SNAP/lib/python3.8/site-packages:$SNAP/usr/lib/python3/dist-packages + PYTHONWARNINGS: ignore + # Needed by a targetcli dependency. + GI_TYPELIB_PATH: $SNAP/usr/lib/x86_64-linux-gnu/girepository-1.0 system-usernames: snap_daemon: shared layout: - # Libvirt/Qemu libs + /usr/local/bin: + bind: $SNAP/usr/local/bin /usr/lib/$SNAPCRAFT_ARCH_TRIPLET/ceph: bind: $SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/ceph /usr/lib/$SNAPCRAFT_ARCH_TRIPLET/qemu: @@ -32,17 +38,38 @@ layout: # Keystone really wants to look here for its fernet keys for some reason /etc/keystone/fernet-keys: bind: $SNAP_COMMON/fernet-keys - + # Libvirt has a hard-coded path for the apparmor directory. + /etc/apparmor.d: + bind: $SNAP_COMMON/etc/apparmor.d + # os-brick has a hard-coded location for the iscsi initiator name file. + /etc/iscsi: + bind: $SNAP_COMMON/etc/iscsi + # NOTE: /var/target is there since vhost-scsi module is a holder of target-core-mod (see + # the content of /sys/module/target_core_mod/holders). This prevents rtslib from being + # able to change the value of /sys/kernel/config/target/dbroot to /etc/target. At the same time, + # rtslib saves configuration to /etc/target/saveconfig.json by default. Hence both bind mounts + # point to the same target directory. + # LIO: the preferred db_root directory documented both in the kernel target code and rtslib. + /etc/target: + bind: $SNAP_COMMON/etc/target + # LIO: the default db_root directory documented both in the kernel target code and rtslib. + /var/target: + bind: $SNAP_COMMON/etc/target + # LIO: Debian targetcli and rtslib carry a patch that changes the preferred dbroot from /etc/target + # to /etc/rtslib-fb-target. + /etc/rtslib-fb-target: + bind: $SNAP_COMMON/etc/target apps: # Openstack openstack: - command: openstack-wrapper openstack + command: bin/openstack-wrapper openstack plugs: - network + # TODO: - microstack-support # OpenStack Service Configuration init: - command: microstack_init + command: bin/microstack_init plugs: - network - mount-observe # rabbitmq-ctl @@ -52,53 +79,49 @@ apps: - ssh-keys # write ssh key - system-observe # rabbitmq ? - hardware-observe # rabbitmq ? - - remove: - command: microstack_remove + # TODO: - microstack-support # Keystone keystone-uwsgi: - command: snap-openstack launch keystone-uwsgi + command: bin/snap-openstack launch keystone-uwsgi daemon: simple plugs: - network-bind - network - network-control + # TODO: - microstack-support keystone-manage: - command: snap-openstack launch keystone-manage + command: bin/snap-openstack launch keystone-manage plugs: - network + # TODO: - microstack-support - # Nova - nova-uwsgi: - command: snap-openstack launch nova-uwsgi - daemon: simple - plugs: - - network - - network-bind - - network-control nova-api: - command: snap-openstack launch nova-api-os-compute + command: bin/snap-openstack launch nova-api-os-compute daemon: simple plugs: - network - network-bind - network-control + # TODO: - microstack-support nova-conductor: - command: snap-openstack launch nova-conductor + command: bin/snap-openstack launch nova-conductor daemon: simple plugs: - network - network-control + # TODO: - microstack-support nova-scheduler: - command: snap-openstack launch nova-scheduler + command: bin/snap-openstack launch nova-scheduler daemon: simple plugs: - network - network-bind - network-control + # TODO: - microstack-support nova-compute: - command: snap-openstack launch nova-compute + command: bin/snap-openstack launch nova-compute + after: [libvirtd] daemon: simple plugs: - network @@ -106,125 +129,139 @@ apps: - network-control - firewall-control - hardware-observe + # TODO: - microstack-support + # Needs access to block devices so that blockdev --flushbufs works. + - block-devices nova-api-metadata: - command: snap-openstack launch nova-api-metadata + command: bin/snap-openstack launch nova-api-metadata daemon: simple plugs: - network - network-bind - firewall-control + # TODO: - microstack-support nova-manage: - command: snap-openstack launch nova-manage + command: bin/snap-openstack launch nova-manage plugs: - network - + # TODO: - microstack-support + nova-spicehtml5proxy: + command: bin/snap-openstack launch nova-spicehtml5proxy + daemon: simple + plugs: + - network + - network-bind + - network-control + # TODO: - microstack-support # Neutron neutron-api: - command: snap-openstack launch neutron-server + command: bin/snap-openstack launch neutron-server daemon: simple plugs: - network - network-bind - network-control - neutron-openvswitch-agent: - command: snap-openstack launch neutron-openvswitch-agent + # TODO: - microstack-support + neutron-ovn-metadata-agent: + command: bin/snap-openstack launch neutron-ovn-metadata-agent daemon: simple plugs: - network - network-bind - network-control - network-observe - - firewall-control - - process-control - - system-observe - neutron-l3-agent: - command: snap-openstack launch neutron-l3-agent - daemon: simple - plugs: - - network - - network-bind - - network-control - - network-observe - - firewall-control - - process-control - - system-observe - neutron-dhcp-agent: - command: snap-openstack launch neutron-dhcp-agent - daemon: simple - plugs: - - network - - network-bind - - network-control - - network-observe - - process-control - - system-observe - neutron-metadata-agent: - command: snap-openstack launch neutron-metadata-agent - daemon: simple - plugs: - - network - - network-bind - - network-control + # TODO: - microstack-support neutron-ovs-cleanup: - command: snap-openstack launch neutron-ovs-cleanup + command: bin/snap-openstack launch neutron-ovs-cleanup plugs: - network - network-control + # TODO: - microstack-support + # NOTE(dmitriis): Upstream Neutron does not support cleaning up ovnmeta- namespaces + # as of Ussuri which is a bug. neutron-netns-cleanup: - command: snap-openstack launch neutron-netns-cleanup + command: bin/snap-openstack launch neutron-netns-cleanup plugs: - network - network-control + # TODO: - microstack-support neutron-db-manage: - command: snap-openstack launch neutron-db-manage + command: bin/snap-openstack launch neutron-db-manage plugs: - network + # TODO: - microstack-support # Glance glance-api: - command: snap-openstack launch glance-api + command: bin/snap-openstack launch glance-api daemon: simple plugs: - network - network-bind - mount-observe - network-control + # TODO: - microstack-support registry: - command: snap-openstack launch glance-registry + command: bin/snap-openstack launch glance-registry daemon: simple plugs: - network - network-bind - network-control + # TODO: - microstack-support glance-manage: - command: snap-openstack launch glance-manage + command: bin/snap-openstack launch glance-manage plugs: - network + # TODO: - microstack-support + + # Placement + placement-uwsgi: + command: bin/snap-openstack launch placement-uwsgi + daemon: simple + plugs: + - network + - network-bind + - mount-observe + - network-control + # TODO: - microstack-support + + placement-manage: + command: bin/snap-openstack launch placement-manage + plugs: + - network + # TODO: - microstack-support # Openstack Shared Services nginx: - command: snap-openstack launch nginx + command: bin/snap-openstack launch nginx daemon: forking plugs: - network - network-bind + # TODO: - microstack-support # Openvswitch ovs-vswitchd: - command: ovs-wrapper $SNAP/share/openvswitch/scripts/ovs-ctl --no-ovsdb-server --no-monitor --system-id=random start - stop-command: ovs-wrapper $SNAP/share/openvswitch/scripts/ovs-ctl --no-ovsdb-server stop + command: bin/ovs-wrapper $SNAP/usr/local/share/openvswitch/scripts/ovs-ctl --no-ovsdb-server --no-monitor start + stop-command: bin/ovs-wrapper $SNAP/usr/local/share/openvswitch/scripts/ovs-ctl --no-ovsdb-server stop after: [ovsdb-server] daemon: forking plugs: - network - network-bind - network-control + # NETLINK_NETFILTER SOCK_RAW sockets + - firewall-control - openvswitch-support - process-control - system-trace + - hardware-observe + - hugepages-control + # TODO: - microstack-support ovsdb-server: - command: ovs-wrapper $SNAP/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd --no-monitor --system-id=random start - stop-command: ovs-wrapper $SNAP/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd stop + command: bin/ovs-wrapper $SNAP/usr/local/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd --no-monitor start + stop-command: bin/ovs-wrapper $SNAP/usr/local/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd stop daemon: forking plugs: - network @@ -233,30 +270,128 @@ apps: - openvswitch-support - process-control - system-trace + # TODO: - microstack-support + ovn-ovsdb-server-nb: + command: bin/ovn-wrapper $SNAP/usr/local/share/ovn/scripts/ovn-ctl start_nb_ovsdb + stop-command: bin/ovn-wrapper $SNAP/usr/local/share/ovn/scripts/ovn-ctl stop_nb_ovsdb + daemon: forking + plugs: + - network + - network-bind + - network-control + - openvswitch-support + - process-control + - system-trace + # TODO: - microstack-support + ovn-ovsdb-server-sb: + command: bin/ovn-wrapper $SNAP/usr/local/share/ovn/scripts/ovn-ctl start_sb_ovsdb + stop-command: bin/ovn-wrapper $SNAP/usr/local/share/ovn/scripts/ovn-ctl stop_sb_ovsdb + daemon: forking + plugs: + - network + - network-bind + - network-control + - openvswitch-support + - process-control + - system-trace + # TODO: - microstack-support + ovn-northd: + command: bin/ovn-wrapper $SNAP/usr/local/share/ovn/scripts/ovn-ctl start_northd + stop-command: bin/ovn-wrapper $SNAP/usr/local/share/ovn/scripts/ovn-ctl stop_northd + daemon: forking + plugs: + - network + - network-bind + - network-control + - openvswitch-support + - process-control + - system-trace + # TODO: - microstack-support + after: + - ovn-ovsdb-server-nb + - ovn-ovsdb-server-sb + ovn-controller: + command: bin/ovn-wrapper $SNAP/usr/local/share/ovn/scripts/ovn-ctl start_controller + stop-command: bin/ovn-wrapper $SNAP/usr/local/share/ovn/scripts/ovn-ctl stop_controller + daemon: forking + plugs: + - network + - network-bind + - network-control + - openvswitch-support + - process-control + - system-trace + # TODO: - microstack-support + after: + - ovsdb-server + - ovs-vswitchd + + ovn-appctl: + command: bin/ovn-wrapper $SNAP/usr/local/bin/ovn-appctl + plugs: + - network + - network-bind + # TODO: - microstack-support + ovn-trace: + command: bin/ovn-wrapper $SNAP/usr/local/bin/ovn-trace + plugs: + - network + - network-bind + # TODO: - microstack-support + ovn-detrace: + command: bin/ovn-wrapper $SNAP/usr/local/bin/ovn-detrace + plugs: + - network + - network-bind + # TODO: - microstack-support + ovn-nbctl: + command: bin/ovn-wrapper $SNAP/usr/local/bin/ovn-nbctl + plugs: + - network + - network-bind + # TODO: - microstack-support + ovn-sbctl: + command: bin/ovn-wrapper $SNAP/usr/local/bin/ovn-sbctl + plugs: + - network + - network-bind + - process-control + # TODO: - microstack-support + ovs-vsctl: - command: ovs-wrapper bin/ovs-vsctl + command: bin/ovs-wrapper $SNAP/usr/local/bin/ovs-vsctl plugs: - network + - process-control + # TODO: - microstack-support ovs-appctl: - command: ovs-wrapper bin/ovs-appctl + command: bin/ovs-wrapper $SNAP/usr/local/bin/ovs-appctl plugs: - network + - process-control + # TODO: - microstack-support ovs-ofctl: - command: ovs-wrapper bin/ovs-ofctl + command: bin/ovs-wrapper $SNAP/usr/local/bin/ovs-ofctl plugs: - network + - process-control + # TODO: - microstack-support ovs-dpctl: - command: ovs-wrapper bin/ovs-dpctl + command: bin/ovs-wrapper $SNAP/usr/local/bin/ovs-dpctl plugs: - network + - process-control + # TODO: - microstack-support external-bridge: - command: wait-on-init setup-br-ex + command: bin/wait-on-init setup-br-ex daemon: oneshot after: [ovs-vswitchd] plugs: - network - network-control + - process-control + # TODO: - microstack-support # Libvirt/Qemu libvirtd: @@ -272,133 +407,210 @@ apps: - opengl - raw-usb - hardware-observe + - hugepages-control - kvm - kernel-module-observe + - process-control + - mount-observe + - block-devices + # TODO: - microstack-support + # Make sure KillMode=process is set in the resulting systemd unit instead of + # the default KillMode=control-group to avoid the situation where stopping + # libvirtd causes the forced power off scenario for all VMs; it also affects + # snap refreshes as libvirtd is restarted in this case. + # Removing the snap will still result in VMs being forcefully powered off which + # is intended. + stop-mode: sigterm virtlogd: - command: virtlogd --pid $SNAP_DATA/virtlogd.pid + command: usr/sbin/virtlogd --pid $SNAP_DATA/virtlogd.pid daemon: simple plugs: - network - network-bind - network-control + # TODO: - microstack-support virsh: - command: virsh + command: usr/bin/virsh + plugs: + - network + # TODO: - microstack-support # MySQL mysqld: - command: mysql-start-server + command: bin/mysql-start-server daemon: simple plugs: - process-control - network - network-bind + # TODO: - microstack-support mysql: - command: mysql-start-client + command: bin/mysql-start-client plugs: - process-control - network + # TODO: - microstack-support # RabbitMQ rabbitmq-server: - command: rabbitmq-server + command: usr/sbin/rabbitmq-server daemon: simple plugs: - network-bind - network - mount-observe - log-observe + # TODO: - microstack-support environment: HOME: $SNAP_COMMON/lib/rabbitmq rabbitmqctl: - command: rabbitmqctl + command: usr/sbin/rabbitmqctl plugs: - network - mount-observe + # TODO: - microstack-support environment: HOME: $SNAP_COMMON/lib/rabbitmq rabbitmq-plugins: - command: rabbitmq-plugins + command: usr/sbin/rabbitmq-plugins environment: HOME: $SNAP_COMMON/lib/rabbitmq + plugs: [] + # TODO: - microstack-support # Memcached memcached: - command: memcached -u root -v + command: usr/bin/memcached -u root -v daemon: simple plugs: - network - network-bind + # TODO: - microstack-support # Cinder cinder-uwsgi: - command: snap-openstack launch cinder-uwsgi + command: bin/snap-openstack launch cinder-uwsgi daemon: simple plugs: - network - network-bind + # TODO: - microstack-support cinder-backup: - command: snap-openstack launch cinder-backup + command: bin/snap-openstack launch cinder-backup daemon: simple plugs: - network - mount-observe + # TODO: - microstack-support cinder-manage: - command: snap-openstack launch cinder-manage + command: bin/snap-openstack launch cinder-manage plugs: - network - mount-observe + # TODO: - microstack-support cinder-scheduler: - command: snap-openstack launch cinder-scheduler + command: bin/snap-openstack launch cinder-scheduler daemon: simple plugs: - network - mount-observe + # TODO: - microstack-support cinder-volume: - command: snap-openstack launch cinder-volume + command: bin/snap-openstack launch cinder-volume daemon: simple plugs: - network + - network-bind - mount-observe + - block-devices + # setpriority by LVM tools. + - process-control + # TODO: - microstack-support + after: + - setup-lvm-loopdev + + # Restores LIO kernel target configuration. + target: + command: bin/target-start + stop-command: bin/target-stop + daemon: oneshot + plugs: + - network + - network-bind + - mount-observe + - block-devices + # TODO: - microstack-support + after: + - setup-lvm-loopdev + environment: + TARGETCLI_HOME: $SNAP_USER_COMMON/.targetcli + + iscsid: + command: bin/iscsid-start + daemon: forking + plugs: + - network + - network-bind + # NETLINK_ISCSI protocol sockets + - network-control + # TODO: - microstack-support + - block-devices + - process-control + after: + # Make sure iscsid is stopped before target configuration is cleaned up. + - target + + # TODO: this is a --devmode workaround since the microstack-support interface is not merged yet. + load-modules: + command: bin/load-modules + daemon: oneshot + plugs: + - kernel-module-control + + setup-lvm-loopdev: + command: bin/setup-lvm-loopdev + daemon: oneshot + plugs: + - network + - block-devices + - mount-observe + # loop device control etc. + # LVM tools use setpriority to set it to -18 which is prohibited in the default snapd templates. + - process-control + # TODO: - microstack-support + after: + - load-modules # Horizon horizon-uwsgi: - command: snap-openstack launch horizon-uwsgi + command: bin/snap-openstack launch horizon-uwsgi daemon: simple plugs: - network - network-bind + # TODO: - microstack-support # Utility to launch a vm. Creates security groups, floating ips, # and other necessities as well. launch: - command: microstack_launch + command: bin/microstack_launch plugs: - network - network-control # could also be mount-observe + # TODO: - microstack-support # Cluster cluster-server: - command: flask run -p 10002 --host 0.0.0.0 # TODO: run as a uwsgi app + command: bin/flask run -p 10002 --host 0.0.0.0 # TODO: run as a uwsgi app daemon: simple environment: LC_ALL: C.UTF-8 # Makes flask happy LANG: C.UTF-8 # Makes flask happy - FLASK_APP: $SNAP/lib/python3.6/site-packages/cluster/daemon.py + FLASK_APP: $SNAP/lib/python3.8/site-packages/cluster/daemon.py plugs: - network - network-bind - - join: - command: python3 $SNAP/lib/python3.6/site-packages/cluster/client.py - - - filebeat: - # this is to avoid conflict with filebeat package - command: bin/filebeat - daemon: simple - - nrpe: command: bin/nrpe daemon: simple @@ -406,6 +618,15 @@ apps: command: bin/telegraf daemon: simple + filebeat: + # this is to avoid conflict with filebeat package + command: bin/filebeat + daemon: simple + + nrpe: + command: bin/nrpe + daemon: simple + parts: # Add Ubuntu Cloud Archive sources. # Allows us to fetch things such as updated libvirt. @@ -415,33 +636,63 @@ parts: - ubuntu-cloud-keyring - software-properties-common - dpkg-dev # We'll need to manipulate sources later. + #stage-packages: + # The two dependencies below are added to avoid errors during + # the build step of the openstack-projects part (see LP: #1864589). + - python3-dev + - python3-distutils override-build: | - add-apt-repository -s cloud-archive:stein - apt-key update - apt update + # add-apt-repository -s cloud-archive:ussuri + # apt-key update + # apt update snapcraftctl build # OpenStack Python Projects, keyed off Keystone openstack-projects: plugin: python - python-version: python3 + # python-version: python3 after: - uca-sources - libvirt - patches constraints: - ${SNAPCRAFT_STAGE}/patches/upper-constraints.txt - source: http://tarballs.openstack.org/keystone/keystone-stable-stein.tar.gz + source: http://tarballs.openstack.org/keystone/keystone-stable-ussuri.tar.gz + build-packages: + - gcc + - git + - libffi-dev + - libssl-dev + - libxml2-dev + - libxslt1-dev + - python3-dev + - libsystemd-dev + stage-packages: + - procps # sysctl utility is used by Neutron agents. + - conntrack + - coreutils + - haproxy + - libmagic1 + - petname + - libsystemd0 + - libpython3.8 + - spice-html5 + - sg3-utils + # LIO userspace dependencies + # Will also pull rtslib-fb to work with ConfigFS. + - targetcli-fb python-packages: + - wheel - libvirt-python - oslo.cache[dogpile] - pymysql - uwsgi - git+https://opendev.org/x/snap.openstack#egg=snap.openstack - - http://tarballs.openstack.org/nova/nova-stable-stein.tar.gz + - http://tarballs.openstack.org/nova/nova-stable-ussuri.tar.gz - neutron - - http://tarballs.openstack.org/glance/glance-stable-stein.tar.gz - - http://tarballs.openstack.org/cinder/cinder-stable-stein.tar.gz + - https://tarballs.opendev.org/openstack/glance/glance-stable-ussuri.tar.gz + - https://tarballs.opendev.org/openstack/cinder/cinder-stable-ussuri.tar.gz + - https://tarballs.opendev.org/openstack/placement/placement-stable-ussuri.tar.gz - horizon - python-cinderclient - python-openstackclient @@ -451,20 +702,15 @@ parts: # above. We include it here so that we don't try to rebuild dist # stuff: - petname - build-packages: - - gcc - - git - - libffi-dev - - libssl-dev - - libxml2-dev - - libxslt1-dev - stage-packages: - - conntrack - - coreutils - - haproxy - - libmagic1 - - python3-dev - - python3-systemd + - systemd-python + # Used by spice-html5 proxy. + - websockify + # For debugging purposes. + - rpdb + build-environment: &python-build-environment + - PIP_COMPILE: 'false' # disable .pyc generation in pip + - PYTHONDONTWRITEBYTECODE: 'false' # disable .pyc generation by setup.py + - SOURCE_DATE_EPOCH: '1591640328' override-prime: | snapcraftctl prime # Patch Openstack Dashboard (horizon) settings files. @@ -479,16 +725,39 @@ parts: done # Now that everything is built, let's disable user site-packages # as stated in PEP-0370 - sed -i usr/lib/python3.6/site.py -e 's/^ENABLE_USER_SITE = None$/ENABLE_USER_SITE = False/' + # sed -i usr/lib/python3.8/site.py -e 's/^ENABLE_USER_SITE = None$/ENABLE_USER_SITE = False/' # Uncomment the below line for better python-libvirt debugging output. # patch --batch --forward --strip 1 --input ../project/patches/nova/nova_log_error.patch organize: - lib/python3.6/site-packages/openstack_dashboard/local/local_settings.py.example: lib/python3.6/site-packages/openstack_dashboard/local/local_settings.py + lib/python3.8/site-packages/openstack_dashboard/local/local_settings.py.example: lib/python3.8/site-packages/openstack_dashboard/local/local_settings.py + override-build: | + snapcraftctl build + rm $SNAPCRAFT_PART_INSTALL/pyvenv.cfg + --strip 1 \ + --input "$patch" + echo + done + # Now that everything is built, let's disable user site-packages + # as stated in PEP-0370 + # sed -i usr/lib/python3.8/site.py -e 's/^ENABLE_USER_SITE = None$/ENABLE_USER_SITE = False/' + # Uncomment the below line for better python-libvirt debugging output. + # patch --batch --forward --strip 1 --input ../project/patches/nova/nova_log_error.patch + organize: + lib/python3.8/site-packages/openstack_dashboard/local/local_settings.py.example: lib/python3.8/site-packages/openstack_dashboard/local/local_settings.py + override-build: | + snapcraftctl build + rm $SNAPCRAFT_PART_INSTALL/pyvenv.cfg + rm $SNAPCRAFT_PART_INSTALL/bin/activate + rm $SNAPCRAFT_PART_INSTALL/bin/activate.csh + rm $SNAPCRAFT_PART_INSTALL/bin/activate.fish + # see LP: #1882994 + rm $SNAPCRAFT_PART_INSTALL/bin/python3 + rm $SNAPCRAFT_PART_INSTALL/bin/python keystone-config: after: [openstack-projects] plugin: dump - source: http://tarballs.openstack.org/keystone/keystone-stable-stein.tar.gz + source: http://tarballs.openstack.org/keystone/keystone-stable-ussuri.tar.gz organize: etc/*.conf: etc/keystone/ etc/*.ini: etc/keystone/ @@ -518,7 +787,7 @@ parts: nova-config: after: [openstack-projects] plugin: dump - source: http://tarballs.openstack.org/nova/nova-stable-stein.tar.gz + source: http://tarballs.openstack.org/nova/nova-stable-ussuri.tar.gz filesets: etc: - etc/nova/* @@ -528,7 +797,7 @@ parts: glance-config: after: [openstack-projects] plugin: dump - source: http://tarballs.openstack.org/glance/glance-stable-stein.tar.gz + source: http://tarballs.openstack.org/glance/glance-stable-ussuri.tar.gz organize: etc/*.conf: etc/glance/ etc/*.ini: etc/glance/ @@ -539,10 +808,24 @@ parts: stage: [$etc] prime: [$etc] + placement-config: + after: [openstack-projects] + plugin: dump + source: http://tarballs.openstack.org/placement/placement-stable-ussuri.tar.gz + organize: + etc/*.conf: etc/placement/ + etc/*.ini: etc/placement/ + etc/*.json: etc/placement/ + filesets: + etc: + - etc/placement/* + stage: [$etc] + prime: [$etc] + cinder-config: after: [openstack-projects] plugin: dump - source: http://tarballs.openstack.org/cinder/cinder-stable-stein.tar.gz + source: http://tarballs.openstack.org/cinder/cinder-stable-ussuri.tar.gz organize: etc/*.conf: etc/cinder/ etc/*.ini: etc/cinder/ @@ -557,7 +840,7 @@ parts: horizon-config: after: [openstack-projects] plugin: dump - source: http://tarballs.openstack.org/horizon/horizon-stable-stein.tar.gz + source: http://tarballs.openstack.org/horizon/horizon-stable-ussuri.tar.gz organize: etc/*.conf: etc/horizon/ filesets: @@ -576,40 +859,63 @@ parts: - curl override-build: | # Create ${SNAPCRAFT_STAGE}/patches/upper-constraints - # Fetch upstream file - curl https://raw.githubusercontent.com/openstack/requirements/stable/stein/upper-constraints.txt --output ./upper-constraints.txt - - # Fix https://bugs.launchpad.net/microstack/+bug/1867360 - sed -e 's/pyScss===1.3.4/git+https:\/\/github.com\/petevg\/pyScss.git#egg=pyScss/' -i ./upper-constraints.txt - + curl https://raw.githubusercontent.com/openstack/requirements/stable/ussuri/upper-constraints.txt --output ./upper-constraints.txt snapcraftctl build organize: '*' : patches/ - 'usr/sbin/dnsmasq': patches/neutron/dnsmasq prime: - -* openvswitch: - source: http://openvswitch.org/releases/openvswitch-2.11.1.tar.gz + source: https://github.com/openvswitch/ovs/archive/v2.13.0.tar.gz plugin: autotools build-packages: - - libssl-dev - - try: [libnuma-dev] + - autoconf + - binutils + - bsdmainutils + - build-essential + - gettext + - git + - libarchive-zip-perl - libcap-ng-dev - libpcap-dev + - on arm64: + - libnuma-dev + - on ppc64el: + - libnuma-dev + - on amd64: + - libnuma-dev + - libssl-dev + - libtimedate-perl + - libtool - libunbound-dev - - python-all - - python-six + - pkg-config + - python-all-dev - python-setuptools + - python3-all-dev + - python3-setuptools + - python3-sphinx + - sphinx-common + - zlib1g-dev stage-packages: + - libevent-2.1-7 + - kmod + - libcap-ng0 + - libssl1.1 + - libunbound8 + - netbase + - ethtool + - procps - uuid-runtime - - libunbound2 - configflags: + autotools-configure-parameters: - "--localstatedir=/var/snap/$SNAPCRAFT_PROJECT_NAME/common" - "--sysconfdir=/var/snap/$SNAPCRAFT_PROJECT_NAME/etc" after: - patches + override-build: | + snapcraftctl build + tar -cvzf $SNAPCRAFT_STAGE/openvswitch.tar.gz . ovs-wrapper: source: ./snap-wrappers/ovs @@ -618,13 +924,62 @@ parts: - openvswitch organize: ovs-wrapper: bin/ovs-wrapper + # Nova os-vif code makes libvirt plug a VM interface into an integration bridge + # which relies on ovs-vsctl execution. Since we use modified paths, some environment + # variables need to be set. + ovs-vsctl: bin/ovs-vsctl + + ovn: + after: [openvswitch, patches] + source: https://github.com/ovn-org/ovn/archive/v20.03.0.tar.gz + plugin: autotools + autotools-configure-parameters: + - --sysconfdir=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/etc + - --localstatedir=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/var + - --with-ovs-source=$SNAPCRAFT_PART_BUILD/openvswitch + override-pull: | + snapcraftctl pull + rm -rf openvswitch + mkdir openvswitch + tar -C openvswitch -xvzf $SNAPCRAFT_STAGE/openvswitch.tar.gz + rm -f $SNAPCRAFT_STAGE/openvswitch.tar.gz + override-build: | + snapcraftctl build + build-packages: + - autoconf + - binutils + - build-essential + - git + - libssl-dev + - libtool + - python3-all-dev + - sphinx-common + stage-packages: + - libevent-2.1-7 + - libunbound8 + organize: + snap/$SNAPCRAFT_PROJECT_NAME/current/share: /share + filesets: + exclude-duplicate-manpages: + - -share/man/man*/ovsdb* + - -usr/local/share/man/man*/ovsdb* + stage: + - $exclude-duplicate-manpages + prime: + - $exclude-duplicate-manpages + + ovn-wrapper: + source: ./snap-wrappers/ovn + plugin: dump + after: + - ovn + organize: + ovn-wrapper: bin/ovn-wrapper nginx: - source: http://www.nginx.org/download/nginx-1.13.0.tar.gz + source: http://www.nginx.org/download/nginx-1.19.0.tar.gz plugin: autotools - after: - - patches - configflags: + autotools-configure-parameters: - --prefix=/usr - "--http-log-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/log/nginx-access.log" - "--error-log-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/log/nginx-error.log" @@ -653,44 +1008,12 @@ parts: done snapcraftctl build - nginx: - source: http://www.nginx.org/download/nginx-1.13.0.tar.gz - plugin: autotools - after: - - patches - configflags: - - --prefix=/usr - - "--http-log-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/log/nginx-access.log" - - "--error-log-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/log/nginx-error.log" - - "--lock-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/lock/nginx.lock" - - "--pid-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/run/nginx.pid" - - "--http-client-body-temp-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/lib/nginx_client_body" - - "--http-proxy-temp-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/lib/nginx_proxy" - - "--http-fastcgi-temp-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/lib/nginx_fastcgi" - - "--http-uwsgi-temp-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/lib/nginx_uwsgi" - - "--http-scgi-temp-path=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/lib/nginx_scgi" - - --with-http_ssl_module - build-packages: - - libpcre3-dev - - libssl-dev - - python-six - override-build: | - # Apply patches - for patch in ${SNAPCRAFT_STAGE}/patches/nginx/*.patch; do - echo "Applying $(basename "$patch") ..." - patch \ - --batch \ - --forward \ - --strip 1 \ - --input "$patch" - echo - done - snapcraftctl build # libvirt/qemu qemu: - source: . - source-subdir: qemu-3.1+dfsg + source: https://git.launchpad.net/ubuntu/+source/qemu + source-type: git + source-branch: ubuntu/focal-updates plugin: autotools after: - uca-sources @@ -701,16 +1024,15 @@ parts: - seabios - ipxe-qemu - freeglut3 # provides libglut.so.3 - - try: - - libnuma1 - - libspice-server1 + - libnuma1 + - libspice-server1 - libasound2 - libasyncns0 - libbluetooth3 - - libboost-iostreams1.62.0 - - libboost-random1.62.0 - - libboost-system1.62.0 - - libboost-thread1.62.0 + - libboost-iostreams1.67.0 + - libboost-random1.67.0 + - libboost-system1.67.0 + - libboost-thread1.67.0 - libcaca0 - libfdt1 - libflac8 @@ -737,9 +1059,25 @@ parts: - libxau6 - libxcb1 - libxdmcp6 - - libxen-dev - libxext6 - libpng16-16 + - libaio1 + - libasn1-8-heimdal + - libbrotli1 + - libcurl3-gnutls + - libgssapi3-heimdal + - libhcrypto4-heimdal + - libheimbase1-heimdal + - libheimntlm0-heimdal + - libhx509-5-heimdal + - libkrb5-26-heimdal + - libldap-2.4-2 + - libnghttp2-14 + - libpsl5 + - libroken18-heimdal + - librtmp1 + - libssh-4 + - libwind0-heimdal build-packages: - acpica-tools - libaio-dev @@ -772,7 +1110,7 @@ parts: - wget - dpkg-dev - gcc - configflags: + autotools-configure-parameters: - --disable-blobs - --prefix=/usr - --localstatedir=/var/snap/$SNAPCRAFT_PROJECT_NAME/common @@ -784,12 +1122,9 @@ parts: - --disable-vhost-user - --enable-system - --target-list=x86_64-softmmu + - --disable-xen override-build: | - # I think that snapcraft is creating this dir, which breaks the below. - rm -Rf qemu-3.1+dfsg - apt source qemu - # Fix issue with running apt source as root - dpkg-source --before-build qemu-3.1+dfsg + dpkg-source --before-build . snapcraftctl build kvm-support: @@ -798,8 +1133,10 @@ parts: - try: [msr-tools] libvirt: - source: . - source-subdir: libvirt-5.0.0 + source: https://git.launchpad.net/ubuntu/+source/libvirt + source-type: git + source-branch: ubuntu/focal-updates + source-subdir: build-subdir after: - qemu - uca-sources @@ -828,6 +1165,9 @@ parts: - wget - dpkg-dev - xsltproc + - libapparmor-dev + - rsync + - open-iscsi stage-packages: - dmidecode - dnsmasq @@ -839,7 +1179,12 @@ parts: - libcurl3-gnutls - libpcap0.8 - libpciaccess0 - configflags: + # libvirt uses the pm-is-supported tool. + - pm-utils + - ebtables + - apparmor + - libapparmor1 + autotools-configure-parameters: - --with-qemu - --without-bhyve - --without-xen @@ -858,32 +1203,71 @@ parts: - --without-storage-rbd - --without-storage-lvm - --without-selinux + - --with-chrdev-lock-files=/run/lock + - --with-storage-iscsi + # TODO(dmitriis): re-enable once a workaround is found for subprocess' RPATH handling. + # - --with-apparmor + - --without-apparmor + # NOTE: the install prefix will be different from the actual location on snap installation. - --prefix=/usr + - --bindir=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/bin + - --sbindir=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin + - --libexecdir=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/libexec + - --libdir=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/lib + - --includedir=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/include + - --oldincludedir=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/include - --localstatedir=/var/snap/$SNAPCRAFT_PROJECT_NAME/common - --sysconfdir=/var/snap/$SNAPCRAFT_PROJECT_NAME/common - DNSMASQ=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin/dnsmasq - DMIDECODE=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin/dmidecode - - OVSVSCTL=/snap/$SNAPCRAFT_PROJECT_NAME/current/bin/ovs-vsctl - - IPTABLES_PATH=/snap/$SNAPCRAFT_PROJECT_NAME/current/sbin/iptables - - IP6TABLES_PATH=/snap/$SNAPCRAFT_PROJECT_NAME/current/sbin/ip6tables + - OVSVSCTL=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/local/bin/ovs-vsctl + - IPTABLES_PATH=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin/iptables-legacy + - IP6TABLES_PATH=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin/ip6tables-legacy + - EBTABLES_PATH=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin/ebtables-legacy + build-environment: + # Libraries under /snap/$SNAPCRAFT_PROJECT_NAME/current/usr/lib/x86_64-linux-gnu are not added to the + # runpath by default. This is OK for parent processes which get LD_LIBRARY_PATH set properly but not + # for the child processes they spawn since the environment variables are not passed down to children by default after execve(2). + # `readelf -d /snap/microstack/current/usr/libexec/virt-aa-helper` should return something like: + # (RUNPATH) Library runpath: [/snap/microstack/current/usr/lib:/snap/microstack/current/usr/lib/x86_64-linux-gnu:...] + - LDFLAGS: '$LDFLAGS -Wl,-rpath=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/lib -Wl,-rpath=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/lib/x86_64-linux-gnu -Wl,-rpath=/snap/$SNAPCRAFT_PROJECT_NAME/current/lib -Wl,-rpath=/lib/x86_64-linux-gnu -Wl,-rpath=/lib/' override-build: | - # I think that snapcraft is creating this dir, which breaks the below. - rm -Rf libvirt-5.0.0 - apt source libvirt - # Fix issue with running apt source as root - dpkg-source --before-build libvirt-5.0.0 + dpkg-source --before-build . + logger `echo -n $PWD` # Prevent libvirt from attempting to run setgroups echo " #undef HAVE_SETGROUPS #undef HAVE_SETEUID - " >> libvirt-5.0.0/config-post.h + " >> config-post.h + # See https://bugs.launchpad.net/snapcraft/+bug/1882255 + mkdir build-subdir + cd build-subdir + ../autogen.sh + + echo "#!/bin/sh + exit 0 + " >> autogen.sh + chmod +x autogen.sh + ln -s ../configure configure + cd .. + # end of a workaround for LP: #1882255 # Build snapcraftctl build - organize: - 'usr/sbin/dnsmasq-orig': usr/sbin/dnsmasq + rsync --remove-source-files -arhvP $SNAPCRAFT_PART_INSTALL/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/* $SNAPCRAFT_PART_INSTALL/usr/ + rm -rf $SNAPCRAFT_PART_INSTALL/snap/$SNAPCRAFT_PROJECT_NAME/current/usr + + # Copy the generated apparmor template into the install dir + # it will be used by libvirt at runtime. + mkdir -p $SNAPCRAFT_PART_INSTALL/etc/apparmor.d/libvirt/ + mkdir -p $SNAPCRAFT_PART_INSTALL/etc/apparmor.d/abstractions + cp $SNAPCRAFT_PART_BUILD/src/security/apparmor/TEMPLATE.qemu $SNAPCRAFT_PART_INSTALL/etc/apparmor.d/libvirt/TEMPLATE.qemu + cp $SNAPCRAFT_PART_BUILD/src/security/apparmor/libvirt-qemu $SNAPCRAFT_PART_INSTALL/etc/apparmor.d/abstractions/libvirt-qemu + # While LXC is not used, the AppArmor code errors out if the template for LXC is not there. + cp $SNAPCRAFT_PART_BUILD/src/security/apparmor/TEMPLATE.lxc $SNAPCRAFT_PART_INSTALL/etc/apparmor.d/libvirt/TEMPLATE.lxc + cp $SNAPCRAFT_PART_BUILD/src/security/apparmor/libvirt-lxc $SNAPCRAFT_PART_INSTALL/etc/apparmor.d/abstractions/libvirt-lxc # MySQL mysql-server: plugin: dump @@ -905,11 +1289,11 @@ parts: rm -f $SNAPCRAFT_PART_INSTALL/usr/bin/erl rm -f $SNAPCRAFT_PART_INSTALL/usr/lib/rabbitmq/bin/rabbitmq-script-wrapper # NOTE(jamespage): The versioned path below is brittle. - rm -f $SNAPCRAFT_PART_INSTALL/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/sbin/rabbitmq-defaults + rm -f $SNAPCRAFT_PART_INSTALL/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.2/sbin/rabbitmq-defaults snapcraftctl build organize: rabbitmq-script-wrapper: usr/lib/rabbitmq/bin/rabbitmq-script-wrapper - rabbitmq-defaults: usr/lib/rabbitmq/lib/rabbitmq_server-3.6.10/sbin/rabbitmq-defaults + rabbitmq-defaults: usr/lib/rabbitmq/lib/rabbitmq_server-3.8.2/sbin/rabbitmq-defaults erl: usr/bin/erl # Memcached Token Caching @@ -920,12 +1304,101 @@ parts: # Cinder LVM backend lvm2: - source: http://www.sourceware.org/pub/lvm2/releases/LVM2.2.02.172.tgz + after: + - libvirt + source: https://git.launchpad.net/ubuntu/+source/lvm2 + source-type: git + source-branch: ubuntu/focal + build-packages: + - python3-pyudev + stage-packages: + - thin-provisioning-tools plugin: autotools - configflags: + autotools-configure-parameters: - --with-confdir=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/etc/ - --with-default-system-dir=/var/snap/$SNAPCRAFT_PROJECT_NAME/common/etc/ - build-attributes: [no-patchelf] + - --with-cache=internal + - --with-device-uid=0 + # Assumes that the "disk" group is present on the host and has 6 for its gid. + # This is common but not fully reliable not knowing the target distribution. + - --with-device-gid=6 + - --with-device-mode=0660 + - --with-default-pid-dir=/run + - --with-default-run-dir=/run/lvm + - --with-default-locking-dir=/run/lock/lvm + - --with-thin=internal + - --with-thin-check=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin/thin_check + - --with-thin-dump=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin/thin_dump + - --with-thin-repair=/snap/$SNAPCRAFT_PROJECT_NAME/current/usr/sbin/thin_repair + - --enable-applib + - --enable-blkid_wiping + - --enable-cmdlib + - --enable-dmeventd + - --enable-dbus-service + # - --enable-lvmlockd-dlm + # - --enable-lvmlockd-sanlock + # - --enable-lvmpolld + # - --enable-notify-dbus + - --enable-pkgconfig + - --enable-readline + # The two below are especially important to make sure lvm tools do not try to use + # mknod themselves to create block special files for device mapper devices and + # so that cookies are used to synchronize udev and LVM tools operations. + - --enable-udev_rules + - --enable-udev_sync + + open-iscsi: + after: + - patches + source: https://git.launchpad.net/ubuntu/+source/open-iscsi + source-type: git + source-branch: ubuntu/focal + build-packages: + - autotools-dev + - bison + - bzip2 + - debhelper + - dh-autoreconf + - dh-exec + - dh-systemd + - dpkg-dev + - flex + - libisns-dev + - libmount-dev + - po-debconf + stage-packages: + - libisns0 + plugin: dump + organize: + usr/iscsid: sbin/iscsid + usr/iscsiadm: sbin/iscsiadm + usr/iscsistart: sbin/iscsistart + utils/iscsi_discovery: sbin/iscsi_discovery + utils/iscsi-iname: sbin/iscsi-iname + filesets: + iscsi-binaries: + - sbin/* + iscsi-deps: + - lib/* + - usr/lib/* + stage: [$iscsi-binaries, $iscsi-deps] + prime: [$iscsi-binaries, $iscsi-deps] + override-build: | + # Distro patches. + dpkg-source --before-build . + # snap-specific patches. + for patch in ${SNAPCRAFT_STAGE}/patches/open-iscsi/*.patch; do + echo "Applying $(basename "$patch") ..." + patch \ + --batch \ + --forward \ + --strip 1 \ + --input "$patch" + echo + done + + dh build-arch --with autotools_dev,autoreconf,systemd + snapcraftctl build # Templates and other files to be overlaid over the snap. overlay: @@ -935,43 +1408,96 @@ parts: # Optionally interactive init script init: plugin: python - python-version: python3 + # python-version: python3 requirements: - - requirements.txt # Relative to source path, so tools/init/req...txt + - requirements.txt # Relative to source path, so tools/init/req...txt source: tools/init + stage-packages: + # note(dmitriis) in order to avoid conflicts about lib64/ld-linux-x86-64.so.2 + # with other parts. + - libc6 + - rsync + build-environment: *python-build-environment + after: [openstack-projects] override-build: | - # Hack to avoid conflict between openstack-projects' - # site-customize and this one. snapcraftctl build - rm $SNAPCRAFT_PART_INSTALL/usr/lib/python3.6/sitecustomize.py + `find $SNAPCRAFT_PART_INSTALL -name '__pycache__' | xargs rm -r` + `find $SNAPCRAFT_PART_INSTALL -name 'RECORD' | xargs rm` + rm $SNAPCRAFT_PART_INSTALL/pyvenv.cfg + rm $SNAPCRAFT_PART_INSTALL/bin/activate + rm $SNAPCRAFT_PART_INSTALL/bin/activate.csh + rm $SNAPCRAFT_PART_INSTALL/bin/activate.fish + # see LP: #1882994 + rm $SNAPCRAFT_PART_INSTALL/bin/python3 + rm $SNAPCRAFT_PART_INSTALL/bin/python + rm $SNAPCRAFT_PART_INSTALL/lib/python3.8/site-packages/netifaces.cpython-38-x86_64-linux-gnu.so # Launch script launch: plugin: python - python-version: python3 + # python-version: python3 requirements: - - requirements.txt + - requirements.txt source: tools/launch + stage-packages: + # note(dmitriis) in order to avoid conflicts about lib64/ld-linux-x86-64.so.2 + # with other parts. + - libc6 + build-environment: *python-build-environment + after: [init] override-build: | - # Hack to avoid conflict between openstack-projects' - # site-customize and this one. snapcraftctl build - rm $SNAPCRAFT_PART_INSTALL/usr/lib/python3.6/sitecustomize.py + `find $SNAPCRAFT_PART_INSTALL -name '__pycache__' | xargs rm -r` + `find $SNAPCRAFT_PART_INSTALL -name 'RECORD' | xargs rm` + rm $SNAPCRAFT_PART_INSTALL/pyvenv.cfg + rm $SNAPCRAFT_PART_INSTALL/bin/activate + rm $SNAPCRAFT_PART_INSTALL/bin/activate.csh + rm $SNAPCRAFT_PART_INSTALL/bin/activate.fish + rm $SNAPCRAFT_PART_INSTALL/bin/python3 + rm $SNAPCRAFT_PART_INSTALL/bin/python # Clustering client and server cluster: plugin: python - python-version: python3 + # python-version: python3 requirements: - requirements.txt constraints: - ${SNAPCRAFT_STAGE}/patches/upper-constraints.txt source: tools/cluster + # note(dmitriis) in order to avoid conflicts about lib64/ld-linux-x86-64.so.2 + # with other parts. + stage-packages: + - libc6 + build-packages: + - python3-dev + build-environment: *python-build-environment + after: [launch] override-build: | - # Hack to avoid conflict between openstack-projects' - # site-customize and this one. snapcraftctl build - rm $SNAPCRAFT_PART_INSTALL/usr/lib/python3.6/sitecustomize.py + `find $SNAPCRAFT_PART_INSTALL -name '__pycache__' | xargs rm -r` + `find $SNAPCRAFT_PART_INSTALL -name 'RECORD' | xargs rm` + rm $SNAPCRAFT_PART_INSTALL/pyvenv.cfg + rm $SNAPCRAFT_PART_INSTALL/bin/activate + rm $SNAPCRAFT_PART_INSTALL/bin/activate.csh + rm $SNAPCRAFT_PART_INSTALL/bin/activate.fish + rm $SNAPCRAFT_PART_INSTALL/bin/python + rm $SNAPCRAFT_PART_INSTALL/bin/python3 + + setup-lvm-loopdev: + source-type: local + source: ./tools/setup-lvm-loopdev + plugin: dump + organize: + setup-lvm-loopdev: bin/setup-lvm-loopdev + stage-packages: + # note(dmitriis) in order to avoid conflicts about lib64/ld-linux-x86-64.so.2 + # with other parts. + - libc6 + # fallocate + - util-linux + # losetup + - mount ### LMA stack ### filebeat: @@ -1037,8 +1563,18 @@ parts: hooks: install: - plugs: [network] + plugs: [ network ] # TODO: , microstack-support] configure: - plugs: [network] + plugs: [ network ] # TODO:, microstack-support] post-refresh: - plugs: [network] + plugs: [ network ] #, microstack-support] + remove: + # NOTE: network-bind is here due to LP: #1644573. + plugs: + - network + - network-bind + - network-control + - firewall-control + - hardware-observe + # TODO: - microstack-support + - block-devices diff --git a/tests/basic-test.sh b/tests/basic-test.sh index de14741..d4099f5 100755 --- a/tests/basic-test.sh +++ b/tests/basic-test.sh @@ -36,8 +36,8 @@ do esac done -if [ ! -f microstack_stein_amd64.snap ]; then - echo "microstack_stein_amd64.snap not found." +if [ ! -f microstack_ussuri_amd64.snap ]; then + echo "microstack_ussuri_amd64.snap not found." echo "Please run snapcraft before executing the tests." exit 1 fi @@ -72,7 +72,7 @@ if [ "$PREFIX" == "multipass" ]; then PREFIX="multipass exec $MACHINE --" multipass launch --cpus 2 --mem 16G $DISTRO --name $MACHINE - multipass copy-files microstack_stein_amd64.snap $MACHINE: + multipass copy-files microstack_ussuri_amd64.snap $MACHINE: HORIZON_IP=`multipass info $MACHINE | grep IPv4 | cut -d":" -f2 \ | tr -d '[:space:]'` @@ -80,11 +80,32 @@ fi # Possibly install a release of the snap before running a test. if [ "${UPGRADE_FROM}" != "none" ]; then - $PREFIX sudo snap install --classic --${UPGRADE_FROM} microstack + $PREFIX sudo snap install --${UPGRADE_FROM} microstack fi # Install the snap under test -- try again if the machine is not yet ready. -$PREFIX sudo snap install --classic --dangerous microstack*.snap +$PREFIX sudo snap install --dangerous microstack*.snap +$PREFIX sudo snap connect microstack:libvirt +$PREFIX sudo snap connect microstack:netlink-audit +$PREFIX sudo snap connect microstack:firewall-control +$PREFIX sudo snap connect microstack:hardware-observe +$PREFIX sudo snap connect microstack:kernel-module-observe +$PREFIX sudo snap connect microstack:kvm +$PREFIX sudo snap connect microstack:log-observe +$PREFIX sudo snap connect microstack:mount-observe +$PREFIX sudo snap connect microstack:netlink-connector +$PREFIX sudo snap connect microstack:network-observe +$PREFIX sudo snap connect microstack:openvswitch-support +$PREFIX sudo snap connect microstack:process-control +$PREFIX sudo snap connect microstack:system-observe +$PREFIX sudo snap connect microstack:network-control +$PREFIX sudo snap connect microstack:system-trace +$PREFIX sudo snap connect microstack:block-devices +$PREFIX sudo snap connect microstack:raw-usb +$PREFIX sudo snap connect microstack:hugepages-control +# $PREFIX sudo snap connect microstack:microstack-support + + $PREFIX sudo /snap/bin/microstack.init --auto # Comment out the above and uncomment below to install the version of diff --git a/tests/framework.py b/tests/framework.py index 07aef6f..aa57de5 100644 --- a/tests/framework.py +++ b/tests/framework.py @@ -82,7 +82,7 @@ class Host(): self.machine = '' self.distro = os.environ.get('DISTRO') or 'bionic' self.snap = os.environ.get('SNAP_FILE') or \ - 'microstack_stein_amd64.snap' + 'microstack_ussuri_amd64.snap' self.horizon_ip = '10.20.20.1' self.host_type = 'localhost' @@ -91,17 +91,34 @@ class Host(): print("Booting a Multipass VM ...") self.multipass() + self.microstack_test() + def install(self, snap=None, channel='dangerous'): if snap is None: snap = self.snap print("Installing {}".format(snap)) - check(*self.prefix, 'sudo', 'snap', 'install', '--devmode', - '--{}'.format(channel), snap) + check(*self.prefix, 'sudo', 'snap', 'install', + '--{}'.format(channel), '--devmode', snap) - def init(self, flag='auto'): - print("Initializing the snap with --{}".format(flag)) - check(*self.prefix, 'sudo', 'microstack.init', '--{}'.format(flag)) + # TODO: add microstack-support once it is merged into snapd. + connections = [ + 'microstack:libvirt', 'microstack:netlink-audit', + 'microstack:firewall-control', 'microstack:hardware-observe', + 'microstack:kernel-module-observe', 'microstack:kvm', + 'microstack:log-observe', 'microstack:mount-observe', + 'microstack:netlink-connector', 'microstack:network-observe', + 'microstack:openvswitch-support', 'microstack:process-control', + 'microstack:system-observe', 'microstack:network-control', + 'microstack:system-trace', 'microstack:block-devices', + 'microstack:raw-usb' + ] + for connection in connections: + check('sudo', 'snap', 'connect', connection) + + def init(self, args=['--auto']): + print(f"Initializing the snap with {args}") + check(*self.prefix, 'sudo', 'microstack.init', *args) def multipass(self): self.machine = petname.generate() @@ -119,6 +136,9 @@ class Host(): info = json.loads(info) self.horizon_ip = info['info'][self.machine]['ipv4'][0] + def microstack_test(self): + check('sudo', 'snap', 'install', 'microstack-test') + def dump_logs(self): # TODO: make unique log name if check_output('whoami') == 'zuul': diff --git a/tests/test_basic.py b/tests/test_basic.py index 14ad7e5..3d0764d 100755 --- a/tests/test_basic.py +++ b/tests/test_basic.py @@ -17,6 +17,7 @@ Web IDE. import os import sys import time +import json import unittest sys.path.append(os.getcwd()) @@ -35,7 +36,11 @@ class TestBasics(Framework): """ host = self.get_host() host.install() - host.init() + host.init([ + '--auto', + '--setup-loop-based-cinder-lvm-backend', + '--loop-device-file-size=32' + ]) prefix = host.prefix endpoints = check_output( @@ -71,13 +76,35 @@ class TestBasics(Framework): # Check to verify that our bridge is there. self.assertTrue('br-ex' in check_output(*prefix, 'ip', 'a')) - # Try to uninstall snap without sudo. - self.assertFalse(call(*prefix, '/snap/bin/microstack.remove', - '--purge', '--auto')) + check(*prefix, 'sudo', 'mkdir', '-p', '/tmp/snap.microstack-test/tmp') + check(*prefix, 'sudo', 'cp', + '/var/snap/microstack/common/etc/microstack.json', + '/tmp/snap.microstack-test/tmp/microstack.json') + check(*prefix, 'microstack-test.rally', 'db', 'recreate') + check(*prefix, 'microstack-test.rally', 'deployment', 'create', + '--filename', '/tmp/microstack.json', + '--name', 'snap_generated') + check(*prefix, 'microstack-test.tempest-init') + check(*prefix, 'microstack-test.rally', 'verify', 'start', + '--load-list', + '/snap/microstack-test/current/2020.06-test-list.txt', + '--detailed', '--concurrency', '2') + check(*prefix, 'microstack-test.rally', 'verify', 'report', + '--type', 'json', '--to', + '/tmp/verification-report.json') + report = json.loads(check_output( + *prefix, 'sudo', 'cat', + '/tmp/snap.microstack-test/tmp/verification-report.json')) + # Make sure there are no verification failures in the report. + failures = list(report['verifications'].values())[0]['failures'] + self.assertEqual(failures, 0, 'Verification tests had failure.') + + # Try to remove the snap without sudo. + self.assertFalse( + call(*prefix, 'snap', 'remove', '--purge', 'microstack')) # Retry with sudo (should succeed). - check(*prefix, 'sudo', '/snap/bin/microstack.remove', - '--purge', '--auto') + check(*prefix, 'sudo', 'snap', 'remove', '--purge', 'microstack') # Verify that MicroStack is gone. self.assertFalse(call(*prefix, 'snap', 'list', 'microstack')) diff --git a/tests/test_cluster.py b/tests/test_cluster.py index 58a6c3c..9df21d2 100755 --- a/tests/test_cluster.py +++ b/tests/test_cluster.py @@ -34,7 +34,7 @@ class TestCluster(Framework): openstack = '/snap/bin/microstack.openstack' control_host = self.get_host() control_host.install() - control_host.init(flag='control') + control_host.init(['--control']) control_prefix = control_host.prefix cluster_password = check_output(*control_prefix, 'sudo', 'snap', diff --git a/tests/test_control.py b/tests/test_control.py index 53956e8..bab1610 100755 --- a/tests/test_control.py +++ b/tests/test_control.py @@ -26,7 +26,7 @@ class TestControlNode(Framework): host = self.get_host() host.install() - host.init(flag='control') + host.init(['--control']) print("Checking output of services ...") services = check_output( diff --git a/tools/cluster/cluster/client.py b/tools/cluster/cluster/client.py old mode 100644 new mode 100755 index 3334566..c6efb54 --- a/tools/cluster/cluster/client.py +++ b/tools/cluster/cluster/client.py @@ -1,3 +1,5 @@ +#!/usr/bin/env python3 + import json import requests diff --git a/tools/init/init/main.py b/tools/init/init/main.py index ac3b33d..db15c0a 100644 --- a/tools/init/init/main.py +++ b/tools/init/init/main.py @@ -34,6 +34,7 @@ import logging import secrets import string import sys +import socket from functools import wraps @@ -55,6 +56,15 @@ def requires_sudo(func): return wrapper +def check_file_size_positive(value): + ival = int(value) + if ival < 1: + raise argparse.ArgumentTypeError( + f'The file size for a loop device' + f' must be larger than 1GB, current: {value}') + return ival + + def parse_init_args(): parser = argparse.ArgumentParser() parser.add_argument('--auto', '-a', action='store_true', @@ -63,6 +73,18 @@ def parse_init_args(): parser.add_argument('--compute', action='store_true') parser.add_argument('--control', action='store_true') parser.add_argument('--debug', action='store_true') + parser.add_argument( + '--setup-loop-based-cinder-lvm-backend', + action='store_true', + help='(experimental) set up a loop device-backed' + ' LVM backend for Cinder.' + ) + parser.add_argument( + '--loop-device-file-size', + type=check_file_size_positive, default=32, + help=('File size in GB (10^9) of a file to be exposed as a loop' + ' device for the Cinder LVM backend.') + ) args = parser.parse_args() return args @@ -100,6 +122,12 @@ def process_init_args(args): if args.debug: log.setLevel(logging.DEBUG) + check('snapctl', 'set', + f'config.cinder.setup-loop-based-cinder-lvm-backend=' + f'{str(args.setup_loop_based_cinder_lvm_backend).lower()}') + check('snapctl', 'set', + f'config.cinder.loop-device-file-size={args.loop_device_file_size}G') + return auto @@ -110,7 +138,8 @@ def init() -> None: question_list = [ questions.Clustering(), - questions.Dns(), + questions.DnsServers(), + questions.DnsDomain(), questions.NetworkSettings(), questions.OsPassword(), # TODO: turn this off if COMPUTE. questions.ForceQemu(), @@ -120,11 +149,15 @@ def init() -> None: questions.DashboardAccess(), questions.RabbitMq(), questions.DatabaseSetup(), + questions.PlacementSetup(), questions.NovaHypervisor(), questions.NovaControlPlane(), + questions.NovaSpiceConsoleSetup(), questions.NeutronControlPlane(), questions.GlanceSetup(), questions.SecurityRules(), + questions.CinderSetup(), + questions.CinderVolumeLVMSetup(), questions.PostSetup(), questions.ExtraServicesQuestion(), ] @@ -160,7 +193,8 @@ def set_network_info() -> None: check('snapctl', 'set', 'config.network.ext-gateway={}'.format(gate)) check('snapctl', 'set', 'config.network.ext-cidr={}'.format(cidr)) check('snapctl', 'set', 'config.network.control-ip={}'.format(ip)) - check('snapctl', 'set', 'config.network.control-ip={}'.format(ip)) + check('snapctl', 'set', + 'config.network.node-fqdn={}'.format(socket.getfqdn())) @requires_sudo diff --git a/tools/init/init/questions/__init__.py b/tools/init/init/questions/__init__.py index 0eb2e4d..1cbb7ea 100644 --- a/tools/init/init/questions/__init__.py +++ b/tools/init/init/questions/__init__.py @@ -28,7 +28,7 @@ from time import sleep from os import path from init.shell import (check, call, check_output, sql, nc_wait, log_wait, - restart, download) + start, restart, download, disable, enable) from init.config import Env, log from init.questions.question import Question from init.questions import clustering, network, uninstall # noqa F401 @@ -104,7 +104,7 @@ class Clustering(Question): # Turn off cluster server # TODO: it would be more secure to reverse this -- only enable # to service if we are doing clustering. - check('snapctl', 'stop', '--disable', 'microstack.cluster-server') + disable('cluster-server') class ConfigQuestion(Question): @@ -138,32 +138,36 @@ class ConfigQuestion(Question): _env[key.strip()] = val.strip() -class Dns(Question): - """Possibly override default dns.""" +class DnsServers(ConfigQuestion): + """Provide default DNS forwarders for MicroStack to use.""" _type = 'string' - _question = 'DNS to use' - config_key = 'config.network.dns' + _question = 'Upstream DNS servers to be used by instances (VMs)' + config_key = 'config.network.dns-servers' def yes(self, answer: str): - """Override the default dhcp_agent.ini file.""" - - file_path = '{SNAP_COMMON}/etc/neutron/dhcp_agent.ini'.format(**_env) - - with open(file_path, 'w') as f: - f.write("""\ -[DEFAULT] -interface_driver = openvswitch -dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq -enable_isolated_metadata = True -dnsmasq_dns_servers = {answer} -""".format(answer=answer)) - # Neutron is not actually started at this point, so we don't # need to restart. # TODO: This isn't idempotent, because it will behave # differently if we re-run this script when neutron *is* # started. Need to figure that out. + pass + + +class DnsDomain(ConfigQuestion): + """An internal DNS domain to be used for ML2 DNS.""" + + _type = 'string' + _question = 'An internal DNS domain to be used for ML2 DNS' + config_key = 'config.network.dns-domain' + + def yes(self, answer: str): + # Neutron is not actually started at this point, so we don't + # need to restart. + # TODO: This isn't idempotent, because it will behave + # differently if we re-run this script when neutron *is* + # started. Need to figure that out. + pass class NetworkSettings(Question): @@ -174,19 +178,46 @@ class NetworkSettings(Question): def yes(self, answer): log.info('Configuring networking ...') - # OpenvSwitch services may not have started up properly - restart('ovsdb-server') - restart('ovs-vswitchd') + role = check_output('snapctl', 'get', 'config.cluster.role') + + # Enable and start the services. + enable('ovsdb-server') + enable('ovs-vswitchd') + enable('ovn-ovsdb-server-sb') + enable('ovn-ovsdb-server-nb') network.ExtGateway().ask() network.ExtCidr().ask() + if role == 'control': + nb_conn = 'unix:{SNAP_COMMON}/run/ovn/ovnnb_db.sock'.format(**_env) + sb_conn = 'unix:{SNAP_COMMON}/run/ovn/ovnsb_db.sock'.format(**_env) + elif role == 'compute': + control_ip = check_output('snapctl', 'get', + 'config.network.control-ip') + sb_conn = f'tcp:{control_ip}:6642' + # Not used by any compute node services. + nb_conn = '' + else: + raise Exception(f'Unexpected node role: {role}') + + # Configure OVN SB and NB sockets based on the role node. For + # single-node deployments there is no need to use a TCP socket. + check_output('snapctl', 'set', + f'config.network.ovn-nb-connection={nb_conn}') + check_output('snapctl', 'set', + f'config.network.ovn-sb-connection={sb_conn}') + # Now that we have default or overriden values, setup the # bridge and write all the proper values into our config # files. check('setup-br-ex') check('snap-openstack', 'setup') + if role == 'control': + enable('ovn-northd') + enable('ovn-controller') + network.IpForwarding().ask() @@ -296,7 +327,7 @@ class RabbitMq(Question): def no(self, answer: str): log.info('Disabling local rabbit ...') - check('snapctl', 'stop', '--disable', 'microstack.rabbitmq-server') + disable('rabbitmq-server') class DatabaseSetup(Question): @@ -315,16 +346,17 @@ class DatabaseSetup(Question): def _create_dbs(self) -> None: # TODO: actually use passwords here. for db in ('neutron', 'nova', 'nova_api', 'nova_cell0', 'cinder', - 'glance', 'keystone'): - sql("CREATE DATABASE IF NOT EXISTS {db};".format(db=db)) - sql( - "GRANT ALL PRIVILEGES ON {db}.* TO {db}@{control_ip} \ - IDENTIFIED BY '{db}';".format(db=db, **_env)) + 'glance', 'keystone', 'placement'): + sql("CREATE USER IF NOT EXISTS '{db}'@'{control_ip}'" + " IDENTIFIED BY '{db}';".format(db=db, **_env)) + sql("CREATE DATABASE IF NOT EXISTS `{db}`;".format(db=db)) + sql("GRANT ALL PRIVILEGES ON {db}.* TO '{db}'@'{control_ip}';" + "".format(db=db, **_env)) # Grant nova user access to cell0 sql( - "GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'{control_ip}' \ - IDENTIFIED BY \'nova';".format(**_env)) + "GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'{control_ip}';" + "".format(**_env)) def _bootstrap(self) -> None: @@ -337,7 +369,8 @@ class DatabaseSetup(Question): '--bootstrap-password', _env['ospassword'], '--bootstrap-admin-url', bootstrap_url, '--bootstrap-internal-url', bootstrap_url, - '--bootstrap-public-url', bootstrap_url) + '--bootstrap-public-url', bootstrap_url, + '--bootstrap-region-id', 'microstack') def yes(self, answer: str) -> None: """Setup Databases. @@ -355,8 +388,8 @@ class DatabaseSetup(Question): # Start keystone-uwsgi. We use snapctl, because systemd # doesn't yet know about the service. - check('snapctl', 'start', 'microstack.nginx') - check('snapctl', 'start', 'microstack.keystone-uwsgi') + start('nginx') + start('keystone-uwsgi') log.info('Configuring Keystone Fernet Keys ...') check('snap-openstack', 'launch', 'keystone-manage', @@ -382,7 +415,7 @@ class DatabaseSetup(Question): check('snapctl', 'set', 'database.ready=true') log.info('Disabling local MySQL ...') - check('snapctl', 'stop', '--disable', 'microstack.mysqld') + disable('mysqld') class NovaHypervisor(Question): @@ -404,11 +437,63 @@ class NovaHypervisor(Question): 'microstack', 'compute', endpoint, 'http://{compute_ip}:8774/v2.1'.format(**_env)) - check('snapctl', 'start', 'microstack.nova-compute') + start('nova-compute') def no(self, answer): log.info('Disabling nova compute service ...') - check('snapctl', 'stop', '--disable', 'microstack.nova-compute') + disable('nova-compute') + + +class NovaSpiceConsoleSetup(Question): + """Run the Spice HTML5 console proxy service""" + + _type = 'boolean' + config_key = 'config.services.spice-console' + + def yes(self, answer): + log.info('Configuring the Spice HTML5 console service...') + start('nova-spicehtml5proxy') + + def no(self, answer): + log.info('Disabling nova compute service ...') + disable('nova-spicehtml5proxy') + + +class PlacementSetup(Question): + """Setup Placement services.""" + + _type = 'boolean' + config_key = 'config.services.control-plane' + + def yes(self, answer: str) -> None: + log.info('Configuring the Placement service...') + + if not call('openstack', 'user', 'show', 'placement'): + check('openstack', 'user', 'create', '--domain', 'default', + '--password', 'placement', 'placement') + check('openstack', 'role', 'add', '--project', 'service', + '--user', 'placement', 'admin') + + if not call('openstack', 'service', 'show', 'placement'): + check('openstack', 'service', 'create', '--name', + 'placement', '--description', '"Placement API"', + 'placement') + + for endpoint in ['public', 'internal', 'admin']: + call('openstack', 'endpoint', 'create', '--region', + 'microstack', 'placement', endpoint, + 'http://{control_ip}:8778'.format(**_env)) + + start('placement-uwsgi') + + log.info('Running Placement DB migrations...') + check('snap-openstack', 'launch', 'placement-manage', 'db', 'sync') + + restart('placement-uwsgi') + + def no(self, answer): + log.info('Disabling the Placement service...') + disable('placement-uwsgi') class NovaControlPlane(Question): @@ -446,31 +531,14 @@ class NovaControlPlane(Question): check('openstack', 'role', 'add', '--project', 'service', '--user', 'nova', 'admin') - if not call('openstack', 'user', 'show', 'placement'): - check('openstack', 'user', 'create', '--domain', 'default', - '--password', 'placement', 'placement') - check('openstack', 'role', 'add', '--project', 'service', - '--user', 'placement', 'admin') - - if not call('openstack', 'service', 'show', 'placement'): - check('openstack', 'service', 'create', '--name', - 'placement', '--description', '"Placement API"', - 'placement') - - for endpoint in ['public', 'internal', 'admin']: - call('openstack', 'endpoint', 'create', '--region', - 'microstack', 'placement', endpoint, - 'http://{control_ip}:8778'.format(**_env)) - # Use snapctl to start nova services. We need to call them # out manually, because systemd doesn't know about them yet. # TODO: parse the output of `snapctl services` to get this # list automagically. - for service in [ - 'microstack.nova-api', - ]: - check('snapctl', 'start', service) + start('nova-api') + log.info('Running Nova API DB migrations' + ' (this will take a lot of time)...') check('snap-openstack', 'launch', 'nova-manage', 'api_db', 'sync') if 'cell0' not in check_output('snap-openstack', 'launch', @@ -485,18 +553,19 @@ class NovaControlPlane(Question): check('snap-openstack', 'launch', 'nova-manage', 'cell_v2', 'create_cell', '--name=cell1', '--verbose') + log.info('Running Nova DB migrations' + ' (this will take a lot of time)...') check('snap-openstack', 'launch', 'nova-manage', 'db', 'sync') restart('nova-api') restart('nova-compute') for service in [ - 'microstack.nova-api-metadata', - 'microstack.nova-conductor', - 'microstack.nova-scheduler', - 'microstack.nova-uwsgi', + 'nova-api-metadata', + 'nova-conductor', + 'nova-scheduler', ]: - check('snapctl', 'start', service) + start(service) nc_wait(_env['compute_ip'], '8774') @@ -509,13 +578,92 @@ class NovaControlPlane(Question): log.info('Disabling nova control plane services ...') for service in [ - 'microstack.nova-uwsgi', - 'microstack.nova-api', - 'microstack.nova-conductor', - 'microstack.nova-scheduler', - 'microstack.nova-api-metadata']: + 'nova-api', + 'nova-conductor', + 'nova-scheduler', + 'nova-api-metadata']: + disable(service) - check('snapctl', 'stop', '--disable', service) + +class CinderSetup(Question): + """Setup Placement services.""" + + _type = 'boolean' + config_key = 'config.services.control-plane' + + def yes(self, answer: str) -> None: + log.info('Configuring the Cinder services...') + + if not call('openstack', 'user', 'show', 'cinder'): + check('openstack', 'user', 'create', '--domain', 'default', + '--password', 'cinder', 'cinder') + check('openstack', 'role', 'add', '--project', 'service', + '--user', 'cinder', 'admin') + + control_ip = _env['control_ip'] + for endpoint in ['public', 'internal', 'admin']: + for api_version in ['v2', 'v3']: + if not call('openstack', 'service', 'show', + f'cinder{api_version}'): + check('openstack', 'service', 'create', '--name', + f'cinder{api_version}', '--description', + f'"Cinder {api_version} API"', + f'volume{api_version}') + if not check_output( + 'openstack', 'endpoint', 'list', + '--service', f'volume{api_version}', '--interface', + endpoint): + check( + 'openstack', 'endpoint', 'create', '--region', + 'microstack', f'volume{api_version}', endpoint, + f'http://{control_ip}:8776/{api_version}/' + '$(project_id)s' + ) + restart('cinder-uwsgi') + + log.info('Running Cinder DB migrations...') + check('snap-openstack', 'launch', 'cinder-manage', 'db', 'sync') + + restart('cinder-uwsgi') + restart('cinder-scheduler') + + def no(self, answer): + log.info('Disabling Cinder services...') + + for service in [ + 'cinder-uwsgi', + 'cinder-scheduler', + 'cinder-volume', + 'cinder-backup']: + disable(service) + + +class CinderVolumeLVMSetup(Question): + """Setup cinder-volume with LVM.""" + + _type = 'boolean' + config_key = 'config.cinder.setup-loop-based-cinder-lvm-backend' + _question = ('(experimental) Do you want to setup a loop device-backed LVM' + ' volume backend for Cinder?') + interactive = True + + def yes(self, answer: bool) -> None: + check('snapctl', 'set', + f'config.cinder.setup-loop-based-cinder-lvm-backend' + f'={str(answer).lower()}') + log.info('Setting up cinder-volume service with the LVM backend...') + enable('setup-lvm-loopdev') + enable('cinder-volume') + enable('target') + enable('iscsid') + + def no(self, answer: bool) -> None: + check('snapctl', 'set', f'config.cinder.lvm.setup-file-backed-lvm=' + f'{str(answer).lower()}') + disable('setup-lvm-loopdev') + disable('cinder-volume') + disable('iscsid') + disable('target') class NeutronControlPlane(Question): @@ -541,26 +689,16 @@ class NeutronControlPlane(Question): 'microstack', 'network', endpoint, 'http://{control_ip}:9696'.format(**_env)) - for service in [ - 'microstack.neutron-api', - 'microstack.neutron-dhcp-agent', - 'microstack.neutron-l3-agent', - 'microstack.neutron-metadata-agent', - 'microstack.neutron-openvswitch-agent', - ]: - check('snapctl', 'start', service) + start('neutron-api') check('snap-openstack', 'launch', 'neutron-db-manage', 'upgrade', 'head') for service in [ - 'microstack.neutron-api', - 'microstack.neutron-dhcp-agent', - 'microstack.neutron-l3-agent', - 'microstack.neutron-metadata-agent', - 'microstack.neutron-openvswitch-agent', + 'neutron-api', + 'neutron-ovn-metadata-agent', ]: - check('snapctl', 'restart', service) + restart(service) nc_wait(_env['control_ip'], '9696') @@ -594,20 +732,23 @@ class NeutronControlPlane(Question): neutron on this machine. """ - # Make sure that the agent is running. + # Make sure the necessary services are enabled and started. for service in [ - 'microstack.neutron-openvswitch-agent', + 'ovs-vswitchd', + 'ovsdb-server', + 'ovn-controller', + 'neutron-ovn-metadata-agent' ]: - check('snapctl', 'start', service) + enable(service) # Disable the other services. for service in [ - 'microstack.neutron-api', - 'microstack.neutron-dhcp-agent', - 'microstack.neutron-metadata-agent', - 'microstack.neutron-l3-agent', + 'neutron-api', + 'ovn-northd', + 'ovn-ovsdb-server-sb', + 'ovn-ovsdb-server-nb', ]: - check('snapctl', 'stop', '--disable', service) + disable(service) class GlanceSetup(Question): @@ -660,10 +801,10 @@ class GlanceSetup(Question): 'http://{compute_ip}:9292'.format(**_env)) for service in [ - 'microstack.glance-api', - 'microstack.registry', # TODO rename to glance-registery + 'glance-api', + 'registry', # TODO rename to glance-registery ]: - check('snapctl', 'start', service) + start(service) check('snap-openstack', 'launch', 'glance-manage', 'db_sync') @@ -677,8 +818,8 @@ class GlanceSetup(Question): self._fetch_cirros() def no(self, answer): - check('snapctl', 'stop', '--disable', 'microstack.glance-api') - check('snapctl', 'stop', '--disable', 'microstack.registry') + disable('glance-api') + disable('registry') class SecurityRules(Question): @@ -725,9 +866,9 @@ class PostSetup(Question): # TODO: fix issue. restart('libvirtd') restart('virtlogd') + restart('nova-compute') - # Start horizon - check('snapctl', 'start', 'microstack.horizon-uwsgi') + restart('horizon-uwsgi') check('snapctl', 'set', 'initialized=true') log.info('Complete. Marked microstack as initialized!') @@ -739,13 +880,13 @@ class SimpleServiceQuestion(Question): log.info('enabling and starting ' + self.__class__.__name__) for service in self.services: - check('snapctl', 'start', '--enable', service) + enable(service) log.info(self.__class__.__name__ + ' enabled') def no(self, answer): for service in self.services: - check('snapctl', 'stop', '--disable', service) + disable(service) class ExtraServicesQuestion(Question): diff --git a/tools/init/init/questions/uninstall.py b/tools/init/init/questions/uninstall.py index d3fa020..bd4accf 100644 --- a/tools/init/init/questions/uninstall.py +++ b/tools/init/init/questions/uninstall.py @@ -2,7 +2,7 @@ import sys from init.config import Env, log from init.questions.question import Question -from init.shell import check, call +from init.shell import call _env = Env().get_env() @@ -29,7 +29,6 @@ class DeleteBridge(Question): # TODO: cleanup system optimizations # TODO: cleanup kernel modules? -# TODO: cleanup iptables rules class RemoveMicrostack(Question): @@ -40,8 +39,4 @@ class RemoveMicrostack(Question): def yes(self, answer): """Uninstall MicroStack, passing any command line options to snapd.""" - log.info('Uninstalling MicroStack (this may take a while) ...') - check('snap', 'remove', '{SNAP_INSTANCE_NAME}'.format(**_env), - *ARGS) - log.info('MicroStack has been removed from your system!') diff --git a/tools/init/init/shell.py b/tools/init/init/shell.py index 88f7414..b331895 100644 --- a/tools/init/init/shell.py +++ b/tools/init/init/shell.py @@ -129,6 +129,16 @@ def log_wait(log: str, message: str) -> None: sleep(1) +def start(service: str) -> None: + """Start a microstack service. + + :param service: the service(s) to be started. Can contain wild cards. + e.g. *rabbit* + + """ + check('snapctl', 'start', 'microstack.{}'.format(service)) + + def restart(service: str) -> None: """Restart a microstack service. @@ -139,6 +149,16 @@ def restart(service: str) -> None: check('snapctl', 'restart', 'microstack.{}'.format(service)) +def enable(service: str) -> None: + """Disable and mask a service. + + :param service: the service(s) to be enabled. Can contain wild cards. + e.g. *rabbit* + + """ + check('snapctl', 'start', '--enable', 'microstack.{}'.format(service)) + + def disable(service: str) -> None: """Disable and mask a service. diff --git a/tools/init/requirements.txt b/tools/init/requirements.txt index ce73c29..fbc66bb 100644 --- a/tools/init/requirements.txt +++ b/tools/init/requirements.txt @@ -1,4 +1,5 @@ -netaddr +# netaddr is pinned to match the upper-constraints.txt file of Ussuri +netaddr===0.7.19 netifaces -pymysql +pymysql==0.9.3 wget diff --git a/tools/init/setup.py b/tools/init/setup.py index 3650c80..8754cce 100644 --- a/tools/init/setup.py +++ b/tools/init/setup.py @@ -9,7 +9,6 @@ setup( 'console_scripts': [ 'microstack_init = init.main:init', 'set_network_info = init.main:set_network_info', - 'microstack_remove = init.main:remove', ], }, ) diff --git a/tools/init/tests/test_question.py b/tools/init/tests/test_question.py index d00f491..f7bc94f 100644 --- a/tools/init/tests/test_question.py +++ b/tools/init/tests/test_question.py @@ -7,7 +7,7 @@ import mock # TODO: drop in test runner and get rid of this line. sys.path.append(os.getcwd()) # noqa -from init.questions.question import (Question, InvalidQuestion, InvalidAnswer) +from init.questions.question import (Question, InvalidQuestion, InvalidAnswer) # noqa ############################################################################## diff --git a/tools/lxd_build.sh b/tools/lxd_build.sh index 4471a20..af64f60 100755 --- a/tools/lxd_build.sh +++ b/tools/lxd_build.sh @@ -9,9 +9,11 @@ sudo apt update sudo apt install -y firefox-geckodriver python3-petname python3-selenium # Setup snapd and snapcraft +# Install snapd if it isn't installed yet (needed to install the snapd snap itself). sudo apt install -y snapd -# Build our snap! +sudo snap install snapd + sudo snap install --classic snapcraft sudo snap install lxd @@ -22,4 +24,6 @@ newgrp lxd << END set -ex lxd init --auto snapcraft --use-lxd -END \ No newline at end of file +# Delete the build container to free the storage space on a test node. +lxc delete snapcraft-microstack +END diff --git a/tools/make-a-microstack.sh b/tools/make-a-microstack.sh index 9fe6999..4c46d18 100755 --- a/tools/make-a-microstack.sh +++ b/tools/make-a-microstack.sh @@ -20,9 +20,49 @@ MACHINE=$(petname) multipass launch --cpus 2 --mem 16G $DISTRO --name $MACHINE # Install the snap -multipass copy-files microstack_stein_amd64.snap $MACHINE: +multipass copy-files microstack_ussuri_amd64.snap $MACHINE: multipass exec $MACHINE -- \ - sudo snap install --classic --dangerous microstack*.snap + sudo snap install --dangerous microstack*.snap + +multipass exec $MACHINE -- \ + sudo snap connect microstack:libvirt +multipass exec $MACHINE -- \ + sudo snap connect microstack:netlink-audit +multipass exec $MACHINE -- \ + sudo snap connect microstack:firewall-control +multipass exec $MACHINE -- \ + sudo snap connect microstack:hardware-observe +multipass exec $MACHINE -- \ + sudo snap connect microstack:kernel-module-observe +multipass exec $MACHINE -- \ + sudo snap connect microstack:kvm +multipass exec $MACHINE -- \ + sudo snap connect microstack:log-observe +multipass exec $MACHINE -- \ + sudo snap connect microstack:mount-observe +multipass exec $MACHINE -- \ + sudo snap connect microstack:netlink-connector +multipass exec $MACHINE -- \ + sudo snap connect microstack:network-observe +multipass exec $MACHINE -- \ + sudo snap connect microstack:openvswitch-support +multipass exec $MACHINE -- \ + sudo snap connect microstack:process-control +multipass exec $MACHINE -- \ + sudo snap connect microstack:system-observe +multipass exec $MACHINE -- \ + sudo snap connect microstack:network-control +multipass exec $MACHINE -- \ + sudo snap connect microstack:system-trace +multipass exec $MACHINE -- \ + sudo snap connect microstack:block-devices +multipass exec $MACHINE -- \ + sudo snap connect microstack:raw-usb +multipass exec $MACHINE -- \ + sudo snap connect microstack:hugepages-control +# TODO: add the below once the interface is merge into snapd. +# multipass exec $MACHINE -- \ +# sudo snap connect microstack:microstack-support # Drop the user into a snap shell, as root. multipass exec $MACHINE -- \ diff --git a/tools/setup-lvm-loopdev/setup-lvm-loopdev b/tools/setup-lvm-loopdev/setup-lvm-loopdev new file mode 100755 index 0000000..cdcc27c --- /dev/null +++ b/tools/setup-lvm-loopdev/setup-lvm-loopdev @@ -0,0 +1,28 @@ +#!/bin/bash + +set -ex + +cinder_volumes_vg=`snapctl get config.cinder.lvm-backend-volume-group` +if [ `snapctl get config.cinder.setup-loop-based-cinder-lvm-backend` = 'true' ] +then + loop_file=$SNAP_COMMON/cinder-lvm.img + loop_file_size=`snapctl get config.cinder.loop-device-file-size` + # Create a file to hold an LVM PV+VG + LVs if it does not exist. + test -f $loop_file || fallocate -l $loop_file_size $loop_file + # Unless this file already has an associated loop device, associate a free loop device with it. + if [ -z `losetup -j $loop_file` ] + then + until losetup -f $loop_file + do + echo 'Waiting until the device cgroup entry is updated, see LP# 1892895' + sleep 1 + done + fi + + allocated_loop_dev=`losetup -j $loop_file | cut -d':' -f 1` + # Create a PV on the allocated loop device unless there is already one on it. + lvmdiskscan -l --config 'devices { filter = [ "a|'$allocated_loop_dev'|", "r|.*|" ] }' | grep -q '1 LVM' || (pvcreate $allocated_loop_dev && vgcreate $cinder_volumes_vg $allocated_loop_dev && exit 0) +fi + +# Activate the logical volumes (relevant on node reboot). +lvchange -a y $cinder_volumes_vg