Commit Graph

73 Commits

Author SHA1 Message Date
Zuul
814a6ac6e3 Merge "Use default role of member for horizon" 2022-01-10 11:09:42 +00:00
Zuul
417cde364d Merge "Use microstack-support interface, drop devmode" 2021-12-03 20:19:33 +00:00
Billy Olsen
4123da81ef Use default role of member for horizon
Default role configured for horizon is set to _member_ but this
is not a role which is configured in keystone. Change to use the
default role of 'member'.

Closes-Bug: #1861371
Change-Id: I08c171cb58ac669c0425274ebb9ceeb17bb9b3f9
2021-12-03 13:02:56 -07:00
Dmitrii Shcherbakov
35ad8fbef0 Use microstack-support interface, drop devmode
With https://github.com/snapcore/snapd/pull/8926 merged and snapd 2.53.2
released we can now test changes without devmode.

Change-Id: I9980b171e537530d67f0a7eed332147f06fe7c3c
2021-11-26 18:23:03 +03:00
Billy Olsen
19d74ff9ba Add PKI API for compute nodes certificates
Treat the control node as a CA for certificates at compute nodes.
Upon joining a cluster, the compute node will request a certificate to
be created by generating a CSR and asking the control node to sign the
certificate.

This adds new config options for the compute private keys and
certificate locations in use.

Change-Id: I8e8b1a86cf7df752b6cb34cfdf65a87a72934ec5
2021-10-20 11:50:43 -07:00
Corey Bryant
4d4c142b07 Add debug logging for various nginx sites
A previous patch provided debug logging support, including
nginx.conf.j2, but failed to provide support to any nginx
site templates.

Change-Id: I7375fa52f2db847deee5c8181e47e68079714ab9
2021-05-26 16:40:46 -04:00
Corey Bryant
064aae8458 Add TLS OpenStack API endpoints
This patch provides TLS endpoints secured by a self-signed
certificate. Another patch will provide support for trusted CA-signed
certificates.

A new config.tls.generate-cert option is added that defaults to true.
When true, a self-signed certificate will be generated and OpenStack
API endpoints will be configured to use TLS with that self-signed
certificate. The following config options are added:

snap get microstack config.tls.generate-self-signed
snap get microstack config.tls.cacert-path
snap get microstack config.tls.cert-path
snap get microstack config.tls.key-path

Users can provide their own self-signed certificate by setting
generate-self-signed to false and storing their own certificates/key
at the paths specified by cacert-path, cert-path, and key-path.
'snap set' can also be used to change the cert/key file names.

If using clustering, the certificates/key will be copied from the
control node to the compute nodes. The config for cacert-path,
cert-path, and key-path will be set to the same values as on the
control node.

Other notable changes:
* The existing generate_selfsigned() function is modified to change
  the subject alternative name to be made up of the hostname and
  optionally an IP. The controller hostname and IP are used when
  generating the certificate for self-signed TLS endpoints. The
  hostname is now used instead of 'microstack.run' when generating
  the clustering certificate.
* This change also aligns logging for nginx and corresponding sites
  and moves all nginx sites to {snap_common}/etc/nginx/sites-enabled.

Change-Id: Iceea3127822404a3275fcf8a221cbedc4b52c217
2021-05-26 16:39:33 -04:00
Zuul
525b11fd21 Merge "Disable client_max_body_size checks in nginx" 2021-04-29 06:58:28 +00:00
Billy Olsen
727c562f2d Disable client_max_body_size checks in nginx
By default, nginx has a small (1MB) limit on the size of uploads,
which prevents using horizon interface to upload an image. Disabling
the client_max_body_size check allows for the glance configuration
to govern the maximum size image that should be accepted. This change
also disables proxy_buffering and proxy_request_buffering in order
to reduce buffering latency.

Closes-Bug: #1868503
Change-Id: I0a89e0845d6c7d2805556f87685d280b4e72122a
2021-04-28 13:35:04 -07:00
Billy Olsen
6bb7c63cd1 Enable libvirt daemon to listen for remote connections
Enable the libvirt daemon to listen for remote connections. This
enables the live migration of instances between nodes in a microstack
cluster. Note, this is using TCP based connections and not secured
TLS based connections. That work should be done as part of enabling
TLS everywhere.

Closes-Bug: #1925707
Change-Id: If00d825c52c2d0dd12bc652ba26f67160dc7a6c5
2021-04-27 14:47:47 -07:00
Billy Olsen
0d7785f233 Fix snapcraft build issues
The latest version of snapcraft fails to build microstack. Chasing
it down is due to multiple staged parts which have conflicting
information. Fixing that bit causes the uwsgi services not to run
correctly.

This patch fixes the uwsgi services by no longer overriding the python
home directory, since python3 is staged into the snap we can use the
default python home.

This patch also removes the libc6 staging into the package, which should
generally be avoided.

Change-Id: I8c176689083831a0b8b56a192a9fbdfb50edbb66
2021-04-26 22:20:33 -07:00
Corey Bryant
0ef39f2865 Add debug logging and individual OpenStack log files
This change introduces config.logging.debug that defaults
to False. Setting to True will enable debug logging for
OpenStack and nginx services.

Change-Id: I2eb428851d795e145c542879faf22b2fd9f8a29f
2021-03-29 14:24:39 +03:00
Corey Bryant
1f30e10b5b Drop glance-registry
Glance-registry has been deprecated since Queens and were removed
from the upstream source in Train.

Change-Id: Ia993bfce039cd46ced3442c9064e4af8547fa54f
2021-03-17 17:07:12 -04:00
Corey Bryant
0ac2f83505 Only include loop devices in LVM global_filter
The following tempest test is failing regularly in the gate:
tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments

The theory behind this fix is that tests are creating/deleting /dev/sdX
devices and LVM ends up attempting to open an already removed device
which causes LVM to temporarily block. Setting the global_filter will
limit the block devices that are used by LVM system components.
Microstack only uses a loopback device for LVM.

Closes-Bug: #1918306
Change-Id: I8cccf7a1b1af2e15106b11023652af23c7715e6f
2021-03-17 17:01:29 -04:00
Dmitrii Shcherbakov
a904cb6804 Rework the test framework & the clustering test
* Remove the dead code;
* Rework the test types;
* Restore the instance connectivity check;
* Rework the clustering test to support the new node addition workflow;
* Check whether a machine where MicroStack is installed has hardware
  virtualization capabilities for different architectures. If not, use
  software emulation;
  * the host model is used with KVM since the default QEMU CPU models on
    x86_64 are subject to vulnerabilities without certain CPU-specific
    features. This conflicts with being able to use live migration
    reliably across hosts with different CPUs.
* Add a default-source-ip init argument to allow controlling the source
  IP of the installation host that will be used as a control ip or
  compute ip locally.
  * used in the clustering test so that the local host IP on the
    multipass network is used as a control IP instead of the IP
    through which the default gateway is available;
  * the IP through which the default gateway is accessible is
    used as a fallback for default-source-ip;
* Given upstream CI has a low amount of resources allocated per machine
  use LXD to set up a dummy compute node;
  * Set RLIMIT_MEMLOCK to 'unlimited' in the LXD container profile
    (see the discussion in LP: #1906280);
  * set remember_owner to 0 in qemu.conf for libvirt to avoid the
    uses of XATTRS (the root user is used anyway so there is no
    need to remember a file owner), otherwise libvirt errors out
    in an unprivileged LXD container.
* Use numeric versions of OpenStack packages in the python-packages
  section of the openstack-projects part since the resolver change in
  recent versions of pip disallows for constraints dependencies of
  packages that come from a URL or a path.
  https://github.com/pypa/pip/issues/8210
  * The newest released version of pip is always used during builds
    since snapcraft uses venv to set up virtual environments and the
    ensurepip package is invoked such that a pip version shipped with
    the distro version of python is upgraded:
    https://github.com/python/cpython/blob/3.8/Lib/venv/__init__.py#L282-L289
            cmd = [context.env_exe, '-Im', 'ensurepip', '--upgrade',
                                                    '--default-pip']
  * Environment variables are ignored when pip is installed in the venv:
    https://docs.python.org/3/using/cmdline.html#id2 (-I option)
    So there is no way to use the old pip version resolver.

Minor clustering client and add-compute changes:

* use stderr for diagnostic messages;
* use stdout to output the connection string so that it can be easily
  picked up by CLI tools without parsing.

Change-Id: I5cb3872c5d142c34da2c8b073652c67021d9ef55
2021-01-15 15:58:03 +03:00
Dmitrii Shcherbakov
c19525476d Drop pci-stub from the load-modules service
pci-stub is available on Ubuntu from modules-extra kernel packages,
however, those are not always installed.

By the looks of it, this module is no longer supported by Libvirt either
(presumably in favor of VFIO):
b8e7e9be9a

Change-Id: I0db4d43d458893493232e150ae2f5b2bb7e05772
Closes-Bug: #1900113
2020-10-21 16:07:17 +03:00
Dmitrii Shcherbakov
e690b22381 Add a [placement] section into Neutron config
It appears to be that the Neutron Segment plugin needs to instantiate a
placement service client which requires credentials and other connection
details:
https://opendev.org/openstack/neutron/src/branch/stable/ussuri/neutron/services/segments/plugin.py#L188

Otherwise, the following exception can be seen periodically in the
Neutron log:

keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin:
 An auth plugin is required to determine endpoint URL

Change-Id: I480292a1c74c376db5c9797b1fcc8469e0e5507a
2020-10-15 14:30:15 +03:00
Dmitrii Shcherbakov
0ba5358865 Add Secure Clustering
* Add a connection-string based workflow to MicroStack;
  * microstack add-compute command can be run at the Control node in
    order to generate a connection string (an ASCII blob for the user);
  * the connection string contains:
    * an address of the control node;
    * a sha256 fingerprint of the TLS certificate used by the clustering
      service at the control node (which is used during verification
      similar to the Certificate Pinning approach);
    * an application credential id;
    * an application credential secret (short expiration time, reader
      role on the service project, restricted to listing the service
      catalog);
  * a MicroStack admin is expected to have ssh access to all nodes that
    will participate in a cluster - prior trust establishment is on
    them to figure out which is normal since they provision the nodes;
  * a MicroStack admin is expected to securely copy a connection string
    to a compute node via ssh. Since it is short-lived and does not
    carry service secrets, there is no risk of a replay at a later time;
  * If the compute role is specified during microstack.init, a
    connection string is requested and used to perform a request to the
    clustering service and validate the certificate fingerprint. The
    credential ID and secret are POSTed for verification to the
    clustering service which responds with the necessary config data
    for the compute node upon successful authorization.
* Set up TLS termination for the clustering service;
  * run the flask app as a UWSGI daemon behind nginx;
  * configure nginx to use a TLS certificate;
  * generate a self-signed TLS certificate.

This setup does not require PKI to be present for its own purposes of
joining compute nodes to the cluster. However, this does not mean that
PKI will not be used for TLS termination of the OpenStack endpoints.

Control node init workflow (non-interactive):

sudo microstack init --auto --control
microstack add-compute
<the connection string to be used at the compute node>

Compute node init workflow (non-interactive):

sudo microstack init --auto --compute --join <connection-string>

Change-Id: I9596fe1e6e5c1a325cc71fd3bf0c78b660b9a83e
2020-10-15 01:37:33 +03:00
Dmitrii Shcherbakov
81cbaa4433 Fix the Neutron OVN metadata service setup
* Set the Nova metadata server address properly so that
  neutron-ovn-metadata-agents running on compute nodes forward the
  requests to the right place instead of trying to use 127.0.0.1;
* generate a random secret instead of hard-coding one.

Change-Id: I6525a4150808ef257bb7a8f49589c1151ca279b0
2020-10-10 08:32:11 +03:00
Dmitrii Shcherbakov
32ad5af7f4 Generate random passwords instead of hard-coding
* The prototype stage hard-coding of passwords is replaced by random
  generation of passwords for:
  * all API services;
  * RabbitMQ;
  * MySQL;
  * OpenStack admin user;
  * OpenStack service users;
* Passwords are not replaced upon successive microstack.init calls to
  preserve idempotency.

Change-Id: Ic3d6108a81d09bdd09e986f80b3040b030605178
2020-10-08 11:25:25 +03:00
Dmitrii Shcherbakov
71ad68d36a Fix Clustering after a rebase to Ussuri + OVN
The previous work included incorrect handling of
configuration for the multi-node case in terms of
OVN configuration.

This change addresses that in addition to other
minor fixes related to the clustering setup.

Change-Id: Ibf04af95271d1746f59192d11831d6129ba5b8d0
2020-10-05 02:37:02 +03:00
Dmitrii Shcherbakov
9cfed61a07 Minor fixes per review 738242
* Set max_header_size to 38 per the upstream Neutron guide albeit the
header size can be variable:
https://tools.ietf.org/html/draft-ietf-nvo3-geneve-16#page-14

* The empty "plugs" entry results in the following error raised by the
auto-review tool of the snap store:

"invalid plugs entry (empty) lint-snap-v2_app_plugs (rabbitmq-plugins)"

Let's avoid this by removing the section altogether.

Change-Id: I1b192140c8ca3445bd817f7e583f303d1bb0a338
2020-09-28 19:23:55 +03:00
Dmitrii Shcherbakov
780a4c4ead Use focal/core20/Ussuri/OVN & enable confinement
Major changes:

* Plumbing necessary for strict confinement with
  the microstack-support interface
  https://github.com/snapcore/snapd/pull/8926
  * Until the interface is merged, devmode will be used and kernel
    modules will be loaded via an auxiliary service.
* upgraded OpenStack components to Focal (20.04) and OpenStack Ussuri;
  * reworked the old patches;
  * added the Placement service since it is now separate;
  * addressed various build issues due to changes in snapcraft and
    built dependencies:
    * e.g. libvirt requires the build directory to be separate from the
      source directory) and LP: #1882255;
    * LP: #1882535 and https://github.com/pypa/pip/issues/8414
    * LP: #1882839
    * LP: #1885294
    * https://storyboard.openstack.org/#!/story/2007806
    * LP: #1864589
    * LP: #1777121
    * LP: #1881590
* ML2/OVS replated with ML2/OVN;
  * dnsmasq is not used anymore;
  * neutron l3 and DHCP agents are not used anymore;
  * Linux network namespaces are only used for
    neutron-ovn-metadata-agent.
  * ML2 DNS support is done via native OVN mechanisms;
  * OVN-related database services (southbound and northbound dbs);
  * OVN-related control plane services (ovn-controller, ovn-northd);
* core20 base support (bionic hosts are supported);
* the removal procedure now relies on the "remove" hook since `snap
remove` cannot be used from the confined environment anymore;
* prerequisites to enabling AppArmor confinement for QEMU processes
  created by the confined libvirtd.
* Added the Spice html5 console proxy service to enable clients to
  retrieve and use it via
  `microstack.openstack console url show --spice <servername>`.
* Added missing Cinder templates and DB migrations for the Cinder DB.
* Added experimental support for a loop device-based LVM backend for
  Cinder. Due to LP: #1892895 this is not recommended to be used in
  production except for tempest testing with an applied workaround;
  * includes iscsid and iscsi-tcp kernel module loading;
  * includes LIO and loading of relevant kernel modules;
  * An LVM PV is created on top of a loop device with a backing file
  present in $SNAP_COMMON/cinder-lvm.img;
  * A VG is created on top of the PV;
  * LVs are created by Cinder and exported via LIO over iscsi to iscsid
  which hot-plugs new SCSI devices. Those SCSI devices are then
  propagated by Nova to libvirt and QEMU during volume attachment;
* Added post-deployment testing via rally and tempest (via the
  microstack-test snap). A set of tests included into Refstack 2018.02
  is executed (except for object storage tests due to the lack of object
  storage support).

Change-Id: Ic70770095860a57d5e0a55a8a9451f9db6be7448
2020-09-25 13:20:12 +00:00
Pete Vander Giessen
c123af21d3 Strict confinement related fixes
(Not complete strict confinement, but these don't break anything
devmode related, and get us closer to having strict confinement
working.)

Added more needed interfaces to snapcraft.yaml.

Created a wrapper around dnsmasq so that we can run as the snap_daemon
user. Added snap_daemon user to snapcraft.yaml.

Added a utility script for connecting interfaces that don't auto
connect (tools/connect.sh). Not useful for production, but saves a lot
of time when testing.

libvirt no longer uses unix sock group "sudo" (can't run setguid in
strict confinement).

Got rid of "find_missing_plugins" in init script. By the time we
release strict confinement to production, all those plugins will auto
connect.

Change-Id: I8324ac7bd0332c41cac17703eb15d7301e7babf3
2020-04-02 17:31:54 +00:00
Pete Vander Giessen
5611bc9759 Strict confinement (devmode)
Make MicroStack strictly confined, albeit in devmode for now.

Addresses unpredictable breakages with apt package upgrades in eoan
and focal, and sets the stage for a better isolated, less fragile snap
going forward.

We now use layouts to handle libvirt and qemu setting paths at compile
time. This is cleaner than the organize hack.

Moved away from calls to systemctl in init, as a strictly confined
snap cannot call systemctl on a non snappy system.

Disabled call to sysctl to set ipv4_fowarding, as we don't have access
to sysctl in a strictly confined snap. This may break some users, and
we need to figure out a way to address the breakage.

Got rid of questions.shell.shell routine, moving rabbitmq setup into a
bash script instead (it's just cleaner).

Moved keypair creation into launch script, as it's difficult to do
sensible things with keypair creation in the init script, which is
running using sudo, and therefore doesn't have access to
/home/<someuser>/snap

Added (but commented out) code that will check to verify that plugs
are connected before running microstack.init or ovs-vsctl. This code
may go away entirely, as we plan on auto connecting all of our
interfaces, and don't technically need to guard against not having
them connected.

Added temporary local upper-constraints file, to fix an issue where
upstream upper-constraints was breaking pip install by setting a
neutron version. This needs a better long term fix, but works for now.

Closes-bug: 1860660
Change-Id: Iaf1f1482609f05285ed9061317b32e90bffd2da0
2020-03-05 09:31:15 +00:00
Pete Vander Giessen
d262cbed49 Revert "Initial DPDK support"
This reverts commit ce5e82e319.

MicroStack cannot currently install due to a missing ovs-related
library. This is possibly due to recent changes in spacraft, or
possibly due to the workarounds for those changes. Regardless, it
appears that backing out the DPDK changes gets us back to a state
where we can install.

Partial-Bug: 1862911
Change-Id: I060c1a0095470639f9158cb9e9ebe8281a65a678
2020-02-12 20:10:59 +00:00
Nikolay Vinogradov
9019ad3529 Added Filebeat, NRPE and Telegraf to Microstack
- Snapped binary packages of Filebeat, NRPE and Telegraf (disabled by default)
- Added W/A of Telegraf segfault after ELF patching by snapcraft
- Implemented IPMI input tuning for Telegraf
- Allowed to run NRPE as root:root (from custom PPA)
- Implemented Filebeat, NRPE and Telegraf control scripts and config on top of snap-overlay
- Added support for checking Microstack systemd services by NRPE
- Added few generic and Microstack-specific NRPE checks
- Added possibility to override default config paths for the daemons
- Added support for in-band IPMI input to Telegraf
- Stick LMA wrappers and services naming to Microstack conventions
- Increase build timeout in .zuul conf by 30min

Change-Id: I68dbdb11248cf0c1e22e9333af3cf0f88954f557
2020-02-11 14:33:26 +00:00
Zuul
be4e35ab29 Merge "Initial DPDK support" 2020-02-11 01:43:32 +00:00
Pete Vander Giessen
a89f5574c3 Added microstack.remove command
Running microstack.remove will remove the br-ex virtual bridge device,
then uninstall MicroStack.

We do this because we can't use ovs-ctl to remove the bridge as part
of a remove hook, as the Open vSwitch daemons are not running at that
point. The microstack.remove command gives operators a way to cleanly
uninstall the snap, without needing to reboot to get rid of br-ex.

Added test exercising the code to test_basic.py.

Rerranged entry points a bit (moved some things into main.py) to make
code sharing easier, and to prevent a proliferation of entry point
scripts in our root dir.

Change-Id: I9ff25864cd96ada3a9b3da8992c2b33955eff0b4
Closes-Bug: #1852147
2019-12-17 17:11:32 +00:00
Gabor Meszaros
ce5e82e319 Initial DPDK support
Adding initial support for OVS DPDK.

Signed-off-by: Gabor Meszaros <gabor@meszaros.pw>
Change-Id: I1f87188bd8b4b928108674a35e9a9c309e9f6d17
2019-12-09 16:45:11 +01:00
Pete Vander Giessen
1a25e50a17 Made horizon, rabbit and mysql ports configurable
Addresses requests to make it easier to avoid conflicts between the
Horizon dashboard and http services that might already be running on
the machine.

Configurable via snap config. Exposing via arguments to .init and
testing post init configuration is left for a separate PR.

Eventually, these may move to non standard ports by default. This PR
sets the stage for that, but further discussion is needed before we
decide whether to implement.

(This commit also contains a sneaky fix for the username display at the
end of the launch script.)

Closes-Bug: 1814829
Change-Id: If728d6ec8024bca4d3e809637fbdcc03ed4e6934
2019-11-27 15:27:54 +00:00
Pete Vander Giessen
2915caf98e Fixed dashboard_allowed_hosts config setting.
Now happens in a template, just like all the other values, which fixes
an issue where it doesn't get overridden during an upgrade.

Change-Id: Ied84ddc0282c77de6797f90efc8923ae66a9d59e
2019-11-21 08:47:04 +01:00
Pete Vander Giessen
590a7bc3d5 Refresh now sets missing config values
Broke default config settings into a separate script, which the
install hook always called, and post-refresh calls if it can't find
the new config hierarchy.

This fixes issues w/ upgrading from beta to edge.

Change-Id: I6b11109c2a2f6aca142a18c9fa274b332891d5c8
2019-11-21 08:46:41 +01:00
Pete Vander Giessen
f7d2064561 set-br-ex now idempotent
When refreshing a snap, set-br-ex could fail because the iptables
rules that it sets were already setup. We now exit zero if this is
true, which prevents us from breaking on upgrades.

Change-Id: Ibfee98cabfa3e35bf53dbd191de2cf46f3709a51
2019-11-21 08:46:15 +01:00
Pete Vander Giessen
5404a261aa Clustering prototype
This enables basic clustering functionality. We add:

tools/cluster/cluster/daemon.py: A server that handles validation of
cluster passwords.

tools/cluster/cluster/client.py: A client for this server.

Important Note: This prototype does not support TLS, and the
functionality in the client and server is basic. Before we roll
clustering out to production, we need to have those two chat over TLS,
and be much more careful about verifying credentials.

Also included ...

Various fixes and changes to the init script and config templates to
support cluster configuration, and allow for the fact that we may have
endpoint references for two network ips.

Updates to snapcraft.yaml, adding the new tooling.

A more formalized config infrastructure. It's still a TODO to move the
specification out of the implicit definition in the install hook, and
into a nice, explicit, well documented yaml file.

Added nesting to the Question classes in the init script, as well as
strings pointing at config keys, rather than having the config be
implicitly indicated by the Question subclass' name. (This allows us
to put together a config spec that doesn't require the person reading
the spec to understand what Questions are, and how they are
implemented.)

Renamed and unified the "unit" and "lint" tox environments, to allow
for the multiple Python tools that we want to lint and test.

Added hooks in the init script to make it possible to do automated
testing, and added an automated test for a cluster. Run with "tox -e
cluster".

Added cirros image to snap, to work around sporadic issues downloading
it from download.cirros.net.

Removed ping logic from snap, to workaround failures in gate. Need to
add it back in once we fix them.

Change-Id: I44ccd16168a7ed41486464df8c9e22a14d71ccfd
2019-11-04 13:03:41 +00:00
Pete Vander Giessen
0399955cf1 Ported and updated launch script
Moved security rules and keypair creation into init first.

Launch script now takes image name as positional argument, and name of
instance as a named argument. This makes it work more like launch in
other Canonical tools.

Written in Python, for ease of maintenance.

--retry and --wait args allow it to behave like tests expect it to,
while humans will get a much more intuitive (and much less noisy)
experience.

Also increased time we wait for a ping on the host, to allow for
slower, pure qemu, emulation times, and bring it in line with what
Tempest does in similar situations.

Change-Id: I11dcc098012468e9c88dcc7af78cde6920f31ecd
2019-10-23 01:09:42 +00:00
Pete Vander Giessen
dfd1d5ec68 Added testing for control nodes
Ported basic-test.sh to test_basic.py, and folded in
test_horizonlogin.py.

Made a testing framework for shared components.

Added test_control.py

Got rid of default .stestr.conf, as we're going to have multiple tests
running, and one conf is confusing.

Manually ordering functional tests for now, as stestr noms too much
output, and runs things in parallel, which doesn't work for our
functional tests.

Skipping compute node test for now, as it won't work until we can
connect to a control node with databases and such.

Moved very-basic-test.sh to tools/make-a-microstack.sh. It's really
more of a tool for manual testing than an automated test.

Added test-requirements and updated gitignore.

Moved auto-detection of kvm extensions to init, rather than test, as
it makes more sense there.

Change-Id: Iba7f7fe07cbb066790f802cf2a7c87c68994062c
2019-10-16 15:44:38 +00:00
Pete Vander Giessen
7525ebcded Enable loading and saving of question answers.
This lays the groundwork for interactive init, as well as being able
to specify control and compute nodes.

Added preliminary config lists for control and compute nodes. Added
appropriate default snapctl config settings in install script.

Also changed "binary" questions to "boolean" questions, as that's
better wording, and it means that my docstrings are not a confusing
mix of "boolean" and "binary" when I forget which term I used.

Snuck in a fix for the "basic" testing environment -- it was missing
the Python requirements, and was therefore failing!

Change-Id: I7f95ab68f924fa4d4280703c372b807cc7c77758
2019-10-10 15:14:00 +00:00
Pete Vander Giessen
b4f90c6eca Make logging better
Move logging output for most services to systemd.

Add a hook in snap.openstack to tell OpenStack services to wait
until we set database.ready in the snap config before starting. This
prevents spamming systemd with error messages before we run
microstack.init (See matching PR against snap.openstack, coming soon.)

Incidentally fix issue w/ the way that shell.py was running
CalledProcessError and parsing output.

Order patches part after uca-sources, to work around an issue we
discovered with apt update while those two parts are running in
parallel. (python-apt segfaults, and no fun is had by anyone.)

Remaining gaps in our logging: systemd still displays some errors
during init, which might be fixable with further ordering of snapctl
start invocations. We're also relying on MySQL and RabbitMQ log output
to know when those services are started, so we haven't moved their
output to systemd just yet.

Dropped in a fix to work w/ updated version of snap.openstack.

Change-Id: I130ed730c14ab35b8b677b9c9f573fa6fe1e8f13
2019-10-08 13:13:23 +00:00
Pete Vander Giessen
73d39dc8ce Make Horizon Dashboard settings operator overrideable.
We want to allow operators to override Horizon's default
settings. This involves moving local_settings.d out of the read only
snap filesystem, and into $SNAP_COMMON. This is a little bit tricky.

First, we patch settings.py and local_settings.py as we're building the
snap, to include a LOCAL_PATH in $SNAP_COMMON.

Then, we add a template with the rest of our default overrides,
and write it out to $SNAP_COMMON/horizon/local_settings.d

Finally we tweak our tests so that we can give our overrides a
spin. As a bonus, this makes test_horizonglogin.py a lot easier to run
in our multipass testing scenario!

`tox -e basic` now also runs selenium tests, as well.

Change-Id: Ic0ce18cfa1b97a93191da749095d8aa2270d5aeb
2019-09-24 18:27:39 +00:00
Pete Vander Giessen
0b4c7a22a3 Fix horizon dashboard
Port the python2.7 local settings overrides to the python3.6 directory
structure.

Move all local_settings.py overrides into _05_snap_tweaks.py as part
of troubleshooting some remaining problems. Everything is more
organized and functional now :-)

Added selenium tests.

Change-Id: I54923e1dc9c7ffa47c2ef6fb90ea9d224b0d2eee
2019-09-16 13:25:39 +00:00
Pete Vander Giessen
93f412fc93 Better init script
Written in Python. Easier to maintain. Easier to make interactive.

Change-Id: Ib579b43c1564b55165de5c2f3d20387122448b19
2019-08-23 16:30:31 +00:00
Pete Vander Giessen
92a6ea8dc7 Broke configuration out into a manual step.
Renamed the old and outdated "configure-openstack" script to "init.sh"

Updated init.sh and folded most of the configure hook into it.

Removed database installation step from install hook.

We can now install microstack without a database dump, which helps
immensely in updating. And we have a logical place to put additional
configuraiton, including some of the manual steps in DEMO.md, which
could be scripted if we gave users a chance to skip the system changes
that they wanted to skip.

Also updated README and DEMO file to match new flow. Updated test
files.

Future cleanup and features documented in Trello, but not included in
this PR, which is big enough already :-)

Change-Id: I8d926a8b463124494ddb7a4696adbe86f89db7d5
2019-08-01 21:12:05 +00:00
Pete Vander Giessen
95efd789b4 Added snap configuration.
Moved openstack env variables out of snapcraft.yaml, and into a
"microtack.rc".

Made openstack-wrapper, which auto sources microstack.rc.

Key off of revised snap.openstack, which supports snapctl config
values in templates. Moved a lot of the overlay into
snap-overlay/templates, and added config values to them. Also writes
templates just once, when you run snap-openstack setup.

Change-Id: Ib0f2e3fc97f491d9ed9dfbafc61dc8e27a8a2b48
2019-07-25 16:37:51 +00:00
Pete Vander Giessen
8ea5dc8679 Added automated testing via tox and zuul.
Tweaked tests/basic_test.sh functional test so that it no longer
requires multpass, and can run without kvm cpu extensions being
enabled (not all machines in the game have cpu extensions.)

Added tox.ini, wrapper script for building and installation, etc.

Change-Id: I968116dd7bec412a55813c896d60cfc86c7070db
2019-07-24 08:04:38 +01:00
Pete Vander Giessen
7b4e61201b
Cruft cleanup. (#75)
* Cruft cleanup.

Organized snapcraft.yaml better, and got rid of unecessary bits.

* Cleaned up scripts.

Moved launch.sh into snap-overlay, to be more consistent with
everything else.

Got rid of outdated and confusing configure-the-things script.

* Clarified a comment.

* Got rid of erroneous config file.

* Undeleted config file -- wasn't erroneous!
2019-06-25 13:45:00 -04:00
Pete Vander Giessen
f45bfd913d
Refactor snap to work with core18. (#73)
Refactor snap to work with core18.

Giving the snapcraft.yaml a base property helps tremendously with the
efficiency of the build process, and I believe that it puts us in a
better position to reliably support non Ubuntu distros going forward.

This also bases us on long supported bionic libraries, and gives us a
nice place to work from as we add Python 3 and Stein support, as well
as general polish and fixes.
2019-06-25 11:31:42 -04:00
Ryan Beisner
dfeb664207
Merge pull request #63 from CanonicalLtd/bug/1823016-default-dns
Added default dns server for microstack guests.
2019-04-11 15:08:25 -05:00
Pete Vander Giessen
dfdd85ecb4 Added newline at end of dhcp_agent.ini 2019-04-10 14:57:44 -04:00
Pete Vander Giessen
d8c34fa1dc Added default dns server for microstack guests. 2019-04-10 14:56:30 -04:00