183 Commits

Author SHA1 Message Date
Pete Vander Giessen
e0fd42e55c Updated "Building MicroStack" section of CONTRIBUTING.md
Now reflects current build process, and has correct url to source code.

Also changed references to "microstack" and "Microstack" to
"MicroStack".

Change-Id: I60f6a28c3e47ed785692e59c0de79e298e437900
Closes-Bug: 1846979
2019-11-14 17:36:25 +00:00
Zuul
2f5847d6ad Merge "Allow off-host access to horizon dashboard by default." 2019-11-06 07:42:39 +00:00
Pete Vander Giessen
0824a570ec Clean up tox.ini a bit
Remove some of the redundancy in tox.ini.

Fixed some lint issues that weren't caught before due to gaps in the
linting coverage.

I think that there's more work to be done here, but this does make
things better than they were before.

Change-Id: I82487dbb9366f3de16b25615bd081b6315671655
2019-11-05 15:36:24 +00:00
Pete Vander Giessen
9e6b8c1e21 Allow off-host access to horizon dashboard by default.
Added a question which allows off host access to horizon
dashboard. Activated it by default, as that's probably what people are
going to actually want.

Change-Id: I0d5bccb3b2eb2b409072d8ae5f8b923942386119
2019-11-05 13:34:32 +00:00
Zuul
a89cb36883 Merge "Fix Eoan." 2019-11-05 13:26:53 +00:00
Pete Vander Giessen
cb940157ab Fix Eoan.
Moved to pure Python where clib conflicts arose in using command line
tools.

Fixed erroneous assumptions about the presence and reliability of a
$HOME variable while running init.

Added tests specific to eoan, disco and xenial. They are not yet part
of the gate.

Change-Id: I2fc74fcc2ae9876442bb87a3446aef48d0428f2f
2019-11-05 03:22:42 +00:00
Peter Matulis
ec52230cc7 Update README since doc publication
Change-Id: I9d2d4f8d47d1c2b560e4d82abbd30a9cbf2d4fd4
2019-11-04 15:23:51 +00:00
Pete Vander Giessen
5404a261aa Clustering prototype
This enables basic clustering functionality. We add:

tools/cluster/cluster/daemon.py: A server that handles validation of
cluster passwords.

tools/cluster/cluster/client.py: A client for this server.

Important Note: This prototype does not support TLS, and the
functionality in the client and server is basic. Before we roll
clustering out to production, we need to have those two chat over TLS,
and be much more careful about verifying credentials.

Also included ...

Various fixes and changes to the init script and config templates to
support cluster configuration, and allow for the fact that we may have
endpoint references for two network ips.

Updates to snapcraft.yaml, adding the new tooling.

A more formalized config infrastructure. It's still a TODO to move the
specification out of the implicit definition in the install hook, and
into a nice, explicit, well documented yaml file.

Added nesting to the Question classes in the init script, as well as
strings pointing at config keys, rather than having the config be
implicitly indicated by the Question subclass' name. (This allows us
to put together a config spec that doesn't require the person reading
the spec to understand what Questions are, and how they are
implemented.)

Renamed and unified the "unit" and "lint" tox environments, to allow
for the multiple Python tools that we want to lint and test.

Added hooks in the init script to make it possible to do automated
testing, and added an automated test for a cluster. Run with "tox -e
cluster".

Added cirros image to snap, to work around sporadic issues downloading
it from download.cirros.net.

Removed ping logic from snap, to workaround failures in gate. Need to
add it back in once we fix them.

Change-Id: I44ccd16168a7ed41486464df8c9e22a14d71ccfd
2019-11-04 13:03:41 +00:00
Pete Vander Giessen
0399955cf1 Ported and updated launch script
Moved security rules and keypair creation into init first.

Launch script now takes image name as positional argument, and name of
instance as a named argument. This makes it work more like launch in
other Canonical tools.

Written in Python, for ease of maintenance.

--retry and --wait args allow it to behave like tests expect it to,
while humans will get a much more intuitive (and much less noisy)
experience.

Also increased time we wait for a ping on the host, to allow for
slower, pure qemu, emulation times, and bring it in line with what
Tempest does in similar situations.

Change-Id: I11dcc098012468e9c88dcc7af78cde6920f31ecd
2019-10-23 01:09:42 +00:00
James Page
e007da2fb9 Align ovs version with Stein UCA
Bump ovs to 2.11.1 to ensure alignment with Stein Cloud Archive.

This also means that the patches for OVS can be dropped as the
change was included in the 2.11.1 release.

Change-Id: I9896f1dbd80a2344565a752d6c697f0a3219c210
2019-10-18 15:40:41 +00:00
Zuul
78ba031251 Merge "Fix up ssh key and improve wording" 2019-10-17 19:31:54 +00:00
Pete Vander Giessen
dfd1d5ec68 Added testing for control nodes
Ported basic-test.sh to test_basic.py, and folded in
test_horizonlogin.py.

Made a testing framework for shared components.

Added test_control.py

Got rid of default .stestr.conf, as we're going to have multiple tests
running, and one conf is confusing.

Manually ordering functional tests for now, as stestr noms too much
output, and runs things in parallel, which doesn't work for our
functional tests.

Skipping compute node test for now, as it won't work until we can
connect to a control node with databases and such.

Moved very-basic-test.sh to tools/make-a-microstack.sh. It's really
more of a tool for manual testing than an automated test.

Added test-requirements and updated gitignore.

Moved auto-detection of kvm extensions to init, rather than test, as
it makes more sense there.

Change-Id: Iba7f7fe07cbb066790f802cf2a7c87c68994062c
2019-10-16 15:44:38 +00:00
Peter Matulis
0587b9c4b1 Fix up ssh key and improve wording
We will probably defer to the documentation once it's
ready and keep a more streamlined storefront here.

Change-Id: Ifb5dc23440eca6f4a706457aba7990299e5faf7b
Closes-Bug: 1841791
2019-10-15 18:25:06 -04:00
Pete Vander Giessen
7525ebcded Enable loading and saving of question answers.
This lays the groundwork for interactive init, as well as being able
to specify control and compute nodes.

Added preliminary config lists for control and compute nodes. Added
appropriate default snapctl config settings in install script.

Also changed "binary" questions to "boolean" questions, as that's
better wording, and it means that my docstrings are not a confusing
mix of "boolean" and "binary" when I forget which term I used.

Snuck in a fix for the "basic" testing environment -- it was missing
the Python requirements, and was therefore failing!

Change-Id: I7f95ab68f924fa4d4280703c372b807cc7c77758
2019-10-10 15:14:00 +00:00
Pete Vander Giessen
b4f90c6eca Make logging better
Move logging output for most services to systemd.

Add a hook in snap.openstack to tell OpenStack services to wait
until we set database.ready in the snap config before starting. This
prevents spamming systemd with error messages before we run
microstack.init (See matching PR against snap.openstack, coming soon.)

Incidentally fix issue w/ the way that shell.py was running
CalledProcessError and parsing output.

Order patches part after uca-sources, to work around an issue we
discovered with apt update while those two parts are running in
parallel. (python-apt segfaults, and no fun is had by anyone.)

Remaining gaps in our logging: systemd still displays some errors
during init, which might be fixable with further ordering of snapctl
start invocations. We're also relying on MySQL and RabbitMQ log output
to know when those services are started, so we haven't moved their
output to systemd just yet.

Dropped in a fix to work w/ updated version of snap.openstack.

Change-Id: I130ed730c14ab35b8b677b9c9f573fa6fe1e8f13
2019-10-08 13:13:23 +00:00
Pete Vander Giessen
73d39dc8ce Make Horizon Dashboard settings operator overrideable.
We want to allow operators to override Horizon's default
settings. This involves moving local_settings.d out of the read only
snap filesystem, and into $SNAP_COMMON. This is a little bit tricky.

First, we patch settings.py and local_settings.py as we're building the
snap, to include a LOCAL_PATH in $SNAP_COMMON.

Then, we add a template with the rest of our default overrides,
and write it out to $SNAP_COMMON/horizon/local_settings.d

Finally we tweak our tests so that we can give our overrides a
spin. As a bonus, this makes test_horizonglogin.py a lot easier to run
in our multipass testing scenario!

`tox -e basic` now also runs selenium tests, as well.

Change-Id: Ic0ce18cfa1b97a93191da749095d8aa2270d5aeb
2019-09-24 18:27:39 +00:00
Pete Vander Giessen
c5f679a673 Quieter init script
Piped output of subprocess commands to log.

Set default log level to INFO.

Change-Id: I583af3f6812195a9061a76b611714aced7c8861f
2019-09-18 12:15:00 +00:00
Zuul
fb4498b331 Merge "Fix horizon dashboard" 2019-09-18 12:10:00 +00:00
Pete Vander Giessen
0b4c7a22a3 Fix horizon dashboard
Port the python2.7 local settings overrides to the python3.6 directory
structure.

Move all local_settings.py overrides into _05_snap_tweaks.py as part
of troubleshooting some remaining problems. Everything is more
organized and functional now :-)

Added selenium tests.

Change-Id: I54923e1dc9c7ffa47c2ef6fb90ea9d224b0d2eee
2019-09-16 13:25:39 +00:00
Pete Vander Giessen
b0501437bb Updated README.
Added a better summary of what MicroStack is, and what you might do
with it.

Change-Id: Ic2e1775209fc61ce48b4fe23f67596e5e4d1aaaf
2019-09-16 06:54:14 +00:00
Pete Vander Giessen
30f09c0c9a Upgrade to Python3/Stein
Move openstack-projects part from python2 to python3.

Add cloud archive.

Update qemu and libvirt versions to those from cloud archive (they
work with python3, while the distro packages versions don't).

Switch from rocky to stein.

Fetch libvirt and qemu sources via "apt source". Gets rid of sub
version hard coding in snapcraft.

Update hard coded references in tests to rocky from stein.

Change-Id: Idb38717998a13feaaf0782e880e540f28bc452a8
2019-08-30 17:15:05 +00:00
Pete Vander Giessen
93f412fc93 Better init script
Written in Python. Easier to maintain. Easier to make interactive.

Change-Id: Ib579b43c1564b55165de5c2f3d20387122448b19
2019-08-23 16:30:31 +00:00
Pete Vander Giessen
b5835060ca Added "basic" option to tox test environments
Just runs `tests/basic_test.sh -m` without building the snap
first. Useful for when you want to re-run the basic_test to verify
that a sporadic issue has been fixed, without rebuilding the snap each
time.

Counterargument: you could always just run `tests/basic_test.sh -m` on
its own. I find it easier to keep one test command in my fingers,
however, rather than having to switch between "tox plus args" and
"something else plus args"

Change-Id: I5ec28864ed41a4e11726df8aee1d31b8ef097f03
2019-08-21 14:13:04 +00:00
Pete Vander Giessen
92a6ea8dc7 Broke configuration out into a manual step.
Renamed the old and outdated "configure-openstack" script to "init.sh"

Updated init.sh and folded most of the configure hook into it.

Removed database installation step from install hook.

We can now install microstack without a database dump, which helps
immensely in updating. And we have a logical place to put additional
configuraiton, including some of the manual steps in DEMO.md, which
could be scripted if we gave users a chance to skip the system changes
that they wanted to skip.

Also updated README and DEMO file to match new flow. Updated test
files.

Future cleanup and features documented in Trello, but not included in
this PR, which is big enough already :-)

Change-Id: I8d926a8b463124494ddb7a4696adbe86f89db7d5
2019-08-01 21:12:05 +00:00
Zuul
de77b4a426 Merge "Added snap configuration." 2019-07-25 18:21:32 +00:00
Pete Vander Giessen
95efd789b4 Added snap configuration.
Moved openstack env variables out of snapcraft.yaml, and into a
"microtack.rc".

Made openstack-wrapper, which auto sources microstack.rc.

Key off of revised snap.openstack, which supports snapctl config
values in templates. Moved a lot of the overlay into
snap-overlay/templates, and added config values to them. Also writes
templates just once, when you run snap-openstack setup.

Change-Id: Ib0f2e3fc97f491d9ed9dfbafc61dc8e27a8a2b48
2019-07-25 16:37:51 +00:00
Zuul
1a34307f3b Merge "Added core18 snap path to PATH." 2019-07-25 14:24:00 +00:00
Pete Vander Giessen
59551ca2cd Added support for testing in multipass vms back to basic_test
Script now takes a "-m" argument. This helps to test locally in a
fresh environment.

Change-Id: I7848070cfdf809f084a107f23112fcfc71e11bf4
2019-07-24 22:59:15 +00:00
Pete Vander Giessen
65441d5b2a Added core18 snap path to PATH.
This should fix the issue where the configure hook doesn't run
properly on systems without netcat installed.

Also got rid of call to install netcat in tools/install_and_build.sh.

Doubled timeout in .zuul.yaml to deal with timeout that happened
during this test (I think that we were pushing the hour timeout as it was).

Change-Id: Ie9cd729109a871c32c12108163d22008d91155f9
2019-07-24 19:59:43 +00:00
Pete Vander Giessen
8ea5dc8679 Added automated testing via tox and zuul.
Tweaked tests/basic_test.sh functional test so that it no longer
requires multpass, and can run without kvm cpu extensions being
enabled (not all machines in the game have cpu extensions.)

Added tox.ini, wrapper script for building and installation, etc.

Change-Id: I968116dd7bec412a55813c896d60cfc86c7070db
2019-07-24 08:04:38 +01:00
James Page
96c94f0cf5 Update libvirt package version (#79) 2019-07-15 11:14:26 -04:00
Pete Vander Giessen
2af4747bf4 Got rid of .travis.yml 2019-07-02 10:14:55 -04:00
Pete Vander Giessen
8436824c20
Renamed travis.yaml to travis.yml. (#76) 2019-06-26 11:25:21 -04:00
Pete Vander Giessen
7b4e61201b
Cruft cleanup. (#75)
* Cruft cleanup.

Organized snapcraft.yaml better, and got rid of unecessary bits.

* Cleaned up scripts.

Moved launch.sh into snap-overlay, to be more consistent with
everything else.

Got rid of outdated and confusing configure-the-things script.

* Clarified a comment.

* Got rid of erroneous config file.

* Undeleted config file -- wasn't erroneous!
2019-06-25 13:45:00 -04:00
Pete Vander Giessen
f45bfd913d
Refactor snap to work with core18. (#73)
Refactor snap to work with core18.

Giving the snapcraft.yaml a base property helps tremendously with the
efficiency of the build process, and I believe that it puts us in a
better position to reliably support non Ubuntu distros going forward.

This also bases us on long supported bionic libraries, and gives us a
nice place to work from as we add Python 3 and Stein support, as well
as general polish and fixes.
2019-06-25 11:31:42 -04:00
Tim McNamara
575a7ee54c Formatting improvements; add Juju discourse link (#69)
This commit makes a few stylistic changes which aid readability.
2019-05-17 17:40:56 -04:00
Nikolay Vinogradov
e5127e0012 Fix snapcraft build after qemu_2.5 source package rebuilding (#71)
qemu_2.5+dfsg-5ubuntu10.36.debian.tar.xz (404) -> 10.38
2019-05-16 11:12:37 -04:00
Pete Vander Giessen
343f6b4eb9 Sneaky update of DEMO.md, to reflect live changes made during the workshop. 2019-05-03 11:32:31 -06:00
Ryan Beisner
30159a1f1d
Merge pull request #68 from CanonicalLtd/move-images-location
Moved cirros image to /var/snap/microstack/common/images
2019-04-22 14:34:29 -05:00
Pete Vander Giessen
6e13cc4112 Moved cirros image to /var/snap/microstack/common/images
This way, it will get cleaned up when we uninstall the snap.

Also added test hooks to test upgrades, to verify that we aren't
breaking existing snaps.
2019-04-19 17:17:29 -04:00
Ryan Beisner
d814e9e122
Merge pull request #64 from CanonicalLtd/demo-docs
Added DEMO.md
2019-04-19 11:07:58 -05:00
Pete Vander Giessen
6475d60d19 Updated language to point to beta rather than edge. 2019-04-19 11:29:33 -04:00
Pete Vander Giessen
fbb44e4ab4 Got rid of WIP. 2019-04-19 10:40:54 -04:00
Pete Vander Giessen
c1efa221f8
Fixed test cleanup. (#65) 2019-04-18 09:26:16 -04:00
Ryan Beisner
dfeb664207
Merge pull request #63 from CanonicalLtd/bug/1823016-default-dns
Added default dns server for microstack guests.
2019-04-11 15:08:25 -05:00
Ryan Beisner
bf1927be79
Merge pull request #62 from CanonicalLtd/bug/fix-slow-vms
Switched back to kvm and host-passthrough vm defaults.
2019-04-11 15:08:10 -05:00
Pete Vander Giessen
6dc3155cbb spelling. 2019-04-11 15:37:56 -04:00
Pete Vander Giessen
d889e3595a Updated minimum requirements. 2019-04-11 15:15:00 -04:00
Pete Vander Giessen
c3a5a00898 Added DEMO.md
These are instructions for runnning the demo that we're planning on
running at the Denver Openstack summit. It walks the user through
deploying microstack, and then deploying a functioning kubernetes
cloud on top of it.
2019-04-11 15:12:44 -04:00
Pete Vander Giessen
dfdd85ecb4 Added newline at end of dhcp_agent.ini 2019-04-10 14:57:44 -04:00