x/microstack
Code Issues Proposed changes

Clustering prototype

Browse Source 
This enables basic clustering functionality. We add:

tools/cluster/cluster/daemon.py: A server that handles validation of
cluster passwords.

tools/cluster/cluster/client.py: A client for this server.

Important Note: This prototype does not support TLS, and the
functionality in the client and server is basic. Before we roll
clustering out to production, we need to have those two chat over TLS,
and be much more careful about verifying credentials.

Also included ...

Various fixes and changes to the init script and config templates to
support cluster configuration, and allow for the fact that we may have
endpoint references for two network ips.

Updates to snapcraft.yaml, adding the new tooling.

A more formalized config infrastructure. It's still a TODO to move the
specification out of the implicit definition in the install hook, and
into a nice, explicit, well documented yaml file.

Added nesting to the Question classes in the init script, as well as
strings pointing at config keys, rather than having the config be
implicitly indicated by the Question subclass' name. (This allows us
to put together a config spec that doesn't require the person reading
the spec to understand what Questions are, and how they are
implemented.)

Renamed and unified the "unit" and "lint" tox environments, to allow
for the multiple Python tools that we want to lint and test.

Added hooks in the init script to make it possible to do automated
testing, and added an automated test for a cluster. Run with "tox -e
cluster".

Added cirros image to snap, to work around sporadic issues downloading
it from download.cirros.net.

Removed ping logic from snap, to workaround failures in gate. Need to
add it back in once we fix them.

Change-Id: I44ccd16168a7ed41486464df8c9e22a14d71ccfd
This commit is contained in:
Pete Vander Giessen 2019-10-15 21:08:33 +00:00
parent 0399955cf1
commit 5404a261aa
41 changed files with 941 additions and 205 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
README.md
View File

@ -123,3 +123,40 @@ sudo snap enable microstack
## Raising a Bug
Please report bugs to the microstack project on launchpad: https://bugs.launchpad.net/microstack
## Clustering Preview
The latests --edge version of the clustering snap contains a preview of microstack's clustering functionality. If you're interested in building a small "edge" cloud with microstack, please take a look at the notes below. Keep in mind that this is preview functionality. Interfaces may not be stable, and the security of the preview is light, and not suitable for production use!
To setup a cluster, you first must setup a control node. Do so with the following commands:
```
sudo snap install microstack
sudo microstack.init
```
Answer the questions in the interactive prompt as follows:
```
Clustering: yes
Role: control
IP Address: Note and accept the default
```
On a second machine, run:
```
sudo snap install microstack
sudo microstack.init
```
Answer the questions in the interactive prompt as follows:
```
Setup clustering: yes
Role: compute
Control IP: the ip address noted above
Compute IP: accept the default
```
You should now have a small, two node cloud, with the first node serving as both the control plane and a hypvervisor, and the second node serving as a hypervisor. You can create vms on both.

2
snap-overlay/bin/setup-br-ex
View File

@ -9,7 +9,7 @@
set -ex
extcidr=$(snapctl get questions.ext-cidr)
extcidr=$(snapctl get config.network.ext-cidr)
# Create external integration bridge
ovs-vsctl --retry --may-exist add-br br-ex

BIN
snap-overlay/images/cirros-0.4.0-x86_64-disk.img Normal file
View File

Binary file not shown.

15
snap-overlay/snap-openstack.yaml
View File

@ -52,15 +52,22 @@ setup:
neutron.keystone.conf.j2: "{snap_common}/etc/neutron/neutron.conf.d/keystone.conf"
neutron.nova.conf.j2: "{snap_common}/etc/neutron/neutron.conf.d/nova.conf"
neutron.database.conf.j2: "{snap_common}/etc/neutron/neutron.conf.d/database.conf"
neutron.conf.d.rabbitmq.conf.j2: "{snap_common}/etc/neutron/neutron.conf.d/rabbitmq.conf"
chmod:
"{snap_common}/instances": 0755
"{snap_common}/etc/microstack.rc": 0644
snap-config-keys:
ospassword: 'questions.os-password'
extgateway: 'questions.ext-gateway'
extcidr: 'questions.ext-cidr'
dns: 'questions.dns'
ospassword: 'config.credentials.os-password'
nova_password: 'config.credentials.nova-password'
neutron_password: 'config.credentials.neutron-password'
placement_password: 'config.credentials.placement-password'
glance_password: 'config.credentials.glance-password'
control_ip: 'config.network.control-ip'
compute_ip: 'config.network.compute-ip'
extgateway: 'config.network.ext-gateway'
extcidr: 'config.network.ext-cidr'
dns: 'config.network.dns'
entry_points:
keystone-manage:
binary: "{snap}/bin/keystone-manage"

10
snap-overlay/templates/clustering-nginx.conf Normal file
View File

@ -0,0 +1,10 @@
server {
listen 8011;
error_log syslog:server=unix:/dev/log;
access_log syslog:server=unix:/dev/log;
location / {
include {{ snap }}/usr/conf/uwsgi_params;
uwsgi_param SCRIPT_NAME '';
uwsgi_pass unix://{{ snap_common }}/run/keystone-api.sock;
}
}

8
snap-overlay/templates/glance.conf.d.keystone.conf.j2
View File

@ -1,13 +1,13 @@
[keystone_authtoken]
auth_uri = http://{{ extgateway }}:5000
auth_url = http://{{ extgateway }}:5000
memcached_servers = {{ extgateway }}:11211
auth_uri = http://{{ control_ip }}:5000
auth_url = http://{{ control_ip }}:5000
memcached_servers = {{ control_ip }}:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = glance
password = glance
password = {{ glance_password }}
[paste_deploy]
flavor = keystone

2
snap-overlay/templates/glance.database.conf.j2
View File

@ -1,2 +1,2 @@
[database]
connection = mysql+pymysql://glance:glance@{{ extgateway }}/glance
connection = mysql+pymysql://glance:glance@{{ control_ip }}/glance

2
snap-overlay/templates/keystone.database.conf.j2
View File

@ -1,2 +1,2 @@
[database]
connection = mysql+pymysql://keystone:keystone@{{ extgateway }}/keystone
connection = mysql+pymysql://keystone:keystone@{{ control_ip }}/keystone

2
snap-overlay/templates/microstack.rc.j2
View File

@ -3,7 +3,7 @@ export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD={{ ospassword }}
export OS_AUTH_URL=http://{{ extgateway }}:5000
export OS_AUTH_URL=http://{{ control_ip }}:5000
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

2
snap-overlay/templates/neutron.conf.d.rabbitmq.conf.j2 Normal file
View File

@ -0,0 +1,2 @@
[DEFAULT]
transport_url = rabbit://openstack:rabbitmq@{{ control_ip }}

2
snap-overlay/templates/neutron.database.conf.j2
View File

@ -1,2 +1,2 @@
[database]
connection = mysql+pymysql://neutron:neutron@{{ extgateway }}/neutron
connection = mysql+pymysql://neutron:neutron@{{ control_ip }}/neutron

8
snap-overlay/templates/neutron.keystone.conf.j2
View File

@ -2,12 +2,12 @@
auth_strategy = keystone
[keystone_authtoken]
auth_uri = http://{{ extgateway }}:5000
auth_url = http://{{ extgateway }}:5000
memcached_servers = {{ extgateway }}:11211
auth_uri = http://{{ control_ip }}:5000
auth_url = http://{{ control_ip }}:5000
memcached_servers = {{ control_ip }}:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = neutron
password = neutron
password = {{ neutron_password }}

4
snap-overlay/templates/neutron.nova.conf.j2
View File

@ -3,11 +3,11 @@ notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
[nova]
auth_url = http://{{ extgateway }}:5000
auth_url = http://{{ control_ip }}:5000
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = microstack
project_name = service
username = nova
password = nova
password = {{ nova_password }}

4
snap-overlay/templates/nova.conf.d.database.conf.j2
View File

@ -1,5 +1,5 @@
[database]
connection = mysql+pymysql://nova:nova@{{ extgateway }}/nova
connection = mysql+pymysql://nova:nova@{{ control_ip }}/nova
[api_database]
connection = mysql+pymysql://nova_api:nova_api@{{ extgateway }}/nova_api
connection = mysql+pymysql://nova_api:nova_api@{{ control_ip }}/nova_api

2
snap-overlay/templates/nova.conf.d.glance.conf.j2
View File

@ -1,2 +1,2 @@
[glance]
api_servers = http://{{ extgateway }}:9292
api_servers = http://{{ control_ip }}:9292

8
snap-overlay/templates/nova.conf.d.keystone.conf.j2
View File

@ -1,13 +1,13 @@
[keystone_authtoken]
auth_uri = http://{{ extgateway }}:5000
auth_url = http://{{ extgateway }}:5000
memcached_servers = {{ extgateway }}:11211
auth_uri = http://{{ control_ip }}:5000
auth_url = http://{{ control_ip }}:5000
memcached_servers = {{ control_ip }}:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = nova
password = {{ nova_password }}
[paste_deploy]
flavor = keystone

8
snap-overlay/templates/nova.conf.d.neutron.conf.j2
View File

@ -1,13 +1,13 @@
[neutron]
url = http://{{ extgateway }}:9696
auth_url = http://{{ extgateway }}:5000
memcached_servers = {{ extgateway }}:11211
url = http://{{ control_ip }}:9696
auth_url = http://{{ control_ip }}:5000
memcached_servers = {{ control_ip }}:11211
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = microstack
project_name = service
username = neutron
password = neutron
password = {{ neutron_password }}
service_metadata_proxy = True
metadata_proxy_shared_secret = supersecret

4
snap-overlay/templates/nova.conf.d.nova-placement.conf.j2
View File

@ -4,6 +4,6 @@ project_domain_name = default
project_name = service
auth_type = password
user_domain_name = default
auth_url = http://{{ extgateway }}:5000
auth_url = http://{{ control_ip }}:5000
username = placement
password = placement
password = {{ placement_password }}

2
snap-overlay/templates/nova.conf.d.rabbitmq.conf.j2
View File

@ -1,2 +1,2 @@
[DEFAULT]
transport_url = rabbit://openstack:rabbitmq@{{ extgateway }}
transport_url = rabbit://openstack:rabbitmq@{{ control_ip }}

63
snap/hooks/install
View File

@ -1,29 +1,62 @@
#!/bin/bash
set -ex
# Config
# Set default answers to the questions that microstack.init asks.
# Initialize Config
# TODO: put this in a nice yaml format, and parse it.
# General snap-wide settings
snapctl set \
questions.ip-forwarding=true \
questions.dns=1.1.1.1 \
questions.ext-gateway=10.20.20.1 \
questions.ext-cidr=10.20.20.1/24 \
questions.os-password=keystone \
questions.rabbit-mq=true \
questions.database-setup=true \
questions.nova-setup=true \
questions.neutron-setup=true \
questions.glance-setup=true \
questions.key-pair="id_microstack" \
questions.security-rules=true \
questions.post-setup=true \
config.clustered=false \
config.post-setup=true \
;
# Networking related settings.
snapctl set \
config.network.dns=1.1.1.1 \
config.network.ext-gateway=10.20.20.1 \
config.network.control-ip=10.20.20.1 \
config.network.compute-ip=10.20.20.1 \
config.network.ext-cidr=10.20.20.1/24 \
config.network.security-rules=true \
;
# Passwords, certs, etc.
snapctl set \
config.credentials.os-password=keystone \
config.credentials.key-pair=id_microstack \
config.credentials.nova-password=nova \
config.credentials.neutron-password=neutron \
config.credentials.placement-password=placement \
config.credentials.glance-password=glance \
;
# Host optimizations and fixes.
snapctl set \
config.host.ip-forwarding=true \
config.host.check-qemu=true \
;
# Enable or disable groups of services.
snapctl set \
config.services.control-plane=true \
config.services.hypervisor=true \
;
# Clustering roles
snapctl set \
cluster.role=control \
cluster.password=null \
;
# MySQL snapshot for speedy install
# snapshot is a mysql data dir with
# rocky keystone,nova,glance,neutron dbs.
mkdir -p ${SNAP_COMMON}/lib
# Put cirros (and potentially other) images in a user writeable place.
mkdir -p ${SNAP_COMMON}/images
cp ${SNAP}/images/* ${SNAP_COMMON}/images/
# Install conf.d configuration from snap for db etc
echo "Installing configuration for OpenStack Services"
for project in neutron nova keystone glance; do

21
snapcraft.yaml
View File

@ -307,6 +307,19 @@ apps:
# plugs:
# - network
# Cluster
cluster-server:
command: flask run -p 10002 --host=0.0.0.0 # TODO: run as a uwsgi app
daemon: simple
environment:
LC_ALL: C.UTF-8 # Makes flask happy
LANG: C.UTF-8 # Makes flask happy
FLASK_APP: ${SNAP}/lib/python3.6/site-packages/cluster/daemon.py
join:
command: python3 ${SNAP}/lib/python3.6/site-packages/cluster/client.py
parts:
# Add Ubuntu Cloud Archive sources.
# Allows us to fetch things such as updated libvirt.
@ -828,3 +841,11 @@ parts:
requirements:
- requirements.txt
source: tools/launch
# Clustering client and server
cluster:
plugin: python
python-version: python3
requirements:
- requirements.txt
source: tools/cluster

43
tests/test_basic.py
View File

@ -14,10 +14,8 @@ Web IDE.
"""
import json
import os
import sys
import time
import unittest
import xvfbwrapper
from selenium import webdriver
@ -52,7 +50,7 @@ class TestBasics(Framework):
"""
launch = '/snap/bin/microstack.launch'
openstack = '/snap/bin/microstack.openstack'
# openstack = '/snap/bin/microstack.openstack'
print("Testing microstack.launch ...")
@ -68,45 +66,6 @@ class TestBasics(Framework):
# Endpoints should not contain localhost
self.assertFalse("localhost" in endpoints)
# Verify that microstack.launch completed successfully
# Ping the instance
ip = None
servers = check_output(*self.PREFIX, openstack,
'server', 'list', '--format', 'json')
servers = json.loads(servers)
for server in servers:
if server['Name'] == 'breakfast':
ip = server['Networks'].split(",")[1].strip()
break
self.assertTrue(ip)
pings = 1
max_pings = 600 # ~10 minutes!
while not call(*self.PREFIX, 'ping', '-c1', '-w1', ip):
pings += 1
if pings > max_pings:
self.assertFalse(True, msg='Max pings reached!')
print("Testing instances' ability to connect to the Internet")
# Test Internet connectivity
attempts = 1
max_attempts = 300 # ~10 minutes!
username = check_output(*self.PREFIX, 'whoami')
while not call(
*self.PREFIX,
'ssh',
'-oStrictHostKeyChecking=no',
'-i', '/home/{}/.ssh/id_microstack'.format(username),
'cirros@{}'.format(ip),
'--', 'ping', '-c1', '91.189.94.250'):
attempts += 1
if attempts > max_attempts:
self.assertFalse(True, msg='Unable to access the Internet!')
time.sleep(1)
if 'multipass' in self.PREFIX:
print("Opening {}:80 up to the outside world".format(
self.HORIZON_IP))

131
tests/test_cluster.py Executable file
View File

@ -0,0 +1,131 @@
#!/usr/bin/env python
"""
cluster_test.py
This is a test to verify that we can setup a small, two node cluster.
The host running this test must have at least 16GB of RAM, four cpu
cores, a large amount of disk space, and the ability to run multipass
vms.
"""
import json
import os
import petname
import sys
import unittest
sys.path.append(os.getcwd())
from tests.framework import Framework, check, check_output, call # noqa E402
os.environ['MULTIPASS'] = 'true' # TODO better way to do this.
class TestCluster(Framework):
INIT_FLAG = 'control'
def _compute_node(self, channel='dangerous'):
"""Make a compute node.
TODO: refactor framework so that we can fold a lot of this
into the parent framework. There's a lot of dupe code here.
"""
machine = petname.generate()
prefix = ['multipass', 'exec', machine, '--']
check('multipass', 'launch', '--cpus', '2', '--mem', '8G',
self.DISTRO, '--name', machine)
check('multipass', 'copy-files', self.SNAP, '{}:'.format(machine))
check(*prefix, 'sudo', 'snap', 'install', '--classic',
'--{}'.format(channel), self.SNAP)
return machine, prefix
def test_cluster(self):
# After the setUp step, we should have a control node running
# in a multipass vm. Let's look up its cluster password and ip
# address.
openstack = '/snap/bin/microstack.openstack'
cluster_password = check_output(*self.PREFIX, 'sudo', 'snap',
'get', 'microstack',
'config.cluster.password')
control_ip = check_output(*self.PREFIX, 'sudo', 'snap',
'get', 'microstack',
'config.network.control-ip')
self.assertTrue(cluster_password)
self.assertTrue(control_ip)
compute_machine, compute_prefix = self._compute_node()
# TODO add the following to args for init
check(*compute_prefix, 'sudo', 'snap', 'set', 'microstack',
'config.network.control-ip={}'.format(control_ip))
check(*compute_prefix, 'sudo', 'microstack.init', '--compute',
'--cluster-password', cluster_password, '--debug')
# Verify that our services look setup properly on compute node.
services = check_output(
*compute_prefix, 'systemctl', 'status', 'snap.microstack.*',
'--no-page')
self.assertTrue('nova-compute' in services)
self.assertFalse('keystone-' in services)
check(*compute_prefix, '/snap/bin/microstack.launch', 'cirros',
'--name', 'breakfast', '--retry',
'--availability-zone', 'nova:{}'.format(compute_machine))
# TODO: verify horizon dashboard on control node.
# Verify endpoints
compute_ip = check_output(*compute_prefix, 'sudo', 'snap',
'get', 'microstack',
'config.network.compute-ip')
self.assertFalse(compute_ip == control_ip)
# Ping the instance
ip = None
servers = check_output(*compute_prefix, openstack,
'server', 'list', '--format', 'json')
servers = json.loads(servers)
for server in servers:
if server['Name'] == 'breakfast':
ip = server['Networks'].split(",")[1].strip()
break
self.assertTrue(ip)
pings = 1
max_pings = 60 # ~1 minutes
# Ping the machine from the control node (we don't have
# networking wired up for the other nodes).
while not call(*self.PREFIX, 'ping', '-c1', '-w1', ip):
pings += 1
if pings > max_pings:
self.assertFalse(
True,
msg='Max pings reached for instance on {}!'.format(
compute_machine))
self.passed = True
# Compute machine cleanup
check('sudo', 'multipass', 'delete', compute_machine)
if __name__ == '__main__':
# Run our tests, ignoring deprecation warnings and warnings about
# unclosed sockets. (TODO: setup a selenium server so that we can
# move from PhantomJS, which is deprecated, to to Selenium headless.)
unittest.main(warnings='ignore')

5
tests/test_control.py
View File

@ -26,6 +26,10 @@ class TestControlNode(Framework):
INIT_FLAG = 'control'
def test_control_node(self):
"""A control node has all services running, so this shouldn't be any
different than our standard setup.
"""
print("Checking output of services ...")
services = check_output(
@ -35,7 +39,6 @@ class TestControlNode(Framework):
print("services: @@@")
print(services)
self.assertFalse('nova-' in services)
self.assertTrue('neutron-' in services)
self.assertTrue('keystone-' in services)

0
tools/cluster/cluster/__init__.py Normal file
View File

41
tools/cluster/cluster/client.py Normal file
View File

@ -0,0 +1,41 @@
import json
import requests
from cluster.shell import check, check_output, write_tunnel_config
def join():
"""Join an existing cluster as a compute node."""
config = json.loads(check_output('snapctl', 'get', 'config'))
password = config['cluster']['password']
control_ip = config['network']['control-ip']
my_ip = config['network']['compute-ip']
if not password:
raise Exception("No cluster password specified!")
resp = requests.post(
'http://{}:10002/join'.format(control_ip),
json={'password': password, 'ip_address': my_ip})
if resp.status_code != 200:
# TODO better error and formatting.
raise Exception('Failed to get info from control node: {}'.format(
resp.json))
resp = resp.json()
# TODO: add better error handling to the below
os_password = resp['config']['credentials']['os-password']
# Write out tunnel config and restart neutron openvswitch agent.
write_tunnel_config(my_ip)
check('snapctl', 'restart', 'microstack.neutron-openvswitch-agent')
# Set passwords and such
check('snapctl', 'set', 'config.credentials.os-password={}'.format(
os_password))
if __name__ == '__main__':
join()

55
tools/cluster/cluster/daemon.py Normal file
View File

@ -0,0 +1,55 @@
import json
from flask import Flask, request
from cluster.shell import check, check_output, write_tunnel_config
app = Flask(__name__)
class Unauthorized(Exception):
pass
def join_info(password, ip_address):
our_password = check_output('snapctl', 'get', 'config.cluster.password')
if password.strip() != our_password.strip():
raise Unauthorized()
# Load config
# TODO: be selective about what we return. For now, we just get everything.
config = json.loads(check_output('snapctl', 'get', 'config'))
# Write out tunnel config and restart neutron openvswitch agent.
write_tunnel_config(config['network']['control-ip'])
check('snapctl', 'restart', 'microstack.neutron-openvswitch-agent')
info = {'config': config}
return info
@app.route('/')
def home():
status = {
'status': 'running',
'info': 'Microstack clustering daemon.'
}
return json.dumps(status)
@app.route('/join', methods=['POST'])
def join():
req = request.json # TODO: better error messages on failed parse.
password = req.get('password')
ip_address = req.get('ip_address')
if not password:
return 'No password specified', 500
try:
return json.dumps(join_info(password, ip_address))
except Unauthorized:
return (json.dumps({'error': 'Incorrect password.'}), 500)

50
tools/cluster/cluster/shell.py Normal file
View File

@ -0,0 +1,50 @@
import os
import pymysql
import subprocess
def sql(cmd) -> None:
"""Execute some SQL!
Really simply wrapper around a pymysql connection, suitable for
passing the limited CREATE and GRANT commands that we need to pass
here.
:param cmd: sql to execute.
# TODO: move this into a shared shell library.
"""
mysql_conf = '${SNAP_USER_COMMON}/etc/mysql/my.cnf'.format(**os.environ)
connection = pymysql.connect(host='localhost', user='root',
read_default_file=mysql_conf)
with connection.cursor() as cursor:
cursor.execute(cmd)
def check_output(*args):
"""Execute a shell command, returning the output of the command."""
return subprocess.check_output(args, env=os.environ,
universal_newlines=True).strip()
def check(*args):
"""Execute a shell command, raising an error on failed excution.
:param args: strings to be composed into the bash call.
"""
return subprocess.check_call(args, env=os.environ)
def write_tunnel_config(local_ip):
"""Write tunnel config file for neutron agent."""
path_ = '{SNAP_COMMON}/etc/neutron/neutron.conf.d/tunnel.conf'.format(
**os.environ)
with open(path_, 'w') as file_:
file_.write("""\
[OVS]
local_ip = {local_ip}
""".format(local_ip=local_ip))

2
tools/cluster/requirements.txt Normal file
View File

@ -0,0 +1,2 @@
flask
requests

13
tools/cluster/setup.py Normal file
View File

@ -0,0 +1,13 @@
from setuptools import setup, find_packages
setup(
name="microstack_cluster",
description="Clustering client and server.",
packages=find_packages(exclude=("tests",)),
version="0.0.1",
entry_points={
'console_scripts': [
'microstack_join = cluster.client:join',
],
}
)

94
tools/init/init/main.py
View File

@ -28,23 +28,72 @@ See the License for the specific language governing permissions and
limitations under the License.
"""
import argparse
import logging
import secrets
import string
import sys
from init.config import log
from init import questions
from init.shell import check
# Figure out whether to prompt for user input, and which type of node
# we're running.
# TODO drop in argparse and formalize this.
COMPUTE = '--compute' in sys.argv
CONTROL = '--control' in sys.argv
AUTO = ('--auto' in sys.argv) or COMPUTE or CONTROL
from init import questions
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('--auto', '-a', action='store_true',
help='Run non interactively.')
parser.add_argument('--cluster-password')
parser.add_argument('--compute', action='store_true')
parser.add_argument('--control', action='store_true')
parser.add_argument('--debug', action='store_true')
args = parser.parse_args()
return args
def process_args(args):
"""Look through our args object and set the proper default config
values in our snap config, based on those args.
"""
auto = args.auto or args.control or args.compute
if args.compute or args.control:
check('snapctl', 'set', 'config.clustered=true')
if args.compute:
check('snapctl', 'set', 'config.cluster.role=compute')
if args.control:
# If both compute and control are passed for some reason, we
# wind up with the role of 'control', which is best, as a
# control node also serves as a compute node in our hyper
# converged architecture.
check('snapctl', 'set', 'config.cluster.role=control')
if args.cluster_password:
check('snapctl', 'set', 'config.cluster.password={}'.format(
args.cluster_password))
if auto and not args.cluster_password:
alphabet = string.ascii_letters + string.digits
password = ''.join(secrets.choice(alphabet) for i in range(10))
check('snapctl', 'set', 'config.cluster.password={}'.format(password))
if args.debug:
log.setLevel(logging.DEBUG)
return auto
def main() -> None:
args = parse_args()
auto = process_args(args)
question_list = [
questions.Clustering(),
questions.Dns(),
questions.ExtGateway(),
questions.ExtCidr(),
@ -56,36 +105,19 @@ def main() -> None:
# questions.FileHandleLimits(),
questions.RabbitMq(),
questions.DatabaseSetup(),
questions.NovaSetup(),
questions.NeutronSetup(),
questions.NovaHypervisor(),
questions.NovaControlPlane(),
questions.NeutronControlPlane(),
questions.GlanceSetup(),
questions.KeyPair(),
questions.SecurityRules(),
questions.PostSetup(),
]
# If we are setting up a "control" or "compute" node, override
# some of the default yes/no questions.
# TODO: move this into a nice little yaml parsing lib, and
# allow people to pass in a config file from the command line.
if CONTROL:
check('snapctl', 'set', 'questions.nova-setup=false')
check('snapctl', 'set', 'questions.key-pair=nil')
check('snapctl', 'set', 'questions.security-rules=false')
if COMPUTE:
check('snapctl', 'set', 'questions.rabbit-mq=false')
check('snapctl', 'set', 'questions.database-setup=false')
check('snapctl', 'set', 'questions.neutron-setup=false')
check('snapctl', 'set', 'questions.glance-setup=false')
for question in question_list:
if AUTO:
# If we are automatically answering questions, replace the
# prompt for user input with a function that returns None,
# causing the question to fall back to the already set
# default
question._input_func = lambda prompt: None
if auto:
# Force all questions to be non-interactive if we passed --auto.
question.interactive = False
try:
question.ask()

230
tools/init/init/questions.py → tools/init/init/questions/__init__.py
View File

@ -30,7 +30,8 @@ from os import path
from init.shell import (check, call, check_output, shell, sql, nc_wait,
log_wait, restart, download)
from init.config import Env, log
from init.question import Question
from init.questions.question import Question
from init.questions import clustering
_env = Env().get_env()
@ -43,6 +44,69 @@ class ConfigError(Exception):
"""
class Clustering(Question):
"""Possibly setup clustering."""
_type = 'boolean'
_question = 'Do you want to setup clustering?'
config_key = 'config.clustered'
interactive = True
def yes(self, answer: bool):
log.info('Configuring clustering ...')
questions = [
clustering.Role(),
clustering.Password(),
clustering.ControlIp(),
clustering.ComputeIp(), # Automagically skipped role='control'
]
for question in questions:
if not self.interactive:
question.interactive = False
question.ask()
role = check_output('snapctl', 'get', 'config.cluster.role')
control_ip = check_output('snapctl', 'get',
'config.network.control-ip')
password = check_output('snapctl', 'get', 'config.cluster.password')
log.debug('Role: {}, IP: {}, Password: {}'.format(
role, control_ip, password))
# TODO: raise an exception if any of the above are None (can
# happen if we're automatig and mess up our params.)
if role == 'compute':
log.info('I am a compute node.')
# Gets config info and sets local env vals.
check_output('microstack_join')
# Set default question answers.
check('snapctl', 'set', 'config.services.control-plane=false')
check('snapctl', 'set', 'config.services.hypervisor=true')
if role == 'control':
log.info('I am a control node.')
check('snapctl', 'set', 'config.services.control-plane=true')
# We want to run a hypervisor on our control plane nodes
# -- this is essentially a hyper converged cloud.
check('snapctl', 'set', 'config.services.hypervisor=true')
# TODO: if this is run after init has already been called,
# need to restart services.
# Write templates
check('snap-openstack', 'setup')
def no(self, answer: bool):
# Turn off cluster server
# TODO: it would be more secure to reverse this -- only enable
# to service if we are doing clustering.
check('systemctl', 'disable', 'snap.microstack.cluster-server')
class ConfigQuestion(Question):
"""Question class that simply asks for and sets a config value.
@ -78,6 +142,7 @@ class Dns(Question):
_type = 'string'
_question = 'DNS to use'
config_key = 'config.network.dns'
def yes(self, answer: str):
"""Override the default dhcp_agent.ini file."""
@ -105,11 +170,21 @@ class ExtGateway(ConfigQuestion):
_type = 'string'
_question = 'External Gateway'
config_key = 'config.network.ext-gateway'
def yes(self, answer):
# Preserve old behavior.
# TODO: update this
_env['extgateway'] = answer
clustered = check_output('snapctl', 'get', 'config.clustered')
if clustered.lower() != 'true':
check('snapctl', 'set', 'config.network.control-ip={}'.format(
answer))
check('snapctl', 'set', 'config.network.compute-ip={}'.format(
answer))
_env['control_ip'] = _env['compute_ip'] = answer
else:
_env['control_ip'] = check_output('snapctl', 'get',
'config.network.control-ip')
_env['compute_ip'] = check_output('snapctl', 'get',
'config.network.compute-ip')
class ExtCidr(ConfigQuestion):
@ -117,20 +192,18 @@ class ExtCidr(ConfigQuestion):
_type = 'string'
_question = 'External Ip Range'
config_key = 'config.network.ext-cidr'
def yes(self, answer):
# Preserve old behavior.
# TODO: update this
_env['extcidr'] = answer
class OsPassword(ConfigQuestion):
_type = 'string'
_question = 'Openstack Admin Password'
config_key = 'config.credentials.os-password'
def yes(self, answer):
# Preserve old behavior.
# TODO: update this
_env['ospassword'] = answer
# TODO obfuscate the password!
@ -141,6 +214,7 @@ class IpForwarding(Question):
_type = 'boolean' # Auto for now, to maintain old behavior.
_question = 'Do you wish to setup ip forwarding? (recommended)'
config_key = 'config.host.ip-forwarding'
def yes(self, answer: str) -> None:
"""Use sysctl to setup ip forwarding."""
@ -150,7 +224,8 @@ class IpForwarding(Question):
class ForceQemu(Question):
_type = 'auto'
_type = 'boolean'
config_key = 'config.host.check-qemu'
def yes(self, answer: str) -> None:
"""Possibly force us to use qemu emulation rather than kvm."""
@ -204,9 +279,10 @@ class RabbitMq(Question):
"""Wait for Rabbit to start, then setup permissions."""
_type = 'boolean'
config_key = 'config.services.control-plane'
def _wait(self) -> None:
nc_wait(_env['extgateway'], '5672')
nc_wait(_env['control_ip'], '5672')
log_file = '{SNAP_COMMON}/log/rabbitmq/startup_log'.format(**_env)
log_wait(log_file, 'completed')
@ -230,31 +306,42 @@ class RabbitMq(Question):
self._configure()
log.info('RabbitMQ Configured!')
def no(self, answer: str):
log.info('Disabling local rabbit ...')
check('systemctl', 'disable', 'snap.microstack.rabbitmq-server')
class DatabaseSetup(Question):
"""Setup keystone permissions, then setup all databases."""
_type = 'boolean'
config_key = 'config.services.control-plane'
def _wait(self) -> None:
nc_wait(_env['extgateway'], '3306')
nc_wait(_env['control_ip'], '3306')
log_wait('{SNAP_COMMON}/log/mysql/error.log'.format(**_env),
'mysqld: ready for connections.')
def _create_dbs(self) -> None:
# TODO: actually use passwords here.
for db in ('neutron', 'nova', 'nova_api', 'nova_cell0', 'cinder',
'glance', 'keystone'):
sql("CREATE DATABASE IF NOT EXISTS {db};".format(db=db))
sql(
"GRANT ALL PRIVILEGES ON {db}.* TO {db}@{extgateway} \
"GRANT ALL PRIVILEGES ON {db}.* TO {db}@{control_ip} \
IDENTIFIED BY '{db}';".format(db=db, **_env))
# Grant nova user access to cell0
sql(
"GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'{control_ip}' \
IDENTIFIED BY \'nova';".format(**_env))
def _bootstrap(self) -> None:
if call('openstack', 'user', 'show', 'admin'):
return
bootstrap_url = 'http://{extgateway}:5000/v3/'.format(**_env)
bootstrap_url = 'http://{control_ip}:5000/v3/'.format(**_env)
check('snap-openstack', 'launch', 'keystone-manage', 'bootstrap',
'--bootstrap-password', _env['ospassword'],
@ -300,11 +387,45 @@ class DatabaseSetup(Question):
log.info('Keystone configured!')
def no(self, answer: str):
# We assume that the control node has a connection setup for us.
check('snapctl', 'set', 'database.ready=true')
class NovaSetup(Question):
"""Create all relevant nova users and services."""
log.info('Disabling local MySQL ...')
check('systemctl', 'disable', 'snap.microstack.mysqld')
class NovaHypervisor(Question):
"""Run the nova compute hypervisor."""
_type = 'boolean'
config_key = 'config.services.hypervisor'
def yes(self, answer):
log.info('Configuring nova compute hypervisor ...')
if not call('openstack', 'service', 'show', 'compute'):
check('openstack', 'service', 'create', '--name', 'nova',
'--description', '"Openstack Compute"', 'compute')
# TODO make sure that we are the control plane before executing
# TODO if control plane is not hypervisor, still create this
for endpoint in ['public', 'internal', 'admin']:
call('openstack', 'endpoint', 'create', '--region',
'microstack', 'compute', endpoint,
'http://{compute_ip}:8774/v2.1'.format(**_env))
check('snapctl', 'start', 'microstack.nova-compute')
def no(self, answer):
log.info('Disabling nova compute service ...')
check('systemctl', 'disable', 'snap.microstack.nova-compute')
class NovaControlPlane(Question):
"""Create all control plane nova users and services."""
_type = 'boolean'
config_key = 'config.services.control-plane'
def _flavors(self) -> None:
"""Create default flavors."""
@ -327,7 +448,7 @@ class NovaSetup(Question):
'm1.xlarge')
def yes(self, answer: str) -> None:
log.info('Configuring nova ...')
log.info('Configuring nova control plane services ...')
if not call('openstack', 'user', 'show', 'nova'):
check('openstack', 'user', 'create', '--domain',
@ -341,14 +462,6 @@ class NovaSetup(Question):
check('openstack', 'role', 'add', '--project', 'service',
'--user', 'placement', 'admin')
if not call('openstack', 'service', 'show', 'compute'):
check('openstack', 'service', 'create', '--name', 'nova',
'--description', '"Openstack Compute"', 'compute')
for endpoint in ['public', 'internal', 'admin']:
call('openstack', 'endpoint', 'create', '--region',
'microstack', 'compute', endpoint,
'http://{extgateway}:8774/v2.1'.format(**_env))
if not call('openstack', 'service', 'show', 'placement'):
check('openstack', 'service', 'create', '--name',
'placement', '--description', '"Placement API"',
@ -357,12 +470,7 @@ class NovaSetup(Question):
for endpoint in ['public', 'internal', 'admin']:
call('openstack', 'endpoint', 'create', '--region',
'microstack', 'placement', endpoint,
'http://{extgateway}:8778'.format(**_env))
# Grant nova user access to cell0
sql(
"GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'{extgateway}' \
IDENTIFIED BY \'nova';".format(**_env))
'http://{control_ip}:8778'.format(**_env))
# Use snapctl to start nova services. We need to call them
# out manually, because systemd doesn't know about them yet.
@ -371,7 +479,6 @@ class NovaSetup(Question):
for service in [
'microstack.nova-api',
'microstack.nova-api-metadata',
'microstack.nova-compute',
'microstack.nova-conductor',
'microstack.nova-scheduler',
'microstack.nova-uwsgi',
@ -396,18 +503,31 @@ class NovaSetup(Question):
restart('nova-*')
nc_wait(_env['extgateway'], '8774')
nc_wait(_env['compute_ip'], '8774')
sleep(5) # TODO: log_wait
log.info('Creating default flavors...')
self._flavors()
def no(self, answer):
log.info('Disabling nova control plane services ...')
class NeutronSetup(Question):
for service in [
'snap.microstack.nova-uwsgi',
'snap.microstack.nova-api',
'snap.microstack.nova-conductor',
'snap.microstack.nova-scheduler',
'snap.microstack.nova-api-metadata']:
check('systemctl', 'disable', service)
class NeutronControlPlane(Question):
"""Create all relevant neutron services and users."""
_type = 'boolean'
config_key = 'config.services.control-plane'
def yes(self, answer: str) -> None:
log.info('Configuring Neutron')
@ -424,7 +544,7 @@ class NeutronSetup(Question):
for endpoint in ['public', 'internal', 'admin']:
call('openstack', 'endpoint', 'create', '--region',
'microstack', 'network', endpoint,
'http://{extgateway}:9696'.format(**_env))
'http://{control_ip}:9696'.format(**_env))
for service in [
'microstack.neutron-api',
@ -440,7 +560,7 @@ class NeutronSetup(Question):
restart('neutron-*')
nc_wait(_env['extgateway'], '9696')
nc_wait(_env['control_ip'], '9696')
sleep(5) # TODO: log_wait
@ -467,27 +587,49 @@ class NeutronSetup(Question):
check('openstack', 'router', 'set', '--external-gateway',
'external', 'test-router')
def no(self, answer):
"""Create endpoints pointed at control node if we're not setting up
neutron on this machine.
"""
# Make sure that the agent is running.
for service in [
'microstack.neutron-openvswitch-agent',
]:
check('snapctl', 'start', service)
# Disable the other services.
for service in [
'snap.microstack.neutron-api',
'snap.microstack.neutron-dhcp-agent',
'snap.microstack.neutron-metadata-agent',
'snap.microstack.neutron-l3-agent',
]:
check('systemctl', 'disable', service)
class GlanceSetup(Question):
"""Setup glance, and download an initial Cirros image."""
_type = 'boolean'
config_key = 'config.services.control-plane'
def _fetch_cirros(self) -> None:
if call('openstack', 'image', 'show', 'cirros'):
return
log.info('Adding cirros image ...')
env = dict(**_env)
env['VER'] = '0.4.0'
env['IMG'] = 'cirros-{VER}-x86_64-disk.img'.format(**env)
log.info('Fetching cirros image ...')
cirros_path = '{SNAP_COMMON}/images/{IMG}'.format(**env)
if not path.exists(cirros_path):
check('mkdir', '-p', '{SNAP_COMMON}/images'.format(**env))
log.info('Downloading cirros image ...')
download(
'http://download.cirros-cloud.net/{VER}/{IMG}'.format(**env),
'{SNAP_COMMON}/images/{IMG}'.format(**env))
@ -513,11 +655,11 @@ class GlanceSetup(Question):
for endpoint in ['internal', 'admin', 'public']:
check('openstack', 'endpoint', 'create', '--region',
'microstack', 'image', endpoint,
'http://{extgateway}:9292'.format(**_env))
'http://{compute_ip}:9292'.format(**_env))
for service in [
'microstack.glance-api',
'microstack.registry', # TODO rename this to glance-registery
'microstack.registry', # TODO rename to glance-registery
]:
check('snapctl', 'start', service)
@ -525,12 +667,16 @@ class GlanceSetup(Question):
restart('glance*')
nc_wait(_env['extgateway'], '9292')
nc_wait(_env['compute_ip'], '9292')
sleep(5) # TODO: log_wait
self._fetch_cirros()
def no(self, answer):
check('systemctl', 'disable', 'snap.microstack.glance-api')
check('systemctl', 'disable', 'snap.microstack.registry')
class KeyPair(Question):
"""Create a keypair for ssh access to instances.
@ -541,6 +687,7 @@ class KeyPair(Question):
questions at the beginning.)
"""
_type = 'string'
config_key = 'config.credentials.key-pair'
def yes(s