Files

Stephen Finucane c1c6a0e916 docs: Replace/remove shade-specific docs

Lots of minor tweaks, none of them worthy of their own change.

Change-Id: Iba8649b8e12f367760849fd185cf76b94b703706
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>

2023-05-30 10:28:59 +01:00

21 KiB

Raw Blame History

Multi-Cloud Demo

This document contains a presentation in presentty format. If you want to walk through it like a presentation, install presentty and run:

presentty doc/source/user/multi-cloud-demo.rst

The content is hopefully helpful even if it's not being narrated, so it's being included in the openstacksdk docs.

Who am I?

Monty Taylor

OpenStack Infra Core
irc: mordred
twitter: @e_monty

What are we going to talk about?

OpenStackSDK

a task and end-user oriented Python library
abstracts deployment differences
designed for multi-cloud
simple to use
massive scale
- optional advanced features to handle 20k servers a day
Initial logic/design extracted from nodepool
Librified to re-use in Ansible

OpenStackSDK is Free Software

https://opendev.org/openstack/openstacksdk
openstack-discuss@lists.openstack.org
#openstack-sdks on oftc

This talk is Free Software, too

Written for presentty (https://pypi.org/project/presentty)
doc/source/user/multi-cloud-demo.rst
examples in examples/cloud
Paths subject to change - this is the first presentation in tree!

Complete Example

from openstack import cloud as openstack

# Initialize and turn on debug logging
openstack.enable_logging(debug=True)

for cloud_name, region_name in [
    ('my-vexxhost', 'ca-ymq-1'),
    ('my-citycloud', 'Buf1'),
    ('my-internap', 'ams01'),
]:
    # Initialize cloud
    cloud = openstack.connect(cloud=cloud_name, region_name=region_name)

    # Upload an image to the cloud
    image = cloud.create_image(
        'devuan-jessie',
        filename='devuan-jessie.qcow2',
        wait=True,
    )

    # Find a flavor with at least 512M of RAM
    flavor = cloud.get_flavor_by_ram(512)

    # Boot a server, wait for it to boot, and then do whatever is needed
    # to get a public ip for it.
    cloud.create_server(
        'my-server',
        image=image,
        flavor=flavor,
        wait=True,
        auto_ip=True,
    )

Let's Take a Few Steps Back

Multi-cloud is easy, but you need to know a few things.

Terminology
Config
OpenStackSDK API

Cloud Terminology

Let's define a few terms, so that we can use them with ease:

cloud - logically related collection of services
region - completely independent subset of a given cloud
patron - human who has an account
user - account on a cloud
project - logical collection of cloud resources
domain - collection of users and projects

Cloud Terminology Relationships

A cloud has one or more regions
A patron has one or more users
A patron has one or more projects
A cloud has one or more domains
In a cloud with one domain it is named "default"
Each patron may have their own domain
Each user is in one domain
Each project is in one domain
A user has one or more roles on one or more projects

HTTP Sessions

HTTP interactions are authenticated via keystone
Authenticating returns a token
An authenticated HTTP Session is shared across a region

Cloud Regions

A cloud region is the basic unit of REST interaction.

A cloud has a service catalog
The service catalog is returned in the token
The service catalog lists endpoint for each service in each region
A region is completely autonomous

Users, Projects and Domains

In clouds with multiple domains, project and user names are only unique within a region.

Names require domain information for uniqueness. IDs do not.
Providing domain information when not needed is fine.
project_name requires project_domain_name or project_domain_id
project_id does not
username requires user_domain_name or user_domain_id
user_id does not

Confused Yet?

Don't worry - you don't have to deal with most of that.

Auth per cloud, select per region

In general, the thing you need to know is:

Configure authentication per cloud
Select config to use by cloud and region

clouds.yaml

Information about the clouds you want to connect to is stored in a file called clouds.yaml.

clouds.yaml can be in your homedir: ~/.config/openstack/clouds.yaml or system-wide: /etc/openstack/clouds.yaml.

Information in your homedir, if it exists, takes precedence.

Full docs on clouds.yaml are at https://docs.openstack.org/os-client-config/latest/

What about Mac and Windows?

USER_CONFIG_DIR is different on Linux, OSX and Windows.

Linux: ~/.config/openstack
OSX: ~/Library/Application Support/openstack
Windows: C:UsersUSERNAMEAppDataLocalOpenStackopenstack

SITE_CONFIG_DIR is different on Linux, OSX and Windows.

Linux: /etc/openstack
OSX: /Library/Application Support/openstack
Windows: C:ProgramDataOpenStackopenstack

Config Terminology

For multi-cloud, think of two types:

profile - Facts about the cloud that are true for everyone
cloud - Information specific to a given user

Apologies for the use of cloud twice.

Environment Variables and Simple Usage

Environment variables starting with OS_ go into a cloud called envvars
If you only have one cloud, you don't have to specify it
OS_CLOUD and OS_REGION_NAME are default values for cloud and region_name

TOO MUCH TALKING - NOT ENOUGH CODE

basic clouds.yaml for the example code

Simple example of a clouds.yaml

Config for a named cloud "my-citycloud"
Reference a well-known "named" profile: citycloud
os-client-config has a built-in list of profiles at https://docs.openstack.org/openstacksdk/latest/user/config/vendor-support.html
Vendor profiles contain various advanced config
cloud name can match profile name (using different names for clarity)

clouds:
  my-citycloud:
    profile: citycloud
    auth:
      username: mordred
      project_id: 65222a4d09ea4c68934fa1028c77f394
      user_domain_id: d0919bd5e8d74e49adf0e145807ffc38
      project_domain_id: d0919bd5e8d74e49adf0e145807ffc38

Where's the password?

secure.yaml

Optional additional file just like clouds.yaml
Values overlaid on clouds.yaml
Useful if you want to protect secrets more stringently

Example secure.yaml

No, my password isn't XXXXXXXX
cloud name should match clouds.yaml
Optional - I actually keep mine in my clouds.yaml

clouds:
  my-citycloud:
    auth:
      password: XXXXXXXX

more clouds.yaml

More information can be provided.

Use v3 of the identity API - even if others are present
Use https://image-ca-ymq-1.vexxhost.net/v2 for image API instead of what's in the catalog

my-vexxhost:
  identity_api_version: 3
  image_endpoint_override: https://image-ca-ymq-1.vexxhost.net/v2
  profile: vexxhost
  auth:
    user_domain_id: default
    project_domain_id: default
    project_name: d8af8a8f-a573-48e6-898a-af333b970a2d
    username: 0b8c435b-cc4d-4e05-8a47-a2ada0539af1

Much more complex clouds.yaml example

Not using a profile - all settings included
In the ams01 region there are two networks with undiscoverable qualities
Each one are labeled here so choices can be made
Any of the settings can be specific to a region if needed
region settings override cloud settings
cloud does not support floating-ips

my-internap:
  auth:
    auth_url: https://identity.api.cloud.inap.com
    username: api-55f9a00fb2619
    project_name: inap-17037
  identity_api_version: 3
  floating_ip_source: None
  regions:
  - name: ams01
    values:
      networks:
      - name: inap-17037-WAN1654
        routes_externally: true
        default_interface: true
      - name: inap-17037-LAN3631
        routes_externally: false

Complete Example Again

from openstack import cloud as openstack

# Initialize and turn on debug logging
openstack.enable_logging(debug=True)

for cloud_name, region_name in [
        ('my-vexxhost', 'ca-ymq-1'),
        ('my-citycloud', 'Buf1'),
        ('my-internap', 'ams01')]:
    # Initialize cloud
    cloud = openstack.connect(cloud=cloud_name, region_name=region_name)

    # Upload an image to the cloud
    image = cloud.create_image(
        'devuan-jessie', filename='devuan-jessie.qcow2', wait=True)

    # Find a flavor with at least 512M of RAM
    flavor = cloud.get_flavor_by_ram(512)

    # Boot a server, wait for it to boot, and then do whatever is needed
    # to get a public ip for it.
    cloud.create_server(
        'my-server', image=image, flavor=flavor, wait=True, auto_ip=True)

Step By Step

Import the library

from openstack import cloud as openstack

Logging

openstacksdk uses standard python logging
openstack.enable_logging does easy defaults
Squelches some meaningless warnings
- debug
  - Logs openstacksdk loggers at debug level
- http_debug Implies debug, turns on HTTP tracing

# Initialize and turn on debug logging
openstack.enable_logging(debug=True)

Example with Debug Logging

examples/cloud/debug-logging.py

from openstack import cloud as openstack
openstack.enable_logging(debug=True)

cloud = openstack.connect(cloud='my-vexxhost', region_name='ca-ymq-1')
cloud.get_image('Ubuntu 16.04.1 LTS [2017-03-03]')

Example with HTTP Debug Logging

examples/cloud/http-debug-logging.py

from openstack import cloud as openstack
openstack.enable_logging(http_debug=True)

cloud = openstack.connect(
    cloud='my-vexxhost', region_name='ca-ymq-1')
cloud.get_image('Ubuntu 16.04.1 LTS [2017-03-03]')

Cloud Regions

cloud constructor needs cloud and region_name
openstack.connect is a helper factory function

for cloud_name, region_name in [
    ('my-vexxhost', 'ca-ymq-1'),
    ('my-citycloud', 'Buf1'),
    ('my-internap', 'ams01')
]:
    # Initialize cloud
    cloud = openstack.connect(cloud=cloud_name, region_name=region_name)

Upload an Image

Picks the correct upload mechanism
SUGGESTION Always upload your own base images

# Upload an image to the cloud
image = cloud.create_image(
    'devuan-jessie',
    filename='devuan-jessie.qcow2',
    wait=True,
)

Always Upload an Image

Ok. You don't have to. But, for multi-cloud...

Images with same content are named different on different clouds
Images with same name on different clouds can have different content
Upload your own to all clouds, both problems go away
Download from OS vendor or build with diskimage-builder

Find a flavor

Flavors are all named differently on clouds
Flavors can be found via RAM
get_flavor_by_ram finds the smallest matching flavor

# Find a flavor with at least 512M of RAM
flavor = cloud.get_flavor_by_ram(512)

Create a server

my-vexxhost
- Boot server
- Wait for status==ACTIVE
my-internap
- Boot server on network inap-17037-WAN1654
- Wait for status==ACTIVE
my-citycloud
- Boot server
- Wait for status==ACTIVE
- Find the port for the fixed_ip for server
- Create floating-ip on that port
- Wait for floating-ip to attach

# Boot a server, wait for it to boot, and then do whatever is needed
# to get a public ip for it.
cloud.create_server(
    'my-server', image=image, flavor=flavor, wait=True, auto_ip=True)

Wow. We didn't even deploy Wordpress!

Image and Flavor by Name or ID

Pass string to image/flavor
Image/Flavor will be found by name or ID
Common pattern
examples/cloud/create-server-name-or-id.py

from openstack import cloud as openstack

# Initialize and turn on debug logging
openstack.enable_logging(debug=True)

for cloud_name, region_name, image, flavor in [
        ('my-vexxhost', 'ca-ymq-1',
         'Ubuntu 16.04.1 LTS [2017-03-03]', 'v1-standard-4'),
        ('my-citycloud', 'Buf1',
         'Ubuntu 16.04 Xenial Xerus', '4C-4GB-100GB'),
        ('my-internap', 'ams01',
         'Ubuntu 16.04 LTS (Xenial Xerus)', 'A1.4')]:
    # Initialize cloud
    cloud = openstack.connect(cloud=cloud_name, region_name=region_name)

    # Boot a server, wait for it to boot, and then do whatever is needed
    # to get a public ip for it.
    server = cloud.create_server(
        'my-server', image=image, flavor=flavor, wait=True, auto_ip=True)
    print(server.name)
    print(server['name'])
    cloud.pprint(server)
    # Delete it - this is a demo
    cloud.delete_server(server, wait=True, delete_ips=True)

Delete Servers

delete_ips Delete any floating_ips the server may have

cloud.delete_server('my-server', wait=True, delete_ips=True)

Image and Flavor by Dict

Pass dict to image/flavor
If you know if the value is Name or ID
Common pattern
examples/cloud/create-server-dict.py

from openstack import cloud as openstack

# Initialize and turn on debug logging
openstack.enable_logging(debug=True)

for cloud_name, region_name, image, flavor_id in [
        ('my-vexxhost', 'ca-ymq-1', 'Ubuntu 16.04.1 LTS [2017-03-03]',
         '5cf64088-893b-46b5-9bb1-ee020277635d'),
        ('my-citycloud', 'Buf1', 'Ubuntu 16.04 Xenial Xerus',
         '0dab10b5-42a2-438e-be7b-505741a7ffcc'),
        ('my-internap', 'ams01', 'Ubuntu 16.04 LTS (Xenial Xerus)',
         'A1.4')]:
    # Initialize cloud
    cloud = openstack.connect(cloud=cloud_name, region_name=region_name)

    # Boot a server, wait for it to boot, and then do whatever is needed
    # to get a public ip for it.
    server = cloud.create_server(
        'my-server', image=image, flavor=dict(id=flavor_id),
        wait=True, auto_ip=True)
    # Delete it - this is a demo
    cloud.delete_server(server, wait=True, delete_ips=True)

Munch Objects

Behave like a dict and an object
examples/cloud/munch-dict-object.py

from openstack import cloud as openstack
openstack.enable_logging(debug=True)

cloud = openstack.connect(cloud='zetta', region_name='no-osl1')
image = cloud.get_image('Ubuntu 14.04 (AMD64) [Local Storage]')
print(image.name)
print(image['name'])

API Organized by Logical Resource

list_servers
search_servers
get_server
create_server
delete_server
update_server

For other things, it's still {verb}_{noun}

attach_volume
wait_for_server
add_auto_ip

Cleanup Script

Sometimes my examples had bugs
examples/cloud/cleanup-servers.py

from openstack import cloud as openstack

# Initialize and turn on debug logging
openstack.enable_logging(debug=True)

for cloud_name, region_name in [
        ('my-vexxhost', 'ca-ymq-1'),
        ('my-citycloud', 'Buf1'),
        ('my-internap', 'ams01')]:
    # Initialize cloud
    cloud = openstack.connect(cloud=cloud_name, region_name=region_name)
    for server in cloud.search_servers('my-server'):
        cloud.delete_server(server, wait=True, delete_ips=True)

Normalization

examples/cloud/normalization.py

from openstack import cloud as openstack
openstack.enable_logging()

cloud = openstack.connect(cloud='fuga', region_name='cystack')
image = cloud.get_image(
    'Ubuntu 16.04 LTS - Xenial Xerus - 64-bit - Fuga Cloud Based Image')
cloud.pprint(image)

Strict Normalized Results

Return only the declared model
examples/cloud/strict-mode.py

from openstack import cloud as openstack
openstack.enable_logging()

cloud = openstack.connect(
    cloud='fuga', region_name='cystack', strict=True)
image = cloud.get_image(
    'Ubuntu 16.04 LTS - Xenial Xerus - 64-bit - Fuga Cloud Based Image')
cloud.pprint(image)

How Did I Find the Image Name for the Last Example?

I often make stupid little utility scripts
examples/cloud/find-an-image.py

from openstack import cloud as openstack
openstack.enable_logging()

cloud = openstack.connect(cloud='fuga', region_name='cystack')
cloud.pprint([
    image for image in cloud.list_images()
    if 'ubuntu' in image.name.lower()])

Added / Modified Information

Servers need more extra help
Fetch addresses dict from neutron
Figure out which IPs are good
detailed - defaults to True, add everything
bare - no extra calls - don't even fix broken things
bare is still normalized
examples/cloud/server-information.py

from openstack import cloud as openstack
openstack.enable_logging(debug=True)

cloud = openstack.connect(cloud='my-citycloud', region_name='Buf1')
try:
    server = cloud.create_server(
        'my-server', image='Ubuntu 16.04 Xenial Xerus',
        flavor=dict(id='0dab10b5-42a2-438e-be7b-505741a7ffcc'),
        wait=True, auto_ip=True)

    print("\n\nFull Server\n\n")
    cloud.pprint(server)

    print("\n\nTurn Detailed Off\n\n")
    cloud.pprint(cloud.get_server('my-server', detailed=False))

    print("\n\nBare Server\n\n")
    cloud.pprint(cloud.get_server('my-server', bare=True))

finally:
    # Delete it - this is a demo
    cloud.delete_server(server, wait=True, delete_ips=True)

Exceptions

All openstacksdk exceptions are subclasses of OpenStackCloudException
Direct REST calls throw OpenStackCloudHTTPError
OpenStackCloudHTTPError subclasses OpenStackCloudException and requests.exceptions.HTTPError
OpenStackCloudURINotFound for 404
OpenStackCloudBadRequest for 400

User Agent Info

Set app_name and app_version for User Agents
(ssh ... region_name is optional if the cloud has one region)
examples/cloud/user-agent.py

from openstack import cloud as openstack
openstack.enable_logging(http_debug=True)

cloud = openstack.connect(
    cloud='datacentred',
    app_name='AmazingApp',
    app_version='1.0',
)
cloud.list_networks()

Uploading Large Objects

swift has a maximum object size
Large Objects are uploaded specially
openstacksdk figures this out and does it
multi-threaded
examples/cloud/upload-object.py

from openstack import cloud as openstack
openstack.enable_logging(debug=True)

cloud = openstack.connect(cloud='ovh', region_name='SBG1')
cloud.create_object(
    container='my-container',
    name='my-object',
    filename='/home/mordred/briarcliff.sh3d',
)
cloud.delete_object('my-container', 'my-object')
cloud.delete_container('my-container')

Uploading Large Objects

Default max_file_size is 5G
This is a conference demo
Let's force a segment_size
One MILLION bytes
examples/cloud/upload-object.py

from openstack import cloud as openstack
openstack.enable_logging(debug=True)

cloud = openstack.connect(cloud='ovh', region_name='SBG1')
cloud.create_object(
    container='my-container',
    name='my-object',
    filename='/home/mordred/briarcliff.sh3d',
    segment_size=1000000,
)
cloud.delete_object('my-container', 'my-object')
cloud.delete_container('my-container')

Service Conditionals

from openstack import cloud as openstack
openstack.enable_logging(debug=True)

cloud = openstack.connect(cloud='kiss', region_name='region1')
print(cloud.has_service('network'))
print(cloud.has_service('container-orchestration'))

Service Conditional Overrides

Sometimes clouds are weird and figuring that out won't work

from openstack import cloud as openstack
openstack.enable_logging(debug=True)

cloud = openstack.connect(cloud='rax', region_name='DFW')
print(cloud.has_service('network'))

clouds:
  rax:
    profile: rackspace
    auth:
      username: mordred
      project_id: 245018
    # This is already in profile: rackspace
    has_network: false

21 KiB Raw Blame History

Multi-Cloud Demo

Who am I?

What are we going to talk about?

OpenStackSDK is Free Software

This talk is Free Software, too

Complete Example

Let's Take a Few Steps Back

Cloud Terminology

Cloud Terminology Relationships

HTTP Sessions

Cloud Regions

Users, Projects and Domains

Confused Yet?

Auth per cloud, select per region

clouds.yaml

What about Mac and Windows?

Config Terminology

Environment Variables and Simple Usage

TOO MUCH TALKING - NOT ENOUGH CODE

basic clouds.yaml for the example code

secure.yaml

Example secure.yaml

more clouds.yaml

Much more complex clouds.yaml example

Complete Example Again

Step By Step

Import the library

Logging

Example with Debug Logging

Example with HTTP Debug Logging

Cloud Regions

Upload an Image

Always Upload an Image

Find a flavor

Create a server

Wow. We didn't even deploy Wordpress!

Image and Flavor by Name or ID

Delete Servers

Image and Flavor by Dict

Munch Objects

API Organized by Logical Resource

Cleanup Script

Normalization

Strict Normalized Results

How Did I Find the Image Name for the Last Example?

Added / Modified Information

Exceptions

User Agent Info

Uploading Large Objects

Uploading Large Objects

Service Conditionals

Service Conditional Overrides

FIN

21 KiB

Raw Blame History