42 KiB

Raw Permalink Blame History

Everything you need to know about using the OpenStack API

This document contains a presentation in presentty format. If you want to walk through it like a presentation, install presentty and run:

presentty doc/source/user/multi-cloud-demo.rst

The content is hopefully helpful even if it's not being narrated, so it's being included in the shade docs.

Everything you need to know about using the OpenStack API

Who am I?

Monty Taylor

OpenStack Infra Core
irc: mordred
twitter: @e_monty

This Talk is Free Software

https://opendev.org/inaugust/inaugust.com/src/branch/master/src/everything-you-need-to-know.rst

What are we going to talk about?

Overview of some terminology
Discussion of SDKs
'Simple' example
Step by step through in much more detail
- Auth
- The Catalog
- Service Types
- Version Discovery
- Actually doing things
- Microversions

Expected Ranting

Fixed vs. Floating IPs
Build your own images
Boot-time scripting with cloud-init

Relevant Docs

Cloud Terminology

Let's define a few terms, so that we can use them with ease:

cloud - logically related collection of services
region - completely independent subset of a given cloud
patron - human who has an account
user - account on a cloud
project - logical collection of cloud resources
domain - collection of users and projects

Cloud Terminology Relationships

A cloud has one or more regions
A patron has one or more users
A patron has one or more projects
A cloud has one or more domains
In a cloud with one domain it is named "default"
Each patron may have their own domain
Each user is in one domain
Each project is in one domain
A user has one or more roles on one or more projects

HTTP Sessions

HTTP interactions are authenticated via keystone
Authenticating returns a token
An authenticated HTTP Session is shared across a region

Cloud Regions

A cloud region is the basic unit of REST interaction.

A cloud has a service catalog
The service catalog is returned in the token
The service catalog lists endpoint for each service in each region
A region is completely autonomous

Availability Zones

If you have exposure to AWS - OpenStack Availability Zones are not what you think they are.
OpenStack Availability Zones are tagged groupings of hypervisor hosts, storage nodes and network nodes.

Users, Projects and Domains

In clouds with multiple domains, project and user names are only unique within a region.

Names require domain information for uniqueness. IDs do not.
Providing domain information when not needed is fine.
project_name requires project_domain_name or project_domain_id
project_id does not
username requires user_domain_name or user_domain_id
user_id does not

Auth vs. Identity

Both provided by the keystone service
identity refers to CRUD operations on users, projects, etc.
auth refers to a user getting a token and catalog

Keystone v2 vs. Keystone v3

Two major versions of identity API
v2 is deprecated/legacy
project was called tenant in v2
domain did not exist in v2

Confused Yet?

Awesome

SDKs

REST

You can always do direct REST
Auth and discovery are a bit of a pain
Once you have those, it's all pretty easy

python

First:

python-openstackclient ... CLI tool openstack

Five main libraries to be aware of:

keystoneauth
os-client-config
shade
python-openstacksdk
python-*client

python-*client

Per-service client libraries
Avoid them. They're not written for you. They make things harder.

keystoneauth

wrapper around python-requests that handles auth, catalog endpoint and version discovery
"REST" library

os-client-config

Library to handle client configuration
Can create python REST clients for you
Used under the covers by shade, python-openstacksdk and python-openstackclient
Handles clouds.yaml files, environment variables and command line arguments

shade

a task and end-user oriented Python library
abstracts deployment differences
designed for multi-cloud
simple to use
massive scale
- optional advanced features to handle 20k servers a day
Initial logic/design extracted from nodepool
Librified to re-use in Ansible

python-openstacksdk

Object oriented interface to OpenStack REST APIs
Just got adopted by the shade team
Interface is going to break soon
shade, python-openstacksdk and os-client-config are going to merge
shade is safe to use - there will always be backwards compat for shade

Multi-Cloud Shade Example

import shade

# Initialize and turn on debug logging
shade.simple_logging(debug=True)

for cloud_name, region_name in [
        ('my-vexxhost', 'ca-ymq-1'),
        ('my-citycloud', 'Buf1'),
        ('my-internap', 'ams01')]:
    # Initialize cloud
    cloud = shade.openstack_cloud(cloud=cloud_name, region_name=region_name)

    # Upload an image to the cloud
    image = cloud.get_image('Fedora 24 (Cloud Edition) [2016-09-23]')

    # Find a flavor with at least 512M of RAM
    flavor = cloud.get_flavor_by_ram(512)

    # Boot a server, wait for it to boot, and then do whatever is needed
    # to get a public ip for it.
    cloud.create_server(
        'my-server', image=image, flavor=flavor, wait=True, auto_ip=True)

It's Not Possible To Cover Everything That Does

Simpler Example

It still may not be possible to cover it all.

import shade

# Initialize and turn on debug logging
shade.simple_logging(debug=True)

# Initialize cloud
cloud = shade.openstack_cloud(cloud='vexxhost')

# Upload an image to the cloud
image = cloud.get_image('Fedora 24 (Cloud Edition) [2016-09-23]')

# Find a flavor with at least 512M of RAM
flavor = cloud.get_flavor_by_ram(512)

# Boot a server, wait for it to boot, and then do whatever is needed
# to get a public ip for it.
cloud.create_server(
    'my-server', image=image, flavor=flavor, wait=True, auto_ip=True)

Get Some Stuff Installed

virtualenv venv
source venv/bin/activate
pip install shade

Essential pieces of Auth Information

auth_url
password
(username + (user_domain_name or user_domain_id)) or user_id)
(project_name + (project_domain_name or project_domain_id)) or project_id

Note: It's important to know if you have a name or id for your user and: project.

I Lied

Essential pieces of Auth Information

auth_url
auth API version
auth_type

If nobody tells you an auth_type, assume password

If nobody tells you an auth_url, look at

https://docs.openstack.org/os-client-config/latest/user/vendor-support.html

Exercise - Find Auth URL for known cloud

The cloud is Vexxhost.

Answer

https://auth.vexxhost.net

Version Discovery

Each OpenStack service has a 'Version Discovery' document at its unversioned root URL. We call this the 'Unversioned Discovery Document'
- Except for swift. (pending some work with discovery caching)
Each OpenStack service ALSO has a 'Version Discovery' document at the root of each Versioned URL. We call this the 'Versioned Discovery Document'
Version Discovery documents should not require Authentication
- If they do on your cloud, your cloud is broken, please file a bug

Exercise #2 - Find Auth API Version from Auth URL

curl -i https://auth.vexxhost.net

Auth Version Discovery Response

{
  "versions": {
    "values": [
      {
        "status": "stable",
        "updated": "2017-02-22T00:00:00Z",
        "media-types": [
          {
            "base": "application/json",
            "type": "application/vnd.openstack.identity-v3+json"
          }
        ],
        "id": "v3.8",
        "links": [
          {
            "href": "https://auth.vexxhost.net/v3/",
            "rel": "self"
          }
        ]
      },

Auth Version Discovery Response Part 2

::

{: "status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json" } ], "id": "v2.0", "links": [ { "href": "https://auth.vexxhost.net/v2.0/", "rel": "self" }, { "href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby" } ]

]

}

Versioned Discovery Document

Sometimes a cloud gives you a versioned auth_url.

curl -i https://auth.vexxhost.net/v3

Auth Version Discovery Resonse

{
  "version": {
    "status": "stable",
    "updated": "2017-02-22T00:00:00Z",
    "media-types": [
      {
        "base": "application/json",
        "type": "application/vnd.openstack.identity-v3+json"
      }
    ],
    "id": "v3.8",
    "links": [
      {
        "href": "https://auth.vexxhost.net/v3/",
        "rel": "self"
      }
    ]
  }
}

Finding a Better Version

Sometimes a cloud gives you a v2 versioned auth_url.

curl -i https://auth.vexxhost.net/v2.0

ALWAYS prefer v3 auth if it's available.

Old Auth Version Discovery Resonse

{
  "version": {
    "status": "deprecated",
    "updated": "2016-08-04T00:00:00Z",
    "media-types": [
      {
        "base": "application/json",
        "type": "application/vnd.openstack.identity-v2.0+json"
      }
    ],
    "id": "v2.0",
    "links": [
      {
        "href": "https://auth.vexxhost.net/v2.0/",
        "rel": "self"
      },
      {
        "href": "https://docs.openstack.org/",
        "type": "text/html",
        "rel": "describedby"
      }
    ]
  }
}

Status Deprecated

Strip the trailing v.* portion from the URL and try again.

{
  "version": {
    "status": "deprecated",

Strip the trailing v.* portion from the URL and try again.

If v3 is not available, complain to someone. Keystone v3 api is considered stable as of February 20, 2013.

Isn't this fun?

To Domain or Not To Domain

In v3:

Project Names and User Names are only unique within a domain
If you don't have domain info, assume domain_id is default

In v2:

No such thing as domains
Projects are called "tenant"

Essential pieces of Auth Information

auth_url - https://auth.vexxhost.net
auth version - 3
versioned auth endpoint - https://auth.vexxhost.net/v3
username - osdn-user-1
user domain id - default
project_name - osdn-project-1
project domain id = default
password - XXXXXXXX

Now we can authenticate!!!

curl -g -i -X POST https://auth.vexxhost.net/v3/auth/tokens -H "Content-Type: application/json" -H "Accept: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": {"user": {"domain": {"id": "default"}, "name": "osdn-user-1", "password": "XXXXXXXX"}}},"scope": {"project": {"domain": {"id": "default"}, "name": "osdn-project-1"}}}}'

What, you can't read that?

curl -g -i -X POST https://auth.vexxhost.net/v3/auth/tokens \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" -d '{
    "auth": {
      "identity": {
        "methods": ["password"],
      "password": {
        "user": {
          "domain": {"id": "default"},
          "name": "osdn-user-1",
          "password": "XXXXXXXX"}}},
      "scope": {
        "project": {
          "domain": {"id": "default"},
          "name": "osdn-project-1"}}}}'

Authentication Response

Readable version of payload: http://paste.openstack.org/raw/623947/

HTTP/1.1 201 Created
Date: Tue, 17 Oct 2017 23:16:05 GMT
Server: Apache/2.4.6 (CentOS)
X-Subject-Token: gAAAAABZ5o81tyDsyJMXl6hVcVOqHgzKQ2WmpvepCSr2NWYM1D97VoaBy-i5OLT8-BbkWBZJq_aqrt03eutp2eEpORI-V4CYpYcWBDkR5jLNfq2gD-TDB9lzvRo7rHKu-PWSLIxEF7whsFJ8bVGDPMN4yc_ZwgfWcR13GxAKutU9_rzR9Gvewjg
Vary: X-Auth-Token
x-openstack-request-id: req-29d8c4e6-1924-42e9-8efa-4f154e813ee6
Content-Length: 11696
Content-Type: application/json

{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "fd6d2130b7784b82aa6b49fd5901f70b", "name": "Member"}], "expires_at": "2017-10-18T23:16:05.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "bbb1e479a9d04f34941470f37c7a0ecb", "name": "osdn-project-1"}, "catalog": [{"endpoints": [{"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "522041f846934310b928a6afab47241e"}, {"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "660a742372554b5581c1f4870d4d42cf"}, {"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "c1c32be0de8441eea4306b8c22442ba3"}], "type": "key-manager", "id": "023de54c09654931856403e708b60a22", "name": "barbican"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "public", "id": "289f7dcf096743fb9a17924bd598e716"}, {"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "admin", "id": "61b6d283ef4b419a8ad23fc53dea70df"}, {"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "internal", "id": "e1e7c79ce7834ec0aa74308326e2f96c"}], "type": "container-infra", "id": "041eac2489bb495b8a900c40b93b2065", "name": "magnum"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "203fd4e466b44569aa9ab8c78ef55bad"}, {"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "da3d087e0c724338ba12c9a1168ef80c"}, {"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "f0a311cb5dbf4ae788670107a3433ac2"}], "type": "orchestration", "id": "0e852aec60e6412b88b066e62c2cd940", "name": "heat"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "4f1fafb25e58421e91a2139dc0382c53"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "5808ab12037f4ab49c37cba0ac9cd58e"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "a680b7674b3b430c9f5e008f72f3dfe4"}], "type": "s3", "id": "13f4cf29c7394a7a801d8aeacca24378", "name": "swift_s3"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "internal", "id": "0bba70ede2ef4bc0a998ed73110406cb"}, {"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "public", "id": "25ee42ace1ad4707b485494baa5d4693"}, {"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "admin", "id": "cf817bfde96d4aa6807cdb4be21eca3b"}], "type": "dns", "id": "1b6455f4855e44bd85be449258340a02", "name": "designate"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://identity-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "8df18f47fcdc4c348d521d4724a5b7ac"}, {"region_id": "ca-ymq-1", "url": "https://auth.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "c0dd9f5f8db248b093d6735b167e1af6"}, {"region_id": "ca-ymq-1", "url": "https://auth.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "c8505f07c349413aa7cd61d42337af99"}], "type": "identity", "id": "1ca1ba37600a436185c317bef6d88c36", "name": "keystone"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "299df431ac7942c18e2137613d8e03d8"}, {"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "bec896cde16646d6a3b56cff997d08cc"}, {"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "ce70c7029b6a45b78e008bc833187559"}], "type": "placement", "id": "37077d282f2b4961ad65f9fb87f58789", "name": "placement"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "4babd2f2b8d842e4b2303468fbf5d3ac"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "6b3ddcf427e04182a7aa4e24d4649581"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "e8e1eb90f7394f5999aec5c8f8c75c88"}], "type": "volume", "id": "40686c5446374c2b92771e67249521b3", "name": "cinder"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "218d9460bd6e4eabb32b980fc580a26a"}, {"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "21a2898a0fcf4ec783974cf75d774916"}, {"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "8842c03d2c51449ebf9ff36778cf17c1"}], "type": "image", "id": "49e5dc4cfe294aa0971e797d3bb1db20", "name": "glance"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "admin", "id": "83fbacf0d0014a27955384fb8632de4c"}, {"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "internal", "id": "9308a40f3a6f4dff9d20afa2dfa6b186"}, {"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "public", "id": "e5cb7e4e6f3440a386d397ec7e00ca44"}], "type": "cloudformation", "id": "4e77820be3494d26a0c39301082da4d7", "name": "heat-cfn"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "5fdfd118a3d6460ab45c1995ca21b7f1"}, {"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "ab9e70b4fde943e49d92c41d2f0892d9"}, {"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "cfa16dc27b82420a8cf7db5ea79aa3dd"}], "type": "data-processing", "id": "523b8e9f4bd44e51aec65c524c953c32", "name": "sahara"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "37860b492dd947daa738f461b9084d2a"}, {"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "7a095734e4984cc7b8ac581aa6131f23"}, {"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "96357df3d6694477b0ad17fef6091210"}], "type": "network", "id": "98f33898880441ddb91522bac15410b5", "name": "neutron"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "01fdd8e07ca74c9daf80a8b66dcc8bf6"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "a25efaf48347441a8d36ce302f31d527"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "ceea6017e14e4b549882b6fe359d07e0"}], "type": "volumev2", "id": "9ec8ed2478e44b368f51dd71a6f2759c", "name": "cinderv2"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "2f582f99db974766af7548dda56c3b50"}, {"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "39e28fa99b69466195aad1fbfb610323"}, {"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "4d38fa91197e4712a2f2d3f89fcd7dad"}], "type": "compute", "id": "a30d9b8a170144f0a7bca3ce028c9ed3", "name": "nova"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "5d9aa667f2fb4c83bf14ee8ae52b541a"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "96bcaf12b54d4060bc9973ded5ce4093"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "ac9138596d3942babe1f9241f93a5e04"}], "type": "volumev3", "id": "a5ae83af15b146559711905c6531c721", "name": "cinderv3"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "39830aba2eea44a3a432ea0bd219c1ab"}, {"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "5883ae0fb85842c5aca05ed51327ac35"}, {"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "80bd3d21df274c43ac8457aa2230598f"}], "type": "octavia", "id": "b6a1520b90a146f4a96321960e7a243a", "name": "octavia"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "http://secure.vexxhost.com/console/api", "region": "ca-ymq-1", "interface": "public", "id": "156bf3730b834d2bacf7eff6c542f394"}], "type": "dashboard", "id": "b7bb9d5476c649f483dfbcb49359c17b", "name": "cloudconsole"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "20b24181722b49a3983d17d42147a22c"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "79fa33ff42ec45118ae8b36789fcb8ae"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "b073b767f10d44f895d9d14fbc3e3d6b"}], "type": "object-store", "id": "c1efec5454ec49b1b016721938401182", "name": "swift"}], "user": {"password_expires_at": null, "domain": {"id": "default", "name": "Default"}, "id": "bc2260d5b7cb40ff8270043fd7af3a08", "name": "osdn-user-1"}, "audit_ids": ["6bu2KObmSwGfO2gsIcmPNw"], "issued_at": "2017-10-17T23:16:05.000000Z"}}

Authentication Response Readable Version

http://paste.openstack.org/raw/623947/

Listing Images

Find 'image' Endpoint from the Catalog
Find image api version
Make API call

Consuming Service Catalog

https://specs.openstack.org/openstack/api-wg/guidelines/consuming-catalog.html

Docs on version discovery and cloud profile:

https://review.openstack.org/#/c/459869 Add guideline describing a cloud profile document
https://review.openstack.org/#/c/459710 Add guidelines on Version Discovery
https://review.openstack.org/#/c/459405 Add document describing consuming version discovery

Find Endpoint

Most important piece of info: service_type (type in catalog)
service_name or name carries no meaning

Service Types Authority

https://service-types.openstack.org

Registry of known service types
Listing of historical aliases for those service types
https://service-types.openstack.org/service-types.json
Not release versoined - most current file is always accurate
Python library: os-service-types

Find Image Endpoint

{'endpoints': [
  {'id': '218d9460bd6e4eabb32b980fc580a26a',
   'interface': 'internal',
   'region': 'ca-ymq-1',
   'region_id': 'ca-ymq-1',
   'url': 'https://image-ca-ymq-1.vexxhost.net'},
  {'id': '21a2898a0fcf4ec783974cf75d774916',
   'interface': 'admin',
   'region': 'ca-ymq-1',
   'region_id': 'ca-ymq-1',
   'url': 'https://image-ca-ymq-1.vexxhost.net'},
  {'id': '8842c03d2c51449ebf9ff36778cf17c1',
   'interface': 'public',
   'region': 'ca-ymq-1',
   'region_id': 'ca-ymq-1',
   'url': 'https://image-ca-ymq-1.vexxhost.net'}],
 'id': '49e5dc4cfe294aa0971e797d3bb1db20',
 'name': 'glance',
 'type': 'image'},

Endpoint Interface

public - Public/External API access - default value
internal - Internal network API access
admin - Historical - was used for keystone - not used anymore

Endpoint Region Name

If there is more than one region, region_name must be specified
If there is only one, it can be omitted

Example with more than one region:

http://paste.openstack.org/raw/623900/

Endpoints Found

cloud=vexxhost1,service_type=image

https://image-ca-ymq-1.vexxhost.net

cloud=citycloud1,service_type=image,region_name=Sto2

https://sto2.citycloud.com:9292

Finding Image API Version

curl -i https://image-ca-ymq-1.vexxhost.net

http://paste.openstack.org/raw/623901/

CURRENT - Best current version

https://image-ca-ymq-1.vexxhost.net/v2/

Listing Images

Send value of X-Subject-Token from auth request in X-Auth-Token

curl -i -X GET https://image-ca-ymq-1.vexxhost.net/v2/images \
    -H "Accept: application/json" \
    -H "X-Auth-Token: gAAAAABZ5qpnytYif9wk-4fsqvgm3Ik81WQl_L4tdlfka7Vr4b-EG0w7QGF-JU_T4zkJsuql4XYSU_9epUnx-aE_UeKMMdaKAyecGG9om6TR3CA6yeeHbMJBTVFfdxvW-QFA76_JXoibCvXGIcKlpotK01BBl3DG5qYNxUFkSquk5cIQh-w63AY"

HTTP/1.1 200 OK
Content-Length: 24153
Content-Type: application/json
X-Openstack-Request-Id: req-8a602280-1bba-46b4-9406-97addcdd9b6c
Date: Wed, 18 Oct 2017 01:12:43 GMT

Content: http://paste.openstack.org/raw/623902/
Note pagination links

Getting Auth information

Download a clouds.yaml file from horizon
Download an legacy openrc file from horizon
Investigate dashboard
Carrier pigeon

Sometimes the files are wrong and you have to apply the earlier rules.

Incorrect openrc file (horizon bug)

This is downloaded from my account. Who can spot the problems?

export OS_AUTH_URL=https://auth.vexxhost.net/v2.0
export OS_PROJECT_ID=bbb1e479a9d04f34941470f37c7a0ecb
export OS_PROJECT_NAME="osdn-project-1"
export OS_USER_DOMAIN_NAME="None"
export OS_USERNAME="osdn-user-1"
export OS_REGION_NAME="ca-ymq-1"
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3

Put the info in a clouds.yaml file

clouds.yaml

Information about the clouds you want to connect to is stored in a file called clouds.yaml.

clouds.yaml can be in your homedir: ~/.config/openstack/clouds.yaml or system-wide: /etc/openstack/clouds.yaml.

Information in your homedir, if it exists, takes precedence.

Full docs on clouds.yaml are at https://docs.openstack.org/developer/os-client-config/

What about Mac and Windows?

USER_CONFIG_DIR is different on Linux, OSX and Windows.

Linux: ~/.config/openstack
OSX: ~/Library/Application Support/openstack
Windows: C:UsersUSERNAMEAppDataLocalOpenStackopenstack

SITE_CONFIG_DIR is different on Linux, OSX and Windows.

Linux: /etc/openstack
OSX: /Library/Application Support/openstack
Windows: C:ProgramDataOpenStackopenstack

Config Terminology

For multi-cloud, think of two types:

profile - Facts about the cloud that are true for everyone
cloud - Information specific to a given user

Apologies for the use of cloud twice.

Example clouds.yaml file

clouds:
  vexxhost1:
    identity_api_version: 3
    auth:
      auth_url: https://auth.vexxhost.net
      user_domain_id: default
      project_domain_id: default
      project_name: osdn-project-1
      username: osdn-user-1
      password: XXXXXXXX

Vexxhost is known

clouds:
  vexxhost1:
    identity_api_version: 3
    profile: vexxhost
    auth:
      user_domain_id: default
      project_domain_id: default
      project_name: osdn-project-1
      username: osdn-user-1
      password: XXXXXXXX

More than one cloud

Yes- I created the same user with the same password on CityCloud:

clouds:
  citycloud1:
    profile: citycloud
    auth:
      username: osdn-user-1
      password: XXXXXXXX
      project_name: osdn-project-1
      user_domain_id: d0919bd5e8d74e49adf0e145807ffc38
      project_domain_id: d0919bd5e8d74e49adf0e145807ffc38
  vexxhost1:
    profile: vexxhost
    auth:
      user_domain_id: default
      project_domain_id: default
      project_name: osdn-project-1
      username: osdn-user-1
      password: XXXXXXXX

Quick note about SSL

You can disable SSL verification:

unitedstack:
  profile: unitedstack
  verify: False

You can set specific SSL arguments:

openstackci-infracloud-vanilla:
  region_name: RegionOne
  cacert: /home/nodepool/.config/openstack/infracloud_vanilla_cacert.pem
  auth:
    auth_url: https://controller00.vanilla.ic.openstack.org:5000
    username: openstackci

Auth per cloud, select per region

In general, the thing you need to know is:

Configure authentication per cloud
Select config to use by cloud and region

Environment Variables and Simple Usage

Environment variables starting with OS_ go into a cloud called envvars
If you only have one cloud, you don't have to specify it
OS_CLOUD and OS_REGION_NAME are default values for cloud and region_name

Easier version of the above REST

What you should be able to do

import os_client_config
client = os_client_config.make_rest_client('image', cloud='vexxhost1')
print(client.get('/images').json())

There's a bug with make_rest_client and glance - I'll get it fixed this week. Workaround:

import os_client_config
from keystoneauth1 import adapter
config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1')
client = adapter.Adapter(
  config.get_session(), service_type='image', version='latest')
print(client.get('/images').json())

Sessions and Adapters

A Session is an Authenticated HTTP connection to a cloud

config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1')
session = config.get_session()
session.get('https://image-ca-ymq-1.vexxhost.net/v2/images')

An "Adapter" is a Session that is 'mounted' on a base endpoint

config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1')
client = adapter.Adapter(
  config.get_session(), service_type='image', version='latest')
session.get('/images')

os_client_config.make_rest_client returns a mounted Adapter
Latest keystoneauth library handles version discovery properly in Adapter

Creating keystoneauth session directly

from keystoneauth1 import adapter, loading, session
loader = loading.get_plugin_loader('password')
auth = loader.load_from_options(
  auth_url='https://auth.vexxhost.net',
  username='osdn-user-1',
  user_domain_id='default',
  password='XXXXXXXX',
  project_name='osdn-project-1',
  project_domain_id='default')
# verify and ssl cert info go here too
session = session.Session(auth)
# interface, region_name also go here
client = adapter.Adapter(session, service_type='image', version='latest')
client.get('/images')

Command Line

openstack --os-cloud=vexxhost1 image list

Back to shade

import shade
cloud = shade.openstack_cloud(cloud='vexxhost')
cloud.pprint(cloud.list_images())

Shade Logging

shade uses standard python logging
simple_logging does easy defaults
Squelches some meaningless warnings
- debug
  - Logs shade loggers at debug level
- http_debug Implies debug, turns on HTTP tracing

# Initialize and turn on debug logging
shade.simple_logging(debug=True)

Find out what REST calls shade is making

import shade
shade.simple_logging(http_debug=True)
cloud = shade.openstack_cloud(cloud='vexxhost1')
cloud.pprint(cloud.list_images())

http://paste.openstack.org/raw/623906/

Real tokens are not logged, only a sha of the token. Reusing the printed curl commands reuquires replacing the token with a real one.

"X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e"

Microversions

Per-HTTP-request behavior differences.
Services declare microversion support ranges in Version Discovery Document

curl -i https://compute-ca-ymq-1.vexxhost.net

Nova Compute Version Document Part One

HTTP/1.1 200 OK
Date: Wed, 18 Oct 2017 01:58:51 GMT
Server: Apache/2.4.6 (CentOS)
Content-Length: 399
Vary: Accept-Encoding
Content-Type: application/json

{
  "versions": [
    {
      "status": "SUPPORTED",
      "version": "",
      "updated": "2011-01-21T11:33:21Z",
      "id": "v2.0",
      "links": [
        {
          "href": "https://compute-ca-ymq-1.vexxhost.net/v2/",
          "rel": "self"
        }
      ],
      "min_version": ""
    },

Nova Compute Version Document Part Two

Note 'version' and 'min_version'.

{
  "status": "CURRENT",
  "version": "2.53",
  "updated": "2013-07-23T11:33:21Z",
  "id": "v2.1",
  "links": [
    {
      "href": "https://compute-ca-ymq-1.vexxhost.net/v2.1/",
      "rel": "self"
    }
  ],
  "min_version": "2.1"
}

Microversion Field Names

The min and max available microversion for the given Major API version.

In some services they are:

min_version
version

In other services they are:

min_version
max_version

Request a Microversion

Add a OpenStack-API-Version header with service-type and version:

OpenStack-API-Version: compute 2.13
Response will contain the same header specifying which version was used:

OpenStack-API-Version: compute 2.13
Older versions of Nova (pre Newton - or versions that do not support a max version of 2.27 or higher), use a different form:

X-OpenStack-Nova-API-Version: 2.13
It doesn't hurt anything to send both - which is what keystoneauth does under the covers.

Documentation on Nova Microversion Changes

https://docs.openstack.org/nova/latest/reference/api-microversion-history.html

Simple Microversion Example with Default

Microversion 2.36 removed some calls. Before 2.36 nova had a /images API that would proxy to glance. It is removed in 2.36 and later.

Making a GET call defaults to the minimum (2.1 in this case)

import os_client_config

client = os_client_config.make_rest_client('compute', cloud='vexxhost')
client.get('/images')

curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/db92b20496ae4fbda850a689ea9d563f/images -H "X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e" -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2"
[200] Date: Wed, 18 Oct 2017 02:10:49 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-3e6bc318-dfd6-493b-9675-8bb1f9285cac x-compute-request-id: req-3e6bc318-dfd6-493b-9675-8bb1f9285cac Content-Encoding: gzip Content-Length: 2411 Content-Type: application/json

Simple Microversion Example with Microversion

Sending microversion 2.36 for the same call tells Nova to use the 2.36 behavior, which results in a 404.

import os_client_config

client = os_client_config.make_rest_client('compute', cloud='vexxhost')
client.get('/images', microversion='2.36')

curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/db92b20496ae4fbda850a689ea9d563f/images -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" -H "X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e" -H "OpenStack-API-Version: compute 2.36" -H "X-OpenStack-Nova-API-Version: 2.36"
[404] Date: Wed, 18 Oct 2017 02:11:00 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.36 X-OpenStack-Nova-API-Version: 2.36 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version x-openstack-request-id: req-d733583e-d2c3-4ca4-bdfb-5e658e194028 x-compute-request-id: req-d733583e-d2c3-4ca4-bdfb-5e658e194028 Content-Length: 78 Content-Type: application/json; charset=UTF-8
{"itemNotFound": {"message": "The resource could not be found.", "code": 404}}

Create Server Group

The server group doesn't matter - this is just to have an object to show.

import os_client_config

client = os_client_config.make_rest_client('compute', cloud='vexxhost')
client.post(
    '/os-server-groups',
    json=dict(server_group=dict(name='test', policies=['anti-affinity'])))

Example of No Microversion Content Behavior

No microversion specified.

client.get('/os-server-groups')

curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb/os-server-groups -H "X-Auth-Token: {SHA1}fe541768b55bbe23f79f97595b137fed3326166a" -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2"
[200] Date: Wed, 18 Oct 2017 02:18:30 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-85e061c0-d474-497c-bd17-96b629086169 x-compute-request-id: req-85e061c0-d474-497c-bd17-96b629086169 Content-Encoding: gzip Content-Length: 137 Content-Type: application/json
{"server_groups": [{"members": [], "metadata": {}, "id": "f88c7aef-b34a-4b0e-b0c7-9076eab88de2", "policies": ["anti-affinity"], "name": "test"}]}

Example of Microversion Content Behavior

Request microversion 2.13, which adds user_id and project_id.

client.get('/os-server-groups', microversion='2.13').json()

curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb/os-server-groups -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" -H "X-Auth-Token: {SHA1}fe541768b55bbe23f79f97595b137fed3326166a" -H "OpenStack-API-Version: compute 2.13" -H "X-OpenStack-Nova-API-Version: 2.13"
[200] Date: Wed, 18 Oct 2017 02:20:13 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.13 X-OpenStack-Nova-API-Version: 2.13 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-6c71b008-185a-46c1-8762-5c2718be1741 x-compute-request-id: req-6c71b008-185a-46c1-8762-5c2718be1741 Content-Encoding: gzip Content-Length: 195 Content-Type: application/json
{"server_groups": [{"user_id": "bc2260d5b7cb40ff8270043fd7af3a08", "policies": ["anti-affinity"], "name": "test", "members": [], "project_id": "bbb1e479a9d04f34941470f37c7a0ecb", "id": "f88c7aef-b34a-4b0e-b0c7-9076eab88de2", "metadata": {}}]}

42 KiB Raw Permalink Blame History

Everything you need to know about using the OpenStack API

Everything you need to know about using the OpenStack API

Who am I?

This Talk is Free Software

What are we going to talk about?

Expected Ranting

Relevant Docs

Cloud Terminology

Cloud Terminology Relationships

HTTP Sessions

Cloud Regions

Availability Zones

Users, Projects and Domains

Auth vs. Identity

Keystone v2 vs. Keystone v3

Confused Yet?

SDKs

REST

python

python-*client

keystoneauth

os-client-config

shade

python-openstacksdk

Multi-Cloud Shade Example

It's Not Possible To Cover Everything That Does

Simpler Example

Get Some Stuff Installed

Essential pieces of Auth Information

I Lied

Essential pieces of Auth Information

Exercise - Find Auth URL for known cloud

Answer

Version Discovery

Exercise #2 - Find Auth API Version from Auth URL

Auth Version Discovery Response

Auth Version Discovery Response Part 2

]

Versioned Discovery Document

Auth Version Discovery Resonse

Finding a Better Version

Old Auth Version Discovery Resonse

Status Deprecated

Isn't this fun?

To Domain or Not To Domain

Essential pieces of Auth Information

Now we can authenticate!!!

What, you can't read that?

Authentication Response

Authentication Response Readable Version

Listing Images

Consuming Service Catalog

Find Endpoint

Service Types Authority

Find Image Endpoint

Endpoint Interface

Endpoint Region Name

Endpoints Found

Finding Image API Version

Listing Images

Getting Auth information

Incorrect openrc file (horizon bug)

Put the info in a clouds.yaml file

clouds.yaml

What about Mac and Windows?

Config Terminology

Example clouds.yaml file

Vexxhost is known

More than one cloud

Quick note about SSL

Auth per cloud, select per region

Environment Variables and Simple Usage

Easier version of the above REST

Sessions and Adapters

Creating keystoneauth session directly

Command Line

Back to shade

Shade Logging

Find out what REST calls shade is making

42 KiB

Raw Permalink Blame History