276 lines
16 KiB
Plaintext
276 lines
16 KiB
Plaintext
DefCore Scoring Worksheet
|
|
=========================
|
|
|
|
This file is used to maintain capabilities scoring for DefCore.
|
|
It is considered "working materials" under the 2016A process and
|
|
is not an authoritative, Board-approved source of information
|
|
as to which capabilities are required for a particular Guideline.
|
|
Rather, it is a point-in-time listing used by DefCore Committee
|
|
members while determining which capabilities may be included in
|
|
a future Guideline. Each line takes the following form:
|
|
|
|
Capability Name: [Widely Deployed,Used By Tools,Used by Clients],
|
|
[TC Future Direction, Complete, Stable],
|
|
[Discoverable, Doc'd, Core in Last Release],
|
|
[Foundational, Atomic, Proximity],
|
|
[Non-Admin] [Total]
|
|
|
|
Possible scores: 0, 1, ? (not scored yet or needs discussion)
|
|
|
|
The Total score is calculated by multiplying the score for each
|
|
Criteria by the weight assigned to that Criteria in the Guideline
|
|
JSON file. It is not necessary to calculate the score by hand:
|
|
use the tabulate_scores.py script found in this directory. The
|
|
script will also append an asterisk (*) to the end of the line if
|
|
the total score is greater than or equal to the cutoff_score listed
|
|
in the Guideline JSON file, indicating that the Capability
|
|
has scored high enough to merit inclusion in the Guideline.
|
|
|
|
Networking
|
|
----------
|
|
network-floating-ips-associate: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
|
|
network-floating-ips-create: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
|
|
network-floating-ips-update: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
|
|
network-floating-ips-delete: [1,1,1] [1,1,1] [0,1,0] [0,1,1] [1] [74]*
|
|
network-l2-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l2-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l2-external: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l2-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l2-port: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l2-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l2-update: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l3-add: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l3-convert: [0,0,0] [1,0,0] [0,0,0] [0,0,0] [1] [11]
|
|
network-l3-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l3-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l3-router: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l3-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-l3-update: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
networks-list-api-versions: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
|
|
network-security-groups-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-security-groups-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-security-groups-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-security-groups-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
network-extensions-provider-add: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
|
|
network-extensions-provider-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
|
|
network-extensions-provider-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
|
|
network-extensions-provider-show: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [0] [0]
|
|
networks-subnet-pools-CRUD: [1,0,1] [1,1,1] [1,1,0] [0,1,1] [1] [76]*
|
|
|
|
Notes:
|
|
* DVR-related testcases (if any) will not be added until DVR gains wider
|
|
acceptance and isn't driver-specific.
|
|
* IPv6 is currently not widely deployed, so v6 versions of these tests need
|
|
not be added at this time. Vendor feedback on v6 support is requested,
|
|
though it is understood that IPv6 capabilities may be driver-dependent.
|
|
* A few folks have asked about provider network extensions as they are
|
|
frequently used for high performance networks and a simplified networking
|
|
model that provides for use cases similar to nova-net. However with default
|
|
policy settings, using provider networks requires admin access and therefore
|
|
is disqualified. I've listed them here anyway just to show that they were
|
|
considered but can't be added to the Guideline.
|
|
* Based on feedback from earlier patchsets, capabilities will be grouped
|
|
together where they had previously been listed separately but are adjacent.
|
|
For example, there are several tests for network creation, but only one or
|
|
few for deletion, listing, and updating. Combining these into a single
|
|
"CRUD" group reduces the chance the chance of running into problems with
|
|
DefCore's 2015A process (specifically section C3(4)) in cases where there
|
|
is only a single test for a capability and the test needs to be flagged on
|
|
account of bugs or other valid reasons. The combined groups are:
|
|
* networks-floating-ips-CRUD-and-associate
|
|
* networks-floating-ips-associate
|
|
* networks-floating-ips-create
|
|
* networks-floating-ips-update
|
|
* networks-floating-ips-delete
|
|
* networks-l2-CRUD
|
|
* networks-l2-create
|
|
* networks-l2-delete
|
|
* networks-l2-external
|
|
* networks-l2-list
|
|
* networks-l2-port
|
|
* networks-l2-show
|
|
* networks-l2-update
|
|
* networks-l3-CRUD
|
|
* networks-l3-add
|
|
* networks-l3-create
|
|
* networks-l3-delete
|
|
* networks-l3-show
|
|
* networks-l3-update
|
|
* networks-l3-router
|
|
* networks-security-groups-CRUD
|
|
* networks-security-groups-create
|
|
* networks-security-groups-delete
|
|
* networks-security-groups-router
|
|
* networks-security-groups-show
|
|
* networks-security-groups-update
|
|
* Subnet pools were deemed not ready in 2016.08 but a good candidate for
|
|
the future. 2017.01 seems like the right time to make them advisory.
|
|
Subnet pools were introduced in Kilo but had incomplete support in
|
|
Horizon at the time. A Heat resource for subnet pools was added
|
|
in Mitaka, and OpenStackClient added support late in the Mitaka cycle
|
|
as well (https://review.openstack.org/#/c/279918/). If we make subnet
|
|
pools advisory now, they could become required in 2017.08--in which
|
|
the earliest release in the Guideline will be Mitaka. Toolkits like
|
|
jClouds, Fog, and libcloud do not yet support subnet pools that I see.
|
|
Subnet pools will be used by Neutron's upcoming get-me-a-network
|
|
capability and Kuryr appears to have plans to use them as part of
|
|
Kubernets integration.
|
|
* Extended port attribute tests were not considered because the capability
|
|
generally requires admin privileges.
|
|
* tempest.api.network.test_ports.PortsTestJSON.test_create_update_port_with_second_ip
|
|
was considered but was removed because not all Neutron plugins support
|
|
more than one IP per port (and some providers limit the number of
|
|
IP's per port for security reasons, e.g. spoofguard).
|
|
|
|
Compute
|
|
-------
|
|
compute-list-api-versions: [1,0,0] [1,1,1] [1,1,1] [1,1,1] [1] [84]*
|
|
compute-servers-suspend-resume: [1,1,1] [1,1,1] [1,1,0] [0,1,1] [1] [82]*
|
|
compute-flavors-list: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
|
|
compute-availability-zones-list: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
|
|
|
|
Image
|
|
-----
|
|
images-v1-delete: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
images-v1-index: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
images-v1-list: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
images-v1-register: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
images-v1-shared-images-add: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
images-v1-shared-images-delete: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
images-v1-shared-images-get: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
images-v1-shared-images-remove: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
images-v1-update: [0,1,1] [0,1,1] [0,1,0] [1,0,1] [1] [58]
|
|
|
|
images-v2-remove: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
|
|
images-v2-update: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
|
|
images-v2-share: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
|
|
images-v2-import: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
|
|
images-v2-list: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
|
|
images-v2-delete: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
|
|
images-v2-get: [1,0,1] [1,1,1] [1,1,0] [1,0,1] [1] [79]*
|
|
|
|
Notes:
|
|
* Image creation is captured under the register operation.
|
|
* No public image tests have explicit support for task API.
|
|
* Scoring for v1 remains in place, but v2 is preferred interop
|
|
standard (as reflected in worksheet).
|
|
* images-v2-share inherently requires more than one credential,
|
|
and currently out of scope for DefCore capabilities.
|
|
* images-v2-remove has no test as of Auguest, 2016.
|
|
|
|
Volume
|
|
----------
|
|
volumes-list-api-versions: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v2-create-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
volumes-v2-attach-detach: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
volumes-v2-snapshot-create-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
volumes-v2-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
volumes-v2-list: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
volumes-v2-update: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
volumes-v2-copy-image-to-volume: [1,0,0] [1,1,1] [1,1,1] [1,1,1] [1] [84]*
|
|
volumes-v2-clone: [1,0,1] [1,1,1] [1,1,1] [1,1,1] [1] [94]*
|
|
volumes-v2-availability-zones: [1,0,0] [1,1,1] [1,1,1] [1,1,0] [1] [76]*
|
|
volumes-v2-extensions: [1,0,0] [0,1,1] [1,1,1] [1,0,0] [1] [59]
|
|
volumes-v2-metadata: [1,0,1] [1,1,1] [1,1,1] [1,1,0] [1] [86]*
|
|
volumes-v2-reserve: [1,0,0] [1,1,1] [1,1,1] [1,1,0] [1] [76]*
|
|
volumes-v2-readonly: [1,0,1] [1,1,1] [1,1,1] [1,1,0] [1] [86]*
|
|
volumes-v2-upload: [1,0,1] [1,1,1] [1,1,0] [1,1,0] [1] [77]*
|
|
volumes-v3-create-delete: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v3-attach-detach: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v3-snapshot-create-delete: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v3-get: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v3-list: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v3-update: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v3-copy-image-to-volume: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v3-clone: [1,0,0] [1,0,1] [1,1,0] [1,1,1] [1] [67]
|
|
volumes-v3-availability-zones: [1,0,0] [1,0,1] [1,1,0] [1,1,0] [1] [59]
|
|
volumes-v3-extensions: [1,0,0] [0,0,1] [1,1,0] [1,0,0] [1] [42]
|
|
volumes-v3-metadata: [1,0,0] [1,0,1] [1,1,0] [1,1,0] [1] [59]
|
|
volumes-v3-reserve: [1,0,0] [1,0,1] [1,1,0] [1,1,0] [1] [59]
|
|
volumes-v3-readonly: [1,0,0] [1,0,1] [1,1,0] [1,1,0] [1] [59]
|
|
volumes-v3-upload: [1,0,0] [1,0,1] [1,1,0] [1,1,0] [1] [59]
|
|
|
|
Notes:
|
|
* volumes-v3 uses microversioning and is compatible with v2 including tests
|
|
therefore v3 will be included as advisory to ensure that the community is
|
|
aware of this transitory state between v2/v3
|
|
* volumes-list-api-versions is implemented but currently does not have any
|
|
tests inside tempest
|
|
* volumes-v2-qos is an admin-only capability
|
|
* test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_with_volume_in_use
|
|
was previously put in advisory status, but was dropped due to the
|
|
fact that the force flag necessary to create a snapshot on an in-use
|
|
volume wasn't added until Liberty.
|
|
|
|
Identity
|
|
--------
|
|
identity-v3-api-discovery: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
|
|
identity-v3-catalog: [1,0,1] [1,1,1] [1,1,0] [1,1,1] [1] [85]*
|
|
identity-v3-list-projects: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
|
|
identity-v3-list-groups: [1,1,1] [1,1,1] [1,1,0] [0,1,0] [1] [74]*
|
|
|
|
Notes:
|
|
* identity-v3-catalog is returned when the api for
|
|
identity-v3-tokens-create is called (GET /v3/auth/tokens). It is
|
|
important to consider it because end users may be relying on this
|
|
catalog for their apps (even though there are other API calls that
|
|
also show the catalog such as GET /v3/auth/catalog). There is one test
|
|
available for this capability but it is in the admin part of the test
|
|
suite, so not yet tested for non-admin users. Even though it scores enough
|
|
to be included as advisory, we cannot do this due to lack of non-admin
|
|
test case.
|
|
* identity-v3-list-projects and identity-v3-list-groups didn't have usable
|
|
tests in the past, but one was added for identity-v3-list-projects last year.
|
|
Providers like Fog.io
|
|
now actually use the /v3/users/{user_id}/[projects|groups] API's:
|
|
https://git.io/vX9S6
|
|
https://git.io/vX9SP
|
|
* identity-v3-change-password was considered here but it's applicability is
|
|
a bit hard to gauge: many systems using third-party authentication (such as
|
|
an LDAP/AD server, an external oauth system, etc) require password changes
|
|
to be done on the backend system. It probably needs further study to see
|
|
if it's really interoperable, but it seems unlikely at this point (I also
|
|
don't see it being supported by many external tools, etc).
|
|
|
|
Object Store
|
|
------------
|
|
|
|
objectstore-object-copy: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
objectstore-object-create: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
objectstore-object-delete: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
objectstore-object-get: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
objectstore-temp-url-put: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
objectstore-object-versioned: [1,1,1] [1,1,1] [1,1,1] [1,1,1] [1] [100]*
|
|
objectstore-temp-url-get: [1,1,1] [1,1,1] [1,1,1] [1,1,0] [1] [92]*
|
|
|
|
objectstore-account-quotas: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
objectstore-account-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
objectstore-container-acl: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
objectstore-container-quotas: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
objectstore-container-create: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
objectstore-container-delete: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
objectstore-container-list: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
|
|
objectstore-bulk-operations: [1,0,0] [1,1,1] [1,1,0] [0,1,0] [1] [58]
|
|
objectstore-info-request: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
objectstore-container-metadata: [1,1,1] [1,1,1] [1,1,0] [1,1,1] [1] [91]*
|
|
objectstore-staticweb: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58]
|
|
objectstore-crossdomain: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58]
|
|
objectstore-healthcheck: [1,0,0] [1,1,1] [0,1,0] [0,1,1] [1] [58]
|
|
|
|
Notes:
|
|
objectstore-info-request is a new capability through re-orginization. The test
|
|
it uses is currently under "objectstore-account-list". Re-org as per PTL
|
|
request: https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L87
|
|
|
|
objectstore-object-upload removed and test moved to objectstore-object-create as per PTL
|
|
request: https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L209
|
|
|
|
objectstore-object-put renamed to objectstore-temp-url-put as per PTL request:
|
|
https://gist.github.com/notmyname/102e4aba7084598638f47cee47f62bb1#file-defcore_updates-txt-L251
|
|
|
|
objectstore-container-metadata used in Fog:
|
|
https://github.com/fog/fog-openstack/blob/master/docs/storage.md#additional-parameters
|
|
Also in jClouds: https://jclouds.apache.org/guides/openstack/#swift
|
|
|