Image Signature Certificate Validation ====================================== Nova can determine if the certificate used to generate and verify the signature of a signed image (see `Glance Image Signature Verification documentation`_) is trusted by the user. This feature is called certificate validation and can be applied to the creation or rebuild of an instance. Certificate validation is meant to be performed jointly with image signature verification but each feature has its own Nova configuration option, to be specified in the ``[glance]`` section of the ``nova.conf`` configuration file. To enable certificate validation, set :oslo.config:option:`glance.enable_certificate_validation` to True. To enable signature validation, set :oslo.config:option:`glance.verify_glance_signatures` to True. Conversely, to disable either of these features, set their option to False or do not include the option in the Nova configurations at all. Certificate validation operates in concert with signature validation in `Cursive`_. It takes in a list of trusted certificate IDs and verifies that the certificate used to sign the image being booted is cryptographically linked to at least one of the provided trusted certificates. This provides the user with confidence in the identity and integrity of the image being booted. Certificate validation will only be performed if image signature validation is enabled. However, the presence of trusted certificate IDs overrides the ``enable_certificate_validation`` and ``verify_glance_signatures`` settings. In other words, if a list of trusted certificate IDs is provided to the instance create or rebuild commands, signature verification and certificate validation will be performed, regardless of their settings in the Nova configurations. See `Using Signature Verification`_ for details. .. _Cursive: http://opendev.org/x/cursive/ .. _Glance Image Signature Verification documentation: https://docs.openstack.org/glance/latest/user/signature.html .. note:: Certificate validation configuration options must be specified in the Nova configuration file that controls the ``nova-osapi_compute`` and ``nova-compute`` services, as opposed to other Nova services (conductor, scheduler, etc.). Requirements ------------ Key manager that is a backend to the `Castellan Interface`_. Possible key managers are: * `Barbican`_ * `Vault`_ .. _Castellan Interface: https://docs.openstack.org/castellan/latest/ .. _Barbican: https://docs.openstack.org/barbican/latest/contributor/devstack.html .. _Vault: https://www.vaultproject.io/ Limitations ----------- * As of the 18.0.0 Rocky release, only the libvirt compute driver supports trusted image certification validation. The feature is not, however, driver specific so other drivers should be able to support this feature over time. See the `feature support matrix`_ for information on which drivers support the feature at any given release. * As of the 18.0.0 Rocky release, image signature and trusted image certification validation is not supported with the Libvirt compute driver when using the ``rbd`` image backend (``[libvirt]/images_type=rbd``) and ``RAW`` formatted images. This is due to the images being cloned directly in the ``RBD`` backend avoiding calls to download and verify on the compute. * As of the 18.0.0 Rocky release, trusted image certification validation is not supported with volume-backed (:term:`boot from volume `) instances. The block storage service support may be available in a future release: https://blueprints.launchpad.net/cinder/+spec/certificate-validate * Trusted image certification support can be controlled via `policy configuration`_ if it needs to be disabled. See the ``os_compute_api:servers:create:trusted_certs`` and ``os_compute_api:servers:rebuild:trusted_certs`` policy rules. .. _feature support matrix: https://docs.openstack.org/nova/latest/user/support-matrix.html#operation_trusted_certs .. _policy configuration: https://docs.openstack.org/nova/latest/configuration/policy.html Configuration ------------- Nova will use the key manager defined by the Castellan key manager interface, which is the Barbican key manager by default. To use a different key manager, update the ``backend`` value in the ``[key_manager]`` group of the nova configuration file. For example:: [key_manager] backend = barbican .. note:: If these lines do not exist, then simply add them to the end of the file. Using Signature Verification ---------------------------- An image will need a few properties for signature verification to be enabled: ``img_signature`` Signature of your image. Signature restrictions are: * 255 character limit ``img_signature_hash_method`` Method used to hash your signature. Possible hash methods are: * SHA-224 * SHA-256 * SHA-384 * SHA-512 ``img_signature_key_type`` Key type used for your image. Possible key types are: * RSA-PSS * DSA * ECC-CURVES * SECT571K1 * SECT409K1 * SECT571R1 * SECT409R1 * SECP521R1 * SECP384R1 ``img_signature_certificate_uuid`` UUID of the certificate that you uploaded to the key manager. Possible certificate types are: * X_509 Using Certificate Validation ---------------------------- Certificate validation is triggered by one of two ways: 1. The Nova configuration options ``verify_glance_signatures`` and ``enable_certificate_validation`` are both set to True:: [glance] verify_glance_signatures = True enable_certificate_validation = True 2. A list of trusted certificate IDs is provided by one of three ways: .. note:: The command line support is pending changes https://review.opendev.org/#/c/500396/ and https://review.opendev.org/#/c/501926/ to python-novaclient and python-openstackclient, respectively. Environment Variable Use the environment variable ``OS_TRUSTED_IMAGE_CERTIFICATE_IDS`` to define a comma-delimited list of trusted certificate IDs. For example: .. code-block:: console $ export OS_TRUSTED_IMAGE_CERTIFICATE_IDS=79a6ad17-3298-4e55-8b3a-1672dd93c40f,b20f5600-3c9d-4af5-8f37-3110df3533a0 Command-Line Flag If booting or rebuilding an instance using the :command:`nova` commands, use the ``--trusted-image-certificate-id`` flag to define a single trusted certificate ID. The flag may be used multiple times to specify multiple trusted certificate IDs. For example: .. code-block:: console $ nova boot myInstanceName \ --flavor 1 \ --image myImageId \ --trusted-image-certificate-id 79a6ad17-3298-4e55-8b3a-1672dd93c40f \ --trusted-image-certificate-id b20f5600-3c9d-4af5-8f37-3110df3533a0 If booting or rebuilding an instance using the :command:`openstack server` commands, use the ``--trusted-image-certificate-id`` flag to define a single trusted certificate ID. The flag may be used multiple times to specify multiple trusted certificate IDs. For example: .. code-block:: console $ openstack --os-compute-api-version=2.63 server create myInstanceName \ --flavor 1 \ --image myImageId \ --nic net-id=fd25c0b2-b36b-45a8-82e4-ab52516289e5 \ --trusted-image-certificate-id 79a6ad17-3298-4e55-8b3a-1672dd93c40f \ --trusted-image-certificate-id b20f5600-3c9d-4af5-8f37-3110df3533a0 Nova Configuration Option Use the Nova configuration option :oslo.config:option:`glance.default_trusted_certificate_ids` to define a comma-delimited list of trusted certificate IDs. This configuration value is only used if ``verify_glance_signatures`` and ``enable_certificate_validation`` options are set to True, and the trusted certificate IDs are not specified anywhere else. For example:: [glance] default_trusted_certificate_ids=79a6ad17-3298-4e55-8b3a-1672dd93c40f,b20f5600-3c9d-4af5-8f37-3110df3533a0 Example Usage ------------- For these instructions, we will construct a 4-certificate chain to illustrate that it is possible to have a single trusted root certificate. We will upload all four certificates to Barbican. Then, we will sign an image and upload it to Glance, which will illustrate image signature verification. Finally, we will boot the signed image from Glance to show that certificate validation is enforced. Enable certificate validation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Enable image signature verification and certificate validation by setting both of their Nova configuration options to True:: [glance] verify_glance_signatures = True enable_certificate_validation = True Create a certificate chain ^^^^^^^^^^^^^^^^^^^^^^^^^^ As mentioned above, we will construct a 4-certificate chain to illustrate that it is possible to have a single trusted root certificate. Before we begin to build our certificate chain, we must first create files for OpenSSL to use for indexing and serial number tracking: .. code-block:: console $ touch index.txt $ echo '01' > serial.txt Create a certificate configuration file """"""""""""""""""""""""""""""""""""""" For these instructions, we will create a single configuration file called ``ca.conf``, which contains various sections that we can specify for use on the command-line during certificate requests and generation. Note that this certificate will be able to sign other certificates because it is a certificate authority. Also note the root CA's unique common name ("root"). The intermediate certificates' common names will be specified on the command-line when generating the corresponding certificate requests. ``ca.conf``:: [ req ] prompt = no distinguished_name = dn-param x509_extensions = ca_cert_extensions [ ca ] default_ca = ca_default [ dn-param ] C = US CN = Root CA [ ca_cert_extensions ] keyUsage = keyCertSign, digitalSignature basicConstraints = CA:TRUE, pathlen:2 [ ca_default ] new_certs_dir = . # Location for new certs after signing database = ./index.txt # Database index file serial = ./serial.txt # The current serial number default_days = 1000 default_md = sha256 policy = signing_policy email_in_dn = no [ intermediate_cert_extensions ] keyUsage = keyCertSign, digitalSignature basicConstraints = CA:TRUE, pathlen:1 [client_cert_extensions] keyUsage = keyCertSign, digitalSignature basicConstraints = CA:FALSE [ signing_policy ] countryName = optional stateOrProvinceName = optional localityName = optional organizationName = optional organizationalUnitName = optional commonName = supplied emailAddress = optional Generate the certificate authority (CA) and corresponding private key """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" For these instructions, we will save the certificate as ``cert_ca.pem`` and the private key as ``key_ca.pem``. This certificate will be a self-signed root certificate authority (CA) that can sign other CAs and non-CA certificates. .. code-block:: console $ openssl req \ -x509 \ -nodes \ -newkey rsa:1024 \ -config ca.conf \ -keyout key_ca.pem \ -out cert_ca.pem Generating a 1024 bit RSA private key ............................++++++ ...++++++ writing new private key to 'key_ca.pem' ----- Create the first intermediate certificate """"""""""""""""""""""""""""""""""""""""" Create a certificate request for the first intermediate certificate. For these instructions, we will save the certificate request as ``cert_intermeidate_a.csr`` and the private key as ``key_intermediate_a.pem``. .. code-block:: console $ openssl req \ -nodes \ -newkey rsa:2048 \ -subj '/CN=First Intermediate Certificate' \ -keyout key_intermediate_a.pem \ -out cert_intermediate_a.csr Generating a 2048 bit RSA private key .............................................................................................................+++ .....+++ writing new private key to 'key_intermediate_a.pem' ----- Generate the first intermediate certificate by signing its certificate request with the CA. For these instructions we will save the certificate as ``cert_intermediate_a.pem``. .. code-block:: console $ openssl ca \ -config ca.conf \ -extensions intermediate_cert_extensions \ -cert cert_ca.pem \ -keyfile key_ca.pem \ -out cert_intermediate_a.pem \ -infiles cert_intermediate_a.csr Using configuration from ca.conf Check that the request matches the signature Signature ok The Subject's Distinguished Name is as follows commonName :ASN.1 12:'First Intermediate Certificate' Certificate is to be certified until Nov 15 16:24:21 2020 GMT (1000 days) Sign the certificate? [y/n]:y 1 out of 1 certificate requests certified, commit? [y/n]y Write out database with 1 new entries Data Base Updated Create the second intermediate certificate """""""""""""""""""""""""""""""""""""""""" Create a certificate request for the second intermediate certificate. For these instructions, we will save the certificate request as ``cert_intermeidate_b.csr`` and the private key as ``key_intermediate_b.pem``. .. code-block:: console $ openssl req \ -nodes \ -newkey rsa:2048 \ -subj '/CN=Second Intermediate Certificate' \ -keyout key_intermediate_b.pem \ -out cert_intermediate_b.csr Generating a 2048 bit RSA private key ..........+++ ............................................+++ writing new private key to 'key_intermediate_b.pem' ----- Generate the second intermediate certificate by signing its certificate request with the first intermediate certificate. For these instructions we will save the certificate as ``cert_intermediate_b.pem``. .. code-block:: console $ openssl ca \ -config ca.conf \ -extensions intermediate_cert_extensions \ -cert cert_intermediate_a.pem \ -keyfile key_intermediate_a.pem \ -out cert_intermediate_b.pem \ -infiles cert_intermediate_b.csr Using configuration from ca.conf Check that the request matches the signature Signature ok The Subject's Distinguished Name is as follows commonName :ASN.1 12:'Second Intermediate Certificate' Certificate is to be certified until Nov 15 16:25:42 2020 GMT (1000 days) Sign the certificate? [y/n]:y 1 out of 1 certificate requests certified, commit? [y/n]y Write out database with 1 new entries Data Base Updated Create the client certificate """"""""""""""""""""""""""""" Create a certificate request for the client certificate. For these instructions, we will save the certificate request as ``cert_client.csr`` and the private key as ``key_client.pem``. .. code-block:: console $ openssl req \ -nodes \ -newkey rsa:2048 \ -subj '/CN=Client Certificate' \ -keyout key_client.pem \ -out cert_client.csr Generating a 2048 bit RSA private key .............................................................................................................................+++ ..............................................................................................+++ writing new private key to 'key_client.pem' ----- Generate the client certificate by signing its certificate request with the second intermediate certificate. For these instructions we will save the certificate as ``cert_client.pem``. .. code-block:: console $ openssl ca \ -config ca.conf \ -extensions client_cert_extensions \ -cert cert_intermediate_b.pem \ -keyfile key_intermediate_b.pem \ -out cert_client.pem \ -infiles cert_client.csr Using configuration from ca.conf Check that the request matches the signature Signature ok The Subject's Distinguished Name is as follows commonName :ASN.1 12:'Client Certificate' Certificate is to be certified until Nov 15 16:26:46 2020 GMT (1000 days) Sign the certificate? [y/n]:y 1 out of 1 certificate requests certified, commit? [y/n]y Write out database with 1 new entries Data Base Updated Upload the generated certificates to the key manager ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order interact with the key manager, the user needs to have a `creator` role. To list all users with a `creator` role, run the following command as an admin: .. code-block:: console $ openstack role assignment list --role creator --names +---------+-----------------------------+-------+-------------------+--------+-----------+ | Role | User | Group | Project | Domain | Inherited | +---------+-----------------------------+-------+-------------------+--------+-----------+ | creator | project_a_creator_2@Default | | project_a@Default | | False | | creator | project_b_creator@Default | | project_b@Default | | False | | creator | project_a_creator@Default | | project_a@Default | | False | +---------+-----------------------------+-------+-------------------+--------+-----------+ To give the `demo` user a `creator` role in the `demo` project, run the following command as an admin: .. code-block:: console $ openstack role add --user demo --project demo creator .. note:: This command provides no output. If the command fails, the user will see a "4xx Client error" indicating that "Secret creation attempt not allowed" and to "please review your user/project privileges". .. note:: The following "openstack secret" commands require that the `python-barbicanclient `_ package is installed. .. code-block:: console $ openstack secret store \ --name CA \ --algorithm RSA \ --expiration 2018-06-29 \ --secret-type certificate \ --payload-content-type "application/octet-stream" \ --payload-content-encoding base64 \ --payload "$(base64 cert_ca.pem)" $ openstack secret store \ --name IntermediateA \ --algorithm RSA \ --expiration 2018-06-29 \ --secret-type certificate \ --payload-content-type "application/octet-stream" \ --payload-content-encoding base64 \ --payload "$(base64 cert_intermediate_a.pem)" $ openstack secret store \ --name IntermediateB \ --algorithm RSA \ --expiration 2018-06-29 \ --secret-type certificate \ --payload-content-type "application/octet-stream" \ --payload-content-encoding base64 \ --payload "$(base64 cert_intermediate_b.pem)" $ openstack secret store \ --name Client \ --algorithm RSA \ --expiration 2018-06-29 \ --secret-type certificate \ --payload-content-type "application/octet-stream" \ --payload-content-encoding base64 \ --payload "$(base64 cert_client.pem)" The responses should look something like this: .. code-block:: console +---------------+------------------------------------------------------------------------------+ | Field | Value | +---------------+------------------------------------------------------------------------------+ | Secret href | http://127.0.0.1/key-manager/v1/secrets/8fbcce5d-d646-4295-ba8a-269fc9451eeb | | Name | CA | | Created | None | | Status | None | | Content types | {u'default': u'application/octet-stream'} | | Algorithm | RSA | | Bit length | 256 | | Secret type | certificate | | Mode | cbc | | Expiration | 2018-06-29T00:00:00+00:00 | +---------------+------------------------------------------------------------------------------+ Save off the certificate UUIDs (found in the secret href): .. code-block:: console $ cert_ca_uuid=8fbcce5d-d646-4295-ba8a-269fc9451eeb $ cert_intermediate_a_uuid=0b5d2c72-12cc-4ba6-a8d7-3ff5cc1d8cb8 $ cert_intermediate_b_uuid=674736e3-f25c-405c-8362-bbf991e0ce0a $ cert_client_uuid=125e6199-2de4-46e3-b091-8e2401ef0d63 Create a signed image ^^^^^^^^^^^^^^^^^^^^^ For these instructions, we will download a small CirrOS image: .. code-block:: console $ wget -nc -O cirros.tar.gz http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-source.tar.gz --2018-02-19 11:37:52-- http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-source.tar.gz Resolving download.cirros-cloud.net (download.cirros-cloud.net)... 64.90.42.85 Connecting to download.cirros-cloud.net (download.cirros-cloud.net)|64.90.42.85|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 434333 (424K) [application/x-tar] Saving to: ‘cirros.tar.gz’ cirros.tar.gz 100%[===================>] 424.15K --.-KB/s in 0.1s 2018-02-19 11:37:54 (3.79 MB/s) - ‘cirros.tar.gz’ saved [434333/434333] Sign the image with the generated client private key: .. code-block:: console $ openssl dgst \ -sha256 \ -sign key_client.pem \ -sigopt rsa_padding_mode:pss \ -out cirros.self_signed.signature \ cirros.tar.gz .. note:: This command provides no output. Save off the base64 encoded signature: .. code-block:: console $ base64_signature=$(base64 -w 0 cirros.self_signed.signature) Upload the signed image to Glance: .. code-block:: console $ openstack image create \ --public \ --container-format bare \ --disk-format qcow2 \ --property img_signature="$base64_signature" \ --property img_signature_certificate_uuid="$cert_client_uuid" \ --property img_signature_hash_method='SHA-256' \ --property img_signature_key_type='RSA-PSS' \ --file cirros.tar.gz \ cirros_client_signedImage +------------------+------------------------------------------------------------------------+ | Field | Value | +------------------+------------------------------------------------------------------------+ | checksum | d41d8cd98f00b204e9800998ecf8427e | | container_format | bare | | created_at | 2019-02-06T06:29:56Z | | disk_format | qcow2 | | file | /v2/images/17f48a6c-e592-446e-9c91-00fbc436d47e/file | | id | 17f48a6c-e592-446e-9c91-00fbc436d47e | | min_disk | 0 | | min_ram | 0 | | name | cirros_client_signedImage | | owner | 45e13e63606f40d6b23275c3cd91aec2 | | properties | img_signature='swA/hZi3WaNh35VMGlnfGnBWuXMlUbdO8h306uG7W3nwOyZP6dGRJ3 | | | Xoi/07Bo2dMUB9saFowqVhdlW5EywQAK6vgDsi9O5aItHM4u7zUPw+2e8eeaIoHlGhTks | | | kmW9isLy0mYA9nAfs3coChOIPXW4V8VgVXEfb6VYGHWm0nShiAP1e0do9WwitsE/TVKoS | | | QnWjhggIYij5hmUZ628KAygPnXklxVhqPpY/dFzL+tTzNRD0nWAtsc5wrl6/8HcNzZsaP | | | oexAysXJtcFzDrf6UQu66D3UvFBVucRYL8S3W56It3Xqu0+InLGaXJJpNagVQBb476zB2 | | | ZzZ5RJ/4Zyxw==', | | | img_signature_certificate_uuid='125e6199-2de4-46e3-b091-8e2401ef0d63', | | | img_signature_hash_method='SHA-256', | | | img_signature_key_type='RSA-PSS', | | | os_hash_algo='sha512', | | | os_hash_value='cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a92 | | | 1d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927d | | | a3e', | | | os_hidden='False' | | protected | False | | schema | /v2/schemas/image | | size | 0 | | status | active | | tags | | | updated_at | 2019-02-06T06:29:56Z | | virtual_size | None | | visibility | public | +------------------+------------------------------------------------------------------------+ .. note:: Creating the image can fail if validation does not succeed. This will cause the image to be deleted and the Glance log to report that "Signature verification failed" for the given image ID. Boot the signed image ^^^^^^^^^^^^^^^^^^^^^ Boot the signed image without specifying trusted certificate IDs: .. code-block:: console $ nova boot myInstance \ --flavor m1.tiny \ --image cirros_client_signedImage .. note:: The instance should fail to boot because certificate validation fails when the feature is enabled but no trusted image certificates are provided. The Nova log output should indicate that "Image signature certificate validation failed" because "Certificate chain building failed". Boot the signed image with trusted certificate IDs: .. code-block:: console $ nova boot myInstance \ --flavor m1.tiny \ --image cirros_client_signedImage \ --trusted-image-certificate-id $cert_ca_uuid,$cert_intermediate_a_uuid \ --trusted-image-certificate-id $cert_intermediate_b_uuid .. note:: The instance should successfully boot and certificate validation should succeed. The Nova log output should indicate that "Image signature certificate validation succeeded". Other Links ----------- * https://etherpad.openstack.org/p/mitaka-glance-image-signing-instructions * https://etherpad.openstack.org/p/queens-nova-certificate-validation * https://wiki.openstack.org/wiki/OpsGuide/User-Facing_Operations * http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/nova-validate-certificates.html