Initial commit of Marshal code base
Change-Id: Icc44c918bb3558ca7b2d0e7d17f1dc55f5b39972
This commit is contained in:
Dave McCowan
parent
b2aad62b28
commit
0f79632f94
126
README.md
Normal file
126
README.md
Normal file
@ -0,0 +1,126 @@
|
||||
## Marshal
|
||||
### Overview
|
||||
|
||||
* Marshal is an agent service running inside virtual machines, which will be responsible for securely fetching encryption keys from ia KMS like Barbican.
|
||||
* This agent will be interfacing with the disk encryption subsystem of the underlying operating system to encrypt/decrypt the disk I/O.
|
||||
* In the case of Linux-based virtual machines this agent will be interfacing with dm-crypt and for Windows OS it will be interfacing with Bit-locker.
|
||||
* The agent provides an abstraction service and can be integrated with other encryption subsystem as required.
|
||||
* When the agent reads a key from the KMS, the key is only stored briefly in a secure temporary file until it can be transferred to the disk encryption subsystem.
|
||||
|
||||
|
||||
**Table of Contents**
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Features](#features)
|
||||
- [Architecture](#architecture)
|
||||
- [Getting Started](#getting-started)
|
||||
- [Software Requirements](#software-requirements)
|
||||
- [Deployment Procedure](#deployment-procedure)
|
||||
- [Documentation](#documentation)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Core Components and Features](#core-components-and-features)
|
||||
- [Security](#security)
|
||||
- [Operations](#operations)
|
||||
- [Platform Support](#platform-support)
|
||||
- [Development](#development)
|
||||
- [License](#license)
|
||||
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* Disk encryption subsystem abstraction allowing for a consistent interface
|
||||
* KMS system abstraction allowing for a consistent interface
|
||||
* Encryption at various levels including full disk encryption, partition encryption including root partition
|
||||
|
||||
|
||||
### Architecture
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
|
||||
### Getting Started
|
||||
#### Deployment
|
||||
#####For production purposes, Marshal is intended to be deployed as a Debian Package embedded into OpenStack VMs
|
||||
###### Deploying Using Debian Package
|
||||
[Building and testing debian package](docs/debian-package-building.md)
|
||||
|
||||
##### For test purposes, Marshal can be cloned using normal Git semantics:
|
||||
|
||||
#### Clone to local repository:
|
||||
|
||||
#####Via SSH:
|
||||
```$ git clone git@github.com:CiscoCloud/marshal.git ```
|
||||
|
||||
#####Via HTTPS:
|
||||
```$ git clone https://github.com/CiscoCloud/marshal.git ```
|
||||
|
||||
|
||||
### Software Requirements
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
* Python 2.7.8
|
||||
* Cryptsetup (if Linux OS)
|
||||
|
||||
### Deployment Procedure
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### Please refer to the [Getting Started Guide](docs/Getting%20Started.md), which covers deployment, configuration, and example usage.
|
||||
|
||||
### Documentation
|
||||
|
||||
###### All documentation is located [here](docs)
|
||||
|
||||
|
||||
### Roadmap
|
||||
* KMS for infrastructure tenants
|
||||
* Volume encryption (With Marshal)
|
||||
* Certificate provisioning
|
||||
* Object Encryption
|
||||
* High key use tenants and IOT
|
||||
* KMaaS
|
||||
|
||||
|
||||
### Core Components and Features
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### List core components and features here
|
||||
- [x] Orchestration
|
||||
|
||||
|
||||
|
||||
### Security
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### List the security services it provides
|
||||
- [x] Encryption
|
||||
|
||||
|
||||
|
||||
### Operations
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### Disk encryption
|
||||
###### Automatic key retreival from a KMS
|
||||
|
||||
|
||||
|
||||
### Platform Support
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### Currently, only the Linux platform is supported using dm_crypt. Support Windows using bitlocker currently in the planning stages.
|
||||
###### Currently, only the OpenStack Barbican KMS is supported. Support for other KMSs is currently in the planning stages.
|
||||
###### Currently, only cloud-based KMSs are supported. Support for local KMSs is currently in the planning stages.
|
||||
|
||||
|
||||
### Development
|
||||
###### Write about the details of how anyone can contribute to the project.
|
||||
|
||||
|
||||
### Getting Support
|
||||
###### Write about the support details of the project.In case of any issue how anyone can get the support.
|
||||
|
||||
|
||||
### License
|
||||
###### Write about the license details of the project.
|
||||
1
debian/compat
vendored
Normal file
1
debian/compat
vendored
Normal file
@ -0,0 +1 @@
|
||||
9
|
||||
27
debian/control
vendored
Normal file
27
debian/control
vendored
Normal file
@ -0,0 +1,27 @@
|
||||
Source: marshal
|
||||
Section: misc
|
||||
Priority: optional
|
||||
Maintainer: Marshal Team <arvtiwar@cisco.com>
|
||||
Build-Depends:
|
||||
debhelper (>= 9~),
|
||||
python-all (>= 2.6),
|
||||
Build-Depends-Indep:
|
||||
python-pbr (>= 0.6),
|
||||
python-setuptools,
|
||||
XS-Python-Version: >= 2.6
|
||||
Standards-Version: 3.9.6
|
||||
Vcs-Browser: https://github.com/CiscoCloud/marshal
|
||||
Vcs-Git: git://github.com/CiscoCloud/marshal.git
|
||||
|
||||
Package: marshal
|
||||
Architecture: all
|
||||
Depends:
|
||||
python-six,
|
||||
python-blist,
|
||||
python-paste,
|
||||
python-openssl,
|
||||
python-requests,
|
||||
${misc:Depends},
|
||||
${python:Depends},
|
||||
Description: CiscoCloud marshal
|
||||
CiscoCloud marshal
|
||||
34
debian/copyright
vendored
Normal file
34
debian/copyright
vendored
Normal file
@ -0,0 +1,34 @@
|
||||
Format: http://dep.debian.net/deps/dep5
|
||||
Upstream-Name: marshal
|
||||
Source: https://github.com/CiscoCloud/marshal
|
||||
|
||||
Files: *
|
||||
Copyright: 2015 Cisco Systems
|
||||
License: Apache-2
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
.
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
.
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
.
|
||||
On Debian-based systems the full text of the Apache version 2.0 license
|
||||
can be found in `/usr/share/common-licenses/Apache-2.0'
|
||||
|
||||
Files: tools/rfc.sh
|
||||
Copyright: Copyright (c) 2010-2011 Gluster, Inc
|
||||
License: GPL-v3
|
||||
GNU GENERAL PUBLIC LICENSE
|
||||
Version 3, 29 June 2007
|
||||
.
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
.
|
||||
On Debian-based systems the full text of the Apache version 2.0 license
|
||||
can be found in `/usr/share/common-licenses/GPL-3'
|
||||
1
debian/marshal.dirs
vendored
Normal file
1
debian/marshal.dirs
vendored
Normal file
@ -0,0 +1 @@
|
||||
/var/log/marshal
|
||||
1
debian/marshal.install
vendored
Normal file
1
debian/marshal.install
vendored
Normal file
@ -0,0 +1 @@
|
||||
etc/marshal/marshal.conf etc/marshal
|
||||
5
debian/pydist-overrides
vendored
Normal file
5
debian/pydist-overrides
vendored
Normal file
@ -0,0 +1,5 @@
|
||||
oslo.config python-oslo-config
|
||||
oslo.serialization python-oslo-serialization
|
||||
oslo.i18n python-oslo-i18n
|
||||
oslo.utils python-oslo-utils
|
||||
oslo.log python-oslo-log
|
||||
7
debian/rules
vendored
Executable file
7
debian/rules
vendored
Executable file
@ -0,0 +1,7 @@
|
||||
#!/usr/bin/make -f
|
||||
|
||||
# Uncomment this to turn on verbose mode.
|
||||
#export DH_VERBOSE=1
|
||||
|
||||
%:
|
||||
dh $@ --with python2
|
||||
1
debian/source/format
vendored
Normal file
1
debian/source/format
vendored
Normal file
@ -0,0 +1 @@
|
||||
3.0 (native)
|
||||
336
docs/Getting Started.md
Normal file
336
docs/Getting Started.md
Normal file
@ -0,0 +1,336 @@
|
||||
###Marshal Getting Started Guide
|
||||
|
||||
####Deploying Marshal
|
||||
Marshal is currently deployable for Debian-based Linux systems and a Debian package is available and has been tested in Ubuntu 14.04. The intention is to have Marshal pre-installed on certain images. However, in cases where Marshal is not pre-installed, it can be installed as a Debian package. The procedure to install the Debian package is given here:
|
||||
|
||||
######ToDo: get clean Debian installation procedure
|
||||
|
||||
To check if Marshal is installed, run the following command:
|
||||
|
||||
```
|
||||
sudo marshal -h
|
||||
```
|
||||
|
||||
There are several ways to configure Marshal. The primary way to configure Marshal defaults is in the /etc/marshal/marshal.conf file:
|
||||
|
||||
```
|
||||
[DEFAULT]
|
||||
# Show more verbose log output (sets INFO log level output)
|
||||
# verbose = True
|
||||
|
||||
# Show debugging output in logs (sets DEBUG log level output)
|
||||
# debug = True
|
||||
|
||||
# log file location
|
||||
log_file = /var/log/marshal/marshal.log
|
||||
|
||||
[KM-OPT]
|
||||
# This section intended for dev/test purposes only
|
||||
# Default Auth Endpoint - for dev/test purposes only
|
||||
keystone_endpoint=http://173.39.224.159:35357/v3/auth/tokens
|
||||
# Default KMS fields - for dev/test purposes only
|
||||
kms_base=[HOST]
|
||||
kms_get_key_api=[some API]
|
||||
kms_key_id=[some key id]
|
||||
kms_project_id=[some project id]
|
||||
|
||||
[crypt]
|
||||
#This section for
|
||||
lf=/tmp/license.json
|
||||
ci=aes-cbc-essiv:sha256
|
||||
ks=256
|
||||
|
||||
```
|
||||
|
||||
More sensitive configuration details are configured in a JSON-formatted license file. The license file can come in 3 different flavors depending on the type of credentials to be used by the Marshal agent to authenticate. The 3 types are as follows:
|
||||
|
||||
* user-based
|
||||
* certififcate
|
||||
* trust
|
||||
|
||||
Example license configuration files for each of these 3 types are given here:
|
||||
|
||||
user-based license:
|
||||
```
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"http://[KEYSTONE_HOST]/v3/auth/tokens"
|
||||
},
|
||||
"project": {
|
||||
"id":"f383613fbcd74d6f8f9d4a40721ef811",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"user",
|
||||
"user": {
|
||||
"id":"4c49397e2d9f41e392498b8079c65343",
|
||||
"password":"changeit"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"e2ccc708-7c8d-437d-aaac-12bad476dd25"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
certificate-based license:
|
||||
```
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"http://[KEYSTONE_HOST]/v3/auth/tokens"
|
||||
},
|
||||
"project": {
|
||||
"id":"12345",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"cert",
|
||||
"cert": {
|
||||
"subject":"some_subject"
|
||||
"signature":"some_signature"
|
||||
"pub_key":"some_key"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"1b13ffdc-1b79-40ce-b94e-f4a2f9253d91"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
trust-based license:
|
||||
```
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"http://[KEYSTONE_HOST]/v3/auth/tokens"
|
||||
},
|
||||
"project": {
|
||||
"id":"12345",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"trust",
|
||||
"trust": {
|
||||
"id":"4c49397e2d9f41e392498b8079c65343"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"1b13ffdc-1b79-40ce-b94e-f4a2f9253d91"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The license path+file can be specified in the Marshal configuration file, or passed as a parameter with the '--crypt-lf' switch. Eg:
|
||||
|
||||
```
|
||||
sudo marshal --crypt-action open --crypt-dev /dev/vdc2 --crypt-mn priv_part --crypt-lf /tmp/license.json
|
||||
```
|
||||
It is recommended that the license file be placed in the /tmp folder or otherwise be disposed of once the desired encryption state is achieved.
|
||||
|
||||
####Understanding Marshal
|
||||
#####Authentication and Key retrieval
|
||||
######General Behavior
|
||||
Using the Keystone endpoint given by the configuration license:identity:endpoint, Marshal will attempt to authenticate using the configuration license:credentials:type and associated credential details.
|
||||
|
||||
######Barbican-specifc behavior
|
||||
Upon successfully authenticating, Marshal will receive an OpenStack token which should provide a Barbican Key Management Store (KMS) endpoint. Marshal will then attempt to retrieve the binary key associated with the key id given in the configuration
|
||||
|
||||
######Other-KMS bevahior
|
||||
Other KMS systems (non-Barbican) are not currently supported, but are expected to be in the near future.
|
||||
|
||||
#####Encryption operations
|
||||
Once Marshal has successfully retrieved the key, an encryption operation can commence against that key. Currently, only volume and volume partition encryption operations are supported, but other operations are anitipated in the future possibly including encrypted communications. The Marshal 'set' action will open an encrypted volume (if it can find the key), and it will format the target using LUKS formatting, if needed. The Marshal 'unset' command will close the target.
|
||||
|
||||
####Example Marshal operations:
|
||||
#####Formatting a block device:
|
||||
```
|
||||
sudo marshal --crypt-action format --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
|
||||
Example:
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
vdc 253:32 0 2G 0 disk
|
||||
|
||||
sudo marshal --crypt-action format --crypt-dev /dev/vdb --crypt-mn backup
|
||||
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action format --crypt-dev /dev/vdb --crypt-mn backup
|
||||
Device /dev/vdb is not a valid LUKS device.
|
||||
Command failed with code 22: Device /dev/vdb is not a valid LUKS device.
|
||||
Could not establish /dev/vdb as a valid LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully formatted.
|
||||
|
||||
```
|
||||
|
||||
#####Opening a block device:
|
||||
```
|
||||
sudo marshal --crypt-action open --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action open --crypt-dev /dev/vdb --crypt-mn backup
|
||||
/dev/vdb is a LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully opened.
|
||||
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
└─backup (dm-0) 252:0 0 2G 0 crypt
|
||||
vdc 253:32 0 2G 0 disk
|
||||
|
||||
```
|
||||
|
||||
#####Unsetting/Closing a block device:
|
||||
Unset=close. Either command can be used to achieve the same result.
|
||||
```
|
||||
sudo marshal --crypt-action unset --crypt-dev <device> --crypt-mn <managed name>
|
||||
sudo marshal --crypt-action close --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action close --crypt-dev /dev/vdb --crypt-mn backup
|
||||
/dev/vdb is a LUKS device.
|
||||
The volume was successfully closed.
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
vdc 253:32 0 2G 0 disk
|
||||
|
||||
```
|
||||
|
||||
#####Setting a block device:
|
||||
To "set" the device means to open it as a LUKS volume if possible, and if not then LUKS format and then open the device.
|
||||
```
|
||||
sudo marshal --crypt-action set --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
Example requiring format
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action set --crypt-dev /dev/vdc --crypt-mn backup2
|
||||
Device /dev/vdc is not a valid LUKS device.
|
||||
Command failed with code 22: Device /dev/vdc is not a valid LUKS device.
|
||||
Could not establish /dev/vdc as a valid LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully formatted.
|
||||
The volume was successfully opened.
|
||||
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
└─backup (dm-0) 252:0 0 2G 0 crypt
|
||||
vdc 253:32 0 2G 0 disk
|
||||
└─backup2 (dm-1) 252:1 0 2G 0 crypt
|
||||
```
|
||||
|
||||
Example not requiring format
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action set --crypt-dev /dev/vdb --crypt-mn backup
|
||||
/dev/vdb is a LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully opened.
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
└─backup (dm-0) 252:0 0 2G 0 crypt
|
||||
vdc 253:32 0 2G 0 disk
|
||||
└─backup2 (dm-1) 252:1 0 2G 0 crypt
|
||||
```
|
||||
|
||||
#####Statusing a block device:
|
||||
```
|
||||
sudo marshal --crypt-action status --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action status --crypt-dev /dev/vdc --crypt-mn backup2
|
||||
/dev/vdc is a LUKS device.
|
||||
```
|
||||
|
||||
#####Statusing a block device with verbosity:
|
||||
Adding the -v flag enables INFO messages to appear at the CLI
|
||||
```
|
||||
sudo marshal --crypt-action status --crypt-dev <device> --crypt-mn <managed name> -v
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action status --crypt-dev /dev/vdc --crypt-mn backup2 -v
|
||||
2015-10-06 22:26:16.186 1669 INFO marshal_agent.common.config [-] status action requested.
|
||||
/dev/vdc is a LUKS device.
|
||||
2015-10-06 22:26:16.192 1669 INFO marshal_agent.common.config [-] Status output was: /dev/mapper/backup2 is active.
|
||||
type: LUKS1
|
||||
cipher: aes-cbc-essiv:sha256
|
||||
keysize: 256 bits
|
||||
device: /dev/vdc
|
||||
offset: 4096 sectors
|
||||
size: 4190208 sectors
|
||||
mode: read/write
|
||||
Command successful.
|
||||
```
|
||||
|
||||
#####Overriding the license file default as set in the configiration file:
|
||||
```
|
||||
sudo marshal --crypt-action <action> --crypt-dev <device> --crypt-mn <managed name> --crypt-lf <license file>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ mv /tmp/license.json .
|
||||
License file not found.
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action status --crypt-dev /dev/vdc --crypt-mn backup2
|
||||
License file not found.
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action status --crypt-dev /dev/vdc --crypt-mn backup2 --crypt-lf license.json
|
||||
/dev/vdc is a LUKS device.
|
||||
```
|
||||
#####Overriding the cipher default as set in the configiration file:
|
||||
```
|
||||
sudo marshal --crypt-action <action> --crypt-dev <device> --crypt-mn <managed name> --crypt-ci <cipher>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action format --crypt-dev /dev/vdc --crypt-mn backup2 --crypt-ci aes-xts-plain64
|
||||
/dev/vdc is a LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully formatted.
|
||||
```
|
||||
|
||||
#####Overriding the key size default as set ub the configuration file:
|
||||
```
|
||||
sudo marshal --crypt-action <action> --crypt-dev <device> --crypt-mn <managed name> --crypt-ks <key size>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action format --crypt-dev /dev/vdc --crypt-mn backup2 --crypt-ci aes-xts-plain64 --crypt-ks 512
|
||||
/dev/vdc is a LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully formatted.
|
||||
|
||||
```
|
||||
35
docs/debian-package-building.md
Normal file
35
docs/debian-package-building.md
Normal file
@ -0,0 +1,35 @@
|
||||
**1. Adding official Openstack Kilo PPA repository.**
|
||||
It's needed because some marshal dependencies available only from it.
|
||||
```
|
||||
sudo apt-get install ubuntu-cloud-keyring
|
||||
echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu" \
|
||||
"trusty-updates/kilo main" \
|
||||
| sudo tee /etc/apt/sources.list.d/cloudarchive-kilo.list
|
||||
```
|
||||
**2. Update index of system packages.**
|
||||
```
|
||||
sudo apt-get update
|
||||
```
|
||||
**3. Install build tools and Marshal dependencies**
|
||||
```
|
||||
sudo apt-get install build-essential debhelper fakeroot git python-setuptools python-pbr python-all
|
||||
```
|
||||
**4. Clone fresh Marshal repo**
|
||||
```
|
||||
git clone https://github.com/CiscoCloud/marshal
|
||||
cd marshal
|
||||
```
|
||||
**5. Build deb package**
|
||||
```
|
||||
dpkg-buildpackage -us -uc
|
||||
```
|
||||
**6. Install package to the target system**
|
||||
```
|
||||
sudo dpkg -i ../marshal_*_all.deb # Errors on this step are normal, next step fixes them.
|
||||
sudo apt-get -f install # (Optional) This installs broken marshal dependencies.
|
||||
```
|
||||
**7. And try to use it**
|
||||
```
|
||||
sudo marshal --help
|
||||
sudo marshal.sh -h
|
||||
```
|
||||
93
docs/disk_commands
Normal file
93
docs/disk_commands
Normal file
@ -0,0 +1,93 @@
|
||||
http://jootamam.net/howto-basic-cryptsetup.htm
|
||||
|
||||
http://www.finnie.org/2009/07/26/keyfile-based-luks-encryption-in-debian/
|
||||
|
||||
http://sleepyhead.de/howto/?href=cryptpart
|
||||
|
||||
|
||||
sudo cryptsetup isLuks /dev/vdb
|
||||
sudo cryptsetup -y luksFormat /dev/vdb --key-file /tmp/mykey.txt
|
||||
sudo cryptsetup --key-file /tmp/mykey.txt luksOpen /dev/vdb backup
|
||||
|
||||
|
||||
ubuntu@dm-crypt:~$ lsblk -o KNAME,TYPE,SIZE,MODEL
|
||||
KNAME TYPE SIZE MODEL
|
||||
vda disk 30G
|
||||
vda1 part 30G
|
||||
vdb disk 4G
|
||||
ubuntu@dm-crypt:~$ lsblk -d -n -oNAME,RO | grep '0$' | awk {'print $1'}
|
||||
vda
|
||||
vdb
|
||||
|
||||
|
||||
|
||||
http://www.bogotobogo.com/DevOps/AWS/aws_attaching_Amazon_EBS_volume_to_instance.php
|
||||
ubuntu@dm-crypt:~$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 30G 0 disk
|
||||
└─vda1 253:1 0 30G 0 part /
|
||||
vdb 253:16 0 4G 0 disk
|
||||
ubuntu@dm-crypt:~$ sudo file -s /dev/vdb
|
||||
sudo: unable to resolve host dm-crypt
|
||||
/dev/vdb: data
|
||||
ubuntu@dm-crypt:~$ sudo file -s /dev/vda1
|
||||
sudo: unable to resolve host dm-crypt
|
||||
/dev/vda1: Linux rev 1.0 ext4 filesystem data, UUID=50879377-446a-412a-99bf-e0d42a78c1b3, volume name "cloudimg-rootfs" (needs journal recovery) (extents) (large files) (huge files)
|
||||
ubuntu@dm-crypt:~$ sudo mkfs -t ext4 /dev/vdb
|
||||
sudo: unable to resolve host dm-crypt
|
||||
mke2fs 1.42.9 (4-Feb-2014)
|
||||
Filesystem label=
|
||||
OS type: Linux
|
||||
Block size=4096 (log=2)
|
||||
Fragment size=4096 (log=2)
|
||||
Stride=0 blocks, Stripe width=0 blocks
|
||||
262144 inodes, 1048576 blocks
|
||||
52428 blocks (5.00%) reserved for the super user
|
||||
First data block=0
|
||||
Maximum filesystem blocks=1073741824
|
||||
32 block groups
|
||||
32768 blocks per group, 32768 fragments per group
|
||||
8192 inodes per group
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736
|
||||
|
||||
Allocating group tables: done
|
||||
Writing inode tables: done
|
||||
Creating journal (32768 blocks): done
|
||||
Writing superblocks and filesystem accounting information: done
|
||||
|
||||
ubuntu@dm-crypt:~$ sudo file -s /dev/vdb
|
||||
sudo: unable to resolve host dm-crypt
|
||||
/dev/vdb: Linux rev 1.0 ext4 filesystem data, UUID=a6e166a8-7b98-4f61-8318-6ba4b878bfee (extents) (large files) (huge files)
|
||||
ubuntu@dm-crypt:~$ sudo mkdir /mydisk
|
||||
sudo: unable to resolve host dm-crypt
|
||||
ubuntu@dm-crypt:~$ sudo mount /dev/vd /mydisk
|
||||
vda vda1 vdb
|
||||
ubuntu@dm-crypt:~$ sudo mount /dev/vdb /mydisk
|
||||
sudo: unable to resolve host dm-crypt
|
||||
ubuntu@dm-crypt:~$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 30G 0 disk
|
||||
└─vda1 253:1 0 30G 0 part /
|
||||
vdb 253:16 0 4G 0 disk /mydisk7379716029
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
http://unix.stackexchange.com/questions/52078/how-to-mount-a-cryptsetup-container-just-with-mount
|
||||
|
||||
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [[ $(mount | grep ${2%%/} | wc -l) -gt 0 ]]; then
|
||||
echo "Path $2 is already mounted!" >&2
|
||||
exit 9
|
||||
else
|
||||
MAPPER=$(mktemp -up /dev/mapper)
|
||||
cryptsetup luksOpen $1 $(basename $MAPPER)
|
||||
shift
|
||||
mount $MAPPER $* || cryptsetup luksClose $(basename $MAPPER)
|
||||
fi
|
||||
|
||||
|
||||
BIN
docs/images/marshal_within_openstack.png
Executable file
BIN
docs/images/marshal_within_openstack.png
Executable file
Binary file not shown.
|
After Width: | Height: | Size: 92 KiB |
137
docs/marshal-demo.md
Normal file
137
docs/marshal-demo.md
Normal file
@ -0,0 +1,137 @@
|
||||
1.Create symmetric keys using Barbican Order API.
|
||||
|
||||
2.Inject license file into VM using a cloud-config script:
|
||||
|
||||
```
|
||||
#cloud-config
|
||||
write_files:
|
||||
- content: |
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"[KMS endpoint url]"
|
||||
},
|
||||
"project": {
|
||||
"id":"f383613fbcd74d6f8f9d4a40721ef811",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"user",
|
||||
"user": {
|
||||
"id":"[user_id]",
|
||||
"password":"[user_password]"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"[key id]"
|
||||
}
|
||||
}
|
||||
}
|
||||
owner: root:root
|
||||
path: /tmp/license.json
|
||||
permissions: '0644'
|
||||
|
||||
```
|
||||
3.Create volume and attach to vm using Horizon
|
||||
|
||||
4.Look for the attached disk
|
||||
|
||||
```
|
||||
lsblk
|
||||
|
||||
lsblk -d -n -oNAME,RO | grep '0$' | awk {'print $1'}
|
||||
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
```
|
||||
|
||||
5.Set v-disk with marshal and dm-crypt
|
||||
|
||||
```
|
||||
sudo marshal.sh set -d /dev/vdb -m vdb1 -l /tmp/license.json
|
||||
```
|
||||
|
||||
7.Look for the attached disk
|
||||
|
||||
```
|
||||
lsblk
|
||||
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
└─vdb1 (dm-0) 252:0 0 2G 0 crypt
|
||||
```
|
||||
|
||||
8.Check status
|
||||
|
||||
```
|
||||
sudo marshal.sh status -d /dev/vdb -m vdb1 -v
|
||||
|
||||
/dev/mapper/vdb1 is active.
|
||||
type: LUKS1
|
||||
cipher: aes-cbc-essiv:sha256
|
||||
keysize: 256 bits
|
||||
device: /dev/vdb
|
||||
offset: 4096 sectors
|
||||
size: 4190208 sectors
|
||||
mode: read/write
|
||||
Command successful.
|
||||
```
|
||||
|
||||
9.create file system before mount
|
||||
|
||||
```
|
||||
sudo mkfs.ext4 /dev/mapper/vdb1
|
||||
```
|
||||
|
||||
10.Create mount point and mount the device point
|
||||
|
||||
```
|
||||
sudo mkdir /cryptdisk1
|
||||
sudo mount /dev/mapper/vdb1 /cryptdisk1
|
||||
```
|
||||
|
||||
11.Verify file system on device
|
||||
|
||||
```
|
||||
sudo df -PTH /dev/mapper/vdb1
|
||||
|
||||
or
|
||||
sudo df -PTH /dev/mapper/vdb1 | awk '{print $2}' | sed -n '1!p'
|
||||
```
|
||||
|
||||
12.Write something
|
||||
|
||||
```
|
||||
cd /cryptdisk1
|
||||
sudo vi crypttest.txt
|
||||
```
|
||||
|
||||
13.Search content
|
||||
|
||||
```
|
||||
sudo grep "super" /cryptdisk1/*
|
||||
```
|
||||
|
||||
14.Unmount disk
|
||||
|
||||
```
|
||||
sudo umount /cryptdisk1
|
||||
```
|
||||
|
||||
15.Unset disk
|
||||
|
||||
```
|
||||
sudo ./marshal.sh unset -d /dev/vdb -m vdb1
|
||||
```
|
||||
16.Search content on device
|
||||
|
||||
```
|
||||
sudo grep "super" /dev/vdb
|
||||
```
|
||||
|
||||
27
etc/marshal/marshal.conf
Normal file
27
etc/marshal/marshal.conf
Normal file
@ -0,0 +1,27 @@
|
||||
[DEFAULT]
|
||||
# Show more verbose log output (sets INFO log level output)
|
||||
# verbose = True
|
||||
|
||||
# Show debugging output in logs (sets DEBUG log level output)
|
||||
# debug = True
|
||||
|
||||
# log file location
|
||||
log_file = /var/log/marshal/marshal.log
|
||||
|
||||
[KM-OPT]
|
||||
# Key Manager Options
|
||||
# kms_type currently supported options are: barbican and vault. defaults to barbican
|
||||
# kms_base and kms_get_ket_api are only used for non-Keystone Barbican API testing or for non-Barbican kms_type
|
||||
#kms_type=barbican
|
||||
kms_type=vault
|
||||
#barbican
|
||||
#kms_base=http://173.39.229.98:9311/v1
|
||||
#kms_get_key_api=/secrets/
|
||||
#vault
|
||||
#kms_base=http://173.39.225.119:80/v1
|
||||
kms_get_key_api=/secret/project/name/apikey
|
||||
|
||||
[crypt]
|
||||
lf=/tmp/license.json
|
||||
ci=aes-cbc-essiv:sha256
|
||||
ks=256
|
||||
@ -1,19 +0,0 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import pbr.version
|
||||
|
||||
|
||||
__version__ = pbr.version.VersionInfo(
|
||||
'marshal').version_string()
|
||||
0
marshal_agent/agent/__init__.py
Normal file
0
marshal_agent/agent/__init__.py
Normal file
119
marshal_agent/agent/auth.py
Normal file
119
marshal_agent/agent/auth.py
Normal file
@ -0,0 +1,119 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Class to handle Marshal authentication against Keystone
|
||||
"""
|
||||
|
||||
import requests
|
||||
import json
|
||||
import abc
|
||||
import six
|
||||
|
||||
from marshal_agent.i18n import _
|
||||
|
||||
from marshal_agent.common import config
|
||||
from marshal_agent.common import exception
|
||||
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
@six.add_metaclass(abc.ABCMeta)
|
||||
class AuthBase(object):
|
||||
@abc.abstractmethod
|
||||
def get_token_wrapper(self, key_id, project_id):
|
||||
raise NotImplementedError
|
||||
|
||||
def get_key_wrapper(self, key_id, project_id):
|
||||
raise NotImplementedError
|
||||
|
||||
def get_key_binary(self):
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class Auth(AuthBase):
|
||||
|
||||
def __init__(self, conf=CONF, lic=None, kms_type=None):
|
||||
self.conf = conf
|
||||
self.lic = lic
|
||||
self.kms_type = kms_type
|
||||
|
||||
def get_token_wrapper(self):
|
||||
return self._get_token_from_keystone()
|
||||
|
||||
def get_token(self):
|
||||
if self.kms_type == 'vault':
|
||||
token = self.lic.x_vault_token
|
||||
return token, None
|
||||
else:
|
||||
return self._get_token_from_keystone()
|
||||
|
||||
def get_key_wrapper(self):
|
||||
return self._get_key_from_kms()
|
||||
|
||||
def _get_token_from_keystone(self):
|
||||
""" Get token from Keystone"""
|
||||
token = None
|
||||
kms_endpoint = None
|
||||
|
||||
payload = {
|
||||
"auth": {
|
||||
"identity": {
|
||||
"methods": [
|
||||
"password"
|
||||
],
|
||||
"password": {
|
||||
"user": {
|
||||
"id": self.lic.user_id,
|
||||
"password": self.lic.user_pass
|
||||
}
|
||||
}
|
||||
},
|
||||
"scope": {
|
||||
"project": {
|
||||
"id": self.lic.project_id,
|
||||
"domain": {
|
||||
"id": "default"
|
||||
},
|
||||
"name": self.lic.project_name
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
self.json_data = json.dumps(payload)
|
||||
hdrs = {
|
||||
'Accept': 'application/json',
|
||||
'Content-Type': 'application/json; charset=UTF-8'
|
||||
}
|
||||
pr = requests.post(self.lic.keystone_endpoint,
|
||||
data=json.dumps(payload), headers=hdrs)
|
||||
if pr.status_code != 201:
|
||||
log_msg = _('Unable to get identity from Keystone. Response Code\
|
||||
was: '+str(pr.status_code))
|
||||
client_msg = _('Marshal was unable to authenticate.')
|
||||
raise exception.MarshalHTTPException(log_msg, client_msg,
|
||||
pr.status_code)
|
||||
else:
|
||||
LOG.debug("Successfully authenticated against Keystone.")
|
||||
token = pr.headers['X-Subject-Token']
|
||||
pr_j = json.loads(pr.content)
|
||||
catalog = pr_j['token']['catalog']
|
||||
for endpoint in catalog:
|
||||
if endpoint.get('type') == 'kms':
|
||||
kms_endpoint = endpoint
|
||||
break
|
||||
|
||||
return token, kms_endpoint
|
||||
165
marshal_agent/agent/keyRunner.py
Normal file
165
marshal_agent/agent/keyRunner.py
Normal file
@ -0,0 +1,165 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Class to handle KMS key retrieval
|
||||
"""
|
||||
|
||||
import requests
|
||||
import json
|
||||
import abc
|
||||
import six
|
||||
|
||||
from marshal_agent.i18n import _
|
||||
from marshal_agent.common import config
|
||||
from marshal_agent.common import exception
|
||||
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
@six.add_metaclass(abc.ABCMeta)
|
||||
class KeyAgentBase(object):
|
||||
@abc.abstractmethod
|
||||
def get_key_wrapper(self, key_id, project_id):
|
||||
raise NotImplementedError
|
||||
|
||||
def get_key_binary(self):
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class KeyRunner(KeyAgentBase):
|
||||
|
||||
def __init__(self, lic=None, token=None, endpoint=None, conf=CONF,
|
||||
key_id=None, project_id=None):
|
||||
|
||||
'''
|
||||
if conf is not None:
|
||||
CONF = conf;
|
||||
else:
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
self.LOG = LOG
|
||||
# LOG = CONF.LOG
|
||||
# self.conf = conf
|
||||
# self.conf = conf
|
||||
#CONF = conf.CONF
|
||||
'''
|
||||
self.config = config
|
||||
self.conf = CONF
|
||||
KM_OPT_GRP_NAME = config.KM_OPT_GRP_NAME
|
||||
conf_opts = getattr(CONF, KM_OPT_GRP_NAME)
|
||||
self.kms_type = conf_opts.kms_type.lower()
|
||||
|
||||
if self.kms_type is None or self.kms_type == 'barbican':
|
||||
# For Barbican, we normally get endpoint from Keystone, but this
|
||||
# allows bypass of keystone for testing Barbican interface:
|
||||
if lic is None:
|
||||
if key_id is None or project_id is None:
|
||||
self.key_id = conf_opts.kms_key_id
|
||||
self.project_id = conf_opts.kms_project_id
|
||||
else:
|
||||
self.key_id = key_id
|
||||
self.project_id = project_id
|
||||
else:
|
||||
self.project_id = lic.project_id
|
||||
self.key_id = lic.key_id
|
||||
if endpoint is None:
|
||||
# Allows manually configured Barbican or other KMS:
|
||||
self.kms_endpoint = conf_opts.kms_base+conf_opts.\
|
||||
kms_get_key_api
|
||||
LOG.debug('Using Barbican endpoint from CONF')
|
||||
else:
|
||||
self.kms_endpoint = endpoint.get('endpoints')[0].\
|
||||
get('url')+'/secrets/'
|
||||
LOG.debug('Using Barbican endpoint from Keystone')
|
||||
elif self.kms_type == 'vault':
|
||||
# For vault we want to be able to provide kms endpoint details in
|
||||
# license file or conf file or, some mixture of the two.
|
||||
# Give license file the priority
|
||||
try:
|
||||
self.kms_base = lic.kms_base
|
||||
except AttributeError:
|
||||
if conf_opts.kms_base is not None:
|
||||
self.kms_base = conf_opts.kms_base
|
||||
try:
|
||||
self.kms_get_key_api = lic.kms_get_key_api
|
||||
except AttributeError:
|
||||
if conf_opts.kms_get_key_api is not None:
|
||||
self.kms_get_key_api = conf_opts.kms_get_key_api
|
||||
try:
|
||||
self.kms_endpoint = self.kms_base+self.kms_get_key_api
|
||||
except AttributeError:
|
||||
raise exception.MissingKMSConfigurationError
|
||||
|
||||
self.lic = lic
|
||||
self.token = token
|
||||
|
||||
def get_key_wrapper(self, key_id, project_id):
|
||||
return self._get_key_from_kms(key_id, project_id)
|
||||
|
||||
def get_key_binary(self):
|
||||
return self._get_key_from_kms(accept='application/octet-stream')
|
||||
|
||||
def _get_key_from_kms(self, accept=None):
|
||||
if self.kms_type is None or self.kms_type == 'barbican':
|
||||
if accept:
|
||||
headers = {
|
||||
'Accept': accept,
|
||||
'X-Project-Id': self.project_id
|
||||
}
|
||||
else:
|
||||
headers = {
|
||||
'Accept': 'application/json',
|
||||
'X-Project-Id': self.project_id
|
||||
}
|
||||
|
||||
if self.token is not None:
|
||||
headers['X-Auth-Token'] = self.token
|
||||
key_manager_url = self.kms_endpoint+format(self.key_id)
|
||||
elif self.kms_type == 'vault':
|
||||
if self.token is not None:
|
||||
headers = {
|
||||
'Accept': 'application/json'
|
||||
}
|
||||
headers['X-Vault-Token'] = self.token
|
||||
key_manager_url = self.kms_endpoint
|
||||
|
||||
LOG.debug('Calling KMS API at: %s', key_manager_url)
|
||||
|
||||
content = None
|
||||
|
||||
r = requests.get(key_manager_url, headers=headers)
|
||||
if r.status_code != 200:
|
||||
log_msg = _('Unable to get key from KMS. Response Code was: ' +
|
||||
str(r.status_code))
|
||||
client_msg = _('Unable to get key from KMS')
|
||||
raise exception.MarshalHTTPException(log_msg, client_msg,
|
||||
r.status_code)
|
||||
elif r.content is None or r.content == '' or r.content == 'None':
|
||||
LOG.info('KMS returned a blank key!')
|
||||
else:
|
||||
LOG.info('Successfully retrieved key from KMS.')
|
||||
content = r.content
|
||||
if self.kms_type is None or self.kms_type == 'barbican':
|
||||
key = content
|
||||
elif self.kms_type == 'vault':
|
||||
try:
|
||||
gr_j = json.loads(content)
|
||||
key = gr_j['data']['value']
|
||||
except (ValueError, KeyError, TypeError):
|
||||
msg = _('Unable to parse JSON response from Key Manager')
|
||||
raise exception.PayloadDecodingError(msg)
|
||||
|
||||
return key
|
||||
84
marshal_agent/agent/license.py
Normal file
84
marshal_agent/agent/license.py
Normal file
@ -0,0 +1,84 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Utility class to handle license data extraction and management
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
from marshal_agent.common import config
|
||||
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
class License(object):
|
||||
|
||||
def __init__(self, license_file, kms_type=None):
|
||||
self.keystone_endpoint = None
|
||||
self.project_id = None
|
||||
self.project_name = None
|
||||
self.key_id = None
|
||||
self.mode = None
|
||||
self.user_id = None
|
||||
self.user_pass = None
|
||||
self.kms_type = kms_type
|
||||
self.parse_license_file(license_file)
|
||||
|
||||
def parse_license_file(self, lf):
|
||||
try:
|
||||
with open(lf, 'r') as lf_h:
|
||||
l_data = (lf_h.read()).rstrip()
|
||||
try:
|
||||
jl_data = json.loads(l_data)
|
||||
except ValueError as e:
|
||||
LOG.debug("Unable to parse license file: %s", str(e))
|
||||
print 'Unable to parse license file: '+str(e)
|
||||
sys.exit(1)
|
||||
|
||||
if self.kms_type == 'vault':
|
||||
self.x_vault_token = \
|
||||
jl_data['license']['identity']['token']
|
||||
try:
|
||||
self.kms_base = \
|
||||
jl_data['license']['endpoint']['kms_base']
|
||||
except KeyError:
|
||||
pass
|
||||
try:
|
||||
self.kms_get_key_api = \
|
||||
jl_data['license']['endpoint']['kms_get_key_api']
|
||||
except KeyError:
|
||||
pass
|
||||
else:
|
||||
self.keystone_endpoint = \
|
||||
jl_data['license']['identity']['endpoint']
|
||||
self.project_id = jl_data['license']['project']['id']
|
||||
self.project_name = jl_data['license']['project']['name']
|
||||
self.key_id = jl_data['license']['key']['id']
|
||||
self.mode = jl_data['license']['credentials']['type']
|
||||
if self.mode == 'user':
|
||||
self.user_id = \
|
||||
jl_data['license']['credentials']['user']['id']
|
||||
self.user_pass = \
|
||||
jl_data['license']['credentials']['user']['password\
|
||||
']
|
||||
|
||||
except IOError:
|
||||
LOG.info("Unable to locate license file: %s", lf)
|
||||
print 'License file not found.'
|
||||
sys.exit(1)
|
||||
except KeyError as e:
|
||||
LOG.info("Unable to parse license file: %s", str(e))
|
||||
print 'Unable to parse license file.'
|
||||
sys.exit(1)
|
||||
210
marshal_agent/agent/marshal.py
Normal file
210
marshal_agent/agent/marshal.py
Normal file
@ -0,0 +1,210 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
The Marshal module is intended to facilitate flexible volume encryption with
|
||||
dynamic key management
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import errno
|
||||
import tempfile
|
||||
from stat import S_ISBLK
|
||||
from marshal_agent.agent.keyRunner import KeyRunner
|
||||
from marshal_agent.agent.license import License
|
||||
from marshal_agent.agent.auth import Auth
|
||||
from marshal_agent.agent.volCrypt import VolCrypt
|
||||
from marshal_agent.common import config
|
||||
from marshal_agent.common.exception import MarshalHTTPException
|
||||
from marshal_agent.common.exception import PayloadDecodingError
|
||||
from marshal_agent.common.exception import MissingKMSConfigurationError
|
||||
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
def check_block_device(device):
|
||||
try:
|
||||
mode = os.stat(device).st_mode
|
||||
except OSError as e:
|
||||
print 'The specified device is invalid.'
|
||||
if e[0] == errno.ENOENT:
|
||||
LOG.info('Device %s was specified, but the file could not be found.\
|
||||
', device)
|
||||
sys.exit(1)
|
||||
|
||||
if not S_ISBLK(mode):
|
||||
print 'The specified device is invalid.'
|
||||
LOG.info('Device was specified, but the file is not a valid block devic\
|
||||
e file.')
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def check_if_luks(vc, device, managed_name):
|
||||
LOG.debug('Performing isLuks check.')
|
||||
if device is None:
|
||||
print 'Please provide the device path.'
|
||||
LOG.info('isLuks check cancelled. No device specified.')
|
||||
sys.exit(1)
|
||||
|
||||
check_block_device(device)
|
||||
|
||||
response = vc.is_luks(device)
|
||||
if response:
|
||||
print '{} is a LUKS device.'.format(device)
|
||||
return True
|
||||
else:
|
||||
print 'Could not establish {} as a valid LUKS device.'.format(device)
|
||||
return False
|
||||
|
||||
|
||||
def missing_managed_name():
|
||||
LOG.info("Managed name is a required parameter for this operation.")
|
||||
print 'Please provide the managed name.'
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def agent_main():
|
||||
LOG.info('Starting Marshal...')
|
||||
KM_OPT_GRP_NAME = config.KM_OPT_GRP_NAME
|
||||
km_conf_opts = getattr(CONF, KM_OPT_GRP_NAME)
|
||||
kms_type = km_conf_opts.kms_type.lower()
|
||||
VOL_CRYPT_GRP_NAME = config.VOL_CRYPT_GRP_NAME
|
||||
vc_conf_opts = getattr(CONF, VOL_CRYPT_GRP_NAME)
|
||||
action = vc_conf_opts.action
|
||||
device = vc_conf_opts.dev
|
||||
managed_name = vc_conf_opts.mn
|
||||
license_file = vc_conf_opts.lf
|
||||
key_size = str(vc_conf_opts.ks)
|
||||
cipher = vc_conf_opts.ci
|
||||
|
||||
LOG.info('KMS type: %s', kms_type)
|
||||
lic = License(license_file, kms_type)
|
||||
vc = VolCrypt(device, managed_name)
|
||||
|
||||
action = action.lower()
|
||||
LOG.info('%s action requested.', action)
|
||||
if action == 'unset':
|
||||
action = 'close'
|
||||
if action == 'isluks':
|
||||
check_if_luks(vc, device, managed_name)
|
||||
sys.exit(1)
|
||||
|
||||
elif action == 'open' or action == 'format' or action == 'set':
|
||||
setNeedsFormat = False
|
||||
if not check_if_luks(vc, device, managed_name):
|
||||
if action == 'open':
|
||||
# For debugging, comment this exit and uncomment the pass
|
||||
sys.exit(1)
|
||||
# pass
|
||||
elif action == 'set':
|
||||
setNeedsFormat = True
|
||||
|
||||
auth = Auth(CONF, lic, kms_type)
|
||||
try:
|
||||
token, kms_endpoint = auth.get_token()
|
||||
except MarshalHTTPException:
|
||||
print 'Authentication failed.'
|
||||
LOG.info('Unable to authenticate against Keystone.')
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
key_runner = KeyRunner(lic, token, kms_endpoint)
|
||||
except MissingKMSConfigurationError as e:
|
||||
LOG.info(e.message)
|
||||
print e.message
|
||||
sys.exit(1)
|
||||
try:
|
||||
print 'Attempting to fetch key from KMS...'
|
||||
binary_key = key_runner.get_key_binary()
|
||||
print 'Key successfully retrieved from KMS...'
|
||||
except PayloadDecodingError:
|
||||
LOG.info('Unable to parse KMS response.')
|
||||
print 'Unable to parse KMS response.'
|
||||
sys.exit(1)
|
||||
except MarshalHTTPException as e:
|
||||
LOG.info('Error requesting key from from KMS: %s', e.status_code)
|
||||
print 'Unable to access KMS.'
|
||||
sys.exit(1)
|
||||
|
||||
tmpdir = tempfile.mkdtemp()
|
||||
key_file_name = 'key.bin'
|
||||
|
||||
# Ensure the file is read/write by the creator only
|
||||
saved_umask = os.umask(0077)
|
||||
|
||||
path = os.path.join(tmpdir, key_file_name)
|
||||
try:
|
||||
with open(path, "wb") as tmp:
|
||||
tmp.write(binary_key)
|
||||
if action == 'open':
|
||||
if managed_name is None:
|
||||
missing_managed_name()
|
||||
result = vc.open_volume(key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully opened.'
|
||||
elif action == 'format':
|
||||
result = vc.format_volume(cipher=cipher, key_size=key_size,
|
||||
key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully formatted.'
|
||||
elif action == 'set':
|
||||
if setNeedsFormat:
|
||||
result = vc.format_volume(cipher=cipher, key_size=key_size,
|
||||
key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully formatted.'
|
||||
if managed_name is None:
|
||||
missing_managed_name()
|
||||
result = vc.open_volume(key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully opened.'
|
||||
elif result == 2:
|
||||
LOG.info("Open failed on set. Attempting format...")
|
||||
result = vc.format_volume(cipher=cipher, key_size=key_size,
|
||||
key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully formatted.'
|
||||
result = vc.open_volume(key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully opened.'
|
||||
|
||||
except IOError as e:
|
||||
print 'IOError'
|
||||
else:
|
||||
pass
|
||||
finally:
|
||||
os.remove(path)
|
||||
os.umask(saved_umask)
|
||||
os.rmdir(tmpdir)
|
||||
|
||||
elif action == 'close':
|
||||
if not check_if_luks(vc, device, managed_name):
|
||||
sys.exit(1)
|
||||
if managed_name is None:
|
||||
missing_managed_name()
|
||||
result = vc.close_volume()
|
||||
if result == 0:
|
||||
print 'The volume was successfully closed.'
|
||||
|
||||
elif action == 'status':
|
||||
if not check_if_luks(vc, device, managed_name):
|
||||
sys.exit(1)
|
||||
if managed_name is None:
|
||||
missing_managed_name()
|
||||
vc.status_volume()
|
||||
|
||||
if __name__ == '__main__':
|
||||
agent_main()
|
||||
135
marshal_agent/agent/volCrypt.py
Normal file
135
marshal_agent/agent/volCrypt.py
Normal file
@ -0,0 +1,135 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
The volcrypt module interfaces with cryptsetup
|
||||
"""
|
||||
|
||||
from marshal_agent.common import config
|
||||
|
||||
import subprocess
|
||||
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
class VolCrypt():
|
||||
|
||||
def __init__(self, dev_path, mapped_name):
|
||||
self.dev_path = dev_path
|
||||
self.mapped_name = mapped_name
|
||||
|
||||
def is_luks(self, dev_path):
|
||||
"""Checks if the specified device uses LUKS for encryption.
|
||||
:param device: the device to check
|
||||
:returns: true if the specified device uses LUKS; false otherwise
|
||||
"""
|
||||
try:
|
||||
# check to see if the device uses LUKS: exit status is 0
|
||||
# if the device is a LUKS partition and non-zero if not
|
||||
cmd = ["cryptsetup", 'isLuks', '--verbose', dev_path]
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.debug(output)
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("isLuks exited with (status %(exit_code)s): "),
|
||||
{"exit_code": e.returncode})
|
||||
return False
|
||||
|
||||
def open_volume(self, **kwargs):
|
||||
"""Opens the LUKS partition on the volume using the provided key file
|
||||
"""
|
||||
LOG.debug("opening encrypted volume %s", self.dev_path)
|
||||
try:
|
||||
cmd = ["cryptsetup", "luksOpen"]
|
||||
|
||||
key_file = kwargs.get("key_file", None)
|
||||
if key_file is not None:
|
||||
cmd.extend(["--key-file", key_file])
|
||||
|
||||
cmd.extend([self.dev_path])
|
||||
cmd.extend([self.mapped_name])
|
||||
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.info("Successfully opened the volume. Opening output was: %s",
|
||||
output)
|
||||
return 0
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("luksOpen exited with (status %(exit_code)s)"),
|
||||
{"exit_code": e.returncode})
|
||||
return e.returncode
|
||||
|
||||
def close_volume(self, **kwargs):
|
||||
"""Closes the LUKS partition on the volume
|
||||
"""
|
||||
LOG.debug("closing encrypted volume %s", self.dev_path)
|
||||
try:
|
||||
cmd = ["cryptsetup", "luksClose"]
|
||||
|
||||
cmd.extend([self.mapped_name])
|
||||
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.info("Successfully closed the volume. Closing output was: %s",
|
||||
output)
|
||||
return 0
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("luksClose exited with (status %(exit_code)s): "),
|
||||
{"exit_code": e.returncode})
|
||||
return e.returncode
|
||||
|
||||
def format_volume(self, **kwargs):
|
||||
"""Creates a LUKS header on the volume.
|
||||
"""
|
||||
LOG.debug("formatting encrypted volume %s", self.dev_path)
|
||||
try:
|
||||
cmd = ["cryptsetup", "--batch-mode", "luksFormat"]
|
||||
|
||||
cipher = kwargs.get("cipher", None)
|
||||
if cipher is not None:
|
||||
cmd.extend(["--cipher", cipher])
|
||||
key_size = kwargs.get("key_size", None)
|
||||
if key_size is not None:
|
||||
cmd.extend(["--key-size", key_size])
|
||||
key_file = kwargs.get("key_file", None)
|
||||
if key_file is not None:
|
||||
cmd.extend(["--key-file", key_file])
|
||||
cmd.extend([self.dev_path])
|
||||
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.info("Successfully formatted the volume. Format output was: \
|
||||
%s", output)
|
||||
return 0
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("luksFormat exited with (status %(exit_code)s): "),
|
||||
{"exit_code": e.returncode})
|
||||
return e.returncode
|
||||
|
||||
def status_volume(self, **kwargs):
|
||||
"""Statuses the LUKS partition on the volume
|
||||
"""
|
||||
LOG.debug("Stating encrypted volume %s", self.dev_path)
|
||||
try:
|
||||
cmd = ["cryptsetup", "-v", "status"]
|
||||
|
||||
cmd.extend([self.mapped_name])
|
||||
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.info("Status output was: %s", output)
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("status exited with (status %(exit_code)s): "),
|
||||
{"exit_code": e.returncode})
|
||||
0
marshal_agent/common/__init__.py
Normal file
0
marshal_agent/common/__init__.py
Normal file
122
marshal_agent/common/config.py
Normal file
122
marshal_agent/common/config.py
Normal file
@ -0,0 +1,122 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Configuration setup for Marshal.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import errno
|
||||
from oslo_config import cfg
|
||||
from oslo_log import log
|
||||
|
||||
import marshal_agent.i18n as u
|
||||
# import marshal.version
|
||||
|
||||
CONFIG_FILE = "/etc/marshal/marshal.conf"
|
||||
# Check if we have root perms. May need tweaking here for SELinux, ACLs, etc..
|
||||
# Comment out for debugging without root perms
|
||||
if __builtins__.get('TESTING_MARSHAL', False) is False:
|
||||
print "NOT TESTING!"
|
||||
if os.getuid() != 0:
|
||||
print "Marshal needs to be run with root permissions."
|
||||
sys.exit(1)
|
||||
else:
|
||||
if os.path.exists("test_marshal.conf"):
|
||||
CONFIG_FILE = "test_marshal.conf"
|
||||
elif os.path.exists("marshal_agent/tests/test_marshal.conf"):
|
||||
CONFIG_FILE = "marshal_agent/tests/test_marshal.conf"
|
||||
|
||||
KMS_TYPE = 'Barbican'
|
||||
KMS_BASE = None
|
||||
KMS_API = None
|
||||
SECRET_ID = None
|
||||
TENANT_ID = None
|
||||
KEYSTONE_ENDPOINT = None
|
||||
|
||||
KM_OPT_GRP_NAME = 'KM-OPT'
|
||||
VOL_CRYPT_GRP_NAME = 'crypt'
|
||||
|
||||
opt_group = cfg.OptGroup(name=KM_OPT_GRP_NAME,
|
||||
title='Key Manager config settings')
|
||||
|
||||
openam = [
|
||||
cfg.StrOpt('kms_type', default=KMS_TYPE,
|
||||
help=('Key Management Store Type.')),
|
||||
cfg.StrOpt('kms_base', default=KMS_BASE,
|
||||
help=('Key management service base url')),
|
||||
cfg.StrOpt('kms_get_key_api', default=KMS_API,
|
||||
help=('Key management service key retrieval API')),
|
||||
cfg.StrOpt('kms_key_id', default=SECRET_ID,
|
||||
help=('Key management service key ID')),
|
||||
cfg.StrOpt('kms_project_id', default=TENANT_ID,
|
||||
help=('Key management service project/tenant ID')),
|
||||
cfg.StrOpt('keystone_endpoint', default=KEYSTONE_ENDPOINT,
|
||||
help=('Keystone endpoint for authentication'))
|
||||
]
|
||||
|
||||
vol_crypt_opt_group = cfg.OptGroup(name=VOL_CRYPT_GRP_NAME,
|
||||
title='Volume Encryption Options')
|
||||
|
||||
vol_crypt_opts = [
|
||||
cfg.StrOpt('action', default='isLuks',
|
||||
help=u._('One of: set, unset, isLuks, open, close, format,\
|
||||
status')),
|
||||
cfg.StrOpt('dev', default=None,
|
||||
help=u._('The target device.')),
|
||||
cfg.StrOpt('mn', default=None,
|
||||
help=u._('The managed name for the device.')),
|
||||
cfg.StrOpt('lf', default='license.json',
|
||||
help=u._('The key license file.')),
|
||||
# Direct keyfile input not supported at this time for security reasons.
|
||||
# cfg.StrOpt('kf', default=None,
|
||||
# help=u._('The key file.')),
|
||||
cfg.IntOpt('ks', default=256,
|
||||
help=u._('Limits the key size to the specified number of bytes.\
|
||||
')),
|
||||
cfg.StrOpt('ci', default='aes-cbc-essiv:sha256',
|
||||
help=u._('Cipher. The encryption algorithm.'))
|
||||
]
|
||||
|
||||
|
||||
def new_config():
|
||||
conf = cfg.ConfigOpts()
|
||||
log.register_options(conf)
|
||||
conf.register_cli_opts(vol_crypt_opts, group=vol_crypt_opt_group)
|
||||
return conf
|
||||
|
||||
|
||||
def parse_args(conf, args=None, usage=None, default_config_files=None):
|
||||
conf(args=args,
|
||||
project='marshal',
|
||||
prog='marshal',
|
||||
# version=marshal.version.__version__,
|
||||
version='0.1',
|
||||
usage=usage,
|
||||
default_config_files=[CONFIG_FILE])
|
||||
|
||||
CONF = new_config()
|
||||
CONF.register_group(opt_group)
|
||||
CONF.register_opts(openam, opt_group)
|
||||
formatter = log.logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - \
|
||||
%(message)s')
|
||||
log.set_defaults(formatter)
|
||||
parse_args(CONF)
|
||||
try:
|
||||
log.setup(CONF, 'marshal')
|
||||
except IOError as e:
|
||||
if (e[0] == errno.ENOENT):
|
||||
print "Could not access logfile! Continuing without logging..."
|
||||
LOG = log.getLogger(__name__)
|
||||
445
marshal_agent/common/exception.py
Normal file
445
marshal_agent/common/exception.py
Normal file
@ -0,0 +1,445 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
# Copyright (c) 2013-2014 Rackspace, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Marshal exception subclasses
|
||||
"""
|
||||
|
||||
import urlparse
|
||||
|
||||
import marshal_agent.i18n as u
|
||||
|
||||
_FATAL_EXCEPTION_FORMAT_ERRORS = False
|
||||
|
||||
|
||||
class RedirectException(Exception):
|
||||
def __init__(self, url):
|
||||
self.url = urlparse.urlparse(url)
|
||||
|
||||
|
||||
class MarshalException(Exception):
|
||||
"""Base Marshal Exception
|
||||
|
||||
To correctly use this class, inherit from it and define
|
||||
a 'message' property. That message will get printf'd
|
||||
with the keyword arguments provided to the constructor.
|
||||
"""
|
||||
message = u._("An unknown exception occurred")
|
||||
|
||||
def __init__(self, message_arg=None, *args, **kwargs):
|
||||
if not message_arg:
|
||||
message_arg = self.message
|
||||
try:
|
||||
self.message = message_arg % kwargs
|
||||
except Exception as e:
|
||||
if _FATAL_EXCEPTION_FORMAT_ERRORS:
|
||||
raise e
|
||||
else:
|
||||
# at least get the core message out if something happened
|
||||
pass
|
||||
super(MarshalException, self).__init__(self.message)
|
||||
|
||||
|
||||
class MarshalHTTPException(MarshalException):
|
||||
"""Base Marshal Exception to handle HTTP responses
|
||||
|
||||
To correctly use this class, inherit from it and define the following
|
||||
properties:
|
||||
|
||||
- message: The message that will be displayed in the server log.
|
||||
- client_message: The message that will actually be outputted to the
|
||||
client.
|
||||
- status_code: The HTTP status code that should be returned.
|
||||
The default status code is 500.
|
||||
"""
|
||||
client_message = u._("failure seen - please contact site administrator.")
|
||||
|
||||
def __init__(self, message_arg=None, client_message=None, status_code=500,
|
||||
*args, **kwargs):
|
||||
self.status_code = status_code
|
||||
if not client_message:
|
||||
client_message = self.client_message
|
||||
try:
|
||||
self.client_message = client_message % kwargs
|
||||
except Exception as e:
|
||||
if _FATAL_EXCEPTION_FORMAT_ERRORS:
|
||||
raise e
|
||||
else:
|
||||
# at least get the core message out if something happened
|
||||
pass
|
||||
super(MarshalHTTPException, self).__init__(
|
||||
message_arg, self.client_message, *args, **kwargs)
|
||||
|
||||
|
||||
class MissingKMSConfigurationError(MarshalException):
|
||||
message = u._("One or more KMS configuration elements could not be found.")
|
||||
|
||||
|
||||
class MissingArgumentError(MarshalException):
|
||||
message = u._("Missing required argument.")
|
||||
|
||||
|
||||
class MissingCredentialError(MarshalException):
|
||||
message = u._("Missing required credential: %(required)s")
|
||||
|
||||
|
||||
class MissingMetadataField(MarshalHTTPException):
|
||||
message = u._("Missing required metadata field for %(required)s")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class InvalidSubjectDN(MarshalHTTPException):
|
||||
message = u._("Invalid subject DN: %(subject_dn)s")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class InvalidContainer(MarshalHTTPException):
|
||||
message = u._("Invalid container: %(reason)s")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class InvalidExtensionsData(MarshalHTTPException):
|
||||
message = u._("Invalid extensions data.")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class InvalidCMCData(MarshalHTTPException):
|
||||
message = u._("Invalid CMC Data")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class InvalidPKCS10Data(MarshalHTTPException):
|
||||
message = u._("Invalid PKCS10 Data: %(reason)s")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class InvalidCertificateRequestType(MarshalHTTPException):
|
||||
message = u._("Invalid Certificate Request Type")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class CertificateExtensionsNotSupported(MarshalHTTPException):
|
||||
message = u._("Extensions are not yet supported. "
|
||||
"Specify a valid profile instead.")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class FullCMCNotSupported(MarshalHTTPException):
|
||||
message = u._("Full CMC Requests are not yet supported.")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class BadAuthStrategy(MarshalException):
|
||||
message = u._("Incorrect auth strategy, expected \"%(expected)s\" but "
|
||||
"received \"%(received)s\"")
|
||||
|
||||
|
||||
class NotFound(MarshalException):
|
||||
message = u._("An object with the specified identifier was not found.")
|
||||
|
||||
|
||||
class UnknownScheme(MarshalException):
|
||||
message = u._("Unknown scheme '%(scheme)s' found in URI")
|
||||
|
||||
|
||||
class BadStoreUri(MarshalException):
|
||||
message = u._("The Store URI was malformed.")
|
||||
|
||||
|
||||
class Duplicate(MarshalException):
|
||||
message = u._("An object with the same identifier already exists.")
|
||||
|
||||
|
||||
class StorageFull(MarshalException):
|
||||
message = u._("There is not enough disk space on the image storage media.")
|
||||
|
||||
|
||||
class StorageWriteDenied(MarshalException):
|
||||
message = u._("Permission to write image storage media denied.")
|
||||
|
||||
|
||||
class AuthBadRequest(MarshalException):
|
||||
message = u._("Connect error/bad request to Auth service at URL %(url)s.")
|
||||
|
||||
|
||||
class AuthUrlNotFound(MarshalException):
|
||||
message = u._("Auth service at URL %(url)s not found.")
|
||||
|
||||
|
||||
class AuthorizationFailure(MarshalException):
|
||||
message = u._("Authorization failed.")
|
||||
|
||||
|
||||
class NotAuthenticated(MarshalException):
|
||||
message = u._("You are not authenticated.")
|
||||
|
||||
|
||||
class Forbidden(MarshalException):
|
||||
message = u._("You are not authorized to complete this action.")
|
||||
|
||||
|
||||
class NotSupported(MarshalException):
|
||||
message = u._("Operation is not supported.")
|
||||
|
||||
|
||||
class ForbiddenPublicImage(Forbidden):
|
||||
message = u._("You are not authorized to complete this action.")
|
||||
|
||||
|
||||
class ProtectedImageDelete(Forbidden):
|
||||
message = u._("Image %(image_id)s is protected and cannot be deleted.")
|
||||
|
||||
|
||||
# NOTE(bcwaldon): here for backwards-compatibility, need to deprecate.
|
||||
class NotAuthorized(Forbidden):
|
||||
message = u._("You are not authorized to complete this action.")
|
||||
|
||||
|
||||
class Invalid(MarshalException):
|
||||
message = u._("Data supplied was not valid.")
|
||||
|
||||
|
||||
class NoDataToProcess(MarshalHTTPException):
|
||||
message = u._("No data supplied to process.")
|
||||
client_message = message
|
||||
status_code = 400
|
||||
|
||||
|
||||
class InvalidSortKey(Invalid):
|
||||
message = u._("Sort key supplied was not valid.")
|
||||
|
||||
|
||||
class InvalidFilterRangeValue(Invalid):
|
||||
message = u._("Unable to filter using the specified range.")
|
||||
|
||||
|
||||
class ReadonlyProperty(Forbidden):
|
||||
message = u._("Attribute '%(property)s' is read-only.")
|
||||
|
||||
|
||||
class ReservedProperty(Forbidden):
|
||||
message = u._("Attribute '%(property)s' is reserved.")
|
||||
|
||||
|
||||
class AuthorizationRedirect(MarshalException):
|
||||
message = u._("Redirecting to %(uri)s for authorization.")
|
||||
|
||||
|
||||
class DatabaseMigrationError(MarshalException):
|
||||
message = u._("There was an error migrating the database.")
|
||||
|
||||
|
||||
class ClientConnectionError(MarshalException):
|
||||
message = u._("There was an error connecting to a server")
|
||||
|
||||
|
||||
class ClientConfigurationError(MarshalException):
|
||||
message = u._("There was an error configuring the client.")
|
||||
|
||||
|
||||
class MultipleChoices(MarshalException):
|
||||
message = u._("The request returned a 302 Multiple Choices. This "
|
||||
"generally means that you have not included a version "
|
||||
"indicator in a request URI.\n\nThe body of response "
|
||||
"returned:\n%(body)s")
|
||||
|
||||
|
||||
class LimitExceeded(MarshalHTTPException):
|
||||
message = u._("The request returned a 413 Request Entity Too Large. This "
|
||||
"generally means that rate limiting or a quota threshold "
|
||||
"was breached.\n\nThe response body:\n%(body)s")
|
||||
client_message = u._("Provided information too large to process")
|
||||
status_code = 413
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(LimitExceeded, self).__init__(*args, **kwargs)
|
||||
self.retry_after = (int(kwargs['retry']) if kwargs.get('retry')
|
||||
else None)
|
||||
|
||||
|
||||
class ServiceUnavailable(MarshalException):
|
||||
message = u._("The request returned 503 Service Unavilable. This "
|
||||
"generally occurs on service overload or other transient "
|
||||
"outage.")
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(ServiceUnavailable, self).__init__(*args, **kwargs)
|
||||
self.retry_after = (int(kwargs['retry']) if kwargs.get('retry')
|
||||
else None)
|
||||
|
||||
|
||||
class ServerError(MarshalException):
|
||||
message = u._("The request returned 500 Internal Server Error.")
|
||||
|
||||
|
||||
class UnexpectedStatus(MarshalException):
|
||||
message = u._("The request returned an unexpected status: %(status)s."
|
||||
"\n\nThe response body:\n%(body)s")
|
||||
|
||||
|
||||
class InvalidContentType(MarshalException):
|
||||
message = u._("Invalid content type %(content_type)s")
|
||||
|
||||
|
||||
class InvalidContentEncoding(MarshalException):
|
||||
message = u._("Invalid content encoding %(content_encoding)s")
|
||||
|
||||
|
||||
class BadRegistryConnectionConfiguration(MarshalException):
|
||||
message = u._("Registry was not configured correctly on API server. "
|
||||
"Reason: %(reason)s")
|
||||
|
||||
|
||||
class BadStoreConfiguration(MarshalException):
|
||||
message = u._("Store %(store_name)s could not be configured correctly. "
|
||||
"Reason: %(reason)s")
|
||||
|
||||
|
||||
class BadDriverConfiguration(MarshalException):
|
||||
message = u._("Driver %(driver_name)s could not be configured correctly. "
|
||||
"Reason: %(reason)s")
|
||||
|
||||
|
||||
class StoreDeleteNotSupported(MarshalException):
|
||||
message = u._("Deleting images from this store is not supported.")
|
||||
|
||||
|
||||
class StoreAddDisabled(MarshalException):
|
||||
message = u._("Configuration for store failed. Adding images to this "
|
||||
"store is disabled.")
|
||||
|
||||
|
||||
class InvalidNotifierStrategy(MarshalException):
|
||||
message = u._("'%(strategy)s' is not an available notifier strategy.")
|
||||
|
||||
|
||||
class MaxRedirectsExceeded(MarshalException):
|
||||
message = u._("Maximum redirects (%(redirects)s) was exceeded.")
|
||||
|
||||
|
||||
class InvalidRedirect(MarshalException):
|
||||
message = u._("Received invalid HTTP redirect.")
|
||||
|
||||
|
||||
class NoServiceEndpoint(MarshalException):
|
||||
message = u._("Response from Keystone does not contain a "
|
||||
"Marshal endpoint.")
|
||||
|
||||
|
||||
class RegionAmbiguity(MarshalException):
|
||||
message = u._("Multiple 'image' service matches for region %(region)s. "
|
||||
"This generally means that a region is required and you "
|
||||
"have not supplied one.")
|
||||
|
||||
|
||||
class WorkerCreationFailure(MarshalException):
|
||||
message = u._("Server worker creation failed: %(reason)s.")
|
||||
|
||||
|
||||
class SchemaLoadError(MarshalException):
|
||||
message = u._("Unable to load schema: %(reason)s")
|
||||
|
||||
|
||||
class InvalidObject(MarshalHTTPException):
|
||||
status_code = 400
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
self.invalid_property = kwargs.get('property')
|
||||
self.message = u._("Failed to validate JSON information: ")
|
||||
self.client_message = u._("Provided object does not match "
|
||||
"schema '{schema}': "
|
||||
"{reason}").format(*args, **kwargs)
|
||||
self.message = self.message + self.client_message
|
||||
super(InvalidObject, self).__init__(*args, **kwargs)
|
||||
|
||||
|
||||
class PayloadDecodingError(MarshalHTTPException):
|
||||
status_code = 400
|
||||
message = u._("Error while attempting to decode payload.")
|
||||
client_message = u._("Unable to decode request data.")
|
||||
|
||||
|
||||
class UnsupportedField(MarshalHTTPException):
|
||||
message = u._("No support for value set on field '%(field)s' on "
|
||||
"schema '%(schema)s': %(reason)s")
|
||||
client_message = u._("Provided field value is not supported")
|
||||
status_code = 400
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(UnsupportedField, self).__init__(*args, **kwargs)
|
||||
self.invalid_field = kwargs.get('field')
|
||||
|
||||
|
||||
class FeatureNotImplemented(MarshalException):
|
||||
message = u._("Feature not implemented for value set on field "
|
||||
"'%(field)s' on " "schema '%(schema)s': %(reason)s")
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(FeatureNotImplemented, self).__init__(*args, **kwargs)
|
||||
self.invalid_field = kwargs.get('field')
|
||||
|
||||
|
||||
class UnsupportedHeaderFeature(MarshalException):
|
||||
message = u._("Provided header feature is unsupported: %(feature)s")
|
||||
|
||||
|
||||
class InUseByStore(MarshalException):
|
||||
message = u._("The image cannot be deleted because it is in use through "
|
||||
"the backend store outside of Marshal.")
|
||||
|
||||
|
||||
class ImageSizeLimitExceeded(MarshalException):
|
||||
message = u._("The provided image is too large.")
|
||||
|
||||
|
||||
class StoredKeyContainerNotFound(MarshalException):
|
||||
message = u._("Container %(container_id)s does not exist for stored "
|
||||
"key certificate generation.")
|
||||
|
||||
|
||||
class StoredKeyPrivateKeyNotFound(MarshalException):
|
||||
message = u._("Container %(container_id)s does not reference a private "
|
||||
"key needed for stored key certificate generation.")
|
||||
|
||||
|
||||
class InvalidUUIDInURI(MarshalHTTPException):
|
||||
message = u._("The provided UUID in the URI (%(uuid_string)s) is "
|
||||
"malformed.")
|
||||
client_message = u._("The provided UUID in the URI is malformed.")
|
||||
status_code = 404
|
||||
|
||||
|
||||
class InvalidCAID(MarshalHTTPException):
|
||||
message = u._("Invalid CA_ID: %(ca_id)s")
|
||||
client_message = u._("The ca_id provided in the request is invalid")
|
||||
status_code = 400
|
||||
|
||||
|
||||
class CANotDefinedForProject(MarshalHTTPException):
|
||||
message = u._("CA specified by ca_id %(ca_id)s not defined for project: "
|
||||
"%(project_id)s")
|
||||
client_message = u._("The ca_id provided in the request is not defined "
|
||||
"for this project")
|
||||
status_code = 403
|
||||
30
marshal_agent/i18n.py
Normal file
30
marshal_agent/i18n.py
Normal file
@ -0,0 +1,30 @@
|
||||
# Copyright 2010-2011 OpenStack LLC.
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
import oslo_i18n as i18n
|
||||
|
||||
_translators = i18n.TranslatorFactory(domain='marshal')
|
||||
|
||||
# The primary translation function using the well-known name "_"
|
||||
_ = _translators.primary
|
||||
|
||||
# Translators for log levels.
|
||||
#
|
||||
# The abbreviated names are meant to reflect the usual use of a short
|
||||
# name like '_'. The "L" is for "log" and the other letter comes from
|
||||
# the level.
|
||||
_LI = _translators.log_info
|
||||
_LW = _translators.log_warning
|
||||
_LE = _translators.log_error
|
||||
_LC = _translators.log_critical
|
||||
0
marshal_agent/openstack/__init__.py
Normal file
0
marshal_agent/openstack/__init__.py
Normal file
0
marshal_agent/openstack/common/__init__.py
Normal file
0
marshal_agent/openstack/common/__init__.py
Normal file
45
marshal_agent/openstack/common/_i18n.py
Normal file
45
marshal_agent/openstack/common/_i18n.py
Normal file
@ -0,0 +1,45 @@
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
"""oslo.i18n integration module.
|
||||
|
||||
See http://docs.openstack.org/developer/oslo.i18n/usage.html
|
||||
|
||||
"""
|
||||
|
||||
try:
|
||||
import oslo_i18n
|
||||
|
||||
# NOTE(dhellmann): This reference to o-s-l-o will be replaced by the
|
||||
# application name when this module is synced into the separate
|
||||
# repository. It is OK to have more than one translation function
|
||||
# using the same domain, since there will still only be one message
|
||||
# catalog.
|
||||
_translators = oslo_i18n.TranslatorFactory(domain='custodian')
|
||||
|
||||
# The primary translation function using the well-known name "_"
|
||||
_ = _translators.primary
|
||||
|
||||
# Translators for log levels.
|
||||
#
|
||||
# The abbreviated names are meant to reflect the usual use of a short
|
||||
# name like '_'. The "L" is for "log" and the other letter comes from
|
||||
# the level.
|
||||
_LI = _translators.log_info
|
||||
_LW = _translators.log_warning
|
||||
_LE = _translators.log_error
|
||||
_LC = _translators.log_critical
|
||||
except ImportError:
|
||||
# NOTE(dims): Support for cases where a project wants to use
|
||||
# code from oslo-incubator, but is not ready to be internationalized
|
||||
# (like tempest)
|
||||
_ = _LI = _LW = _LE = _LC = lambda x: x
|
||||
151
marshal_agent/openstack/common/eventlet_backdoor.py
Normal file
151
marshal_agent/openstack/common/eventlet_backdoor.py
Normal file
@ -0,0 +1,151 @@
|
||||
# Copyright (c) 2012 OpenStack Foundation.
|
||||
# Administrator of the National Aeronautics and Space Administration.
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import copy
|
||||
import errno
|
||||
import gc
|
||||
import logging
|
||||
import os
|
||||
import pprint
|
||||
import socket
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
import eventlet.backdoor
|
||||
import greenlet
|
||||
from oslo_config import cfg
|
||||
|
||||
from custodian.openstack.common._i18n import _LI
|
||||
|
||||
help_for_backdoor_port = (
|
||||
"Acceptable values are 0, <port>, and <start>:<end>, where 0 results "
|
||||
"in listening on a random tcp port number; <port> results in listening "
|
||||
"on the specified port number (and not enabling backdoor if that port "
|
||||
"is in use); and <start>:<end> results in listening on the smallest "
|
||||
"unused port number within the specified range of port numbers. The "
|
||||
"chosen port is displayed in the service's log file.")
|
||||
eventlet_backdoor_opts = [
|
||||
cfg.StrOpt('backdoor_port',
|
||||
help="Enable eventlet backdoor. %s" % help_for_backdoor_port)
|
||||
]
|
||||
|
||||
CONF = cfg.CONF
|
||||
CONF.register_opts(eventlet_backdoor_opts)
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def list_opts():
|
||||
"""Entry point for oslo-config-generator.
|
||||
"""
|
||||
return [(None, copy.deepcopy(eventlet_backdoor_opts))]
|
||||
|
||||
|
||||
class EventletBackdoorConfigValueError(Exception):
|
||||
def __init__(self, port_range, help_msg, ex):
|
||||
msg = ('Invalid backdoor_port configuration %(range)s: %(ex)s. '
|
||||
'%(help)s' %
|
||||
{'range': port_range, 'ex': ex, 'help': help_msg})
|
||||
super(EventletBackdoorConfigValueError, self).__init__(msg)
|
||||
self.port_range = port_range
|
||||
|
||||
|
||||
def _dont_use_this():
|
||||
print("Don't use this, just disconnect instead")
|
||||
|
||||
|
||||
def _find_objects(t):
|
||||
return [o for o in gc.get_objects() if isinstance(o, t)]
|
||||
|
||||
|
||||
def _print_greenthreads():
|
||||
for i, gt in enumerate(_find_objects(greenlet.greenlet)):
|
||||
print(i, gt)
|
||||
traceback.print_stack(gt.gr_frame)
|
||||
print()
|
||||
|
||||
|
||||
def _print_nativethreads():
|
||||
for threadId, stack in sys._current_frames().items():
|
||||
print(threadId)
|
||||
traceback.print_stack(stack)
|
||||
print()
|
||||
|
||||
|
||||
def _parse_port_range(port_range):
|
||||
if ':' not in port_range:
|
||||
start, end = port_range, port_range
|
||||
else:
|
||||
start, end = port_range.split(':', 1)
|
||||
try:
|
||||
start, end = int(start), int(end)
|
||||
if end < start:
|
||||
raise ValueError
|
||||
return start, end
|
||||
except ValueError as ex:
|
||||
raise EventletBackdoorConfigValueError(port_range, ex,
|
||||
help_for_backdoor_port)
|
||||
|
||||
|
||||
def _listen(host, start_port, end_port, listen_func):
|
||||
try_port = start_port
|
||||
while True:
|
||||
try:
|
||||
return listen_func((host, try_port))
|
||||
except socket.error as exc:
|
||||
if (exc.errno != errno.EADDRINUSE or
|
||||
try_port >= end_port):
|
||||
raise
|
||||
try_port += 1
|
||||
|
||||
|
||||
def initialize_if_enabled():
|
||||
backdoor_locals = {
|
||||
'exit': _dont_use_this, # So we don't exit the entire process
|
||||
'quit': _dont_use_this, # So we don't exit the entire process
|
||||
'fo': _find_objects,
|
||||
'pgt': _print_greenthreads,
|
||||
'pnt': _print_nativethreads,
|
||||
}
|
||||
|
||||
if CONF.backdoor_port is None:
|
||||
return None
|
||||
|
||||
start_port, end_port = _parse_port_range(str(CONF.backdoor_port))
|
||||
|
||||
# NOTE(johannes): The standard sys.displayhook will print the value of
|
||||
# the last expression and set it to __builtin__._, which overwrites
|
||||
# the __builtin__._ that gettext sets. Let's switch to using pprint
|
||||
# since it won't interact poorly with gettext, and it's easier to
|
||||
# read the output too.
|
||||
def displayhook(val):
|
||||
if val is not None:
|
||||
pprint.pprint(val)
|
||||
sys.displayhook = displayhook
|
||||
|
||||
sock = _listen('localhost', start_port, end_port, eventlet.listen)
|
||||
|
||||
# In the case of backdoor port being zero, a port number is assigned by
|
||||
# listen(). In any case, pull the port number out here.
|
||||
port = sock.getsockname()[1]
|
||||
LOG.info(
|
||||
_LI('Eventlet backdoor listening on %(port)s for process %(pid)d') %
|
||||
{'port': port, 'pid': os.getpid()}
|
||||
)
|
||||
eventlet.spawn_n(eventlet.backdoor.backdoor_server, sock,
|
||||
locals=backdoor_locals)
|
||||
return port
|
||||
45
marshal_agent/openstack/common/local.py
Normal file
45
marshal_agent/openstack/common/local.py
Normal file
@ -0,0 +1,45 @@
|
||||
# Copyright 2011 OpenStack Foundation.
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
"""Local storage of variables using weak references"""
|
||||
|
||||
import threading
|
||||
import weakref
|
||||
|
||||
|
||||
class WeakLocal(threading.local):
|
||||
def __getattribute__(self, attr):
|
||||
rval = super(WeakLocal, self).__getattribute__(attr)
|
||||
if rval:
|
||||
# NOTE(mikal): this bit is confusing. What is stored is a weak
|
||||
# reference, not the value itself. We therefore need to lookup
|
||||
# the weak reference and return the inner value here.
|
||||
rval = rval()
|
||||
return rval
|
||||
|
||||
def __setattr__(self, attr, value):
|
||||
value = weakref.ref(value)
|
||||
return super(WeakLocal, self).__setattr__(attr, value)
|
||||
|
||||
|
||||
# NOTE(mikal): the name "store" should be deprecated in the future
|
||||
store = WeakLocal()
|
||||
|
||||
# A "weak" store uses weak references and allows an object to fall out of scope
|
||||
# when it falls out of scope in the code that uses the thread local storage. A
|
||||
# "strong" store will hold a reference to the object so that it never falls out
|
||||
# of scope.
|
||||
weak_store = WeakLocal()
|
||||
strong_store = threading.local()
|
||||
147
marshal_agent/openstack/common/loopingcall.py
Normal file
147
marshal_agent/openstack/common/loopingcall.py
Normal file
@ -0,0 +1,147 @@
|
||||
# Copyright 2010 United States Government as represented by the
|
||||
# Administrator of the National Aeronautics and Space Administration.
|
||||
# Copyright 2011 Justin Santa Barbara
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
|
||||
from eventlet import event
|
||||
from eventlet import greenthread
|
||||
|
||||
from custodian.openstack.common._i18n import _LE, _LW
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
# NOTE(zyluo): This lambda function was declared to avoid mocking collisions
|
||||
# with time.time() called in the standard logging module
|
||||
# during unittests.
|
||||
_ts = lambda: time.time()
|
||||
|
||||
|
||||
class LoopingCallDone(Exception):
|
||||
"""Exception to break out and stop a LoopingCallBase.
|
||||
|
||||
The poll-function passed to LoopingCallBase can raise this exception to
|
||||
break out of the loop normally. This is somewhat analogous to
|
||||
StopIteration.
|
||||
|
||||
An optional return-value can be included as the argument to the exception;
|
||||
this return-value will be returned by LoopingCallBase.wait()
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self, retvalue=True):
|
||||
""":param retvalue: Value that LoopingCallBase.wait() should return."""
|
||||
self.retvalue = retvalue
|
||||
|
||||
|
||||
class LoopingCallBase(object):
|
||||
def __init__(self, f=None, *args, **kw):
|
||||
self.args = args
|
||||
self.kw = kw
|
||||
self.f = f
|
||||
self._running = False
|
||||
self.done = None
|
||||
|
||||
def stop(self):
|
||||
self._running = False
|
||||
|
||||
def wait(self):
|
||||
return self.done.wait()
|
||||
|
||||
|
||||
class FixedIntervalLoopingCall(LoopingCallBase):
|
||||
"""A fixed interval looping call."""
|
||||
|
||||
def start(self, interval, initial_delay=None):
|
||||
self._running = True
|
||||
done = event.Event()
|
||||
|
||||
def _inner():
|
||||
if initial_delay:
|
||||
greenthread.sleep(initial_delay)
|
||||
|
||||
try:
|
||||
while self._running:
|
||||
start = _ts()
|
||||
self.f(*self.args, **self.kw)
|
||||
end = _ts()
|
||||
if not self._running:
|
||||
break
|
||||
delay = end - start - interval
|
||||
if delay > 0:
|
||||
LOG.warn(_LW('task %(func_name)r run outlasted '
|
||||
'interval by %(delay).2f sec'),
|
||||
{'func_name': self.f, 'delay': delay})
|
||||
greenthread.sleep(-delay if delay < 0 else 0)
|
||||
except LoopingCallDone as e:
|
||||
self.stop()
|
||||
done.send(e.retvalue)
|
||||
except Exception:
|
||||
LOG.exception(_LE('in fixed duration looping call'))
|
||||
done.send_exception(*sys.exc_info())
|
||||
return
|
||||
else:
|
||||
done.send(True)
|
||||
|
||||
self.done = done
|
||||
|
||||
greenthread.spawn_n(_inner)
|
||||
return self.done
|
||||
|
||||
|
||||
class DynamicLoopingCall(LoopingCallBase):
|
||||
"""A looping call which sleeps until the next known event.
|
||||
|
||||
The function called should return how long to sleep for before being
|
||||
called again.
|
||||
"""
|
||||
|
||||
def start(self, initial_delay=None, periodic_interval_max=None):
|
||||
self._running = True
|
||||
done = event.Event()
|
||||
|
||||
def _inner():
|
||||
if initial_delay:
|
||||
greenthread.sleep(initial_delay)
|
||||
|
||||
try:
|
||||
while self._running:
|
||||
idle = self.f(*self.args, **self.kw)
|
||||
if not self._running:
|
||||
break
|
||||
|
||||
if periodic_interval_max is not None:
|
||||
idle = min(idle, periodic_interval_max)
|
||||
LOG.debug('Dynamic looping call %(func_name)r sleeping '
|
||||
'for %(idle).02f seconds',
|
||||
{'func_name': self.f, 'idle': idle})
|
||||
greenthread.sleep(idle)
|
||||
except LoopingCallDone as e:
|
||||
self.stop()
|
||||
done.send(e.retvalue)
|
||||
except Exception:
|
||||
LOG.exception(_LE('in dynamic looping call'))
|
||||
done.send_exception(*sys.exc_info())
|
||||
return
|
||||
else:
|
||||
done.send(True)
|
||||
|
||||
self.done = done
|
||||
|
||||
greenthread.spawn(_inner)
|
||||
return self.done
|
||||
503
marshal_agent/openstack/common/service.py
Normal file
503
marshal_agent/openstack/common/service.py
Normal file
@ -0,0 +1,503 @@
|
||||
# Copyright 2010 United States Government as represented by the
|
||||
# Administrator of the National Aeronautics and Space Administration.
|
||||
# Copyright 2011 Justin Santa Barbara
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
"""Generic Node base class for all workers that run on hosts."""
|
||||
|
||||
import errno
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
|
||||
try:
|
||||
# Importing just the symbol here because the io module does not
|
||||
# exist in Python 2.6.
|
||||
from io import UnsupportedOperation # noqa
|
||||
except ImportError:
|
||||
# Python 2.6
|
||||
UnsupportedOperation = None
|
||||
|
||||
import eventlet
|
||||
from eventlet import event
|
||||
from oslo_config import cfg
|
||||
|
||||
from custodian.openstack.common import eventlet_backdoor
|
||||
from custodian.openstack.common._i18n import _LE, _LI, _LW
|
||||
from custodian.openstack.common import systemd
|
||||
from custodian.openstack.common import threadgroup
|
||||
|
||||
|
||||
CONF = cfg.CONF
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _sighup_supported():
|
||||
return hasattr(signal, 'SIGHUP')
|
||||
|
||||
|
||||
def _is_daemon():
|
||||
# The process group for a foreground process will match the
|
||||
# process group of the controlling terminal. If those values do
|
||||
# not match, or ioctl() fails on the stdout file handle, we assume
|
||||
# the process is running in the background as a daemon.
|
||||
# http://www.gnu.org/software/bash/manual/bashref.html#Job-Control-Basics
|
||||
try:
|
||||
is_daemon = os.getpgrp() != os.tcgetpgrp(sys.stdout.fileno())
|
||||
except OSError as err:
|
||||
if err.errno == errno.ENOTTY:
|
||||
# Assume we are a daemon because there is no terminal.
|
||||
is_daemon = True
|
||||
else:
|
||||
raise
|
||||
except UnsupportedOperation:
|
||||
# Could not get the fileno for stdout, so we must be a daemon.
|
||||
is_daemon = True
|
||||
return is_daemon
|
||||
|
||||
|
||||
def _is_sighup_and_daemon(signo):
|
||||
if not (_sighup_supported() and signo == signal.SIGHUP):
|
||||
# Avoid checking if we are a daemon, because the signal isn't
|
||||
# SIGHUP.
|
||||
return False
|
||||
return _is_daemon()
|
||||
|
||||
|
||||
def _signo_to_signame(signo):
|
||||
signals = {signal.SIGTERM: 'SIGTERM',
|
||||
signal.SIGINT: 'SIGINT'}
|
||||
if _sighup_supported():
|
||||
signals[signal.SIGHUP] = 'SIGHUP'
|
||||
return signals[signo]
|
||||
|
||||
|
||||
def _set_signals_handler(handler):
|
||||
signal.signal(signal.SIGTERM, handler)
|
||||
signal.signal(signal.SIGINT, handler)
|
||||
if _sighup_supported():
|
||||
signal.signal(signal.SIGHUP, handler)
|
||||
|
||||
|
||||
class Launcher(object):
|
||||
"""Launch one or more services and wait for them to complete."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize the service launcher.
|
||||
|
||||
:returns: None
|
||||
|
||||
"""
|
||||
self.services = Services()
|
||||
self.backdoor_port = eventlet_backdoor.initialize_if_enabled()
|
||||
|
||||
def launch_service(self, service):
|
||||
"""Load and start the given service.
|
||||
|
||||
:param service: The service you would like to start.
|
||||
:returns: None
|
||||
|
||||
"""
|
||||
service.backdoor_port = self.backdoor_port
|
||||
self.services.add(service)
|
||||
|
||||
def stop(self):
|
||||
"""Stop all services which are currently running.
|
||||
|
||||
:returns: None
|
||||
|
||||
"""
|
||||
self.services.stop()
|
||||
|
||||
def wait(self):
|
||||
"""Waits until all services have been stopped, and then returns.
|
||||
|
||||
:returns: None
|
||||
|
||||
"""
|
||||
self.services.wait()
|
||||
|
||||
def restart(self):
|
||||
"""Reload config files and restart service.
|
||||
|
||||
:returns: None
|
||||
|
||||
"""
|
||||
cfg.CONF.reload_config_files()
|
||||
self.services.restart()
|
||||
|
||||
|
||||
class SignalExit(SystemExit):
|
||||
def __init__(self, signo, exccode=1):
|
||||
super(SignalExit, self).__init__(exccode)
|
||||
self.signo = signo
|
||||
|
||||
|
||||
class ServiceLauncher(Launcher):
|
||||
def _handle_signal(self, signo, frame):
|
||||
# Allow the process to be killed again and die from natural causes
|
||||
_set_signals_handler(signal.SIG_DFL)
|
||||
raise SignalExit(signo)
|
||||
|
||||
def handle_signal(self):
|
||||
_set_signals_handler(self._handle_signal)
|
||||
|
||||
def _wait_for_exit_or_signal(self, ready_callback=None):
|
||||
status = None
|
||||
signo = 0
|
||||
|
||||
LOG.debug('Full set of CONF:')
|
||||
CONF.log_opt_values(LOG, logging.DEBUG)
|
||||
|
||||
try:
|
||||
if ready_callback:
|
||||
ready_callback()
|
||||
super(ServiceLauncher, self).wait()
|
||||
except SignalExit as exc:
|
||||
signame = _signo_to_signame(exc.signo)
|
||||
LOG.info(_LI('Caught %s, exiting'), signame)
|
||||
status = exc.code
|
||||
signo = exc.signo
|
||||
except SystemExit as exc:
|
||||
status = exc.code
|
||||
finally:
|
||||
self.stop()
|
||||
|
||||
return status, signo
|
||||
|
||||
def wait(self, ready_callback=None):
|
||||
systemd.notify_once()
|
||||
while True:
|
||||
self.handle_signal()
|
||||
status, signo = self._wait_for_exit_or_signal(ready_callback)
|
||||
if not _is_sighup_and_daemon(signo):
|
||||
return status
|
||||
self.restart()
|
||||
|
||||
|
||||
class ServiceWrapper(object):
|
||||
def __init__(self, service, workers):
|
||||
self.service = service
|
||||
self.workers = workers
|
||||
self.children = set()
|
||||
self.forktimes = []
|
||||
|
||||
|
||||
class ProcessLauncher(object):
|
||||
_signal_handlers_set = set()
|
||||
|
||||
@classmethod
|
||||
def _handle_class_signals(cls, *args, **kwargs):
|
||||
for handler in cls._signal_handlers_set:
|
||||
handler(*args, **kwargs)
|
||||
|
||||
def __init__(self):
|
||||
"""Constructor."""
|
||||
|
||||
self.children = {}
|
||||
self.sigcaught = None
|
||||
self.running = True
|
||||
rfd, self.writepipe = os.pipe()
|
||||
self.readpipe = eventlet.greenio.GreenPipe(rfd, 'r')
|
||||
self.handle_signal()
|
||||
|
||||
def handle_signal(self):
|
||||
self._signal_handlers_set.add(self._handle_signal)
|
||||
_set_signals_handler(self._handle_class_signals)
|
||||
|
||||
def _handle_signal(self, signo, frame):
|
||||
self.sigcaught = signo
|
||||
self.running = False
|
||||
|
||||
# Allow the process to be killed again and die from natural causes
|
||||
_set_signals_handler(signal.SIG_DFL)
|
||||
|
||||
def _pipe_watcher(self):
|
||||
# This will block until the write end is closed when the parent
|
||||
# dies unexpectedly
|
||||
self.readpipe.read()
|
||||
|
||||
LOG.info(_LI('Parent process has died unexpectedly, exiting'))
|
||||
|
||||
sys.exit(1)
|
||||
|
||||
def _child_process_handle_signal(self):
|
||||
# Setup child signal handlers differently
|
||||
def _sigterm(*args):
|
||||
signal.signal(signal.SIGTERM, signal.SIG_DFL)
|
||||
raise SignalExit(signal.SIGTERM)
|
||||
|
||||
def _sighup(*args):
|
||||
signal.signal(signal.SIGHUP, signal.SIG_DFL)
|
||||
raise SignalExit(signal.SIGHUP)
|
||||
|
||||
signal.signal(signal.SIGTERM, _sigterm)
|
||||
if _sighup_supported():
|
||||
signal.signal(signal.SIGHUP, _sighup)
|
||||
# Block SIGINT and let the parent send us a SIGTERM
|
||||
signal.signal(signal.SIGINT, signal.SIG_IGN)
|
||||
|
||||
def _child_wait_for_exit_or_signal(self, launcher):
|
||||
status = 0
|
||||
signo = 0
|
||||
|
||||
# NOTE(johannes): All exceptions are caught to ensure this
|
||||
# doesn't fallback into the loop spawning children. It would
|
||||
# be bad for a child to spawn more children.
|
||||
try:
|
||||
launcher.wait()
|
||||
except SignalExit as exc:
|
||||
signame = _signo_to_signame(exc.signo)
|
||||
LOG.info(_LI('Child caught %s, exiting'), signame)
|
||||
status = exc.code
|
||||
signo = exc.signo
|
||||
except SystemExit as exc:
|
||||
status = exc.code
|
||||
except BaseException:
|
||||
LOG.exception(_LE('Unhandled exception'))
|
||||
status = 2
|
||||
finally:
|
||||
launcher.stop()
|
||||
|
||||
return status, signo
|
||||
|
||||
def _child_process(self, service):
|
||||
self._child_process_handle_signal()
|
||||
|
||||
# Reopen the eventlet hub to make sure we don't share an epoll
|
||||
# fd with parent and/or siblings, which would be bad
|
||||
eventlet.hubs.use_hub()
|
||||
|
||||
# Close write to ensure only parent has it open
|
||||
os.close(self.writepipe)
|
||||
# Create greenthread to watch for parent to close pipe
|
||||
eventlet.spawn_n(self._pipe_watcher)
|
||||
|
||||
# Reseed random number generator
|
||||
random.seed()
|
||||
|
||||
launcher = Launcher()
|
||||
launcher.launch_service(service)
|
||||
return launcher
|
||||
|
||||
def _start_child(self, wrap):
|
||||
if len(wrap.forktimes) > wrap.workers:
|
||||
# Limit ourselves to one process a second (over the period of
|
||||
# number of workers * 1 second). This will allow workers to
|
||||
# start up quickly but ensure we don't fork off children that
|
||||
# die instantly too quickly.
|
||||
if time.time() - wrap.forktimes[0] < wrap.workers:
|
||||
LOG.info(_LI('Forking too fast, sleeping'))
|
||||
time.sleep(1)
|
||||
|
||||
wrap.forktimes.pop(0)
|
||||
|
||||
wrap.forktimes.append(time.time())
|
||||
|
||||
pid = os.fork()
|
||||
if pid == 0:
|
||||
launcher = self._child_process(wrap.service)
|
||||
while True:
|
||||
self._child_process_handle_signal()
|
||||
status, signo = self._child_wait_for_exit_or_signal(launcher)
|
||||
if not _is_sighup_and_daemon(signo):
|
||||
break
|
||||
launcher.restart()
|
||||
|
||||
os._exit(status)
|
||||
|
||||
LOG.info(_LI('Started child %d'), pid)
|
||||
|
||||
wrap.children.add(pid)
|
||||
self.children[pid] = wrap
|
||||
|
||||
return pid
|
||||
|
||||
def launch_service(self, service, workers=1):
|
||||
wrap = ServiceWrapper(service, workers)
|
||||
|
||||
LOG.info(_LI('Starting %d workers'), wrap.workers)
|
||||
while self.running and len(wrap.children) < wrap.workers:
|
||||
self._start_child(wrap)
|
||||
|
||||
def _wait_child(self):
|
||||
try:
|
||||
# Block while any of child processes have exited
|
||||
pid, status = os.waitpid(0, 0)
|
||||
if not pid:
|
||||
return None
|
||||
except OSError as exc:
|
||||
if exc.errno not in (errno.EINTR, errno.ECHILD):
|
||||
raise
|
||||
return None
|
||||
|
||||
if os.WIFSIGNALED(status):
|
||||
sig = os.WTERMSIG(status)
|
||||
LOG.info(_LI('Child %(pid)d killed by signal %(sig)d'),
|
||||
dict(pid=pid, sig=sig))
|
||||
else:
|
||||
code = os.WEXITSTATUS(status)
|
||||
LOG.info(_LI('Child %(pid)s exited with status %(code)d'),
|
||||
dict(pid=pid, code=code))
|
||||
|
||||
if pid not in self.children:
|
||||
LOG.warning(_LW('pid %d not in child list'), pid)
|
||||
return None
|
||||
|
||||
wrap = self.children.pop(pid)
|
||||
wrap.children.remove(pid)
|
||||
return wrap
|
||||
|
||||
def _respawn_children(self):
|
||||
while self.running:
|
||||
wrap = self._wait_child()
|
||||
if not wrap:
|
||||
continue
|
||||
while self.running and len(wrap.children) < wrap.workers:
|
||||
self._start_child(wrap)
|
||||
|
||||
def wait(self):
|
||||
"""Loop waiting on children to die and respawning as necessary."""
|
||||
|
||||
systemd.notify_once()
|
||||
LOG.debug('Full set of CONF:')
|
||||
CONF.log_opt_values(LOG, logging.DEBUG)
|
||||
|
||||
try:
|
||||
while True:
|
||||
self.handle_signal()
|
||||
self._respawn_children()
|
||||
# No signal means that stop was called. Don't clean up here.
|
||||
if not self.sigcaught:
|
||||
return
|
||||
|
||||
signame = _signo_to_signame(self.sigcaught)
|
||||
LOG.info(_LI('Caught %s, stopping children'), signame)
|
||||
if not _is_sighup_and_daemon(self.sigcaught):
|
||||
break
|
||||
|
||||
for pid in self.children:
|
||||
os.kill(pid, signal.SIGHUP)
|
||||
self.running = True
|
||||
self.sigcaught = None
|
||||
except eventlet.greenlet.GreenletExit:
|
||||
LOG.info(_LI("Wait called after thread killed. Cleaning up."))
|
||||
|
||||
self.stop()
|
||||
|
||||
def stop(self):
|
||||
"""Terminate child processes and wait on each."""
|
||||
self.running = False
|
||||
for pid in self.children:
|
||||
try:
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
except OSError as exc:
|
||||
if exc.errno != errno.ESRCH:
|
||||
raise
|
||||
|
||||
# Wait for children to die
|
||||
if self.children:
|
||||
LOG.info(_LI('Waiting on %d children to exit'), len(self.children))
|
||||
while self.children:
|
||||
self._wait_child()
|
||||
|
||||
|
||||
class Service(object):
|
||||
"""Service object for binaries running on hosts."""
|
||||
|
||||
def __init__(self, threads=1000):
|
||||
self.tg = threadgroup.ThreadGroup(threads)
|
||||
|
||||
# signal that the service is done shutting itself down:
|
||||
self._done = event.Event()
|
||||
|
||||
def reset(self):
|
||||
# NOTE(Fengqian): docs for Event.reset() recommend against using it
|
||||
self._done = event.Event()
|
||||
|
||||
def start(self):
|
||||
pass
|
||||
|
||||
def stop(self, graceful=False):
|
||||
self.tg.stop(graceful)
|
||||
self.tg.wait()
|
||||
# Signal that service cleanup is done:
|
||||
if not self._done.ready():
|
||||
self._done.send()
|
||||
|
||||
def wait(self):
|
||||
self._done.wait()
|
||||
|
||||
|
||||
class Services(object):
|
||||
|
||||
def __init__(self):
|
||||
self.services = []
|
||||
self.tg = threadgroup.ThreadGroup()
|
||||
self.done = event.Event()
|
||||
|
||||
def add(self, service):
|
||||
self.services.append(service)
|
||||
self.tg.add_thread(self.run_service, service, self.done)
|
||||
|
||||
def stop(self):
|
||||
# wait for graceful shutdown of services:
|
||||
for service in self.services:
|
||||
service.stop()
|
||||
service.wait()
|
||||
|
||||
# Each service has performed cleanup, now signal that the run_service
|
||||
# wrapper threads can now die:
|
||||
if not self.done.ready():
|
||||
self.done.send()
|
||||
|
||||
# reap threads:
|
||||
self.tg.stop()
|
||||
|
||||
def wait(self):
|
||||
self.tg.wait()
|
||||
|
||||
def restart(self):
|
||||
self.stop()
|
||||
self.done = event.Event()
|
||||
for restart_service in self.services:
|
||||
restart_service.reset()
|
||||
self.tg.add_thread(self.run_service, restart_service, self.done)
|
||||
|
||||
@staticmethod
|
||||
def run_service(service, done):
|
||||
"""Service start wrapper.
|
||||
|
||||
:param service: service to run
|
||||
:param done: event to wait on until a shutdown is triggered
|
||||
:returns: None
|
||||
|
||||
"""
|
||||
service.start()
|
||||
done.wait()
|
||||
|
||||
|
||||
def launch(service, workers=1):
|
||||
if workers is None or workers == 1:
|
||||
launcher = ServiceLauncher()
|
||||
launcher.launch_service(service)
|
||||
else:
|
||||
launcher = ProcessLauncher()
|
||||
launcher.launch_service(service, workers=workers)
|
||||
|
||||
return launcher
|
||||
105
marshal_agent/openstack/common/systemd.py
Normal file
105
marshal_agent/openstack/common/systemd.py
Normal file
@ -0,0 +1,105 @@
|
||||
# Copyright 2012-2014 Red Hat, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
"""
|
||||
Helper module for systemd service readiness notification.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _abstractify(socket_name):
|
||||
if socket_name.startswith('@'):
|
||||
# abstract namespace socket
|
||||
socket_name = '\0%s' % socket_name[1:]
|
||||
return socket_name
|
||||
|
||||
|
||||
def _sd_notify(unset_env, msg):
|
||||
notify_socket = os.getenv('NOTIFY_SOCKET')
|
||||
if notify_socket:
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
try:
|
||||
sock.connect(_abstractify(notify_socket))
|
||||
sock.sendall(msg)
|
||||
if unset_env:
|
||||
del os.environ['NOTIFY_SOCKET']
|
||||
except EnvironmentError:
|
||||
LOG.debug("Systemd notification failed", exc_info=True)
|
||||
finally:
|
||||
sock.close()
|
||||
|
||||
|
||||
def notify():
|
||||
"""Send notification to Systemd that service is ready.
|
||||
|
||||
For details see
|
||||
http://www.freedesktop.org/software/systemd/man/sd_notify.html
|
||||
"""
|
||||
_sd_notify(False, 'READY=1')
|
||||
|
||||
|
||||
def notify_once():
|
||||
"""Send notification once to Systemd that service is ready.
|
||||
|
||||
Systemd sets NOTIFY_SOCKET environment variable with the name of the
|
||||
socket listening for notifications from services.
|
||||
This method removes the NOTIFY_SOCKET environment variable to ensure
|
||||
notification is sent only once.
|
||||
"""
|
||||
_sd_notify(True, 'READY=1')
|
||||
|
||||
|
||||
def onready(notify_socket, timeout):
|
||||
"""Wait for systemd style notification on the socket.
|
||||
|
||||
:param notify_socket: local socket address
|
||||
:type notify_socket: string
|
||||
:param timeout: socket timeout
|
||||
:type timeout: float
|
||||
:returns: 0 service ready
|
||||
1 service not ready
|
||||
2 timeout occurred
|
||||
"""
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
|
||||
sock.settimeout(timeout)
|
||||
sock.bind(_abstractify(notify_socket))
|
||||
try:
|
||||
msg = sock.recv(512)
|
||||
except socket.timeout:
|
||||
return 2
|
||||
finally:
|
||||
sock.close()
|
||||
if 'READY=1' in msg:
|
||||
return 0
|
||||
else:
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# simple CLI for testing
|
||||
if len(sys.argv) == 1:
|
||||
notify()
|
||||
elif len(sys.argv) >= 2:
|
||||
timeout = float(sys.argv[1])
|
||||
notify_socket = os.getenv('NOTIFY_SOCKET')
|
||||
if notify_socket:
|
||||
retval = onready(notify_socket, timeout)
|
||||
sys.exit(retval)
|
||||
149
marshal_agent/openstack/common/threadgroup.py
Normal file
149
marshal_agent/openstack/common/threadgroup.py
Normal file
@ -0,0 +1,149 @@
|
||||
# Copyright 2012 Red Hat, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
import logging
|
||||
import threading
|
||||
|
||||
import eventlet
|
||||
from eventlet import greenpool
|
||||
|
||||
from custodian.openstack.common import loopingcall
|
||||
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _thread_done(gt, *args, **kwargs):
|
||||
"""Callback function to be passed to GreenThread.link() when we spawn()
|
||||
Calls the :class:`ThreadGroup` to notify if.
|
||||
|
||||
"""
|
||||
kwargs['group'].thread_done(kwargs['thread'])
|
||||
|
||||
|
||||
class Thread(object):
|
||||
"""Wrapper around a greenthread, that holds a reference to the
|
||||
:class:`ThreadGroup`. The Thread will notify the :class:`ThreadGroup` when
|
||||
it has done so it can be removed from the threads list.
|
||||
"""
|
||||
def __init__(self, thread, group):
|
||||
self.thread = thread
|
||||
self.thread.link(_thread_done, group=group, thread=self)
|
||||
|
||||
def stop(self):
|
||||
self.thread.kill()
|
||||
|
||||
def wait(self):
|
||||
return self.thread.wait()
|
||||
|
||||
def link(self, func, *args, **kwargs):
|
||||
self.thread.link(func, *args, **kwargs)
|
||||
|
||||
|
||||
class ThreadGroup(object):
|
||||
"""The point of the ThreadGroup class is to:
|
||||
|
||||
* keep track of timers and greenthreads (making it easier to stop them
|
||||
when need be).
|
||||
* provide an easy API to add timers.
|
||||
"""
|
||||
def __init__(self, thread_pool_size=10):
|
||||
self.pool = greenpool.GreenPool(thread_pool_size)
|
||||
self.threads = []
|
||||
self.timers = []
|
||||
|
||||
def add_dynamic_timer(self, callback, initial_delay=None,
|
||||
periodic_interval_max=None, *args, **kwargs):
|
||||
timer = loopingcall.DynamicLoopingCall(callback, *args, **kwargs)
|
||||
timer.start(initial_delay=initial_delay,
|
||||
periodic_interval_max=periodic_interval_max)
|
||||
self.timers.append(timer)
|
||||
|
||||
def add_timer(self, interval, callback, initial_delay=None,
|
||||
*args, **kwargs):
|
||||
pulse = loopingcall.FixedIntervalLoopingCall(callback, *args, **kwargs)
|
||||
pulse.start(interval=interval,
|
||||
initial_delay=initial_delay)
|
||||
self.timers.append(pulse)
|
||||
|
||||
def add_thread(self, callback, *args, **kwargs):
|
||||
gt = self.pool.spawn(callback, *args, **kwargs)
|
||||
th = Thread(gt, self)
|
||||
self.threads.append(th)
|
||||
return th
|
||||
|
||||
def thread_done(self, thread):
|
||||
self.threads.remove(thread)
|
||||
|
||||
def _stop_threads(self):
|
||||
current = threading.current_thread()
|
||||
|
||||
# Iterate over a copy of self.threads so thread_done doesn't
|
||||
# modify the list while we're iterating
|
||||
for x in self.threads[:]:
|
||||
if x is current:
|
||||
# don't kill the current thread.
|
||||
continue
|
||||
try:
|
||||
x.stop()
|
||||
except eventlet.greenlet.GreenletExit:
|
||||
pass
|
||||
except Exception as ex:
|
||||
LOG.exception(ex)
|
||||
|
||||
def stop_timers(self):
|
||||
for x in self.timers:
|
||||
try:
|
||||
x.stop()
|
||||
except Exception as ex:
|
||||
LOG.exception(ex)
|
||||
self.timers = []
|
||||
|
||||
def stop(self, graceful=False):
|
||||
"""stop function has the option of graceful=True/False.
|
||||
|
||||
* In case of graceful=True, wait for all threads to be finished.
|
||||
Never kill threads.
|
||||
* In case of graceful=False, kill threads immediately.
|
||||
"""
|
||||
self.stop_timers()
|
||||
if graceful:
|
||||
# In case of graceful=True, wait for all threads to be
|
||||
# finished, never kill threads
|
||||
self.wait()
|
||||
else:
|
||||
# In case of graceful=False(Default), kill threads
|
||||
# immediately
|
||||
self._stop_threads()
|
||||
|
||||
def wait(self):
|
||||
for x in self.timers:
|
||||
try:
|
||||
x.wait()
|
||||
except eventlet.greenlet.GreenletExit:
|
||||
pass
|
||||
except Exception as ex:
|
||||
LOG.exception(ex)
|
||||
current = threading.current_thread()
|
||||
|
||||
# Iterate over a copy of self.threads so thread_done doesn't
|
||||
# modify the list while we're iterating
|
||||
for x in self.threads[:]:
|
||||
if x is current:
|
||||
continue
|
||||
try:
|
||||
x.wait()
|
||||
except eventlet.greenlet.GreenletExit:
|
||||
pass
|
||||
except Exception as ex:
|
||||
LOG.exception(ex)
|
||||
115
marshal_agent/tests/config.py
Normal file
115
marshal_agent/tests/config.py
Normal file
@ -0,0 +1,115 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Configuration setup for Testing Marshal.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import errno
|
||||
from oslo_config import cfg
|
||||
from oslo_log import log
|
||||
|
||||
import marshal_agent.i18n as u
|
||||
# import marshal.version
|
||||
|
||||
__builtins__['TESTING_MARSHAL'] = True
|
||||
|
||||
# Check if we have root perms. May need tweaking here for SELinux, ACLs, etc..
|
||||
if __builtins__.get('TESTING_MARSHAL', False) is False and os.getuid() != 0:
|
||||
print "Marshal needs to be run with root permissions."
|
||||
sys.exit(1)
|
||||
|
||||
KMS_TYPE = 'Barbican'
|
||||
KMS_BASE = 'a_test_base'
|
||||
KMS_API = 'a_test_api'
|
||||
SECRET_ID = 'a_test_secret_id'
|
||||
TENANT_ID = None
|
||||
KEYSTONE_ENDPOINT = None
|
||||
|
||||
KM_OPT_GRP_NAME = 'KM-OPT'
|
||||
VOL_CRYPT_GRP_NAME = 'crypt'
|
||||
|
||||
opt_group = cfg.OptGroup(name=KM_OPT_GRP_NAME,
|
||||
title='Key Manager config settings')
|
||||
|
||||
openam = [
|
||||
cfg.StrOpt('kms_type', default=KMS_TYPE,
|
||||
help=('Key Management Store Type.')),
|
||||
cfg.StrOpt('kms_base', default=KMS_BASE,
|
||||
help=('Key management service base url')),
|
||||
cfg.StrOpt('kms_get_key_api', default=KMS_API,
|
||||
help=('Key management service key retrieval API')),
|
||||
cfg.StrOpt('kms_key_id', default=SECRET_ID,
|
||||
help=('Key management service key ID')),
|
||||
cfg.StrOpt('kms_project_id', default=TENANT_ID,
|
||||
help=('Key management service project/tenant ID')),
|
||||
cfg.StrOpt('keystone_endpoint', default=KEYSTONE_ENDPOINT,
|
||||
help=('Keystone endpoint for authentication'))
|
||||
]
|
||||
|
||||
vol_crypt_opt_group = cfg.OptGroup(name=VOL_CRYPT_GRP_NAME,
|
||||
title='Volume Encryption Options')
|
||||
|
||||
vol_crypt_opts = [
|
||||
cfg.StrOpt('action', default='isLuks',
|
||||
help=u._('One of: set, unset, isLuks, open, close, format,\
|
||||
status')),
|
||||
cfg.StrOpt('dev', default=None,
|
||||
help=u._('The target device.')),
|
||||
cfg.StrOpt('mn', default=None,
|
||||
help=u._('The managed name for the device.')),
|
||||
cfg.StrOpt('lf', default='license.json',
|
||||
help=u._('The key license file.')),
|
||||
# Direct keyfile input not supported at this time for security reasons.
|
||||
# cfg.StrOpt('kf', default=None,
|
||||
# help=u._('The key file.')),
|
||||
cfg.IntOpt('ks', default=256,
|
||||
help=u._('Limits the key size to the specified number of bytes.\
|
||||
')),
|
||||
cfg.StrOpt('ci', default='aes-cbc-essiv:sha256',
|
||||
help=u._('Cipher. The encryption algorithm.'))
|
||||
]
|
||||
|
||||
|
||||
def new_config():
|
||||
conf = cfg.ConfigOpts()
|
||||
log.register_options(conf)
|
||||
conf.register_cli_opts(vol_crypt_opts, group=vol_crypt_opt_group)
|
||||
return conf
|
||||
|
||||
|
||||
def parse_args(conf, args=None, usage=None, default_config_files=None):
|
||||
conf(args=args,
|
||||
project='marshal',
|
||||
prog='marshal',
|
||||
# version=marshal.version.__version__,
|
||||
version='0.1',
|
||||
usage=usage,
|
||||
default_config_files=["test_marshal.conf"])
|
||||
|
||||
CONF = new_config()
|
||||
CONF.register_group(opt_group)
|
||||
CONF.register_opts(openam, opt_group)
|
||||
formatter = log.logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - \
|
||||
%(message)s')
|
||||
log.set_defaults(formatter)
|
||||
parse_args(CONF)
|
||||
try:
|
||||
log.setup(CONF, 'marshal')
|
||||
except IOError as e:
|
||||
if (e[0] == errno.ENOENT):
|
||||
print "Could not access logfile! Continuing without logging..."
|
||||
LOG = log.getLogger(__name__)
|
||||
0
marshal_agent/tests/functional_tests/temp.txt
Normal file
0
marshal_agent/tests/functional_tests/temp.txt
Normal file
50
marshal_agent/tests/functional_tests/testKeyRunner.py
Normal file
50
marshal_agent/tests/functional_tests/testKeyRunner.py
Normal file
@ -0,0 +1,50 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Test class dedicated to testing the KeyRunner class end-to-end
|
||||
"""
|
||||
|
||||
import unittest
|
||||
from unittest import TestCase
|
||||
from marshal_agent.agent.keyRunner import KeyRunner
|
||||
# from marshal_agent.common import config
|
||||
from oslo_log import log as logging
|
||||
|
||||
# CONF = config.CONF
|
||||
# LOG = config.LOG
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TestKeyRunner(TestCase):
|
||||
|
||||
"""
|
||||
Functionality Test
|
||||
"""
|
||||
def test_success_without_mocking(self):
|
||||
LOG.info("TestKeyRunner test_success_without_mocking: initializing...")
|
||||
keyRunner = KeyRunner()
|
||||
response = keyRunner.get_key_binary()
|
||||
self.assertNotEqual(response, None, "TestKeyRunner test_success_without\
|
||||
_mocking: Failed!")
|
||||
if response is not None:
|
||||
LOG.info("TestKeyRunner test_success_without_mocking: Key Retrieved\
|
||||
!")
|
||||
print "binary key is: " + response
|
||||
with open('key.bin', 'wb') as fh:
|
||||
fh.write(response)
|
||||
LOG.info("TestKeyRunner test_success_without_mocking: finalizing...")
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
114
marshal_agent/tests/testKeyRunner.py
Normal file
114
marshal_agent/tests/testKeyRunner.py
Normal file
@ -0,0 +1,114 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Test class dedicated to testing the KeyRunner class in isolation / with mocks
|
||||
"""
|
||||
|
||||
import unittest
|
||||
from marshal_agent.common.exception import MarshalHTTPException
|
||||
import responses
|
||||
|
||||
# Stash the command-line args so the oslo config doesn't hoover
|
||||
# the unittest args
|
||||
import sys
|
||||
tempArgs = []
|
||||
for x in sys.argv:
|
||||
tempArgs.append(x)
|
||||
del sys.argv[1:]
|
||||
__builtins__['TESTING_MARSHAL'] = True
|
||||
from marshal_agent.agent.keyRunner import KeyRunner
|
||||
for x in tempArgs:
|
||||
sys.argv.append(x)
|
||||
|
||||
|
||||
class TestKeyRunner(unittest.TestCase):
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(TestKeyRunner, self).__init__(*args, **kwargs)
|
||||
|
||||
def setUp(self):
|
||||
super(TestKeyRunner, self).setUp()
|
||||
|
||||
"""
|
||||
Mocked / Isolation Tests
|
||||
"""
|
||||
"""
|
||||
Testing 200 with key received
|
||||
"""
|
||||
@responses.activate
|
||||
def test_success_with_mocking(self):
|
||||
keyRunner = KeyRunner()
|
||||
conf = keyRunner.conf
|
||||
KM_OPT_GRP_NAME = keyRunner.config.KM_OPT_GRP_NAME
|
||||
conf_opts = getattr(conf, KM_OPT_GRP_NAME)
|
||||
kms_base = conf_opts.kms_base
|
||||
kms_get_key_api = conf_opts.kms_get_key_api
|
||||
kms_key_id = conf_opts.kms_key_id
|
||||
URL = kms_base+kms_get_key_api+kms_key_id
|
||||
responses.add(responses.GET, URL,
|
||||
status=200,
|
||||
body='aVeryVerySecretKey',
|
||||
content_type="application/json")
|
||||
response = keyRunner.get_key_binary()
|
||||
self.assertNotEqual(response, None,
|
||||
"TestKeyRunner test_success_with_mocking: Failed!")
|
||||
if response is not None:
|
||||
assert response == 'aVeryVerySecretKey'
|
||||
|
||||
"""
|
||||
Testing 200 with no key received
|
||||
"""
|
||||
@responses.activate
|
||||
def test_partial_success_with_mocking(self):
|
||||
keyRunner = KeyRunner()
|
||||
conf = keyRunner.conf
|
||||
KM_OPT_GRP_NAME = keyRunner.config.KM_OPT_GRP_NAME
|
||||
conf_opts = getattr(conf, KM_OPT_GRP_NAME)
|
||||
kms_base = conf_opts.kms_base
|
||||
kms_get_key_api = conf_opts.kms_get_key_api
|
||||
kms_key_id = conf_opts.kms_key_id
|
||||
URL = kms_base+kms_get_key_api+kms_key_id
|
||||
responses.add(responses.GET, URL,
|
||||
status=200,
|
||||
body=None,
|
||||
content_type="application/json")
|
||||
response = keyRunner.get_key_binary()
|
||||
self.assertEqual(response, None, "TestKeyRunner \
|
||||
test_partial_success_with_mocking: Failed!")
|
||||
"""
|
||||
Testing 403
|
||||
"""
|
||||
@responses.activate
|
||||
def test_successful_failure_code_403_with_mocking(self):
|
||||
keyRunner = KeyRunner()
|
||||
conf = keyRunner.conf
|
||||
KM_OPT_GRP_NAME = keyRunner.config.KM_OPT_GRP_NAME
|
||||
conf_opts = getattr(conf, KM_OPT_GRP_NAME)
|
||||
kms_base = conf_opts.kms_base
|
||||
kms_get_key_api = conf_opts.kms_get_key_api
|
||||
kms_key_id = conf_opts.kms_key_id
|
||||
URL = kms_base+kms_get_key_api+kms_key_id
|
||||
responses.add(responses.GET, URL,
|
||||
status=403,
|
||||
body='aVeryVerySecretKey',
|
||||
content_type="application/json")
|
||||
with self.assertRaises(MarshalHTTPException) as cm:
|
||||
keyRunner.get_key_binary()
|
||||
self.assertEqual(cm.exception.status_code, 403, "TestKeyRunner\
|
||||
test_successful_failure_code_403_with_mocking:\
|
||||
Failed! Response Code="+str(cm.exception.status_code))
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
28
marshal_agent/tests/test_marshal.conf
Normal file
28
marshal_agent/tests/test_marshal.conf
Normal file
@ -0,0 +1,28 @@
|
||||
[DEFAULT]
|
||||
# Show more verbose log output (sets INFO log level output)
|
||||
verbose = True
|
||||
|
||||
# Show debugging output in logs (sets DEBUG log level output)
|
||||
debug = True
|
||||
|
||||
# log file location
|
||||
log_file = test_marshal.log
|
||||
|
||||
[KM-OPT]
|
||||
# Key Manager Options
|
||||
# kms_type currently supported options are: barbican and vault. defaults to barbican
|
||||
# kms_base and kms_get_ket_api are only used for non-Keystone Barbican API testing or for non-Barbican kms_type
|
||||
kms_type=barbican
|
||||
#barbican
|
||||
kms_base=http://a_test_base/
|
||||
kms_get_key_api=a_test_api/
|
||||
kms_key_id=a_key_id
|
||||
#vault
|
||||
#kms_type=vault
|
||||
#kms_base=http://173.39.225.119:80/v1
|
||||
#kms_get_key_api=/secret/project/name/apikey
|
||||
|
||||
[crypt]
|
||||
lf=/tmp/license.json
|
||||
ci=aes-cbc-essiv:sha256
|
||||
ks=256
|
||||
@ -1,6 +1,26 @@
|
||||
# The order of packages is significant, because pip processes them in the order
|
||||
# of appearance. Changing the order has an impact on the overall integration
|
||||
# process, which may cause wedges in the gate later.
|
||||
#cryptography>=0.4 # Apache-2.0
|
||||
#iso8601>=0.1.9
|
||||
#jsonschema>=2.0.0,<3.0.0
|
||||
#netaddr>=0.7.12
|
||||
|
||||
oslo.i18n>=1.5.0 # Apache-2.0
|
||||
oslo.log>=1.8.0 # Apache-2.0
|
||||
oslo.config>=2.3.0 # Apache-2.0
|
||||
oslo.serialization>=1.4.0 # Apache-2.0
|
||||
oslo.utils>=2.4.0,!=2.6.0 # Apache-2.0
|
||||
|
||||
Paste
|
||||
|
||||
pyOpenSSL>=0.14
|
||||
#keystonemiddleware>=1.0.0
|
||||
six>=1.9.0
|
||||
pep8==1.5.7
|
||||
#blist>=1.3.6
|
||||
requests>=2.5.2,!=2.8.0
|
||||
|
||||
|
||||
#json
|
||||
|
||||
pbr>=1.6
|
||||
Babel>=1.3
|
||||
|
||||
22
setup.cfg
22
setup.cfg
@ -1,6 +1,7 @@
|
||||
[metadata]
|
||||
name = marshal
|
||||
summary = OpenStack Boilerplate contains all the boilerplate you need to create an OpenStack package.
|
||||
version = 0.0.1
|
||||
summary = OpenStack Marshal
|
||||
description-file =
|
||||
README.rst
|
||||
author = OpenStack
|
||||
@ -21,7 +22,7 @@ classifier =
|
||||
|
||||
[files]
|
||||
packages =
|
||||
marshal
|
||||
marshal_agent
|
||||
|
||||
[build_sphinx]
|
||||
source-dir = doc/source
|
||||
@ -44,3 +45,20 @@ input_file = marshal/locale/marshal.pot
|
||||
keywords = _ gettext ngettext l_ lazy_gettext
|
||||
mapping_file = babel.cfg
|
||||
output_file = marshal/locale/marshal.pot
|
||||
|
||||
data_files =
|
||||
etc/marshal =
|
||||
etc/marshal/marshal.conf
|
||||
scripts =
|
||||
bin/marshal.sh
|
||||
|
||||
[global]
|
||||
setup-hooks =
|
||||
pbr.hooks.setup_hook
|
||||
|
||||
[entry_points]
|
||||
console_scripts =
|
||||
marshal = marshal_agent.agent.marshal:agent_main
|
||||
|
||||
[wheel]
|
||||
universal = 1
|
||||
|
||||
@ -13,3 +13,18 @@ oslotest>=1.10.0 # Apache-2.0
|
||||
testrepository>=0.0.18
|
||||
testscenarios>=0.4
|
||||
testtools>=1.4.0
|
||||
|
||||
mock>=1.2
|
||||
|
||||
|
||||
#testrepository>=0.0.18
|
||||
#testtools>=0.9.36,!=1.2.0
|
||||
#fixtures>=0.3.14
|
||||
#requests>=2.5.2,!=2.8.0
|
||||
#responses>=0.5.0
|
||||
|
||||
|
||||
# Documentation build requirements
|
||||
#sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3
|
||||
#oslosphinx>=2.2.0 # Apache-2.0
|
||||
|
||||
|
||||
23
tmp/license.json
Normal file
23
tmp/license.json
Normal file
@ -0,0 +1,23 @@
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"http://[HOST]/v3/auth/tokens"
|
||||
},
|
||||
"project": {
|
||||
"id":"f383613fbcd74d6f8f9d4a40721ef811",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"user",
|
||||
"user": {
|
||||
"id":"4c49397e2d9f41e392498b8079c65343",
|
||||
"password":"changeit"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"e2ccc708-7c8d-437d-aaac-12bad476dd25"
|
||||
}
|
||||
}
|
||||
}
|
||||
12
tmp/license_vault.json
Normal file
12
tmp/license_vault.json
Normal file
@ -0,0 +1,12 @@
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"token":"e391ee37-b8fb-69c0-e3fd-7d485b2b516a"
|
||||
},
|
||||
"endpoint": {
|
||||
"kms_base":"http://173.39.225.119:80/v1",
|
||||
"kms_get_key_api":"/secret/project/name/apikey"
|
||||
}
|
||||
}
|
||||
}
|
||||
15
tox.ini
15
tox.ini
@ -1,18 +1,23 @@
|
||||
[tox]
|
||||
minversion = 1.6
|
||||
envlist = py34,py27,pypy,pep8
|
||||
envlist = py27,pep8
|
||||
skipsdist = True
|
||||
|
||||
[testenv]
|
||||
usedevelop = True
|
||||
install_command = pip install -U {opts} {packages}
|
||||
install_command =
|
||||
pip install -U {opts} {packages}
|
||||
setenv =
|
||||
VIRTUAL_ENV={envdir}
|
||||
deps = -r{toxinidir}/test-requirements.txt
|
||||
commands = python setup.py test --slowest --testr-args='{posargs}'
|
||||
deps =
|
||||
-r{toxinidir}/test-requirements.txt
|
||||
blist
|
||||
responses
|
||||
commands =
|
||||
python setup.py test --slowest --testr-args='{posargs}'
|
||||
|
||||
[testenv:pep8]
|
||||
commands = flake8
|
||||
commands = pep8 marshal_agent/.
|
||||
|
||||
[testenv:venv]
|
||||
commands = {posargs}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user