Initial commit of Marshal code base
Change-Id: Icc44c918bb3558ca7b2d0e7d17f1dc55f5b39972
This commit is contained in:
Dave McCowan
parent
b2aad62b28
commit
0f79632f94
126
README.md
Normal file
126
README.md
Normal file
@ -0,0 +1,126 @@
|
||||
## Marshal
|
||||
### Overview
|
||||
|
||||
* Marshal is an agent service running inside virtual machines, which will be responsible for securely fetching encryption keys from ia KMS like Barbican.
|
||||
* This agent will be interfacing with the disk encryption subsystem of the underlying operating system to encrypt/decrypt the disk I/O.
|
||||
* In the case of Linux-based virtual machines this agent will be interfacing with dm-crypt and for Windows OS it will be interfacing with Bit-locker.
|
||||
* The agent provides an abstraction service and can be integrated with other encryption subsystem as required.
|
||||
* When the agent reads a key from the KMS, the key is only stored briefly in a secure temporary file until it can be transferred to the disk encryption subsystem.
|
||||
|
||||
|
||||
**Table of Contents**
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Features](#features)
|
||||
- [Architecture](#architecture)
|
||||
- [Getting Started](#getting-started)
|
||||
- [Software Requirements](#software-requirements)
|
||||
- [Deployment Procedure](#deployment-procedure)
|
||||
- [Documentation](#documentation)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Core Components and Features](#core-components-and-features)
|
||||
- [Security](#security)
|
||||
- [Operations](#operations)
|
||||
- [Platform Support](#platform-support)
|
||||
- [Development](#development)
|
||||
- [License](#license)
|
||||
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* Disk encryption subsystem abstraction allowing for a consistent interface
|
||||
* KMS system abstraction allowing for a consistent interface
|
||||
* Encryption at various levels including full disk encryption, partition encryption including root partition
|
||||
|
||||
|
||||
### Architecture
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
|
||||
### Getting Started
|
||||
#### Deployment
|
||||
#####For production purposes, Marshal is intended to be deployed as a Debian Package embedded into OpenStack VMs
|
||||
###### Deploying Using Debian Package
|
||||
[Building and testing debian package](docs/debian-package-building.md)
|
||||
|
||||
##### For test purposes, Marshal can be cloned using normal Git semantics:
|
||||
|
||||
#### Clone to local repository:
|
||||
|
||||
#####Via SSH:
|
||||
```$ git clone git@github.com:CiscoCloud/marshal.git ```
|
||||
|
||||
#####Via HTTPS:
|
||||
```$ git clone https://github.com/CiscoCloud/marshal.git ```
|
||||
|
||||
|
||||
### Software Requirements
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
* Python 2.7.8
|
||||
* Cryptsetup (if Linux OS)
|
||||
|
||||
### Deployment Procedure
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### Please refer to the [Getting Started Guide](docs/Getting%20Started.md), which covers deployment, configuration, and example usage.
|
||||
|
||||
### Documentation
|
||||
|
||||
###### All documentation is located [here](docs)
|
||||
|
||||
|
||||
### Roadmap
|
||||
* KMS for infrastructure tenants
|
||||
* Volume encryption (With Marshal)
|
||||
* Certificate provisioning
|
||||
* Object Encryption
|
||||
* High key use tenants and IOT
|
||||
* KMaaS
|
||||
|
||||
|
||||
### Core Components and Features
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### List core components and features here
|
||||
- [x] Orchestration
|
||||
|
||||
|
||||
|
||||
### Security
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### List the security services it provides
|
||||
- [x] Encryption
|
||||
|
||||
|
||||
|
||||
### Operations
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### Disk encryption
|
||||
###### Automatic key retreival from a KMS
|
||||
|
||||
|
||||
|
||||
### Platform Support
|
||||
-----------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
###### Currently, only the Linux platform is supported using dm_crypt. Support Windows using bitlocker currently in the planning stages.
|
||||
###### Currently, only the OpenStack Barbican KMS is supported. Support for other KMSs is currently in the planning stages.
|
||||
###### Currently, only cloud-based KMSs are supported. Support for local KMSs is currently in the planning stages.
|
||||
|
||||
|
||||
### Development
|
||||
###### Write about the details of how anyone can contribute to the project.
|
||||
|
||||
|
||||
### Getting Support
|
||||
###### Write about the support details of the project.In case of any issue how anyone can get the support.
|
||||
|
||||
|
||||
### License
|
||||
###### Write about the license details of the project.
|
||||
1
debian/compat
vendored
Normal file
1
debian/compat
vendored
Normal file
@ -0,0 +1 @@
|
||||
9
|
||||
27
debian/control
vendored
Normal file
27
debian/control
vendored
Normal file
@ -0,0 +1,27 @@
|
||||
Source: marshal
|
||||
Section: misc
|
||||
Priority: optional
|
||||
Maintainer: Marshal Team <arvtiwar@cisco.com>
|
||||
Build-Depends:
|
||||
debhelper (>= 9~),
|
||||
python-all (>= 2.6),
|
||||
Build-Depends-Indep:
|
||||
python-pbr (>= 0.6),
|
||||
python-setuptools,
|
||||
XS-Python-Version: >= 2.6
|
||||
Standards-Version: 3.9.6
|
||||
Vcs-Browser: https://github.com/CiscoCloud/marshal
|
||||
Vcs-Git: git://github.com/CiscoCloud/marshal.git
|
||||
|
||||
Package: marshal
|
||||
Architecture: all
|
||||
Depends:
|
||||
python-six,
|
||||
python-blist,
|
||||
python-paste,
|
||||
python-openssl,
|
||||
python-requests,
|
||||
${misc:Depends},
|
||||
${python:Depends},
|
||||
Description: CiscoCloud marshal
|
||||
CiscoCloud marshal
|
||||
34
debian/copyright
vendored
Normal file
34
debian/copyright
vendored
Normal file
@ -0,0 +1,34 @@
|
||||
Format: http://dep.debian.net/deps/dep5
|
||||
Upstream-Name: marshal
|
||||
Source: https://github.com/CiscoCloud/marshal
|
||||
|
||||
Files: *
|
||||
Copyright: 2015 Cisco Systems
|
||||
License: Apache-2
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
.
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
.
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
.
|
||||
On Debian-based systems the full text of the Apache version 2.0 license
|
||||
can be found in `/usr/share/common-licenses/Apache-2.0'
|
||||
|
||||
Files: tools/rfc.sh
|
||||
Copyright: Copyright (c) 2010-2011 Gluster, Inc
|
||||
License: GPL-v3
|
||||
GNU GENERAL PUBLIC LICENSE
|
||||
Version 3, 29 June 2007
|
||||
.
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
.
|
||||
On Debian-based systems the full text of the Apache version 2.0 license
|
||||
can be found in `/usr/share/common-licenses/GPL-3'
|
||||
1
debian/marshal.dirs
vendored
Normal file
1
debian/marshal.dirs
vendored
Normal file
@ -0,0 +1 @@
|
||||
/var/log/marshal
|
||||
1
debian/marshal.install
vendored
Normal file
1
debian/marshal.install
vendored
Normal file
@ -0,0 +1 @@
|
||||
etc/marshal/marshal.conf etc/marshal
|
||||
5
debian/pydist-overrides
vendored
Normal file
5
debian/pydist-overrides
vendored
Normal file
@ -0,0 +1,5 @@
|
||||
oslo.config python-oslo-config
|
||||
oslo.serialization python-oslo-serialization
|
||||
oslo.i18n python-oslo-i18n
|
||||
oslo.utils python-oslo-utils
|
||||
oslo.log python-oslo-log
|
||||
7
debian/rules
vendored
Executable file
7
debian/rules
vendored
Executable file
@ -0,0 +1,7 @@
|
||||
#!/usr/bin/make -f
|
||||
|
||||
# Uncomment this to turn on verbose mode.
|
||||
#export DH_VERBOSE=1
|
||||
|
||||
%:
|
||||
dh $@ --with python2
|
||||
1
debian/source/format
vendored
Normal file
1
debian/source/format
vendored
Normal file
@ -0,0 +1 @@
|
||||
3.0 (native)
|
||||
336
docs/Getting Started.md
Normal file
336
docs/Getting Started.md
Normal file
@ -0,0 +1,336 @@
|
||||
###Marshal Getting Started Guide
|
||||
|
||||
####Deploying Marshal
|
||||
Marshal is currently deployable for Debian-based Linux systems and a Debian package is available and has been tested in Ubuntu 14.04. The intention is to have Marshal pre-installed on certain images. However, in cases where Marshal is not pre-installed, it can be installed as a Debian package. The procedure to install the Debian package is given here:
|
||||
|
||||
######ToDo: get clean Debian installation procedure
|
||||
|
||||
To check if Marshal is installed, run the following command:
|
||||
|
||||
```
|
||||
sudo marshal -h
|
||||
```
|
||||
|
||||
There are several ways to configure Marshal. The primary way to configure Marshal defaults is in the /etc/marshal/marshal.conf file:
|
||||
|
||||
```
|
||||
[DEFAULT]
|
||||
# Show more verbose log output (sets INFO log level output)
|
||||
# verbose = True
|
||||
|
||||
# Show debugging output in logs (sets DEBUG log level output)
|
||||
# debug = True
|
||||
|
||||
# log file location
|
||||
log_file = /var/log/marshal/marshal.log
|
||||
|
||||
[KM-OPT]
|
||||
# This section intended for dev/test purposes only
|
||||
# Default Auth Endpoint - for dev/test purposes only
|
||||
keystone_endpoint=http://173.39.224.159:35357/v3/auth/tokens
|
||||
# Default KMS fields - for dev/test purposes only
|
||||
kms_base=[HOST]
|
||||
kms_get_key_api=[some API]
|
||||
kms_key_id=[some key id]
|
||||
kms_project_id=[some project id]
|
||||
|
||||
[crypt]
|
||||
#This section for
|
||||
lf=/tmp/license.json
|
||||
ci=aes-cbc-essiv:sha256
|
||||
ks=256
|
||||
|
||||
```
|
||||
|
||||
More sensitive configuration details are configured in a JSON-formatted license file. The license file can come in 3 different flavors depending on the type of credentials to be used by the Marshal agent to authenticate. The 3 types are as follows:
|
||||
|
||||
* user-based
|
||||
* certififcate
|
||||
* trust
|
||||
|
||||
Example license configuration files for each of these 3 types are given here:
|
||||
|
||||
user-based license:
|
||||
```
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"http://[KEYSTONE_HOST]/v3/auth/tokens"
|
||||
},
|
||||
"project": {
|
||||
"id":"f383613fbcd74d6f8f9d4a40721ef811",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"user",
|
||||
"user": {
|
||||
"id":"4c49397e2d9f41e392498b8079c65343",
|
||||
"password":"changeit"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"e2ccc708-7c8d-437d-aaac-12bad476dd25"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
certificate-based license:
|
||||
```
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"http://[KEYSTONE_HOST]/v3/auth/tokens"
|
||||
},
|
||||
"project": {
|
||||
"id":"12345",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"cert",
|
||||
"cert": {
|
||||
"subject":"some_subject"
|
||||
"signature":"some_signature"
|
||||
"pub_key":"some_key"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"1b13ffdc-1b79-40ce-b94e-f4a2f9253d91"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
trust-based license:
|
||||
```
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"http://[KEYSTONE_HOST]/v3/auth/tokens"
|
||||
},
|
||||
"project": {
|
||||
"id":"12345",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"trust",
|
||||
"trust": {
|
||||
"id":"4c49397e2d9f41e392498b8079c65343"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"1b13ffdc-1b79-40ce-b94e-f4a2f9253d91"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The license path+file can be specified in the Marshal configuration file, or passed as a parameter with the '--crypt-lf' switch. Eg:
|
||||
|
||||
```
|
||||
sudo marshal --crypt-action open --crypt-dev /dev/vdc2 --crypt-mn priv_part --crypt-lf /tmp/license.json
|
||||
```
|
||||
It is recommended that the license file be placed in the /tmp folder or otherwise be disposed of once the desired encryption state is achieved.
|
||||
|
||||
####Understanding Marshal
|
||||
#####Authentication and Key retrieval
|
||||
######General Behavior
|
||||
Using the Keystone endpoint given by the configuration license:identity:endpoint, Marshal will attempt to authenticate using the configuration license:credentials:type and associated credential details.
|
||||
|
||||
######Barbican-specifc behavior
|
||||
Upon successfully authenticating, Marshal will receive an OpenStack token which should provide a Barbican Key Management Store (KMS) endpoint. Marshal will then attempt to retrieve the binary key associated with the key id given in the configuration
|
||||
|
||||
######Other-KMS bevahior
|
||||
Other KMS systems (non-Barbican) are not currently supported, but are expected to be in the near future.
|
||||
|
||||
#####Encryption operations
|
||||
Once Marshal has successfully retrieved the key, an encryption operation can commence against that key. Currently, only volume and volume partition encryption operations are supported, but other operations are anitipated in the future possibly including encrypted communications. The Marshal 'set' action will open an encrypted volume (if it can find the key), and it will format the target using LUKS formatting, if needed. The Marshal 'unset' command will close the target.
|
||||
|
||||
####Example Marshal operations:
|
||||
#####Formatting a block device:
|
||||
```
|
||||
sudo marshal --crypt-action format --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
|
||||
Example:
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
vdc 253:32 0 2G 0 disk
|
||||
|
||||
sudo marshal --crypt-action format --crypt-dev /dev/vdb --crypt-mn backup
|
||||
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action format --crypt-dev /dev/vdb --crypt-mn backup
|
||||
Device /dev/vdb is not a valid LUKS device.
|
||||
Command failed with code 22: Device /dev/vdb is not a valid LUKS device.
|
||||
Could not establish /dev/vdb as a valid LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully formatted.
|
||||
|
||||
```
|
||||
|
||||
#####Opening a block device:
|
||||
```
|
||||
sudo marshal --crypt-action open --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action open --crypt-dev /dev/vdb --crypt-mn backup
|
||||
/dev/vdb is a LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully opened.
|
||||
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
└─backup (dm-0) 252:0 0 2G 0 crypt
|
||||
vdc 253:32 0 2G 0 disk
|
||||
|
||||
```
|
||||
|
||||
#####Unsetting/Closing a block device:
|
||||
Unset=close. Either command can be used to achieve the same result.
|
||||
```
|
||||
sudo marshal --crypt-action unset --crypt-dev <device> --crypt-mn <managed name>
|
||||
sudo marshal --crypt-action close --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action close --crypt-dev /dev/vdb --crypt-mn backup
|
||||
/dev/vdb is a LUKS device.
|
||||
The volume was successfully closed.
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
vdc 253:32 0 2G 0 disk
|
||||
|
||||
```
|
||||
|
||||
#####Setting a block device:
|
||||
To "set" the device means to open it as a LUKS volume if possible, and if not then LUKS format and then open the device.
|
||||
```
|
||||
sudo marshal --crypt-action set --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
Example requiring format
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action set --crypt-dev /dev/vdc --crypt-mn backup2
|
||||
Device /dev/vdc is not a valid LUKS device.
|
||||
Command failed with code 22: Device /dev/vdc is not a valid LUKS device.
|
||||
Could not establish /dev/vdc as a valid LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully formatted.
|
||||
The volume was successfully opened.
|
||||
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
└─backup (dm-0) 252:0 0 2G 0 crypt
|
||||
vdc 253:32 0 2G 0 disk
|
||||
└─backup2 (dm-1) 252:1 0 2G 0 crypt
|
||||
```
|
||||
|
||||
Example not requiring format
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action set --crypt-dev /dev/vdb --crypt-mn backup
|
||||
/dev/vdb is a LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully opened.
|
||||
cloud-user@marshal-test-8:~/marshal$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
└─backup (dm-0) 252:0 0 2G 0 crypt
|
||||
vdc 253:32 0 2G 0 disk
|
||||
└─backup2 (dm-1) 252:1 0 2G 0 crypt
|
||||
```
|
||||
|
||||
#####Statusing a block device:
|
||||
```
|
||||
sudo marshal --crypt-action status --crypt-dev <device> --crypt-mn <managed name>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action status --crypt-dev /dev/vdc --crypt-mn backup2
|
||||
/dev/vdc is a LUKS device.
|
||||
```
|
||||
|
||||
#####Statusing a block device with verbosity:
|
||||
Adding the -v flag enables INFO messages to appear at the CLI
|
||||
```
|
||||
sudo marshal --crypt-action status --crypt-dev <device> --crypt-mn <managed name> -v
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action status --crypt-dev /dev/vdc --crypt-mn backup2 -v
|
||||
2015-10-06 22:26:16.186 1669 INFO marshal_agent.common.config [-] status action requested.
|
||||
/dev/vdc is a LUKS device.
|
||||
2015-10-06 22:26:16.192 1669 INFO marshal_agent.common.config [-] Status output was: /dev/mapper/backup2 is active.
|
||||
type: LUKS1
|
||||
cipher: aes-cbc-essiv:sha256
|
||||
keysize: 256 bits
|
||||
device: /dev/vdc
|
||||
offset: 4096 sectors
|
||||
size: 4190208 sectors
|
||||
mode: read/write
|
||||
Command successful.
|
||||
```
|
||||
|
||||
#####Overriding the license file default as set in the configiration file:
|
||||
```
|
||||
sudo marshal --crypt-action <action> --crypt-dev <device> --crypt-mn <managed name> --crypt-lf <license file>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ mv /tmp/license.json .
|
||||
License file not found.
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action status --crypt-dev /dev/vdc --crypt-mn backup2
|
||||
License file not found.
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action status --crypt-dev /dev/vdc --crypt-mn backup2 --crypt-lf license.json
|
||||
/dev/vdc is a LUKS device.
|
||||
```
|
||||
#####Overriding the cipher default as set in the configiration file:
|
||||
```
|
||||
sudo marshal --crypt-action <action> --crypt-dev <device> --crypt-mn <managed name> --crypt-ci <cipher>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action format --crypt-dev /dev/vdc --crypt-mn backup2 --crypt-ci aes-xts-plain64
|
||||
/dev/vdc is a LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully formatted.
|
||||
```
|
||||
|
||||
#####Overriding the key size default as set ub the configuration file:
|
||||
```
|
||||
sudo marshal --crypt-action <action> --crypt-dev <device> --crypt-mn <managed name> --crypt-ks <key size>
|
||||
```
|
||||
Example
|
||||
```
|
||||
cloud-user@marshal-test-8:~/marshal$ sudo marshal --crypt-action format --crypt-dev /dev/vdc --crypt-mn backup2 --crypt-ci aes-xts-plain64 --crypt-ks 512
|
||||
/dev/vdc is a LUKS device.
|
||||
Attempting to fetch key from KMS...
|
||||
Key successfully retrieved from KMS...
|
||||
The volume was successfully formatted.
|
||||
|
||||
```
|
||||
35
docs/debian-package-building.md
Normal file
35
docs/debian-package-building.md
Normal file
@ -0,0 +1,35 @@
|
||||
**1. Adding official Openstack Kilo PPA repository.**
|
||||
It's needed because some marshal dependencies available only from it.
|
||||
```
|
||||
sudo apt-get install ubuntu-cloud-keyring
|
||||
echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu" \
|
||||
"trusty-updates/kilo main" \
|
||||
| sudo tee /etc/apt/sources.list.d/cloudarchive-kilo.list
|
||||
```
|
||||
**2. Update index of system packages.**
|
||||
```
|
||||
sudo apt-get update
|
||||
```
|
||||
**3. Install build tools and Marshal dependencies**
|
||||
```
|
||||
sudo apt-get install build-essential debhelper fakeroot git python-setuptools python-pbr python-all
|
||||
```
|
||||
**4. Clone fresh Marshal repo**
|
||||
```
|
||||
git clone https://github.com/CiscoCloud/marshal
|
||||
cd marshal
|
||||
```
|
||||
**5. Build deb package**
|
||||
```
|
||||
dpkg-buildpackage -us -uc
|
||||
```
|
||||
**6. Install package to the target system**
|
||||
```
|
||||
sudo dpkg -i ../marshal_*_all.deb # Errors on this step are normal, next step fixes them.
|
||||
sudo apt-get -f install # (Optional) This installs broken marshal dependencies.
|
||||
```
|
||||
**7. And try to use it**
|
||||
```
|
||||
sudo marshal --help
|
||||
sudo marshal.sh -h
|
||||
```
|
||||
93
docs/disk_commands
Normal file
93
docs/disk_commands
Normal file
@ -0,0 +1,93 @@
|
||||
http://jootamam.net/howto-basic-cryptsetup.htm
|
||||
|
||||
http://www.finnie.org/2009/07/26/keyfile-based-luks-encryption-in-debian/
|
||||
|
||||
http://sleepyhead.de/howto/?href=cryptpart
|
||||
|
||||
|
||||
sudo cryptsetup isLuks /dev/vdb
|
||||
sudo cryptsetup -y luksFormat /dev/vdb --key-file /tmp/mykey.txt
|
||||
sudo cryptsetup --key-file /tmp/mykey.txt luksOpen /dev/vdb backup
|
||||
|
||||
|
||||
ubuntu@dm-crypt:~$ lsblk -o KNAME,TYPE,SIZE,MODEL
|
||||
KNAME TYPE SIZE MODEL
|
||||
vda disk 30G
|
||||
vda1 part 30G
|
||||
vdb disk 4G
|
||||
ubuntu@dm-crypt:~$ lsblk -d -n -oNAME,RO | grep '0$' | awk {'print $1'}
|
||||
vda
|
||||
vdb
|
||||
|
||||
|
||||
|
||||
http://www.bogotobogo.com/DevOps/AWS/aws_attaching_Amazon_EBS_volume_to_instance.php
|
||||
ubuntu@dm-crypt:~$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 30G 0 disk
|
||||
└─vda1 253:1 0 30G 0 part /
|
||||
vdb 253:16 0 4G 0 disk
|
||||
ubuntu@dm-crypt:~$ sudo file -s /dev/vdb
|
||||
sudo: unable to resolve host dm-crypt
|
||||
/dev/vdb: data
|
||||
ubuntu@dm-crypt:~$ sudo file -s /dev/vda1
|
||||
sudo: unable to resolve host dm-crypt
|
||||
/dev/vda1: Linux rev 1.0 ext4 filesystem data, UUID=50879377-446a-412a-99bf-e0d42a78c1b3, volume name "cloudimg-rootfs" (needs journal recovery) (extents) (large files) (huge files)
|
||||
ubuntu@dm-crypt:~$ sudo mkfs -t ext4 /dev/vdb
|
||||
sudo: unable to resolve host dm-crypt
|
||||
mke2fs 1.42.9 (4-Feb-2014)
|
||||
Filesystem label=
|
||||
OS type: Linux
|
||||
Block size=4096 (log=2)
|
||||
Fragment size=4096 (log=2)
|
||||
Stride=0 blocks, Stripe width=0 blocks
|
||||
262144 inodes, 1048576 blocks
|
||||
52428 blocks (5.00%) reserved for the super user
|
||||
First data block=0
|
||||
Maximum filesystem blocks=1073741824
|
||||
32 block groups
|
||||
32768 blocks per group, 32768 fragments per group
|
||||
8192 inodes per group
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376, 294912, 819200, 884736
|
||||
|
||||
Allocating group tables: done
|
||||
Writing inode tables: done
|
||||
Creating journal (32768 blocks): done
|
||||
Writing superblocks and filesystem accounting information: done
|
||||
|
||||
ubuntu@dm-crypt:~$ sudo file -s /dev/vdb
|
||||
sudo: unable to resolve host dm-crypt
|
||||
/dev/vdb: Linux rev 1.0 ext4 filesystem data, UUID=a6e166a8-7b98-4f61-8318-6ba4b878bfee (extents) (large files) (huge files)
|
||||
ubuntu@dm-crypt:~$ sudo mkdir /mydisk
|
||||
sudo: unable to resolve host dm-crypt
|
||||
ubuntu@dm-crypt:~$ sudo mount /dev/vd /mydisk
|
||||
vda vda1 vdb
|
||||
ubuntu@dm-crypt:~$ sudo mount /dev/vdb /mydisk
|
||||
sudo: unable to resolve host dm-crypt
|
||||
ubuntu@dm-crypt:~$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 30G 0 disk
|
||||
└─vda1 253:1 0 30G 0 part /
|
||||
vdb 253:16 0 4G 0 disk /mydisk7379716029
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
http://unix.stackexchange.com/questions/52078/how-to-mount-a-cryptsetup-container-just-with-mount
|
||||
|
||||
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [[ $(mount | grep ${2%%/} | wc -l) -gt 0 ]]; then
|
||||
echo "Path $2 is already mounted!" >&2
|
||||
exit 9
|
||||
else
|
||||
MAPPER=$(mktemp -up /dev/mapper)
|
||||
cryptsetup luksOpen $1 $(basename $MAPPER)
|
||||
shift
|
||||
mount $MAPPER $* || cryptsetup luksClose $(basename $MAPPER)
|
||||
fi
|
||||
|
||||
|
||||
BIN
docs/images/marshal_within_openstack.png
Executable file
BIN
docs/images/marshal_within_openstack.png
Executable file
Binary file not shown.
|
After Width: | Height: | Size: 92 KiB |
137
docs/marshal-demo.md
Normal file
137
docs/marshal-demo.md
Normal file
@ -0,0 +1,137 @@
|
||||
1.Create symmetric keys using Barbican Order API.
|
||||
|
||||
2.Inject license file into VM using a cloud-config script:
|
||||
|
||||
```
|
||||
#cloud-config
|
||||
write_files:
|
||||
- content: |
|
||||
{
|
||||
"license":
|
||||
{
|
||||
"identity": {
|
||||
"version":"v3",
|
||||
"endpoint":"[KMS endpoint url]"
|
||||
},
|
||||
"project": {
|
||||
"id":"f383613fbcd74d6f8f9d4a40721ef811",
|
||||
"name":"marshal-demo"
|
||||
},
|
||||
"credentials": {
|
||||
"type":"user",
|
||||
"user": {
|
||||
"id":"[user_id]",
|
||||
"password":"[user_password]"
|
||||
}
|
||||
},
|
||||
"key": {
|
||||
"id":"[key id]"
|
||||
}
|
||||
}
|
||||
}
|
||||
owner: root:root
|
||||
path: /tmp/license.json
|
||||
permissions: '0644'
|
||||
|
||||
```
|
||||
3.Create volume and attach to vm using Horizon
|
||||
|
||||
4.Look for the attached disk
|
||||
|
||||
```
|
||||
lsblk
|
||||
|
||||
lsblk -d -n -oNAME,RO | grep '0$' | awk {'print $1'}
|
||||
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
```
|
||||
|
||||
5.Set v-disk with marshal and dm-crypt
|
||||
|
||||
```
|
||||
sudo marshal.sh set -d /dev/vdb -m vdb1 -l /tmp/license.json
|
||||
```
|
||||
|
||||
7.Look for the attached disk
|
||||
|
||||
```
|
||||
lsblk
|
||||
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
vda 253:0 0 50G 0 disk
|
||||
└─vda1 253:1 0 50G 0 part /
|
||||
vdb 253:16 0 2G 0 disk
|
||||
└─vdb1 (dm-0) 252:0 0 2G 0 crypt
|
||||
```
|
||||
|
||||
8.Check status
|
||||
|
||||
```
|
||||
sudo marshal.sh status -d /dev/vdb -m vdb1 -v
|
||||
|
||||
/dev/mapper/vdb1 is active.
|
||||
type: LUKS1
|
||||
cipher: aes-cbc-essiv:sha256
|
||||
keysize: 256 bits
|
||||
device: /dev/vdb
|
||||
offset: 4096 sectors
|
||||
size: 4190208 sectors
|
||||
mode: read/write
|
||||
Command successful.
|
||||
```
|
||||
|
||||
9.create file system before mount
|
||||
|
||||
```
|
||||
sudo mkfs.ext4 /dev/mapper/vdb1
|
||||
```
|
||||
|
||||
10.Create mount point and mount the device point
|
||||
|
||||
```
|
||||
sudo mkdir /cryptdisk1
|
||||
sudo mount /dev/mapper/vdb1 /cryptdisk1
|
||||
```
|
||||
|
||||
11.Verify file system on device
|
||||
|
||||
```
|
||||
sudo df -PTH /dev/mapper/vdb1
|
||||
|
||||
or
|
||||
sudo df -PTH /dev/mapper/vdb1 | awk '{print $2}' | sed -n '1!p'
|
||||
```
|
||||
|
||||
12.Write something
|
||||
|
||||
```
|
||||
cd /cryptdisk1
|
||||
sudo vi crypttest.txt
|
||||
```
|
||||
|
||||
13.Search content
|
||||
|
||||
```
|
||||
sudo grep "super" /cryptdisk1/*
|
||||
```
|
||||
|
||||
14.Unmount disk
|
||||
|
||||
```
|
||||
sudo umount /cryptdisk1
|
||||
```
|
||||
|
||||
15.Unset disk
|
||||
|
||||
```
|
||||
sudo ./marshal.sh unset -d /dev/vdb -m vdb1
|
||||
```
|
||||
16.Search content on device
|
||||
|
||||
```
|
||||
sudo grep "super" /dev/vdb
|
||||
```
|
||||
|
||||
27
etc/marshal/marshal.conf
Normal file
27
etc/marshal/marshal.conf
Normal file
@ -0,0 +1,27 @@
|
||||
[DEFAULT]
|
||||
# Show more verbose log output (sets INFO log level output)
|
||||
# verbose = True
|
||||
|
||||
# Show debugging output in logs (sets DEBUG log level output)
|
||||
# debug = True
|
||||
|
||||
# log file location
|
||||
log_file = /var/log/marshal/marshal.log
|
||||
|
||||
[KM-OPT]
|
||||
# Key Manager Options
|
||||
# kms_type currently supported options are: barbican and vault. defaults to barbican
|
||||
# kms_base and kms_get_ket_api are only used for non-Keystone Barbican API testing or for non-Barbican kms_type
|
||||
#kms_type=barbican
|
||||
kms_type=vault
|
||||
#barbican
|
||||
#kms_base=http://173.39.229.98:9311/v1
|
||||
#kms_get_key_api=/secrets/
|
||||
#vault
|
||||
#kms_base=http://173.39.225.119:80/v1
|
||||
kms_get_key_api=/secret/project/name/apikey
|
||||
|
||||
[crypt]
|
||||
lf=/tmp/license.json
|
||||
ci=aes-cbc-essiv:sha256
|
||||
ks=256
|
||||
@ -1,19 +0,0 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import pbr.version
|
||||
|
||||
|
||||
__version__ = pbr.version.VersionInfo(
|
||||
'marshal').version_string()
|
||||
0
marshal_agent/agent/__init__.py
Normal file
0
marshal_agent/agent/__init__.py
Normal file
119
marshal_agent/agent/auth.py
Normal file
119
marshal_agent/agent/auth.py
Normal file
@ -0,0 +1,119 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Class to handle Marshal authentication against Keystone
|
||||
"""
|
||||
|
||||
import requests
|
||||
import json
|
||||
import abc
|
||||
import six
|
||||
|
||||
from marshal_agent.i18n import _
|
||||
|
||||
from marshal_agent.common import config
|
||||
from marshal_agent.common import exception
|
||||
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
@six.add_metaclass(abc.ABCMeta)
|
||||
class AuthBase(object):
|
||||
@abc.abstractmethod
|
||||
def get_token_wrapper(self, key_id, project_id):
|
||||
raise NotImplementedError
|
||||
|
||||
def get_key_wrapper(self, key_id, project_id):
|
||||
raise NotImplementedError
|
||||
|
||||
def get_key_binary(self):
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class Auth(AuthBase):
|
||||
|
||||
def __init__(self, conf=CONF, lic=None, kms_type=None):
|
||||
self.conf = conf
|
||||
self.lic = lic
|
||||
self.kms_type = kms_type
|
||||
|
||||
def get_token_wrapper(self):
|
||||
return self._get_token_from_keystone()
|
||||
|
||||
def get_token(self):
|
||||
if self.kms_type == 'vault':
|
||||
token = self.lic.x_vault_token
|
||||
return token, None
|
||||
else:
|
||||
return self._get_token_from_keystone()
|
||||
|
||||
def get_key_wrapper(self):
|
||||
return self._get_key_from_kms()
|
||||
|
||||
def _get_token_from_keystone(self):
|
||||
""" Get token from Keystone"""
|
||||
token = None
|
||||
kms_endpoint = None
|
||||
|
||||
payload = {
|
||||
"auth": {
|
||||
"identity": {
|
||||
"methods": [
|
||||
"password"
|
||||
],
|
||||
"password": {
|
||||
"user": {
|
||||
"id": self.lic.user_id,
|
||||
"password": self.lic.user_pass
|
||||
}
|
||||
}
|
||||
},
|
||||
"scope": {
|
||||
"project": {
|
||||
"id": self.lic.project_id,
|
||||
"domain": {
|
||||
"id": "default"
|
||||
},
|
||||
"name": self.lic.project_name
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
self.json_data = json.dumps(payload)
|
||||
hdrs = {
|
||||
'Accept': 'application/json',
|
||||
'Content-Type': 'application/json; charset=UTF-8'
|
||||
}
|
||||
pr = requests.post(self.lic.keystone_endpoint,
|
||||
data=json.dumps(payload), headers=hdrs)
|
||||
if pr.status_code != 201:
|
||||
log_msg = _('Unable to get identity from Keystone. Response Code\
|
||||
was: '+str(pr.status_code))
|
||||
client_msg = _('Marshal was unable to authenticate.')
|
||||
raise exception.MarshalHTTPException(log_msg, client_msg,
|
||||
pr.status_code)
|
||||
else:
|
||||
LOG.debug("Successfully authenticated against Keystone.")
|
||||
token = pr.headers['X-Subject-Token']
|
||||
pr_j = json.loads(pr.content)
|
||||
catalog = pr_j['token']['catalog']
|
||||
for endpoint in catalog:
|
||||
if endpoint.get('type') == 'kms':
|
||||
kms_endpoint = endpoint
|
||||
break
|
||||
|
||||
return token, kms_endpoint
|
||||
165
marshal_agent/agent/keyRunner.py
Normal file
165
marshal_agent/agent/keyRunner.py
Normal file
@ -0,0 +1,165 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Class to handle KMS key retrieval
|
||||
"""
|
||||
|
||||
import requests
|
||||
import json
|
||||
import abc
|
||||
import six
|
||||
|
||||
from marshal_agent.i18n import _
|
||||
from marshal_agent.common import config
|
||||
from marshal_agent.common import exception
|
||||
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
@six.add_metaclass(abc.ABCMeta)
|
||||
class KeyAgentBase(object):
|
||||
@abc.abstractmethod
|
||||
def get_key_wrapper(self, key_id, project_id):
|
||||
raise NotImplementedError
|
||||
|
||||
def get_key_binary(self):
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class KeyRunner(KeyAgentBase):
|
||||
|
||||
def __init__(self, lic=None, token=None, endpoint=None, conf=CONF,
|
||||
key_id=None, project_id=None):
|
||||
|
||||
'''
|
||||
if conf is not None:
|
||||
CONF = conf;
|
||||
else:
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
self.LOG = LOG
|
||||
# LOG = CONF.LOG
|
||||
# self.conf = conf
|
||||
# self.conf = conf
|
||||
#CONF = conf.CONF
|
||||
'''
|
||||
self.config = config
|
||||
self.conf = CONF
|
||||
KM_OPT_GRP_NAME = config.KM_OPT_GRP_NAME
|
||||
conf_opts = getattr(CONF, KM_OPT_GRP_NAME)
|
||||
self.kms_type = conf_opts.kms_type.lower()
|
||||
|
||||
if self.kms_type is None or self.kms_type == 'barbican':
|
||||
# For Barbican, we normally get endpoint from Keystone, but this
|
||||
# allows bypass of keystone for testing Barbican interface:
|
||||
if lic is None:
|
||||
if key_id is None or project_id is None:
|
||||
self.key_id = conf_opts.kms_key_id
|
||||
self.project_id = conf_opts.kms_project_id
|
||||
else:
|
||||
self.key_id = key_id
|
||||
self.project_id = project_id
|
||||
else:
|
||||
self.project_id = lic.project_id
|
||||
self.key_id = lic.key_id
|
||||
if endpoint is None:
|
||||
# Allows manually configured Barbican or other KMS:
|
||||
self.kms_endpoint = conf_opts.kms_base+conf_opts.\
|
||||
kms_get_key_api
|
||||
LOG.debug('Using Barbican endpoint from CONF')
|
||||
else:
|
||||
self.kms_endpoint = endpoint.get('endpoints')[0].\
|
||||
get('url')+'/secrets/'
|
||||
LOG.debug('Using Barbican endpoint from Keystone')
|
||||
elif self.kms_type == 'vault':
|
||||
# For vault we want to be able to provide kms endpoint details in
|
||||
# license file or conf file or, some mixture of the two.
|
||||
# Give license file the priority
|
||||
try:
|
||||
self.kms_base = lic.kms_base
|
||||
except AttributeError:
|
||||
if conf_opts.kms_base is not None:
|
||||
self.kms_base = conf_opts.kms_base
|
||||
try:
|
||||
self.kms_get_key_api = lic.kms_get_key_api
|
||||
except AttributeError:
|
||||
if conf_opts.kms_get_key_api is not None:
|
||||
self.kms_get_key_api = conf_opts.kms_get_key_api
|
||||
try:
|
||||
self.kms_endpoint = self.kms_base+self.kms_get_key_api
|
||||
except AttributeError:
|
||||
raise exception.MissingKMSConfigurationError
|
||||
|
||||
self.lic = lic
|
||||
self.token = token
|
||||
|
||||
def get_key_wrapper(self, key_id, project_id):
|
||||
return self._get_key_from_kms(key_id, project_id)
|
||||
|
||||
def get_key_binary(self):
|
||||
return self._get_key_from_kms(accept='application/octet-stream')
|
||||
|
||||
def _get_key_from_kms(self, accept=None):
|
||||
if self.kms_type is None or self.kms_type == 'barbican':
|
||||
if accept:
|
||||
headers = {
|
||||
'Accept': accept,
|
||||
'X-Project-Id': self.project_id
|
||||
}
|
||||
else:
|
||||
headers = {
|
||||
'Accept': 'application/json',
|
||||
'X-Project-Id': self.project_id
|
||||
}
|
||||
|
||||
if self.token is not None:
|
||||
headers['X-Auth-Token'] = self.token
|
||||
key_manager_url = self.kms_endpoint+format(self.key_id)
|
||||
elif self.kms_type == 'vault':
|
||||
if self.token is not None:
|
||||
headers = {
|
||||
'Accept': 'application/json'
|
||||
}
|
||||
headers['X-Vault-Token'] = self.token
|
||||
key_manager_url = self.kms_endpoint
|
||||
|
||||
LOG.debug('Calling KMS API at: %s', key_manager_url)
|
||||
|
||||
content = None
|
||||
|
||||
r = requests.get(key_manager_url, headers=headers)
|
||||
if r.status_code != 200:
|
||||
log_msg = _('Unable to get key from KMS. Response Code was: ' +
|
||||
str(r.status_code))
|
||||
client_msg = _('Unable to get key from KMS')
|
||||
raise exception.MarshalHTTPException(log_msg, client_msg,
|
||||
r.status_code)
|
||||
elif r.content is None or r.content == '' or r.content == 'None':
|
||||
LOG.info('KMS returned a blank key!')
|
||||
else:
|
||||
LOG.info('Successfully retrieved key from KMS.')
|
||||
content = r.content
|
||||
if self.kms_type is None or self.kms_type == 'barbican':
|
||||
key = content
|
||||
elif self.kms_type == 'vault':
|
||||
try:
|
||||
gr_j = json.loads(content)
|
||||
key = gr_j['data']['value']
|
||||
except (ValueError, KeyError, TypeError):
|
||||
msg = _('Unable to parse JSON response from Key Manager')
|
||||
raise exception.PayloadDecodingError(msg)
|
||||
|
||||
return key
|
||||
84
marshal_agent/agent/license.py
Normal file
84
marshal_agent/agent/license.py
Normal file
@ -0,0 +1,84 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Utility class to handle license data extraction and management
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
from marshal_agent.common import config
|
||||
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
class License(object):
|
||||
|
||||
def __init__(self, license_file, kms_type=None):
|
||||
self.keystone_endpoint = None
|
||||
self.project_id = None
|
||||
self.project_name = None
|
||||
self.key_id = None
|
||||
self.mode = None
|
||||
self.user_id = None
|
||||
self.user_pass = None
|
||||
self.kms_type = kms_type
|
||||
self.parse_license_file(license_file)
|
||||
|
||||
def parse_license_file(self, lf):
|
||||
try:
|
||||
with open(lf, 'r') as lf_h:
|
||||
l_data = (lf_h.read()).rstrip()
|
||||
try:
|
||||
jl_data = json.loads(l_data)
|
||||
except ValueError as e:
|
||||
LOG.debug("Unable to parse license file: %s", str(e))
|
||||
print 'Unable to parse license file: '+str(e)
|
||||
sys.exit(1)
|
||||
|
||||
if self.kms_type == 'vault':
|
||||
self.x_vault_token = \
|
||||
jl_data['license']['identity']['token']
|
||||
try:
|
||||
self.kms_base = \
|
||||
jl_data['license']['endpoint']['kms_base']
|
||||
except KeyError:
|
||||
pass
|
||||
try:
|
||||
self.kms_get_key_api = \
|
||||
jl_data['license']['endpoint']['kms_get_key_api']
|
||||
except KeyError:
|
||||
pass
|
||||
else:
|
||||
self.keystone_endpoint = \
|
||||
jl_data['license']['identity']['endpoint']
|
||||
self.project_id = jl_data['license']['project']['id']
|
||||
self.project_name = jl_data['license']['project']['name']
|
||||
self.key_id = jl_data['license']['key']['id']
|
||||
self.mode = jl_data['license']['credentials']['type']
|
||||
if self.mode == 'user':
|
||||
self.user_id = \
|
||||
jl_data['license']['credentials']['user']['id']
|
||||
self.user_pass = \
|
||||
jl_data['license']['credentials']['user']['password\
|
||||
']
|
||||
|
||||
except IOError:
|
||||
LOG.info("Unable to locate license file: %s", lf)
|
||||
print 'License file not found.'
|
||||
sys.exit(1)
|
||||
except KeyError as e:
|
||||
LOG.info("Unable to parse license file: %s", str(e))
|
||||
print 'Unable to parse license file.'
|
||||
sys.exit(1)
|
||||
210
marshal_agent/agent/marshal.py
Normal file
210
marshal_agent/agent/marshal.py
Normal file
@ -0,0 +1,210 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
The Marshal module is intended to facilitate flexible volume encryption with
|
||||
dynamic key management
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import errno
|
||||
import tempfile
|
||||
from stat import S_ISBLK
|
||||
from marshal_agent.agent.keyRunner import KeyRunner
|
||||
from marshal_agent.agent.license import License
|
||||
from marshal_agent.agent.auth import Auth
|
||||
from marshal_agent.agent.volCrypt import VolCrypt
|
||||
from marshal_agent.common import config
|
||||
from marshal_agent.common.exception import MarshalHTTPException
|
||||
from marshal_agent.common.exception import PayloadDecodingError
|
||||
from marshal_agent.common.exception import MissingKMSConfigurationError
|
||||
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
def check_block_device(device):
|
||||
try:
|
||||
mode = os.stat(device).st_mode
|
||||
except OSError as e:
|
||||
print 'The specified device is invalid.'
|
||||
if e[0] == errno.ENOENT:
|
||||
LOG.info('Device %s was specified, but the file could not be found.\
|
||||
', device)
|
||||
sys.exit(1)
|
||||
|
||||
if not S_ISBLK(mode):
|
||||
print 'The specified device is invalid.'
|
||||
LOG.info('Device was specified, but the file is not a valid block devic\
|
||||
e file.')
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def check_if_luks(vc, device, managed_name):
|
||||
LOG.debug('Performing isLuks check.')
|
||||
if device is None:
|
||||
print 'Please provide the device path.'
|
||||
LOG.info('isLuks check cancelled. No device specified.')
|
||||
sys.exit(1)
|
||||
|
||||
check_block_device(device)
|
||||
|
||||
response = vc.is_luks(device)
|
||||
if response:
|
||||
print '{} is a LUKS device.'.format(device)
|
||||
return True
|
||||
else:
|
||||
print 'Could not establish {} as a valid LUKS device.'.format(device)
|
||||
return False
|
||||
|
||||
|
||||
def missing_managed_name():
|
||||
LOG.info("Managed name is a required parameter for this operation.")
|
||||
print 'Please provide the managed name.'
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def agent_main():
|
||||
LOG.info('Starting Marshal...')
|
||||
KM_OPT_GRP_NAME = config.KM_OPT_GRP_NAME
|
||||
km_conf_opts = getattr(CONF, KM_OPT_GRP_NAME)
|
||||
kms_type = km_conf_opts.kms_type.lower()
|
||||
VOL_CRYPT_GRP_NAME = config.VOL_CRYPT_GRP_NAME
|
||||
vc_conf_opts = getattr(CONF, VOL_CRYPT_GRP_NAME)
|
||||
action = vc_conf_opts.action
|
||||
device = vc_conf_opts.dev
|
||||
managed_name = vc_conf_opts.mn
|
||||
license_file = vc_conf_opts.lf
|
||||
key_size = str(vc_conf_opts.ks)
|
||||
cipher = vc_conf_opts.ci
|
||||
|
||||
LOG.info('KMS type: %s', kms_type)
|
||||
lic = License(license_file, kms_type)
|
||||
vc = VolCrypt(device, managed_name)
|
||||
|
||||
action = action.lower()
|
||||
LOG.info('%s action requested.', action)
|
||||
if action == 'unset':
|
||||
action = 'close'
|
||||
if action == 'isluks':
|
||||
check_if_luks(vc, device, managed_name)
|
||||
sys.exit(1)
|
||||
|
||||
elif action == 'open' or action == 'format' or action == 'set':
|
||||
setNeedsFormat = False
|
||||
if not check_if_luks(vc, device, managed_name):
|
||||
if action == 'open':
|
||||
# For debugging, comment this exit and uncomment the pass
|
||||
sys.exit(1)
|
||||
# pass
|
||||
elif action == 'set':
|
||||
setNeedsFormat = True
|
||||
|
||||
auth = Auth(CONF, lic, kms_type)
|
||||
try:
|
||||
token, kms_endpoint = auth.get_token()
|
||||
except MarshalHTTPException:
|
||||
print 'Authentication failed.'
|
||||
LOG.info('Unable to authenticate against Keystone.')
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
key_runner = KeyRunner(lic, token, kms_endpoint)
|
||||
except MissingKMSConfigurationError as e:
|
||||
LOG.info(e.message)
|
||||
print e.message
|
||||
sys.exit(1)
|
||||
try:
|
||||
print 'Attempting to fetch key from KMS...'
|
||||
binary_key = key_runner.get_key_binary()
|
||||
print 'Key successfully retrieved from KMS...'
|
||||
except PayloadDecodingError:
|
||||
LOG.info('Unable to parse KMS response.')
|
||||
print 'Unable to parse KMS response.'
|
||||
sys.exit(1)
|
||||
except MarshalHTTPException as e:
|
||||
LOG.info('Error requesting key from from KMS: %s', e.status_code)
|
||||
print 'Unable to access KMS.'
|
||||
sys.exit(1)
|
||||
|
||||
tmpdir = tempfile.mkdtemp()
|
||||
key_file_name = 'key.bin'
|
||||
|
||||
# Ensure the file is read/write by the creator only
|
||||
saved_umask = os.umask(0077)
|
||||
|
||||
path = os.path.join(tmpdir, key_file_name)
|
||||
try:
|
||||
with open(path, "wb") as tmp:
|
||||
tmp.write(binary_key)
|
||||
if action == 'open':
|
||||
if managed_name is None:
|
||||
missing_managed_name()
|
||||
result = vc.open_volume(key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully opened.'
|
||||
elif action == 'format':
|
||||
result = vc.format_volume(cipher=cipher, key_size=key_size,
|
||||
key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully formatted.'
|
||||
elif action == 'set':
|
||||
if setNeedsFormat:
|
||||
result = vc.format_volume(cipher=cipher, key_size=key_size,
|
||||
key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully formatted.'
|
||||
if managed_name is None:
|
||||
missing_managed_name()
|
||||
result = vc.open_volume(key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully opened.'
|
||||
elif result == 2:
|
||||
LOG.info("Open failed on set. Attempting format...")
|
||||
result = vc.format_volume(cipher=cipher, key_size=key_size,
|
||||
key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully formatted.'
|
||||
result = vc.open_volume(key_file=path)
|
||||
if result == 0:
|
||||
print 'The volume was successfully opened.'
|
||||
|
||||
except IOError as e:
|
||||
print 'IOError'
|
||||
else:
|
||||
pass
|
||||
finally:
|
||||
os.remove(path)
|
||||
os.umask(saved_umask)
|
||||
os.rmdir(tmpdir)
|
||||
|
||||
elif action == 'close':
|
||||
if not check_if_luks(vc, device, managed_name):
|
||||
sys.exit(1)
|
||||
if managed_name is None:
|
||||
missing_managed_name()
|
||||
result = vc.close_volume()
|
||||
if result == 0:
|
||||
print 'The volume was successfully closed.'
|
||||
|
||||
elif action == 'status':
|
||||
if not check_if_luks(vc, device, managed_name):
|
||||
sys.exit(1)
|
||||
if managed_name is None:
|
||||
missing_managed_name()
|
||||
vc.status_volume()
|
||||
|
||||
if __name__ == '__main__':
|
||||
agent_main()
|
||||
135
marshal_agent/agent/volCrypt.py
Normal file
135
marshal_agent/agent/volCrypt.py
Normal file
@ -0,0 +1,135 @@
|
||||
# Copyright (c) 2015 Cisco Systems
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
The volcrypt module interfaces with cryptsetup
|
||||
"""
|
||||
|
||||
from marshal_agent.common import config
|
||||
|
||||
import subprocess
|
||||
|
||||
CONF = config.CONF
|
||||
LOG = config.LOG
|
||||
|
||||
|
||||
class VolCrypt():
|
||||
|
||||
def __init__(self, dev_path, mapped_name):
|
||||
self.dev_path = dev_path
|
||||
self.mapped_name = mapped_name
|
||||
|
||||
def is_luks(self, dev_path):
|
||||
"""Checks if the specified device uses LUKS for encryption.
|
||||
:param device: the device to check
|
||||
:returns: true if the specified device uses LUKS; false otherwise
|
||||
"""
|
||||
try:
|
||||
# check to see if the device uses LUKS: exit status is 0
|
||||
# if the device is a LUKS partition and non-zero if not
|
||||
cmd = ["cryptsetup", 'isLuks', '--verbose', dev_path]
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.debug(output)
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("isLuks exited with (status %(exit_code)s): "),
|
||||
{"exit_code": e.returncode})
|
||||
return False
|
||||
|
||||
def open_volume(self, **kwargs):
|
||||
"""Opens the LUKS partition on the volume using the provided key file
|
||||
"""
|
||||
LOG.debug("opening encrypted volume %s", self.dev_path)
|
||||
try:
|
||||
cmd = ["cryptsetup", "luksOpen"]
|
||||
|
||||
key_file = kwargs.get("key_file", None)
|
||||
if key_file is not None:
|
||||
cmd.extend(["--key-file", key_file])
|
||||
|
||||
cmd.extend([self.dev_path])
|
||||
cmd.extend([self.mapped_name])
|
||||
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.info("Successfully opened the volume. Opening output was: %s",
|
||||
output)
|
||||
return 0
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("luksOpen exited with (status %(exit_code)s)"),
|
||||
{"exit_code": e.returncode})
|
||||
return e.returncode
|
||||
|
||||
def close_volume(self, **kwargs):
|
||||
"""Closes the LUKS partition on the volume
|
||||
"""
|
||||
LOG.debug("closing encrypted volume %s", self.dev_path)
|
||||
try:
|
||||
cmd = ["cryptsetup", "luksClose"]
|
||||
|
||||
cmd.extend([self.mapped_name])
|
||||
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.info("Successfully closed the volume. Closing output was: %s",
|
||||
output)
|
||||
return 0
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("luksClose exited with (status %(exit_code)s): "),
|
||||
{"exit_code": e.returncode})
|
||||
return e.returncode
|
||||
|
||||
def format_volume(self, **kwargs):
|
||||
"""Creates a LUKS header on the volume.
|
||||
"""
|
||||
LOG.debug("formatting encrypted volume %s", self.dev_path)
|
||||
try:
|
||||
cmd = ["cryptsetup", "--batch-mode", "luksFormat"]
|
||||
|
||||
cipher = kwargs.get("cipher", None)
|
||||
if cipher is not None:
|
||||
cmd.extend(["--cipher", cipher])
|
||||
key_size = kwargs.get("key_size", None)
|
||||
if key_size is not None:
|
||||
cmd.extend(["--key-size", key_size])
|
||||
key_file = kwargs.get("key_file", None)
|
||||
if key_file is not None:
|
||||
cmd.extend(["--key-file", key_file])
|
||||
cmd.extend([self.dev_path])
|
||||
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.info("Successfully formatted the volume. Format output was: \
|
||||
%s", output)
|
||||
return 0
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("luksFormat exited with (status %(exit_code)s): "),
|
||||
{"exit_code": e.returncode})
|
||||
return e.returncode
|
||||
|
||||
def status_volume(self, **kwargs):
|
||||
"""Statuses the LUKS partition on the volume
|
||||
"""
|
||||
LOG.debug("Stating encrypted volume %s", self.dev_path)
|
||||
try:
|
||||
cmd = ["cryptsetup", "-v", "status"]
|
||||
|
||||
cmd.extend([self.mapped_name])
|
||||
|
||||
output = subprocess.check_output(cmd, shell=False)
|
||||
LOG.info("Status output was: %s", output)
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
LOG.info(("status exited with (status %(exit_code)s): "),
|
||||
{"exit_code": e.returncode})
|
||||
0
0