sahara/doc/source/restapi/v02.rst
Alexander Ignatov a2892cb4ec Pre-release 0.1.2 doc changes
Change-Id: I7d1cc835522fff0a99015d21019a4aef6f3220f3
2013-05-30 07:28:08 -07:00

44 KiB

Savanna REST API v0.2

Note

REST API v0.2 corresponds to Savanna v0.1.X

1 General API information

This section contains base info about the Savanna REST API design.

1.1 Authentication and Authorization

The Savanna API uses the Keystone Identity Service as the default authentication service. When Keystone is enabled, users who submit requests to the Savanna service must provide an authentication token in X-Auth-Token request header. User could obtain the token by authenticating to the Keystone endpoint. For more information about Keystone, see the OpenStack Identity Developer Guide.

Also with each request user must specify OpenStack tenant in url path like that: '/v0.2/{tenant_id}/clusters'. Savanna will perform the requested operation in that tenant using provided credentials. Therefore, user will be able to create and manage clusters only within tenants he have access to.

Currently Savanna does not perform its own authorization checks. We are planning to implement them later. Each of the Savanna objects described in section 2 ('Object Model') will belong to some tenant. User will be able to perform operations only on objects he created or if he is an admin in the tenant to which object belongs.

1.2 Request / Response Types

The Savanna API supports the JSON data serialization format. This means that for requests that contain a body, the Content-Type header must be set to the MIME type value "application/json". Also, clients should accept JSON serialized responses by specifying the Accept header with the MIME type value "application/json" or adding ".json" extension to the resource name. The default response format is "application/json" if the client omits to specify an Accept header or append the ".json" extension in the URL path.

Example:

GET /v0.2/{tenant_id}/clusters.json

or

GET /v0.2/{tenant_id}/clusters
Accept: application/json

1.3 Filtering and Column Selection

(It's not implemented yet)

The Savanna API supports filtering based on all top level attributes of a resource. Filters are applicable to all list requests.

For example, the following request returns all clusters named 'foobar':

GET /v0.2/{tenant_id}/clusters?name=foobar

When you specify multiple filters, the Savanna API returns only objects that meet all filtering criteria. The operation applies an AND condition among the filters, for example:

GET /v0.2/{tenant_id}/clusters?name=foo1&some_attr=some_value

Currently, there is no OR mechanism for filtering. Alternatively, you can issue a distinct request for each filter and build a response set from the received responses on the client-side.

By default, the Savanna service returns all attributes for any show or list call. The Savanna API has a mechanism to limit the set of attributes returned. You can use the fields query parameter to control the attributes returned from the Savanna API.

For example, the following request returns only id, name and status for each cluster:

GET /v0.2/{tenant_id}/clusters.json?fields=id&fields=name&fields=status

Such selector could be useful for displaying some info about running clusters on GUI.

Filtering and column selection features could be used together too.

1.4 Faults

The Savanna API returns an error response if a failure occurs while processing a request. Savanna uses only standard HTTP error codes. 4xx errors indicate problems in the particular request being sent from the client and 5xx errors indicate server-side problems.

Here we can add table with all used error codes.

The response body will contain richer information about the cause of the error. An error response follows the format illustrated by the following example:

HTTP/1.1 400 BAD REQUEST
Content-type: application/json
Content-length: 126

{
    "error_name": "CLUSTER_NAME_ALREADY_EXISTS",
    "error_message": "Cluster with name 'test-cluster' already exists",
    "error_code": 400
}

The 'error_code' attribute equals to HTTP response code. The 'error_name' attribute indicates generic error type without any concrete ids or names, etc. And the last attribute is 'error_message' that contains human readable error description.

2 Object Model

2.1 Node Template object

2.1.1 Description

Node Template object is an abstraction for storing VM hardware parameters (vCPU, RAM, HDD, etc.), VM node type (JobTracker+NameNode, TaskTracker+DataNode, JobTracker only, etc.) and some Hadoop-relative configurations (processes heap sizes, # of MapReduce tasks per node, etc.).

Expected node types:

JT+NN - node with both JobTracker and NameNode running;

JT - node with only JobTracker running;

NN - node with only NameNode running;

TT+DN - node with both TaskTracker and DataNode running;

TT - node with only TaskTracker running;

DN - node with only DataNode running.

All node types are predefined and new types might be added in the future.

2.1.2 Object Fields

Name Type Description
id [required for update/get] string
name string A name of the node template
node_type string/object That parameter type depends on the operation: if create, then node type name should be provided as string. List and retrieve operations return node type as object (see below its fields)
node_type.name string Name of the node type
node_type.processes array of strings List of process names that will be executed on node
flavor_id string Name of OpenStack flavor which will be used to provision nodes. Actually this should be an id, we use names temporarily for our convenience
job_tracker section Section for Job Tracker configuration
job_tracker.heap_size [JT required] int Job Tracker heap size in MB
name_node section Section for Name Node configuration
name_node.heap_size [NN required] int Name Node heap size in MB
task_tracker section Section for Task Tracker configuration
task_tracker.heap_size [TT required] int Task Tracker heap size in MB
task_tracker.max_map_tasks [TT optional] int Max number of map tasks per task tracker
task_tracker.max_reduce_tasks [TT optional] int Max number of reduce tasks per task tracker
task_tracker.task_heap_size [TT required] int Task (map or reduce) heap size in MB
data_node section Section for Data Node configuration
data_node.heap_size [DN required] int Data Node heap size in megabytes

2.1.3 Example

The example below corresponds to Node Template returned by retrieve operation. For example of Node Template used in create operation see

that operation details further below

{
    "node_template": {
        "id": "3412",
        "name": "taskTracker_and_dataNode.medium",
        "node_type": {
            "name": "TT+DN",
            "processes": ["task_tracker", "data_node"]
        },
        "flavor_id": "m1.medium",
        "task_tracker": {
            "heap_size": 384,
            "max_map_tasks": 3,
            "max_reduce_tasks": 1,
            "task_heap_size": 640
        },
        "data_node": {
            "heap_size": 384
        }
    }
}

2.2 Cluster Create/Update object

2.2.1 Description

This object is used to create and update existing clusters. It contains cluster's name and 'node_templates' section that provides information about number of instances of Node Templates that we want to use. Additionally, it includes the 'base_image_id' id which specifies image that should be the base for VMs creation. Such image should be prepared by installing Hadoop with special layout. The 'id' field should not be specified during cluster creation, but it should be specified while updating the existing cluster.

2.2.2 Object fields

Name Type Description
id [required for update] string
name string A name of the cluster
node_templates map (object) Map Node Template -> count
base_image_id string An id of the image stored in Glance that will be used for VM creation (the image should have Hadoop installed with the specific layout)

2.2.3 Example

{
    "cluster": {
        "id": "1234",
        "name": "dev-cluster",
        "node_templates": {
            "jobTracker_and_nameNode.medium": 1,
            "taskTracker_and_dataNode.medium": 4
        },
        "base_image_id": "123456"
    }
}

2.3 Cluster object

2.3.1 Description

This object used in REST API responses to return information about clusters.

2.3.2 Object fields

Name Type Description
id [required for update] string
name string A name of the cluster. Name could consist of english letters, digits and hyphens
node_templates map (object) Map <Node Template name> -> count
base_image_id string An id of the image stored in Glance that will be used for VM creation (the image should have Hadoop installed with the specific layout)
status string Status of the cluster
nodes objects array List of vms used by the Hadoop cluster
service_urls section Section with urls for installed services
service_urls.job_tracker string Url for JobTracker
service_urls.name_node string Url for NameNode

2.3.3 Example

{
    "cluster": {
        "id": "1234",
        "name": "dev-cluster",
        "node_templates": {
            "jobTracker_and_nameNode.medium": 1,
            "taskTracker_and_dataNode.medium": 10
        },
        "base_image_id": "123456",
        "status": "Active",
        "nodes": [
            "<Node object>",
            "<Node object>"
        ],
        "service_urls": {
            "job_tracker": "http://10.0.1.10:50030",
            "name_node": "http://10.0.1.10:50070"
        }
    }
}

2.4 Node object

2.4.1 Description

This object is used in REST API responses to return information about cluster's nodes. In fact, a node is a virtual machine managed by OpenStack Nova. This object used only as a nested element of Cluster objects.

2.4.2 Object fields

Name Type Description
vm_id string VM id in OpenStack cluster
node_template object Node Template used to create this node
node_template.id string Node Template id used to create this node
node_template.name string Node Template name used to create this node

2.4.3 Example

{
    "vm_id": "dee040e1-f58f-4d54-846b-fec8e2bd9b21",
    "node_template": {
        "id": "123456",
        "name": "jt_nn.xlarge"
    }
}

3 API operations

3.1 Node Templates ops

Summary

Use the following APIs to manage templates:

Verb URI Description
GET /v0.2/{tenant_id}/node-templates Lists summary for all Node Templates in Savanna which belong to the Tenant with id <tenant_id>
POST /v0.2/{tenant_id}/node-templates Creates a new Node Template
GET /v0.2/{tenant_id}/node-templates/{node_template_id} Lists information for the specified Node Template
PUT /v0.2/{tenant_id}/node-templates/{node_template_id} Updates Node Template identified by node_template_id (only unused Node Templates could be modified)
DELETE /v0.2/{tenant_id}/node-templates/{node_template_id} Destroyed a specified Node Template (only unused Node Templates could be removed)

3.1.2 List all Node Templates

Normal Response Code: 200 (OK)

Errors: none

This operations returns the list of all Node Templates associated with specified tenant (url attribute 'tenant_id').

This operation does not require a request body.

This operation returns a response body. It returns a (potentially empty) list, each element in the list is a "Node Template object" described in section 2.1.

Example:

request

GET http://savanna/v0.2/775181/node-templates
Accept: application/json
X-Auth-Token: SOME_AUTH_TOKEN

response

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: XX
{
    "node_templates": [
        {
            "id": "3412",
            "name": "tt_dn.medium",
            "node_type": {
                "name": "TT+DN",
                "processes": ["task_tracker", "data_node"]
            },
            "flavor_id": "m1.medium",
            "task_tracker": {
                "heap_size": 384,
                "max_map_tasks": 3,
                "max_reduce_tasks": 1,
                "task_heap_size": 640
            },
            "data_node": {
                "heap_size": 384
            }
        },
        {
            "id": "3413",
            "name": "tt_dn.xlarge",
            "node_type": {
                "name": "TT+DN",
                "processes": ["task_tracker", "data_node"]
            },
            "flavor_id": "m1.medium",
            "task_tracker": {
                "heap_size": 2048,
                "max_map_tasks": 6,
                "max_reduce_tasks": 3,
                "task_heap_size": 2048
            },
            "data_node": {
                "heap_size": 2048
            }
        }
    ]
}

3.1.3 Create a Node Template

Normal Response Code: 202 (Accepted)

Errors:

400, VALIDATION_ERROR, "Some input field is specified incorrectly or is missing"

400, MALFORMED_REQUEST_BODY, "Malformed message body with 'reason'"

400, NODE_TEMPLATE_ALREADY_EXISTS, "Node Template with name 'custom_user_template' already exists"

400, NODE_TYPE_NOT_FOUND, "NodeType 'wrong-node-type' not found"

400, NODE_PROCESS_DISCREPANCY, "Discrepancies in Node Processes. Required: ['process1', 'process2']"

400, FLAVOR_NOT_FOUND, "Cannot find flavor with name 'some-flavor'"

This operation creates a new Node Template from the parameters specified in request object. All parameters has been described in section 2.1.

Node Template will be created in the specified tenant (url attribute 'tenant_id').

This operation requires request body.

This operation returns a response body. It contains newly created node template as "Node Template object", defined in section 2.1.

Example:

request

POST http://savanna/v0.2/775181/node-templates
{
    "node_template": {
        "name": "custom_user_template",
        "node_type": "TT+DN",
        "flavor_id": "m1.medium",
        "task_tracker": {
            "heap_size": 384,
            "max_map_tasks": 3,
            "max_reduce_tasks": 1,
            "task_heap_size": 640
        },
        "data_node": {
            "heap_size": 384
        }
    }
}

response

HTTP/1.1 202 Accepted
Content-Type: application/json
Content-Length: XX
{
    "node_template": {
        "id": "3412",
        "name": "custom_user_template",
        "node_type": {
            "name": "TT+DN",
            "processes": ["task_tracker", "data_node"]
        },
        "flavor_id": "m1.medium",
        "task_tracker": {
            "heap_size": 384,
            "max_map_tasks": 3,
            "max_reduce_tasks": 1,
            "task_heap_size": 640
        },
        "data_node": {
            "heap_size": 384
        }
    }
}

3.1.4 Retrieve a specific Node Template

Normal Response Code: 200 (OK)

Errors:

404, NODE_TEMPLATE_NOT_FOUND, "NodeTemplate 'node_template_id' not found"

This operation returns a Node Template object identified by node_template_id.

This operation does not require a request body.

This operation returns a response body. It contains node template as "Node Template object", defined in section 2.1.

Example:

request

GET http://savanna/v0.2/775181/node-templates/3421

response

HTTP/1.1 200 OK
Content-Type: application/json
{
    "node_template": {
        "id": "3421",
        "name": "tt_dn.medium",
        "node_type": {
            "name": "TT+DN",
            "processes": ["task_tracker", "data_node"]
        },
        "flavor_id": "m1.medium",
        "task_tracker": {
            "heap_size": 384,
            "max_map_tasks": 3,
            "max_reduce_tasks": 1,
            "task_heap_size": 640
        },
        "data_node": {
            "heap_size": 384
        }
    }
}

3.1.5 Update a Node Template

Normal Response Code: 202 (Accepted)

Errors:

400, VALIDATION_ERROR, "Some input field is specified incorrectly or is missing"

400, MALFORMED_REQUEST_BODY, "Malformed message body with 'reason'"

400, NODE_TEMPLATE_ALREADY_EXISTS, "Node Template with name 'custom_user_template' already exists"

400, NODE_TYPE_NOT_FOUND, "NodeType 'wrong-node-type' not found"

400, NODE_PROCESS_DISCREPANCY, "Discrepancies in Node Processes. Required: ['process1', 'process2']"

400, FLAVOR_NOT_FOUND, "Cannot find flavor with name 'some-flavor'"

400, NODE_TEMPLATE_ALREADY_IN_USE, "NodeTemplate 'node_template_id' is used by running cluster"

404, NODE_TEMPLATE_NOT_FOUND, "NodeTemplate 'node_template_id' not found"

This operation updates the attributes of the node template identified by node_template_id. All attributes should be specified while updating resource. Only node templates that are not in use could be updated. So, if any cluster uses a node template, than such node template could not be modified or removed.

This operation requires request body. It should contain the full Node Template object (defined in section 2.1) with new data.

This operation returns response body. It contains updated Node Template object defined in section 2.1.

Example:

request

PUT http://savanna/v0.2/775181/node-templates/3413
{
    "node_template": {
        "id": "3413",
        "name": "tt_dn.medium",
        "node_type": {
            "name": "TT+DN",
            "processes": ["task_tracker", "data_node"]
        },
        "flavor_id": "m1.medium",
        "task_tracker": {
            "heap_size": 402,
            "max_map_tasks": 3,
            "max_reduce_tasks": 1,
            "task_heap_size": 500
        },
        "data_node": {
            "heap_size": 402
        }
    }
}

response

HTTP/1.1 202 Accepted
Content-Type: application/json
{
    "node_template": {
        "id": "3413",
        "name": "tt_dn.medium",
        "node_type": {
            "name": "TT+DN",
            "processes": ["task_tracker", "data_node"]
        },
        "flavor_id": "m1.medium",
        "task_tracker": {
            "heap_size": 402,
            "max_map_tasks": 3,
            "max_reduce_tasks": 1,
            "task_heap_size": 500
        },
        "data_node": {
            "heap_size": 402
        }
    }
}

3.1.6 Remove a Node Template

Normal Response Code: 204 (NO CONTENT)

Errors:

404, NODE_TEMPLATE_NOT_FOUND, "NodeTemplate 'node_template_id' not found"

This operation removes the specified Node Template and its associated resources. Any and all data is immediately purged and is not recoverable. Only Node Templates that are not in use could be removed. So, if any cluster uses a Node Template, than such Node Template could not be modified or removed.

This operation does not require a request body.

This operation does not return a response body.

Example:

request

DELETE http://savanna/v0.2/775181/node-templates/3421

response

HTTP/1.1 204 No Content

3.2 Clusters ops

Summary

Use the following APIs to manage clusters:

Verb URI Description
GET /v0.2/{tenant_id}/clusters Lists summary for all clusters in Savanna which belong to the specified tenant
POST /v0.2/{tenant_id}/clusters Creates a new cluster
GET /v0.2/{tenant_id}/clusters/{cluster_id} Lists detailed information for the specified cluster
PUT /v0.2/{tenant_id}/clusters/{cluster_id} Updates cluster identified by cluster_id
DELETE /v0.2/{tenant_id}/clusters/{cluster_id} Destroys specified cluster

3.2.2 List all clusters

Normal Response Code: 200 (OK)

Errors: none

This operations returns the list of all clusters associated with the specified tenant (url attribute 'tenant_id').

This operation does not require a request body.

This operation returns a response body. It returns a (potentially empty) list, each element in the list is a "Cluster object" described in section 2.2.

Example:

request

GET http://savanna/v0.2/775181/clusters

response

HTTP/1.1 200 OK
Content-Type: application/json
{
    "clusters": [
        {
            "id": "1234",
            "name": "dev-cluster",
            "node_templates": {
                "jt_nn.medium": 1,
                "tt_dn.medium": 10
            },
            "base_image_id": "123456",
            "status": "Active",
            "service_urls": {
                "job_tracker": "10.0.1.10:50030",
                "name_node": "10.0.1.10:50070"
            },
            "nodes": [
                "<Node object>",
                "<Node object>"
            ]
        },
        {
            "id": "1235",
            "name": "qa-cluster",
            "node_templates": {
                "jobTracker.medium": 1,
                "nameNode.medium": 1,
                "tt_dn.medium": 10
            },
            "base_image_id": "123456",
            "status": "Active",
            "service_urls": {
                "job_tracker": "10.0.1.25:50030",
                "name_node": "10.0.1.25:50070"
            },
            "nodes": [
                "<Node object>",
                "<Node object>"
            ]
        }
     ]
}

3.2.3 Create a cluster

Normal Response Code: 202 (Accepted)

Errors:

400, VALIDATION_ERROR, "Some input field is specified incorrectly or is missing"

400, MALFORMED_REQUEST_BODY, "Malformed message body with 'reason'"

400, CLUSTER_ALREADY_EXISTS, "Cluster with name 'custom_cluster' already exists"

400, NODE_TEMPLATE_NOT_FOUND, "NodeTemplate 'node_template_id' not found"

400, NOT_SINGLE_NAME_NODE, "Hadoop cluster should contain only 1 NameNode. "Actual NN count is 'nn_count'"

400, NOT_SINGLE_JOB_TRACKER, "Hadoop cluster should contain only 1 JobTracker. "Actual JT count is 'jt_count'"

400, IMAGE_NOT_FOUND, "Cannot find image with id 'base_image_id'"

400, NOT_ENOUGH_RESOURCES, "Nova available instances='instances_count', VCPUs='vcpus_count', RAM='ram_count'. Requested instances='req_instances', VCPUs='req_vcpus', RAM='req_ram'"

This operation creates a new cluster from the parameters specified in request object. All parameters have been described in section 2.2.

This operation requires request body.

This operation returns a response body. It contains newly created Node Template as "Cluster Create/Update object", defined in section 2.2.

The main idea is to specify number of required instances of some Node Template. It means that if you want 10 nodes with both TaskTracker and DataNode than you should specify Node Template with type "TT+DN" and specify number of instances 10. Number of master nodes should be equal to 1 for each master node type, i.e. there must be exactly 1 JobTracker node and 1 NameNode node in the cluster.

Example:

request

POST http://savanna/v0.2/775181/clusters
{
    "cluster": {
        "name": "dev-cluster",
        "node_templates": {
            "jt_nn.medium": 1,
            "tt_dn.medium": 10
        },
        "base_image_id": "123456"
    }
}

response

HTTP/1.1 202 Accepted
Content-Type: application/json
{
    "cluster": {
        "id": "1234",
        "name": "dev-cluster",
        "node_templates": {
            "jt_nn.medium": 1,
            "tt_dn.medium": 10
        },
        "base_image_id": "123456",
        "status": "Starting",
        "service_urls": {},
        "nodes": []
    }
}

3.2.4 Retrieve a specific cluster

Normal Response Code: 200 (OK)

Errors:

404, CLUSTER_NOT_FOUND, "Cluster 'cluster_id' not found"

This operation returns a cluster object identified by id.

This operation does not require a request body.

This operation returns a response body. It contains cluster as "Cluster object", defined in section 2.2.

Example:

request

GET http://savanna/v0.2/775181/clusters/1234

response

HTTP/1.1 200 OK
Content-Type: application/json
{
    "cluster": {
        "id": "1234",
        "name": "dev-cluster",
        "node_templates": {
            "jt_nn.medium": 1,
            "tt_dn.medium": 10
        },
        "base_image_id": "123456",
        "status": "Active",
        "service_urls": {
            "job_tracker": "10.0.1.10:50030",
            "name_node": "10.0.1.10:50070"
        },
        "nodes": [
            "<Node object>",
            "<Node object>"
        ]
    }
}

3.2.5 Update a cluster

Normal Response Code: 202 (Accepted)

Errors:

400, VALIDATION_ERROR, "Some input field is specified incorrectly or is missing"

400, MALFORMED_REQUEST_BODY, "Malformed message body with 'reason'"

400, CLUSTER_ALREADY_EXISTS, "Cluster with name 'custom_cluster' already exists"

400, NODE_TEMPLATE_NOT_FOUND, "NodeTemplate 'node_template_id' not found"

400, NOT_SINGLE_NAME_NODE, "Hadoop cluster should contain only 1 NameNode. "Actual NN count is 'nn_count'"

400, NOT_SINGLE_JOB_TRACKER, "Hadoop cluster should contain only 1 JobTracker. "Actual JT count is 'jt_count'"

400, IMAGE_NOT_FOUND, "Cannot find image with id 'base_image_id'"

400, NOT_ENOUGH_RESOURCES, "Nova available instances='instances_count', VCPUs='vcpus_count', RAM='ram_count'. Requested instances='req_instances', VCPUs='req_vcpus', RAM='req_ram'"

404, CLUSTER_NOT_FOUND, "Cluster 'cluster_id' not found"

This operation updates the attributes of the cluster identified by cluster_id. All attributes should be specified while updating resource.

This operation requires request body. It should contains the full "Cluster Create/Update object" (defined in section 2.2) with new data.

This operation returns response body. It contains updated cluster object defined in section 2.2.

This operation should be used for adding and removing nodes. For more information, please, take a look on cluster creation operation. All cluster Node Templates should be specified while scaling cluster , not only new Node Templates or those with changed number of instances. All missing Node Templates will be removed from cluster.

Example:

request

PUT http://savanna/v0.2/775181/clusters/1234
{
    "cluster": {
        "id": "1234",
        "name": "dev-cluster",
        "node_templates": {
            "jt_nn.medium": 1,
            "tt_dn.medium": 15
        },
        "base_image_id": "123456"
    }
}

response

HTTP/1.1 202 Accepted
Content-Type: application/json
{
    "cluster": {
        "id": "1234",
        "name": "dev-cluster",
        "node_templates": {
            "jt_nn.medium": 1,
            "tt_dn.medium": 15
        },
        "base_image_id": "123456",
        "status": "Active",
        "service_urls": {
            "job_tracker": "10.0.1.10:50030",
            "name_node": "10.0.1.10:50070"
        },
        "nodes": [
            "<Node object>",
            "<Node object>"
        ]
    }
}

3.2.6 Remove a cluster

Normal Response Code: 204 (NO CONTENT)

Errors:

404, CLUSTER_NOT_FOUND, "Cluster 'cluster_id' not found"

This operation removes the specified cluster (identified by cluster_id) and its associated instances or other resources. Any and all data is immediately purged and is not recoverable.

This operation does not require a request body.

This operation does not return a response body.

Example:

request

DELETE http://savanna/v0.2/775181/clusters/1234

response

HTTP/1.1 204 No Content