Merge "Asynchronous Zone Import"

This commit is contained in:
Jenkins 2015-07-22 14:52:10 +00:00 committed by Gerrit Code Review
commit 7ea92a2d02
1 changed files with 307 additions and 0 deletions

View File

@ -0,0 +1,307 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
This template should be in ReSTructured text. The filename in the git
repository should match the launchpad URL, for example a URL of
https://blueprints.launchpad.net/designate/+spec/awesome-thing should be named
awesome-thing.rst . Please do not delete any of the sections in this
template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
===============================
Asynchronous Zone Import/Export
===============================
https://blueprints.launchpad.net/designate/+spec/async-import-export
Problem description
===================
Large zone imports have the potential to take a lot longer than the
typically allowed time for API responses. Parsing large zone files, and inserting
the necessary records scales linearly with the amount of records that need to be
imported.
Proposed change
===============
The API for Zone imports needs to be changed to be asynchronous. This would involve
a new database table for zone import "statuses", and a more robust system for managing
the parsing and insertion of the zone data. Exporting zones will also return to the v2 API.
API Changes
-----------
GET /v2/zones/tasks/imports
^^^^^^^^^^^^^^^^^^^^^^^^^^^
This allows the user to view the statuses of their zone import requests. If the import has
completed successfully, they can get the zone id and a link to the zone.
.. code-block:: http
GET /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
Accept: application/json
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66
{
"imports": [
{
"id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
"zone_id": "agqm44f0-s638-15e3-f4d3-1893572c9a67",
"status": "SUCCESS",
"links":{
"self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66",
"href": "http://127.0.0.1:9001/v2/zones/agqm44f0-s638-15e3-f4d3-1893572c9a67"
},
{
"id": "addda8f0-f558-11e3-a3ac-0800200c9a66",
"zone_id": "qgqm44f0-s638-15e3-f4d3-1893572c9a67",
"status": "SUCCESS",
"links":{
"self": "http://127.0.0.1:9001/v2/zones/tasks/import/addda8f0-f558-11e3-a3ac-0800200c9a66",
"href": "http://127.0.0.1:9001/v2/zones/qgqm44f0-s638-15e3-f4d3-1893572c9a67"
}
]
}
POST /v2/zones/tasks/imports
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This creates a request to import a zone. The zone data is passed via the request body,
and the Content-Type header must be set to 'text/dns'.
This returns an id for the import request, for the user to query.
.. code-block:: http
POST /v2/zones/tasks/import HTTP/1.1
Accept: application/json
Content-Type: text/dns
$ORIGIN example.com.
example.com. 42 IN SOA ns.example.com. nsadmin.example.com. 42 42 42 42 42
example.com. 42 IN NS ns.example.com.
example.com. 42 IN MX 10 mail.example.com.
ns.example.com. 42 IN A 10.0.0.1
mail.example.com. 42 IN A 10.0.0.2
HTTP/1.1 201 Accepted
Content-Type: application/json; charset=UTF-8
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66
{
"id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
"zone_id": null,
"status": "PENDING",
"links":{
"self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66"
}
}
GET /v2/zones/tasks/imports/<id>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This allows the user to view the status of their zone import request. If the import has
completed successfully, they can get the zone id and a link to the zone.
.. code-block:: http
GET /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
Accept: application/json
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66
{
"id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
"zone_id": "agqm44f0-s638-15e3-f4d3-1893572c9a67",
"status": "SUCCESS",
"links":{
"self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66",
"href": "http://127.0.0.1:9001/v2/zones/agqm44f0-s638-15e3-f4d3-1893572c9a67"
}
}
GET /v2/zones/<id>
^^^^^^^^^^^^^^^^^^
This request, with the header "Accept:text/dns" exports the zone in DNS zonefile format to the user.
.. code-block:: http
GET /v2/zones/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
Accept: text/dns
HTTP/1.1 200 OK
Content-Type: text/dns; charset=UTF-8
Location: /v2/zones/cddda8f0-f558-11e3-a3ac-0800200c9a66
HTTP/1.1 200 OK
Content-Type: text/dns
$ORIGIN example.com.
$TTL 42
example.com. IN SOA ns.designate.com. nsadmin.example.com. (
1394213803 ; serial
3600 ; refresh
600 ; retry
86400 ; expire
3600 ; minimum
)
example.com. IN NS ns.designate.com.
example.com. IN MX 10 mail.example.com.
ns.example.com. IN A 10.0.0.1
mail.example.com. IN A 10.0.0.2
Central Changes
---------------
create_import_domain(body)
^^^^^^^^^^^^^^^^^^^^^^^^^^
+---------------+-----------------------------------------------+--------------+
| **Parameter** | **Description** | **Required** |
+===============+===============================================+==============+
| *body* | Unserialized request data from the API request| Yes |
+---------------+-----------------------------------------------+--------------+
1. Create an entry in the zone_imports table to track the request with status
PENDING.
2. Kicks off a thread to _import_domain with the request body.
3. Returns a zone_import object.
get_import_domain(id)
^^^^^^^^^^^^^^^^^^^^^
+---------------+-----------------------------------------------+--------------+
| **Parameter** | **Description** | **Required** |
+===============+===============================================+==============+
| *import_id* | The id of the zone import | Yes |
+---------------+-----------------------------------------------+--------------+
1. Calls storage.find_import to get a specific zone_import record from the
zone_imports table.
2. Returns the zone_import object
find_import_domains(context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+---------------+-----------------------------------------------+--------------+
| **Parameter** | **Description** | **Required** |
+===============+===============================================+==============+
| *context* | Context to be passed to storage.find_imports | Yes |
+---------------+-----------------------------------------------+--------------+
1. Calls off to the storage.find_imports to find all the zone imports for a
tenant.
2. Returns the zone_import_list object
delete_import_domain(id)
^^^^^^^^^^^^^^^^^^^^^^^^
+---------------+-----------------------------------------------+--------------+
| **Parameter** | **Description** | **Required** |
+===============+===============================================+==============+
| *import_id* | The id of the zone import | Yes |
+---------------+-----------------------------------------------+--------------+
1. Calls off to storage.delete_zone_import to delete the zone import record from
the zone_imports table.
_import_domain(body)
""""""""""""""""""""
+---------------+-----------------------------------------------+--------------+
| **Parameter** | **Description** | **Required** |
+===============+===============================================+==============+
| *body* | Unserialized request data from the API request| Yes |
+---------------+-----------------------------------------------+--------------+
| *zone_import* | zone_import object from the original request | Yes |
+---------------+-----------------------------------------------+--------------+
1. Tries to use dnspython's from_text method to convert the zone into a dnspython
object. If that doesn't work, updates the zone_import object in storage to ERROR
to indicate that the zone could not be imported.
2. Converts the dnspython zone to a Designate domain object.
3. Calls central.create_domain with the converted object.
4. Takes the return object's id and updates the zone_import object in storage
with the new zone_id and updates the status of the import to COMPLETE, or similar.
Storage Changes
---------------
A new table for zone import records will be added, along with the boilerplate
get, set, delete, update methods.
New Table - zone_imports
^^^^^^^^^^^^^^^^^^^^^^^^
+---------+---------+-----------+---------+-------------------------------------+
| Row | Type | Nullable? | Unique? | Notes |
+=========+=========+===========+=========+=====================================+
| id | uuid | No | Yes | Primary key |
+---------+---------+-----------+---------+-------------------------------------+
| zone_id | uuid | Yes | Yes | Zone id when the import completes |
+---------+---------+-----------+---------+-------------------------------------+
| status | ENUM | Yes | No | One of [COMPLETE, PENDING, ERROR] |
+---------+---------+-----------+---------+-------------------------------------+
| message | VARCHAR | Yes | No | A message letting the user know why |
| | | | | their import failed. For example: |
| | | | | "Malformed zonefile", "Complete" |
+---------+---------+-----------+---------+-------------------------------------+
Other Changes
-------------
New DesignateObjects for the Imports will have be created, ZoneImport, ZoneImportList.
Exporting zones will also return to the v2 API, but should remain relatively unchanged.
Alternatives
------------
Instead of creating a new zone_import table and object, it may be possible
to create an empty domain object and call central._create_domain_in_storage
and make an empty domain that is then updated with the result of the import.
Only then could the call to the pool manager be made.
This might make the code simpler. But provides very little in the way of
letting the user know that their import failed.
You could add a status like MALFORMED_ZONEFILE or something, but that
would still require the user to delete the zone before they tried again.
Unless you soft-delete the zone when it fails and modify the default
find_domains criterion to find zones that have been deleted only if they
have that status.
Assignee(s)
-----------
Primary assignee:
tim-simmons-t
Milestones
----------
Target Milestone for completion:
Liberty-1
Work Items
----------
- Implement the database table, migration.
- Implement the storage methods.
- Implement the central methods.
- Fix the API to use the new code.
- Move the APIs back into the v2 namespace