Merge "Asynchronous Zone Import"
This commit is contained in:
commit
7ea92a2d02
|
@ -0,0 +1,307 @@
|
|||
..
|
||||
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
..
|
||||
This template should be in ReSTructured text. The filename in the git
|
||||
repository should match the launchpad URL, for example a URL of
|
||||
https://blueprints.launchpad.net/designate/+spec/awesome-thing should be named
|
||||
awesome-thing.rst . Please do not delete any of the sections in this
|
||||
template. If you have nothing to say for a whole section, just write: None
|
||||
For help with syntax, see http://sphinx-doc.org/rest.html
|
||||
To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
|
||||
|
||||
===============================
|
||||
Asynchronous Zone Import/Export
|
||||
===============================
|
||||
|
||||
https://blueprints.launchpad.net/designate/+spec/async-import-export
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Large zone imports have the potential to take a lot longer than the
|
||||
typically allowed time for API responses. Parsing large zone files, and inserting
|
||||
the necessary records scales linearly with the amount of records that need to be
|
||||
imported.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
The API for Zone imports needs to be changed to be asynchronous. This would involve
|
||||
a new database table for zone import "statuses", and a more robust system for managing
|
||||
the parsing and insertion of the zone data. Exporting zones will also return to the v2 API.
|
||||
|
||||
API Changes
|
||||
-----------
|
||||
|
||||
GET /v2/zones/tasks/imports
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This allows the user to view the statuses of their zone import requests. If the import has
|
||||
completed successfully, they can get the zone id and a link to the zone.
|
||||
|
||||
.. code-block:: http
|
||||
|
||||
GET /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
|
||||
Accept: application/json
|
||||
|
||||
HTTP/1.1 200 OK
|
||||
Content-Type: application/json; charset=UTF-8
|
||||
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66
|
||||
|
||||
{
|
||||
"imports": [
|
||||
{
|
||||
"id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
|
||||
"zone_id": "agqm44f0-s638-15e3-f4d3-1893572c9a67",
|
||||
"status": "SUCCESS",
|
||||
"links":{
|
||||
"self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66",
|
||||
"href": "http://127.0.0.1:9001/v2/zones/agqm44f0-s638-15e3-f4d3-1893572c9a67"
|
||||
},
|
||||
{
|
||||
"id": "addda8f0-f558-11e3-a3ac-0800200c9a66",
|
||||
"zone_id": "qgqm44f0-s638-15e3-f4d3-1893572c9a67",
|
||||
"status": "SUCCESS",
|
||||
"links":{
|
||||
"self": "http://127.0.0.1:9001/v2/zones/tasks/import/addda8f0-f558-11e3-a3ac-0800200c9a66",
|
||||
"href": "http://127.0.0.1:9001/v2/zones/qgqm44f0-s638-15e3-f4d3-1893572c9a67"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
POST /v2/zones/tasks/imports
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This creates a request to import a zone. The zone data is passed via the request body,
|
||||
and the Content-Type header must be set to 'text/dns'.
|
||||
|
||||
This returns an id for the import request, for the user to query.
|
||||
|
||||
.. code-block:: http
|
||||
|
||||
POST /v2/zones/tasks/import HTTP/1.1
|
||||
Accept: application/json
|
||||
Content-Type: text/dns
|
||||
|
||||
$ORIGIN example.com.
|
||||
example.com. 42 IN SOA ns.example.com. nsadmin.example.com. 42 42 42 42 42
|
||||
example.com. 42 IN NS ns.example.com.
|
||||
example.com. 42 IN MX 10 mail.example.com.
|
||||
ns.example.com. 42 IN A 10.0.0.1
|
||||
mail.example.com. 42 IN A 10.0.0.2
|
||||
|
||||
HTTP/1.1 201 Accepted
|
||||
Content-Type: application/json; charset=UTF-8
|
||||
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66
|
||||
|
||||
{
|
||||
"id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
|
||||
"zone_id": null,
|
||||
"status": "PENDING",
|
||||
"links":{
|
||||
"self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66"
|
||||
}
|
||||
}
|
||||
|
||||
GET /v2/zones/tasks/imports/<id>
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This allows the user to view the status of their zone import request. If the import has
|
||||
completed successfully, they can get the zone id and a link to the zone.
|
||||
|
||||
.. code-block:: http
|
||||
|
||||
GET /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
|
||||
Accept: application/json
|
||||
|
||||
HTTP/1.1 200 OK
|
||||
Content-Type: application/json; charset=UTF-8
|
||||
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66
|
||||
|
||||
{
|
||||
"id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
|
||||
"zone_id": "agqm44f0-s638-15e3-f4d3-1893572c9a67",
|
||||
"status": "SUCCESS",
|
||||
"links":{
|
||||
"self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66",
|
||||
"href": "http://127.0.0.1:9001/v2/zones/agqm44f0-s638-15e3-f4d3-1893572c9a67"
|
||||
}
|
||||
}
|
||||
|
||||
GET /v2/zones/<id>
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This request, with the header "Accept:text/dns" exports the zone in DNS zonefile format to the user.
|
||||
|
||||
.. code-block:: http
|
||||
|
||||
GET /v2/zones/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
|
||||
Accept: text/dns
|
||||
|
||||
HTTP/1.1 200 OK
|
||||
Content-Type: text/dns; charset=UTF-8
|
||||
Location: /v2/zones/cddda8f0-f558-11e3-a3ac-0800200c9a66
|
||||
|
||||
HTTP/1.1 200 OK
|
||||
Content-Type: text/dns
|
||||
|
||||
$ORIGIN example.com.
|
||||
$TTL 42
|
||||
example.com. IN SOA ns.designate.com. nsadmin.example.com. (
|
||||
1394213803 ; serial
|
||||
3600 ; refresh
|
||||
600 ; retry
|
||||
86400 ; expire
|
||||
3600 ; minimum
|
||||
)
|
||||
example.com. IN NS ns.designate.com.
|
||||
example.com. IN MX 10 mail.example.com.
|
||||
ns.example.com. IN A 10.0.0.1
|
||||
mail.example.com. IN A 10.0.0.2
|
||||
|
||||
Central Changes
|
||||
---------------
|
||||
|
||||
create_import_domain(body)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
| **Parameter** | **Description** | **Required** |
|
||||
+===============+===============================================+==============+
|
||||
| *body* | Unserialized request data from the API request| Yes |
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
|
||||
1. Create an entry in the zone_imports table to track the request with status
|
||||
PENDING.
|
||||
2. Kicks off a thread to _import_domain with the request body.
|
||||
3. Returns a zone_import object.
|
||||
|
||||
get_import_domain(id)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
| **Parameter** | **Description** | **Required** |
|
||||
+===============+===============================================+==============+
|
||||
| *import_id* | The id of the zone import | Yes |
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
|
||||
1. Calls storage.find_import to get a specific zone_import record from the
|
||||
zone_imports table.
|
||||
2. Returns the zone_import object
|
||||
|
||||
find_import_domains(context)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
| **Parameter** | **Description** | **Required** |
|
||||
+===============+===============================================+==============+
|
||||
| *context* | Context to be passed to storage.find_imports | Yes |
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
|
||||
1. Calls off to the storage.find_imports to find all the zone imports for a
|
||||
tenant.
|
||||
2. Returns the zone_import_list object
|
||||
|
||||
delete_import_domain(id)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
| **Parameter** | **Description** | **Required** |
|
||||
+===============+===============================================+==============+
|
||||
| *import_id* | The id of the zone import | Yes |
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
|
||||
1. Calls off to storage.delete_zone_import to delete the zone import record from
|
||||
the zone_imports table.
|
||||
|
||||
_import_domain(body)
|
||||
""""""""""""""""""""
|
||||
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
| **Parameter** | **Description** | **Required** |
|
||||
+===============+===============================================+==============+
|
||||
| *body* | Unserialized request data from the API request| Yes |
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
| *zone_import* | zone_import object from the original request | Yes |
|
||||
+---------------+-----------------------------------------------+--------------+
|
||||
|
||||
|
||||
1. Tries to use dnspython's from_text method to convert the zone into a dnspython
|
||||
object. If that doesn't work, updates the zone_import object in storage to ERROR
|
||||
to indicate that the zone could not be imported.
|
||||
2. Converts the dnspython zone to a Designate domain object.
|
||||
3. Calls central.create_domain with the converted object.
|
||||
4. Takes the return object's id and updates the zone_import object in storage
|
||||
with the new zone_id and updates the status of the import to COMPLETE, or similar.
|
||||
|
||||
|
||||
Storage Changes
|
||||
---------------
|
||||
|
||||
A new table for zone import records will be added, along with the boilerplate
|
||||
get, set, delete, update methods.
|
||||
|
||||
|
||||
New Table - zone_imports
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
+---------+---------+-----------+---------+-------------------------------------+
|
||||
| Row | Type | Nullable? | Unique? | Notes |
|
||||
+=========+=========+===========+=========+=====================================+
|
||||
| id | uuid | No | Yes | Primary key |
|
||||
+---------+---------+-----------+---------+-------------------------------------+
|
||||
| zone_id | uuid | Yes | Yes | Zone id when the import completes |
|
||||
+---------+---------+-----------+---------+-------------------------------------+
|
||||
| status | ENUM | Yes | No | One of [COMPLETE, PENDING, ERROR] |
|
||||
+---------+---------+-----------+---------+-------------------------------------+
|
||||
| message | VARCHAR | Yes | No | A message letting the user know why |
|
||||
| | | | | their import failed. For example: |
|
||||
| | | | | "Malformed zonefile", "Complete" |
|
||||
+---------+---------+-----------+---------+-------------------------------------+
|
||||
|
||||
Other Changes
|
||||
-------------
|
||||
|
||||
New DesignateObjects for the Imports will have be created, ZoneImport, ZoneImportList.
|
||||
|
||||
Exporting zones will also return to the v2 API, but should remain relatively unchanged.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Instead of creating a new zone_import table and object, it may be possible
|
||||
to create an empty domain object and call central._create_domain_in_storage
|
||||
and make an empty domain that is then updated with the result of the import.
|
||||
Only then could the call to the pool manager be made.
|
||||
|
||||
This might make the code simpler. But provides very little in the way of
|
||||
letting the user know that their import failed.
|
||||
You could add a status like MALFORMED_ZONEFILE or something, but that
|
||||
would still require the user to delete the zone before they tried again.
|
||||
Unless you soft-delete the zone when it fails and modify the default
|
||||
find_domains criterion to find zones that have been deleted only if they
|
||||
have that status.
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
tim-simmons-t
|
||||
|
||||
Milestones
|
||||
----------
|
||||
|
||||
Target Milestone for completion:
|
||||
Liberty-1
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
- Implement the database table, migration.
|
||||
- Implement the storage methods.
|
||||
- Implement the central methods.
|
||||
- Fix the API to use the new code.
|
||||
- Move the APIs back into the v2 namespace
|
Loading…
Reference in New Issue