openstack
/
designate
Code Issues Proposed changes
designate/designate/tests/unit
History
Takahito Hirose 00d0cb71e6 Fix DBDeadLock error resulting into 500 
When user requests the record registration request continuously,
sometimes designate hits DBDeadLock resuting into 500 InternalServerError.

We get below error:

2019-02-21 21:30:39.925 49752 ERROR designate.api.middleware RemoteError:
Remote error: DBDeadlock (pymysql.err.InternalError)
(1213, u'Deadlock found when trying to get lock; try restarting transaction')
[SQL: u'UPDATE records SET version=(records.version + %(version_1)s),
updated_at=%(updated_at)s, data=%(data)s, hash=%(hash)s, status=%(status)s,
action=%(action)s, serial=%(serial)s WHERE records.id = %(id_1)s']
[parameters: {'status': 'PENDING', 'hash': '39795ee18c6e3c9ad1c0190c6a3d8d4f',
'updated_at': datetime.datetime(2019, 2, 21, 12, 30, 39, 909846), u'version_1': 1,
u'id_1': '7a655eeda4d446cdaa81caf19ab55fcc', 'action': 'UPDATE',
'serial': 1550752338,
'data': u'ns2.example.jp. domain.example.com. 1550752338 3552 600 86400 3600'}]

In the process of record registeration, designate first tried to update
the reocrd and then update the zone status.

Updating the zone_status and registering the record process[1] and after synced
update record_status and zone_status process[2] are in reverse order. So If user
request the registering record many time and same time, Designate will get the
DBDeadLock, when these processes run the same time.

We observed that changing the order of the operations solves this issue.

[1] https://github.com/openstack/designate/blob/master/designate/central/service.py#L1292-L1320
[2] https://github.com/openstack/designate/blob/master/designate/central/service.py#L2310-L2322

1. transaction [1]-1 updating zone status process <- run    ---> table_name-zone
2. transaction [2]-1 updating record status process <- run  ---> table_name-record
3. transaction [1]-2 registering record process <- run and wait  ---> table_name-record
4. transaction [2]-2 updating zone process <-deadlock!           ---> table_name-zone

Change-Id: Icd6e690ac84a2fe0db0f4a8a513de47f7916f5ea
Related-Bug: #1785459
(cherry picked from commit f828654a3d)
 		2019-04-08 02:29:22 +00:00
..
test_agent Fix unit tests for oslo.config 3.18.0 2016-10-21 14:21:19 -06:00
test_api Replace assertDictEqual() with assertEqual() 2016-08-08 01:47:04 +00:00
test_backend Use designate.utils.generate_uuid instead of str(uuid.uuid4) 2017-01-16 16:12:08 +07:00
test_central Fix DBDeadLock error resulting into 500 2019-04-08 02:29:22 +00:00
test_mdns Skip test failing after oslo.messaging update 2017-09-09 12:01:11 +00:00
test_metrics Fix noop metrics timed decorator 2017-01-26 16:58:37 -07:00
test_objects Merge "Add 'DELETED', 'NO_ZONE' and 'SUCCESS' status for objects." 2017-06-09 13:04:14 +00:00
test_pool_manager Implement policy in code (1) 2017-10-24 09:23:25 +00:00
test_producer Fix task tests under py3 2017-06-29 10:02:03 +02:00
test_scheduler Remove usage of parameter enforce_type 2017-05-22 22:45:14 +08:00
README Create unit tests dir, move objects tests there 2015-07-20 14:15:42 +01:00
__init__.py Actually poll for zone deletes 2016-02-16 22:30:37 +00:00
test_pool.py Fix pool tests in py3 2017-06-29 10:01:11 +02:00
test_service_status.py Add support for getting Service Status 2016-04-12 16:32:17 +02:00

README 

This directory contains pure unit tests.

Examples:

ostestr --regex designate.tests.test_unit

python setup.py testr --coverage -t designate.tests.test_unit