From 10e48bb6bba928d54c9cbad8915d0170cf209ea4 Mon Sep 17 00:00:00 2001
From: melanie witt <melwittt@gmail.com>
Date: Tue, 16 May 2017 10:25:42 +0000
Subject: [PATCH] Cache database and message queue connection objects

Recently in the gate we have seen a trace on some work-in-progress
patches:

  OperationalError: (pymysql.err.OperationalError)
                    (1040, u'Too many connections')

and at least one operator has reported that the number of database
connections increased significantly going from Mitaka to Newton.

It was suspected that the increase was caused by creating new oslo.db
transaction context managers on-the-fly when switching database
connections for cells. Comparing the dstat --tcp output of runs of the
gate-tempest-dsvm-neutron-full-ubuntu-xenial job with and without
caching of the database connections showed a difference of 445 active
TCP connections and 1495 active TCP connections, respectively [1].

This adds caching of the oslo.db transaction context managers and the
oslo.messaging transports to avoid creating a large number of objects
that are not being garbage-collected as expected.

Closes-Bug: #1691545

[1] https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing

Change-Id: I17e0eb836dd87aac5859f506e7d771d42753d31a
---
 nova/test.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/nova/test.py b/nova/test.py
index d6e4d59a6..64d548640 100644
--- a/nova/test.py
+++ b/nova/test.py
@@ -242,6 +242,7 @@ class TestCase(testtools.TestCase):
         # NOTE(danms): Reset the cached list of cells
         from nova.compute import api
         api.CELLS = []
+        context.CELL_CACHE = {}
 
         self.cell_mappings = {}
         self.host_mappings = {}