Fix the init command global conf dir path

This commit fixes how we use the global configuration path as part of
the tempest init command. The concept behind the global config dir is
that is used as an initial preseed of the local etc/ dirs that get
used in each workspace. However, right now the init command relies on
python packaging to try and figure out where this dir lives. This is
because we implicitly are relying on the data_files construct in the
python packaging ecosystem to create this global configuration dir.
This however is a mistake and causes nondeterminism and lots of bugs
because python packaging is never consistent in how it handles these.

Instead of futily attempting to guess where python might put the data
files and hoping that we got it right, this commit switches to an
opinionated stance on where these directories live, it becomes a
documented set of places and the burden is switched to the installer
and or user to make sure these directories are correctly populated.
While this requires an extra step in some installation scenarios it's
a necessary extra step to ensure things actually work. As part of
this change it was also necessary to update certain functions to be
more resilient against the absence of any files in the global config
dir.

Partially-Implements: bp tempest-run-cmd
Change-Id: Ic4e67362db053848e6ad03b0eae9e55faa87766f

releasenotes/notes/tempest-init-global-config-dir-location-changes-12260255871d3a2b.yaml

@@ -0,0 +1,12 @@
+---
+upgrade:
+  - The location on disk that the *tempest init* command looks for has changed.
+    Previously it would attempt to use python packaging's data files to guess
+    where setuptools/distutils were installing data files, which was incredibly
+    unreliable and depended on how you installed tempest and which versions of
+    setuptools, distutils, and python you had installed. Instead, now it will
+    use either /etc/tempest, $XDG_CONFIG_PATH/.config/tempest, or
+    ~/.tempest/etc (attempted in that order). If none of these exist it will
+    create an empty ~/.tempest/etc directory. If you were relying on the
+    previous behavior and none of these directories were being used you will
+    need to move the files to live in one of these directories.

tempest/cmd/init.py

@@ -40,45 +40,28 @@ group_regex=([^\.]*\.)*
 def get_tempest_default_config_dir():
     """Get default config directory of tempest
 
-    Returns the correct default config dir to support both cases of
-    tempest being or not installed in a virtualenv.
-    Cases considered:
-    - no virtual env, python2: real_prefix and base_prefix not set
-    - no virtual env, python3: real_prefix not set, base_prefix set and
-      identical to prefix
-    - virtualenv, python2: real_prefix and prefix are set and different
-    - virtualenv, python3: real_prefix not set, base_prefix and prefix are
-      set and identical
-    - pyvenv, any python version: real_prefix not set, base_prefix and prefix
-      are set and different
+    There are 3 dirs that get tried in priority order. First is /etc/tempest,
+    if that doesn't exist it looks for a tempest dir in the XDG_CONFIG_HOME
+    dir (defaulting to ~/.config/tempest) and last it tries for a
+    ~/.tempest/etc directory. If none of these exist a ~/.tempest/etc
+    directory will be created.
 
     :return: default config dir
     """
-    real_prefix = getattr(sys, 'real_prefix', None)
-    base_prefix = getattr(sys, 'base_prefix', None)
-    prefix = sys.prefix
     global_conf_dir = '/etc/tempest'
-    if (real_prefix is None and
-            (base_prefix is None or base_prefix == prefix) and
-            os.path.isdir(global_conf_dir)):
-        # Probably not running in a virtual environment.
-        # NOTE(andreaf) we cannot distinguish this case from the case of
-        # a virtual environment created with virtualenv, and running python3.
-        # Also if it appears we are not in virtual env and fail to find
-        # global config: '/etc/tempest', fall back to
-        # '[sys.prefix]/etc/tempest'
+    xdg_config = os.environ.get('XDG_CONFIG_HOME',
+                                os.path.expanduser('~/.config'))
+    user_xdg_global_path = os.path.join(xdg_config, 'tempest')
+    user_global_path = os.path.join(os.path.expanduser('~'), '.tempest/etc')
+    if os.path.isdir(global_conf_dir):
         return global_conf_dir
+    elif os.path.isdir(user_xdg_global_path):
+        return user_xdg_global_path
+    elif os.path.isdir(user_global_path):
+        return user_global_path
     else:
-        conf_dir = os.path.join(prefix, 'etc/tempest')
-        if os.path.isdir(conf_dir):
-            return conf_dir
-        else:
-            # NOTE: The prefix is gotten from the path which pyconfig.h is
-            # installed under. Some envs contain it under /usr/include, not
-            # /user/local/include. Then prefix becomes /usr on such envs.
-            # However, etc/tempest is installed under /usr/local and the bove
-            # path logic mismatches. This is a workaround for such envs.
-            return os.path.join(prefix, 'local/etc/tempest')
+        os.makedirs(user_global_path)
+        return user_global_path
 
 
 class TempestInit(command.Command):