libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available

Because of the libvirt issue[1], there is a bug[2] that if we set cache mode
whose write semantic is not O_DIRECT (.i.e unsafe, writeback or writethrough),
there will be a problem with the volume drivers
(.i.e nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver,
nova.virt.libvirt.volume.LibvirtNFSVolumeDriver and so on), which designate
native io explicitly.

That problem will generate a libvirt xml for the instance,
whose content contains

```
...
<disk ... >
  <driver ... cache='unsafe/writeback/writethrough' io='native' />
</disk>
...
```
In turn, it will fail to start the instance or attach the disk.

> When qemu is configured with a block device that has aio=native set, but
> the cache mode doesn't use O_DIRECT (i.e. isn't cache=none/directsync or any
> unnamed mode with explicit cache.direct=on), then the raw-posix block driver
> for local files and block devices will silently fall back to aio=threads.
> The blockdev-add interface rejects such combinations, but qemu can't
> change the existing legacy interfaces that libvirt uses today.

[1]: 058384003d
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1086704

Closes-Bug: #1841363
Change-Id: If9acc054100a6733f3659a15dd9fc2d462e84d64

nova/tests/unit/virt/libvirt/test_driver.py

@@ -17893,6 +17893,51 @@ class LibvirtConnTestCase(test.NoDBTestCase,
         drvr._set_cache_mode(fake_conf)
         self.assertEqual('none', fake_conf.driver_cache)
 
+    def _make_fake_conf(self, cache=None):
+        if cache:
+            self.flags(disk_cachemodes=['block=' + cache], group='libvirt')
+        else:
+            self.flags(group='libvirt')
+        drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True)
+        fake_conf = FakeConfigGuestDisk()
+        fake_conf.source_type = 'block'
+        fake_conf.driver_io = 'native'
+        drvr._set_cache_mode(fake_conf)
+        return fake_conf
+
+    def test_set_cache_mode_driver_io_writeback(self):
+        """Tests that when conf.driver_io is 'native' and driver_cache is
+        'writeback', then conf.driver_io is forced to 'threads'
+        """
+        fake_conf = self._make_fake_conf('writeback')
+        self.assertEqual('writeback', fake_conf.driver_cache)
+        self.assertEqual('threads', fake_conf.driver_io)
+
+    def test_set_cache_mode_driver_io_writethrough(self):
+        """Tests that when conf.driver_io is 'native' and driver_cache is
+        'writethrough', then conf.driver_io is forced to 'threads'
+        """
+        fake_conf = self._make_fake_conf('writethrough')
+        self.assertEqual('writethrough', fake_conf.driver_cache)
+        self.assertEqual('threads', fake_conf.driver_io)
+
+    def test_set_cache_mode_driver_io_unsafe(self):
+        """Tests that when conf.driver_io is 'native' and driver_cache is
+        'unsafe', then conf.driver_io is forced to 'threads'
+        """
+        fake_conf = self._make_fake_conf('unsafe')
+        self.assertEqual('unsafe', fake_conf.driver_cache)
+        self.assertEqual('threads', fake_conf.driver_io)
+
+    def test_without_set_cache_mode_driver_io(self):
+        """Tests that when conf.driver_io is 'native' and driver_cache is
+        not set(this is default settings), then conf.driver_io is kept as
+        'native'
+        """
+        fake_conf = self._make_fake_conf()
+        self.assertIsNone(fake_conf.driver_cache)
+        self.assertEqual('native', fake_conf.driver_io)
+
     def test_set_cache_mode_invalid_mode(self):
         self.flags(disk_cachemodes=['file=FAKE'], group='libvirt')
         drvr = libvirt_driver.LibvirtDriver(fake.FakeVirtAPI(), True)

nova/virt/libvirt/driver.py

@@ -590,6 +590,22 @@ class LibvirtDriver(driver.ComputeDriver):
                                                   driver_cache)
             conf.driver_cache = cache_mode
 
+        # NOTE(acewit): If the [libvirt]disk_cachemodes is set as
+        # `block=writeback` or `block=writethrough` or `block=unsafe`,
+        # whose correponding Linux's IO semantic is not O_DIRECT in
+        # file nova.conf, then it will result in an attachment failure
+        # because of the libvirt bug
+        # (https://bugzilla.redhat.com/show_bug.cgi?id=1086704)
+        if ((getattr(conf, 'driver_io', None) == "native") and
+                conf.driver_cache not in [None, 'none', 'directsync']):
+            conf.driver_io = "threads"
+            LOG.warning("The guest disk driver io mode has fallen back "
+                        "from 'native' to 'threads' because the "
+                        "disk cache mode is set as %(cachemode)s, which does"
+                        "not use O_DIRECT. See the following bug report "
+                        "for more details: https://launchpad.net/bugs/1841363",
+                        {'cachemode': conf.driver_cache})
+
     def _do_quality_warnings(self):
         """Warn about potential configuration issues.
 

releasenotes/notes/bug-1841363-fallback-to-threaded-io-when-native-io-is-not-supported-fe56014e9648a518.yaml

@@ -0,0 +1,14 @@
+---
+fixes:
+  - |
+    Since Libvirt v.1.12.0 and the introduction of the `libvirt issue`_ ,
+    there is a fact that if we set cache mode whose write semantic is not
+    O_DIRECT (i.e. "unsafe", "writeback" or "writethrough"), there will
+    be a problem with the volume drivers (i.e. LibvirtISCSIVolumeDriver,
+    LibvirtNFSVolumeDriver and so on), which designate native io explicitly.
+
+    When the driver_cache (default is none) has been configured as neither
+    "none" nor "directsync", the libvirt driver will ensure the driver_io
+    to be "threads" to avoid an instance spawning failure.
+
+    .. _`libvirt issue`: https://bugzilla.redhat.com/show_bug.cgi?id=1086704