Browse Source

Enhance docs rendering; update documentation

Enhanced rendering of docs, expanded introductory section.

Documented `perm` module, alphabetically sorted documenation for
modules, replaced dead recorded demo links with new links,
documented apt package blacklisting capability.

Change-Id: Ifd889efe73287c13d839ab40b1a78ffa357fd00e
Roman Gorshunov 1 month ago
parent
commit
dfdadbe970
3 changed files with 249 additions and 184 deletions
  1. 1
    0
      .gitignore
  2. 36
    6
      README.rst
  3. 212
    178
      doc/source/index.rst

+ 1
- 0
.gitignore View File

@@ -52,6 +52,7 @@ ChangeLog
52 52
 *~
53 53
 .*.swp
54 54
 .*sw?
55
+.vscode/
55 56
 
56 57
 # Files created by releasenotes build
57 58
 releasenotes/build

+ 36
- 6
README.rst View File

@@ -1,22 +1,52 @@
1
+==========
1 2
 Divingbell
2 3
 ==========
3 4
 
5
+|Doc Status|
6
+
7
+Introduction
8
+============
9
+
4 10
 Divingbell is a lightweight solution for:
5 11
 
6
-1. Bare metal configuration management for a few very targeted use cases
7
-2. Bare metal package manager orchestration
12
+1. Bare metal configuration management for a few very targeted use
13
+cases via the following modules:
14
+
15
+- *apparmor*
16
+- *ethtool*
17
+- *exec* (run arbitrary scripts)
18
+- system *limits*
19
+- *mounts*
20
+- permissions (*perm*)
21
+- *sysctl* values
22
+- basic user account management (*uamlite*)
23
+
24
+2. Bare metal package manager orchestration using *apt* module
8 25
 
9 26
 What problems does it solve?
10 27
 ----------------------------
11 28
 
12 29
 The needs identified for Divingbell were:
13 30
 
14
-1. To plug gaps in day 1 tools (e.g., Drydock) for node configuration
31
+1. To plug gaps in day 1 tools (e.g., `Drydock`_) for node configuration
15 32
 2. To provide a day 2 solution for managing these configurations going forward
16 33
 3. [Future] To provide a day 2 solution for system level host patching
17 34
 
35
+.. include-marker
36
+
18 37
 Documentation
19
--------------
38
+=============
39
+
40
+Find more documentation for Divingbell on `Read the Docs`_.
41
+
42
+Further Reading
43
+===============
44
+
45
+`Airship`_.
20 46
 
21
-Find more documentation for Divingbell on
22
-`Read the Docs <https://airship-divingbell.readthedocs.io/>`_.
47
+.. |Doc Status| image:: https://readthedocs.org/projects/airship-divingbell/badge/?version=latest
48
+   :target: https://airship-divingbell.readthedocs.io/
49
+   :alt: Documentation Status
50
+.. _Read the Docs: https://airship-divingbell.readthedocs.io
51
+.. _Drydock: https://airship-drydock.readthedocs.io
52
+.. _Airship: https://www.airshipit.org

+ 212
- 178
doc/source/index.rst View File

@@ -14,139 +14,96 @@
14 14
       License for the specific language governing permissions and limitations
15 15
       under the License.
16 16
 
17
-Divingbell
18
-==========
19
-
20
-Divingbell is a lightweight solution for:
21
-
22
-1. Bare metal configuration management for a few very targeted use cases
23
-2. Bare metal package manager orchestration
24
-
25
-What problems does it solve?
26
-----------------------------
27
-
28
-The needs identified for Divingbell were:
29
-
30
-1. To plug gaps in day 1 tools (e.g., Drydock) for node configuration
31
-2. To provide a day 2 solution for managing these configurations going forward
32
-3. [Future] To provide a day 2 solution for system level host patching
17
+.. include:: ../../README.rst
18
+  :end-before: include-marker
33 19
 
34 20
 Design and Implementation
35
--------------------------
21
+=========================
36 22
 
37
-Divingbell daemonsets run as privileged containers which mount the host
23
+Divingbell DaemonSets run as privileged containers which mount the host
38 24
 filesystem and chroot into that filesystem to enforce configuration and package
39
-state. (The `diving bell <http://bit.ly/2hSXlai>`_ analogue can be thought of as something that descends
40
-into the deeps to facilitate work done down below the surface.)
25
+state, or executes scripts in a namespace of ``systemd`` (PID=1). (The
26
+`diving bell <http://bit.ly/2hSXlai>`_ analogue can be thought of as something
27
+that descends into the deeps to facilitate work done down below the surface).
41 28
 
42
-We use the daemonset construct as a way of getting a copy of each pod on every
29
+We use the DaemonSet construct as a way of getting a copy of each pod on every
43 30
 node, but the work done by this chart's pods behaves like an event-driven job.
44 31
 In practice this means that the chart internals run once on pod startup,
45 32
 followed by an infinite sleep such that the pods always report a "Running"
46
-status that k8s recognizes as the healthy (expected) result for a daemonset.
33
+status that k8s recognizes as the healthy (expected) result for a DaemonSet.
47 34
 
48 35
 In order to keep configuration as isolated as possible from other systems that
49
-manage common files like /etc/fstab and /etc/sysctl.conf, Divingbell daemonsets
50
-manage all of their configuration in separate files (e.g. by writing unique
51
-files to /etc/sysctl.d or defining unique Systemd units) to avoid potential
52
-conflicts. Another example is limit management, Divingbell daemonset writes
53
-separate files to /etc/security/limits.d.
36
+manage common files like ``/etc/fstab`` and ``/etc/sysctl.conf``, Divingbell
37
+DaemonSets manage all of their configuration in separate files (e.g. by writing
38
+unique files to ``/etc/sysctl.d`` or defining unique Systemd units) to avoid
39
+potential conflicts. Another example is ``limits`` management, where Divingbell
40
+DaemonSets write separate files to ``/etc/security/limits.d``.
54 41
 
55
-To maximize robustness and utility, the daemonsets in this chart are made to be
42
+To maximize robustness and utility, the DaemonSets in this chart are made to be
56 43
 idempotent. In addition, they are designed to implicitly restore the original
57 44
 system state after previously defined states are undefined. (e.g., removing a
58 45
 previously defined mount from the yaml manifest, with no record of the original
59 46
 mount in the updated manifest).
60 47
 
61 48
 Lifecycle management
62
---------------------
49
+====================
63 50
 
64
-This chart's daemonsets will be spawned by Armada. They run in an event-driven
65
-fashion: the idempotent automation for each daemonset will only re-run when
51
+This chart's DaemonSets will be spawned by `Armada`_. They run in an event-driven
52
+fashion: the idempotent automation for each DaemonSet only re-runs when
66 53
 Armada spawns/respawns the container, or if information relevant to the host
67 54
 changes in the configmap.
68 55
 
69
-Daemonset configs
70
------------------
71
-
72
-sysctl
73
-^^^^^^
56
+DaemonSet configs
57
+=================
74 58
 
75
-Used to manage host level sysctl tunables. Ex::
76
-
77
-    conf:
78
-      sysctl:
79
-        net/ipv4/ip_forward: 1
80
-        net/ipv6/conf/all/forwarding: 1
81
-
82
-limits
83
-^^^^^^
84
-
85
-Used to manage host level limits. Ex::
86
-
87
-  conf:
88
-    limits:
89
-      nofile:
90
-        domain: 'root'
91
-        type: 'soft'
92
-        item: 'nofile'
93
-        value: '101'
94
-      core_dump:
95
-        domain: '0:'
96
-        type: 'hard'
97
-        item: 'core'
98
-        value: 0
99
-
100
-Previous values of newly set limits are backed up to /var/divingbell/limits
101
-
102
-
103
-mounts
104
-^^^^^^
59
+apparmor
60
+--------
105 61
 
106
-used to manage host level mounts (outside of those in /etc/fstab). Ex::
62
+Used to manage host level apparmor profiles/rules. Ex.::
107 63
 
108 64
     conf:
109
-      mounts:
110
-        mnt:
111
-          mnt_tgt: /mnt
112
-          device: tmpfs
113
-          type: tmpfs
114
-          options: 'defaults,noatime,nosuid,nodev,noexec,mode=1777,size=1024M'
115
-
116
-ethtool
117
-^^^^^^^
118
-
119
-Used to manage host level NIC tunables. Ex::
65
+      apparmor:
66
+        complain_mode: "true"
67
+        profiles:
68
+          profile-1: |
69
+            #include <tunables/global>
70
+              /usr/sbin/profile-1 {
71
+                #include <abstractions/apache2-common>
72
+                #include <abstractions/base>
73
+                #include <abstractions/nis>
120 74
 
121
-    conf:
122
-      ethtool:
123
-        ens3:
124
-          tx-tcp-segmentation: off
125
-          tx-checksum-ip-generic: on
75
+                capability dac_override,
76
+                capability dac_read_search,
77
+                capability net_bind_service,
78
+                capability setgid,
79
+                capability setuid,
126 80
 
127
-uamlite
128
-^^^^^^^
81
+                /data/www/safe/* r,
82
+                deny /data/www/unsafe/* r,
83
+              }
84
+          profile-2: |
85
+            #include <tunables/global>
86
+              /usr/sbin/profile-2 {
87
+                #include <abstractions/apache2-common>
88
+                #include <abstractions/base>
89
+                #include <abstractions/nis>
129 90
 
130
-Used to manage host level local user accounts, their SSH keys, and their sudo
131
-access. Ex::
91
+                capability dac_override,
92
+                capability dac_read_search,
93
+                capability net_bind_service,
94
+                capability setgid,
95
+                capability setuid,
132 96
 
133
-    conf:
134
-      uamlite:
135
-        purge_expired_users: false
136
-        users:
137
-        - user_name: testuser
138
-          user_crypt_passwd: $6$...
139
-          user_sudo: true
140
-          user_sshkeys:
141
-          - ssh-rsa AAAAB3N... key1-comment
142
-          - ssh-rsa AAAAVY6... key2-comment
97
+                /data/www/safe/* r,
98
+                deny /data/www/unsafe/* r,
99
+              }
143 100
 
144 101
 apt
145
-^^^
102
+---
146 103
 
147
-``apt`` daemonset does package management. It is able to install a package of
104
+``apt`` DaemonSet does package management. It is able to install a package of
148 105
 a specific version (or upgrade an existing one to requested version). Version
149
-is optional, and if not provided the latest available package is installed.
106
+is optional, and if not provided, the latest available package is installed.
150 107
 It can also remove packages that were previously installed by divingbell (it is
151 108
 done by excluding the packages you want to remove from the configuration).
152 109
 Here is an example configuration for it::
@@ -158,7 +115,15 @@ Here is an example configuration for it::
158 115
           version: <VERSION1>
159 116
         - name: <PACKAGE2>
160 117
 
161
-It is also possible to provide debconf settings for packages the following
118
+There is a possibility to blacklist packages, e.g. ``telnetd`` and ``nis``::
119
+
120
+    conf:
121
+      apt:
122
+        blacklistpkgs:
123
+        - name: telnetd
124
+        - name: nis
125
+
126
+It is also possible to provide ``debconf`` settings for packages the following
162 127
 way::
163 128
 
164 129
     conf:
@@ -170,10 +135,21 @@ way::
170 135
             question_type: boolean
171 136
             answer: false
172 137
 
138
+ethtool
139
+-------
140
+
141
+Used to manage host level NIC tunables. Ex.::
142
+
143
+    conf:
144
+      ethtool:
145
+        ens3:
146
+          tx-tcp-segmentation: off
147
+          tx-checksum-ip-generic: on
148
+
173 149
 exec
174
-^^^^
150
+----
175 151
 
176
-Used to execute scripts on nodes, ex::
152
+Used to execute scripts on nodes in ``systemd`` (PID=1) namespace, for ex.::
177 153
 
178 154
     exec:
179 155
       002-script2.sh:
@@ -195,112 +171,146 @@ Used to execute scripts on nodes, ex::
195 171
           echo env: $env1 $env2 $env3
196 172
 
197 173
 Scripts are executed in alphanumeric order with the key names used. Therefore
198
-in this example, 001-script1.sh runs first, followed by 002-script2.sh.
174
+in this example, ``001-script1.sh`` runs first, followed by ``002-script2.sh``.
199 175
 Targeting of directives to specific nodes by hostname or node label is
200
-achievable by use of the overrides capability described below.
176
+achievable by the use of the overrides capability described below.
201 177
 
202
-The following set of options are fully implemeneted::
178
+The following set of options is fully implemeneted:
203 179
 
204
-    ``rerun_policy`` may be optionally set to ``always``, ``never``, or
180
+  - ``rerun_policy`` may optionally be set to ``always``, ``never``, or
205 181
     ``once_successfully`` for a given script. That script would always be rerun,
206 182
     never be rerun, or rerun until the first successful execution respectively.
207 183
     Default value is ``always``. This is tracked via a hash of the dict object
208 184
     for the script (i.e. script name, script data, script args, script env, etc).
209 185
     If any of that info changes, so will the hash, and it will be seen as a new
210 186
     object which will be executed regardless of this setting.
211
-
212
-    ``script_timeout`` may optionally be set to the number of seconds to wait for
187
+  - ``script_timeout`` may optionally be set to the number of seconds to wait for
213 188
     script completion before termination. Default value is ``1800`` (30 min).
214
-
215
-    ``rerun_interval`` may be optionally set to the number of seconds to wait
189
+  - ``rerun_interval`` may optionally be set to the number of seconds to wait
216 190
     between rerunning a given script which ran successfully the previous time.
217 191
     Default value is ``infinite``.
218
-
219
-    ``retry_interval`` may be optionally set to the number of seconds to wait
192
+  - ``retry_interval`` may optionally be set to the number of seconds to wait
220 193
     between rerunning a given script which did not run successfully the previous
221 194
     time. Default behavior is to match the ``rerun_interval``.
222 195
 
223
-The following set of options are partially implemeneted::
196
+The following set of options is partially implemeneted:
224 197
 
225
-    ``blocking_policy`` may optionally be set to ``background``, ``foreground``,
198
+  - ``blocking_policy`` may optionally be set to ``background``, ``foreground``,
226 199
     or ``foreground_halt_pod_on_failure`` for a given script. This may be used to
227 200
     run a script in the background (running in parallel, i.e. non-blocking) or
228 201
     in the foreground (blocking). In either case, a failure of the script does
229
-    not cause a failure (crashloop) of the pod. The third option may be used
202
+    not cause a failure (CrashLoop) of the pod. The third option may be used
230 203
     where the reverse behavior is desired (i.e., it would not proceed with
231 204
     running the next script in the sequence until the current script ran
232 205
     successfully). ``background`` option is not yet implemeneted. Default value
233
-    Deafult value is ``foreground``.
206
+    is ``foreground``.
234 207
 
235
-The following set of options are not yet implemeneted::
208
+The following set of options is not yet implemeneted:
236 209
 
237
-    ``rerun_interval_persist`` may be optionally set to ``false`` for a given
210
+  - ``rerun_interval_persist`` may optionally be set to ``false`` for a given
238 211
     script. This makes the script execute on pod/node startup regardless of the
239 212
     interval since the last successful execution. Default value is ``true``.
240
-
241
-    ``rerun_max_count`` may be optionally set to the maximum number of times a
213
+  - ``rerun_max_count`` may optionally be set to the maximum number of times a
242 214
     succeeding script should be retried. Successful exec count does not persist
243 215
     through pod/node restart. Default value is ``infinite``.
244
-
245
-    ``retry_interval_persist`` may be optionally set to ``false`` for a given
216
+  - ``retry_interval_persist`` may optionally be set to ``false`` for a given
246 217
     script. This makes the script execute on pod/node startup, regardless of the
247 218
     time since the last execution. Default value is ``true``.
248
-
249
-    ``retry_max_count`` may be optionally set to the maximum number of times a
219
+  - ``retry_max_count`` may optionally be set to the maximum number of times a
250 220
     failing script should be retried. Failed exec count does not persist
251 221
     through pod/node restart. Default value is ``infinite``.
252 222
 
253
-apparmor
254
-^^^^^^^^
223
+limits
224
+------
255 225
 
256
-Used to manage host level apparmor profiles/rules, Ex::
226
+Used to manage host level limits. Ex.::
227
+
228
+  conf:
229
+    limits:
230
+      nofile:
231
+        domain: 'root'
232
+        type: 'soft'
233
+        item: 'nofile'
234
+        value: '101'
235
+      core_dump:
236
+        domain: '0:'
237
+        type: 'hard'
238
+        item: 'core'
239
+        value: 0
240
+
241
+Previous values of newly set limits are backed up to ``/var/divingbell/limits``.
242
+
243
+mounts
244
+------
245
+
246
+Used to manage host level mounts (outside of those in ``/etc/fstab``). Ex.::
257 247
 
258 248
     conf:
259
-      apparmor:
260
-        complain_mode: "true"
261
-        profiles:
262
-          profile-1: |
263
-            #include <tunables/global>
264
-              /usr/sbin/profile-1 {
265
-                #include <abstractions/apache2-common>
266
-                #include <abstractions/base>
267
-                #include <abstractions/nis>
249
+      mounts:
250
+        mnt:
251
+          mnt_tgt: /mnt
252
+          device: tmpfs
253
+          type: tmpfs
254
+          options: 'defaults,noatime,nosuid,nodev,noexec,mode=1777,size=1024M'
268 255
 
269
-                capability dac_override,
270
-                capability dac_read_search,
271
-                capability net_bind_service,
272
-                capability setgid,
273
-                capability setuid,
256
+perm
257
+----
274 258
 
275
-                /data/www/safe/* r,
276
-                deny /data/www/unsafe/* r,
277
-              }
278
-          profile-2: |
279
-            #include <tunables/global>
280
-              /usr/sbin/profile-2 {
281
-                #include <abstractions/apache2-common>
282
-                #include <abstractions/base>
283
-                #include <abstractions/nis>
259
+Used to manage permissions. Ex.::
284 260
 
285
-                capability dac_override,
286
-                capability dac_read_search,
287
-                capability net_bind_service,
288
-                capability setgid,
289
-                capability setuid,
261
+    conf:
262
+      perms:
263
+        -
264
+          path: '/etc/shadow'
265
+          owner: 'root'
266
+          group: 'shadow'
267
+          permissions: '0640'
268
+        -
269
+          path: '/etc/passwd'
270
+          owner: 'root'
271
+          group: 'root'
272
+          permissions: '0644'
273
+
274
+Module supports ``rerun_policy`` and ``rerun_interval`` options (like in
275
+``exec`` module). Previous values of newly set permissions are backed up to
276
+``/var/divingbell/perm``.
290 277
 
291
-                /data/www/safe/* r,
292
-                deny /data/www/unsafe/* r,
293
-              }
278
+sysctl
279
+------
280
+
281
+Used to manage host level sysctl tunables. Ex.::
282
+
283
+    conf:
284
+      sysctl:
285
+        net/ipv4/ip_forward: 1
286
+        net/ipv6/conf/all/forwarding: 1
287
+
288
+uamlite
289
+-------
290
+
291
+Used to manage host level local user accounts, their SSH keys, and their sudo
292
+access. Ex.::
293
+
294
+    conf:
295
+      uamlite:
296
+        purge_expired_users: false
297
+        users:
298
+        - user_name: testuser
299
+          user_crypt_passwd: $6$...
300
+          user_sudo: true
301
+          user_sshkeys:
302
+          - ssh-rsa AAAAB3N... key1-comment
303
+          - ssh-rsa AAAAVY6... key2-comment
294 304
 
295 305
 Operations
296
-----------
306
+==========
297 307
 
298 308
 Setting apparmor profiles
299
-^^^^^^^^^^^^^^^^^^^^^^^^^
309
+-------------------------
300 310
 
301 311
 The way apparmor loading/unloading implemented is through saving
302 312
 settings to a file and than running ``apparmor_parser`` command.
303
-The daemonset supports both enforcement and complain mode,
313
+The DaemonSet supports both enforcement and complain mode,
304 314
 enforcement being the default. To request complain mode for the
305 315
 profiles, add ``complain_mode: "true"`` nested under apparmor entry.
306 316
 
@@ -330,7 +340,7 @@ net_bind_service), but not both. Such problems are hard to debug, so
330 340
 caution needed while setting configs up.
331 341
 
332 342
 Setting user passwords
333
-^^^^^^^^^^^^^^^^^^^^^^
343
+----------------------
334 344
 
335 345
 Including ``user_crypt_passwd`` to set a user password is optional.
336 346
 
@@ -352,7 +362,7 @@ network access is unavailable, console username/password access will be the only
352 362
 login option.
353 363
 
354 364
 Setting user sudo
355
-^^^^^^^^^^^^^^^^^
365
+-----------------
356 366
 
357 367
 Including ``user_sudo`` to set user sudo access is optional. The default value
358 368
 is ``false``.
@@ -361,7 +371,7 @@ At least one user must be defined with sudo access in order for the built-in
361 371
 ``ubuntu`` account to be disabled.
362 372
 
363 373
 SSH keys
364
-^^^^^^^^
374
+--------
365 375
 
366 376
 Including ``user_sshkeys`` for defining one or more user SSH keys is optional.
367 377
 
@@ -380,7 +390,7 @@ At least one user must be defined with an SSH key and sudo in order for the
380 390
 built-in ``ubuntu`` account to be disabled.
381 391
 
382 392
 Purging expired users
383
-^^^^^^^^^^^^^^^^^^^^^
393
+---------------------
384 394
 
385 395
 Including the ``purge_expired_users`` key-value pair is optional. The default
386 396
 value is ``false``.
@@ -392,23 +402,23 @@ maintain UID consistency (in the event the same accounts gets re-added later,
392 402
 they regain access to their home directory files without UID mismatching).
393 403
 
394 404
 Node specific configurations
395
-----------------------------
405
+============================
396 406
 
397
-Although we expect these daemonsets to run indiscriminately on all nodes in the
407
+Although we expect these DaemonSets to run indiscriminately on all nodes in the
398 408
 infrastructure, we also expect that different nodes will need to be given a
399 409
 different set of data depending on the node role/function. This chart supports
400 410
 establishing value overrides for nodes with specific label value pairs and for
401 411
 targeting nodes with specific hostnames. The overridden configuration is merged
402 412
 with the normal config data, with the override data taking precedence.
403 413
 
404
-The chart will then generate one daemonset for each host and label override, in
405
-addition to a default daemonset for which no overrides are applied.
406
-Each daemonset generated will also exclude from its scheduling criteria all
407
-other hosts and labels defined in other overrides for the same daemonset, to
408
-ensure that there is no overlap of daemonsets (i.e., one and only one daemonset
414
+The chart will then generate one DaemonSet for each host and label override, in
415
+addition to a default DaemonSet for which no overrides are applied.
416
+Each DaemonSet generated will also exclude from its scheduling criteria all
417
+other hosts and labels defined in other overrides for the same DaemonSet, to
418
+ensure that there is no overlap of DaemonSets (i.e., one and only one DaemonSet
409 419
 of a given type for each node).
410 420
 
411
-Overrides example with sysctl daemonset::
421
+Overrides example with sysctl DaemonSet::
412 422
 
413 423
     conf:
414 424
       sysctl:
@@ -459,7 +469,8 @@ Caveats:
459 469
    contained both of the defined labels.
460 470
 
461 471
 Dev Environment with Vagrant
462
-----------------------------
472
+============================
473
+
463 474
 The point of Dev env to prepare working environment for development.
464 475
 
465 476
 Vagrantfile allows to run on working copy with modifications
@@ -471,6 +482,29 @@ but do not delete the pods and other stuff. You have:
471 482
 3. your not committed test runs in prepared env
472 483
 
473 484
 Recorded Demo
474
--------------
475
-
476
-A recorded demo of using Divingbell can be found `here <https://asciinema.org/a/beJQZpRPdOctowW0Lxkxrhz17>`_.
485
+=============
486
+
487
+Here are a few demo recording of Divingbell in action:
488
+
489
+- Divingbell limits module:
490
+  `link <https://asciinema.org/a/209619>`__,
491
+  `link <https://asciinema.org/a/211450>`__ (with overrides)
492
+- Divingbell sysctl module:
493
+  `link <https://asciinema.org/a/209667>`__,
494
+  `link <https://asciinema.org/a/211433>`__ (with overrides)
495
+- Divingbell perms module:
496
+  `link <https://asciinema.org/a/209509>`__,
497
+  `link <https://asciinema.org/a/213125>`__ (with overrides)
498
+
499
+Further Reading
500
+===============
501
+
502
+`Airship`_.
503
+
504
+.. |Doc Status| image:: https://readthedocs.org/projects/airship-divingbell/badge/?version=latest
505
+   :target: https://airship-divingbell.readthedocs.io/
506
+   :alt: Documentation Status
507
+.. _Read the Docs: https://airship-divingbell.readthedocs.io
508
+.. _Drydock: https://airship-drydock.readthedocs.io
509
+.. _Airship: https://www.airshipit.org
510
+.. _Armada: https://airship-armada.readthedocs.io

Loading…
Cancel
Save