openstack/swift
Code Issues Proposed changes

Create a separate doc for audit watcher

Browse Source 
We have a separate doc page for middlewares that pulls the docstring
from each middleware's docstring[0]. This makes it easy to look up the docs
in our documentation and easy to find the middleware doc by looking in
the code of the middleware itself.

This patch does the same with the audit watchers. There is now a page
that generates a list of audit watchers, even though currently it's only
one, and pulls the docs from their docstrings. Giving us an easy way to
maintain each audit watcher doc along with it's code.

[0] - https://docs.openstack.org/swift/latest/middleware.html

Change-Id: I1456aba0158d29fa0a879dcc2dfb13245c45ad16
This commit is contained in:
Matthew Oliver 2021-01-06 16:17:30 +11:00 committed by Pete Zaitcev
parent 587a4bef33
commit 46a6a09d5b
4 changed files with 38 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
doc/source/audit_watchers.rst Normal file
View File

@ -0,0 +1,12 @@
.. _common_audit_watchers:
*********************
Object Audit Watchers
*********************
.. _dark_data:
Dark Data
=========
.. automodule:: swift.obj.watchers.dark_data

24
doc/source/development_watchers.rst
View File

@ -104,23 +104,9 @@ Dark Data Watcher
The watcher API is assumed to be under development. Operators who
need extensions are welcome to report any needs for more arguments
to see_object(). For now, start by copying the provided template watcher
swift/obj/watchers/dark_data.py and see if it is sufficient.
to see_object().
The name of "Dark Data" refers to the scientific hypothesis of Dark Matter,
which supposes that the universe contains a lot of matter than we cannot
observe. The Dark Data in Swift is the name of objects that are not
accounted in the containers.
The experience of running large scale clusters suggests that Swift does
not have any particular bugs that trigger creation of dark data. So,
this is an excercise in writing watchers, with a plausible function.
When enabled, Dark Data watcher definitely drags down the cluster's overall
performance, as mentioned above. Of course, the load increase can be
mitigated as usual, but at the expense of the total time taken by
the pass of auditor.
Finally, keep in mind that Dark Data watcher needs the container
ring to operate, but runs on an object node. This can come up if
cluster has nodes separated by function.
The :ref:`dark_data` watcher has been provided as an example. If an
operator wants to create their own watcher, start by copying
the provided example template ``swift/obj/watchers/dark_data.py`` and see
if it is sufficient.

1
doc/source/index.rst
View File

@ -166,6 +166,7 @@ Source Documentation
object
misc
middleware
audit_watchers
Indices and tables

20
swift/obj/watchers/dark_data.py
View File

@ -23,6 +23,26 @@
# Which is for the best, because of a large performance impact of this.
#
"""
The name of "Dark Data" refers to the scientific hypothesis of Dark Matter,
which supposes that the universe contains a lot of matter than we cannot
observe. The Dark Data in Swift is the name of objects that are not
accounted in the containers.
The experience of running large scale clusters suggests that Swift does
not have any particular bugs that trigger creation of dark data. So,
this is an excercise in writing watchers, with a plausible function.
When enabled, Dark Data watcher definitely drags down the cluster's overall
performance. Of course, the load increase can be
mitigated as usual, but at the expense of the total time taken by
the pass of auditor.
Finally, keep in mind that Dark Data watcher needs the container
ring to operate, but runs on an object node. This can come up if
cluster has nodes separated by function.
"""
import os
import random
import shutil