Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.
This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.
Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.
As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.
The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?
Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.
Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).
References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.
THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS
With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.
So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.
`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.
The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808
It was deprecated and we discussed on this topic in Denver PTG
for Queen cycle. Main motivation for this work is that deprecated
post_as_copy option and its gate blocks future symlink work.
Change-Id: I411893db1565864ed5beb6ae75c38b982a574476
Add support for a 2+1 EC policy to be optionally used as default
policy when running in process functional tests.
The EC policy may be selected by setting the env var:
SWIFT_TEST_IN_PROCESS_CONF_LOADER=ec tox
when running .functests, or by using the new tox test env:
tox -e func-ec
Change-Id: I02e3553a74a024efdab91dcd609ac1cf4e4f3208
Make development guidelines consistent with recent
changes to tox envs, specifically the removal of
"-in-process" from some test env names [1] and the
removal of the "func-fast-post" test env [2].
[1] Related-Change: I02477d81b836df71780942189d37d616944c4dce
[2] Related-Change: I6faf8fcfa0a1d96aaf0f5e0ad2106b2b416da22f
Change-Id: I08b92c005ee50beff09a92b4331dd7dbeed79bde
Change [1] requires pip >= 7.0.1. In general, test
environments will need to have pip and virtualenv in line
with global requirements. Add a reference to the global
requirements rather than specifying versions that will
inevitably become stale.
Also adds inline literal markup to all occurences of "tox"
for consistency.
[1] Related-Change: I7c60be623b4340ee34ae1aa520f17b303348811d
Change-Id: I7730f22c594a3521973cb9ff264a7d50f2b86a1a
Depends-On: I45eee626713438af5fc676f2b5f636d7ec23f7be
added comments on how to run in_process and specific
test cases
Change-Id: I485755996b15753323d30de09914d35e262fcedc
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Running functional tests in the in-process mode uses
the default value for the pipeline. This patch adds support
to specify the SWIFT_TEST_IN_PROCESS_CONF_LOADER variable
to point to a labeled function that changes the proxy
configuration for the functional test.
The patch also adds a new tox environment
func-in-process-encryption
which runs with the environment variable
SWIFT_TEST_IN_PROCESS_CONF_LOADER=encryption
The motivation for this change is to put support in place for an
upstream CI job that will functionally test using encryption
middleware in the pipeline. The gate job is proposed at:
https://review.openstack.org/#/c/348292/
Change-Id: I15c4b20f1d2be57ae21c69c614f6a9579145bee9
Add an other-requirements.txt file containing a cross-platform list of
dependencies needed for running included tox-based tests. Also include a
tox environment for convenience calling the bindep[*] utility to list
any missing system requirements.
Document bindep and other-requirements usage.
This change is self-testing.
For other-requirements.txt see also
http://docs.openstack.org/infra/manual/drivers.html#package-requirements
[*] http://docs.openstack.org/infra/bindep/
Change-Id: Iea6f5fecba3b7cb9f6dac7029c0f17fc31fc0e3c
Running functional tests in the in-process mode uses
the default value for proxy-server object_post_as_copy,
which is True. This patch adds support for an environment
variable to change this value to False so that the fast-post
mode is tested when running in-process functional tests.
The patch also adds a new tox environment func-in-process-fast-post
which forces in-process functional testing with
object_post_as_copy=False.
The motivation for this change, apart from enabling configurable
local testing, is to put support in place for an upstream CI job
that will actually functionally test the fast-post mode, which is
otherwise only covered by unit tests.
There are currently two gate jobs that run the functional tests:
- gate-swift-dsvm-functional runs the tests *twice* against a
devstack swift service, once using tempauth and once using
keystoneauth. The devstack swift service uses the default
object_post_as_copy=True.
- gate-swift-tox-func runs the func tests in in-process mode
which also uses tempauth and object_post_as_copy=True. This
duplicates one of the config scenarios above.
With this change either the gate-swift-tox-func job or a new job
could run the functional tests using object_post_as_copy=False.
Change-Id: Ia37f6df1dc38e44ef7404fbf0a52f6fc22fae0c2
The doc for these sections was missing because of an rst error - the
source is there in rst file but didn't make it into the html output.
Add doc for per_diff and max_diffs in account and container doc sections.
Also, fix a bunch of other sphinx build errors and most of the warnings.
Change-Id: If9ed2619b2f92c6c65a94f41d8819db8726d3893
The instructions suggest updating the ownership of the swift.egg-info
directory may be required and provide a cmd to do so. Said cmd assumes user
'swift' exists, but tox will require the current user to have access to the
directory to perform updates, moreover user swift may not exist at all.
Change-Id: I63951970553c086ff4f1ac65a81559ba69166494
When functional tests are run in tox and an exception is raised when connecting
to Swift (for example: Swift not running, missing python-keystoneclient package
used by python-swiftclient) 0 tests are executed, but tox returns a success.
An exception is raised during tests, caused by a missing python-keystoneclient
in python-swiftclient. Instead of adding python-keystoneclient as a dependency
in python-swiftclient the package is added to the test-requirements.txt in Swift
itself. Note that adding python-keystoneclient to the test-requirements in
python-swiftclient is not sufficient (it's already in there).
The exception in setup_package() is catched by the openstack.nose_plugin, thus
disabling this plugin for now as well.
Also fixing two test errors seen on the gate regarding the tempurl middleware.
There was also an update to tox, environment variables were no longer passed
with versions >= 2.0 (http://tox.readthedocs.org/en/latest/changelog.html).
Swift test environment variables have been added to the passenv to re-enable the
former behavior, as well as environment variables required to pass proxy
settings.
This also led to skipped tempauth tests, and together with the missing
python-keystoneclient no tests were executed.
Related-Bug: 1461440
Related-Bug: 1455102
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: Ideea071017d04912c60ed0bc76532adbb446c31d
Add ability to select a storage policy when running
the functional tests by defining the environment variable
SWIFT_TEST_POLICY.
This patch was first introduced by Janie Richling in
feature/ec: https://review.openstack.org/#/c/166097
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: Id96354546088df137def0ae7fd434baf64ed0c51
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Currently the in-process tests build a 2 replica, 4 partition,
2 device object ring. This patch allows an alternative policy
and ring to be specified for testing via environment variables
that may optionally be set.
SWIFT_TEST_IN_PROCESS_CONF_DIR - This points the test setup to a
directory which may have a swift.conf file and ring file. The
test setup will then prefer these conf files over the samples
in '/etc'.
SWIFT_TEST_POLICY - This causes the in-process test to
use the specified policy from the swift.conf file and its
associated ring for testing (first copying the conf and ring file
and modifying device parameters to suit in-process testing). If
not set, the tests will use the default policy.
The in-process tests now start sufficient object servers for the
ring file being tested against.
This should allow in-process functional testing of various policies
and rings (e.g. EC policies) without needing to reconfigure an SAIO
for each test scenario.
The refactoring of the in_process test setup code should also
allow easier addition of other 'hard-coded' test policies/rings
in the future.
Change-Id: I24f5a13de3d296b400da1691dcb53423a9f8a463
Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.
It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.
DocImpact
Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>
dnspython was added recently to the list of requirments.
This patch completes the change:
1. The new requirement is documented
2. The need for tox --recreate is documented
(assuming there is no way to automate 2 above)
Change-Id: I49f0cb4b01b79dd7cf4329984a2f8416378ebe19
Fixes: Bug #1176627
Replaced GA code for cross-domain tracking.
Patchset addresses reviewer's comments
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright
Adds current year to each Sphinx-built page.
Addresses only the docs copyright attribution, not code files.
Change-Id: Ib90fd1c92c8fafce2db821bc2b17cef1377cfc1e
* Adds tox config
- based on the config from python-quantumclient and updated for
test, pep8 and coverage execution as per nova's run_tests.sh.
* Adds nosetests defaults in setup.cfg
* Adds runtime dependencies in tools/pip-requires
- dependencies were gathered by referencing the packages used in
creation of a Swift All In One. Versions were determined by
checking the swift-core/trunk ppa or, failing that, the version
available in lucid.
* Adds test dependencies in tools/test-requires
* Updates swift/common/middleware/formpost.py for pep8 compliance
* Adds instructions for executing the tests with Tox to the
developer_guidelines
* Adds instructions for installing openstack.nose_plugin to
developer_saio
* Fixes bug 909177
Change-Id: I5407924d2181e9ab335aaf76bf30c8d40deccbb4
