Previously, if a cluster's combined configured max_meta_name_length and
max_meta_value_length constraints were larger than the configured
max_meta_overall_size, we would accidentally go over the overall size
while intending to just test being exactly at the value length-limit.
Change-Id: I42a5287011509e5b43959aab060f9ec7405ae5b9
This patch will add more granularity to shard operation cache or
backend metrics, and then remove some of existing and duplicated
metrics.
Before this patch, related metrics are:
1.shard_<op>.cache.[hit|miss|skip]
2.shard_<op>.backend.<status_int>
where op is 'listing' or 'updating'.
With this patch, they are going to become:
1.shard_<op>.infocache.hit
cache hits with infocache.
2.shard_<op>.cache.hit
cache hits with memcache.
3.shard_<op>.cache.[miss|bypass|skip|force_skip|disabled|error]
.<status_int>
Those are operations made to backend due to below reasons.
miss: cache misses.
bypass: metadata didn't support a cache lookup
skip: the selective skips per skip percentage config.
force_skip: the request with 'x-newest' header.
disabled: memcache is disabled.
error: memcache connection error.
For each kind of operation metrics, suffix <status_int> will
count operations with different status. Then a sum of all
status sub-metrics will the total metrics of that operation.
UpgradeImpact
=============
Metrics dashboard will need updates to display those changed metrics
correctly, also infocache metrics are newly added, please see above
message for all changes needed.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ib8be30d3969b4b4808664c43e94db53d10e6ef4c
Depending on test order (and possibly whether there were earlier
failures?) the new tests may trip KeyErrors when trying to get
s3_access_key values. Solution seems to be defining
setUpModule() / tearDownModule() like other functional tests.
Also fix up some Content-MD5 handling; if we're using pre-signed URLs,
we can't provide a Content-MD5.
Change-Id: Ifce72ec255b1b618b9914ce5785d04ee0ebd3b8c
Related-Change: I84494123cfc85e234098c554ecd3e77981f8a096
Previously, clients could use XML external entities (XXEs) to read
arbitrary files from proxy-servers and inject the content into the
request. Since many S3 APIs reflect request content back to the user,
this could be used to extract any secrets that the swift user could
read, such as tempauth credentials, keymaster secrets, etc.
Now, disable entity resolution -- any unknown entities will be replaced
with an empty string. Without resolving the entities, the request is
still processed.
[CVE-2022-47950]
Closes-Bug: #1998625
Co-Authored-By: Romain de Joux <romain.de-joux@ovhcloud.com>
Change-Id: I84494123cfc85e234098c554ecd3e77981f8a096
Probe test to produce a scenario where a parent container is stuck
at sharding because of a gap in shard ranges. And the gap is caused
by deleted child shard range which finishes sharding before its
parent does.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I73918776ed91b19ba3fd6deda2fe4ca2820f4dbf
We've seen S3 clients expecting to be able to send request lines like
GET https://cluster.domain/bucket/key HTTP/1.1
instead of the expected
GET /bucket/key HTTP/1.1
Testing against other, independent servers with something like
( echo -n $'GET https://www.google.com/ HTTP/1.1\r\nHost: www.google.com\r\nConnection: close\r\n\r\n' ; sleep 1 ) | openssl s_client -connect www.google.com:443
suggests that it may be reasonable to accept them; the RFC even goes so
far as to say
> To allow for transition to the absolute-form for all requests in some
> future version of HTTP, a server MUST accept the absolute-form in
> requests, even though HTTP/1.1 clients will only send them in
> requests to proxies.
(See https://datatracker.ietf.org/doc/html/rfc7230#section-5.3.2)
Fix it at the protocol level, so everywhere else we can mostly continue
to assume that PATH_INFO starts with a / like we always have.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I04012e523f01e910f41d5a41cdd86d3d2a1b9c59
pytest still complains about some 20k warnings, but the vast majority
are actually because of eventlet, and a lot of those will get cleaned up
when upper-constraints picks up v0.33.2.
Change-Id: If48cda4ae206266bb41a4065cd90c17cbac84b7f
Once a shard container has been created as part of the sharder cycle it
pulls the shards own_shard_range, updates the object_count and
bytes_used and pushes this to the root container. The root container can
use these to display the current container stats.
However, it is not until a shard gets to the CLEAVED state, that it
holds enough information for it's namespace, so before this the number
it returns is incorrect. Further, when we find and create a shard, it
starts out with the number of objects, at the time, that are expected to
go into them. This is better answer then, say, nothing.
So it's better for the shard to send it's current own_shard_range but
don't update the stats until it can be authoritive of that answer.
This patch adds a new SHARD_UPDATE_STAT_STATES that track what
ShardRange states a shard needs to be in in order to be responsible,
current definition is:
SHARD_UPDATE_STAT_STATES = [ShardRange.CLEAVED, ShardRange.ACTIVE,
ShardRange.SHARDING, ShardRange.SHARDED,
ShardRange.SHRINKING, ShardRange.SHRUNK]
As we don't want to update the OSR stats and the meta_timestmap, also
move tombstone updates to only happen when in a SHARD_UPDATE_STAT_STATES
state.
Change-Id: I838dbba3c791fffa6a36ffdcf73eceeaff718373
nose has not seen active development for many years now. With py310, we
can no longer use it due to import errors.
Also update lower contraints
Closes-Bug: #1993531
Change-Id: I215ba0d4654c9c637c3b97953d8659ac80892db8
We need to support the aforementioned headers in our s3 apis
and raise an InvalidArgumentError if a s3 client makes a request
Change-Id: I2c5b18e52da7f33b31ba386cdbd042f90b69ef97
Previously, when fetching a shard range from a container DB using
ContainerBroker.get_own_shard_range(), the stats of the returned shard
range were updated as a side effect. However, the stats were not
persisted in the own shard range row in the DB. Often the extra DB
queries to get the stats are unnecessary because we don't need
up-to-date stats in the returned shard range. The behavior also leads
to potential confusion because the object stats of the returned shard
range object are not necessarily consistent with the object stats of
the same shard range in the DB.
This patch therefore removes the stats updating behavior from
get_own_shard_range() and makes the stats updating happen as an
explicit separate operation, when needed. This is also more consistent
with how the tombstone count is updated.
Up-to-date own shard range stats are persisted when a container is
first enabled for sharding, and then each time a shard container
reports its stats to the root container.
Change-Id: Ib10ef918c8983ca006a3740db8cfd07df2dfecf7
Some subcommands of 'swift-manage-shard-ranges' prompt a user before
taking action. For example, 'find_and_replace --enable' will warn a
user and ask for input if existing shard ranges are found. This patch
improves user input handling to exit gracefully (with err code 3) if
either an EOFError or KeyboardInterrupt is raised while waiting for
user input.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I75c8acd372d449b8a4531ff816d839dfe40d8a56
The test assertion assumes the test environment has exactly one
policy, but the test setup uses whatever number of policies are
currently in POLICIES. This can cause the test to fail in environments
where more than one policy is configured.
Fix the assertion to also use the actual number of policies.
Change-Id: I8fecc583f211913144663c85ab27ad448b5600f8
Add coverage for container the GET path that attempts to use memcache.
When memcache is not available the backend requests will reverse shard
ranges if required by the request params. When memcache is available
the proxy sends a 'X-Backend-Override-Shard-Name-Filter' header to
instruct the backend to always return ShardRanges in their natural
order. Add test coverage for the latter case.
Also add a last_modified item to the shard range dicts in mock
response bodies to be consistent with the real server response.
Change-Id: Ic0454c5f1d37a84258e43427cfe7cd6dfced285b
During normal sharding, parent shard and child shard ranges
may have transient overlaps which will disapear when done.
To avoid this transient overlap shows up in the warning log,
when sharder audits root container, it will not emit
'overlapping ranges in state ...' warning when overlapping
shard ranges are parent-children during the past time period
of reclaim_age.
Change-Id: I88bce3c178923457e3887790fa58aad56086b1e6
The warning string should refer to account not container.
Also, add the account name to the warning string.
Change-Id: I6e7058eb9f5ed4298b5055d46dafb66d24ec5d09
We've seen shards become stuck while sharding because they had
incomplete or stale deleted shard ranges. The root container had more
complete and useful shard ranges into which objects could have been
cleaved, but the shard never merged the root's shard ranges.
While the sharder is auditing shard container DBs it would previously
only merge shard ranges fetched from root into the shard DB if the
shard was shrinking or the shard ranges were known to be children of
the shard. With this patch the sharder will now merge other shard
ranges from root during sharding as well as shrinking.
Shard ranges from root are only merged if they would not result in
overlaps or gaps in the set of shard ranges in the shard DB. Shard
ranges that are known to be ancestors of the shard are never merged,
except the root shard range which may be merged into a shrinking
shard. These checks were not previously applied when merging
shard ranges into a shrinking shard.
The two substantive changes with this patch are therefore:
- shard ranges from root are now merged during sharding,
subject to checks.
- shard ranges from root are still merged during shrinking,
but are now subjected to checks.
Change-Id: I066cfbd9062c43cd9638710882ae9bd85a5b4c37
There are three new metrics added:
1. 'error_limiter.incremented_limit', when one node has accumulated
enough errors to trigger error suppression.
2. 'error_limiter.forced_limit', when one node runs into serious
error like Insufficient Storage.
3. 'error_limiter.is_limited', when one node is error limited
during node selection process due to suppression_limit.
UpgradeImpact:
There is change of log format when 'ERROR Insufficient Storage' is
raised, from '%(msg)s %(node)s' to
'Node will be error limited from now: %s, error: %s'.
Change-Id: Id9bec9b46dad82163517fd71cfdda5b751464588
This patch introduces a check_response method to proxy server
Application class. The method provides common checking and reporting
of backend response errors (i.e. responses with status code >=
500). This checking was previously duplicated in proxy controller
classes.
The formatting and content of error logging messages is now more
unform, which does result in changes to some logged error messages.
Change-Id: I4b935a47f39642c11364f2a8d50a5f2ebb3b84f1
Sometimes we'll get back a 503 on the initial attempt, though the delete
succeeded on the backend. Then when the client automatically retries, it
gets back a 404.
Change-Id: I6d8d5af68884b08e22fd8a332f366a0b81acb7ed
Add more logs for each important cleaving step to help
investigate some slow cleaving processes.
Also fixed some misleading test names.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I4727dcb6268c68c04d86e109aa14f54234a656f9
Tim Burke