The swift functional tests use a config which requires keystone ports,
we're about to make those go away. This exposes the actual auth_uri to
swift for consumption.
Change-Id: I5868dfdb8e5f0972ba04e359d212b04351502436
Currently Devstack starts all Swift services, including those
in charge of "consistency convergence" (remember Swift is eventually
consistent), data scrubbing, hard-deletion (*-reaper services)
cleanup.
But when running with Replication Factor 1 some of those services
are not needed at all. Besides, the fonctionnalities provided by
some of these services are not tested at all (neither in Tempest
nor in Swift functional tests).
Thus, in light of saving some Mo of RAM, this patch introduces a config
flag to start only a minimal set of Swift services, just what's required
to make all of our current tests pass.
The default value for this new config flag is set to start all services,
that is to maintain Devstack's current behavior.
For sake of completeness, here is the list of services that are not
going to be started is the config flag is toggled, and the associated RSS
according to our peakmem_tracker
40004 swift-object-replicator /etc/swift/object-server/1.conf
34320 swift-container-replicator /etc/swift/container-server/1.conf
33584 swift-object-auditor /etc/swift/object-server/1.conf
33328 swift-object-reconstructor /etc/swift/object-server/1.conf
31936 swift-object-updater /etc/swift/object-server/1.conf
31492 swift-account-reaper /etc/swift/account-server/1.conf
31076 swift-account-replicator /etc/swift/account-server/1.conf
29540 swift-container-updater /etc/swift/container-server/1.conf
29220 swift-account-auditor /etc/swift/account-server/1.conf
29036 swift-container-auditor /etc/swift/container-server/1.conf
So we are looking at saving at most ~350Mo of RAM (could be less
because RSS doesn't account for shared memory).
A follow-up patch will soon be proposed in devstack-gate to not run
those additional services in our Gate jobs.
Change-Id: I8a0d03ac0296a74e38efd185beb8513866eaf0c4
During the PTG there was a discussion that the screen developer
workflow wasn't nearly as useful as it once was. There were now too
many services to see them all on one screen, and one of the most
common service restart scenarios was not restarting one service, but a
bunch to get code to take effect.
This implements a 3rd way of running services instead of direct
forking via bash, or running under screen, which is running as systemd
units.
Logging is adjusted because it's redundant to log datetime in oslo.log
when journald has that.
Swift needed to have services launched by absolute path to work.
This is disabled by default, but with instructions on using it. The
long term intent is to make this the way to run devstack, which would
be the same between both the gate and local use.
Some changes were also needed to run_process to pass the run User
in. A hack around the keystone uwsgi launcher was done at the same
time to remove a run_process feature that only keystone uwsgi uses.
Change-Id: I836bf27c4cfdc449628aa7641fb96a5489d5d4e7
The proliferation of internal/admin endpoints is mostly legacy and
based on some specific deployment patterns. These are not used by
everyone, and for the devstack case aren't really that useful. We
should simplify our service catalog down to the minimum we need for
development.
Change-Id: Ided7a65c81b3a0b56f0184847fc82e17c29a771e
* iniuncomment followed by iniset for reseller_prefix just adds a
duplicate line in the config file that configparser does not like
so just remove the uncomment
* fall back to http:// url for glance->swift keystone authentication
* insecure flag to talk to swift
Depends-On: I51d56d16a5b175bd45dee09edc0b2748d72a5d06
Change-Id: I02ed01e20f8dce195c51273e8384130af53384ce
This commit switches how scripts we use to launch the installed version
in the path. Previously the scripts were manually executed in the source
repo, but this has issues if you're trying to run with py3 in a system
where python == py2. Setuptools already does the shebang magic for us
at install time, so we just need to use the installed version of the
script.
Change-Id: Iaa4d80ec607a2aa200400330e16cad3a4ca782ac
Stud is now abandonware (see https://github.com/bumptech/stud) and is
not packaged in xenial. Lets use Apache for SSL termination since its
there already.
Change-Id: Ifcba410f5969521e8b3d30f02795541c1661f83a
lib/keystone sets KEYSTONE_AUTH_URI and KEYSTONE_SERVICE_URI that
other projects should use rather than building the URL themselves.
This will allow us to more easily drop the port altogether.
Change-Id: I7467aae680215f3045d32a088af2187e1eba8169
This patch enables account management by default in Swift. This will be
leveraged by Tempest test cases validating account management APIs.
Depends-On: Id29f5ca48f92cd139535be7064107b8a61b02856
Change-Id: Ic01432939ed9b4cf0cbf20e3244d4d76847f539f
This reverts commit 7d1ec430046256ac3f536f4920a43936e28ef0c4.
This broke the sahara and layer4 dsvm jobs. The layer4 job
is voting on tempest changes so tempest is also broken.
Change-Id: Ide69f10cd85bf7ff0d86bc8cba56dedd26850362
Partial-Bug: #1573868
We really should only have code that create endpoints once, making all
osc calls get_or_set adds 3 seconds per call for no really good
reason.
This also stops creating the internal endpoints in the service
catalog. It's a pattern that we're trying not to propogate, so lets
not have it in devstack any more.
Change-Id: Ia8cefe43753900d62117beae330db46deb6a9fc9
Make it possible to construct the service users in their own seperate
domain. Changing this away from Default will not work for everyone yet,
though it does work for basic service interaction however enabling it
will allow us to start testing and hopefully gating that services aren't
relying on v2 only concepts.
Change-Id: I7e73df5dd1caabf355783da2bc0f3007ade92fba
If the variable SWIFT_STORAGE_IPS contains a space-separated list of
IPs, we can use this to create consistent rings across all proxy and
storage nodes.
Change-Id: If9307196dc7e74e4a842c95503958ae2d7f7acc7
This sets all the internal variables and service users that are set to
use project instead of tenant for clarity.
Change-Id: I4aa833bac2ee2281c5f2881f7ae1fd8e7c759f74
Currently Devstack (and devstack gate) uses the default
max_file_size of for Swift (5GB). However the loopback file
is only 1 or 2 GB is size.
We are looking at setting a default FALLOCATE_RESERVE in swift
https://review.openstack.org/#/c/288011.
Because of this our max file size test fails due to the max_file_size
and devstacks loopback size being too small.
This patch sets this to a more sane size by default inside DevStack.
See the gerrit link above for more details.
Tempest uses a loopback of 6GB, so in this case the swift default of 5GB
should be fine.
Change-Id: If09eab7d16ae67bd252020e00e8812ff252f065b
This replaces the use of TENANT variables with PROJECT ones during the
initial setup. The openrc will still export a OS_TENANT_NAME because
many tools (cinderclient, glanceclient amoung them) will not function
without it. We warn when we do that.
Change-Id: I824b1121842eb5821034071874bf1bb2d7c3631e
This is just another code path for little benefit in devstack which is
going to rot out. We should be opinionated here and only support the
dynamic catalog.
Change-Id: I4e5c7e86aefe72fc21c77d423033e9b169318fec
It was probably finally removed by one of recent refactoring changes.
Now ironic gate fails due to Swift trying to access it to override OS_AUTH_URL.
This change drops this override and just uses OS_AUTH_URL.
Closes-Bug: #1535245
Change-Id: I145bec110c4299e61f2bce49df41dcd82e5d462d
I noticed this when debugging some grenade issues failures.
An include of grenade/functions stores the current value of XTRACE
(on) and disables xtrace for the rest of the import.
We then include devstack's "functions" library, which now overwrites
the stored value of XTRACE the current state; i.e. disabled.
When it finishes it restores the prior state (disabled), and then
grenade restores the same value of XTRACE (disabled).
The result is that xtrace is incorrectly disabled until the next time
it just happens to be turned on.
The solution is to name-space the store of the current-value of xtrace
so when we finish sourcing a file, we always restore the tracing value
to what it was when we entered.
Some files had already discovered this. In general there is
inconsistency around the setting of the variable, and a lot of obvious
copy-paste. This brings consistency across all files by using
_XTRACE_* prefixes for the sotre/restore of tracing values.
Change-Id: Iba7739eada5711d9c269cb4127fa712e9f961695
with the release of osc 1.8.0, modifying object store account
properties is now available. use this mechanism and avoid setting
environment variable that are only helpful for swift CLI.
Change-Id: Ie51e3e2bb86162763f23d0a6bed36208811f89fc
This patch alows specifiying a deviation of the swift default port 8080 with
variable SWIFT_DEFAULT_BIND_PORT. The created endpoints in keystone for
object-store and the backup_swift_url in cinder.conf will use variable
SWIFT_DEFAULT_BIND_PORT instead of the fixed port 8080.
Change-Id: I47bbcf77368c430718fb8f29b7de1ff305e64422
Closes-Bug: #1489767
Ia0957b47187c3dcadd46154b17022c4213781112 proposes to have bashate
find instances of setting a local value. The issue is that "local"
always returns 0, thus hiding any failure in the commands running to
set the variable.
This is an automated replacement of such instances
Depends-On: I676c805e8f0401f75cc5367eee83b3d880cdef81
Change-Id: I9c8912a8fd596535589b207d7fc553b9d951d3fe
The default bind_port (6011-6013) in the sample config files for swift
use port numbers that are in the range registered by X11 (6000-6063) and
can prevent swift from starting if the ports are in use.
We should use an unregistered range (6611-6613).
Change-Id: Ifd95b99004aead5ddc8ae1a8dd3ccd9c4f2abe91
Closes-Bug: #1254328
This reverts commit f768787bdd6dddf2790f83a884618d29677ca77c.
And sets OS_AUTH_VERSION so swift CLI doesn't fall flat when
not using v2 keystone
Change-Id: If44a7e0d85e48020a3c90d8c5c027513129f0f3b
The python-keystoneclient requires a trailing /v<IDENTITY_API_VERSION> to
successfully authenticate, otherwise it fails with a 404 error due to a not
found resource.
This error showed up only when generating Swift tempurls, because the error was
raised when using python-swiftclient.
This change fixes this for python-swiftclient within devstack.
Change-Id: Ibe222d65162898db69acba076b5fe1cb3621fbc3
Closes-Bug: 1492216
The previous approach assumed that devstack in tree service support
would always be a super set of tempest. That's not necessarily
true. Instead when configuring tempest we should look at all the
possible services that tempest could know about, which will let us
disable services we don't have support for.
Change-Id: I9c24705e494689f09a885eb0a640efd50db33fcf
Swift3 should be in the same region as all other cloud.
By default it has regaion name 'US'. It's ok for requests
that signed by version 1 of signature because they haven't
region information in request.
But S3 signature of version 4 protocol sends region name to server
and swift3 plugin checks that input region equals to internal.
And because all cloud lives with 'RegionOne' then swift3 fails
request because it has 'US' region name by default.
Change-Id: Icd817183b1a040110372a8ae5d73fd2f0ec5559c
Related-Bug: #1411078
The openstack role list command doesn't include any identity API version
information and so will fail when running purely with v3. We could add
this information to the command however we already have a function that
does what swift requires so we should use it.
Change-Id: I5d5417eaed432760bfb97cf35bd76a0919c6004d
Currently, the Swift proxy, object, account, and container servers bind
to IPv4 address 0.0.0.0 by default. In the case of a user setting
SERVICE_IP_VERSION=6 in their local.conf file, these Swift servers still
listen on 0.0.0.0 instead of ::, which causes a ./stack.sh run to fail.
This change explicitly sets the bind_ip variable in the Swift server
config files so that the servers bind to either 0.0.0.0 (when
SERVICE_IP_VERSION != 6) or :: (when SERVICE_IP_VERSION == 6).
This patch is related to the following patch for devstack IPv6 support:
https://review.openstack.org/#/c/192329
Change-Id: Ie268c6daf5374e67ef8710a731c3af50ffdb821e
Always use the keystone V3 API when creating services and endpoints. The syntax
here is slightly different but we maintain the function interface.
Change-Id: Ib3a375918a45fd6e37d873a1a5c0c4b26bdbb5d8
Implements: bp keystonev3
By default, most Openstack services are bound to 0.0.0.0
and service endpoints are registered as IPv4 addresses.
With this change we introduce two new variables to control
this behavior:
SERVICE_IP_VERSION - can either be "4" or "6".
When set to "4" (default if not set) devstack will operate
as today - most services will open listen sockets on 0.0.0.0
and service endpoints will be registered using HOST_IP as the
address.
When set to "6" devstack services will open listen sockets on ::
and service endpoints will be registered using HOST_IPV6 as the
address.
There is no support for "4+6", more work is required for that.
HOST_IPV6 - if SERVICE_IP_VERSION=6 this must be an IPv6
address configured on the system.
Some existing services, like the Openvswitch agent, will continue
to use IPv4 addresses for things like tunnel endpoints. This is
a current restriction in the code and can be updated at a later
time. This change is just a first step to supporting IPv6-only
control and data planes in devstack.
This change is also partly based on two previous patches,
https://review.openstack.org/#/c/140519/ and
https://review.openstack.org/#/c/176898/
Change-Id: I5c0b775490ce54ab104fd5e89b20fb700212ae74
Co-Authored-By: Sean Collins <sean@coreitpro.com>
Co-Authored-By: Baodong Li <baoli@cisco.com>
Co-Authored-By: Sridhar Gaddam <sridhar.gaddam@enovance.com>
Co-Authored-By: Adam Kacmarsky <adam.kacmarsky@hp.com>
Co-Authored-By: Jeremy Alvis <jeremy.alvis@hp.com>
The gate/updown.sh calls the unstack.sh with
-ex option. Normally we do not use -e with unstack.sh.
The unstack.sh can fail if the service already stopped,
and it also can have flaky failures on the gate.
For example the stop_swift function tries to kill swift in two
different ways, and if the first one succeeds before the 2th attempt
the pkill fails the whole unstack.sh.
This change accepts kill failure.
Normally the kill can fail if the process does not exits,
or when you do not have permission to the kill operation.
Since the permission issue is very unlikely in our case,
this change does not tries to distinguish the two operation.
The behavior of the unstack.sh wen you are not using -ex should
not be changed by this change.
Change-Id: I64bf3cbe1b60c96f5b271dcfb620c3d4b50de26b
This includes requiring a domain when creating a user. This will allow us to
control where users are created in a later patch.
Adding the token to the user creation call is required because of a bad
interaction between OpenStackClient, os-client-config and keystoneclient
when dealing with v2 authentication but v3 API calls. It will be cleaned
up when we switch to v3 credentials.
Change-Id: I6ef50fd384d423bc0f13ee1016a8bdbb0650ecd9
Implements: bp keystonev3
