diff --git a/specs/juno/tripleo-juno-configurable-mnt-state.rst b/specs/juno/tripleo-juno-configurable-mnt-state.rst new file mode 100644 index 00000000..c3b97d4f --- /dev/null +++ b/specs/juno/tripleo-juno-configurable-mnt-state.rst @@ -0,0 +1,238 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +======================================================= +Configurable directory for persistent and stateful data +======================================================= + +https://blueprints.launchpad.net/tripleo/+spec/tripleo-juno-configurable-mnt-state + +Make the hardcoded /mnt/state path for stateful data be configurable. + +Problem Description +=================== + +1. A hard coded directory of /mnt/state for persistent data is incompatible +with Red Hat based distros available mechanism for a stateful data path. Red +Hat based distros, such as Fedora, RHEL, and CentOS, have a feature that uses +bind mounts for mounting paths onto a stateful data partition and does not +require manually reconfiguring software to use /mnt/state. + +2. Distros that use SELinux have pre-existing policy that allows access to +specific paths. Reconfiguring these paths to be under /mnt/state, results +in SELinux denials for existing services, requiring additional policy to be +written and maintained. + +3. Some operators and administrators find the reconfiguring of many services to +not use well known default values for filesystem paths to be disruptive and +inconsistent. They do not expect these changes when using a distro that they've +come to learn and anticipate certain configurations. These types of changes +also require documentation changes to existing documents and processes. + + +Proposed Change +=============== +Deployers will be able to choose a configurable path instead of the hardcoded +value of /mnt/state for the stateful path. + +A new element, stateful-path will be added that defines the value for the +stateful path. The default will be /mnt/state. + +There are 3 areas that need to respect the configurable path: + +os-apply-config template generation + The stateful-path element will set the stateful path value by installing a + JSON file to a well known location for os-collect-config to use as a local + data source. This will require a new local data source collector to be added + to os-collect-config (See `Dependencies`_). + + The JSON file's contents will be based on $STATEFUL_PATH, e.g.: + + {'stateful-path': '/mnt/state'} + + File templates (files under os-apply-config in an element) will then be + updated to replace the hard coded /mnt/state with {{stateful-path}}. + + Currently, there is a mix of root locations of the os-apply-config templates. + Most are written under /, although some are written under /mnt/state. The + /mnt/state is hard coded in the directory tree under os-apply-config in these + elements, so this will be removed to have the templates just written under /. + Symlinks could instead be used in these elements to setup the correct paths. + Support can also be added to os-apply-config's control file mechanism to + indicate these files should be written under the stateful path. An example + patch that does this is at: https://review.openstack.org/#/c/113651/ + +os-refresh-config scripts run at boot time + In order to make the stateful path configurable, all of the hard coded + references to /mnt/state in os-refresh-config scripts will be replaced with an + environment variable, $STATEFUL_PATH. + + The stateful-path element will provide an environment.d script for + os-refresh-config that reads the value from os-apply-config: + + export STATEFUL_PATH=$(os-apply-config --key stateful-path --type raw) + +Hook scripts run at image build time + The stateful-path element will provide an environment.d script for use at + image build time: + + export STATEFUL_PATH=${STATEFUL_PATH:-"/mnt/state"} + +The use-ephemeral element will depend on the stateful-path element, effectively +making the default stateful path remain /mnt/state. + +The stateful path can be reconfigured by defining $STATEFUL_PATH either A) in +the environment before an image build; or B) in an element with an +environment.d script which runs earlier than the stateful-path environment.d +script. + + +Alternatives +------------ +None come to mind, the point of this spec is to enable an alternative to what's +already existing. There may be additional alternatives out there other folks +may wish to add support for. + +Security Impact +--------------- +None + +Other End User Impact +--------------------- +End users using elements that change the stateful path location from /mnt/state +to something else will see this change reflected in configuration files and in +the directories used for persistent and stateful data. They will have to know +how the stateful path is configured and accessed. + +Different TripleO installs would appear different if used with elements that +configured the stateful path differently. + +This also adds some complexity when reading TripleO code, because instead of +there being an explicit path, there would instead be a reference to a +configurable value. + +Performance Impact +------------------ +There will be additional logic in os-refresh-config to determine and set the +stateful path, and an additional local collector that os-collect-config would +use. However, these are negligible in terms of negatively impacting +performance. + + +Other Deployer Impact +--------------------- +Deployers will be able to choose different elements that may reconfigure the +stateful path or change the value for $STATEFUL_PATH. The default will remain +unchanged however. + +Deployers would have to know what the stateful path is, and if it's different +across their environment, this could be confusing. However, this seems unlikely +as deployers are likely to be standardizing on one set of common elements, +distro, etc. + +In the future, if TripleO CI and CD clouds that are based on Red Hat distros +make use of this feature to enable Red Hat read only root support, then these +clouds would be configured differently from clouds that are configured to use +/mnt/state. As a team, the tripleo-cd-admins will have to know which +configuration has been used. + +Developer Impact +---------------- +1. Developers need to use the $STATEFUL_PATH and {{stateful-path}} +substitutions when they intend to refer to the stateful path. + +2. Code that needs to know the stateful path will need access to the variable +defining the path, it won't be able to assume the path is /mnt/state. A call to +os-apply-config to query the key defining the path could be done to get +the value, as long as os-collect-config has already run at least once. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + james-slagle + +Work Items +---------- + +tripleo-incubator +^^^^^^^^^^^^^^^^^ +* Update troubleshooting docs to mention that /mnt/state is a configurable + path, and could be different in local environments. + +tripleo-image-elements +^^^^^^^^^^^^^^^^^^^^^^ +* Add a new stateful-path element that configures stateful-path and $STATEFUL_PATH + to /mnt/state +* Update os-apply-config templates to replace /mnt/state with {{stateful-path}} +* Update os-refresh-config scripts to replace /mnt/state with $STATEFUL_PATH +* Update all elements that have os-apply-config template files under /mnt/state + to just be under /. + + * update os-apply-config element to call os-apply-config with a --root + $STATEFUL_PATH option + * update elements that have paths to os-apply-config generated files (such + as /etc/nova/nova.conf) to refer to those paths as + $STATEFUL_PATH/path/to/file. + +* make use-ephemeral element depend on stateful-path element + +Dependencies +============ +1. os-collect-config will need a new feature to read from a local data source + directory that elements can install JSON files into, such as a source.d. There + will be a new spec filed on this feature. + https://review.openstack.org/#/c/100965/ + +2. os-apply-config will need an option in its control file to support + generating templates under the configurable stateful path. There is a patch + here: https://review.openstack.org/#/c/113651/ + + +Testing +======= + +There is currently no testing that all stateful and persistent data is actually +written to a stateful partition. + +We should add tempest tests that directly exercise the preserve_ephemeral +option, and have tests that check that all stateful data has been preserved +across a "nova rebuild". Tempest seems like a reasonable place to add these +tests since preserve_ephemeral is a Nova OpenStack feature. Plus, once TripleO +CI is running tempest against the deployed OverCloud, we will be testing this +feature. + +We should also test in TripleO CI that state is preserved across a rebuild by +adding stateful data before a rebuild and verifying it is still present after a +rebuild. + +Documentation Impact +==================== + +We will document the new stateful-path element. + +TripleO documentation will need to mention the potential difference in +configuration files and the location of persistent data if a value other than +/mnt/state is used. + + +References +========== + +os-collect-config local datasource collector spec: + +* https://review.openstack.org/100965 + +Red Hat style stateful partition support this will enable: + +* https://git.fedorahosted.org/cgit/initscripts.git/tree/systemd/fedora-readonly +* https://git.fedorahosted.org/cgit/initscripts.git/tree/sysconfig/readonly-root +* https://git.fedorahosted.org/cgit/initscripts.git/tree/statetab +* https://git.fedorahosted.org/cgit/initscripts.git/tree/rwtab