Merge "stx-6.0: Initial spec for Debian Build"
This commit is contained in:
Zuul
Gerrit Code Review
commit
b345e039f1
|
|
@ -0,0 +1,337 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License. http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=======================
|
||||
StarlingX: Debian Build
|
||||
=======================
|
||||
|
||||
|
||||
Debian Build Spec - Update build system to support transition to Debian
|
||||
=======================================================================
|
||||
|
||||
Storyboard Story:
|
||||
https://storyboard.openstack.org/#!/story/2008846
|
||||
|
||||
A move from CentOS to Debian requires that the existing StarlingX
|
||||
build container definition (Dockerfile), build scripts, and build
|
||||
procedures described in the StarlingX Build Guide and related
|
||||
documents are all modified to be based on Debian. Instead of just
|
||||
swapping CentOS for Debian, this specification proposes using several
|
||||
offload containers along a streamlined StarlingX build container that
|
||||
would work in concert to complete builds, patch creation and related
|
||||
operations. Completion of this spec would allow StarlingX to build
|
||||
StarlingX.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
If we were to simply replicate the existing CentOS based StarlingX
|
||||
build container with a similarly capable Debian StarlingX build
|
||||
container we would find ourselves with similar deficiencies:
|
||||
|
||||
* large container size (3GB+ currently for CentOS)
|
||||
* integration of developer details, (username, uid, gid)
|
||||
* loose internal component versioning
|
||||
* requirement to run as a privileged container
|
||||
* inability to be hosted on github or other container registry (see
|
||||
developer details)
|
||||
* Unable to scale (User interactions and build ops take place in the
|
||||
same container)
|
||||
* Unable to load balance (all operations are tied to the same
|
||||
container)
|
||||
* Awkward handling of shared resources (such as the package feed)
|
||||
* Developers often having to rebuild the STX Build container
|
||||
* Developers having to wait roughly 20 minutes for the STX Build
|
||||
container to be built
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
This change would affect most of the use cases described in the
|
||||
StarlingX Build Guide[1]. Such as, but not limited to:
|
||||
|
||||
* Build packages
|
||||
* Build ISOs
|
||||
* Build installer
|
||||
* Build StarlingX OpenStack Application
|
||||
|
||||
This would impact the StarlingX Developer. Although the proposed
|
||||
design does increase overall complexity this would, as much as
|
||||
possible, be transparent to the developer. The biggest impact is the
|
||||
elimination of every developer having to build a personal container
|
||||
and keeping it up to date, all containers can simply be pulled from a
|
||||
registry and updates will only take as long as necessary to download
|
||||
the updated containers. Do to the potential distributed nature of the
|
||||
proposed design developers will have the option to use smaller systems
|
||||
to host the interactive STX Build container, while running offload
|
||||
containers, such as the Package Build container on more capable
|
||||
systems, this is especially useful for organizations with multiple
|
||||
developers.
|
||||
|
||||
NOTE: The scope of this spec would not result in considerable changes
|
||||
in these use cases, most changes would result in how operations are
|
||||
performed in the background only.
|
||||
|
||||
[1][https://docs.starlingx.io/developer_resources/build_guide.html]
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
A new Debian based STX Build container would be created. As with the
|
||||
existing CentOS Build container a Dockerfile would be written to allow
|
||||
for use with docker-build.
|
||||
|
||||
The new STX Build container would contain tools such as 'repo' ,
|
||||
'git', 'emacs', 'vim', ... to provide a similar development
|
||||
environment as was previously available.
|
||||
|
||||
Debian package build tools such as sbuild/pbuilder, apt, ... may be
|
||||
available to allow developers to work on packages without additional
|
||||
containers. Much as they would following the Debian Maintainers Guide
|
||||
[2].
|
||||
|
||||
The container will make use of tools such as 'fixuid' [3] to ensure no
|
||||
developer attributes are required to be built into the container and
|
||||
thus can be uploaded to dockerhub or other container registries for
|
||||
immediate use.
|
||||
|
||||
Three new containers will be created to 'offload' work from the new
|
||||
STX Build container:
|
||||
|
||||
* Package Builder
|
||||
* Image Builder
|
||||
* Repository Manager
|
||||
|
||||
A tool such as docker-compose or Helm charts will be used to start the
|
||||
containers and ensure connectivity between containers is available. It
|
||||
is possible that some containers become shared resources between many
|
||||
developers so the means by which the 'set' of containers is started
|
||||
and stopped may vary.
|
||||
|
||||
The build scripts will be updated to redirect work from the STX Build
|
||||
container to the appropriate offload container. Configuration defaults
|
||||
will favor the 'STX individual contributor' use case but a
|
||||
configuration file will be used to allow the defaults to be
|
||||
overwritten. For example if docker-compose is used to start all
|
||||
containers on a network called 'stx' the configuration defaults can
|
||||
use container names in URLs, but if the Package Builder container is
|
||||
shared and exists on a secondary machine an IP address might be used
|
||||
in the configuration file for 'builder.url'.
|
||||
|
||||
Communication to offload containers will be via REST, 'docker exec',
|
||||
or any suitable client/server interface. Ideally these communications
|
||||
will be abstracted allowing for flexibility in what actual containers
|
||||
are used for each function, ie. scripts such as the build script would
|
||||
interact with local functions with a defined API instead of directly
|
||||
with the offload container interface.
|
||||
|
||||
The software contained in each of the offload containers will be
|
||||
covered in separate specs and are outside the scope of this spec. Just
|
||||
know that:
|
||||
|
||||
* Package Builder container - will build packages, taking typical
|
||||
Debian input files (orig.tar, debian.diff.tar, .dsc) along with
|
||||
artefacts from a package feed to provide build dependencies, to
|
||||
build packages and output one or more .deb files
|
||||
* Image Builder container - will take a configuration as input and use
|
||||
artefacts from one or more package feeds to generate images such as
|
||||
ISOs and OSTree repositories.
|
||||
* Repository Manager container - will allow for one or more package
|
||||
feeds to be published as well as provide package repository
|
||||
operations such as sync.
|
||||
|
||||
[2][https://www.debian.org/doc/manuals/maint-guide/]
|
||||
[3][https://github.com/boxboat/fixuid]
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
We could simply implement the Debian based STX Build container in a
|
||||
similar way in which the current CentOS based STX Build container is
|
||||
done today. This approach does not address the shortcomings described
|
||||
previously.
|
||||
|
||||
A single STX Build container could be created with all necessary
|
||||
functions required, so again an analog to what the current STX Build
|
||||
container is, but add a REST API or similar to allow functions to be
|
||||
executed from outside the container. For example two of these STX
|
||||
Build containers are started but one would be configured to handle the
|
||||
user interactions while other build operations would be sent to the
|
||||
second container for processing. This would handle some scaling issues
|
||||
but not as many as the proposed approach does. It also doesn't address
|
||||
all issues previously expressed about a large monolithic build
|
||||
container.
|
||||
|
||||
Another approach would be to make use of external services as shared
|
||||
resources, such as running one or a cluster of Open Build Service
|
||||
which the STX Build container would use to complete package
|
||||
builds. This could be a compelling approach for a multi-developer
|
||||
organization but would add significant overhead for the 'Individual
|
||||
STX contributor' use case.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None for StarlingX deployment. Splitting the STX Builder into multiple
|
||||
containers and exploring using some containers as shared resources
|
||||
definitely does introduce security implications during the build which
|
||||
will have to be considered.
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
As previously stated, the added complexity of this design will be
|
||||
abstracted from the developer. Interactions inside the new STX Build
|
||||
container will be similar to what a developer does today, despite
|
||||
operations at times being sent to an offload container(s).
|
||||
|
||||
The use of a purpose built Repository Management container should
|
||||
simplify interactions with the repository feed, removing the need for
|
||||
direct file manipulation as done in the current design.
|
||||
|
||||
Eliminating the need to build a custom container for each developer
|
||||
will simplify the developer's day to day work and should prevent
|
||||
version skew that might exist based on when each developer last
|
||||
rebuilt their Build container.
|
||||
|
||||
For a STX Individual Contributor the system requirements and
|
||||
interactions to complete a build will remain much as they are in the
|
||||
current design. The defaults used will be selected to assume this
|
||||
use case. For multi-developer organizations new configuration options
|
||||
will be available to exploit the distributed capabilities of this
|
||||
design, allowing hardware resources to be selected based on the type
|
||||
of workload. This will also have a side effect of being able to
|
||||
exercise finer permissions control over shared hardware resources.
|
||||
|
||||
Upgrade impact
|
||||
--------------
|
||||
|
||||
None
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Mark Asselstine - markawr
|
||||
|
||||
Other contributors:
|
||||
|
||||
* Yue Tao - wrytao
|
||||
* Hongxu Jia - hjia
|
||||
* Xiao Zhang - xzhang1
|
||||
* Peng Yan - yanpenghit
|
||||
* Haiqing Bai - hqbai
|
||||
|
||||
Repos Impacted
|
||||
--------------
|
||||
|
||||
starlingx/tools
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
See storyboard. Additional items will be added after spec approval.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
This spec is related to the 'new stx tool' spec. The contents of this
|
||||
spec, however, are written such that the 'new stx tool' spec does not
|
||||
exist. For example the assertion in the section labelled 'use cases'
|
||||
that 'changes to the build system would be only in how tasks are
|
||||
performed in the background' assumes similar script names would be
|
||||
reused, and does not assume a move to a new stx tool. If the 'new stx
|
||||
tool' spec is accepted then changes to existing build scripts will be
|
||||
forgone in exchange for new implementations in the stx tool.
|
||||
|
||||
This spec will be related to specs to be written for implementation of
|
||||
a Package Build container, an Image Builder container and a Repository
|
||||
Manager container. The writing of these specs is contingent on this
|
||||
spec being accepted.
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
As the scope of this spec is restricted to the building of StarlingX
|
||||
it does not introduce any additional runtime testing requirements. As
|
||||
this change is proposed to take place alongside the move to Debian,
|
||||
full runtime testing is expected related to that spec.
|
||||
|
||||
The new methods of building full or partial platforms introduced by
|
||||
this spec will require validation to ensure similar outcomes as were
|
||||
expected when building StarlingX for CentOS. All build operations will
|
||||
be tested to ensure that the documented results map to real world
|
||||
use. Where possible test code will be submitted alongside code
|
||||
implementing functionality. The operations described in the current
|
||||
StarlingX Build Guide and other documents will form the basis for
|
||||
testing.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Revisit much of the StarlingX Build Guide
|
||||
https://docs.starlingx.io/developer_resources/build_guide.html
|
||||
|
||||
Revisit much of the StarlingX Layered Build Guide
|
||||
https://docs.starlingx.io/developer_resources/layered_build_guide.html
|
||||
|
||||
Possible updates to the Build StarlingX Docker Images document
|
||||
https://docs.starlingx.io/developer_resources/build_docker_image.html
|
||||
|
||||
NOTE: updates to these documents are already required to be updated
|
||||
for the Debian transition, regardless of the changes suggested in this
|
||||
spec, though the scope with these changes will be larger.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* https://etherpad.opendev.org/p/stx-ptg-planning-april-2021
|
||||
|
||||
* https://docs.starlingx.io/developer_resources/build_guide.html
|
||||
|
||||
* https://www.debian.org/doc/manuals/maint-guide/
|
||||
|
||||
* https://github.com/boxboat/fixuid
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - STX-6.0
|
||||
- Introduced
|
||||
Loading…
Reference in New Issue