Consolidate build guides into single guide
R1 and R2 build guides were identified as being nearly identical. Based on this, they were consolidated into a single guide. - Updated link to reference consolidated guide - Updated index page to reference consolidated guide - Removed old versioned guides - Basic formatting cleanup (remove $prompts, lists, indentation) Change-Id: I83aea4d76147fbc56d555dff3ce768737d15c4e2 Signed-off-by: Kristal Dale <kristal.dale@intel.com>
This commit is contained in:
@@ -1,9 +1,8 @@
|
||||
==========================
|
||||
Build guide StarlingX R2.0
|
||||
==========================
|
||||
=====================
|
||||
StarlingX Build Guide
|
||||
=====================
|
||||
|
||||
This section describes the steps for building a StarlingX ISO from the R2.0
|
||||
StarlingX release.
|
||||
This section describes the steps for building an ISO image from a StarlingX release.
|
||||
|
||||
------------
|
||||
Requirements
|
||||
@@ -15,10 +14,10 @@ Hardware requirements
|
||||
|
||||
A workstation computer with:
|
||||
|
||||
- Processor: x86_64 is the only supported architecture
|
||||
- Memory: At least 32GB RAM
|
||||
- Hard Disk: 500GB HDD
|
||||
- Network: Network adapter with active Internet connection
|
||||
* Processor: x86_64 is the only supported architecture
|
||||
* Memory: At least 32GB RAM
|
||||
* Hard Disk: 500GB HDD
|
||||
* Network: Network adapter with active Internet connection
|
||||
|
||||
*********************
|
||||
Software requirements
|
||||
@@ -26,26 +25,22 @@ Software requirements
|
||||
|
||||
A workstation computer with:
|
||||
|
||||
- Operating System: Ubuntu 16.04 LTS 64-bit
|
||||
- Docker
|
||||
- Android Repo Tool
|
||||
- Proxy settings configured (if required)
|
||||
|
||||
- See
|
||||
http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000136.html
|
||||
for more details
|
||||
|
||||
- Public SSH key
|
||||
* Operating System: Ubuntu 16.04 LTS 64-bit
|
||||
* Docker
|
||||
* Android Repo Tool
|
||||
* Proxy settings configured, if required (See
|
||||
http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000136.html for more details)
|
||||
* Public SSH key
|
||||
|
||||
-----------------------------
|
||||
Development environment setup
|
||||
-----------------------------
|
||||
|
||||
This section describes how to set up a StarlingX development system on a
|
||||
workstation computer. After completing these steps, you can
|
||||
build a StarlingX ISO image on the following Linux distribution:
|
||||
workstation computer. After completing these steps, you can build a StarlingX
|
||||
ISO image on the following Linux distribution:
|
||||
|
||||
- Ubuntu 16.04 LTS 64-bit
|
||||
* Ubuntu 16.04 LTS 64-bit
|
||||
|
||||
****************************
|
||||
Update your operating system
|
||||
@@ -54,100 +49,83 @@ Update your operating system
|
||||
Before proceeding with the build, ensure your Ubuntu distribution is up to date.
|
||||
You first need to update the local database list of available packages:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ sudo apt-get update
|
||||
sudo apt-get update
|
||||
|
||||
******************************************
|
||||
Installation requirements and dependencies
|
||||
******************************************
|
||||
|
||||
^^^^
|
||||
User
|
||||
^^^^
|
||||
#. Set up <user>.
|
||||
|
||||
1. Make sure you are a non-root user with sudo enabled when you build the
|
||||
StarlingX ISO. You also need to either use your existing user or create a
|
||||
separate *<user>*:
|
||||
Make sure you are a non-root user with sudo privileges enabled when you build
|
||||
the StarlingX ISO.
|
||||
|
||||
.. code:: sh
|
||||
Use either your existing user or create a separate *<user>*:
|
||||
|
||||
$ sudo useradd -m -d /home/<user> <user>
|
||||
::
|
||||
|
||||
2. Your *<user>* should have sudo privileges:
|
||||
sudo useradd -m -d /home/<user> <user>
|
||||
sudo sh -c "echo '<user> ALL=(ALL:ALL) ALL' >> /etc/sudoers"
|
||||
sudo su -c <user>
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ sudo sh -c "echo '<user> ALL=(ALL:ALL) ALL' >> /etc/sudoers"
|
||||
$ sudo su -c <user>
|
||||
#. Set up Git.
|
||||
|
||||
^^^
|
||||
Git
|
||||
^^^
|
||||
Install the required Git packages on the Ubuntu host system:
|
||||
|
||||
3. Install the required packages on the Ubuntu host system:
|
||||
::
|
||||
|
||||
.. code:: sh
|
||||
sudo apt-get install make git curl
|
||||
|
||||
$ sudo apt-get install make git curl
|
||||
Set up your identity in git using your actual name and email address:
|
||||
|
||||
4. Make sure to set up your identity using the following two commands.
|
||||
Be sure to provide your actual name and email address:
|
||||
::
|
||||
|
||||
.. code:: sh
|
||||
git config --global user.name "Name LastName"
|
||||
git config --global user.email "Email Address"
|
||||
|
||||
$ git config --global user.name "Name LastName"
|
||||
$ git config --global user.email "Email Address"
|
||||
|
||||
^^^^^^^^^
|
||||
Docker CE
|
||||
^^^^^^^^^
|
||||
#. Install the required Docker CE packages in the Ubuntu host system.
|
||||
|
||||
5. Install the required Docker CE packages in the Ubuntu host system. See
|
||||
`Get Docker CE for
|
||||
Ubuntu <https://docs.docker.com/install/linux/docker-ce/ubuntu/#os-requirements>`__
|
||||
for more information.
|
||||
See
|
||||
`Get Docker CE for Ubuntu <https://docs.docker.com/install/linux/docker-ce/ubuntu/#os-requirements>`__ for more information.
|
||||
|
||||
6. Log out and log in to add your *<user>* to the Docker group:
|
||||
Make sure to log out and log in to add your *<user>* to the Docker group:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ sudo usermod -aG docker <user>
|
||||
sudo usermod -aG docker <user>
|
||||
|
||||
^^^^^^^^^^^^^^^^^
|
||||
Android Repo Tool
|
||||
^^^^^^^^^^^^^^^^^
|
||||
#. Install the Android Repo Tool in the Ubuntu host system.
|
||||
|
||||
7. Install the required Android Repo Tool in the Ubuntu host system. Follow
|
||||
the steps in the `Installing
|
||||
Repo <https://source.android.com/setup/build/downloading#installing-repo>`__
|
||||
Follow the steps in the
|
||||
`Installing Repo <https://source.android.com/setup/build/downloading#installing-repo>`__
|
||||
section.
|
||||
|
||||
**********************
|
||||
Install public SSH key
|
||||
**********************
|
||||
|
||||
#. Follow these instructions on GitHub to `Generate a Public SSH
|
||||
Key <https://help.github.com/articles/connecting-to-github-with-ssh>`__.
|
||||
Then upload your public key to your GitHub and Gerrit account
|
||||
profiles:
|
||||
Follow these instructions on GitHub to
|
||||
`Generate a Public SSH Key <https://help.github.com/articles/connecting-to-github-with-ssh>`__.
|
||||
Then upload your public key to your GitHub and Gerrit account profiles:
|
||||
|
||||
- `Upload to
|
||||
Github <https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account>`__
|
||||
- `Upload to
|
||||
Gerrit <https://review.openstack.org/#/settings/ssh-keys>`__
|
||||
* `Upload to Github <https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account>`__
|
||||
|
||||
* `Upload to Gerrit <https://review.openstack.org/#/settings/ssh-keys>`__
|
||||
|
||||
****************************
|
||||
Create a workspace directory
|
||||
****************************
|
||||
|
||||
#. Create a *starlingx* workspace directory on your system.
|
||||
Best practices dictate creating the workspace directory
|
||||
in your $HOME directory:
|
||||
Create a *starlingx* workspace directory on your system. Best practices dictate
|
||||
creating the workspace directory in your $HOME directory:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ mkdir -p $HOME/starlingx/
|
||||
mkdir -p $HOME/starlingx/
|
||||
|
||||
*********************
|
||||
Install tools project
|
||||
@@ -155,17 +133,17 @@ Install tools project
|
||||
|
||||
#. Under your $HOME directory, clone the <tools> project:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ cd $HOME
|
||||
$ git clone https://opendev.org/starlingx/tools.git
|
||||
cd $HOME
|
||||
git clone https://opendev.org/starlingx/tools.git
|
||||
|
||||
#. Navigate to the *<$HOME/tools>* project
|
||||
directory:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ cd $HOME/tools/
|
||||
cd $HOME/tools/
|
||||
|
||||
-----------------------------
|
||||
Prepare the base Docker image
|
||||
@@ -182,10 +160,10 @@ Configuration values
|
||||
You can customize values for the StarlingX base Docker image using a
|
||||
text-based configuration file named ``localrc``:
|
||||
|
||||
- ``HOST_PREFIX`` points to the directory that hosts the 'designer'
|
||||
* ``HOST_PREFIX`` points to the directory that hosts the 'designer'
|
||||
subdirectory for source code, the 'loadbuild' subdirectory for
|
||||
the build environment, generated RPMs, and the ISO image.
|
||||
- ``HOST_MIRROR_DIR`` points to the directory that hosts the CentOS mirror
|
||||
* ``HOST_MIRROR_DIR`` points to the directory that hosts the CentOS mirror
|
||||
repository.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@@ -194,7 +172,7 @@ localrc configuration file
|
||||
|
||||
Create your ``localrc`` configuration file. For example:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
# tbuilder localrc
|
||||
MYUNAME=<your user name>
|
||||
@@ -212,7 +190,7 @@ to build the base Docker image.
|
||||
#. If necessary, you might have to set http/https proxy in your
|
||||
Dockerfile before building the docker image:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
ENV http_proxy " http://your.actual_http_proxy.com:your_port "
|
||||
ENV https_proxy " https://your.actual_https_proxy.com:your_port "
|
||||
@@ -221,9 +199,9 @@ to build the base Docker image.
|
||||
|
||||
#. The ``tb.sh`` script automates the base Docker image build:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
./tb.sh create
|
||||
./tb.sh create
|
||||
|
||||
----------------------------------
|
||||
Build the CentOS mirror repository
|
||||
@@ -237,14 +215,14 @@ this CentOS mirror repository.
|
||||
Run repository Docker container
|
||||
*******************************
|
||||
|
||||
| Run the following commands under a terminal identified as "**One**":
|
||||
Run the following commands under a terminal identified as "**One**":
|
||||
|
||||
#. Navigate to the *$HOME/tools/centos-mirror-tool* project
|
||||
directory:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ cd $HOME/tools/centos-mirror-tools/
|
||||
cd $HOME/tools/centos-mirror-tools/
|
||||
|
||||
#. Launch the Docker container using the previously created base Docker image
|
||||
*<repository>:<tag>*. As /localdisk is defined as the workdir of the
|
||||
@@ -253,9 +231,9 @@ Run repository Docker container
|
||||
this directory. The container runs from the same directory in which the
|
||||
scripts are stored.
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ docker run -it --volume $(pwd):/localdisk local/$USER-stx-builder:7.4 bash
|
||||
docker run -it --volume $(pwd):/localdisk local/$USER-stx-builder:7.4 bash
|
||||
|
||||
*****************
|
||||
Download packages
|
||||
@@ -266,7 +244,7 @@ Download packages
|
||||
|
||||
::
|
||||
|
||||
# cd localdisk && bash download_mirror.sh
|
||||
cd localdisk && bash download_mirror.sh
|
||||
|
||||
#. Monitor the download of packages until it is complete. When the download
|
||||
is complete, the following message appears:
|
||||
@@ -289,8 +267,8 @@ Verify packages
|
||||
|
||||
::
|
||||
|
||||
# cat logs/*_missing_*.log
|
||||
# cat logs/*_failmoved_*.log
|
||||
cat logs/*_missing_*.log
|
||||
cat logs/*_failmoved_*.log
|
||||
|
||||
#. In case missing or failed packages do exist, which is usually caused by
|
||||
network instability (or timeout), you need to download the packages
|
||||
@@ -333,17 +311,17 @@ as "**Two**", run the following commands:
|
||||
#. From terminal identified as "**Two**", create a *mirror/CentOS*
|
||||
directory under your *starlingx* workspace directory:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ mkdir -p $HOME/starlingx/mirror/CentOS/
|
||||
mkdir -p $HOME/starlingx/mirror/CentOS/
|
||||
|
||||
#. Copy the built CentOS mirror repository built under
|
||||
*$HOME/tools/centos-mirror-tool* to the *$HOME/starlingx/mirror/*
|
||||
workspace directory:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ cp -r $HOME/tools/centos-mirror-tools/output/stx-r1/ $HOME/starlingx/mirror/CentOS/
|
||||
cp -r $HOME/tools/centos-mirror-tools/output/stx-r1/ $HOME/starlingx/mirror/CentOS/
|
||||
|
||||
|
||||
-------------------------
|
||||
@@ -356,33 +334,33 @@ Run building Docker container
|
||||
|
||||
#. From the terminal identified as "**Two**", create the workspace folder:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ mkdir -p $HOME/starlingx/workspace
|
||||
mkdir -p $HOME/starlingx/workspace
|
||||
|
||||
#. Navigate to the *$HOME/tools* project directory:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ cd $HOME/tools
|
||||
cd $HOME/tools
|
||||
|
||||
#. Verify environment variables:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ bash tb.sh env
|
||||
bash tb.sh env
|
||||
|
||||
#. Run the building Docker container:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ bash tb.sh run
|
||||
bash tb.sh run
|
||||
|
||||
#. Execute the buiding Docker container:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ bash tb.sh exec
|
||||
bash tb.sh exec
|
||||
|
||||
*********************************
|
||||
Download source code repositories
|
||||
@@ -391,48 +369,57 @@ Download source code repositories
|
||||
#. From the terminal identified as "**Two**", which is now inside the
|
||||
building Docker container, start the internal environment:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ eval $(ssh-agent)
|
||||
$ ssh-add
|
||||
eval $(ssh-agent)
|
||||
ssh-add
|
||||
|
||||
#. Use the repo tool to create a local clone of the manifest
|
||||
Git repository based on the "master" branch:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ cd $MY_REPO_ROOT_DIR
|
||||
$ repo init -u https://opendev.org/starlingx/manifest -m default.xml
|
||||
cd $MY_REPO_ROOT_DIR
|
||||
repo init -u https://opendev.org/starlingx/manifest -m default.xml
|
||||
|
||||
|
||||
Optionally, specify a specific branch to clone, for example the R2.0 release
|
||||
branch:
|
||||
|
||||
::
|
||||
|
||||
cd $MY_REPO_ROOT_DIR
|
||||
repo init -u https://opendev.org/starlingx/manifest -m default.xml -b r/stx.2.0
|
||||
|
||||
#. Synchronize the repository:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ repo sync -j`nproc`
|
||||
repo sync -j`nproc`
|
||||
|
||||
#. Create a tarballs repository:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ ln -s /import/mirrors/CentOS/stx-r1/CentOS/pike/downloads/ $MY_REPO/stx/
|
||||
ln -s /import/mirrors/CentOS/stx-r1/CentOS/pike/downloads/ $MY_REPO/stx/
|
||||
|
||||
Alternatively, you can run the "populate_downloads.sh" script to copy
|
||||
the tarballs instead of using a symlink:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/
|
||||
populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/
|
||||
|
||||
Outside the container
|
||||
|
||||
#. From another terminal identified as "**Three**", create mirror binaries:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ mkdir -p $HOME/starlingx/mirror/CentOS/stx-installer
|
||||
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/stx-installer/initrd.img
|
||||
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/stx-installer/vmlinuz
|
||||
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/stx-installer/squashfs.img
|
||||
mkdir -p $HOME/starlingx/mirror/CentOS/stx-installer
|
||||
cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/stx-installer/initrd.img
|
||||
cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/stx-installer/vmlinuz
|
||||
cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/stx-installer/squashfs.img
|
||||
|
||||
**************
|
||||
Build packages
|
||||
@@ -449,15 +436,15 @@ Build packages
|
||||
|
||||
#. Update the symbolic links:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/
|
||||
generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/
|
||||
|
||||
#. Build the packages:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ build-pkgs
|
||||
build-pkgs
|
||||
|
||||
#. **Optional!** Generate-Cgcs-Tis-Repo:
|
||||
|
||||
@@ -467,19 +454,19 @@ Build packages
|
||||
need to execute the following command after building modified or new
|
||||
packages.
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ generate-cgcs-tis-repo
|
||||
generate-cgcs-tis-repo
|
||||
|
||||
-------------------
|
||||
Build StarlingX ISO
|
||||
-------------------
|
||||
|
||||
#. Build the image:
|
||||
Build the image:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ build-iso
|
||||
build-iso
|
||||
|
||||
---------------
|
||||
Build installer
|
||||
@@ -493,15 +480,15 @@ every time you upgrade the kernel.
|
||||
|
||||
After running "build-iso", run:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ build-pkgs --installer
|
||||
build-pkgs --installer
|
||||
|
||||
This builds *rpm* and *anaconda* packages. Then run:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ update-pxe-network-installer
|
||||
update-pxe-network-installer
|
||||
|
||||
The *update-pxe-network-installer* covers the steps detailed in
|
||||
*$MY_REPO/stx/stx-metal/installer/initrd/README*. This script
|
||||
@@ -532,11 +519,11 @@ Two ways exist for using these files:
|
||||
|
||||
Recreate the *pxe-network-installer* package and rebuild the image:
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ build-pkgs --clean pxe-network-installer
|
||||
$ build-pkgs pxe-network-installer
|
||||
$ build-iso
|
||||
build-pkgs --clean pxe-network-installer
|
||||
build-pkgs pxe-network-installer
|
||||
build-iso
|
||||
|
||||
Your ISO image should be able to boot.
|
||||
|
||||
@@ -544,13 +531,11 @@ Your ISO image should be able to boot.
|
||||
Additional notes
|
||||
****************
|
||||
|
||||
- In order to get the first boot working, this complete procedure needs
|
||||
to be done. However, once the init files are created, these can be
|
||||
stored in a shared location where different developers can make use
|
||||
of them. Updating these files is not a frequent task and should be
|
||||
done whenever the kernel is upgraded.
|
||||
- StarlingX is in active development. Consequently, it is possible that in the
|
||||
future the **0.2** version will change to a more generic solution.
|
||||
* In order to get the first boot working, this complete procedure needs to be
|
||||
done. However, once the init files are created, these can be stored in a shared location where different developers can make use of them. Updating these files
|
||||
is not a frequent task and should be done whenever the kernel is upgraded.
|
||||
* StarlingX is in active development. Consequently, it is possible that a
|
||||
future version will change to a more generic solution.
|
||||
|
||||
---------------
|
||||
Build avoidance
|
||||
@@ -560,26 +545,22 @@ Build avoidance
|
||||
Purpose
|
||||
*******
|
||||
|
||||
Greatly reduce build times after using "repo" to syncronized a local
|
||||
repository with an upstream source (i.e. "repo sync").
|
||||
Build avoidance works well for designers working
|
||||
within a regional office. Starting from a new workspace, "build-pkgs"
|
||||
typically requires three or more hours to complete. Build avoidance
|
||||
Greatly reduce build times after using "repo" to syncronized a local repository
|
||||
with an upstream source (i.e. "repo sync"). Build avoidance works well for
|
||||
designers working within a regional office. Starting from a new workspace,
|
||||
"build-pkgs" typically requires three or more hours to complete. Build avoidance
|
||||
reduces this step to approximately 20 minutes.
|
||||
|
||||
***********
|
||||
Limitations
|
||||
***********
|
||||
|
||||
- Little or no benefit for designers who refresh a pre-existing
|
||||
workspace at least daily (e.g. download_mirror.sh, repo sync,
|
||||
generate-cgcs-centos-repo.sh, build-pkgs, build-iso). In these cases,
|
||||
an incremental build (i.e. reuse of same workspace without a "build-pkgs
|
||||
--clean") is often just as efficient.
|
||||
- Not likely to be useful to solo designers, or teleworkers that wish
|
||||
to compile on using their home computers. Build avoidance downloads build
|
||||
artifacts from a reference build, and WAN speeds are generally too
|
||||
slow.
|
||||
* Little or no benefit for designers who refresh a pre-existing workspace at
|
||||
least daily (e.g. download_mirror.sh, repo sync, generate-cgcs-centos-repo.sh, build-pkgs, build-iso). In these cases, an incremental build (i.e. reuse of
|
||||
same workspace without a :command:`build-pkgs --clean`) is often just as
|
||||
efficient.
|
||||
* Not likely to be useful to solo designers, or teleworkers that wish to compile
|
||||
on using their home computers. Build avoidance downloads build artifacts from a reference build, and WAN speeds are generally too slow.
|
||||
|
||||
*****************
|
||||
Method (in brief)
|
||||
@@ -587,27 +568,23 @@ Method (in brief)
|
||||
|
||||
#. Reference builds
|
||||
|
||||
- A server in the regional office performs regular (e.g. daily)
|
||||
automated builds using existing methods. These builds are called
|
||||
"reference builds".
|
||||
- The builds are timestamped and preserved for some time (i.e. a
|
||||
number of weeks).
|
||||
- A build CONTEXT, which is a file produced by "build-pkgs"
|
||||
at location *$MY_WORKSPACE/CONTEXT*, is captured. It is a bash script that can
|
||||
cd to each and every Git and checkout the SHA that contributed to
|
||||
the build.
|
||||
- For each package built, a file captures the md5sums of all the
|
||||
source code inputs required to build that package. These files are
|
||||
also produced by "build-pkgs" at location
|
||||
*$MY_WORKSPACE//rpmbuild/SOURCES//srpm_reference.md5*.
|
||||
- All these build products are accessible locally (e.g. a regional
|
||||
office) using "rsync".
|
||||
* A server in the regional office performs regular (e.g. daily) automated
|
||||
builds using existing methods. These builds are called "reference builds".
|
||||
* The builds are timestamped and preserved for some time (i.e. a number of weeks).
|
||||
* A build CONTEXT, which is a file produced by "build-pkgs" at location
|
||||
*$MY_WORKSPACE/CONTEXT*, is captured. It is a bash script that can cd to
|
||||
each and every Git and checkout the SHA that contributed to the build.
|
||||
* For each package built, a file captures the md5sums of all the source code
|
||||
inputs required to build that package. These files are also produced by
|
||||
"build-pkgs" at location *$MY_WORKSPACE//rpmbuild/SOURCES//srpm_reference.md5*.
|
||||
* All these build products are accessible locally (e.g. a regional office)
|
||||
using "rsync".
|
||||
|
||||
**NOTE:** Other protocols can be added later.
|
||||
**NOTE:** Other protocols can be added later.
|
||||
|
||||
#. Designers
|
||||
|
||||
- Request a build avoidance build. Recommended after you have
|
||||
* Request a build avoidance build. Recommended after you have
|
||||
done synchronized the repository (i.e. "repo sync").
|
||||
|
||||
::
|
||||
@@ -617,13 +594,13 @@ Method (in brief)
|
||||
populate_downloads.sh
|
||||
build-pkgs --build-avoidance
|
||||
|
||||
- Use combinations of additional arguments, environment variables, and a
|
||||
* Use combinations of additional arguments, environment variables, and a
|
||||
configuration file unique to the regional office to specify an URL
|
||||
to the reference builds.
|
||||
|
||||
- Using a configuration file to specify the location of your reference build:
|
||||
* Using a configuration file to specify the location of your reference build:
|
||||
|
||||
::
|
||||
::
|
||||
|
||||
mkdir -p $MY_REPO/local-build-data
|
||||
|
||||
@@ -642,18 +619,17 @@ Method (in brief)
|
||||
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
|
||||
EOF
|
||||
|
||||
- Using command-line arguments to specify the location of your reference
|
||||
build:
|
||||
* Using command-line arguments to specify the location of your reference
|
||||
build:
|
||||
|
||||
::
|
||||
::
|
||||
|
||||
build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins
|
||||
|
||||
- Prior to your build attempt, you need to accept the host key.
|
||||
Doing so prevents "rsync" failures on a "yes/no" prompt.
|
||||
You only have to do this once.
|
||||
* Prior to your build attempt, you need to accept the host key. Doing so
|
||||
prevents "rsync" failures on a "yes/no" prompt. You only have to do this once.
|
||||
|
||||
::
|
||||
::
|
||||
|
||||
grep -q $BUILD_AVOIDANCE_HOST $HOME/.ssh/known_hosts
|
||||
if [ $? != 0 ]; then
|
||||
@@ -661,56 +637,52 @@ Method (in brief)
|
||||
fi
|
||||
|
||||
|
||||
- "build-pkgs" does the following:
|
||||
* "build-pkgs" does the following:
|
||||
|
||||
- From newest to oldest, scans the CONTEXTs of the various
|
||||
reference builds. Selects the first (i.e. most recent) context that
|
||||
satisfies the following requirement: every Git the SHA
|
||||
specifies in the CONTEXT is present.
|
||||
- The selected context might be slightly out of date, but not by
|
||||
more than a day. This assumes daily reference builds are run.
|
||||
- If the context has not been previously downloaded, then
|
||||
download it now. This means you need to download select portions of the
|
||||
reference build workspace into the designer's workspace. This
|
||||
includes all the SRPMS, RPMS, MD5SUMS, and miscellaneous supporting
|
||||
files. Downloading these files usually takes about 10 minutes
|
||||
over an office LAN.
|
||||
- The designer could have additional commits or uncommitted changes
|
||||
not present in the reference builds. Affected packages are
|
||||
identified by the differing md5sum's. In these cases, the packages
|
||||
are re-built. Re-builds usually take five or more minutes,
|
||||
depending on the packages that have changed.
|
||||
* From newest to oldest, scans the CONTEXTs of the various reference builds.
|
||||
Selects the first (i.e. most recent) context that satisfies the following
|
||||
requirement: every Git the SHA specifies in the CONTEXT is present.
|
||||
* The selected context might be slightly out of date, but not by more than
|
||||
a day. This assumes daily reference builds are run.
|
||||
* If the context has not been previously downloaded, then download it now.
|
||||
This means you need to download select portions of the reference build
|
||||
workspace into the designer's workspace. This includes all the SRPMS,
|
||||
RPMS, MD5SUMS, and miscellaneous supporting files. Downloading these files
|
||||
usually takes about 10 minutes over an office LAN.
|
||||
* The designer could have additional commits or uncommitted changes not
|
||||
present in the reference builds. Affected packages are identified by the
|
||||
differing md5sum's. In these cases, the packages are re-built. Re-builds
|
||||
usually take five or more minutes, depending on the packages that have changed.
|
||||
|
||||
- What if no valid reference build is found? Then build-pkgs will fall
|
||||
back to a regular build.
|
||||
* What if no valid reference build is found? Then build-pkgs will fall back
|
||||
to a regular build.
|
||||
|
||||
****************
|
||||
Reference builds
|
||||
****************
|
||||
|
||||
- The regional office implements an automated build that pulls the
|
||||
latest StarlingX software and builds it on a regular basis (e.g.
|
||||
daily builds). Jenkins, cron, or similar tools can trigger these builds.
|
||||
- Each build is saved to a unique directory, and preserved for a time
|
||||
that is reflective of how long a designer might be expected to work
|
||||
on a private branch without syncronizing with the master branch.
|
||||
This takes about two weeks.
|
||||
* The regional office implements an automated build that pulls the latest
|
||||
StarlingX software and builds it on a regular basis (e.g. daily builds).
|
||||
Jenkins, cron, or similar tools can trigger these builds.
|
||||
* Each build is saved to a unique directory, and preserved for a time that is
|
||||
reflective of how long a designer might be expected to work on a private branch
|
||||
without syncronizing with the master branch. This takes about two weeks.
|
||||
|
||||
- The *MY_WORKSPACE* directory for the build shall have a common root
|
||||
* The *MY_WORKSPACE* directory for the build shall have a common root
|
||||
directory, and a leaf directory that is a sortable time stamp. The
|
||||
suggested format is *YYYYMMDDThhmmss*.
|
||||
|
||||
.. code:: sh
|
||||
::
|
||||
|
||||
$ sudo apt-get update
|
||||
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
|
||||
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
|
||||
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}
|
||||
sudo apt-get update
|
||||
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
|
||||
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
|
||||
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}
|
||||
|
||||
- Designers can access all build products over the internal network of
|
||||
the regional office. The current prototype employs "rsync". Other
|
||||
protocols that can efficiently share, copy, or transfer large directories
|
||||
of content can be added as needed.
|
||||
* Designers can access all build products over the internal network of the
|
||||
regional office. The current prototype employs "rsync". Other protocols that
|
||||
can efficiently share, copy, or transfer large directories of content can be
|
||||
added as needed.
|
||||
|
||||
^^^^^^^^^^^^^^
|
||||
Advanced usage
|
@@ -1,828 +0,0 @@
|
||||
==========================
|
||||
Build guide StarlingX R1.0
|
||||
==========================
|
||||
|
||||
This section describes the steps for building a StarlingX ISO from the R1.0
|
||||
StarlingX release.
|
||||
|
||||
------------
|
||||
Requirements
|
||||
------------
|
||||
|
||||
The recommended minimum requirements include:
|
||||
|
||||
*********************
|
||||
Hardware requirements
|
||||
*********************
|
||||
|
||||
A workstation computer with:
|
||||
|
||||
- Processor: x86_64 is the only supported architecture
|
||||
- Memory: At least 32GB RAM
|
||||
- Hard Disk: 500GB HDD
|
||||
- Network: Network adapter with active Internet connection
|
||||
|
||||
*********************
|
||||
Software requirements
|
||||
*********************
|
||||
|
||||
A workstation computer with:
|
||||
|
||||
- Operating System: Ubuntu 16.04 LTS 64-bit
|
||||
- Docker
|
||||
- Android Repo Tool
|
||||
- Proxy settings configured (if required)
|
||||
|
||||
- See
|
||||
http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000136.html
|
||||
for more details
|
||||
|
||||
- Public SSH key
|
||||
|
||||
-----------------------------
|
||||
Development environment setup
|
||||
-----------------------------
|
||||
|
||||
This section describes how to set up a StarlingX development system on a
|
||||
workstation computer. After completing these steps, you can
|
||||
build a StarlingX ISO image on the following Linux distribution:
|
||||
|
||||
- Ubuntu 16.04 LTS 64-bit
|
||||
|
||||
****************************
|
||||
Update your operating system
|
||||
****************************
|
||||
|
||||
Before proceeding with the build, ensure your Ubuntu distribution is up to date.
|
||||
You first need to update the local database list of available packages:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ sudo apt-get update
|
||||
|
||||
******************************************
|
||||
Installation requirements and dependencies
|
||||
******************************************
|
||||
|
||||
^^^^
|
||||
User
|
||||
^^^^
|
||||
|
||||
1. Make sure you are a non-root user with sudo enabled when you build the
|
||||
StarlingX ISO. You also need to either use your existing user or create a
|
||||
separate *<user>*:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ sudo useradd -m -d /home/<user> <user>
|
||||
|
||||
2. Your *<user>* should have sudo privileges:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ sudo sh -c "echo '<user> ALL=(ALL:ALL) ALL' >> /etc/sudoers"
|
||||
$ sudo su -c <user>
|
||||
|
||||
^^^
|
||||
Git
|
||||
^^^
|
||||
|
||||
3. Install the required packages on the Ubuntu host system:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ sudo apt-get install make git curl
|
||||
|
||||
4. Make sure to set up your identity using the following two commands.
|
||||
Be sure to provide your actual name and email address:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ git config --global user.name "Name LastName"
|
||||
$ git config --global user.email "Email Address"
|
||||
|
||||
^^^^^^^^^
|
||||
Docker CE
|
||||
^^^^^^^^^
|
||||
|
||||
5. Install the required Docker CE packages in the Ubuntu host system. See
|
||||
`Get Docker CE for
|
||||
Ubuntu <https://docs.docker.com/install/linux/docker-ce/ubuntu/#os-requirements>`__
|
||||
for more information.
|
||||
|
||||
6. Log out and log in to add your *<user>* to the Docker group:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ sudo usermod -aG docker <user>
|
||||
|
||||
^^^^^^^^^^^^^^^^^
|
||||
Android Repo Tool
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
7. Install the required Android Repo Tool in the Ubuntu host system. Follow
|
||||
the steps in the `Installing
|
||||
Repo <https://source.android.com/setup/build/downloading#installing-repo>`__
|
||||
section.
|
||||
|
||||
**********************
|
||||
Install public SSH key
|
||||
**********************
|
||||
|
||||
#. Follow these instructions on GitHub to `Generate a Public SSH
|
||||
Key <https://help.github.com/articles/connecting-to-github-with-ssh>`__.
|
||||
Then upload your public key to your GitHub and Gerrit account
|
||||
profiles:
|
||||
|
||||
- `Upload to
|
||||
Github <https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account>`__
|
||||
- `Upload to
|
||||
Gerrit <https://review.openstack.org/#/settings/ssh-keys>`__
|
||||
|
||||
****************************
|
||||
Create a workspace directory
|
||||
****************************
|
||||
|
||||
#. Create a *starlingx* workspace directory on your system.
|
||||
Best practices dictate creating the workspace directory
|
||||
in your $HOME directory:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ mkdir -p $HOME/starlingx/
|
||||
|
||||
*************************
|
||||
Install stx-tools project
|
||||
*************************
|
||||
|
||||
#. Under your $HOME directory, clone the <stx-tools> project:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ cd $HOME
|
||||
$ git clone https://git.starlingx.io/stx-tools
|
||||
|
||||
#. Navigate to the *<$HOME/stx-tools>* project
|
||||
directory:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ cd $HOME/stx-tools/
|
||||
|
||||
-----------------------------
|
||||
Prepare the base Docker image
|
||||
-----------------------------
|
||||
|
||||
StarlingX base Docker image handles all steps related to StarlingX ISO
|
||||
creation. This section describes how to customize the base Docker image
|
||||
building process.
|
||||
|
||||
********************
|
||||
Configuration values
|
||||
********************
|
||||
|
||||
You can customize values for the StarlingX base Docker image using a
|
||||
text-based configuration file named ``localrc``:
|
||||
|
||||
- ``HOST_PREFIX`` points to the directory that hosts the 'designer'
|
||||
subdirectory for source code, the 'loadbuild' subdirectory for
|
||||
the build environment, generated RPMs, and the ISO image.
|
||||
- ``HOST_MIRROR_DIR`` points to the directory that hosts the CentOS mirror
|
||||
repository.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
localrc configuration file
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Create your ``localrc`` configuration file. For example:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
# tbuilder localrc
|
||||
MYUNAME=<your user name>
|
||||
PROJECT=starlingx
|
||||
HOST_PREFIX=$HOME/starlingx/workspace
|
||||
HOST_MIRROR_DIR=$HOME/starlingx/mirror
|
||||
|
||||
***************************
|
||||
Build the base Docker image
|
||||
***************************
|
||||
|
||||
Once the ``localrc`` configuration file has been customized, it is time
|
||||
to build the base Docker image.
|
||||
|
||||
#. If necessary, you might have to set http/https proxy in your
|
||||
Dockerfile before building the docker image:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
ENV http_proxy " http://your.actual_http_proxy.com:your_port "
|
||||
ENV https_proxy " https://your.actual_https_proxy.com:your_port "
|
||||
ENV ftp_proxy " http://your.actual_ftp_proxy.com:your_port "
|
||||
RUN echo " proxy=http://your-proxy.com:port " >> /etc/yum.conf
|
||||
|
||||
#. The ``tb.sh`` script automates the Base Docker image build:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
./tb.sh create
|
||||
|
||||
----------------------------------
|
||||
Build the CentOS mirror repository
|
||||
----------------------------------
|
||||
|
||||
The creation of the StarlingX ISO relies on a repository of RPM Binaries,
|
||||
RPM Sources, and Tar Compressed files. This section describes how to build
|
||||
this CentOS mirror repository.
|
||||
|
||||
*******************************
|
||||
Run repository Docker container
|
||||
*******************************
|
||||
|
||||
| Run the following commands under a terminal identified as "**One**":
|
||||
|
||||
#. Navigate to the *$HOME/stx-tools/centos-mirror-tool* project
|
||||
directory:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ cd $HOME/stx-tools/centos-mirror-tools/
|
||||
|
||||
#. Launch the Docker container using the previously created base Docker image
|
||||
*<repository>:<tag>*. As /localdisk is defined as the workdir of the
|
||||
container, you should use the same folder name to define the volume.
|
||||
The container starts to run and populate 'logs' and 'output' folders in
|
||||
this directory. The container runs from the same directory in which the
|
||||
scripts are stored.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ docker run -it --volume $(pwd):/localdisk local/$USER-stx-builder:7.4 bash
|
||||
|
||||
*****************
|
||||
Download packages
|
||||
*****************
|
||||
|
||||
#. Inside the Docker container, enter the following commands to download
|
||||
the required packages to populate the CentOS mirror repository:
|
||||
|
||||
::
|
||||
|
||||
# cd localdisk && bash download_mirror.sh
|
||||
|
||||
#. Monitor the download of packages until it is complete. When the download
|
||||
is complete, the following message appears:
|
||||
|
||||
::
|
||||
|
||||
totally 17 files are downloaded!
|
||||
step #3: done successfully
|
||||
IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images
|
||||
for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso"
|
||||
- out/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img
|
||||
- out/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img
|
||||
- out/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz
|
||||
|
||||
***************
|
||||
Verify packages
|
||||
***************
|
||||
|
||||
#. Verify no missing or failed packages exist:
|
||||
|
||||
::
|
||||
|
||||
# cat logs/*_missing_*.log
|
||||
# cat logs/*_failmove_*.log
|
||||
|
||||
#. In case missing or failed packages do exist, which is usually caused by
|
||||
network instability (or timeout), you need to download the packages
|
||||
manually.
|
||||
Doing so assures you get all RPMs listed in
|
||||
*rpms_3rdparties.lst*/*rpms_centos.lst*/*rpms_centos3rdparties.lst*.
|
||||
|
||||
******************
|
||||
Packages structure
|
||||
******************
|
||||
|
||||
The following is a general overview of the packages structure resulting
|
||||
from downloading the packages:
|
||||
|
||||
::
|
||||
|
||||
/home/<user>/stx-tools/centos-mirror-tools/output
|
||||
└── stx-r1
|
||||
└── CentOS
|
||||
└── pike
|
||||
├── Binary
|
||||
│ ├── EFI
|
||||
│ ├── images
|
||||
│ ├── isolinux
|
||||
│ ├── LiveOS
|
||||
│ ├── noarch
|
||||
│ └── x86_64
|
||||
├── downloads
|
||||
│ ├── integrity
|
||||
│ └── puppet
|
||||
└── Source
|
||||
|
||||
*******************************
|
||||
Create CentOS mirror repository
|
||||
*******************************
|
||||
|
||||
Outside your Repository Docker container, in another terminal identified
|
||||
as "**Two**", run the following commands:
|
||||
|
||||
#. From terminal identified as "**Two**", create a *mirror/CentOS*
|
||||
directory under your *starlingx* workspace directory:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ mkdir -p $HOME/starlingx/mirror/CentOS/
|
||||
|
||||
#. Copy the built CentOS Mirror Repository built under
|
||||
*$HOME/stx-tools/centos-mirror-tool* to the *$HOME/starlingx/mirror/*
|
||||
workspace directory:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ cp -r $HOME/stx-tools/centos-mirror-tools/output/stx-r1/ $HOME/starlingx/mirror/CentOS/
|
||||
|
||||
|
||||
-------------------------
|
||||
Create StarlingX packages
|
||||
-------------------------
|
||||
|
||||
*****************************
|
||||
Run building Docker container
|
||||
*****************************
|
||||
|
||||
#. From the terminal identified as "**Two**", create the workspace folder:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ mkdir -p $HOME/starlingx/workspace
|
||||
|
||||
#. Navigate to the *$HOME/stx-tools* project directory:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ cd $HOME/stx-tools
|
||||
|
||||
#. Verify environment variables:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ bash tb.sh env
|
||||
|
||||
#. Run the building Docker container:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ bash tb.sh run
|
||||
|
||||
#. Execute the buiding Docker container:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ bash tb.sh exec
|
||||
|
||||
*********************************
|
||||
Download source code repositories
|
||||
*********************************
|
||||
|
||||
#. From the terminal identified as "**Two**", which is now inside the
|
||||
Building Docker container, start the internal environment:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ eval $(ssh-agent)
|
||||
$ ssh-add
|
||||
|
||||
#. Use the repo tool to create a local clone of the stx-manifest
|
||||
Git repository based on the "r/2018.10" branch:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ cd $MY_REPO_ROOT_DIR
|
||||
$ repo init -u https://git.starlingx.io/stx-manifest -m default.xml -b r/2018.10
|
||||
|
||||
#. Synchronize the repository:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ repo sync -j`nproc`
|
||||
|
||||
#. Create a tarballs repository:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ ln -s /import/mirrors/CentOS/stx-r1/CentOS/pike/downloads/ $MY_REPO/stx/
|
||||
|
||||
Alternatively, you can run the "populate_downloads.sh" script to copy
|
||||
the tarballs instead of using a symlink:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/
|
||||
|
||||
Outside the container
|
||||
|
||||
#. From another terminal identified as "**Three**", create mirror binaries:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ mkdir -p $HOME/starlingx/mirror/CentOS/stx-installer
|
||||
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/stx-installer/initrd.img
|
||||
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/stx-installer/vmlinuz
|
||||
$ cp $HOME/starlingx/mirror/CentOS/stx-r1/CentOS/pike/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/stx-installer/squashfs.img
|
||||
|
||||
**************
|
||||
Build packages
|
||||
**************
|
||||
|
||||
#. Go back to the terminal identified as "**Two**", which is the Building Docker container.
|
||||
|
||||
#. **Temporal!** Build-Pkgs Errors. Be prepared to have some missing /
|
||||
corrupted rpm and tarball packages generated during
|
||||
`Build the CentOS Mirror Repository`_, which will cause the next step
|
||||
to fail. If that step does fail, manually download those missing /
|
||||
corrupted packages.
|
||||
|
||||
#. Update the symbolic links:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ generate-cgcs-centos-repo.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/
|
||||
|
||||
#. Build the packages:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ build-pkgs
|
||||
|
||||
#. **Optional!** Generate-Cgcs-Tis-Repo:
|
||||
|
||||
While this step is optional, it improves performance on subsequent
|
||||
builds. The cgcs-tis-repo has the dependency information that
|
||||
sequences the build order. To generate or update the information, you
|
||||
need to execute the following command after building modified or new
|
||||
packages.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ generate-cgcs-tis-repo
|
||||
|
||||
-------------------
|
||||
Build StarlingX ISO
|
||||
-------------------
|
||||
|
||||
#. Build the image:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ build-iso
|
||||
|
||||
---------------
|
||||
Build installer
|
||||
---------------
|
||||
|
||||
To get your StarlingX ISO ready to use, you must create the initialization
|
||||
files used to boot the ISO, additional controllers, and compute nodes.
|
||||
|
||||
**NOTE:** You only need this procedure during your first build and
|
||||
every time you upgrade the kernel.
|
||||
|
||||
After running "build-iso", run:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ build-pkgs --installer
|
||||
|
||||
This builds *rpm* and *anaconda* packages. Then run:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ update-pxe-network-installer
|
||||
|
||||
The *update-pxe-network-installer* covers the steps detailed in
|
||||
*$MY_REPO/stx/stx-metal/installer/initrd/README*. This script
|
||||
creates three files on
|
||||
*/localdisk/loadbuild/pxe-network-installer/output*.
|
||||
|
||||
::
|
||||
|
||||
new-initrd.img
|
||||
new-squashfs.img
|
||||
new-vmlinuz
|
||||
|
||||
Rename the files as follows:
|
||||
|
||||
::
|
||||
|
||||
initrd.img
|
||||
squashfs.img
|
||||
vmlinuz
|
||||
|
||||
Two ways exist for using these files:
|
||||
|
||||
#. Store the files in the */import/mirror/CentOS/stx-installer/* folder
|
||||
for future use.
|
||||
#. Store the files in an arbitrary location and modify the
|
||||
*$MY_REPO/stx/stx-metal/installer/pxe-network-installer/centos/build_srpm.data*
|
||||
file to point to these files.
|
||||
|
||||
Recreate the *pxe-network-installer* package and rebuild the image:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ build-pkgs --clean pxe-network-installer
|
||||
$ build-pkgs pxe-network-installer
|
||||
$ build-iso
|
||||
|
||||
Your ISO image should be able to boot.
|
||||
|
||||
****************
|
||||
Additional notes
|
||||
****************
|
||||
|
||||
- In order to get the first boot working, this complete procedure needs
|
||||
to be done. However, once the init files are created, these can be
|
||||
stored in a shared location where different developers can make use
|
||||
of them. Updating these files is not a frequent task and should be
|
||||
done whenever the kernel is upgraded.
|
||||
- StarlingX is in active development. Consequently, it is possible that in the
|
||||
future the **0.2** version will change to a more generic solution.
|
||||
|
||||
---------------
|
||||
Build avoidance
|
||||
---------------
|
||||
|
||||
*******
|
||||
Purpose
|
||||
*******
|
||||
|
||||
Greatly reduce build times after using "repo" to syncronized a local
|
||||
repository with an upstream source (i.e. "repo sync").
|
||||
Build avoidance works well for designers working
|
||||
within a regional office. Starting from a new workspace, "build-pkgs"
|
||||
typically requires three or more hours to complete. Build avoidance
|
||||
reduces this step to approximately 20 minutes.
|
||||
|
||||
***********
|
||||
Limitations
|
||||
***********
|
||||
|
||||
- Little or no benefit for designers who refresh a pre-existing
|
||||
workspace at least daily (e.g. download_mirror.sh, repo sync,
|
||||
generate-cgcs-centos-repo.sh, build-pkgs, build-iso). In these cases,
|
||||
an incremental build (i.e. reuse of same workspace without a "build-pkgs
|
||||
--clean") is often just as efficient.
|
||||
- Not likely to be useful to solo designers, or teleworkers that wish
|
||||
to compile on using their home computers. Build avoidance downloads build
|
||||
artifacts from a reference build, and WAN speeds are generally too
|
||||
slow.
|
||||
|
||||
*****************
|
||||
Method (in brief)
|
||||
*****************
|
||||
|
||||
#. Reference Builds
|
||||
|
||||
- A server in the regional office performs regular (e.g. daily)
|
||||
automated builds using existing methods. These builds are called
|
||||
"reference builds".
|
||||
- The builds are timestamped and preserved for some time (i.e. a
|
||||
number of weeks).
|
||||
- A build CONTEXT, which is a file produced by "build-pkgs"
|
||||
at location *$MY_WORKSPACE/CONTEXT*, is captured. It is a bash script that can
|
||||
cd to each and every Git and checkout the SHA that contributed to
|
||||
the build.
|
||||
- For each package built, a file captures the md5sums of all the
|
||||
source code inputs required to build that package. These files are
|
||||
also produced by "build-pkgs" at location
|
||||
*$MY_WORKSPACE//rpmbuild/SOURCES//srpm_reference.md5*.
|
||||
- All these build products are accessible locally (e.g. a regional
|
||||
office) using "rsync".
|
||||
|
||||
**NOTE:** Other protocols can be added later.
|
||||
|
||||
#. Designers
|
||||
|
||||
- Request a build avoidance build. Recommended after you have
|
||||
done synchronized the repository (i.e. "repo sync").
|
||||
|
||||
::
|
||||
|
||||
repo sync
|
||||
generate-cgcs-centos-repo.sh
|
||||
populate_downloads.sh
|
||||
build-pkgs --build-avoidance
|
||||
|
||||
- Use combinations of additional arguments, environment variables, and a
|
||||
configuration file unique to the regional office to specify an URL
|
||||
to the reference builds.
|
||||
|
||||
- Using a configuration file to specify the location of your reference build:
|
||||
|
||||
::
|
||||
|
||||
mkdir -p $MY_REPO/local-build-data
|
||||
|
||||
cat <<- EOF > $MY_REPO/local-build-data/build_avoidance_source
|
||||
# Optional, these are already the default values.
|
||||
BUILD_AVOIDANCE_DATE_FORMAT="%Y%m%d"
|
||||
BUILD_AVOIDANCE_TIME_FORMAT="%H%M%S"
|
||||
BUILD_AVOIDANCE_DATE_TIME_DELIM="T"
|
||||
BUILD_AVOIDANCE_DATE_TIME_POSTFIX="Z"
|
||||
BUILD_AVOIDANCE_DATE_UTC=1
|
||||
BUILD_AVOIDANCE_FILE_TRANSFER="rsync"
|
||||
|
||||
# Required, unique values for each regional office
|
||||
BUILD_AVOIDANCE_USR="jenkins"
|
||||
BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
|
||||
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
|
||||
EOF
|
||||
|
||||
- Using command-line arguments to specify the location of your reference
|
||||
build:
|
||||
|
||||
::
|
||||
|
||||
build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins
|
||||
|
||||
- Prior to your build attempt, you need to accept the host key.
|
||||
Doing so prevents "rsync" failures on a "yes/no" prompt.
|
||||
You only have to do this once.
|
||||
|
||||
::
|
||||
|
||||
grep -q $BUILD_AVOIDANCE_HOST $HOME/.ssh/known_hosts
|
||||
if [ $? != 0 ]; then
|
||||
ssh-keyscan $BUILD_AVOIDANCE_HOST >> $HOME/.ssh/known_hosts
|
||||
fi
|
||||
|
||||
|
||||
- "build-pkgs" does the following:
|
||||
|
||||
- From newest to oldest, scans the CONTEXTs of the various
|
||||
reference builds. Selects the first (i.e. most recent) context that
|
||||
satisfies the following requirement: every Git the SHA
|
||||
specifies in the CONTEXT is present.
|
||||
- The selected context might be slightly out of date, but not by
|
||||
more than a day. This assumes daily reference builds are run.
|
||||
- If the context has not been previously downloaded, then
|
||||
download it now. This means you need to download select portions of the
|
||||
reference build workspace into the designer's workspace. This
|
||||
includes all the SRPMS, RPMS, MD5SUMS, and miscellaneous supporting
|
||||
files. Downloading these files usually takes about 10 minutes
|
||||
over an office LAN.
|
||||
- The designer could have additional commits or uncommitted changes
|
||||
not present in the reference builds. Affected packages are
|
||||
identified by the differing md5sum's. In these cases, the packages
|
||||
are re-built. Re-builds usually take five or more minutes,
|
||||
depending on the packages that have changed.
|
||||
|
||||
- What if no valid reference build is found? Then build-pkgs will fall
|
||||
back to a regular build.
|
||||
|
||||
****************
|
||||
Reference builds
|
||||
****************
|
||||
|
||||
- The regional office implements an automated build that pulls the
|
||||
latest StarlingX software and builds it on a regular basis (e.g.
|
||||
daily builds). Jenkins, cron, or similar tools can trigger these builds.
|
||||
- Each build is saved to a unique directory, and preserved for a time
|
||||
that is reflective of how long a designer might be expected to work
|
||||
on a private branch without syncronizing with the master branch.
|
||||
This takes about two weeks.
|
||||
|
||||
- The *MY_WORKSPACE* directory for the build shall have a common root
|
||||
directory, and a leaf directory that is a sortable time stamp. The
|
||||
suggested format is *YYYYMMDDThhmmss*.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
$ sudo apt-get update
|
||||
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
|
||||
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
|
||||
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}
|
||||
|
||||
- Designers can access all build products over the internal network of
|
||||
the regional office. The current prototype employs "rsync". Other
|
||||
protocols that can efficiently share, copy, or transfer large directories
|
||||
of content can be added as needed.
|
||||
|
||||
^^^^^^^^^^^^^^
|
||||
Advanced usage
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
Can the reference build itself use build avoidance? Yes it can.
|
||||
Can it reference itself? Yes it can.
|
||||
In both these cases, caution is advised. To protect against any possible
|
||||
'divergence from reality', you should limit how many steps you remove
|
||||
a build avoidance build from a full build.
|
||||
|
||||
Suppose we want to implement a self-referencing daily build in an
|
||||
environment where a full build already occurs every Saturday.
|
||||
To protect ourselves from a
|
||||
build failure on Saturday we also want a limit of seven days since
|
||||
the last full build. Your build script might look like this ...
|
||||
|
||||
::
|
||||
|
||||
...
|
||||
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
|
||||
BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
|
||||
FULL_BUILD_DAY="Saturday"
|
||||
MAX_AGE_DAYS=7
|
||||
|
||||
LAST_FULL_BUILD_LINK="$BUILD_AVOIDANCE_DIR/latest_full_build"
|
||||
LAST_FULL_BUILD_DAY=""
|
||||
NOW_DAY=$(date -u "+%A")
|
||||
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
|
||||
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}
|
||||
|
||||
# update software
|
||||
repo init -u ${BUILD_REPO_URL} -b ${BUILD_BRANCH}
|
||||
repo sync --force-sync
|
||||
$MY_REPO_ROOT_DIR/stx-tools/toCOPY/generate-cgcs-centos-repo.sh
|
||||
$MY_REPO_ROOT_DIR/stx-tools/toCOPY/populate_downloads.sh
|
||||
|
||||
# User can optionally define BUILD_METHOD equal to one of 'FULL', 'AVOIDANCE', or 'AUTO'
|
||||
# Sanitize BUILD_METHOD
|
||||
if [ "$BUILD_METHOD" != "FULL" ] && [ "$BUILD_METHOD" != "AVOIDANCE" ]; then
|
||||
BUILD_METHOD="AUTO"
|
||||
fi
|
||||
|
||||
# First build test
|
||||
if [ "$BUILD_METHOD" != "FULL" ] && [ ! -L $LAST_FULL_BUILD_LINK ]; then
|
||||
echo "latest_full_build symlink missing, forcing full build"
|
||||
BUILD_METHOD="FULL"
|
||||
fi
|
||||
|
||||
# Build day test
|
||||
if [ "$BUILD_METHOD" == "AUTO" ] && [ "$NOW_DAY" == "$FULL_BUILD_DAY" ]; then
|
||||
echo "Today is $FULL_BUILD_DAY, forcing full build"
|
||||
BUILD_METHOD="FULL"
|
||||
fi
|
||||
|
||||
# Build age test
|
||||
if [ "$BUILD_METHOD" != "FULL" ]; then
|
||||
LAST_FULL_BUILD_DATE=$(basename $(readlink $LAST_FULL_BUILD_LINK) | cut -d '_' -f 1)
|
||||
LAST_FULL_BUILD_DAY=$(date -d $LAST_FULL_BUILD_DATE "+%A")
|
||||
AGE_SECS=$(( $(date "+%s") - $(date -d $LAST_FULL_BUILD_DATE "+%s") ))
|
||||
AGE_DAYS=$(( $AGE_SECS/60/60/24 ))
|
||||
if [ $AGE_DAYS -ge $MAX_AGE_DAYS ]; then
|
||||
echo "Haven't had a full build in $AGE_DAYS days, forcing full build"
|
||||
BUILD_METHOD="FULL"
|
||||
fi
|
||||
BUILD_METHOD="AVOIDANCE"
|
||||
fi
|
||||
|
||||
#Build it
|
||||
if [ "$BUILD_METHOD" == "FULL" ]; then
|
||||
build-pkgs --no-build-avoidance
|
||||
else
|
||||
build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER
|
||||
fi
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Build failed in build-pkgs"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
build-iso
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Build failed in build-iso"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$BUILD_METHOD" == "FULL" ]; then
|
||||
# A successful full build. Set last full build symlink.
|
||||
if [ -L $LAST_FULL_BUILD_LINK ]; then
|
||||
rm -rf $LAST_FULL_BUILD_LINK
|
||||
fi
|
||||
ln -sf $MY_WORKSPACE $LAST_FULL_BUILD_LINK
|
||||
fi
|
||||
...
|
||||
|
||||
A final note....
|
||||
|
||||
To use the full build day as your avoidance build reference point,
|
||||
modify the "build-pkgs" commands above to use "--build-avoidance-day ",
|
||||
as shown in the following two examples:
|
||||
|
||||
::
|
||||
|
||||
build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $FULL_BUILD_DAY
|
||||
|
||||
# Here is another example with a bit more shuffling of the above script.
|
||||
|
||||
build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $LAST_FULL_BUILD_DAY
|
||||
|
||||
The advantage is that our build is never more than one step removed
|
||||
from a full build. This assumes the full build was successful.
|
||||
|
||||
The disadvantage is that by the end of the week, the reference build is getting
|
||||
rather old. During active weeks, build times could approach build times for
|
||||
full builds.
|
@@ -33,14 +33,13 @@ For information on the StarlingX development process, refer to the following gui
|
||||
Build StarlingX
|
||||
---------------
|
||||
|
||||
For instructions on how to build StarlingX, refer to the following guides (build
|
||||
guides for StarlingX are release-specific).
|
||||
Refer to the StarlingX Build Guide for instructions on how to build a StarlingX
|
||||
ISO image.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
build_guides/r2_release/index
|
||||
build_guides/r1_release/index
|
||||
build_guide
|
||||
|
||||
--------------------
|
||||
Additional resources
|
||||
|
@@ -242,7 +242,7 @@ Unload firewall and disable firewall on boot:
|
||||
Getting the StarlingX ISO image
|
||||
-------------------------------
|
||||
|
||||
Follow the instructions from the :doc:`/contributor/build_guides/r1_release/index` to build a
|
||||
Follow the instructions from the :doc:`/contributor/build_guide` to build a
|
||||
StarlingX ISO image.
|
||||
|
||||
**********
|
||||
|
Reference in New Issue
Block a user