Scott Little 2877e95b34 fix BUILD_IS_SLOW typo

Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: Ic15382ed06635355df2a6c20dbfb5f90d42638e3

2021-09-21 13:39:03 -04:00

48 KiB

Raw Blame History

Packaging Reference

This guide contains information about creating StarlingX packages.

Tips and tricks

Here are some things to consider when creating packages:

Make sure you have the correct BuildRequires dependencies listed. Mock will ensure that these packages are installed in the build environment before trying to build the package in question. Be aware that CentOS often uses -devel for developer packages (which typically include headers and the .so shared library link) whereas StarlingX generally uses -dev. For BuildRequires, you can use the following:
- Instead of virtual/x86_64-wrs-linux-gcc, you can use gcc.
- Instead of glib-2.0, use glib2-devel.
- Instead of sqlite3, use sqlite-devel.
- You can comment out or delete virtual/libc and virtual/x86_64-wrs-linux-compilerlibs.
Typically, you do not need to manually add dependencies due to shared libraries to either Requires or Provides in the spec file. RPM will usually add them automatically.
When your package creates a shared library, make sure to install it with 755 (or other suitably executable) permissions. The automatic dependency handling only runs if the library file is actually executable.
It can be useful to refer to a bitbake-generated spec file, especially for STX-specific packages. Generally the build and install phases are missing from the bitbake-generated spec file and you must add them.
Make sure all necessary files are listed, because the build_srpm phase will only include source files listed in your spec file. In a simple case, this can be just the following line: Source: %{name}-%{version}.tar.gz

STX authored package with a makefile

This section uses fm-common as a sample package to convert.

Locate the root directory for the package.

cd wr-cgcs/layers/cgcs/middleware/fault/recipes-common/fm-common

Create a directory template in the same location as your current bitbake recipe. The presence of a centos directory will identify packages to be compiled for STX on CentOS.
```
mkdir -p centos
mkdir -p centos/files
```
The makefile requires an install build target. However, bitbake will try to use install if it is introduced when it wasn't previously present, so we'll use install_non_bb instead.

The makefile takes the following parameters: DEST_DIR, BIN_DIR, LIB_DIR, INC_DIR, MAJOR, MINOR.

The basic strategy is to construct the directory tree using install -m 755 -d some_dir, then copy files using install -m 755 from_file some_dir/to_file

For versioned libraries, you must create non- or lesser-versioned symlinks, for example:
```
ln -s my_lib.so.1.0 some_dir/my_lib.so.1
```
Typically, you do not need the .so installed on the target (it's generally only needed for building against). If you do, then it would be linked as ln -s my_lib.so.1.0 some_dir/my_lib.so and would usually be part of a -dev package. Don't forget tabs!!!

vi sources/Makefile
install_non_bb:
        install -m 755 -d $(DEST_DIR)$(BIN_DIR)
        install -m 755 -d $(DEST_DIR)$(LIB_DIR)
        install -m 755 -d $(DEST_DIR)$(INC_DIR)
        install -m 755 fmClientCli $(DEST_DIR)$(BIN_DIR)
        install -m 644 fmDbAPI.h $(DEST_DIR)$(INC_DIR)
        install -m 644 fmAPI.h $(DEST_DIR)$(INC_DIR)
        install -m 644 fmThread.h $(DEST_DIR)$(INC_DIR)
        install -m 644 fmAlarm.h $(DEST_DIR)$(INC_DIR)
        install -m 644 $(LIBFMCOMMON_SO).$(MAJOR).$(MINOR) $(DEST_DIR)$(LIB_DIR)
        ln -s $(LIBFMCOMMON_SO).$(MAJOR).$(MINOR) $(DEST_DIR)$(LIB_DIR)/$(LIBFMCOMMON_SO).$(MAJOR)
        ln -s $(LIBFMCOMMON_SO).$(MAJOR).$(MINOR) $(DEST_DIR)$(LIB_DIR)/$(LIBFMCOMMON_SO)

Create an RPM spec file

Put the RPM spec file under the centos directory. This is the rpm version of a bitbake recipe.

vi centos/.spec

RPM spec files support macros and variable. Reference the macro with the %{macro_name} syntax. Many macros are already defined to help you, such as, %{version} refers to the Version: entry in the description section. You can also define your own macros, for example %define local_dir /usr/local creates the macro local_dir. Fill in the following sections:

# A short description of the packaged software
summary:

# Defines what the package will actually be called
Name:

# Should be set to the version of the software being packaged
Version:

# A number that is used to represent the number of times the software, at the present version, has been packaged
Release:

# Give a list of the source files found in the $BUILD_DIR/SOURCES subdirectory
# If git managed, usually just one, a tarball that we will create via build script e.g. Source0: %{name}-%{version}.tar.gz
# Alternatively name the tarball that is brought in via COPY_LIST from $STX_BASE/downloads directory.
# Many source files can be specified... list any additional files in $BUILD_DIR/SOURCES (likely placed there via COPY_LIST)
Source0: %{name}-%{version}.tar.gz
Source1: %{name}.conf
Source2: %(name}.init

# List of build requirements
BuildRequires:

# List of runtime requirements
Requires:

# Describe the package
%description

# Describes how to unpack the compressed packages (the setup macro will do it for you)
%prep
%setup

# invoke 'make'
%build

# invoke 'make install'
%install

# Perhaps call 'make clean' or other cleanup scripts
%clean

# List of files to include in the rpm
%files

# Apply these permission/ownership overrides and the files that follow
%defattr(-,root,root,-)

# Send this file to the standard path for documentation
%doc some_file

# Create a directory when the rpm is installed
%dir some_directory

# Install this file on this path
%some_file_path

One spec file can create multiple related rpm files. One common usage is to place the *.so file (but not the *.so.* files) for a shared library and its .h interface files in a -dev or -devel package. The Bitbake naming convention is -dev, while CentOS uses -devel. Most of the StarlingX packages use the -dev naming convention.

You can create additional packages with %package -n package_name, or %package suffix which creates a package named first_package_name-suffix.

Following %package, add a unique Summary:, and override any values from the first package. These additional packages also require a %description, %files sections, and may require %prep, %setup, %build, %install, %clean sections. All these extra sections use the same -n package_name or suffix arguments to define which package they apply to. Refer to the example below, which uses fm-common-dev.

%package -n fm-common-dev Summary: STX Platform Fault Management
Common Package - Development files Group: devel Requires: fm-common =
%{version}-%{release} ... %description -n fm-common-dev ... %files -n
fm-common-dev ...

vi centos/fm-common.spec
%define local_dir /usr/local
%define local_bindir %{local_dir}/bin

Summary: STX Platform Fault Management Common Package
Name: fm-common
Version: 1.0
Release: r26.0
License: Apache-2.0
Group: base
URL: unknown
Source0: %{name}-%{version}.tar.gz
BuildRequires: util-linux
BuildRequires: postgresql-devel
BuildRequires: libuuid-devel
Requires: postgresql

%package -n fm-common-dev
Summary: STX Platform Fault Management Common Package - Development files
Group: devel
Requires: fm-common = %{version}-%{release}

%description
STX platform Fault Management Client Library that provides APIs for
applications to raise/clear/update active alarms.

%description -n fm-common-dev
STX platform Fault Management Client Library that provides APIs for
applications to raise/clear/update active alarms.  This package contains
symbolic links, header files, and related items necessary for software
development.

%prep
%setup
%build
VER=%{version}
MAJOR=`echo $VER | awk -F . '{print $1}'`
MINOR=`echo $VER | awk -F . '{print $2}'`
make  MAJOR=$MAJOR MINOR=$MINOR %{?_smp_mflags}

%install
rm -rf $RPM_BUILD_ROOT
VER=%{version}
MAJOR=`echo $VER | awk -F . '{print $1}'`
MINOR=`echo $VER | awk -F . '{print $2}'`
make DEST_DIR=$RPM_BUILD_ROOT BIN_DIR=%{local_bindir} LIB_DIR=%{_libdir} INC_DIR=%{_includedir} MAJOR=$MAJOR MINOR=$MINOR install_non_bb

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
%doc LICENSE
%dir %{local_dir}
%dir %{local_bindir}
%{local_bindir}/*

%files -n fm-common-dev
%defattr(-,root,root,-)
%{_includedir}/*
%{_libdir}/*

Set up srpm build script

This section describes two options: you can choose to use the default srpm build script or create your own srpm build script.

Use the default srpm build script, and just provide a few data definitions, in bash format.

vi centos/build_srpm.data
Variables you can define. All paths are relative to PKG_BASE (see below) unless prefixed by another environment variable, eg. STX_BASE
SRC_DIR=my_src_dir
# Tar everything found in this subdirectory. Define this if source need to be collected into a tarball in $BUILD_DIR/SOURCES.
# Tar file name and version are derived from your .spec file and/or PKG-INFO file.  Alternatively you may define TAR_NAME and VERSION in your 'build_srpm.data' file

COPY_LIST="$STX_BASE/downloads/my_tar.tar.gz  my_extra_files_dir  my_script_dir/my_script"
 # A Space separated list of paths to copy.  In the build directory they will by copied into the $BUILD_DIR/SOURCES directory alongside any tarball you may have created via SRC_DIR.

EXCLUDE_LIST_FROM_TAR=my_omit_dir  my_script_dir/my_omit_script"
# Omit these paths under SRC_DIR from the tar

TAR_NAME=nova
VERSION=1.1
# The name and version of the package.  Assuming SRC_DIR is specified, a tarball with the name "$TAR_NAME-$VERSION.tar.gz" will be created from from the contents of $SRC_DIR.
# NOTE: These values are pulled automatically from your .spec ("%global service" or "Name:" and "Version:") or PKG-INFO ("Name:" and "Version:").  You can override the automatic valuation by supplying your own values.

COPY_LIST_TO_TAR=my_extra_files_dir  my_script_dir/my_script"
 # A Space separated list of paths to copy.  These are added to the tarball itself.  Try to avoid using this... we prefer that tarballs of 3rd party origin remain unaltered.

TIS_BASE_SRCREV=9af04d8480da4359f766e59c45ac23f199b138b0
# Last git commit prior to start of TC content

TIS_PATCH_VER=GITREVCOUNT+3
# Version is passed into .spec file as rpmmacro to modify package release string.
# Can contain 'GITREVCOUNT', the count of commits since TIS_BASE_SRCREV

BUILD_IS_BIG=6
# An estimate of the disk requirements in GB.  This package won't build on a tmpfs based build environment small than this value.
# If a build fails on a tmpfs based node due to disk space exhaustion, then at minimum this value must be greater than the tpmfs size
# as seen in the "MOCKCHAIN_RESOURCE_ALLOCATION=0:10:10:5" log.

BUILD_IS_SLOW=5
# An estimate of the build time on disk in minutes.
# A larger value will tell the build system to schedule this job sooner
# rather than later, to maximize parallelism.

Create your own srpm build script. The goal of the script is to use the rpmbuild tool to create one or more .src.rpm files in the $BUILD_DIR/SRPMS directory. The directory itself will be created by rpmbuild.

vi centos/build_srpm

A sample build script might look like this:

source "$SRC_BASE/build-tools/spec-utils"

SRC_DIR=$STX_BASE/git/libvirt

if [ -f PKG-INFO ]: then
   VERSION=$(grep '^Version:' PKG-INFO | awk -F ': ' '{print $2}' | sed -e 's/^[[:space:]]*//')
   TAR_NAME=$(grep '^Name:' PKG-INFO | awk -F ': ' '{print $2}' | sed -e 's/^[[:space:]]*//')
else
   SPEC_PATH=$(ls $BUILD_DIR/SPECS/*.spec)
   VERSION=$(spec_evaluate '%{version}' "$SPEC_PATH" 2>> /dev/null)
   NAME=$(spec_evaluate '%{name} "$SPEC_PATH" 2>> /dev/null)
fi

BUILD_DIR="$RPMBUILD_BASE"
EXTRA_FILES="libvirt-1.2.12/libvirt.logrotate "
EXTRA_FILES+="libvirt-1.2.12/libvirt.lxc "
EXTRA_FILES+="libvirt-1.2.12/libvirt.qemu "
EXTRA_FILES+="libvirt-1.2.12/libvirt.uml "

mkdir -p $BUILD_DIR/SRPMS

TAR="$TAR_NAME-$VERSION.tar.gz"
TAR_PATH="$BUILD_DIR/SOURCES/$TAR"

TAR_NEEDED=0
if [ -f $TAR_PATH ]; then
   n=`find . -cnewer $TAR_PATH -and ! -path './.git*' \
                               -and ! -path './build/*' \
                               -and ! -path './.pc/*' \
                               -and ! -path './patches/*' \
                               -and ! -path './$DISTRO/*' \
                               -and ! -path './pbr-*.egg/*' \
                               | wc -l`
   if [ $n -gt 0 ]; then
       TAR_NEEDED=1
   fi
else
   TAR_NEEDED=1
fi

if [ $TAR_NEEDED -gt 0 ]; then
   tar czvf $TAR_PATH $EXTRA_FILES \
                      -C $(dirname $SRC_DIR) $(basename $SRC_DIR) \
                      --exclude '.git/' \
                      --exclude '.git*' \
                      --transform "s,^libvirt-1.2.12,extra," \
                      --transform "s,^$(basename $SRC_DIR),$TAR_NAME-$VERSION,"
fi


for SPEC in `ls $BUILD_DIR/SPECS`; do
   SPEC_PATH="$BUILD_DIR/SPECS/$SPEC"
   RELEASE=$(grep '^Release:' $SPEC_PATH | awk -F ': ' '{print $2}' | sed -e 's/^[[:space:]]*//')
   NAME=`echo $SPEC | sed 's/.spec$//'`
   SRPM="$NAME-$VERSION-$RELEASE.src.rpm"
   SRPM_PATH="$BUILD_DIR/SRPMS/$SRPM"

   BUILD_NEEDED=0
   if [ -f $SRPM_PATH ]; then
       n=`find . -cnewer $SRPM_PATH | wc -l`
       if [ $n -gt 0 ]; then
           BUILD_NEEDED=1
       fi
   else
       BUILD_NEEDED=1
   fi

   if [ $BUILD_NEEDED -gt 0 ]; then
       rpmbuild -bs $SPEC_PATH --define="%_topdir $BUILD_DIR"
   fi
done

The idea is to tar up any source from git into a tarball into $BUILD_DIR/SOURCES and/or copy tarballs from a downloads directory. Copy any supporting files into $BUILD_DIR/SOURCES also. Finally, use rpmbuild to create a .src.rpm. DO NOT try to compile the binary .rpm.

Details for build-srpm.data

This section describes the contents of the build-srpm.data file, which customizes the setup when you build SRPM (Source RPM) files.

Pre-defined path variables

You may use any of the following:

SRC_BASE      # Usually the same as $MY_REPO

STX_BASE      # $MY_REPO/stx

PKG_BASE      # Base dir for the package. Listed in centos_pkg_dir.
              # Directory in which CentOS directory is found.

PATCHES_BASE  # Directory where source patches for the package can be found, typically: $PKG_BASE/$DISTRO/patches

FILE_BASE     # Directory where supplemental files for the package can be found, typically: $PKG_BASE/$DISTRO/files
              # Files that can be copied into $BUILD_DIR/SOURCES prior to assembling the .src.rpm

CGCS_BASE     # same as $STX_BASE, being phased out in favour of PKG_BASE
              # May be removed in a future release, do not use.

User-defined variables

You may define any of the following variables:

SRC_DIR=my_src_dir
# Tar everything found in this subdirectory. Define this if source need to be collected into a tarball in $BUILD_DIR/SOURCES.
# Tar file name and version are derived from your .spec file and/or PKG-INFO file. Alternatively you may define TAR_NAME and VERSION in your 'build_srpm.data' file

COPY_LIST="$STX_BASE/downloads/my_tar.tar.gz my_extra_files_dir my_script_dir/my_script"
# A Space separated list of paths to copy. In the build directory they will by copied into the $BUILD_DIR/SOURCES directory alongside any tarball you may have created via SRC_DIR.

EXCLUDE_LIST_FROM_TAR=my_omit_dir my_script_dir/my_omit_script"
# Omit these paths under SRC_DIR from the tar

TAR_NAME=nova
VERSION=1.1
# The name and version of the package. Assuming SRC_DIR is specified, a tarball with the name "$TAR_NAME-$VERSION.tar.gz" will be created from from the contents of $SRC_DIR.
# NOTE: These values are pulled automatically from your .spec ("%global service" or "Name:" and "Version:") or PKG-INFO ("Name:" and "Version:"). You can override the automatic valuation by supplying your own values.

COPY_LIST_TO_TAR=my_extra_files_dir my_script_dir/my_script"
# A Space separated list of paths to copy. These are added to the tarball itself. Try to avoid using this... we prefer that tarballs of 3rd party origin remain unaltered.

BUILD_IS_BIG=6
# An estimate of the disk requirements in GB. This package won't build on a tmpfs based build environment small than this value.
# If a build fails on a tmpfs based node due to disk space exhaustion, then at minimum this value must be greater than the tpmfs size
# as seen in the "MOCKCHAIN_RESOURCE_ALLOCATION=0:10:10:5" log.

BUILD_IS_SLOW=5
# An estimate of the build time on disk in minutes.
# A larger value will tell the build system to schedule this job sooner rather than later, to maximize parallelism.

TIS_PATCH_VER=<expression>
                         # An integer, or one of the supported
                         # variables listed below, or the sum of
                         # variables and integers.
                         # e.g.
                         # TIS_PATCH_VER=PKG_GITREVCOUNT+SRC_GITREVCOUNT+5

       PKG_GITREVCOUNT # Count git revisions relative to PKG_BASE.
                         # Optionally only count from PKG_BASE_SRCREV

       SRC_GITREVCOUNT # Count git revisions relative to SRC_DIR.
                         # Optionally only count from SRC_BASE_SRCREV

       GITREVCOUNT # Deprecated, please use SRC_GITREVCOUNT instead.
                         # Count git revisions relative to SRC_DIR.
                         # Optionally only count from TIS_BASE_SRCREV

       OTHER_GITREVCOUNT # count git revisions from all sources excluding
                         # PKG_BASE and SRC_DIR
                         # Optionally only count from
                         # BASE_SRCREV_FOR_PATH[<path>]

PKG_BASE_SRCREV=<sha> # Limit PKG_GITREVCOUNT revision count to
                            # commits since <sha>

SRC_BASE_SRCREV=<sha> # Limit SRC_GITREVCOUNT revision count to
                            # commits since <sha>

TIS_BASE_SRCREV=<sha> # Deprecated, please use SRC_BASE_SRCREV
                            # instead
                            # Limit GITREVCOUNT revision count to commits
                            # since <sha>

BASE_SRCREV_FOR_PATH[<path>]=[<sha>|OTHER_PKG_BASE_SRCREV]
                            # Limit OTHER_GITREVCOUNT revision count for
                            # commits under <path> to commits since <sha>.
                            # If <path> is the PKG_BASE of another package
                            # (not the current package) then the keyword
                            # 'OTHER_PKG_BASE_SRCREV' can be used to extract
                            # the 'PKG_BASE_SRCREV' value of the other
                            # package.
                            #
                            # The <path> can reference variables like
                            # $STX_BASE and $GIT_BASE.

OPT_DEP_LIST=<path-list> # Add a space separated list of paths that
                              # don't contribute to the content of a src.rpm
                              # but do contribute to triggering a rebuild,
                              # and possibly modifying the TIS_PATCH_VER via
                              # use of OTHER_GITREVCOUNT.

OPT_DEP_LIST_FOR_BUILD_TYPE[<build-type>]=<path-list>
                              # For a specific build type only, add a space
                              # separated list of paths that don't
                              # contribute to the content of src.rpm,
                              # but do contribute to triggering a
                              # rebuild, and possibly modifying the
                              # TIS_PATCH_VER via use of OTHER_GITREVCOUNT.

Add your package to package list file

The file name is centos_pkg_dirs. There is a separate package list file for each git repo. When adding your package, use the relative path from the root of the git to the desired PKG_BASE.

Using package libvirt as an example:

Spec file: $MY_REPO/stx/integ/virt/libvirt/centos/libvirt.spec
PKG_BASE: %BLUE%$MY_REPO/stx%ENDCOLOR%/%RED%integ/virt/libvirt%ENDCOLOR%
Root of its git tree: $MY_REPO/stx
centos_pkg_dirs file: %BLUE%$MY_REPO/stx%ENDCOLOR%/centos_pkg_dirs

The new entry needs to be: %RED%integ/virt/libvirt%ENDCOLOR% echo "virt/libvirt" >> "$MY_REPO/stx/integ/centos_pkg_dirs"

Build your package

Complete instructions can be found in the Create StarlingX packages section of the Build Guide <create_stx_pkgs>.

build-rpms fm-common

Commit your changes

git add centos/files/*
git add centos/fm-common.spec
git add centos/build_srpm*

# If you added PKG-INFO
git add PKG-INFO

git add sources/Makefile
git commit

Package without a makefile

In this case, the process is generally similar to the above instructions, but the %build section has alternative commands or may be omitted altogether. The install phase is done directly in the RPM spec file.

For example, the collector package has no makefile. Its spec file is at stx/utilities/tools/collector/centos/collector.spec. There is no "build" phase, because the files are scripts. However, the files must be unpacked and installed into the appropriate locations. The relevant bits look like this:

Source0: %{name}-%{version}.tar.gz

%prep
%setup

%install
mkdir -p %{buildroot}

install -d 755 -d %{buildroot}%{_sysconfdir}/collect.d
install -d 755 -d %{buildroot}%{_sysconfdir}/collect
install -d 755 -d %{buildroot}/usr/local/sbin
install -d 755 -d %{buildroot}/usr/local/bin
install -d 755 -d %{buildroot}%{_sbindir}

install -m 755 collect %{buildroot}/usr/local/sbin/collect
install -m 755 collect_host %{buildroot}/usr/local/sbin/collect_host
install -m 755 collect_date %{buildroot}/usr/local/sbin/collect_date
install -m 755 collect_utils %{buildroot}/usr/local/sbin/collect_utils
install -m 755 collect_parms %{buildroot}/usr/local/sbin/collect_parms
install -m 755 collect_mask_passwords %{buildroot}/usr/local/sbin/collect_mask_passwords
install -m 755 expect_done %{buildroot}/usr/local/sbin/expect_done

install -m 755 collect_sysinv.sh %{buildroot}%{_sysconfdir}/collect.d/collect_sysinv
install -m 755 collect_psqldb.sh %{buildroot}%{_sysconfdir}/collect.d/collect_psqldb
install -m 755 collect_openstack.sh %{buildroot}%{_sysconfdir}/collect.d/collect_openstack
install -m 755 collect_networking.sh %{buildroot}%{_sysconfdir}/collect.d/collect_networking
install -m 755 collect_ceph.sh %{buildroot}%{_sysconfdir}/collect.d/collect_ceph
install -m 755 collect_sm.sh %{buildroot}%{_sysconfdir}/collect.d/collect_sm
install -m 755 collect_tc.sh %{buildroot}%{_sysconfdir}/collect.d/collect_tc
install -m 755 collect_nfv_vim.sh %{buildroot}%{_sysconfdir}/collect.d/collect_nfv_vim
install -m 755 collect_ovs.sh %{buildroot}%{_sysconfdir}/collect.d/collect_ovs
install -m 755 collect_patching.sh %{buildroot}%{_sysconfdir}/collect.d/collect_patching
install -m 755 collect_coredump.sh %{buildroot}%{_sysconfdir}/collect.d/collect_coredump
install -m 755 collect_crash.sh %{buildroot}%{_sysconfdir}/collect.d/collect_crash
install -m 755 collect_ima.sh %{buildroot}%{_sysconfdir}/collect.d/collect_ima
install -m 755 collect_fm.sh %{buildroot}%{_sysconfdir}/collect.d/collect_fm
install -m 755 collect_containerization.sh %{buildroot}%{_sysconfdir}/collect.d/collect_containerization
install -m 755 collect_dc.sh %{buildroot}%{_sysconfdir}/collect.d/collect_dc

install -m 755 etc.exclude %{buildroot}%{_sysconfdir}/collect/etc.exclude
install -m 755 run.exclude %{buildroot}%{_sysconfdir}/collect/run.exclude

ln -sf /usr/local/sbin/collect %{buildroot}/usr/local/bin/collect
ln -sf /usr/local/sbin/collect %{buildroot}%{_sbindir}/collect

Either the setup or autosetup macro can be used to unpack the Source0 tarball in %{_builddir}. Next, a number of files are explicitly installed into %{buildroot}. Most of these come from the unpacked tarball. One file comes from the original source directory, since it wasn't moved over to %{_builddir} in the prep phase. All the other sections in the spec file are treated the same way as the above package.

OpenStack and Python packages

These packages are similar to the above as far as requirements, dependencies, etc. The build phase is a bit different since there is no compilation step involved.

Source0:          %{name}-%{version}.tar.gz

%prep
%setup -q

%build
%{__python2} setup.py build

%install
%{__python2} setup.py install -O1 --skip-build --root %{buildroot}

mkdir -p %{buildroot}%{_sysconfdir}/bash_completion.d
install -pm 644 tools/nova.bash_completion \
    %{buildroot}%{_sysconfdir}/bash_completion.d/nova

In this case, the setup macro unpacks the Source0 tarball in the prep phase. Next the setup.py that is part of the Python package handles the actual process of building and installing the Python code. During the install phase, a separate file is installed which isn't handled by the Python installer.

CentOS package with appended patches

This example uses the parted package from CentOS and appends a few patches of our own.

Assuming this package has never been modified by StarlingX previously, find a home for the package. This example uses $MY_REPO/stx/integ/filesystem/parted.

MY_GIT_ROOT=$MY_REPO/stx/integ
MY_PKG_ROOT=$MY_GIT_ROOT/filesystem/parted
mkdir -p $MY_PKG_ROOT
cd $MY_PKG_ROOT

# You must also add the new package directory to your local centos_pkg_dirs file if not already present
echo filesystem/parted >> $MY_GIT_ROOT/centos_pkg_dirs

Create a directory skeleton. The patches directory contains patches applied to the tarball inside the src.rpm. The meta_patches directory contains patches applied to the rpmbuild directory of the .src.rpm and are used to modify the SPECS/*.spec file, or the SOURCES/* files.
```
mkdir -p centos/meta_patches
mkdir -p centos/patches
touch centos/meta_patches/PATCH_ORDER
echo TIS_PATCH_VER=1 > centos/build_srpm.data
```
Tell build-srpms where to find the src.rpm in the srpm_path file. Two locations are supported for source RPMS: the $MY_REPO/cgcs-centos-repo directory is for source RPMS provided by CentOS. Tell the build system to search for a source RPM using the mirror: prefix when you specify the path to the source RPM.
```
find -L $MY_REPO/cgcs-centos-repo/  -name 'parted*.src.rpm'
/localdisk/<username>/designer/test/cgcs-root/cgcs-centos-repo/Source/parted-3.1-23.el7.src.rpm

echo "mirror:Source/parted-3.1-23.el7.src.rpm" > centos/srpm_path
```
The other location supported by build-srpms is the downloads subgit of your layers (found in $MY_REPO/stx/downloads for CGCS, for example). This should be used for packages that CentOS does not provide. In this case, place the source RPM you downloaded in the directory, and use the repo: prefix when creating your srpm_path file. The base path for repo: is $MY_REPO, so specify paths from that point.
```
cp ~/Downloads/somepackage-1.2-3.src.rpm $MY_REPO/stx/downloads
echo "repo:stx/downloads/somepackage-1.2-3.src.rpm" > centos/srpm_path
```

(Optional) Set up a git environment to create your patch. The SRPM will be extracted, then the source tarball will be extracted and patched.

# clean any prior edit environment that might exist
#
build-srpms --edit --clean parted

#
# Create a working edit environment
#
build-srpms --edit parted
...
===== 'parted' has been extracted to for editing. =====
===== Metadata can be found at: /localdisk/loadbuild/<username>/test/std/srpm_work/parted/rpmbuild
===== Source code can be found at: /localdisk/loadbuild/<username>/test/std/srpm_work/parted/gits/parted.spec/parted-3.1

#
# Create you patch
#
cd  $MY_WORKSPACE/std/srpm_work/parted/gits/parted.spec/parted-3.1

edit

# create a patch
git add
git commit -m 'foo bar'
git format-patch -n HEAD^
   0001-foo-bar.patch

# Copy it to the repo... probabbly best to drop the missleading 0001 prefix
mv 0001-foo-bar.patch $MY_PKG_ROOT/centos/patches/foo-bar.patch

# undo the commit, if patch failed testing and needs more work
git reset --soft HEAD~

#
# Now create a meta_patch to add your patch to the spec file
#
cd $MY_WORKSPACE/std/srpm_work/parted/rpmbuild

# modify Release to include tis patch version, add our patch to the end of the patch list
vi SPECS/parted.spec
   ....
   Release: 23.el7%{?_tis_dist}.%{tis_patch_ver}
   ....
   Patch34: 0034-tests-Use-wait_for_dev_to_-functions.patch

   # WRS
   Patch35: syscalls.patch
   Patch36: foo-bar.patch
   ...
   # if the %prep/%setup section does not use 'git am %{patches}' which automatically applies all patches,
   # then it should contain a list of %patch macros (one per patch) and you will have to append your patch to the end of the list.
   # %patch macros are basically just invocations of the 'patch' command to apply your patches.  e.g. like this
   %patch35 -p1
   %patch36 -p1


git add SPECS/parted.spec

# Optionally edit and add files in SOURCES/
git commit -m 'meta foo bar'
git format-patch -n HEAD^
   0001-meta-foo-bar.patch

# Copy it to the repo... probabbly best to drop the missleading 0001 prefix
mv 0001-meta-foo-bar.patch $MY_PKG_ROOT/centos/meta_patches/meta-foo-bar.patch

# undo the commit, if patch failed testing and needs more work
git reset --soft HEAD~

#
# Finally add the meta patch to the patch order file... so that the patches vs the spec file apply in the correct order
#
echo meta-foo-bar.patch >> $MY_PKG_ROOT/centos/meta_patches/PATCH_ORDER

(Optional) Create a build_srpm.data file and add a COPY_LIST which identifies additional files to copy into SOURCES before finalizing the srpm.
```
vi  $MY_PKG_ROOT/centos/build_srpm.data
COPY_LIST="files/resizepart.sh"
```
Note

The order is SRPM extracted first, then copy centos/patches/* into SOURCES, then apply meta patches, then copy anything named in COPY_LIST into SOURCES. COPY_LIST items are not available for meta patching. Just edit them directly.
Build the final src.rpm with your new changes inserted.
```
build-srpms --clean parted
build-srpms parted
```

Build the rpm.

build-rpms --clean parted
build-rpms parted
Verify contents of rpm:
build-rpms --no-descendants parted
cd $MY_WORKSPACE/rpmbuild
mkdir TMP; cd TMP
rpm2cpio parted-3.1-23.el7.tis.1.x86_64.rpm | cpio -ivmd

Test. If it fails, go back to step 4.

Commit.

cd $MY_PKG_ROOT
git add meta_patches/PATCH_ORDER  meta_patches/meta-foo-bar.patch  patches/foo-bar.patch
# add these as required
git add centos/srpm_path
git add centos/build_srpm.data
git commit -F
... rebase/push...

Discard the working directory and its temporary gits created in step 4.
```
build-srpms --clean --edit parted
```

Up-version a CentOS package with appended patches

Suppose you have a patched srpm resource-agents-3.9.5-54.el7_2.8.src.rpm and you want to migrate to a newer srpm, for example, resource-agents-3.9.5-54.el7_2.8.src.rpm.

First question: How do I know if there is a newer version available? If the local CentOS mirror has been updated, you can scan it for new srpms that have local WRS patches using:

find_patched_srpms_needing_upgrade
   /localdisk/designer/<username>/test/cgcs-root/stx/recipes-cgl/cluster-resource-agents/centos/srpm_path: resource-agents-3.9.5-54.el7_2.8.src.rpm ==> resource-agents-3.9.5-54.el7_2.9.src.rpm

This tells you that there are patches for resource-agents-3.9.5-54.el7_2.8.src.rpm, but a newer version is available (resource-agents-3.9.5-54.el7_2.9.src.rpm). You should probably migrate the patches to the new resource-agents-3.9.5-54.el7_2.9.src.rpm. No output means no migrations are required at this time.

Check out the old version with our patches.

build-srpms resource-agents --edit --clean
build-srpms resource-agents --edit

Point the srpm_path at the new source rpm.

vi $MY_REPO/cgcs-root/stx/recipes-cgl/cluster-resource-agents/centos/srpm_path
  mirror:CentOS/vault.centos.org/7.2.1511/updates/Source/SPackages/resource-agents-3.9.5-54.el7_2.9.src.rpm

Check out the new version without our patches.

build-srpms resource-agents --edit --no-meta-patch
...
===== 'resource-agents' has been extracted for editing. =====
===== Metedata can be found at: /localdisk/loadbuild/<username>/test/std/srpm_work/resource-agents/rpmbuild
===== Source code can be found at: /localdisk/loadbuild/<username>/teststd//srpm_work/resource-agents/gits/resource-agents.spec/ClusterLabs-resource-agents-5434e96

You now have two directories, each a git tree with two branches, one based on the old srpm, one on the new srpm.

cd $MY_WORKSPACE/std/srpm_work/resource-agents/rpmbuild

git branch --all
  master
  resource-agents-3.9.5-54.el7_2.8
* resource-agents-3.9.5-54.el7_2.9

cd $MY_WORKSPACE/std/srpm_work/resource-agents/gits/resource-agents.spec/ClusterLabs-resource-agents-5434e96
git branch --all
  master
  resource-agents-3.9.5-54.el7_2.8
* resource-agents-3.9.5-54.el7_2.9

Show the commit histories of the two branches to identify the work to do.

cd $MY_WORKSPACE/std/srpm_work/resource-agents/gits/resource-agents.spec/ClusterLabs-resource-agents-5434e96

git log resource-agents-3.9.5-54.el7_2.8 --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%ci) %C(bold blue)<%an>%Creset' --abbrev-commit
* be18338 - (HEAD, resource-agents-3.9.5-54.el7_2.8) WRS: Patch99: exportfs_accept_ipv6.patch (2016-04-29 01:11:03 -0400)
* 70308b7 - WRS: Patch98: pgsql.patch (2016-04-29 01:11:02 -0400)
* 802a93a - WRS: Patch97: lvm_vg_activation.patch (2016-04-29 01:11:00 -0400)
* 96acac3 - WRS: Patch96: umount-in-namespace.patch (2016-04-29 01:11:00 -0400)
* 24b1bb1 - WRS: Patch95: copyright.patch (2016-04-29 01:10:59 -0400)
* cb5c3a9 - WRS: Patch94: ipaddr2_check_if_state.patch (2016-04-29 01:10:58 -0400)
* 600cda6 - WRS: Patch93: new_ocf_return_codes.patch (2016-04-29 01:10:57 -0400)
* 5f40f0b - WRS: Patch92: filesystem_rmon.patch (2016-04-29 01:10:56 -0400)
* facf036 - (tag: pre_wrs_resource-agents-3.9.5-54.el7_2.8) Patch91: bz1316633-backup-and-restore-rabbitmq-users-during-resource-re.patch (2016-04-29 01:10:32 -0400)
* 48cbdac - Patch90: bz1311180-rabbitmq-cluster-forget-stopped-cluster-nodes.patch (2016-04-29 01:10:31 -0400)


git log resource-agents-3.9.5-54.el7_2.9 --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%ci) %C(bold blue)<%an>%Creset' --abbrev-commit
* 746eb3d - (HEAD, tag: pre_wrs_resource-agents-3.9.5-54.el7_2.9, resource-agents-3.9.5-54.el7_2.9) Patch94: bz1318744-galera-no-grastate.patch (2016-04-29 01:12:16 -0400)
* f6fc3f0 - Patch93: bz1318744-galera-heuristic-recovered.patch (2016-04-29 01:12:15 -0400)
* dc76b27 - Patch92: bz1318744-galera-crash-recovery.patch (2016-04-29 01:12:14 -0400)
* facf036 - (tag: pre_wrs_resource-agents-3.9.5-54.el7_2.8) Patch91: bz1316633-backup-and-restore-rabbitmq-users-during-resource-re.patch (2016-04-29 01:10:32 -0400)
* 48cbdac - Patch90: bz1311180-rabbitmq-cluster-forget-stopped-cluster-nodes.patch (2016-04-29 01:10:31 -0400)

The image below shows the color-coded results of the comparison.

The patches in green with WRS: in the patch name must be ported.
The commit in orange noted with tag: was the last common commit.
The patches in blue are new patches from upstream that may interfere with our patches.

Package providing an init script

The packages/recipes that provide an init script (found under /etc/init.d/ and corresponding /etc/rc.d/) must be converted to systemd services. Within STX there are two kinds of services:

Services that must be started on boot.
Services that are monitored by pmond (and automatically restarted in case of failure). In this case, we don't want systemd to restart a process in case of failure.

The general idea about converting those services is to still keep the script file installed under /etc/init.d/ in the short term and wrap it in a systemd service. This is needed so that service dependencies can be evaluated by systemd and start them in the right sequence on boot. Re-using the same init script at the moment is more convenient for supporting the same code base for wrlinux and CentOS packaging (no need to support multiple version of the same script).

Given an example init script /etc/init.d/helloworld, create a corresponding helloworld.service file:

[Unit]
Description=Helloworld service

# Configures requirement dependencies on other services. If this service gets activated, the
# services listed here will be activated as well. If one of the other services gets deactivated or
# its activation fails, this service will be deactivated.
Requires=

# A weaker version of Requires. Services listed in this option will be started if the configuring
# unit is. However, if the listed services fail to start, this has no impact on the current service.
Wants=

# The next two directives are important for specifying the services dependancies. This will indicate
# to systemd in which order services should be started. This does not imply a dependency
# relationship ('Requires' or 'Wants') and must be used in conjunction with one of the above
# directives if this is desired.

# The services listed in this directive will be started before starting the current service.
After=syslog.target network.target

# The services listed in this directive will not be started until the current service is marked as
# started if they are activated at the same time.
Before=

[Service]
# Usual types are 'oneshot', 'simple' and 'forking'.
# oneshot: The process is not a daemon running in the background. Systemd will wait that the process
#          exit before continuing.
# simple: Systemd will fork the process.
# forking: The process will call fork().  It is recommended to use 'PIDfile' in this case so that
#          systemd can identify the process.
Type=oneshot

# Commands with their arguments that are executed when this service is started
ExecStart=/etc/init.d/helloworld start

# Additional commands that are executed before command in ExecStart
ExecStartPre=

# Additional commands that are executed after command in ExecStart
ExecStartPost=

# Commands to execute to stop the service
ExecStop=/etc/init.d/helloworld stop

# Additional commands that are executed after the service is stopped
ExecStopPost=

# Commands to execute to trigger a configuration reload in the service
ExecReload=

# Only useful when 'Type' is 'forking'.  PID file is important if the process is monitored by pmond.
PIDFile=/var/run/helloworld.pid

# Configures whether the service shall be restarted when the service process exits.
# Should be 'always' is systemd should monitor the process.
# Should be 'no' (default) if process is monitored by pmond.
Restart=

[Install]
# This enables to service to be started on boot. It is equivalent of init runlevel (see below for equivalence).  This
# means that the current service will be started when the listed unit is started.  All the other services that are
# also dependent on this target will also get started, the order will depends on the 'After' and 'Before' options
# specified above.
WantedBy=multi-user.target

Once the systemd service file is defined, it needs to be packaged in the rpm. The helloworld.service file must be installed in the /etc/systemd/system/ directory (recommended for STX services) by the rpm spec. It's important to set file permissions to 664. The systemd services must be enabled on install and disabled on uninstalled. This is achieved by using the systemd command systemctl enable (and disable). However, it is recommended to use the macros already defined by rpm. For example:

%install
install -d -m 755 %{buildroot}/etc/systemd/system/
install -p -D -m 664 helloworld.service %{buildroot}/etc/systemd/system/helloworld.service

# Enable the service and start it.
%post
%systemd_post helloworld.service

# Disable the service and stop it.
%preun
%systemd_preun helloworld.service

# Try to restart the service.  Usefull on upgrades.
%postun
%systemd_postun_with_restart helloworld.service

Systemd searches these directories for services (in order of precedence, the first item has higher precedence than the one following):

/etc/systemd/system/ Recommended and reserved location for system administrator customized services
/run/systemd/system/ Mainly used by CentOS. Don't pollute this directory.
/usr/lib/systemd/system/

CentOS is distributed with a number of predefined targets that are more or less similar to the standard set of runlevels. Use the following comparison table for specifying your WantedBy target.

runlevel0.target -> poweroff.target
runlevel1.target -> rescue.target
runlevel2.target -> multi-user.target
runlevel3.target -> multi-user.target
runlevel4.target -> multi-user.target
runlevel5.target -> graphical.target
runlevel6.target -> reboot.target

Convert to systemd service (without intermediate init script):

The exact PID file name that is used by pmond can be found by looking on an installed system under /etc/pmon.d/. Look for pidfile= in the corresponding pmond configuration file for the service.
Some of the init scripts are using start-stop-daemon to manage processes. The start-stop-package has been ported to CentOS (it is not supported by CentOS). Make sure the init script is providing the right --pidfile argument that matches the pmond service configuration. A process that is started by start-stop-daemon with the --daemon or -d options indicates that the processes will be forked, in which case the systemd service Type should be forking. Else if the script or process doesn't fork, the Type should be simple which will tell systemd to fork it.
Hints for defining service startup order (Before and After):
- Systemd determines when a service should start using the After= and Before= values found in the [Unit] section of the service file. These values take a space-separated list of other services or targets, where a target is effectively a collection of services. For example, the log management service has the following values:
```
[Unit]
Description=StarlingX Log Management
After=network.target syslog-ng.service iscsid.service sw-patch.service
Before=config.service pmon.service
```
  In this example, the log management service can't start until software is patched, and a network, a disk and the syslog-ng service are all up and running.
- The init script might contain some specific information about the order. Look for Required-Start and Required-Stop in it.

Troubleshooting (at runtime):

Enable a service: systemctl enable
Disable a service: systemctl disable
Start a service: systemctl start
Stop a service: systemctl stop
Get service status: systemctl status
Get list of running units ordered by time to init: systemd-analyze blame
Show systemd logging: journalctl /usr/lib/systemd/systemd
Show logging of a specific service: journalctl _SYSTEMD_UNIT=

After you've converted a package

Each repo has a centos_iso_image.inc file, which contains the list of packages to be included in the installation iso relevant to the repo. The build-iso step merges the individual centos_iso_image.inc files to generate the complete list.

48 KiB Raw Blame History