starlingx/config
Code Issues Proposed changes

StarlingX open source release updates

Browse Source 
Signed-off-by: Dean Troyer <dtroyer@gmail.com>
This commit is contained in:
Dean Troyer
2018-05-30 16:17:02 -07:00
parent ddded39cb9
commit 9b95aa0a35
1076 changed files with 209271 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
CONTRIBUTORS.wrs Normal file
View File

@@ -0,0 +1,12 @@
The following contributors from Wind River have developed the seed code in this
repository. We look forward to community collaboration and contributions for
additional features, enhancements and refactoring.
Contributors:
=============
Bart Wensley <Barton.Wensley@windriver.com>
John Kung <John.Kung@windriver.com>
Don Penney <Don.Penney@windriver.com>
Matt Peters <Matt.Peters@windriver.com>
Tao Liu <Tao.Liu@windriver.com>
David Sullivan <David.Sullivan@windriver.com>

202
LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

5
README.rst Normal file
View File

@@ -0,0 +1,5 @@
==========
stx-config
==========
StarlingX Configuration Management

6
compute-huge/.gitignore vendored Normal file
View File

@@ -0,0 +1,6 @@
!.distro
.distro/centos7/rpmbuild/RPMS
.distro/centos7/rpmbuild/SRPMS
.distro/centos7/rpmbuild/BUILD
.distro/centos7/rpmbuild/BUILDROOT
.distro/centos7/rpmbuild/SOURCES/compute-huge*tar.gz

13
compute-huge/PKG-INFO Normal file
View File

@@ -0,0 +1,13 @@
Metadata-Version: 1.1
Name: compute-huge
Version: 1.0
Summary: Initial compute node hugepages and reserved cpus configuration
Home-page:
Author: Windriver
Author-email: info@windriver.com
License: Apache-2.0
Description: Initial compute node hugepages and reserved cpus configuration
Platform: UNKNOWN

8
compute-huge/bin/topology Normal file
View File

@@ -0,0 +1,8 @@
#!/bin/bash
#
# Copyright (c) 2013-2014 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
python /usr/bin/topology.pyc

4
compute-huge/centos/build_srpm.data Normal file
View File

@@ -0,0 +1,4 @@
SRC_DIR="compute-huge"
COPY_LIST_TO_TAR="bin"
COPY_LIST="$SRC_DIR/LICENSE"
TIS_PATCH_VER=10

85
compute-huge/centos/compute-huge.spec Normal file
View File

@@ -0,0 +1,85 @@
Summary: Initial compute node hugepages and reserved cpus configuration
Name: compute-huge
Version: 1.0
Release: %{tis_patch_ver}%{?_tis_dist}
License: Apache-2.0
Group: base
Packager: Wind River <info@windriver.com>
URL: unknown
Source0: %{name}-%{version}.tar.gz
Source1: LICENSE
BuildRequires: systemd-devel
Requires: systemd
Requires: python
Requires: /bin/systemctl
%description
Initial compute node hugepages and reserved cpus configuration
%define local_bindir /usr/bin/
%define local_etc_initd /etc/init.d/
%define local_etc_nova /etc/nova/
%define local_etc_goenabledd /etc/goenabled.d/
%define debug_package %{nil}
%prep
%setup
%build
%{__python} -m compileall topology.py
%install
# compute init scripts
install -d -m 755 %{buildroot}%{local_etc_initd}
install -p -D -m 755 affine-platform.sh %{buildroot}%{local_etc_initd}/affine-platform.sh
install -p -D -m 755 compute-huge.sh %{buildroot}%{local_etc_initd}/compute-huge.sh
# utility scripts
install -p -D -m 755 cpumap_functions.sh %{buildroot}%{local_etc_initd}/cpumap_functions.sh
install -p -D -m 755 task_affinity_functions.sh %{buildroot}%{local_etc_initd}/task_affinity_functions.sh
install -p -D -m 755 log_functions.sh %{buildroot}%{local_etc_initd}/log_functions.sh
install -d -m 755 %{buildroot}%{local_bindir}
install -p -D -m 755 ps-sched.sh %{buildroot}%{local_bindir}/ps-sched.sh
# TODO: Only ship pyc ?
install -p -D -m 755 topology.py %{buildroot}%{local_bindir}/topology.py
install -p -D -m 755 topology.pyc %{buildroot}%{local_bindir}/topology.pyc
install -p -D -m 755 affine-interrupts.sh %{buildroot}%{local_bindir}/affine-interrupts.sh
install -p -D -m 755 set-cpu-wakeup-latency.sh %{buildroot}%{local_bindir}/set-cpu-wakeup-latency.sh
install -p -D -m 755 bin/topology %{buildroot}%{local_bindir}/topology
# compute config data
install -d -m 755 %{buildroot}%{local_etc_nova}
install -p -D -m 755 compute_reserved.conf %{buildroot}%{local_etc_nova}/compute_reserved.conf
install -p -D -m 755 compute_hugepages_total.conf %{buildroot}%{local_etc_nova}/compute_hugepages_total.conf
# goenabled check
install -d -m 755 %{buildroot}%{local_etc_goenabledd}
install -p -D -m 755 compute-huge-goenabled.sh %{buildroot}%{local_etc_goenabledd}/compute-huge-goenabled.sh
# systemd services
install -d -m 755 %{buildroot}%{_unitdir}
install -p -D -m 664 affine-platform.sh.service %{buildroot}%{_unitdir}/affine-platform.sh.service
install -p -D -m 664 compute-huge.sh.service %{buildroot}%{_unitdir}/compute-huge.sh.service
%post
/bin/systemctl enable affine-platform.sh.service >/dev/null 2>&1
/bin/systemctl enable compute-huge.sh.service >/dev/null 2>&1
%clean
rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
%{local_bindir}/*
%{local_etc_initd}/*
%{local_etc_goenabledd}/*
%config(noreplace) %{local_etc_nova}/compute_reserved.conf
%config(noreplace) %{local_etc_nova}/compute_hugepages_total.conf
%{_unitdir}/compute-huge.sh.service
%{_unitdir}/affine-platform.sh.service

202
compute-huge/compute-huge/LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

62
compute-huge/compute-huge/affine-interrupts.sh Normal file
View File

@@ -0,0 +1,62 @@
#!/bin/bash
################################################################################
# Copyright (c) 2015-2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
################################################################################
#
# Purpose:
# Affine the interface IRQ to specified cpulist.
#
# Usage: /usr/bin/affine-interrupts.sh interface cpulist
#
# Define minimal path
PATH=/bin:/usr/bin:/usr/local/bin
# logger setup
WHOAMI=`basename $0`
LOG_FACILITY=user
LOG_PRIORITY=info
TMPLOG=/tmp/${WHOAMI}.log
# LOG() - generates log and puts in temporary file
function LOG()
{
logger -t "${0##*/}[$$]" -p ${LOG_FACILITY}.${LOG_PRIORITY} "$@"
echo "${0##*/}[$$]" "$@" >> ${TMPLOG}
}
function INFO()
{
MSG="INFO"
LOG "${MSG} $@"
}
function ERROR()
{
MSG="ERROR"
LOG "${MSG} $@"
}
if [ "$#" -ne 2 ]; then
ERROR "Interface name and cpulist are required"
exit 1
fi
interface=$1
cpulist=$2
# Find PCI device matching interface, keep last matching device name
dev=$(find /sys/devices -name "${interface}" | \
perl -ne 'print $1 if /([[:xdigit:]]{4}:[[:xdigit:]]{2}:[[:xdigit:]]{2}\.[[:xdigit:]])\/[[:alpha:]]/;')
# Obtain all IRQs for this device
irq=$(cat /sys/bus/pci/devices/${dev}/irq 2>/dev/null)
msi_irqs=$(ls /sys/bus/pci/devices/${dev}/msi_irqs 2>/dev/null | xargs)
INFO $LINENO "affine ${interface} (dev:${dev} irq:${irq} msi_irqs:${msi_irqs}) with cpus (${cpulist})"
for i in $(echo "${irq} ${msi_irqs}"); do echo $i; done | \
xargs --no-run-if-empty -i{} \
/bin/bash -c "[[ -e /proc/irq/{} ]] && echo ${cpulist} > /proc/irq/{}/smp_affinity_list" 2>/dev/null
exit 0

170
compute-huge/compute-huge/affine-platform.sh Executable file
View File

@@ -0,0 +1,170 @@
#!/bin/bash
################################################################################
# Copyright (c) 2013 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
################################################################################
# Define minimal path
PATH=/bin:/usr/bin:/usr/local/bin
LOG_FUNCTIONS=${LOG_FUNCTIONS:-"/etc/init.d/log_functions.sh"}
CPUMAP_FUNCTIONS=${CPUMAP_FUNCTIONS:-"/etc/init.d/cpumap_functions.sh"}
TASK_AFFINITY_FUNCTIONS=${TASK_AFFINITY_FUNCTIONS:-"/etc/init.d/task_affinity_functions.sh"}
source /etc/init.d/functions
[[ -e ${LOG_FUNCTIONS} ]] && source ${LOG_FUNCTIONS}
[[ -e ${CPUMAP_FUNCTIONS} ]] && source ${CPUMAP_FUNCTIONS}
[[ -e ${TASK_AFFINITY_FUNCTIONS} ]] && source ${TASK_AFFINITY_FUNCTIONS}
linkname=$(readlink -n -f $0)
scriptname=$(basename $linkname)
# Enable debug logs
LOG_DEBUG=1
. /etc/platform/platform.conf
################################################################################
# Affine all running tasks to the CPULIST provided in the first parameter.
################################################################################
function affine_tasks
{
local CPULIST=$1
local PIDLIST
local RET=0
# Affine non-kernel-thread tasks (excluded [kthreadd] and its children) to all available
# cores. They will be reaffined to platform cores later on as part of nova-compute
# launch.
log_debug "Affining all tasks to all available CPUs..."
affine_tasks_to_all_cores
RET=$?
if [ $RET -ne 0 ]; then
log_error "Some tasks failed to be affined to all cores."
fi
# Get number of logical cpus
N_CPUS=$(cat /proc/cpuinfo 2>/dev/null | \
awk '/^[pP]rocessor/ { n +=1 } END { print (n>0) ? n : 1}')
# Calculate platform cores cpumap
PLATFORM_COREMASK=$(cpulist_to_cpumap ${CPULIST} ${N_CPUS})
# Set default IRQ affinity
echo ${PLATFORM_COREMASK} > /proc/irq/default_smp_affinity
# Affine all PCI/MSI interrupts to platform cores; this overrides
# irqaffinity boot arg, since that does not handle IRQs for PCI devices
# on numa nodes that do not intersect with platform cores.
PCIDEVS=/sys/bus/pci/devices
declare -a irqs=()
irqs+=($(cat ${PCIDEVS}/*/irq 2>/dev/null | xargs))
irqs+=($(ls ${PCIDEVS}/*/msi_irqs 2>/dev/null | grep -E '^[0-9]+$' | xargs))
# flatten list of irqs, removing duplicates
irqs=($(echo ${irqs[@]} | tr ' ' '\n' | sort -nu))
log_debug "Affining all PCI/MSI irqs(${irqs[@]}) with cpus (${CPULIST})"
for i in ${irqs[@]}; do
/bin/bash -c "[[ -e /proc/irq/${i} ]] && echo ${CPULIST} > /proc/irq/${i}/smp_affinity_list" 2>/dev/null
done
if [[ "$subfunction" == *"compute,lowlatency" ]]; then
# Affine work queues to platform cores
echo ${PLATFORM_COREMASK} > /sys/devices/virtual/workqueue/cpumask
echo ${PLATFORM_COREMASK} > /sys/bus/workqueue/devices/writeback/cpumask
# On low latency compute reassign the per cpu threads rcuc, ksoftirq,
# ktimersoftd to FIFO along with the specified priority
PIDLIST=$( ps -e -p 2 |grep rcuc | awk '{ print $1; }')
for PID in ${PIDLIST[@]}
do
chrt -p -f 4 ${PID} 2>/dev/null
done
PIDLIST=$( ps -e -p 2 |grep ksoftirq | awk '{ print $1; }')
for PID in ${PIDLIST[@]}
do
chrt -p -f 2 ${PID} 2>/dev/null
done
PIDLIST=$( ps -e -p 2 |grep ktimersoftd | awk '{ print $1; }')
for PID in ${PIDLIST[@]}
do
chrt -p -f 3 ${PID} 2>/dev/null
done
fi
return 0
}
################################################################################
# Start Action
################################################################################
function start
{
local RET=0
echo -n "Starting ${scriptname}: "
## Check whether we are root (need root for taskset)
if [ $UID -ne 0 ]; then
log_error "require root or sudo"
RET=1
return ${RET}
fi
## Define platform cpulist to be thread siblings of core 0
PLATFORM_CPULIST=$(get_platform_cpu_list)
# Affine all tasks to platform cpulist
affine_tasks ${PLATFORM_CPULIST}
RET=$?
if [ ${RET} -ne 0 ]; then
log_error "Failed to affine tasks ${PLATFORM_CPULIST}, rc=${RET}"
return ${RET}
fi
print_status ${RET}
return ${RET}
}
################################################################################
# Stop Action - don't do anything
################################################################################
function stop
{
local RET=0
echo -n "Stopping ${scriptname}: "
print_status ${RET}
return ${RET}
}
################################################################################
# Restart Action
################################################################################
function restart() {
stop
start
}
################################################################################
# Main Entry
#
################################################################################
case "$1" in
start)
start
;;
stop)
stop
;;
restart|reload)
restart
;;
status)
echo -n "OK"
;;
*)
echo $"Usage: $0 {start|stop|restart|reload|status}"
exit 1
esac
exit $?

14
compute-huge/compute-huge/affine-platform.sh.service Normal file
View File

@@ -0,0 +1,14 @@
[Unit]
Description=Titanium Cloud Affine Platform
After=syslog.service network.service dbus.service sw-patch.service
Before=compute-huge.sh.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/init.d/affine-platform.sh start
ExecStop=/etc/init.d/affine-platform.sh stop
ExecReload=/etc/init.d/affine-platform.sh restart
[Install]
WantedBy=multi-user.target

24
compute-huge/compute-huge/compute-huge-goenabled.sh Normal file
View File

@@ -0,0 +1,24 @@
#!/bin/bash
#
# Copyright (c) 2014,2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
#
# compute-huge.sh "goenabled" check.
#
# If a problem was detected during configuration of huge pages and compute
# resources then the board is not allowed to enable.
#
COMPUTE_HUGE_GOENABLED="/var/run/compute_huge_goenabled"
source "/etc/init.d/log_functions.sh"
source "/usr/bin/tsconfig"
if [ -e ${VOLATILE_COMPUTE_CONFIG_COMPLETE} -a ! -f ${COMPUTE_HUGE_GOENABLED} ]; then
log_error "compute-huge.sh CPU configuration check failed. Failing goenabled check."
exit 1
fi
exit 0

1512
compute-huge/compute-huge/compute-huge.sh Executable file
View File

File diff suppressed because it is too large Load Diff

14
compute-huge/compute-huge/compute-huge.sh.service Normal file
View File

@@ -0,0 +1,14 @@
[Unit]
Description=Titanium Cloud Compute Huge
After=syslog.service network.service affine-platform.sh.service sw-patch.service
Before=sshd.service sw-patch-agent.service sysinv-agent.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/init.d/compute-huge.sh start
ExecStop=/etc/init.d/compute-huge.sh stop
ExecReload=/etc/init.d/compute-huge.sh restart
[Install]
WantedBy=multi-user.target

0
compute-huge/compute-huge/compute_hugepages_total.conf Normal file
View File

78
compute-huge/compute-huge/compute_reserved.conf Normal file
View File

@@ -0,0 +1,78 @@
################################################################################
# Copyright (c) 2013-2015 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
################################################################################
# COMPUTE Node configuration parameters for reserved memory and physical cores
# used by Base software and VSWITCH. These are resources that libvirt cannot use.
#
################################################################################
#
# Enable compute-huge.sh console debug logs (uncomment)
#
################################################################################
LOG_DEBUG=1
################################################################################
#
# List of logical CPU instances available in the system. This value is used
# for auditing purposes so that the current configuration can be checked for
# validity against the actual number of logical CPU instances in the system.
#
################################################################################
COMPUTE_CPU_LIST="0-1"
################################################################################
#
# List of Base software resources reserved per numa node. Each array element
# consists of a 3-tuple formatted as: <node>:<memory>:<cores>.
#
# Example: To reserve 1500MB and 1 core on NUMA node0, and 1500MB and 1 core
# on NUMA node1, the variable must be specified as follows.
# COMPUTE_BASE_MEMORY=("node0:1500MB:1" "node1:1500MB:1")
#
################################################################################
COMPUTE_BASE_RESERVED=("node0:8000MB:1" "node1:2000MB:0" "node2:2000MB:0" "node3:2000MB:0")
################################################################################
#
# List of HugeTLB memory descriptors to configure. Each array element
# consists of a 3-tuple descriptor formatted as: <node>:<pgsize>:<pgcount>.
# The NUMA node specified must exist and the HugeTLB pagesize must be a valid
# value such as 2048kB or 1048576kB.
#
# For example, to request 256 x 2MB HugeTLB pages on NUMA node0 and node1 the
# variable must be specified as follows.
# COMPUTE_VSWITCH_MEMORY=("node0:2048kB:256" "node1:2048kB:256")
#
################################################################################
COMPUTE_VSWITCH_MEMORY=("node0:1048576kB:1" "node1:1048576kB:1" "node2:1048576kB:1" "node3:1048576kB:1")
################################################################################
#
# List of VSWITCH physical cores reserved for VSWITCH applications.
#
# Example: To reserve 2 cores on NUMA node0, and 2 cores on NUMA node1, the
# variable must be specified as follows.
# COMPUTE_VSWITCH_CORES=("node0:2" "node1:2")
#
################################################################################
COMPUTE_VSWITCH_CORES=("node0:2" "node1:0" "node2:0" "node3:0")
################################################################################
#
# List of HugeTLB memory descriptors to configure for Libvirt. Each array element
# consists of a 3-tuple descriptor formatted as: <node>:<pgsize>:<pgcount>.
# The NUMA node specified must exist and the HugeTLB pagesize must be a valid
# value such as 2048kB or 1048576kB.
#
# For example, to request 256 x 2MB HugeTLB pages on NUMA node0 and node1 the
# variable must be specified as follows.
# COMPUTE_VM_MEMORY_2M=("node0:2048kB:256" "node1:2048kB:256")
#
################################################################################
COMPUTE_VM_MEMORY_2M=()
COMPUTE_VM_MEMORY_1G=()

399
compute-huge/compute-huge/cpumap_functions.sh Normal file
View File

@@ -0,0 +1,399 @@
#!/bin/bash
################################################################################
# Copyright (c) 2013-2015 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
################################################################################
source /etc/platform/platform.conf
################################################################################
# Utility function to expand a sequence of numbers (e.g., 0-7,16-23)
################################################################################
function expand_sequence
{
SEQUENCE=(${1//,/ })
DELIMITER=${2:-","}
LIST=
for entry in ${SEQUENCE[@]}
do
range=(${entry/-/ })
a=${range[0]}
b=${range[1]:-${range[0]}}
for i in $(seq $a $b)
do
LIST="${LIST}${DELIMITER}${i}"
done
done
echo ${LIST:1}
}
################################################################################
# Append a string to comma separated list string
################################################################################
function append_list() {
local PUSH=$1
local LIST=$2
if [ -z "${LIST}" ]
then
LIST=${PUSH}
else
LIST="${LIST},${PUSH}"
fi
echo ${LIST}
return 0
}
################################################################################
# Condense a sequence of numbers to a list of ranges (e.g, 7-12,15-16)
################################################################################
function condense_sequence() {
local arr=( $(printf '%s\n' "$@" | sort -n) )
local first
local last
local cpulist=""
for ((i=0; i < ${#arr[@]}; i++))
do
num=${arr[$i]}
if [[ -z $first ]]; then
first=$num
last=$num
continue
fi
if [[ num -ne $((last + 1)) ]]; then
if [[ first -eq last ]]; then
cpulist=$(append_list ${first} ${cpulist})
else
cpulist=$(append_list "${first}-${last}" ${cpulist})
fi
first=$num
last=$num
else
: $((last++))
fi
done
if [[ first -eq last ]]; then
cpulist=$(append_list ${first} ${cpulist})
else
cpulist=$(append_list "${first}-${last}" ${cpulist})
fi
echo "$cpulist"
}
################################################################################
# Converts a CPULIST (e.g., 0-7,16-23) to a CPUMAP (e.g., 0x00FF00FF). The
# CPU map is returned as a string representation of a large hexidecimal
# number but without the leading "0x" characters.
#
################################################################################
function cpulist_to_cpumap
{
local CPULIST=$1
local NR_CPUS=$2
local CPUMAP=0
local CPUID=0
if [ -z "${NR_CPUS}" ] || [ ${NR_CPUS} -eq 0 ]
then
echo 0
return 0
fi
for CPUID in $(expand_sequence $CPULIST " ")
do
if [ "${CPUID}" -lt "${NR_CPUS}" ]; then
CPUMAP=$(echo "${CPUMAP} + (2^${CPUID})" | bc -l)
fi
done
echo "obase=16;ibase=10;${CPUMAP}" | bc -l
return 0
}
################################################################################
# Converts a CPUMAP (e.g., 0x00FF00FF) to a CPULIST (e.g., 0-7,16-23). The
# CPUMAP is expected in hexidecimal (base=10) form without the leading "0x"
# characters.
#
################################################################################
function cpumap_to_cpulist
{
local CPUMAP=$(echo "obase=10;ibase=16;$1" | bc -l)
local NR_CPUS=$2
local list=()
local cpulist=""
for((i=0; i < NR_CPUS; i++))
do
## Since 'bc' does not support any bitwise operators this expression:
## if (CPUMAP & (1 << CPUID))
## has to be rewritten like this:
## if (CPUMAP % (2**(CPUID+1)) > ((2**(CPUID)) - 1))
##
ISSET=$(echo "scale=0; (${CPUMAP} % 2^(${i}+1)) > (2^${i})-1" | bc -l)
if [ "${ISSET}" -ne 0 ]
then
list+=($i)
fi
done
cpulist=$(condense_sequence ${list[@]} )
echo "$cpulist"
return 0
}
################################################################################
# Bitwise NOT of a hexidecimal representation of a CPULIST. The value is
# returned as a hexidecimal value but without the leading "0x" characters
#
################################################################################
function invert_cpumap
{
local CPUMAP=$(echo "obase=10;ibase=16;$1" | bc -l)
local NR_CPUS=$2
local INVERSE_CPUMAP=0
for CPUID in $(seq 0 $((NR_CPUS - 1)));
do
## See comment in previous function
ISSET=$(echo "scale=0; (${CPUMAP} % 2^(${CPUID}+1)) > (2^${CPUID})-1" | bc -l)
if [ "${ISSET}" -eq 1 ]; then
continue
fi
INVERSE_CPUMAP=$(echo "${INVERSE_CPUMAP} + (2^${CPUID})" | bc -l)
done
echo "obase=16;ibase=10;${INVERSE_CPUMAP}" | bc -l
return 0
}
################################################################################
# Builds the complement representation of a CPULIST
#
################################################################################
function invert_cpulist
{
local CPULIST=$1
local NR_CPUS=$2
local CPUMAP=$(cpulist_to_cpumap ${CPULIST} ${NR_CPUS})
cpumap_to_cpulist $(invert_cpumap ${CPUMAP} ${NR_CPUS}) ${NR_CPUS}
return 0
}
################################################################################
# in_list() - check whether item is contained in list
# param: item
# param: list (i.e. 0-3,8-11)
# returns: 0 - item is contained in list;
# 1 - item is not contained in list
#
################################################################################
function in_list() {
local item="$1"
local list="$2"
# expand list format 0-3,8-11 to a full sequence {0..3} {8..11}
local exp_list=$(echo ${list} | \
sed -e 's#,# #g' -e 's#\([0-9]*\)-\([0-9]*\)#{\1\.\.\2}#g')
local e
for e in $(eval echo ${exp_list})
do
[[ "$e" == "$item" ]] && return 0
done
return 1
}
################################################################################
# any_in_list() - check if any item of sublist is contained in list
# param: sublist
# param: list
# returns: 0 - an item of sublist is contained in list;
# 1 - no sublist items contained in list
#
################################################################################
function any_in_list() {
local sublist="$1"
local list="$2"
local e
local exp_list
# expand list format 0-3,8-11 to a full sequence {0..3} {8..11}
exp_list=$(echo ${list} | \
sed -e 's#,# #g' -e 's#\([0-9]*\)-\([0-9]*\)#{\1\.\.\2}#g')
declare -A a_list
for e in $(eval echo ${exp_list})
do
a_list[$e]=1
done
# expand list format 0-3,8-11 to a full sequence {0..3} {8..11}
exp_list=$(echo ${sublist} | \
sed -e 's#,# #g' -e 's#\([0-9]*\)-\([0-9]*\)#{\1\.\.\2}#g')
declare -A a_sublist
for e in $(eval echo ${exp_list})
do
a_sublist[$e]=1
done
# Check if any element of sublist is in list
for e in "${!a_sublist[@]}"
do
if [[ "${a_list[$e]}" == 1 ]]
then
return 0 # matches
fi
done
return 1 # no match
}
################################################################################
# Return list of CPUs reserved for platform
################################################################################
function get_platform_cpu_list() {
## Define platform cpulist based on engineering a number of cores and
## whether this is a combo or not, and include SMT siblings.
if [[ $subfunction = *compute* ]]; then
RESERVE_CONF="/etc/nova/compute_reserved.conf"
[[ -e ${RESERVE_CONF} ]] && source ${RESERVE_CONF}
if [ -n "$PLATFORM_CPU_LIST" ];then
echo "$PLATFORM_CPU_LIST"
return 0
fi
fi
local PLATFORM_SOCKET=0
local PLATFORM_START=0
local PLATFORM_CORES=1
if [ "$nodetype" = "controller" ]; then
((PLATFORM_CORES+=1))
fi
local PLATFORM_CPULIST=$(topology_to_cpulist ${PLATFORM_SOCKET} ${PLATFORM_START} ${PLATFORM_CORES})
echo ${PLATFORM_CPULIST}
}
################################################################################
# Return list of CPUs reserved for vswitch
################################################################################
function get_vswitch_cpu_list() {
## Define default avp cpulist based on engineered number of platform cores,
## engineered avp cores, and include SMT siblings.
if [[ $subfunction = *compute* ]]; then
VSWITCH_CONF="/etc/vswitch/vswitch.conf"
[[ -e ${VSWITCH_CONF} ]] && source ${VSWITCH_CONF}
if [ -n "$VSWITCH_CPU_LIST" ];then
echo "$VSWITCH_CPU_LIST"
return 0
fi
fi
local N_CORES_IN_PKG=$(cat /proc/cpuinfo 2>/dev/null | \
awk '/^cpu cores/ {n = $4} END { print (n>0) ? n : 1 }')
# engineer platform cores
local PLATFORM_CORES=1
if [ "$nodetype" = "controller" ]; then
((PLATFORM_CORES+=1))
fi
# engineer AVP cores
local AVP_SOCKET=0
local AVP_START=${PLATFORM_CORES}
local AVP_CORES=1
if [ ${N_CORES_IN_PKG} -gt 4 ]; then
((AVP_CORES+=1))
fi
local AVP_CPULIST=$(topology_to_cpulist ${AVP_SOCKET} ${AVP_START} ${AVP_CORES})
echo ${AVP_CPULIST}
}
################################################################################
# vswitch_expanded_cpu_list() - compute the vswitch cpu list, including it's siblings
################################################################################
function vswitch_expanded_cpu_list() {
list=$(get_vswitch_cpu_list)
# Expand vswitch cpulist
vswitch_cpulist=$(expand_sequence ${list} " ")
cpulist=""
for e in $vswitch_cpulist
do
# claim hyperthread siblings if SMT enabled
SIBLINGS_CPULIST=$(cat /sys/devices/system/cpu/cpu${e}/topology/thread_siblings_list 2>/dev/null)
siblings_cpulist=$(expand_sequence ${SIBLINGS_CPULIST} " ")
for s in $siblings_cpulist
do
in_list ${s} ${cpulist}
if [ $? -eq 1 ]
then
cpulist=$(append_list ${s} ${cpulist})
fi
done
done
echo "$cpulist"
return 0
}
################################################################################
# platform_expanded_cpu_list() - compute the platform cpu list, including it's siblings
################################################################################
function platform_expanded_cpu_list() {
list=$(get_platform_cpu_list)
# Expand platform cpulist
platform_cpulist=$(expand_sequence ${list} " ")
cpulist=""
for e in $platform_cpulist
do
# claim hyperthread siblings if SMT enabled
SIBLINGS_CPULIST=$(cat /sys/devices/system/cpu/cpu${e}/topology/thread_siblings_list 2>/dev/null)
siblings_cpulist=$(expand_sequence ${SIBLINGS_CPULIST} " ")
for s in $siblings_cpulist
do
in_list ${s} ${cpulist}
if [ $? -eq 1 ]
then
cpulist=$(append_list ${s} ${cpulist})
fi
done
done
echo "$cpulist"
return 0
}
################################################################################
# Return list of CPUs based on cpu topology. Select the socket, starting core
# within the socket, select number of cores, and SMT siblings.
################################################################################
function topology_to_cpulist() {
local SOCKET=$1
local CORE_START=$2
local NUM_CORES=$3
local CPULIST=$(cat /proc/cpuinfo 2>/dev/null | perl -sne \
'BEGIN { %T = {}; %H = {}; $L = $P = $C = $S = 0; }
{
if (/processor\s+:\s+(\d+)/) { $L = $1; }
if (/physical id\s+:\s+(\d+)/) { $P = $1; }
if (/core id\s+:\s+(\d+)/) {
$C = $1;
$T{$P}{$C}++;
$S = $T{$P}{$C};
$H{$P}{$C}{$S} = $L;
}
}
END {
@cores = sort { $a <=> $b } keys $T{$socket};
@sel_cores = splice @cores, $core_start, $num_cores;
@lcpus = ();
for $C (@sel_cores) {
for $S (sort {$a <=> $b } keys %{ $H{$socket}{$C} }) {
push @lcpus, $H{$socket}{$C}{$S};
}
}
printf "%s\n", join(",", @lcpus);
}' -- -socket=${SOCKET} -core_start=${CORE_START} -num_cores=${NUM_CORES})
echo ${CPULIST}
}

244
compute-huge/compute-huge/cpumap_functions_unit_test.sh Normal file
View File

@@ -0,0 +1,244 @@
#!/bin/bash
#
# Copyright (c) 2015-2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
source /etc/init.d/cpumap_functions.sh
export NR_CPUS_LIST=("4" "8" "16" "32" "64" "128")
if [ ! -z ${1} ]; then
NR_CPUS_LIST=(${1//,/ })
fi
function test_cpumap_to_cpulist()
{
local NR_CPUS=$1
declare -A CPULISTS
if [ ${NR_CPUS} -ge 4 ]; then
CPULISTS["0"]=""
CPULISTS["1"]="0"
CPULISTS["2"]="1"
CPULISTS["3"]="0-1"
CPULISTS["5"]="0,2"
CPULISTS["7"]="0-2"
CPULISTS["F"]="0-3"
CPULISTS["9"]="0,3"
fi
if [ ${NR_CPUS} -ge 8 ]; then
CPULISTS["00"]=""
CPULISTS["11"]="0,4"
CPULISTS["FF"]="0-7"
CPULISTS["81"]="0,7"
fi
if [ ${NR_CPUS} -ge 16 ]; then
CPULISTS["0000"]=""
CPULISTS["1111"]="0,4,8,12"
CPULISTS["FFF"]="0-11"
CPULISTS["F0F"]="0-3,8-11"
CPULISTS["F0F0"]="4-7,12-15"
CPULISTS["FFFF"]="0-15"
CPULISTS["FFFE"]="1-15"
CPULISTS["8001"]="0,15"
fi
if [ ${NR_CPUS} -ge 32 ]; then
CPULISTS["00000000"]=""
CPULISTS["11111111"]="0,4,8,12,16,20,24,28"
CPULISTS["0F0F0F0F"]="0-3,8-11,16-19,24-27"
CPULISTS["F0F0F0F0"]="4-7,12-15,20-23,28-31"
CPULISTS["FFFFFFFF"]="0-31"
CPULISTS["FFFFFFFE"]="1-31"
CPULISTS["80000001"]="0,31"
fi
if [ ${NR_CPUS} -ge 64 ]; then
CPULISTS["0000000000000000"]=""
CPULISTS["1111111111111111"]="0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60"
CPULISTS["0F0F0F0F0F0F0F0F"]="0-3,8-11,16-19,24-27,32-35,40-43,48-51,56-59"
CPULISTS["F0F0F0F0F0F0F0F0"]="4-7,12-15,20-23,28-31,36-39,44-47,52-55,60-63"
CPULISTS["FFFFFFFFFFFFFFFF"]="0-63"
CPULISTS["FFFFFFFFFFFFFFFE"]="1-63"
CPULISTS["8000000000000001"]="0,63"
fi
if [ ${NR_CPUS} -ge 128 ]; then
CPULISTS["00000000000000000000000000000000"]=""
CPULISTS["11111111111111111111111111111111"]="0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124"
CPULISTS["0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F"]="0-3,8-11,16-19,24-27,32-35,40-43,48-51,56-59,64-67,72-75,80-83,88-91,96-99,104-107,112-115,120-123"
CPULISTS["F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0"]="4-7,12-15,20-23,28-31,36-39,44-47,52-55,60-63,68-71,76-79,84-87,92-95,100-103,108-111,116-119,124-127"
CPULISTS["FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"]="0-127"
CPULISTS["FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE"]="1-127"
CPULISTS["80000000000000000000000000000001"]="0,127"
fi
for CPUMAP in ${!CPULISTS[@]}; do
EXPECTED=${CPULISTS[${CPUMAP}]}
CPULIST=$(cpumap_to_cpulist ${CPUMAP} ${NR_CPUS})
if [ "${CPULIST}" != "${EXPECTED}" ]; then
printf "\n"
echo "error: (cpumap_to_list ${CPUMAP} ${NR_CPUS}) returned \"${CPULIST}\" instead of \"${EXPECTED}\""
fi
printf "."
done
printf "\n"
}
function test_cpulist_to_cpumap()
{
local NR_CPUS=$1
declare -A CPUMAPS
if [ ${NR_CPUS} -ge 4 ]; then
CPUMAPS[" "]="0"
CPUMAPS["0"]="1"
CPUMAPS["1"]="2"
CPUMAPS["0-1"]="3"
CPUMAPS["0,2"]="5"
CPUMAPS["0-2"]="7"
CPUMAPS["0-3"]="F"
CPUMAPS["0,3"]="9"
fi
if [ ${NR_CPUS} -ge 8 ]; then
CPUMAPS["0,4"]="11"
CPUMAPS["0-7"]="FF"
CPUMAPS["0,7"]="81"
fi
if [ ${NR_CPUS} -ge 16 ]; then
CPUMAPS["0,4,8,12"]="1111"
CPUMAPS["0-11"]="FFF"
CPUMAPS["0-3,8-11"]="F0F"
CPUMAPS["4-7,12-15"]="F0F0"
CPUMAPS["0-15"]="FFFF"
CPUMAPS["1-15"]="FFFE"
CPUMAPS["0,15"]="8001"
fi
if [ ${NR_CPUS} -ge 32 ]; then
CPUMAPS["0,4,8,12,16,20,24,28"]="11111111"
CPUMAPS["0-3,8-11,16-19,24-27"]="F0F0F0F"
CPUMAPS["4-7,12-15,20-23,28-31"]="F0F0F0F0"
CPUMAPS["0-31"]="FFFFFFFF"
CPUMAPS["1-31"]="FFFFFFFE"
CPUMAPS["0,31"]="80000001"
fi
if [ ${NR_CPUS} -ge 64 ]; then
CPUMAPS["0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60"]="1111111111111111"
CPUMAPS["0-3,8-11,16-19,24-27,32-35,40-43,48-51,56-59"]="F0F0F0F0F0F0F0F"
CPUMAPS["4-7,12-15,20-23,28-31,36-39,44-47,52-55,60-63"]="F0F0F0F0F0F0F0F0"
CPUMAPS["0-63"]="FFFFFFFFFFFFFFFF"
CPUMAPS["1-63"]="FFFFFFFFFFFFFFFE"
CPUMAPS["0,63"]="8000000000000001"
fi
if [ ${NR_CPUS} -ge 128 ]; then
CPUMAPS["0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124"]="11111111111111111111111111111111"
CPUMAPS["0-3,8-11,16-19,24-27,32-35,40-43,48-51,56-59,64-67,72-75,80-83,88-91,96-99,104-107,112-115,120-123"]="F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F"
CPUMAPS["4-7,12-15,20-23,28-31,36-39,44-47,52-55,60-63,68-71,76-79,84-87,92-95,100-103,108-111,116-119,124-127"]="F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0"
CPUMAPS["0-127"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
CPUMAPS["1-127"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE"
CPUMAPS["0,127"]="80000000000000000000000000000001"
fi
for CPULIST in ${!CPUMAPS[@]}; do
EXPECTED=${CPUMAPS[${CPULIST}]}
CPUMAP=$(cpulist_to_cpumap ${CPULIST} ${NR_CPUS})
if [ "${CPUMAP}" != "${EXPECTED}" ]; then
printf "\n"
echo "error: (cpulist_to_cpumap ${CPULIST} ${NR_CPUS}) returned \"${CPUMAP}\" instead of \"${EXPECTED}\""
fi
printf "."
done
printf "\n"
}
function test_invert_cpumap()
{
local NR_CPUS=$1
declare -A INVERSES
if [ $((${NR_CPUS} % 4)) -ne 0 ]; then
echo "test_invert_cpumap skipping NR_CPUS=${NR_CPUS}; not a multiple of 4"
return 0
fi
if [ ${NR_CPUS} -ge 4 ]; then
INVERSES["0"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
INVERSES["1"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE"
INVERSES["2"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFD"
INVERSES["3"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFC"
INVERSES["5"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFA"
INVERSES["7"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF8"
INVERSES["F"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0"
INVERSES["9"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF6"
fi
if [ ${NR_CPUS} -ge 8 ]; then
INVERSES["11"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEE"
INVERSES["FF"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF00"
INVERSES["F0"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0F"
INVERSES["81"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF7E"
fi
if [ ${NR_CPUS} -ge 16 ]; then
INVERSES["1111"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFEEEE"
INVERSES["FFF"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFF000"
INVERSES["F0F"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFF0F0"
INVERSES["F0F0"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFF0F0F"
INVERSES["0F0F"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFFF0F0"
INVERSES["FFFF"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFF0000"
INVERSES["FFFE"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFF0001"
INVERSES["8001"]="FFFFFFFFFFFFFFFFFFFFFFFFFFFF7FFE"
fi
if [ ${NR_CPUS} -ge 32 ]; then
INVERSES["11111111"]="FFFFFFFFFFFFFFFFFFFFFFFFEEEEEEEE"
INVERSES["0F0F0F0F"]="FFFFFFFFFFFFFFFFFFFFFFFFF0F0F0F0"
INVERSES["F0F0F0F0"]="FFFFFFFFFFFFFFFFFFFFFFFF0F0F0F0F"
INVERSES["FFFFFFFF"]="FFFFFFFFFFFFFFFFFFFFFFFF00000000"
INVERSES["FFFFFFFE"]="FFFFFFFFFFFFFFFFFFFFFFFF00000001"
INVERSES["80000001"]="FFFFFFFFFFFFFFFFFFFFFFFF7FFFFFFE"
fi
if [ ${NR_CPUS} -ge 64 ]; then
INVERSES["1111111111111111"]="FFFFFFFFFFFFFFFFEEEEEEEEEEEEEEEE"
INVERSES["0F0F0F0F0F0F0F0F"]="FFFFFFFFFFFFFFFFF0F0F0F0F0F0F0F0"
INVERSES["F0F0F0F0F0F0F0F0"]="FFFFFFFFFFFFFFFF0F0F0F0F0F0F0F0F"
INVERSES["FFFFFFFFFFFFFFFF"]="FFFFFFFFFFFFFFFF0000000000000000"
INVERSES["FFFFFFFFFFFFFFFE"]="FFFFFFFFFFFFFFFF0000000000000001"
INVERSES["8000000000000001"]="FFFFFFFFFFFFFFFF7FFFFFFFFFFFFFFE"
fi
if [ ${NR_CPUS} -ge 128 ]; then
INVERSES["11111111111111111111111111111111"]="EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE"
INVERSES["0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F"]="F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0"
INVERSES["F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0"]="0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F"
INVERSES["FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"]="00000000000000000000000000000000"
INVERSES["FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE"]="00000000000000000000000000000001"
INVERSES["80000000000000000000000000000001"]="7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFE"
fi
for CPUMAP in ${!INVERSES[@]}; do
EXPECTED=${INVERSES[${CPUMAP}]}
if [ ${NR_CPUS} -lt 128 ]; then
EXPECTED=$(echo ${EXPECTED} | cut --complement -c1-$((32-((${NR_CPUS}+3)/4))))
fi
EXPECTED=$(echo ${EXPECTED} | sed -e "s/^0*//")
if [ -z ${EXPECTED} ]; then
EXPECTED="0"
fi
INVERSE=$(invert_cpumap ${CPUMAP} ${NR_CPUS})
if [ "${INVERSE}" != "${EXPECTED}" ]; then
printf "\n"
echo "error: (invert_cpumap ${CPUMAP} ${NR_CPUS}) returned \"${INVERSE}\" instead of \"${EXPECTED}\""
fi
printf "."
done
printf "\n"
}
for NR_CPUS in ${NR_CPUS_LIST[@]}; do
echo "NR_CPUS=${NR_CPUS}"
test_cpumap_to_cpulist ${NR_CPUS}
test_cpulist_to_cpumap ${NR_CPUS}
test_invert_cpumap ${NR_CPUS}
echo ""
done
exit 0

49
compute-huge/compute-huge/log_functions.sh Normal file
View File

@@ -0,0 +1,49 @@
#!/bin/bash
################################################################################
# Copyright (c) 2013-2015 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
################################################################################
################################################################################
# Log if debug is enabled via LOG_DEBUG
#
################################################################################
function log_debug
{
if [ ! -z "${LOG_DEBUG}" ]; then
logger -p debug -t "$0[${PPID}]" -s "$@" 2>&1
fi
}
################################################################################
# Log unconditionally to STDERR
#
################################################################################
function log_error
{
logger -p error -t "$0[${PPID}]" -s "$@"
}
################################################################################
# Log unconditionally to STDOUT
#
################################################################################
function log
{
logger -p info -t "$0[${PPID}]" -s "$@" 2>&1
}
################################################################################
# Utility function to print the status of a command result
#
################################################################################
function print_status()
{
if [ "$1" -eq "0" ]; then
echo "[ OK ]"
else
echo "[FAILED]"
fi
}

27
compute-huge/compute-huge/ps-sched.sh Executable file
View File

@@ -0,0 +1,27 @@
#!/bin/bash
################################################################################
# Copyright (c) 2013 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
################################################################################
#
# ps-sched.sh -- gives detailed task listing with scheduling attributes
# -- this is cpu and scheduling intensive version (shell/taskset based)
# (note: does not print fields 'group' or 'timeslice')
printf "%6s %6s %6s %1c %2s %4s %6s %4s %-24s %2s %-16s %s\n" "PID" "TID" "PPID" "S" "PO" "NICE" "RTPRIO" "PR" "AFFINITY" "P" "COMM" "COMMAND"
ps -eL -o pid=,lwp=,ppid=,state=,class=,nice=,rtprio=,priority=,psr=,comm=,command= | \
while read pid tid ppid state policy nice rtprio priority psr comm command
do
bitmask=$(taskset -p $tid 2>/dev/null)
aff=${bitmask##*: }
if [ -z "${aff}" ]; then
aff="0x0"
else
aff="0x${aff}"
fi
printf "%6d %6d %6d %1c %2s %4s %6s %4d %-24s %2d %-16s %s\n" $pid $tid $ppid $state $policy $nice $rtprio $priority $aff $psr $comm "$command"
done
exit 0

90
compute-huge/compute-huge/set-cpu-wakeup-latency.sh Normal file
View File

@@ -0,0 +1,90 @@
#!/bin/bash
#
# Copyright (c) 2017 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
# Purpose: set PM QoS resume latency constraints for CPUs.
# Usage: /usr/bin/set-cpu-wakeup-latency.sh policy cpulist
# policy may be either "low" or "high" to set appropriate latency.
# "low" means HALT (C1) is the deepest C-state we allow the CPU to enter.
# "high" means we allow the CPU to sleep as deeply as possible.
# cpulist is for specifying a numerical list of processors.
# It may contain multiple items, separated by comma, and ranges.
# For example, 0,5,7,9-11.
# Define minimal path
PATH=/bin:/usr/bin:/usr/local/bin
LOG_FUNCTIONS=${LOG_FUNCTIONS:-"/etc/init.d/log_functions.sh"}
CPUMAP_FUNCTIONS=${CPUMAP_FUNCTIONS:-"/etc/init.d/cpumap_functions.sh"}
[[ -e ${LOG_FUNCTIONS} ]] && source ${LOG_FUNCTIONS}
[[ -e ${CPUMAP_FUNCTIONS} ]] && source ${CPUMAP_FUNCTIONS}
if [ $UID -ne 0 ]; then
log_error "$0 requires root or sudo privileges"
exit 1
fi
if [ "$#" -ne 2 ]; then
log_error "$0 requires policy and cpulist parameters"
exit 1
fi
POLICY=$1
CPU_LIST=$2
NUMBER_OF_CPUS=$(getconf _NPROCESSORS_CONF 2>/dev/null)
STATUS=1
for CPU_NUM in $(expand_sequence "$CPU_LIST" " ")
do
# Check that we are not setting PM QoS policy for non-existing CPU
if [ "$CPU_NUM" -lt "0" ] || [ "$CPU_NUM" -ge "$NUMBER_OF_CPUS" ]; then
log_error "CPU number ${CPU_NUM} is invalid, available CPUs are 0-${NUMBER_OF_CPUS-1}"
exit 1
fi
# Obtain CPU wakeup latencies for all C-states available starting from operating state to deepest sleep
declare -a LIMITS=()
LIMITS+=($(cat /sys/devices/system/cpu/cpu${CPU_NUM}/cpuidle/state*/latency 2>/dev/null | xargs | sort))
if [ ${#LIMITS[@]} -eq 0 ]; then
log_debug "Failed to get PM QoS latency limits for CPU ${CPU_NUM}"
fi
# Select appropriate CPU wakeup latency based on "low" or "high" policy
case "${POLICY}" in
"low")
# Get first sleep state for "low" policy
if [ ${#LIMITS[@]} -eq 0 ]; then
LATENCY=1
else
LATENCY=${LIMITS[1]}
fi
;;
"high")
# Get deepest sleep state for "high" policy
if [ ${#LIMITS[@]} -eq 0 ]; then
LATENCY=1000
else
LATENCY=${LIMITS[${#LIMITS[@]}-1]}
fi
;;
*)
log_error "Policy is invalid, can be either low or high"
exit 1
esac
# Set the latency for paricular CPU
echo ${LATENCY} > /sys/devices/system/cpu/cpu${CPU_NUM}/power/pm_qos_resume_latency_us 2>/dev/null
RET_VAL=$?
if [ ${RET_VAL} -ne 0 ]; then
log_error "Failed to set PM QoS latency for CPU ${CPU_NUM}, rc=${RET_VAL}"
continue
else
log_debug "Succesfully set PM QoS latency for CPU ${CPU_NUM}, rc=${RET_VAL}"
STATUS=0
fi
done
exit ${STATUS}

330
compute-huge/compute-huge/task_affinity_functions.sh Executable file
View File

@@ -0,0 +1,330 @@
#!/bin/bash
################################################################################
# Copyright (c) 2017 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
################################################################################
# Define minimal path
PATH=/bin:/usr/bin:/usr/local/bin
. /etc/platform/platform.conf
LOG_FUNCTIONS=${LOG_FUNCTIONS:-"/etc/init.d/log_functions.sh"}
CPUMAP_FUNCTIONS=${CPUMAP_FUNCTIONS:-"/etc/init.d/cpumap_functions.sh"}
[[ -e ${LOG_FUNCTIONS} ]] && source ${LOG_FUNCTIONS}
[[ -e ${CPUMAP_FUNCTIONS} ]] && source ${CPUMAP_FUNCTIONS}
# Enable debug logs and tag them
LOG_DEBUG=1
TAG="TASKAFFINITY:"
TASK_AFFINING_INCOMPLETE="/etc/platform/.task_affining_incomplete"
N_CPUS=$(cat /proc/cpuinfo 2>/dev/null | \
awk '/^[pP]rocessor/ { n +=1 } END { print (n>0) ? n : 1}')
FULLSET_CPUS="0-"$((N_CPUS-1))
FULLSET_MASK=$(cpulist_to_cpumap ${FULLSET_CPUS} ${N_CPUS})
PLATFORM_CPUS=$(get_platform_cpu_list)
PLATFORM_CPULIST=$(get_platform_cpu_list| \
perl -pe 's/(\d+)-(\d+)/join(",",$1..$2)/eg'| \
sed 's/,/ /g')
VSWITCH_CPULIST=$(get_vswitch_cpu_list| \
perl -pe 's/(\d+)-(\d+)/join(",",$1..$2)/eg'| \
sed 's/,/ /g')
IDLE_MARK=95.0
KERNEL=`uname -a`
################################################################################
# Check if a given core is one of the platform cores
################################################################################
function is_platform_core()
{
local core=$1
for CPU in ${PLATFORM_CPULIST}; do
if [ $core -eq $CPU ]; then
return 1
fi
done
return 0
}
################################################################################
# Check if a given core is one of the vswitch cores
################################################################################
function is_vswitch_core()
{
local core=$1
for CPU in ${VSWITCH_CPULIST}; do
if [ $core -eq $CPU ]; then
return 1
fi
done
return 0
}
################################################################################
# An audit and corrective action following a swact
################################################################################
function audit_and_reaffine()
{
local mask=$1
local cmd_str=""
local tasklist
cmd_str="ps-sched.sh|awk '(\$9==\"$mask\") {print \$2}'"
tasklist=($(eval $cmd_str))
# log_debug "cmd str = $cmd_str"
log_debug "${TAG} There are ${#tasklist[@]} tasks to reaffine."
for task in ${tasklist[@]}; do
taskset -acp ${PLATFORM_CPUS} $task &> /dev/null
rc=$?
[[ $rc -ne 0 ]] && log_error "Failed to set CPU affinity for pid $pid, rc=$rc"
done
tasklist=($(eval $cmd_str))
[[ ${#tasklist[@]} -eq 0 ]] && return 0 || return 1
}
################################################################################
# The following function is used to verify that any sleeping management tasks
# that are on non-platform cores can be migrated to platform cores as soon as
# they are scheduled. It can be invoked either manually or from goenableCompute
# script as a scheduled job (with a few minute delay) if desired.
# The induced tasks migration should be done after all VMs have been restored
# following a host reboot in AIO, hence the delay.
################################################################################
function move_inactive_threads_to_platform_cores()
{
local tasklist
local cmd_str=""
# Compile a list of non-kernel & non-vswitch/VM related threads that are not
# on platform cores.
# e.g. if the platform cpulist value is "0 8", the resulting command to be
# evaluated should look like this:
# ps-sched.sh|grep -v vswitch|awk '($10!=0 && $10!=8 && $3!=2) {if(NR>1)print $2}'
cmd_str="ps-sched.sh|grep -v vswitch|awk '("
for cpu_num in ${PLATFORM_CPULIST}; do
cmd_str=$cmd_str"\$10!="${cpu_num}" && "
done
cmd_str=$cmd_str"\$3!=2) {if(NR>1)print \$2}'"
echo "selection string = $cmd_str"
tasklist=($(eval $cmd_str))
log_debug "${TAG} There are ${#tasklist[@]} number of tasks to be moved."
# These sleep tasks are stuck on the wrong core(s). They need to be woken up
# so they can be migrated to the right ones. Attaching and detaching strace
# momentarily to the task does the trick.
for task in ${tasklist[@]}; do
strace -p $task 2>/dev/null &
pid=$!
sleep 0.1
kill -SIGINT $pid
done
tasklist=($(eval $cmd_str))
[[ ${#tasklist[@]} -eq 0 ]] && return 0 || return 1
}
################################################################################
# The following function is called by affine-platform.sh to affine tasks to
# all available cores during initial startup and subsequent host reboots.
################################################################################
function affine_tasks_to_all_cores()
{
local pidlist
local rc=0
if [[ "${KERNEL}" == *" RT "* ]]; then
return 0
fi
log_debug "${TAG} Affining all tasks to CPU (${FULLSET_CPUS})"
pidlist=$(ps --ppid 2 -p 2 --deselect -o pid= | awk '{ print $1; }')
for pid in ${pidlist[@]}; do
ppid=$(ps -o ppid= -p $pid |tr -d '[:space:]')
if [ -z $ppid ] || [ $ppid -eq 2 ]; then
continue
fi
log_debug "Affining pid $pid, parent pid = $ppid"
taskset --all-tasks --pid --cpu-list ${FULLSET_CPUS} $pid &> /dev/null
rc=$?
[[ $rc -ne 0 ]] && log_error "Failed to set CPU affinity for pid $pid, rc=$rc"
done
# Write the cpu list to a temp file which will be read and removed when
# the tasks are reaffined back to platform cores later on.
echo ${FULLSET_CPUS} > ${TASK_AFFINING_INCOMPLETE}
return $rc
}
################################################################################
# The following function can be called by any platform service that needs to
# temporarily make use of idle VM cores to run a short-duration, service
# critical and cpu intensive operation in AIO. For instance, sm can levearage
# the idle cores to speed up swact activity.
#
# At the end of the operation, regarless of the result, the service must be
# calling function affine_tasks_to_platform_cores to re-affine platform tasks
# back to their assigned core(s).
#
# Kernel, vswitch and VM related tasks are untouched.
################################################################################
function affine_tasks_to_idle_cores()
{
local cpulist
local cpuocc_list
local vswitch_pid
local pidlist
local idle_cpulist
local platform_cpus
local rc=0
local cpu=0
if [ -f ${TASK_AFFINING_INCOMPLETE} ]; then
read cpulist < ${TASK_AFFINING_INCOMPLETE}
log_debug "${TAG} Tasks have already been affined to CPU ($cpulist)."
return 0
fi
if [[ "${KERNEL}" == *" RT "* ]]; then
return 0
fi
# Compile a list of cpus with idle percentage greater than 95% in the last
# 5 seconds.
cpuocc_list=($(sar -P ALL 1 5|grep Average|awk '{if(NR>2)print $8}'))
for idle_value in ${cpuocc_list[@]}; do
is_vswitch_core $cpu
if [ $? -eq 1 ]; then
((cpu++))
continue
fi
is_platform_core $cpu
if [ $? -eq 1 ]; then
# Platform core is added to the idle list by default
idle_cpulist=$idle_cpulist$cpu","
else
# Non platform core is added to the idle list if it is more than 95% idle
[[ $(echo "$idle_value > ${IDLE_MARK}"|bc) -eq 1 ]] && idle_cpulist=$idle_cpulist$cpu","
fi
((cpu++))
done
idle_cpulist=$(echo $idle_cpulist|sed 's/.$//')
platform_affinity_mask=$(cpulist_to_cpumap ${PLATFORM_CPUS} ${N_CPUS} \
|awk '{print tolower($0)}')
log_debug "${TAG} Affining all tasks to idle CPU ($idle_cpulist)"
vswitch_pid=$(pgrep vswitch)
pidlist=$(ps --ppid 2 -p 2 --deselect -o pid= | awk '{ print $1; }')
for pid in ${pidlist[@]}; do
ppid=$(ps -o ppid= -p $pid |tr -d '[:space:]')
if [ -z $ppid ] || [ $ppid -eq 2 ] || [ "$pid" = "$vswitch_pid" ]; then
continue
fi
pid_affinity_mask=$(taskset -p $pid | awk '{print $6}')
if [ "${pid_affinity_mask}" == "${platform_affinity_mask}" ]; then
# log_debug "Affining pid $pid to idle cores..."
taskset --all-tasks --pid --cpu-list $idle_cpulist $pid &> /dev/null
rc=$?
[[ $rc -ne 0 ]] && log_error "Failed to set CPU affinity for pid $pid, rc=$rc"
fi
done
# Save the cpu list to the temp file which will be read and removed when
# tasks are reaffined to the platform cores later on.
echo $idle_cpulist > ${TASK_AFFINING_INCOMPLETE}
return $rc
}
################################################################################
# The following function is called by either:
# a) nova-compute wrapper script during AIO system initial bringup or reboot
# or
# b) sm at the end of swact sequence
# to re-affine management tasks back to the platform cores.
################################################################################
function affine_tasks_to_platform_cores()
{
local cpulist
local pidlist
local rc=0
local count=0
if [ ! -f ${TASK_AFFINING_INCOMPLETE} ]; then
dbg_str="${TAG} Either tasks have never been affined to all/idle cores or"
dbg_str=$dbg_str" they have already been reaffined to platform cores."
log_debug "$dbg_str"
return 0
fi
read cpulist < ${TASK_AFFINING_INCOMPLETE}
affinity_mask=$(cpulist_to_cpumap $cpulist ${N_CPUS}|awk '{print tolower($0)}')
log_debug "${TAG} Reaffining tasks to platform cores (${PLATFORM_CPUS})..."
pidlist=$(ps --ppid 2 -p 2 --deselect -o pid= | awk '{ print $1; }')
for pid in ${pidlist[@]}; do
# log_debug "Processing pid $pid..."
pid_affinity_mask=$(taskset -p $pid | awk '{print $6}')
# Only management tasks need to be reaffined. Kernel, vswitch and VM related
# tasks were not affined previously so they should have different affinity
# mask(s).
if [ "${pid_affinity_mask}" == "${affinity_mask}" ]; then
((count++))
# log_debug "Affining pid $pid to platform cores..."
taskset --all-tasks --pid --cpu-list ${PLATFORM_CPUS} $pid &> /dev/null
rc=$?
[[ $rc -ne 0 ]] && log_error "Failed to set CPU affinity for pid $pid, rc=$rc"
fi
done
# A workaround for lack of "end of swact" state
fullmask=$(echo ${FULLSET_MASK} | awk '{print tolower($0)}')
if [ "${affinity_mask}" != "${fullmask}" ]; then
log_debug "${TAG} Schedule an audit and cleanup"
(sleep 60; audit_and_reaffine "0x"$affinity_mask) &
fi
rm -rf ${TASK_AFFINING_INCOMPLETE}
log_debug "${TAG} $count tasks were reaffined to platform cores."
return $rc
}
################################################################################
# The following function can be leveraged by cron tasks
################################################################################
function get_most_idle_core()
{
local cpuocc_list
local cpu=0
local most_idle_value=${IDLE_MARK}
local most_idle_cpu=0
if [[ "${KERNEL}" == *" RT "* ]]; then
echo $cpu
return
fi
cpuocc_list=($(sar -P ALL 1 5|grep Average|awk '{if(NR>2)print $8}'))
for idle_value in ${cpuocc_list[@]}; do
is_vswitch_core $cpu
if [ $? -eq 1 ]; then
((cpu++))
continue
fi
if [ $(echo "$idle_value > $most_idle_value"|bc) -eq 1 ]; then
most_idle_value=$idle_value
most_idle_cpu=$cpu
fi
((cpu++))
done
echo $most_idle_cpu
}

241
compute-huge/compute-huge/topology.py Executable file
View File

@@ -0,0 +1,241 @@
#!/usr/bin/env python
################################################################################
# Copyright (c) 2013 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
################################################################################
#
# topology.py -- gives a summary of logical cpu enumeration,
# sockets, cores per package, threads per core,
# total memory, and numa nodes
import os
import sys
import re
class Topology(object):
""" Build up topology information.
(i.e. logical cpu topology, NUMA nodes, memory)
"""
def __init__(self):
self.num_cpus = 0
self.num_nodes = 0
self.num_sockets = 0
self.num_cores_per_pkg = 0
self.num_threads_per_core = 0
self.topology = {}
self.topology_idx = {}
self.total_memory_MiB = 0
self.total_memory_nodes_MiB = []
self._get_cpu_topology()
self._get_total_memory_MiB()
self._get_total_memory_nodes_MiB()
def _get_cpu_topology(self):
'''Enumerate logical cpu topology based on parsing /proc/cpuinfo
as function of socket_id, core_id, and thread_id. This updates
topology and reverse index topology_idx mapping.
:param self
:updates self.num_cpus - number of logical cpus
:updates self.num_nodes - number of sockets; maps to number of numa nodes
:updates self.topology[socket_id][core_id][thread_id] = cpu
:updates self.topology_idx[cpu] = {'s': socket_id, 'c': core_id, 't': thread_id}
:returns None
'''
self.num_cpus = 0
self.num_nodes = 0
self.num_sockets = 0
self.num_cores = 0
self.num_threads = 0
self.topology = {}
self.topology_idx = {}
Thread_cnt = {}
cpu = socket_id = core_id = thread_id = -1
re_processor = re.compile(r'^[Pp]rocessor\s+:\s+(\d+)')
re_socket = re.compile(r'^physical id\s+:\s+(\d+)')
re_core = re.compile(r'^core id\s+:\s+(\d+)')
with open('/proc/cpuinfo', 'r') as infile:
for line in infile:
match = re_processor.search(line)
if match:
cpu = int(match.group(1))
socket_id = -1; core_id = -1; thread_id = -1
self.num_cpus += 1
continue
match = re_socket.search(line)
if match:
socket_id = int(match.group(1))
continue
match = re_core.search(line)
if match:
core_id = int(match.group(1))
if not Thread_cnt.has_key(socket_id):
Thread_cnt[socket_id] = {}
if not Thread_cnt[socket_id].has_key(core_id):
Thread_cnt[socket_id][core_id] = 0
else:
Thread_cnt[socket_id][core_id] += 1
thread_id = Thread_cnt[socket_id][core_id]
if not self.topology.has_key(socket_id):
self.topology[socket_id] = {}
if not self.topology[socket_id].has_key(core_id):
self.topology[socket_id][core_id] = {}
self.topology[socket_id][core_id][thread_id] = cpu
self.topology_idx[cpu] = {'s': socket_id, 'c': core_id, 't': thread_id}
continue
self.num_nodes = len(self.topology.keys())
# In the case topology not detected, hard-code structures
if self.num_nodes == 0:
n_sockets, n_cores, n_threads = (1, self.num_cpus, 1)
self.topology = {}
for socket_id in range(n_sockets):
self.topology[socket_id] = {}
for core_id in range(n_cores):
self.topology[socket_id][core_id] = {}
for thread_id in range(n_threads):
self.topology[socket_id][core_id][thread_id] = 0
# Define Thread-Socket-Core order for logical cpu enumeration
self.topology_idx = {}
cpu = 0
for thread_id in range(n_threads):
for socket_id in range(n_sockets):
for core_id in range(n_cores):
self.topology[socket_id][core_id][thread_id] = cpu
self.topology_idx[cpu] = {'s': socket_id, 'c': core_id, 't': thread_id}
cpu += 1
self.num_nodes = len(self.topology.keys())
self.num_sockets = len(self.topology.keys())
self.num_cores_per_pkg = len(self.topology[0].keys())
self.num_threads_per_core = len(self.topology[0][0].keys())
return None
def _get_total_memory_MiB(self):
"""Get the total memory for VMs (MiB).
:updates: total memory for VMs (MiB)
"""
self.total_memory_MiB = 0
# Total memory
try:
m = open('/proc/meminfo').read().split()
idx_Total = m.index('MemTotal:') + 1
self.total_memory_MiB = int(m[idx_Total]) / 1024
except IOError:
# silently ignore IO errors (eg. file missing)
pass
return None
def _get_total_memory_nodes_MiB(self):
"""Get the total memory per numa node for VMs (MiB).
:updates: total memory per numa node for VMs (MiB)
"""
self.total_memory_nodes_MiB = []
# Memory of each numa node (MiB)
for node in range(self.num_nodes):
Total_MiB = 0
meminfo = "/sys/devices/system/node/node%d/meminfo" % node
try:
m = open(meminfo).read().split()
idx_Total = m.index('MemTotal:') + 1
Total_MiB = int(m[idx_Total]) / 1024
except IOError:
# silently ignore IO errors (eg. file missing)
pass
self.total_memory_nodes_MiB.append(Total_MiB)
return None
def _print_cpu_topology(self):
'''Print logical cpu topology enumeration as function of:
socket_id, core_id, and thread_id.
:param self
:returns None
'''
cpu_list = self.topology_idx.keys()
cpu_list.sort()
total_memory_GiB = self.total_memory_MiB/1024.0
print 'TOPOLOGY:'
print '%16s : %5d' % ('logical cpus', self.num_cpus)
print '%16s : %5d' % ('sockets', self.num_sockets)
print '%16s : %5d' % ('cores_per_pkg', self.num_cores_per_pkg)
print '%16s : %5d' % ('threads_per_core', self.num_threads_per_core)
print '%16s : %5d' % ('numa_nodes', self.num_nodes)
print '%16s : %5.2f %s' % ('total_memory', total_memory_GiB, 'GiB')
print '%16s :' % ('memory_per_node'),
for node in range(self.num_nodes):
node_memory_GiB = self.total_memory_nodes_MiB[node]/1024.0
print '%5.2f' % (node_memory_GiB),
print '%s' % ('GiB')
print
print 'LOGICAL CPU TOPOLOGY:'
print "%9s :" % 'cpu_id',
for cpu in cpu_list:
print "%3d" % cpu,
print
print "%9s :" % 'socket_id',
for cpu in cpu_list:
socket_id = self.topology_idx[cpu]['s']
print "%3d" % socket_id,
print
print "%9s :" % 'core_id',
for cpu in cpu_list:
core_id = self.topology_idx[cpu]['c']
print "%3d" % core_id,
print
print "%9s :" % 'thread_id',
for cpu in cpu_list:
thread_id = self.topology_idx[cpu]['t']
print "%3d" % thread_id,
print
print
print 'CORE TOPOLOGY:'
print "%6s %9s %7s %9s %s" % ('cpu_id', 'socket_id', 'core_id', 'thread_id', 'affinity')
for cpu in cpu_list:
affinity = 1<<cpu
socket_id = self.topology_idx[cpu]['s']
core_id = self.topology_idx[cpu]['c']
thread_id = self.topology_idx[cpu]['t']
print "%6d %9d %7d %9d 0x%x" \
% (cpu, socket_id, core_id, thread_id, affinity)
return None
#-------------------------------------------------------------------------------
''' Main Program
'''
# Get logical cpu topology
topology = Topology()
topology._print_cpu_topology()
sys.exit(0)

6
computeconfig/.gitignore vendored Normal file
View File

@@ -0,0 +1,6 @@
!.distro
.distro/centos7/rpmbuild/RPMS
.distro/centos7/rpmbuild/SRPMS
.distro/centos7/rpmbuild/BUILD
.distro/centos7/rpmbuild/BUILDROOT
.distro/centos7/rpmbuild/SOURCES/computeconfig*tar.gz

13
computeconfig/PKG-INFO Normal file
View File

@@ -0,0 +1,13 @@
Metadata-Version: 1.1
Name: computeconfig
Version: 1.0
Summary: Initial compute node configuration
Home-page:
Author: Windriver
Author-email: info@windriver.com
License: Apache-2.0
Description: Initial compute node configuration
Platform: UNKNOWN

2
computeconfig/centos/build_srpm.data Normal file
View File

@@ -0,0 +1,2 @@
SRC_DIR="computeconfig"
TIS_PATCH_VER=11

99
computeconfig/centos/computeconfig.spec Normal file
View File

@@ -0,0 +1,99 @@
Summary: computeconfig
Name: computeconfig
Version: 1.0
Release: %{tis_patch_ver}%{?_tis_dist}
License: Apache-2.0
Group: base
Packager: Wind River <info@windriver.com>
URL: unknown
Source0: %{name}-%{version}.tar.gz
%define debug_package %{nil}
Requires: systemd
%description
Initial compute node configuration
%package -n computeconfig-standalone
Summary: computeconfig
Group: base
%description -n computeconfig-standalone
Initial compute node configuration
%package -n computeconfig-subfunction
Summary: computeconfig
Group: base
%description -n computeconfig-subfunction
Initial compute node configuration
%define local_etc_initd /etc/init.d/
%define local_goenabledd /etc/goenabled.d/
%define local_etc_systemd /etc/systemd/system/
%prep
%setup
%build
%install
install -d -m 755 %{buildroot}%{local_etc_initd}
install -p -D -m 700 compute_config %{buildroot}%{local_etc_initd}/compute_config
install -p -D -m 700 compute_services %{buildroot}%{local_etc_initd}/compute_services
install -d -m 755 %{buildroot}%{local_goenabledd}
install -p -D -m 755 config_goenabled_check.sh %{buildroot}%{local_goenabledd}/config_goenabled_check.sh
install -d -m 755 %{buildroot}%{local_etc_systemd}
install -d -m 755 %{buildroot}%{local_etc_systemd}/config
install -p -D -m 664 computeconfig.service %{buildroot}%{local_etc_systemd}/config/computeconfig-standalone.service
install -p -D -m 664 computeconfig-combined.service %{buildroot}%{local_etc_systemd}/config/computeconfig-combined.service
#install -p -D -m 664 config.service %{buildroot}%{local_etc_systemd}/config.service
%post -n computeconfig-standalone
if [ ! -e $D%{local_etc_systemd}/computeconfig.service ]; then
cp $D%{local_etc_systemd}/config/computeconfig-standalone.service $D%{local_etc_systemd}/computeconfig.service
else
cmp -s $D%{local_etc_systemd}/config/computeconfig-standalone.service $D%{local_etc_systemd}/computeconfig.service
if [ $? -ne 0 ]; then
rm -f $D%{local_etc_systemd}/computeconfig.service
cp $D%{local_etc_systemd}/config/computeconfig-standalone.service $D%{local_etc_systemd}/computeconfig.service
fi
fi
systemctl enable computeconfig.service
%post -n computeconfig-subfunction
if [ ! -e $D%{local_etc_systemd}/computeconfig.service ]; then
cp $D%{local_etc_systemd}/config/computeconfig-combined.service $D%{local_etc_systemd}/computeconfig.service
else
cmp -s $D%{local_etc_systemd}/config/computeconfig-combined.service $D%{local_etc_systemd}/computeconfig.service
if [ $? -ne 0 ]; then
rm -f $D%{local_etc_systemd}/computeconfig.service
cp $D%{local_etc_systemd}/config/computeconfig-combined.service $D%{local_etc_systemd}/computeconfig.service
fi
fi
systemctl enable computeconfig.service
%clean
# rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
%doc LICENSE
%{local_etc_initd}/*
%files -n computeconfig-standalone
%defattr(-,root,root,-)
%dir %{local_etc_systemd}/config
%{local_etc_systemd}/config/computeconfig-standalone.service
#%{local_etc_systemd}/config.service
%{local_goenabledd}/*
%files -n computeconfig-subfunction
%defattr(-,root,root,-)
%dir %{local_etc_systemd}/config
%{local_etc_systemd}/config/computeconfig-combined.service

202
computeconfig/computeconfig/LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

383
computeconfig/computeconfig/compute_config Normal file
View File

@@ -0,0 +1,383 @@
#!/bin/bash
#
# Copyright (c) 2013-2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
#
# chkconfig: 2345 80 80
#
### BEGIN INIT INFO
# Provides: compute_config
# Short-Description: Compute node config agent
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
. /usr/bin/tsconfig
. /etc/platform/platform.conf
PLATFORM_DIR=/opt/platform
CONFIG_DIR=$CONFIG_PATH
VOLATILE_CONFIG_PASS="/var/run/.config_pass"
VOLATILE_CONFIG_FAIL="/var/run/.config_fail"
LOGFILE="/var/log/compute_config.log"
IMA_POLICY=/etc/ima.policy
# Copy of /opt/platform required for compute_services
VOLATILE_PLATFORM_PATH=$VOLATILE_PATH/cpe_upgrade_opt_platform
DELAY_SEC=600
# If we're on a controller, increase DELAY_SEC to a large value
# to allow for active services to recover from a reboot or DOR
if [ "$nodetype" = "controller" ]
then
DELAY_SEC=900
fi
fatal_error()
{
cat <<EOF
*****************************************************
*****************************************************
$1
*****************************************************
*****************************************************
EOF
touch $VOLATILE_CONFIG_FAIL
logger "Error: $1"
echo "Pausing for 5 seconds..."
sleep 5
exit 1
}
get_ip()
{
local host=$1
# Check /etc/hosts for the hostname
local ipaddr=$(cat /etc/hosts | awk -v host=$host '$2 == host {print $1}')
if [ -n "$ipaddr" ]
then
echo $ipaddr
return
fi
START=$SECONDS
let -i UNTIL=${SECONDS}+${DELAY_SEC}
while [ ${UNTIL} -ge ${SECONDS} ]
do
# Because dnsmasq can resolve both a hostname to both an IPv4 and an IPv6
# address in certain situations, and the last address is the IPv6, which
# would be the management, this is preferred over the IPv4 pxeboot address,
# so take the last address only.
ipaddr=$(dig +short ANY $host|tail -1)
if [[ "$ipaddr" =~ ^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*$ ]]
then
let -i DURATION=$SECONDS-$START
logger -t $0 -p info "DNS query resolved to $ipaddr (took ${DURATION} secs)"
echo $ipaddr
return
fi
if [[ "$ipaddr" =~ ^[0-9a-z]*\:[0-9a-z\:]*$ ]]
then
let -i DURATION=$SECONDS-$START
logger -t $0 -p info "DNS query resolved to $ipaddr (took ${DURATION} secs)"
echo $ipaddr
return
fi
logger -t $0 -p warn "DNS query failed for $host"
sleep 5
done
let -i DURATION=$SECONDS-$START
logger -t $0 -p warn "DNS query failed after max retries for $host (${DURATION} secs)"
}
wait_for_controller_services()
{
SERVICE="platform-nfs-ip"
while [ "$SECONDS" -le "$DELAY_SEC" ]
do
# Check to make sure the service is enabled
sm-query service ${SERVICE} | grep -q "enabled-active"
if [ $? -eq 0 ]
then
return 0
fi
# Not running Let's wait a couple of seconds and check again
sleep 2
done
return 1
}
start()
{
if [ -f /etc/platform/installation_failed ] ; then
fatal_error "/etc/platform/installation_failed flag is set. Aborting."
fi
function=`echo "$subfunction" | cut -f 2 -d','`
if [ "$nodetype" != "compute" -a "$function" != "compute" ] ; then
logger -t $0 -p warn "exiting because this is not compute host"
exit 0
fi
# If we're on a controller, ensure we only run if the controller config is complete
if [ "$nodetype" = "controller" -a ! -f /etc/platform/.initial_controller_config_complete ]
then
logger -t $0 -p warn "exiting because this is controller that has not completed initial config"
exit 0
fi
# Exit in error if called while the fail flag file is present
if [ -e $VOLATILE_CONFIG_FAIL ] ; then
logger -t $0 -p warn "exiting due to presence of $VOLATILE_CONFIG_FAIL file"
exit 1
fi
# remove previous pass flag file so that if this fails we don't
# end up with both pass and fail flag files present
rm -f $VOLATILE_CONFIG_PASS
if [ "$(stat -c %d:%i /)" != "$(stat -c %d:%i /proc/1/root/.)" ]; then
# we are in chroot installer environment
exit 0
fi
echo "Configuring compute node..."
###### SECURITY PROFILE (EXTENDED) #################
# If we are in Extended Security Profile mode, #
# then before anything else, we need to load the #
# IMA Policy so that all configuration operations #
# can be measured and appraised #
# #
# N.B: Only run for compute nodetype since for AIO #
# controllerconfig would have already enabled IMA #
# policy #
#####################################################
if [ "$nodetype" = "compute" -a "${security_profile}" = "extended" ]
then
IMA_LOAD_PATH=/sys/kernel/security/ima/policy
if [ -f ${IMA_LOAD_PATH} ]; then
echo "Loading IMA Policy"
# Best effort operation only, if policy is
# malformed then audit logs will indicate this,
# and customer will need to load policy manually
cat $IMA_POLICY > ${IMA_LOAD_PATH}
[ $? -eq 0 ] || logger -t $0 -p warn "IMA Policy could not be loaded, see audit.log"
else
# the securityfs mount should have been
# created had the IMA module loaded properly.
# This is therefore a fatal error
fatal_error "${IMA_LOAD_PATH} not available. Aborting."
fi
fi
HOST=$(hostname)
if [ -z "$HOST" -o "$HOST" = "localhost" ]
then
fatal_error "Host undefined. Unable to perform config"
fi
date "+%FT%T.%3N" > $LOGFILE
IPADDR=$(get_ip $HOST)
if [ -z "$IPADDR" ]
then
fatal_error "Unable to get IP from host: $HOST"
fi
# wait for controller services to be ready if it is an AIO system
# since ping the loopback interface always returns ok
if [ -e "${PLATFORM_SIMPLEX_FLAG}" ]
then
echo "Wait for the controller services"
wait_for_controller_services
if [ $? -ne 0 ]
then
fatal_error "Controller services are not ready"
fi
else
/usr/local/bin/connectivity_test -t ${DELAY_SEC} -i ${IPADDR} controller-platform-nfs
if [ $? -ne 0 ]
then
# 'controller-platform-nfs' is not available from management address
fatal_error "Unable to contact active controller (controller-platform-nfs) from management address"
fi
fi
# Write the hostname to file so it's persistent
echo $HOST > /etc/hostname
if ! [ -e "${PLATFORM_SIMPLEX_FLAG}" ]
then
# Mount the platform filesystem (if necessary - could be auto-mounted by now)
mkdir -p $PLATFORM_DIR
if [ ! -f $CONFIG_DIR/hosts ]
then
nfs-mount controller-platform-nfs:$PLATFORM_DIR $PLATFORM_DIR > /dev/null 2>&1
RC=$?
if [ $RC -ne 0 ]
then
fatal_error "Unable to mount $PLATFORM_DIR (RC:$RC)"
fi
fi
fi
if [ "$nodetype" = "compute" ]
then
# Check whether our installed load matches the active controller
CONTROLLER_UUID=`curl -sf http://controller/feed/rel-${SW_VERSION}/install_uuid`
if [ $? -ne 0 ]
then
fatal_error "Unable to retrieve installation uuid from active controller"
fi
if [ "$INSTALL_UUID" != "$CONTROLLER_UUID" ]
then
fatal_error "This node is running a different load than the active controller and must be reinstalled"
fi
fi
# banner customization always returns 0, success:
/usr/sbin/install_banner_customization
cp $CONFIG_DIR/hosts /etc/hosts
if [ $? -ne 0 ]
then
fatal_error "Unable to copy $CONFIG_DIR/hosts"
fi
if [ "$nodetype" = "controller" -a "$HOST" = "controller-1" ]
then
# In a small system restore, there may be instance data that we want to
# restore. Copy it and delete it.
MATE_INSTANCES_DIR="$CONFIG_DIR/controller-1_nova_instances"
if [ -d "$MATE_INSTANCES_DIR" ]
then
echo "Restoring instance data from mate controller"
cp -Rp $MATE_INSTANCES_DIR/* /etc/nova/instances/
rm -rf $MATE_INSTANCES_DIR
fi
fi
# Upgrade related checks for controller-1 in combined controller/compute
if [ "$nodetype" = "controller" -a "$HOST" = "controller-1" ]
then
# Check controller activity.
# Prior to the final compile of R5 the service check below had been
# against platform-nfs-ip. However, there was a compute
# subfunction configuration failure when an AIO-DX system controller
# booted up while there was no pingable backup controller. Seems the
# platform-nfs-ip service was not always reaching the enabled-active
# state when this check was performed under this particular failure.
# Seems an earlier launched service of like functionality, namely
# 'platform-export-fs' is reliably enabled at this point there-by
# resolving the issue.
sm-query service platform-export-fs | grep enabled-active > /dev/null 2>&1
if [ $? -ne 0 ]
then
# This controller is not active so it is safe to check the version
# of the mate controller.
VOLATILE_ETC_PLATFORM_MOUNT=$VOLATILE_PATH/etc_platform
mkdir $VOLATILE_ETC_PLATFORM_MOUNT
nfs-mount controller-0:/etc/platform $VOLATILE_ETC_PLATFORM_MOUNT
if [ $? -eq 0 ]
then
# Check whether software versions match on the two controllers
MATE_SW_VERSION=$(source $VOLATILE_ETC_PLATFORM_MOUNT/platform.conf && echo $sw_version)
if [ $SW_VERSION != $MATE_SW_VERSION ]
then
echo "Controllers are running different software versions"
echo "SW_VERSION: $SW_VERSION MATE_SW_VERSION: $MATE_SW_VERSION"
# Since controller-1 is always upgraded first (and downgraded
# last), we know that controller-1 is running a higher release
# than controller-0.
# This controller is not active and is running a higher
# release than the mate controller, so do not launch
# any of the compute services (they will not work with
# a lower version of the controller services).
echo "Disabling compute services until controller activated"
touch $VOLATILE_DISABLE_COMPUTE_SERVICES
# Copy $PLATFORM_DIR into a temporary location for the compute_services script to
# access. This is only required for CPE upgrades
rm -rf $VOLATILE_PLATFORM_PATH
mkdir -p $VOLATILE_PLATFORM_PATH
cp -Rp $PLATFORM_DIR/* $VOLATILE_PLATFORM_PATH/
fi
umount $VOLATILE_ETC_PLATFORM_MOUNT
rmdir $VOLATILE_ETC_PLATFORM_MOUNT
else
rmdir $VOLATILE_ETC_PLATFORM_MOUNT
fatal_error "Unable to mount /etc/platform"
fi
else
# Controller-1 (CPE) is active and is rebooting. This is probably a DOR. Since this
# could happen during an upgrade, we will copy $PLATFORM_DIR into a temporary
# location for the compute_services script to access in case of a future swact.
rm -rf $VOLATILE_PLATFORM_PATH
mkdir -p $VOLATILE_PLATFORM_PATH
cp -Rp $PLATFORM_DIR/* $VOLATILE_PLATFORM_PATH/
fi
fi
# Apply the puppet manifest
HOST_HIERA=${PUPPET_PATH}/hieradata/${IPADDR}.yaml
if [ -f ${HOST_HIERA} ]; then
echo "$0: Running puppet manifest apply"
puppet-manifest-apply.sh ${PUPPET_PATH}/hieradata ${IPADDR} compute
RC=$?
if [ $RC -ne 0 ];
then
fatal_error "Failed to run the puppet manifest (RC:$RC)"
fi
else
fatal_error "Host configuration not yet available for this node ($(hostname)=${IPADDR}); aborting configuration."
fi
# Load Network Block Device
modprobe nbd
if [ $? -ne 0 ]
then
echo "WARNING: Unable to load kernel module: nbd."
logger "WARNING: Unable to load kernel module: nbd."
fi
#Run mount command to mount any NFS filesystems that required network access
/bin/mount -a -t nfs
RC=$?
if [ $RC -ne 0 ]
then
fatal_error "Unable to mount NFS filesystems (RC:$RC)"
fi
touch $VOLATILE_CONFIG_PASS
}
stop ()
{
# Nothing to do
return
}
case "$1" in
start)
start
;;
stop)
stop
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
;;
esac
exit 0

220
computeconfig/computeconfig/compute_services Normal file
View File

@@ -0,0 +1,220 @@
#!/bin/bash
#
# Copyright (c) 2016-2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
#
# This script provides support for CPE upgrades. It will be called during swacts
# by the /usr/local/sbin/sm-notification python script, if we are in a small
# footprint system (CPE)
#
# During a swact to, the script will delete the $VOLATILE_DISABLE_COMPUTE_SERVICES
# flag and re-apply the compute manifests.
# During a swact away from (downgrades), the script re-create the
# $VOLATILE_DISABLE_COMPUTE_SERVICES flag and re-apply the compute manifests.
#
# This script should only re-apply the compute manifests if;
# - It is running on a CPE (small footprint) system
# - It is controller-1
# - Controller-0 has not yet been upgraded
#
# This script logs to /var/log/platform.log
#
. /usr/bin/tsconfig
. /etc/platform/platform.conf
VOLATILE_CONFIG_PASS="/var/run/.config_pass"
VOLATILE_CONFIG_FAIL="/var/run/.config_fail"
IN_PROGRESS="/var/run/.compute_services_in_progress"
TEMP_MATE_ETC_DIR="$VOLATILE_PATH/etc_platform_compute"
TEMP_PUPPET_DIR="$VOLATILE_PATH/puppet_compute"
# Copy of /opt/platform populate by compute_config
VOLATILE_PLATFORM_PATH=$VOLATILE_PATH/cpe_upgrade_opt_platform
# Process id and full filename of this executable
NAME="[$$] $0($1)"
end_exec()
{
rm $IN_PROGRESS
exit 0
}
init()
{
local action_to_perform=$1
# This will log to /var/log/platform.log
logger -t $NAME -p local1.info "Begin ..."
# Check if this program is currently executing, if so sleep for 5 seconds and check again.
# After 10 minutes of waiting assume something is wrong and exit.
count=0
while [ -f $IN_PROGRESS ] ; do
if [ $count -gt 120 ] ; then
logger -t $NAME -p local1.error "Execution completion of previous call is taking more than 10 minutes. Exiting."
end_exec
fi
logger -t $NAME -p local1.info "Sleep for 5 seconds"
let count++
sleep 5
done
touch $IN_PROGRESS
HOST=$(hostname)
if [ -z "$HOST" -o "$HOST" = "localhost" ] ; then
logger -t $NAME -p local1.error "Host undefiled"
end_exec
fi
# this script should only be performed on controller-1
if [ "$HOST" != "controller-1" ] ; then
logger -t $NAME -p local1.info "Exiting because this is not controller-1"
end_exec
fi
# This script should only be called if we are in a CPE system
sub_function=`echo "$subfunction" | cut -f 2 -d','`
if [ $sub_function != "compute" ] ; then
logger -t $NAME -p local1.error "Exiting because this is not CPE host"
end_exec
fi
# Exit if called while the config compute success flag file is not present
if [ ! -f $VOLATILE_CONFIG_PASS ] ; then
logger -t $NAME -p local1.info "Exiting due to non-presence of $VOLATILE_CONFIG_PASS file"
end_exec
fi
# Exit if called while the config compute failure flag file is present
if [ -f $VOLATILE_CONFIG_FAIL ] ; then
logger -t $NAME -p local1.info "Exiting due to presence of $VOLATILE_CONFIG_FAIL file"
end_exec
fi
# Ensure we only run if the controller config is complete
if [ ! -f /etc/platform/.initial_controller_config_complete ] ; then
logger -t $NAME -p local1.warn "exiting because CPE controller that has not completed initial config"
end_exec
fi
IPADDR=$(cat /etc/hosts | awk -v host=$HOST '$2 == host {print $1}')
if [ -z "$IPADDR" ] ; then
logger -t $NAME -p local1.error "Unable to get IP from host: $HOST"
end_exec
fi
# The platform filesystem was mounted in compute_config and copied in a temp
# location
if [ ! -f $VOLATILE_PLATFORM_PATH/config/${SW_VERSION}/hosts ] ; then
logger -t $NAME -p local1.error "Error accessing $VOLATILE_PLATFORM_PATH"
end_exec
fi
# Check the release version of controller-0
mkdir $TEMP_MATE_ETC_DIR
nfs-mount controller-0:/etc/platform $TEMP_MATE_ETC_DIR
if [ $? -eq 0 ] ; then
# Should only be executed when the releases do not match
MATE_SW_VERSION=$(source $TEMP_MATE_ETC_DIR/platform.conf && echo $sw_version)
logger -t $NAME -p local1.info "SW_VERSION: $SW_VERSION MATE_SW_VERSION: $MATE_SW_VERSION"
# Check whether software versions match on the two controllers
# Since controller-1 is always upgraded first (and downgraded
# last), we know that controller-1 is running a higher release
# than controller-0.
if [ $SW_VERSION == $MATE_SW_VERSION ] ; then
logger -t $NAME -p local1.info "Releases matches... do not continue"
umount $TEMP_MATE_ETC_DIR
rmdir $TEMP_MATE_ETC_DIR
end_exec
fi
else
logger -t $NAME -p local1.error "Unable to mount /etc/platform"
rmdir $TEMP_MATE_ETC_DIR
end_exec
fi
umount $TEMP_MATE_ETC_DIR
rmdir $TEMP_MATE_ETC_DIR
# Copy the puppet data into $TEMP_PUPPET_DIR
VOLATILE_PUPPET_PATH=${VOLATILE_PLATFORM_PATH}/puppet/${SW_VERSION}
logger -t $NAME -p local1.info "VOLATILE_PUPPET_PATH = $VOLATILE_PUPPET_PATH"
rm -rf $TEMP_PUPPET_DIR
cp -R $VOLATILE_PUPPET_PATH $TEMP_PUPPET_DIR
if [ $? -ne 0 ] ; then
logger -t $NAME -p local1.error "Failed to copy packstack directory $VOLATILE_PUPPET_PATH to $TEMP_PUPPET_DIR "
end_exec
fi
# Update the VOLATILE_DISABLE_COMPUTE_SERVICES flag and stop nova-compute if in "stop"
if [ $action_to_perform == "stop" ] ; then
logger -t $NAME -p local1.info "Disabling compute services"
# Set the compute services disable flag used by the manifest
touch $VOLATILE_DISABLE_COMPUTE_SERVICES
# Stop nova-compute
logger -t $NAME -p local1.info "Stopping nova-compute"
/etc/init.d/e_nova-init stop
else
logger -t $NAME -p local1.info "Enabling compute services"
# Clear the compute services disable flag used by the manifest
rm $VOLATILE_DISABLE_COMPUTE_SERVICES
fi
# Apply the puppet manifest
HOST_HIERA=${TEMP_PUPPET_DIR}/hieradata/${IPADDR}.yaml
if [ -f ${HOST_HIERA} ]; then
echo "$0: Running puppet manifest apply"
puppet-manifest-apply.sh ${TEMP_PUPPET_DIR}/hieradata ${IPADDR} compute
RC=$?
if [ $RC -ne 0 ];
then
logger -t $NAME -p local1.info "Failed to run the puppet manifest (RC:$RC)"
end_exec
fi
else
logger -t $NAME -p local1.info "Host configuration not yet available for this node ($(hostname)=${IPADDR}); aborting configuration."
end_exec
fi
# Start nova-compute is we are starting compute services
if [ $action_to_perform == "start" ] ; then
logger -t $NAME -p local1.info "Starting nova-compute"
/etc/init.d/e_nova-init start
fi
# Cleanup
rm -rf $TEMP_PUPPET_DIR
logger -t $NAME -p local1.info "... Done"
end_exec
}
case "$1" in
start)
init $1
;;
stop)
init $1
;;
*)
logger -t $NAME -p local1.info "Usage: $0 {start|stop}"
exit 1
;;
esac
end_exec

21
computeconfig/computeconfig/computeconfig-combined.service Normal file
View File

@@ -0,0 +1,21 @@
[Unit]
Description=computeconfig service
After=syslog.target network.service remote-fs.target
After=sw-patch.service
After=affine-platform.sh.service compute-huge.sh.service
After=controllerconfig.service config.service
After=goenabled.service
After=sysinv-agent.service
After=network-online.target
[Service]
Type=simple
ExecStart=/etc/init.d/compute_config start
ExecStop=
ExecReload=
StandardOutput=syslog+console
StandardError=syslog+console
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

22
computeconfig/computeconfig/computeconfig.service Normal file
View File

@@ -0,0 +1,22 @@
[Unit]
Description=computeconfig service
After=syslog.target network.service remote-fs.target
After=sw-patch.service
After=affine-platform.sh.service compute-huge.sh.service
After=opt-platform.service
After=sysinv-agent.service
After=network-online.target
Before=config.service compute-config-gate.service
Before=goenabled.service
[Service]
Type=simple
ExecStart=/etc/init.d/compute_config start
ExecStop=
ExecReload=
StandardOutput=syslog+console
StandardError=syslog+console
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

22
computeconfig/computeconfig/config_goenabled_check.sh Normal file
View File

@@ -0,0 +1,22 @@
#!/bin/bash
#
# Copyright (c) 2014 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
# Configuration "goenabled" check.
# If configuration failed, prevent the node from going enabled.
NAME=$(basename $0)
VOLATILE_CONFIG_FAIL="/var/run/.config_fail"
logfile=/var/log/patching.log
if [ -f $VOLATILE_CONFIG_FAIL ]
then
logger "$NAME: Node configuration has failed. Failing goenabled check."
exit 1
fi
exit 0

13
config-gate/PKG-INFO Normal file
View File

@@ -0,0 +1,13 @@
Metadata-Version: 1.1
Name: config-gate
Version: 1.0
Summary: General config initialization gate
Home-page:
Author: Windriver
Author-email: info@windriver.com
License: Apache-2.0
Description: General config initialization gate
Platform: UNKNOWN

2
config-gate/centos/build_srpm.data Normal file
View File

@@ -0,0 +1,2 @@
SRC_DIR="files"
TIS_PATCH_VER=0

59
config-gate/centos/config-gate.spec Normal file
View File

@@ -0,0 +1,59 @@
Summary: config-gate
Name: config-gate
Version: 1.0
Release: %{tis_patch_ver}%{?_tis_dist}
License: Apache-2.0
Group: base
Packager: Wind River <info@windriver.com>
URL: unknown
Source0: %{name}-%{version}.tar.gz
%define debug_package %{nil}
Requires: systemd
%description
Startup configuration gate
%package -n %{name}-compute
Summary: config-gate-compute
Group: base
%description -n %{name}-compute
Startup compute configuration gate
%define local_etc_systemd /etc/systemd/system/
%prep
%setup
%build
%install
install -d -m 755 %{buildroot}%{_sbindir}
install -p -D -m 555 wait_for_config_init.sh %{buildroot}%{_sbindir}/
install -p -D -m 555 wait_for_compute_config_init.sh %{buildroot}%{_sbindir}/
install -d -m 755 %{buildroot}%{local_etc_systemd}
install -p -D -m 444 config.service %{buildroot}%{local_etc_systemd}/config.service
install -p -D -m 444 compute-config-gate.service %{buildroot}%{local_etc_systemd}/compute-config-gate.service
%post
systemctl enable config.service
%post -n %{name}-compute
systemctl enable compute-config-gate.service
%clean
# rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
%doc LICENSE
%{_sbindir}/wait_for_config_init.sh
%{local_etc_systemd}/config.service
%files -n %{name}-compute
%defattr(-,root,root,-)
%{_sbindir}/wait_for_compute_config_init.sh
%{local_etc_systemd}/compute-config-gate.service

202
config-gate/files/LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

15
config-gate/files/compute-config-gate.service Normal file
View File

@@ -0,0 +1,15 @@
[Unit]
Description=TIS compute config gate
After=sw-patch.service computeconfig.service
Before=serial-getty@ttyS0.service getty@tty1.service
[Service]
Type=oneshot
ExecStart=/usr/sbin/wait_for_compute_config_init.sh
ExecStop=
ExecReload=
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

16
config-gate/files/config.service Normal file
View File

@@ -0,0 +1,16 @@
[Unit]
Description=General TIS config gate
After=sw-patch.service
Before=serial-getty@ttyS0.service getty@tty1.service
# Each config service must have a Before statement against config.service, to ensure ordering
[Service]
Type=oneshot
ExecStart=/usr/sbin/wait_for_config_init.sh
ExecStop=
ExecReload=
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target

20
config-gate/files/wait_for_compute_config_init.sh Normal file
View File

@@ -0,0 +1,20 @@
#!/bin/bash
#
# Copyright (c) 2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
# Wait for compute config service
SERVICE=computeconfig.service
while :
do
systemctl status $SERVICE |grep -q running
if [ $? -ne 0 ]; then
exit 0
fi
sleep 1
done

36
config-gate/files/wait_for_config_init.sh Normal file
View File

@@ -0,0 +1,36 @@
#!/bin/bash
#
# Copyright (c) 2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
# Wait for base node config service
. /etc/platform/platform.conf
SERVICE=
case $nodetype in
controller)
SERVICE=controllerconfig.service
;;
compute)
SERVICE=computeconfig.service
;;
storage)
SERVICE=storageconfig.service
;;
*)
exit 1
;;
esac
while :
do
systemctl status $SERVICE |grep -q running
if [ $? -ne 0 ]; then
exit 0
fi
sleep 1
done

6
configutilities/.gitignore vendored Normal file
View File

@@ -0,0 +1,6 @@
!.distro
.distro/centos7/rpmbuild/RPMS
.distro/centos7/rpmbuild/SRPMS
.distro/centos7/rpmbuild/BUILD
.distro/centos7/rpmbuild/BUILDROOT
.distro/centos7/rpmbuild/SOURCES/configutilities*tar.gz

13
configutilities/PKG-INFO Executable file
View File

@@ -0,0 +1,13 @@
Metadata-Version: 1.1
Name: configutilities
Version: 1.2.0
Summary: Titanium Cloud configuration utilities
Home-page:
Author: Windriver
Author-email: info@windriver.com
License: Apache-2.0
Description: Titanium Cloud configuration utilities
Platform: UNKNOWN

3
configutilities/centos/build_srpm.data Executable file
View File

@@ -0,0 +1,3 @@
SRC_DIR="configutilities"
COPY_LIST="$SRC_DIR/LICENSE"
TIS_PATCH_VER=34

64
configutilities/centos/configutilities.spec Executable file
View File

@@ -0,0 +1,64 @@
Summary: configutilities
Name: configutilities
Version: 3.0.0
Release: %{tis_patch_ver}%{?_tis_dist}
License: Apache-2.0
Group: base
Packager: Wind River <info@windriver.com>
URL: unknown
Source0: %{name}-%{version}.tar.gz
Source1: LICENSE
%define debug_package %{nil}
BuildRequires: python-setuptools
Requires: python-netaddr
#Requires: wxPython
%description
Titanium Cloud Controller configuration utilities
%package -n %{name}-cgts-sdk
Summary: configutilities sdk files
Group: devel
%description -n %{name}-cgts-sdk
SDK files for configutilities
%define local_bindir /usr/bin
%define pythonroot /usr/lib64/python2.7/site-packages
%define cgcs_sdk_deploy_dir /opt/deploy/cgcs_sdk
%define cgcs_sdk_tarball_name wrs-%{name}-%{version}.tgz
%prep
%setup
%build
%{__python} setup.py build
%install
%{__python} setup.py install --root=$RPM_BUILD_ROOT \
--install-lib=%{pythonroot} \
--prefix=/usr \
--install-data=/usr/share \
--single-version-externally-managed
sed -i "s#xxxSW_VERSIONxxx#%{platform_release}#" %{name}/common/validator.py
tar czf %{cgcs_sdk_tarball_name} %{name}
mkdir -p $RPM_BUILD_ROOT%{cgcs_sdk_deploy_dir}
install -m 644 %{cgcs_sdk_tarball_name} $RPM_BUILD_ROOT%{cgcs_sdk_deploy_dir}
%clean
rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
%doc LICENSE
%{local_bindir}/*
%dir %{pythonroot}/%{name}
%{pythonroot}/%{name}/*
%dir %{pythonroot}/%{name}-%{version}-py2.7.egg-info
%{pythonroot}/%{name}-%{version}-py2.7.egg-info/*
%files -n %{name}-cgts-sdk
%{cgcs_sdk_deploy_dir}/%{cgcs_sdk_tarball_name}

202
configutilities/configutilities/LICENSE Executable file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

202
configutilities/configutilities/configutilities/LICENSE Executable file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

76
configutilities/configutilities/configutilities/README Executable file
View File

@@ -0,0 +1,76 @@
Copyright © 2017 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
-----------------------------------------------------------------------
Titanium Cloud Configuration Utilities
---------------------------------------
To facilitate various aspects of Titanium Cloud installation and
configuration, utilities have been created to generate and validate
configuration and setup files which are utilized by the system.
Installing the Configuration Utilities
--------------------------------------
This tarball includes several utilities which can be used to aid in the
configuration of Titanium Cloud. Note that these are optional tools which are run prior
to installation, and not run on the target system.
To install the utilities on a Linux machine follow these steps:
1. Ensure you have the tools necessary to install new python packages (pip and setuptools)
If you do not, you must install them using the appropriate commands for
your version of linux, such as:
sudo apt-get install python-pip # e.g. for Ubuntu or Debian
2. The config_gui tool makes use of external tools which must be
installed as follows:
if using Ubuntu/Debian:
sudo apt-get install python-wxgtk2.8 python-wxtools
if using Fedora:
sudo yum install wxPython python-setuptools
if using CentOS/RedHat, the appropriate rpm can be obtained from EPEL
sudo yum install epel-release
sudo yum install wxPython
Note, if epel-release is not available, it can be obtained as such (specific to
your version)
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
sudo rpm -Uvh epel-release-6*.rpm
sudo yum install wxPython
3. Copy wrs-configutilities-3.0.0.tgz to the python install directory
(i.e. /usr/lib/python2.7/dist-packages or /usr/lib/python2.7/site-packages)
4. Cd to this python install directory
5. Untar the file:
sudo tar xfv wrs-configutilities-3.0.0.tgz
6. Cd configutilities
7. Run setup:
sudo python setup.py install
Using the Configuration Utilities
---------------------------------
There are two tools installed: config_validator and config_gui.
config_validator is a commandline tool which takes a 'controller configuration
input' file of the INI type and does preliminary analysis to ensure its validity.
It can be called as follows:
config_validator --system-config <filename>
config_gui is a GUI-based tool which provides tools for creating a 'controller
configuration input' INI file and/or a 'bulk host' XML file. It can be launched
by calling 'config_gui' from the command line and will walk you through the process
of generating the desired configuration files.

20
configutilities/configutilities/configutilities/__init__.py Executable file
View File

@@ -0,0 +1,20 @@
#
# Copyright (c) 2015-2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
# flake8: noqa
#
from common.validator import validate
from common.configobjects import (Network, DEFAULT_CONFIG, REGION_CONFIG,
DEFAULT_NAMES, HP_NAMES, SUBCLOUD_CONFIG,
MGMT_TYPE, INFRA_TYPE, OAM_TYPE,
NETWORK_PREFIX_NAMES, HOST_XML_ATTRIBUTES,
LINK_SPEED_1G, LINK_SPEED_10G,
DEFAULT_DOMAIN_NAME)
from common.exceptions import ConfigError, ConfigFail, ValidateFail
from common.utils import is_valid_vlan, is_mtu_valid, is_speed_valid, \
validate_network_str, validate_address_str, validate_address, \
ip_version_to_string, lag_mode_to_str, \
validate_openstack_password, extract_openstack_password_rules_from_file

5
configutilities/configutilities/configutilities/common/__init__.py Normal file
View File

@@ -0,0 +1,5 @@
#
# Copyright (c) 2015 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#

381
configutilities/configutilities/configutilities/common/configobjects.py Executable file
View File

@@ -0,0 +1,381 @@
"""
Copyright (c) 2015-2016 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
from netaddr import iter_iprange
from exceptions import ConfigFail, ValidateFail
from utils import is_mtu_valid, is_speed_valid, is_valid_vlan, \
validate_network_str, validate_address_str
DEFAULT_CONFIG = 0
REGION_CONFIG = 1
SUBCLOUD_CONFIG = 2
MGMT_TYPE = 0
INFRA_TYPE = 1
OAM_TYPE = 2
NETWORK_PREFIX_NAMES = [
('MGMT', 'INFRA', 'OAM'),
('CLM', 'BLS', 'CAN')
]
LINK_SPEED_1G = 1000
LINK_SPEED_10G = 10000
LINK_SPEED_25G = 25000
VALID_LINK_SPEED = [LINK_SPEED_1G, LINK_SPEED_10G, LINK_SPEED_25G]
# Additions to this list must be reflected in the hostfile
# generator tool (config->configutilities->hostfiletool.py)
HOST_XML_ATTRIBUTES = ['hostname', 'personality', 'subfunctions',
'mgmt_mac', 'mgmt_ip',
'bm_ip', 'bm_type', 'bm_username',
'bm_password', 'boot_device', 'rootfs_device',
'install_output', 'console', 'vsc_controllers',
'power_on', 'location', 'subtype']
# Network naming types
DEFAULT_NAMES = 0
HP_NAMES = 1
# well-known default domain name
DEFAULT_DOMAIN_NAME = 'Default'
class LogicalInterface(object):
""" Represents configuration for a logical interface.
"""
def __init__(self):
self.name = None
self.mtu = None
self.link_capacity = None
self.lag_interface = False
self.lag_mode = None
self.ports = None
def parse_config(self, system_config, logical_interface):
# Ensure logical interface config is present
if not system_config.has_section(logical_interface):
raise ConfigFail("Missing config for logical interface %s." %
logical_interface)
self.name = logical_interface
# Parse/validate the MTU
self.mtu = system_config.getint(logical_interface, 'INTERFACE_MTU')
if not is_mtu_valid(self.mtu):
raise ConfigFail("Invalid MTU value for %s. "
"Valid values: 576 - 9216" % logical_interface)
# Parse/validate the link_capacity
if system_config.has_option(logical_interface,
'INTERFACE_LINK_CAPACITY'):
self.link_capacity = \
system_config.getint(logical_interface,
'INTERFACE_LINK_CAPACITY')
# link_capacity is optional
if self.link_capacity:
if not is_speed_valid(self.link_capacity,
valid_speeds=VALID_LINK_SPEED):
raise ConfigFail(
"Invalid link-capacity value for %s." % logical_interface)
# Parse the ports
self.ports = filter(None, [x.strip() for x in
system_config.get(logical_interface,
'INTERFACE_PORTS').split(',')])
# Parse/validate the LAG config
lag_interface = system_config.get(logical_interface,
'LAG_INTERFACE')
if lag_interface.lower() == 'y':
self.lag_interface = True
if len(self.ports) != 2:
raise ConfigFail(
"Invalid number of ports (%d) supplied for LAG "
"interface %s" % (len(self.ports), logical_interface))
self.lag_mode = system_config.getint(logical_interface, 'LAG_MODE')
if self.lag_mode < 1 or self.lag_mode > 6:
raise ConfigFail(
"Invalid LAG_MODE value of %d for %s. Valid values: 1-6" %
(self.lag_mode, logical_interface))
elif lag_interface.lower() == 'n':
if len(self.ports) > 1:
raise ConfigFail(
"More than one interface supplied for non-LAG "
"interface %s" % logical_interface)
if len(self.ports) == 0:
raise ConfigFail(
"No interfaces supplied for non-LAG "
"interface %s" % logical_interface)
else:
raise ConfigFail(
"Invalid LAG_INTERFACE value of %s for %s. Valid values: "
"Y or N" % (lag_interface, logical_interface))
class Network(object):
""" Represents configuration for a network.
"""
def __init__(self):
self.vlan = None
self.cidr = None
self.multicast_cidr = None
self.start_address = None
self.end_address = None
self.floating_address = None
self.address_0 = None
self.address_1 = None
self.dynamic_allocation = False
self.gateway_address = None
self.logical_interface = None
def parse_config(self, system_config, config_type, network_type,
min_addresses=0, multicast_addresses=0, optional=False,
naming_type=DEFAULT_NAMES):
network_prefix = NETWORK_PREFIX_NAMES[naming_type][network_type]
network_name = network_prefix + '_NETWORK'
if naming_type == HP_NAMES:
attr_prefix = network_prefix + '_'
else:
attr_prefix = ''
# Ensure network config is present
if not system_config.has_section(network_name):
if not optional:
raise ConfigFail("Missing config for network %s." %
network_name)
else:
# Optional interface - just return
return
# Parse/validate the VLAN
if system_config.has_option(network_name, attr_prefix + 'VLAN'):
self.vlan = system_config.getint(network_name,
attr_prefix + 'VLAN')
if self.vlan:
if not is_valid_vlan(self.vlan):
raise ConfigFail(
"Invalid %s value of %d for %s. Valid values: 1-4094" %
(attr_prefix + 'VLAN', self.vlan, network_name))
# Parse/validate the cidr
cidr_str = system_config.get(network_name, attr_prefix + 'CIDR')
try:
self.cidr = validate_network_str(
cidr_str, min_addresses)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s" %
(attr_prefix + 'CIDR', cidr_str, network_name, e))
# Parse/validate the multicast subnet
if 0 < multicast_addresses and \
system_config.has_option(network_name,
attr_prefix + 'MULTICAST_CIDR'):
multicast_cidr_str = system_config.get(network_name, attr_prefix +
'MULTICAST_CIDR')
try:
self.multicast_cidr = validate_network_str(
multicast_cidr_str, multicast_addresses, multicast=True)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s" %
(attr_prefix + 'MULTICAST_CIDR', multicast_cidr_str,
network_name, e))
if self.cidr.version != self.multicast_cidr.version:
raise ConfigFail(
"Invalid %s value of %s for %s. Multicast "
"subnet and network IP families must be the same." %
(attr_prefix + 'MULTICAST_CIDR', multicast_cidr_str,
network_name))
# Parse/validate the hardwired controller addresses
floating_address_str = None
address_0_str = None
address_1_str = None
if min_addresses == 1:
if (system_config.has_option(
network_name, attr_prefix + 'IP_FLOATING_ADDRESS') or
system_config.has_option(
network_name, attr_prefix + 'IP_UNIT_0_ADDRESS') or
system_config.has_option(
network_name, attr_prefix + 'IP_UNIT_1_ADDRESS') or
system_config.has_option(
network_name, attr_prefix + 'IP_START_ADDRESS') or
system_config.has_option(
network_name, attr_prefix + 'IP_END_ADDRESS')):
raise ConfigFail(
"Only one IP address is required for OAM "
"network, use 'IP_ADDRESS' to specify the OAM IP "
"address")
floating_address_str = system_config.get(
network_name, attr_prefix + 'IP_ADDRESS')
try:
self.floating_address = validate_address_str(
floating_address_str, self.cidr)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s" %
(attr_prefix + 'IP_ADDRESS',
floating_address_str, network_name, e))
self.address_0 = self.floating_address
self.address_1 = self.floating_address
else:
if system_config.has_option(
network_name, attr_prefix + 'IP_FLOATING_ADDRESS'):
floating_address_str = system_config.get(
network_name, attr_prefix + 'IP_FLOATING_ADDRESS')
try:
self.floating_address = validate_address_str(
floating_address_str, self.cidr)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s" %
(attr_prefix + 'IP_FLOATING_ADDRESS',
floating_address_str, network_name, e))
if system_config.has_option(
network_name, attr_prefix + 'IP_UNIT_0_ADDRESS'):
address_0_str = system_config.get(
network_name, attr_prefix + 'IP_UNIT_0_ADDRESS')
try:
self.address_0 = validate_address_str(
address_0_str, self.cidr)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s" %
(attr_prefix + 'IP_UNIT_0_ADDRESS',
address_0_str, network_name, e))
if system_config.has_option(
network_name, attr_prefix + 'IP_UNIT_1_ADDRESS'):
address_1_str = system_config.get(
network_name, attr_prefix + 'IP_UNIT_1_ADDRESS')
try:
self.address_1 = validate_address_str(
address_1_str, self.cidr)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s" %
(attr_prefix + 'IP_UNIT_1_ADDRESS',
address_1_str, network_name, e))
# Parse/validate the start/end addresses
start_address_str = None
end_address_str = None
if system_config.has_option(
network_name, attr_prefix + 'IP_START_ADDRESS'):
start_address_str = system_config.get(
network_name, attr_prefix + 'IP_START_ADDRESS')
try:
self.start_address = validate_address_str(
start_address_str, self.cidr)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s" %
(attr_prefix + 'IP_START_ADDRESS',
start_address_str, network_name, e))
if system_config.has_option(
network_name, attr_prefix + 'IP_END_ADDRESS'):
end_address_str = system_config.get(
network_name, attr_prefix + 'IP_END_ADDRESS')
try:
self.end_address = validate_address_str(
end_address_str, self.cidr)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s " %
(attr_prefix + 'IP_END_ADDRESS',
end_address_str, network_name, e))
if start_address_str or end_address_str:
if not end_address_str:
raise ConfigFail("Missing attribute %s for %s_NETWORK" %
(attr_prefix + 'IP_END_ADDRESS',
network_name))
if not start_address_str:
raise ConfigFail("Missing attribute %s for %s_NETWORK" %
(attr_prefix + 'IP_START_ADDRESS',
network_name))
if not self.start_address < self.end_address:
raise ConfigFail(
"Start address %s not less than end address %s for %s."
% (str(self.start_address), str(self.end_address),
network_name))
address_list = list(iter_iprange(start_address_str,
end_address_str))
if not len(address_list) >= min_addresses:
raise ConfigFail("Address range for %s must contain at "
"least %d addresses." %
(network_name, min_addresses))
if floating_address_str or address_0_str or address_1_str:
if not floating_address_str:
raise ConfigFail("Missing attribute %s for %s_NETWORK" %
(attr_prefix + 'IP_FLOATING_ADDRESS',
network_name))
if not address_0_str:
raise ConfigFail("Missing attribute %s for %s_NETWORK" %
(attr_prefix + 'IP_UNIT_0_ADDRESS',
network_name))
if not address_1_str:
raise ConfigFail("Missing attribute %s for %s_NETWORK" %
(attr_prefix + 'IP_UNIT_1_ADDRESS',
network_name))
if start_address_str and floating_address_str:
raise ConfigFail("Overspecified network: Can only set %s "
"and %s OR %s, %s, and %s for "
"%s_NETWORK" %
(attr_prefix + 'IP_START_ADDRESS',
attr_prefix + 'IP_END_ADDRESS',
attr_prefix + 'IP_FLOATING_ADDRESS',
attr_prefix + 'IP_UNIT_0_ADDRESS',
attr_prefix + 'IP_UNIT_1_ADDRESS',
network_name))
if config_type == DEFAULT_CONFIG:
if not self.start_address:
self.start_address = self.cidr[2]
if not self.end_address:
self.end_address = self.cidr[-2]
# Parse/validate the dynamic IP address allocation
if system_config.has_option(network_name,
'DYNAMIC_ALLOCATION'):
dynamic_allocation = system_config.get(network_name,
'DYNAMIC_ALLOCATION')
if dynamic_allocation.lower() == 'y':
self.dynamic_allocation = True
elif dynamic_allocation.lower() == 'n':
self.dynamic_allocation = False
else:
raise ConfigFail(
"Invalid DYNAMIC_ALLOCATION value of %s for %s. "
"Valid values: Y or N" %
(dynamic_allocation, network_name))
# Parse/validate the gateway (optional)
if system_config.has_option(network_name, attr_prefix + 'GATEWAY'):
gateway_address_str = system_config.get(
network_name, attr_prefix + 'GATEWAY')
try:
self.gateway_address = validate_address_str(
gateway_address_str, self.cidr)
except ValidateFail as e:
raise ConfigFail(
"Invalid %s value of %s for %s.\nReason: %s" %
(attr_prefix + 'GATEWAY',
gateway_address_str, network_name, e))
# Parse/validate the logical interface
logical_interface_name = system_config.get(
network_name, attr_prefix + 'LOGICAL_INTERFACE')
self.logical_interface = LogicalInterface()
self.logical_interface.parse_config(system_config,
logical_interface_name)

98
configutilities/configutilities/configutilities/common/crypt.py Normal file
View File

@@ -0,0 +1,98 @@
# Copyright 2011 OpenStack Foundation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Routines for URL-safe encrypting/decrypting
Cloned from git/glance/common
"""
import base64
import os
import random
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.ciphers import algorithms
from cryptography.hazmat.primitives.ciphers import Cipher
from cryptography.hazmat.primitives.ciphers import modes
from oslo_utils import encodeutils
import six
# NOTE(jokke): simplified transition to py3, behaves like py2 xrange
from six.moves import range
def urlsafe_encrypt(key, plaintext, blocksize=16):
"""
Encrypts plaintext. Resulting ciphertext will contain URL-safe characters.
If plaintext is Unicode, encode it to UTF-8 before encryption.
:param key: AES secret key
:param plaintext: Input text to be encrypted
:param blocksize: Non-zero integer multiple of AES blocksize in bytes (16)
:returns: Resulting ciphertext
"""
def pad(text):
"""
Pads text to be encrypted
"""
pad_length = (blocksize - len(text) % blocksize)
# NOTE(rosmaita): I know this looks stupid, but we can't just
# use os.urandom() to get the bytes because we use char(0) as
# a delimiter
pad = b''.join(six.int2byte(random.SystemRandom().randint(1, 0xFF))
for i in range(pad_length - 1))
# We use chr(0) as a delimiter between text and padding
return text + b'\0' + pad
plaintext = encodeutils.to_utf8(plaintext)
key = encodeutils.to_utf8(key)
# random initial 16 bytes for CBC
init_vector = os.urandom(16)
backend = default_backend()
cypher = Cipher(algorithms.AES(key), modes.CBC(init_vector),
backend=backend)
encryptor = cypher.encryptor()
padded = encryptor.update(
pad(six.binary_type(plaintext))) + encryptor.finalize()
encoded = base64.urlsafe_b64encode(init_vector + padded)
if six.PY3:
encoded = encoded.decode('ascii')
return encoded
def urlsafe_decrypt(key, ciphertext):
"""
Decrypts URL-safe base64 encoded ciphertext.
On Python 3, the result is decoded from UTF-8.
:param key: AES secret key
:param ciphertext: The encrypted text to decrypt
:returns: Resulting plaintext
"""
# Cast from unicode
ciphertext = encodeutils.to_utf8(ciphertext)
key = encodeutils.to_utf8(key)
ciphertext = base64.urlsafe_b64decode(ciphertext)
backend = default_backend()
cypher = Cipher(algorithms.AES(key), modes.CBC(ciphertext[:16]),
backend=backend)
decryptor = cypher.decryptor()
padded = decryptor.update(ciphertext[16:]) + decryptor.finalize()
text = padded[:padded.rfind(b'\0')]
if six.PY3:
text = text.decode('utf-8')
return text

25
configutilities/configutilities/configutilities/common/exceptions.py Normal file
View File

@@ -0,0 +1,25 @@
#
# Copyright (c) 2015 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
class ConfigError(Exception):
"""Base class for configuration exceptions."""
def __init__(self, message=None):
self.message = message
def __str__(self):
return self.message or ""
class ConfigFail(ConfigError):
"""General configuration error."""
pass
class ValidateFail(ConfigError):
"""Validation of data failed."""
pass

295
configutilities/configutilities/configutilities/common/guicomponents.py Executable file
View File

@@ -0,0 +1,295 @@
#
# Copyright (c) 2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
import wx
from exceptions import ValidateFail
import wrs_ico
TEXT_BOX_SIZE = (150, -1)
TEXT_WIDTH = 450
DEBUG = False
VGAP = 5
HGAP = 10
def debug(msg):
if DEBUG:
print msg
# Tracks what type of controls will implement a config question
class TYPES(object):
string = 1
int = 2
radio = 3
choice = 4
checkbox = 5
help = 6
separator = 7
class Field(object):
def __init__(self, text="", type=TYPES.string, transient=False,
initial="", choices=[], shows=[], reverse=False,
enabled=True):
"""Represent a configuration question
:param text: Question prompt text
:param type: The type of wxWidgets control(s) used to implement this
field
:param transient: Whether this field should be written automatically
to the INI file
:param enabled: Whether this field should be enabled or
disabled (greyed-out)
:param initial: Initial value used to populate the control
:param choices: A string list of choices to populate selection-based
fields
:param shows: A list of field key strings that this field should show
when checked. Only checkboxes implement this functionality atm
:param reverse: Switches the 'shows' logic -> checked
will hide fields instead of showing them
:return: the Field object
"""
self.text = text
self.type = type
self.transient = transient
self.initial = initial
self.choices = choices
self.shows = shows
self.reverse = reverse
self.enabled = enabled
# Controls used to implement this field
self.prompt = None
self.input = None
if type is TYPES.help:
self.transient = True
# Sanity to make sure fields are being utilized correctly
if self.shows and self.type is TYPES.help:
raise NotImplementedError()
if not self.shows and self.reverse:
raise NotImplementedError()
def get_value(self):
# Return value of the control (a string or int)
if not self.input:
value = None
elif not self.input.IsShown() or not self.input.IsEnabled():
value = None
elif self.type is TYPES.string:
value = self.input.GetLineText(0)
elif self.type is TYPES.int:
try:
value = self.input.GetLineText(0)
int(value)
except ValueError:
raise ValidateFail(
"Invalid entry for %s. Must enter a numeric value" %
self.text)
elif self.type is TYPES.radio:
value = self.input.GetString(self.input.GetSelection())
elif self.type is TYPES.choice:
value = self.input.GetString(self.input.GetSelection())
elif self.type is TYPES.checkbox:
value = "N"
if self.input.GetValue():
value = "Y"
else:
raise NotImplementedError()
return value
def set_value(self, value):
# Set value of the control (string or int)
if not self.input:
# Can't 'set' help text etc.
raise NotImplementedError()
elif self.type is TYPES.string or self.type is TYPES.int:
self.input.SetValue(value)
elif self.type is TYPES.radio or self.type is TYPES.choice:
index = self.input.FindString(value)
if index == wx.NOT_FOUND:
raise ValidateFail("Invalid value %s for field %s" %
(value, self.text))
self.input.SetSelection(index)
elif self.type is TYPES.checkbox:
self.input.SetValue(value == "Y")
else:
raise NotImplementedError()
def destroy(self):
if self.prompt:
self.prompt.Destroy()
if self.input:
self.input.Destroy()
def show(self, visible):
debug("Setting visibility to %s for field %s prompt=%s" %
(visible, self.text, self.prompt))
if visible:
if self.prompt:
self.prompt.Show()
if self.input:
self.input.Show()
else:
if self.prompt:
self.prompt.Hide()
if self.input:
self.input.Hide()
def prepare_fields(parent, fields, sizer, change_hdlr):
for row, (name, field) in enumerate(fields.items()):
initial = field.initial
# if config.has_option(parent.section, name):
# initial = config.get(parent.section, name)
add_attributes = wx.ALIGN_CENTER_VERTICAL
width = 1
field.prompt = wx.StaticText(parent, label=field.text, name=name)
# Generate different control based on field type
if field.type is TYPES.string or field.type is TYPES.int:
field.input = wx.TextCtrl(parent, value=initial, name=name,
size=TEXT_BOX_SIZE)
elif field.type is TYPES.radio:
field.input = wx.RadioBox(
parent, choices=field.choices, majorDimension=1,
style=wx.RA_SPECIFY_COLS, name=name, id=wx.ID_ANY)
elif field.type is TYPES.choice:
field.input = wx.Choice(
parent, choices=field.choices, name=name)
if initial:
field.input.SetSelection(field.input.FindString(initial))
elif field.type is TYPES.checkbox:
width = 2
field.input = wx.CheckBox(parent, name=name, label=field.text,
) # style=wx.ALIGN_RIGHT)
field.input.SetValue(initial == 'Y')
if field.prompt:
field.prompt.Hide()
field.prompt = None
elif field.type is TYPES.help:
width = 2
field.prompt.Wrap(TEXT_WIDTH)
field.input = None
elif field.type is TYPES.separator:
width = 2
field.prompt = wx.StaticLine(parent, -1)
add_attributes = wx.EXPAND | wx.ALL
field.input = None
else:
raise NotImplementedError()
col = 0
if field.prompt:
sizer.Add(field.prompt, (row, col), span=(1, width),
flag=add_attributes)
col += 1
if field.input:
field.input.Enable(field.enabled)
sizer.Add(field.input, (row, col),
flag=add_attributes)
# Go through again and set show/hide relationships
for name, field in fields.items():
if field.shows:
# Add display handlers
field.input.Bind(wx.EVT_CHECKBOX, change_hdlr)
# todo tsmith add other evts
# Start by hiding target prompt/input controls
for target_name in field.shows:
target = fields[target_name]
if target.prompt:
target.prompt.Hide()
if target.input:
target.input.Hide()
def on_change(parent, fields, event):
obj = event.GetEventObject()
# debug("Checked: " + str(event.Checked()) +
# ", Reverse: " + str(parent.fields[obj.GetName()].reverse) +
# ", Will show: " + str(event.Checked() is not
# parent.fields[obj.GetName()].reverse))
# Hide/Show the targets of the control
# Note: the "is not" implements switching the show logic around
handle_sub_show(
fields,
fields[obj.GetName()].shows,
event.Checked() is not fields[obj.GetName()].reverse)
parent.Layout()
event.Skip()
def handle_sub_show(fields, targets, show):
""" Recursive function to handle showing/hiding of a list of fields
:param targets: [String]
:param show: bool
"""
sub_handled = []
for tgt in targets:
if tgt in sub_handled:
# Handled by newly shown control
continue
tgt_field = fields[tgt]
# Show or hide this field as necessary
tgt_field.show(show)
# If it shows others (checkbox) and is now shown,
# apply it's value decide on showing it's children, not the
# original show
if tgt_field.shows and show:
sub_handled.extend(tgt_field.shows)
handle_sub_show(
fields,
tgt_field.shows,
(tgt_field.get_value() is 'Y') is not fields[tgt].reverse)
def set_icons(parent):
# Icon setting
# todo Make higher resolution icons, verify on different linux desktops
icons = wx.IconBundle()
for sz in [16, 32, 48]:
# try:
# icon = wx.Icon(wrs_ico.windriver_favicon.getIcon(),
# width=sz, height=sz)
icon = wrs_ico.favicon.getIcon()
icons.AddIcon(icon)
# except:
# pass
parent.SetIcons(icons)
# ico = wrs_ico.windriver_favicon.getIcon()
# self.SetIcon(ico)
# self.tbico = wx.TaskBarIcon()
# self.tbico.SetIcon(ico, '')

308
configutilities/configutilities/configutilities/common/utils.py Normal file
View File

@@ -0,0 +1,308 @@
"""
Copyright (c) 2015-2016 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
import ConfigParser
import re
import six
from netaddr import (IPNetwork,
IPAddress,
AddrFormatError)
from exceptions import ValidateFail
EXPECTED_SERVICE_NAME_AND_TYPE = (
{"KEYSTONE_SERVICE_NAME": "keystone",
"KEYSTONE_SERVICE_TYPE": "identity",
"GLANCE_SERVICE_NAME": "glance",
"GLANCE_SERVICE_TYPE": "image",
"NOVA_SERVICE_NAME": "nova",
"NOVA_SERVICE_TYPE": "compute",
"PLACEMENT_SERVICE_NAME": "placement",
"PLACEMENT_SERVICE_TYPE": "placement",
"NEUTRON_SERVICE_NAME": "neutron",
"NEUTRON_SERVICE_TYPE": "network",
"SYSINV_SERVICE_NAME": "sysinv",
"SYSINV_SERVICE_TYPE": "platform",
"PATCHING_SERVICE_NAME": "patching",
"PATCHING_SERVICE_TYPE": "patching",
"HEAT_SERVICE_NAME": "heat",
"HEAT_SERVICE_TYPE": "orchestration",
"HEAT_CFN_SERVICE_NAME": "heat-cfn",
"HEAT_CFN_SERVICE_TYPE": "cloudformation",
"CEILOMETER_SERVICE_NAME": "ceilometer",
"CEILOMETER_SERVICE_TYPE": "metering",
"NFV_SERVICE_NAME": "vim",
"NFV_SERVICE_TYPE": "nfv",
"AODH_SERVICE_NAME": "aodh",
"AODH_SERVICE_TYPE": "alarming",
"PANKO_SERVICE_NAME": "panko",
"PANKO_SERVICE_TYPE": "event"})
def is_valid_vlan(vlan):
"""Determine whether vlan is valid."""
try:
if 0 < int(vlan) < 4095:
return True
else:
return False
except (ValueError, TypeError):
return False
def is_mtu_valid(mtu):
"""Determine whether a mtu is valid."""
try:
if int(mtu) < 576:
return False
elif int(mtu) > 9216:
return False
else:
return True
except (ValueError, TypeError):
return False
def is_speed_valid(speed, valid_speeds=None):
"""Determine whether speed is valid."""
try:
if valid_speeds is not None and int(speed) not in valid_speeds:
return False
else:
return True
except (ValueError, TypeError):
return False
def is_valid_hostname(hostname):
"""Determine whether a hostname is valid as per RFC 1123."""
# Maximum length of 255
if not hostname or len(hostname) > 255:
return False
# Allow a single dot on the right hand side
if hostname[-1] == ".":
hostname = hostname[:-1]
# Create a regex to ensure:
# - hostname does not begin or end with a dash
# - each segment is 1 to 63 characters long
# - valid characters are A-Z (any case) and 0-9
valid_re = re.compile("(?!-)[A-Z\d-]{1,63}(?<!-)$", re.IGNORECASE)
return all(valid_re.match(x) for x in hostname.split("."))
def is_valid_mac(mac):
"""Verify the format of a MAC addres."""
if not mac:
return False
m = "[0-9a-f]{2}([-:])[0-9a-f]{2}(\\1[0-9a-f]{2}){4}$"
return isinstance(mac, six.string_types) and re.match(m, mac.lower())
def validate_network_str(network_str, minimum_size,
existing_networks=None, multicast=False):
"""Determine whether a network is valid."""
try:
network = IPNetwork(network_str)
if network.ip != network.network:
raise ValidateFail("Invalid network address")
elif network.size < minimum_size:
raise ValidateFail("Subnet too small - must have at least %d "
"addresses" % minimum_size)
elif network.version == 6 and network.prefixlen < 64:
raise ValidateFail("IPv6 minimum prefix length is 64")
elif existing_networks:
if any(network.ip in subnet for subnet in existing_networks):
raise ValidateFail("Subnet overlaps with another "
"configured subnet")
elif multicast and not network.is_multicast():
raise ValidateFail("Invalid subnet - must be multicast")
return network
except AddrFormatError:
raise ValidateFail(
"Invalid subnet - not a valid IP subnet")
def is_valid_filename(filename):
return '\0' not in filename
def is_valid_by_path(filename):
return "/dev/disk/by-path" in filename and "-part" not in filename
def validate_address_str(ip_address_str, network):
"""Determine whether an address is valid."""
try:
ip_address = IPAddress(ip_address_str)
if ip_address.version != network.version:
msg = ("Invalid IP version - must match network version " +
ip_version_to_string(network.version))
raise ValidateFail(msg)
elif ip_address == network:
raise ValidateFail("Cannot use network address")
elif ip_address == network.broadcast:
raise ValidateFail("Cannot use broadcast address")
elif ip_address not in network:
raise ValidateFail(
"Address must be in subnet %s" % str(network))
return ip_address
except AddrFormatError:
raise ValidateFail(
"Invalid address - not a valid IP address")
def ip_version_to_string(ip_version):
"""Determine whether a nameserver address is valid."""
if ip_version == 4:
return "IPv4"
elif ip_version == 6:
return "IPv6"
else:
return "IP"
def validate_nameserver_address_str(ip_address_str, subnet_version=None):
"""Determine whether a nameserver address is valid."""
try:
ip_address = IPAddress(ip_address_str)
if subnet_version is not None and ip_address.version != subnet_version:
msg = ("Invalid IP version - must match OAM subnet version " +
ip_version_to_string(subnet_version))
raise ValidateFail(msg)
return ip_address
except AddrFormatError:
msg = "Invalid address - "
"not a valid %s address" % ip_version_to_string(subnet_version)
raise ValidateFail(msg)
def validate_address(ip_address, network):
"""Determine whether an address is valid."""
if ip_address.version != network.version:
msg = ("Invalid IP version - must match network version " +
ip_version_to_string(network.version))
raise ValidateFail(msg)
elif ip_address == network:
raise ValidateFail("Cannot use network address")
elif ip_address == network.broadcast:
raise ValidateFail("Cannot use broadcast address")
elif ip_address not in network:
raise ValidateFail("Address must be in subnet %s" % str(network))
def check_network_overlap(new_network, configured_networks):
""" Validate that new_network does not overlap any configured_networks.
"""
if any(new_network.ip in subnet for subnet in
configured_networks):
raise ValidateFail(
"Subnet %s overlaps with another configured subnet" % new_network)
def lag_mode_to_str(lag_mode):
if lag_mode == 0:
return "balance-rr"
if lag_mode == 1:
return "active-backup"
elif lag_mode == 2:
return "balance-xor"
elif lag_mode == 3:
return "broadcast"
elif lag_mode == 4:
return "802.3ad"
elif lag_mode == 5:
return "balance-tlb"
elif lag_mode == 6:
return "balance-alb"
else:
raise Exception(
"Invalid LAG_MODE value of %d. Valid values: 0-6" % lag_mode)
def validate_openstack_password(password, rules_file,
section="security_compliance"):
try:
config = ConfigParser.RawConfigParser()
parsed_config = config.read(rules_file)
if not parsed_config:
msg = ("Cannot parse rules file: %s" % rules_file)
raise Exception(msg)
if not config.has_section(section):
msg = ("Required section '%s' not found in rules file" % section)
raise Exception(msg)
password_regex = get_optional(config, section, 'password_regex')
password_regex_description = get_optional(config, section,
'password_regex_description')
if not password_regex:
msg = ("Required option 'password_regex' not found in "
"rule file: %s" % rules_file)
raise Exception(msg)
# Even if regex_description is not found, we will proceed
# and give a generic failure warning instead
if not password_regex_description:
password_regex_description = ("Password does not meet "
"complexity criteria")
if not isinstance(password, six.string_types):
msg = ("Password must be a string type")
raise Exception(msg)
try:
# config parser would read in the string as a literal
# representation which would fail regex matching
password_regex = password_regex.strip('"')
if not re.match(password_regex, password):
return False, password_regex_description
except re.error:
msg = ("Unable to validate password due to invalid "
"complexity criteria ('password_regex')")
raise Exception(msg)
except Exception:
raise Exception("Password validation failed")
return True, ""
def extract_openstack_password_rules_from_file(
rules_file, section="security_compliance"):
try:
config = ConfigParser.RawConfigParser()
parsed_config = config.read(rules_file)
if not parsed_config:
msg = ("Cannot parse rules file: %" % rules_file)
raise Exception(msg)
if not config.has_section(section):
msg = ("Required section '%s' not found in rules file" % section)
raise Exception(msg)
rules = config.items(section)
if not rules:
msg = ("section '%s' contains no configuration options" % section)
raise Exception(msg)
return dict(rules)
except Exception:
raise Exception("Failed to extract password rules from file")
def get_optional(conf, section, key):
if conf.has_option(section, key):
return conf.get(section, key)
return None
def get_service(conf, section, key):
if key in EXPECTED_SERVICE_NAME_AND_TYPE:
if conf.has_option(section, key):
value = conf.get(section, key)
if value != EXPECTED_SERVICE_NAME_AND_TYPE[key]:
raise ValidateFail("Unsupported %s: %s " % (key, value))
else:
value = EXPECTED_SERVICE_NAME_AND_TYPE[key]
return value
else:
return conf.get(section, key)

1139
configutilities/configutilities/configutilities/common/validator.py Executable file
View File

File diff suppressed because it is too large Load Diff

37
configutilities/configutilities/configutilities/common/wrs_ico.py Executable file
View File

@@ -0,0 +1,37 @@
# ----------------------------------------------------------------------
# This file was generated by img2py.py
#
#
# Copyright (c) 2015-2016 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
#
# Stylized red Wind River 'W' icon
#
from wx.lib.embeddedimage import PyEmbeddedImage
favicon = PyEmbeddedImage(
"iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABHNCSVQICAgIfAhkiAAAA99J"
"REFUWIXtls1PE2sUxn/z0Q7U0g/AXmNAUBGNxsSYGnWjQQ0YNAISY1y4kLh16T/g1qV/gwtJ"
"JFFATbBCmuBXrhiMitGk0hpBJaQtM4Vpp+3cxXAHSltub1zggpPM4p155pznfd5zznuEv8Fk"
"A03cyOCbBDYJ/BEE5EpqUPiP72YFmHI42bV/f0ng6uDG3BzZubmSTuW6OhzbtmEaBuloFDOd"
"Lo3buhVHIGDhpqcxMxnLfzoaXYlnmiAICJKEoCjWWpKI37tH7MYNyOUKvQoCDbdvU3ftGnlN"
"Y7qvj4VQqGiXgizTeOcOtVeukHr1isjly+QSCYvYdF9fEVvR5aLu6lX8ly4B4OnoQGlqIh2J"
"FOCcTU34urqQa2uhthZ/by/qs2cW8VWmtLTgPXcOyetlcWLCDg4gL4RCJSXLfPtGzcmTyIEA"
"SnMz7hMn0CMRe3cm4GlvR9m1y/7H096Os7mZzNevK6ICnrNncTY2kksmSQ4PF+SCKCwv1j76"
"p0+kXr9eRol4z59HdDptx5LLhf/iRRBXCknZuRPPmTMFOSR7PPi6ugBYfPuWpcnJgiMqW4Z5"
"XScxOGjL6T5+HGXPHszlXbkOH2bL0aOQz2PMzlo4UcTX04Pkctm7dx05wpZgEIDko0dkVbUg"
"TlkCAqCOjpKJxQBwbN+O59Qp+5uvuxvJ5yMTizF765Z9ru5jx6g+dMiSWRDwdXcjut1kf/1i"
"YWSkKM66jSgTiaCGw/ba29mJWFWFo6EBb2cnAOrYGPN376K9eAGA5Pfbkjt37MDb0QGA9vIl"
"+tRUUYWsSyCfy5EcHMQ0DABcwSBV+/ZR09aG0tpKXteJDwyQVVXi/f2Y2axN1BEIUHP6NMru"
"3WCaJIeGyJfoEesSEABtfBz982cA5Pp6/D09+Ht7ESSJpclJUs+fIwALIyPoU1MAVO3di+/C"
"BUsJUSQTi6GOjZXslvJ6BACMmRnUp0+pPnAAgPrr1xHdbgASDx9izM9b3XJmhsSDB1QfPIjg"
"cPDXzZs4AgEA1HCYzJoeUpECYGVyYmiIfCoFWMkoeTwYP36QHB4uxA0MYPz8aanQ2ork82Ea"
"hiX/2i5aKQEBWHzzhqV37wrea+Ew+sePtqwCsPT+vdUJV1n6yxe08fGyl1VF13E2Hif5+LG9"
"Ng2D+P375JeT81/LGwbx/n7yum6/WwiFML5/L+u74nkg+eSJfSMuffiAFg4XXzpY5704MWER"
"0jS79ZYzodKxXFAUatrakP1+MtGoVfdm6V9dwSBVLS3kUinU0VHymvb7BKB4TvhdHFRQhqut"
"kqnn/+DgD5gJNwlsEthwAv8AApOBr7T8BuQAAAAASUVORK5CYII=")

100
configutilities/configutilities/configutilities/config_validator.py Executable file
View File

@@ -0,0 +1,100 @@
"""
Copyright (c) 2015-2016 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
import sys
import os
import ConfigParser
from common.validator import validate
from common.configobjects import DEFAULT_CONFIG, REGION_CONFIG
from common.exceptions import ConfigFail, ValidateFail
def parse_config(config_file):
"""Parse system config file"""
config = ConfigParser.RawConfigParser()
try:
config.read(config_file)
except Exception as e:
raise ConfigFail("Error parsing system config file: %s" % e.message)
return config
def show_help():
print ("Usage: %s\n"
"Perform validation of a given configuration file\n\n"
"--system-config <name> Validate a system configuration file\n"
"--region-config <name> Validate a region configuration file\n"
% sys.argv[0])
exit(1)
def main():
config_file = None
system_config = False
region_config = False
arg = 1
while arg < len(sys.argv):
if sys.argv[arg] == "--system-config":
arg += 1
if arg < len(sys.argv):
config_file = sys.argv[arg]
else:
print "--system-config requires the filename of the config " \
"file"
exit(1)
system_config = True
elif sys.argv[arg] == "--region-config":
arg += 1
if arg < len(sys.argv):
config_file = sys.argv[arg]
else:
print "--region-config requires the filename of the config " \
"file"
exit(1)
region_config = True
elif sys.argv[arg] in ["--help", "-h", "-?"]:
show_help()
else:
print "Invalid option."
show_help()
arg += 1
if [system_config, region_config].count(True) != 1:
print "Invalid combination of options selected"
show_help()
if system_config:
config_type = DEFAULT_CONFIG
else:
config_type = REGION_CONFIG
if not os.path.isfile(config_file):
print("Config file %s does not exist" % config_file)
exit(1)
# Parse the system config file
print "Parsing configuration file... ",
system_config = parse_config(config_file)
print "DONE"
# Validate the system config file
print "Validating configuration file... ",
try:
# we use the presence of tsconfig to determine if we are onboard or
# not since it will not be available in the offboard case
offboard = False
try:
from tsconfig.tsconfig import SW_VERSION # noqa: F401
except ImportError:
offboard = True
validate(system_config, config_type, None, offboard)
except ConfigParser.Error as e:
print("Error parsing configuration file %s: %s" % (config_file, e))
except (ConfigFail, ValidateFail) as e:
print("\nValidation failed: %s" % e)
print "DONE"

1457
configutilities/configutilities/configutilities/configfiletool.py Executable file
View File

File diff suppressed because it is too large Load Diff

114
configutilities/configutilities/configutilities/configgui.py Executable file
View File

@@ -0,0 +1,114 @@
"""
Copyright (c) 2015-2017 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
import wx
from common.guicomponents import set_icons
from common.validator import TiS_VERSION
import configfiletool
import hostfiletool
TEXT_WIDTH = 560
BTN_SIZE = (200, -1)
class WelcomeScreen(wx.Frame):
def __init__(self, *args, **kwargs):
super(WelcomeScreen, self).__init__(*args, **kwargs)
page = Content(self)
set_icons(self)
size = page.main_sizer.Fit(self)
self.SetMinSize(size)
self.Layout()
class Content(wx.Panel):
def __init__(self, *args, **kwargs):
super(Content, self).__init__(*args, **kwargs)
self.title = wx.StaticText(
self, -1,
'Titanium Cloud Configuration Utility')
self.title.SetFont(wx.Font(18, wx.SWISS, wx.NORMAL, wx.BOLD))
# Set up controls for the main page
self.description = wx.StaticText(
self, -1,
' Welcome, The following tools are available for use:')
self.config_desc = wx.StaticText(
self, -1,
"The Titanium Cloud configuration file wizard allows users to "
"create the configuration INI file which is used during the "
"installation process")
self.config_desc.Wrap(TEXT_WIDTH / 2)
self.hosts_desc = wx.StaticText(
self, -1,
"The Titanium Cloud host file tool allows users to create an XML "
"file specifying hosts to be provisioned as part of the Titanium "
"Cloud cloud deployment.")
self.hosts_desc.Wrap(TEXT_WIDTH / 2)
self.config_wiz_btn = wx.Button(
self, -1, "Launch Config File Wizard", size=BTN_SIZE)
self.Bind(wx.EVT_BUTTON, self.launch_config_wiz, self.config_wiz_btn)
self.host_file_tool_btn = wx.Button(
self, -1, "Launch Host File Tool", size=BTN_SIZE)
self.Bind(wx.EVT_BUTTON, self.launch_host_wiz, self.host_file_tool_btn)
self.box1 = wx.StaticBox(self)
self.box2 = wx.StaticBox(self)
# Do layout of controls
self.main_sizer = wx.BoxSizer(wx.VERTICAL)
self.tool1Sizer = wx.StaticBoxSizer(self.box1, wx.HORIZONTAL)
self.tool2Sizer = wx.StaticBoxSizer(self.box2, wx.HORIZONTAL)
self.main_sizer.AddSpacer(10)
self.main_sizer.Add(self.title, flag=wx.ALIGN_CENTER)
self.main_sizer.AddSpacer(10)
self.main_sizer.Add(self.description)
self.main_sizer.AddSpacer(5)
self.main_sizer.Add(self.tool1Sizer, proportion=1, flag=wx.EXPAND)
self.main_sizer.Add(self.tool2Sizer, proportion=1, flag=wx.EXPAND)
self.main_sizer.AddSpacer(5)
self.tool1Sizer.Add(self.config_desc, flag=wx.ALIGN_CENTER)
self.tool1Sizer.AddSpacer(10)
self.tool1Sizer.Add(self.config_wiz_btn, flag=wx.ALIGN_CENTER)
self.tool2Sizer.Add(self.hosts_desc, flag=wx.ALIGN_CENTER)
self.tool2Sizer.AddSpacer(10)
self.tool2Sizer.Add(self.host_file_tool_btn, flag=wx.ALIGN_CENTER)
self.SetSizer(self.main_sizer)
self.Layout()
def launch_config_wiz(self, event):
conf_wizard = configfiletool.ConfigWizard()
conf_wizard.run()
conf_wizard.Destroy()
def launch_host_wiz(self, event):
hostfiletool.HostGUI()
def main():
app = wx.App(0) # Start the application
gui = WelcomeScreen(None, title="Titanium Cloud Configuration Utility v"
+ TiS_VERSION)
gui.Show()
app.MainLoop()
app.Destroy()
if __name__ == '__main__':
main()

510
configutilities/configutilities/configutilities/hostfiletool.py Executable file
View File

@@ -0,0 +1,510 @@
"""
Copyright (c) 2015-2017 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
from collections import OrderedDict
import netaddr
import xml.etree.ElementTree as ET
import wx
from common import utils, exceptions
from common.guicomponents import Field, TYPES, prepare_fields, on_change, \
set_icons, handle_sub_show
from common.configobjects import HOST_XML_ATTRIBUTES
from common.validator import TiS_VERSION
PAGE_SIZE = (200, 200)
WINDOW_SIZE = (570, 700)
CB_TRUE = True
CB_FALSE = False
PADDING = 10
IMPORT_ID = 100
EXPORT_ID = 101
INTERNAL_ID = 105
EXTERNAL_ID = 106
filedir = ""
filename = ""
# Globals
BULK_ADDING = False
class HostPage(wx.Panel):
def __init__(self, parent):
wx.Panel.__init__(self, parent=parent)
self.parent = parent
self.sizer = wx.BoxSizer(wx.VERTICAL)
self.SetSizer(self.sizer)
self.fieldgroup = []
self.fieldgroup.append(OrderedDict())
self.fieldgroup.append(OrderedDict())
self.fieldgroup.append(OrderedDict())
self.fields_sizer1 = wx.GridBagSizer(vgap=10, hgap=10)
self.fields_sizer2 = wx.GridBagSizer(vgap=10, hgap=10)
self.fields_sizer3 = wx.GridBagSizer(vgap=10, hgap=10)
# Basic Fields
self.fieldgroup[0]['personality'] = Field(
text="Personality",
type=TYPES.choice,
choices=['compute', 'controller', 'storage'],
initial='compute'
)
self.fieldgroup[0]['hostname'] = Field(
text="Hostname",
type=TYPES.string,
initial=parent.get_next_hostname()
)
self.fieldgroup[0]['mgmt_mac'] = Field(
text="Management MAC Address",
type=TYPES.string,
initial=""
)
self.fieldgroup[0]['mgmt_ip'] = Field(
text="Management IP Address",
type=TYPES.string,
initial=""
)
self.fieldgroup[0]['location'] = Field(
text="Location",
type=TYPES.string,
initial=""
)
# Board Management
self.fieldgroup[1]['uses_bm'] = Field(
text="This host uses Board Management",
type=TYPES.checkbox,
initial="",
shows=['bm_ip', 'bm_username',
'bm_password', 'power_on'],
transient=True
)
self.fieldgroup[1]['bm_ip'] = Field(
text="Board Management IP Address",
type=TYPES.string,
initial=""
)
self.fieldgroup[1]['bm_username'] = Field(
text="Board Management username",
type=TYPES.string,
initial=""
)
self.fieldgroup[1]['bm_password'] = Field(
text="Board Management password",
type=TYPES.string,
initial=""
)
self.fieldgroup[1]['power_on'] = Field(
text="Power on host",
type=TYPES.checkbox,
initial="N",
transient=True
)
# Installation Parameters
self.fieldgroup[2]['boot_device'] = Field(
text="Boot Device",
type=TYPES.string,
initial=""
)
self.fieldgroup[2]['rootfs_device'] = Field(
text="Rootfs Device",
type=TYPES.string,
initial=""
)
self.fieldgroup[2]['install_output'] = Field(
text="Installation Output",
type=TYPES.choice,
choices=['text', 'graphical'],
initial="text"
)
self.fieldgroup[2]['console'] = Field(
text="Console",
type=TYPES.string,
initial=""
)
prepare_fields(self, self.fieldgroup[0], self.fields_sizer1,
self.on_change)
prepare_fields(self, self.fieldgroup[1], self.fields_sizer2,
self.on_change)
prepare_fields(self, self.fieldgroup[2], self.fields_sizer3,
self.on_change)
# Bind button handlers
self.Bind(wx.EVT_CHOICE, self.on_personality,
self.fieldgroup[0]['personality'].input)
self.Bind(wx.EVT_TEXT, self.on_hostname,
self.fieldgroup[0]['hostname'].input)
# Control Buttons
self.button_sizer = wx.BoxSizer(orient=wx.HORIZONTAL)
self.add = wx.Button(self, -1, "Add a New Host")
self.Bind(wx.EVT_BUTTON, self.on_add, self.add)
self.remove = wx.Button(self, -1, "Remove this Host")
self.Bind(wx.EVT_BUTTON, self.on_remove, self.remove)
self.button_sizer.Add(self.add)
self.button_sizer.Add(self.remove)
# Add fields and spacers
self.sizer.Add(self.fields_sizer1)
self.sizer.AddWindow(wx.StaticLine(self, -1), 0, wx.EXPAND | wx.ALL,
PADDING)
self.sizer.Add(self.fields_sizer2)
self.sizer.AddWindow(wx.StaticLine(self, -1), 0, wx.EXPAND | wx.ALL,
PADDING)
self.sizer.Add(self.fields_sizer3)
self.sizer.AddStretchSpacer()
self.sizer.AddWindow(wx.StaticLine(self, -1), 0, wx.EXPAND | wx.ALL,
PADDING)
self.sizer.Add(self.button_sizer, border=10, flag=wx.CENTER)
def on_hostname(self, event, string=None):
"""Update the List entry text to match the new hostname
"""
string = string or event.GetString()
index = self.parent.GetSelection()
self.parent.SetPageText(index, string)
self.parent.parent.Layout()
def on_personality(self, event, string=None):
"""Remove hostname field if it's a storage or controller
"""
string = string or event.GetString()
index = self.parent.GetSelection()
if string == 'compute':
self.fieldgroup[0]['hostname'].show(True)
self.parent.SetPageText(index,
self.fieldgroup[0]['hostname'].get_value())
return
elif string == 'controller':
self.fieldgroup[0]['hostname'].show(False)
elif string == 'storage':
self.fieldgroup[0]['hostname'].show(False)
self.parent.SetPageText(index, string)
self.parent.Layout()
def on_add(self, event):
try:
self.validate()
except Exception as ex:
wx.LogError("Error on page: " + ex.message)
return
self.parent.new_page()
def on_remove(self, event):
if self.parent.GetPageCount() is 1:
wx.LogError("Must leave at least one host")
return
index = self.parent.GetSelection()
self.parent.DeletePage(index)
def to_xml(self):
"""Create the XML for this host
"""
self.validate()
attrs = ""
# Generic handling
for fgroup in self.fieldgroup:
for name, field in fgroup.items():
if field.transient or not field.get_value():
continue
attrs += "\t\t<" + name + ">" + \
field.get_value() + "</" + name + ">\n"
# Special Fields
if self.fieldgroup[1]['power_on'].get_value() is 'Y':
attrs += "\t\t<power_on/>\n"
if self.fieldgroup[1]['uses_bm'].get_value() is 'Y':
attrs += "\t\t<bm_type>bmc</bm_type>\n"
return "\t<host>\n" + attrs + "\t</host>\n"
def validate(self):
if self.fieldgroup[0]['personality'].get_value() == "compute" and not \
utils.is_valid_hostname(
self.fieldgroup[0]['hostname'].get_value()):
raise exceptions.ValidateFail(
"Hostname %s is not valid" %
self.fieldgroup[0]['hostname'].get_value())
if not utils.is_valid_mac(self.fieldgroup[0]['mgmt_mac'].get_value()):
raise exceptions.ValidateFail(
"Management MAC address %s is not valid" %
self.fieldgroup[0]['mgmt_mac'].get_value())
ip = self.fieldgroup[0]['mgmt_ip'].get_value()
if ip:
try:
netaddr.IPAddress(ip)
except Exception:
raise exceptions.ValidateFail(
"Management IP address %s is not valid" % ip)
if self.fieldgroup[1]['uses_bm'].get_value() == 'Y':
ip = self.fieldgroup[1]['bm_ip'].get_value()
if ip:
try:
netaddr.IPAddress(ip)
except Exception:
raise exceptions.ValidateFail(
"Board Management IP address %s is not valid" % ip)
else:
raise exceptions.ValidateFail(
"Board Management IP is not specified. "
"External Board Management Network requires Board "
"Management IP address.")
def on_change(self, event):
on_change(self, self.fieldgroup[1], event)
def set_field(self, name, value):
for fgroup in self.fieldgroup:
for fname, field in fgroup.items():
if fname == name:
field.set_value(value)
class HostBook(wx.Listbook):
def __init__(self, parent):
wx.Listbook.__init__(self, parent, style=wx.BK_DEFAULT)
self.parent = parent
self.Layout()
# Add a starting host
self.new_page()
self.Bind(wx.EVT_LISTBOOK_PAGE_CHANGED, self.on_changed)
self.Bind(wx.EVT_LISTBOOK_PAGE_CHANGING, self.on_changing)
def on_changed(self, event):
event.Skip()
def on_changing(self, event):
# Trigger page validation before leaving
if BULK_ADDING:
event.Skip()
return
index = self.GetSelection()
try:
if index != -1:
self.GetPage(index).validate()
except Exception as ex:
wx.LogError("Error on page: " + ex.message)
event.Veto()
return
event.Skip()
def new_page(self, hostname=None):
new_page = HostPage(self)
self.AddPage(new_page, hostname or self.get_next_hostname())
self.SetSelection(self.GetPageCount() - 1)
return new_page
def get_next_hostname(self, suggest=None):
prefix = "compute-"
new_suggest = suggest or 0
for existing in range(self.GetPageCount()):
if prefix + str(new_suggest) in self.GetPageText(existing):
new_suggest = self.get_next_hostname(suggest=new_suggest + 1)
if suggest:
prefix = ""
return prefix + str(new_suggest)
def to_xml(self):
"""Create the complete XML and allow user to save
"""
xml = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n" \
"<hosts version=\"" + TiS_VERSION + "\">\n"
for index in range(self.GetPageCount()):
try:
xml += self.GetPage(index).to_xml()
except Exception as ex:
wx.LogError("Error on page number %s: %s" %
(index + 1, ex.message))
return
xml += "</hosts>"
writer = wx.FileDialog(self,
message="Save Host XML File",
defaultDir=filedir or "",
defaultFile=filename or "TiS_hosts.xml",
wildcard="XML file (*.xml)|*.xml",
style=wx.FD_SAVE,
)
if writer.ShowModal() == wx.ID_CANCEL:
return
# Write the XML file to disk
try:
with open(writer.GetPath(), "wb") as f:
f.write(xml.encode('utf-8'))
except IOError:
wx.LogError("Error writing hosts xml file '%s'." %
writer.GetPath())
class HostGUI(wx.Frame):
def __init__(self):
wx.Frame.__init__(self, None, wx.ID_ANY,
"Titanium Cloud Host File Creator v" + TiS_VERSION,
size=WINDOW_SIZE)
self.panel = wx.Panel(self)
self.sizer = wx.BoxSizer(wx.VERTICAL)
self.book = HostBook(self.panel)
self.sizer.Add(self.book, 1, wx.ALL | wx.EXPAND, 5)
self.panel.SetSizer(self.sizer)
set_icons(self)
menu_bar = wx.MenuBar()
# File
file_menu = wx.Menu()
import_item = wx.MenuItem(file_menu, IMPORT_ID, '&Import')
file_menu.AppendItem(import_item)
export_item = wx.MenuItem(file_menu, EXPORT_ID, '&Export')
file_menu.AppendItem(export_item)
menu_bar.Append(file_menu, '&File')
self.Bind(wx.EVT_MENU, self.on_import, id=IMPORT_ID)
self.Bind(wx.EVT_MENU, self.on_export, id=EXPORT_ID)
self.SetMenuBar(menu_bar)
self.Layout()
self.SetMinSize(WINDOW_SIZE)
self.Show()
def on_import(self, e):
global BULK_ADDING
try:
BULK_ADDING = True
msg = ""
reader = wx.FileDialog(self,
"Import Existing Titanium Cloud Host File",
"", "", "XML file (*.xml)|*.xml",
wx.FD_OPEN | wx.FD_FILE_MUST_EXIST)
if reader.ShowModal() == wx.ID_CANCEL:
return
# Read in the config file
try:
with open(reader.GetPath(), 'rb') as f:
contents = f.read()
root = ET.fromstring(contents)
except Exception as ex:
wx.LogError("Cannot parse host file, Error: %s." % ex)
return
# Check version of host file
if root.get('version', "") != TiS_VERSION:
msg += "Warning: This file was created using tools for a " \
"different version of Titanium Cloud than this tool " \
"was designed for (" + TiS_VERSION + ")"
for idx, xmlhost in enumerate(root.findall('host')):
hostname = None
name_elem = xmlhost.find('hostname')
if name_elem is not None:
hostname = name_elem.text
new_host = self.book.new_page()
self.book.GetSelection()
try:
for attr in HOST_XML_ATTRIBUTES:
elem = xmlhost.find(attr)
if elem is not None and elem.text:
# Enable and display bm section if used
if attr == 'bm_type' and elem.text:
new_host.set_field("uses_bm", "Y")
handle_sub_show(
new_host.fieldgroup[1],
new_host.fieldgroup[1]['uses_bm'].shows,
True)
new_host.Layout()
# Basic field setting
new_host.set_field(attr, elem.text)
# Additional functionality for special fields
if attr == 'personality':
# Update hostname visibility and page title
new_host.on_personality(None, elem.text)
# Special handling for presence of power_on element
if attr == 'power_on' and elem is not None:
new_host.set_field(attr, "Y")
new_host.validate()
except Exception as ex:
if msg:
msg += "\n"
msg += "Warning: Added host %s has a validation error, " \
"reason: %s" % \
(hostname or ("with index " + str(idx)),
ex.message)
# No longer delete hosts with validation errors,
# The user can fix them up before exporting
# self.book.DeletePage(new_index)
if msg:
wx.LogWarning(msg)
finally:
BULK_ADDING = False
self.Layout()
def on_export(self, e):
# Do a validation of current page first
index = self.book.GetSelection()
try:
if index != -1:
self.book.GetPage(index).validate()
except Exception as ex:
wx.LogError("Error on page: " + ex.message)
return
# Check for hostname conflicts
hostnames = []
for existing in range(self.book.GetPageCount()):
hostname = self.book.GetPage(
existing).fieldgroup[0]['hostname'].get_value()
if hostname in hostnames:
wx.LogError("Cannot export, duplicate hostname '%s'" %
hostname)
return
# Ignore multiple None hostnames
elif hostname:
hostnames.append(hostname)
self.book.to_xml()
def main():
app = wx.App(0) # Start the application
HostGUI()
app.MainLoop()
if __name__ == '__main__':
main()

29
configutilities/configutilities/configutilities/setup.py Executable file
View File

@@ -0,0 +1,29 @@
"""
Copyright (c) 2016-2017 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
from setuptools import setup, find_packages
setup(
name='wrs-configutility',
description='Titanium Cloud Configuration Utility',
version='3.0.0',
license='Apache-2.0',
platforms=['any'],
provides=['configutilities'],
packages=find_packages(),
install_requires=['netaddr>=0.7.14', 'six'],
package_data={},
include_package_data=False,
entry_points={
'gui_scripts': [
'config_gui = configutilities.configgui:main',
],
'console_scripts': [
'config_validator = configutilities.config_validator:main'
],
}
)

BIN
configutilities/configutilities/favicon.ico Executable file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

26
configutilities/configutilities/setup.py Executable file
View File

@@ -0,0 +1,26 @@
"""
Copyright (c) 2016 Wind River Systems, Inc.
SPDX-License-Identifier: Apache-2.0
"""
from setuptools import setup, find_packages
setup(
name='configutilities',
description='Configuration File Validator',
version='3.0.0',
license='Apache-2.0',
platforms=['any'],
provides=['configutilities'],
packages=find_packages(),
install_requires=['netaddr>=0.7.14'],
package_data={},
include_package_data=False,
entry_points={
'console_scripts': [
'config_validator = configutilities.config_validator:main',
],
}
)

22
configutilities/configutilities/tox.ini Normal file
View File

@@ -0,0 +1,22 @@
# Tox (http://tox.testrun.org/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.
[tox]
envlist = flake8
# Tox does not work if the path to the workdir is too long, so move it to /tmp
toxworkdir = /tmp/{env:USER}_ccutiltox
wrsdir = {toxinidir}/../../../../../../../../..
[testenv]
whitelist_externals = find
install_command = pip install --no-cache-dir {opts} {packages}
[testenv:flake8]
basepython = python2.7
deps = flake8
commands = flake8 {posargs}
[flake8]
ignore = W503

6
controllerconfig/.gitignore vendored Normal file
View File

@@ -0,0 +1,6 @@
!.distro
.distro/centos7/rpmbuild/RPMS
.distro/centos7/rpmbuild/SRPMS
.distro/centos7/rpmbuild/BUILD
.distro/centos7/rpmbuild/BUILDROOT
.distro/centos7/rpmbuild/SOURCES/controllerconfig*tar.gz

13
controllerconfig/PKG-INFO Normal file
View File

@@ -0,0 +1,13 @@
Metadata-Version: 1.1
Name: controllerconfig
Version: 1.0
Summary: Controller Node Configuration
Home-page:
Author: Windriver
Author-email: info@windriver.com
License: Apache-2.0
Description: Controller node configuration
Platform: UNKNOWN

2
controllerconfig/centos/build_srpm.data Executable file
View File

@@ -0,0 +1,2 @@
SRC_DIR="controllerconfig"
TIS_PATCH_VER=140

86
controllerconfig/centos/controllerconfig.spec Normal file
View File

@@ -0,0 +1,86 @@
Summary: Controller node configuration
Name: controllerconfig
Version: 1.0
Release: %{tis_patch_ver}%{?_tis_dist}
License: Apache-2.0
Group: base
Packager: Wind River <info@windriver.com>
URL: unknown
Source0: %{name}-%{version}.tar.gz
BuildRequires: python-setuptools
Requires: systemd
Requires: python-netaddr
Requires: python-keyring
Requires: python-six
Requires: python-iso8601
Requires: psmisc
Requires: lshell
Requires: python-pyudev
Requires: python-netifaces
%description
Controller node configuration
%define local_dir /usr/
%define local_bindir %{local_dir}/bin/
%define local_etc_initd /etc/init.d/
%define local_goenabledd /etc/goenabled.d/
%define local_etc_upgraded /etc/upgrade.d/
%define local_etc_systemd /etc/systemd/system/
%define pythonroot /usr/lib64/python2.7/site-packages
%define debug_package %{nil}
%prep
%setup
%build
%{__python} setup.py build
# TODO: NO_GLOBAL_PY_DELETE (see python-byte-compile.bbclass), put in macro/script
%install
%{__python} setup.py install --root=$RPM_BUILD_ROOT \
--install-lib=%{pythonroot} \
--prefix=/usr \
--install-data=/usr/share \
--single-version-externally-managed
install -d -m 755 %{buildroot}%{local_bindir}
install -p -D -m 700 scripts/keyringstaging %{buildroot}%{local_bindir}/keyringstaging
install -p -D -m 700 scripts/openstack_update_admin_password %{buildroot}%{local_bindir}/openstack_update_admin_password
install -p -D -m 700 scripts/install_clone.py %{buildroot}%{local_bindir}/install_clone
install -p -D -m 700 scripts/finish_install_clone.sh %{buildroot}%{local_bindir}/finish_install_clone.sh
install -d -m 755 %{buildroot}%{local_goenabledd}
install -p -D -m 700 scripts/config_goenabled_check.sh %{buildroot}%{local_goenabledd}/config_goenabled_check.sh
install -d -m 755 %{buildroot}%{local_etc_initd}
install -p -D -m 755 scripts/controller_config %{buildroot}%{local_etc_initd}/controller_config
# Install Upgrade scripts
install -d -m 755 %{buildroot}%{local_etc_upgraded}
install -p -D -m 755 upgrade-scripts/* %{buildroot}%{local_etc_upgraded}/
install -d -m 755 %{buildroot}%{local_etc_systemd}
install -p -D -m 664 scripts/controllerconfig.service %{buildroot}%{local_etc_systemd}/controllerconfig.service
#install -p -D -m 664 scripts/config.service %{buildroot}%{local_etc_systemd}/config.service
%post
systemctl enable controllerconfig.service
%clean
rm -rf $RPM_BUILD_ROOT
%files
%defattr(-,root,root,-)
%doc LICENSE
%{local_bindir}/*
%dir %{pythonroot}/%{name}
%{pythonroot}/%{name}/*
%dir %{pythonroot}/%{name}-%{version}.0-py2.7.egg-info
%{pythonroot}/%{name}-%{version}.0-py2.7.egg-info/*
%{local_goenabledd}/*
%{local_etc_initd}/*
%dir %{local_etc_upgraded}
%{local_etc_upgraded}/*
%{local_etc_systemd}/*

7
controllerconfig/controllerconfig/.coveragerc Normal file
View File

@@ -0,0 +1,7 @@
[run]
branch = True
source = controllerconfig
omit = controllerconfig/tests/*
[report]
ignore_errors = True

5
controllerconfig/controllerconfig/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
*.pyc
.coverage
.testrepository
cover

8
controllerconfig/controllerconfig/.testr.conf Normal file
View File

@@ -0,0 +1,8 @@
[DEFAULT]
test_command=OS_STDOUT_CAPTURE=1 \
OS_STDERR_CAPTURE=1 \
OS_TEST_TIMEOUT=60 \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./controllerconfig/tests} $LISTOPT $IDOPTION
test_id_option=--load-list $IDFILE
test_list_option=--list

202
controllerconfig/controllerconfig/LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

5
controllerconfig/controllerconfig/controllerconfig/__init__.py Normal file
View File

@@ -0,0 +1,5 @@
#
# Copyright (c) 2015 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#

1895
controllerconfig/controllerconfig/controllerconfig/backup_restore.py Normal file
View File

File diff suppressed because it is too large Load Diff

717
controllerconfig/controllerconfig/controllerconfig/clone.py Normal file
View File

@@ -0,0 +1,717 @@
#
# Copyright (c) 2017 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
"""
Clone a Configured System and Install the image on another
identical hardware or the same hardware.
"""
import os
import re
import glob
import time
import shutil
import netaddr
import tempfile
import fileinput
import subprocess
from common import constants
from sysinv.common import constants as si_const
import sysinv_api
import tsconfig.tsconfig as tsconfig
from common import log
from common.exceptions import CloneFail, BackupFail
import utils
import backup_restore
DEBUG = False
LOG = log.get_logger(__name__)
DEVNULL = open(os.devnull, 'w')
CLONE_ARCHIVE_DIR = "clone-archive"
CLONE_ISO_INI = ".cloneiso.ini"
NAME = "name"
INSTALLED = "installed_at"
RESULT = "result"
IN_PROGRESS = "in-progress"
FAIL = "failed"
OK = "ok"
def clone_status():
""" Check status of last install-clone. """
INI_FILE1 = os.path.join("/", CLONE_ARCHIVE_DIR, CLONE_ISO_INI)
INI_FILE2 = os.path.join(tsconfig.PLATFORM_CONF_PATH, CLONE_ISO_INI)
name = "unknown"
result = "unknown"
installed_at = "unknown time"
for ini_file in [INI_FILE1, INI_FILE2]:
if os.path.exists(ini_file):
with open(ini_file) as f:
s = f.read()
for line in s.split("\n"):
if line.startswith(NAME):
name = line.split("=")[1].strip()
elif line.startswith(RESULT):
result = line.split("=")[1].strip()
elif line.startswith(INSTALLED):
installed_at = line.split("=")[1].strip()
break # one file was found, skip the other file
if result != "unknown":
if result == OK:
print("\nInstallation of cloned image [{}] was successful at {}\n"
.format(name, installed_at))
elif result == FAIL:
print("\nInstallation of cloned image [{}] failed at {}\n"
.format(name, installed_at))
else:
print("\ninstall-clone is in progress.\n")
else:
print("\nCloned image is not installed on this node.\n")
def check_size(archive_dir):
""" Check if there is enough space to create iso. """
overhead_bytes = 1024 ** 3 # extra GB for staging directory
# Size of the cloned iso is directly proportional to the
# installed package repository (note that patches are a part of
# the system archive size below).
# 1G overhead size added (above) will accomodate the temporary
# workspace (updating system archive etc) needed to create the iso.
feed_dir = os.path.join('/www', 'pages', 'feed',
'rel-' + tsconfig.SW_VERSION)
overhead_bytes += backup_restore.backup_std_dir_size(feed_dir)
cinder_config = False
backend_services = sysinv_api.get_storage_backend_services()
for services in backend_services.values():
if (services.find(si_const.SB_SVC_CINDER) != -1):
cinder_config = True
break
clone_size = (
overhead_bytes +
backup_restore.backup_etc_size() +
backup_restore.backup_config_size(tsconfig.CONFIG_PATH) +
backup_restore.backup_puppet_data_size(constants.HIERADATA_PERMDIR) +
backup_restore.backup_keyring_size(backup_restore.keyring_permdir) +
backup_restore.backup_ldap_size() +
backup_restore.backup_postgres_size(cinder_config) +
backup_restore.backup_ceilometer_size(
backup_restore.ceilometer_permdir) +
backup_restore.backup_std_dir_size(backup_restore.glance_permdir) +
backup_restore.backup_std_dir_size(backup_restore.home_permdir) +
backup_restore.backup_std_dir_size(backup_restore.patching_permdir) +
backup_restore.backup_std_dir_size(
backup_restore.patching_repo_permdir) +
backup_restore.backup_std_dir_size(backup_restore.extension_permdir) +
backup_restore.backup_std_dir_size(
backup_restore.patch_vault_permdir) +
backup_restore.backup_cinder_size(backup_restore.cinder_permdir))
archive_dir_free_space = \
utils.filesystem_get_free_space(archive_dir)
if clone_size > archive_dir_free_space:
print ("\nArchive directory (%s) does not have enough free "
"space (%s), estimated size to create image is %s." %
(archive_dir,
utils.print_bytes(archive_dir_free_space),
utils.print_bytes(clone_size)))
raise CloneFail("Not enough free space.\n")
def update_bootloader_default(bl_file, host):
""" Update bootloader files for cloned image """
if not os.path.exists(bl_file):
LOG.error("{} does not exist".format(bl_file))
raise CloneFail("{} does not exist".format(os.path.basename(bl_file)))
# Tags should be in sync with common-bsp/files/centos.syslinux.cfg
# and common-bsp/files/grub.cfg
STANDARD_STANDARD = '0'
STANDARD_EXTENDED = 'S0'
AIO_STANDARD = '2'
AIO_EXTENDED = 'S2'
AIO_LL_STANDARD = '4'
AIO_LL_EXTENDED = 'S4'
if "grub.cfg" in bl_file:
STANDARD_STANDARD = 'standard>serial>' + \
si_const.SYSTEM_SECURITY_PROFILE_STANDARD
STANDARD_EXTENDED = 'standard>serial>' + \
si_const.SYSTEM_SECURITY_PROFILE_EXTENDED
AIO_STANDARD = 'aio>serial>' + \
si_const.SYSTEM_SECURITY_PROFILE_STANDARD
AIO_EXTENDED = 'aio>serial>' + \
si_const.SYSTEM_SECURITY_PROFILE_EXTENDED
AIO_LL_STANDARD = 'aio-lowlat>serial>' + \
si_const.SYSTEM_SECURITY_PROFILE_STANDARD
AIO_LL_EXTENDED = 'aio-lowlat>serial>' + \
si_const.SYSTEM_SECURITY_PROFILE_EXTENDED
SUBMENUITEM_TBOOT = 'tboot'
SUBMENUITEM_SECUREBOOT = 'secureboot'
timeout_line = None
default_line = None
default_label_num = STANDARD_STANDARD
if utils.get_system_type() == si_const.TIS_AIO_BUILD:
if si_const.LOWLATENCY in tsconfig.subfunctions:
default_label_num = AIO_LL_STANDARD
else:
default_label_num = AIO_STANDARD
if (tsconfig.security_profile ==
si_const.SYSTEM_SECURITY_PROFILE_EXTENDED):
default_label_num = STANDARD_EXTENDED
if utils.get_system_type() == si_const.TIS_AIO_BUILD:
if si_const.LOWLATENCY in tsconfig.subfunctions:
default_label_num = AIO_LL_EXTENDED
else:
default_label_num = AIO_EXTENDED
if "grub.cfg" in bl_file:
if host.tboot is not None:
if host.tboot == "true":
default_label_num = default_label_num + '>' + \
SUBMENUITEM_TBOOT
else:
default_label_num = default_label_num + '>' + \
SUBMENUITEM_SECUREBOOT
try:
with open(bl_file) as f:
s = f.read()
for line in s.split("\n"):
if line.startswith("timeout"):
timeout_line = line
elif line.startswith("default"):
default_line = line
if "grub.cfg" in bl_file:
replace = "default='{}'\ntimeout=10".format(default_label_num)
else: # isolinux format
replace = "default {}\ntimeout 10".format(default_label_num)
if default_line and timeout_line:
s = s.replace(default_line, "")
s = s.replace(timeout_line, replace)
elif default_line:
s = s.replace(default_line, replace)
elif timeout_line:
s = s.replace(timeout_line, replace)
else:
s = replace + s
s = re.sub(r'boot_device=[^\s]*',
'boot_device=%s' % host.boot_device,
s)
s = re.sub(r'rootfs_device=[^\s]*',
'rootfs_device=%s' % host.rootfs_device,
s)
s = re.sub(r'console=[^\s]*',
'console=%s' % host.console,
s)
with open(bl_file, "w") as f:
LOG.info("rewriting {}: label={} find=[{}][{}] replace=[{}]"
.format(bl_file, default_label_num, timeout_line,
default_line, replace.replace('\n', '<newline>')))
f.write(s)
except Exception as e:
LOG.error("update_bootloader_default failed: {}".format(e))
raise CloneFail("Failed to update bootloader files")
def get_online_cpus():
""" Get max cpu id """
with open('/sys/devices/system/cpu/online') as f:
s = f.read()
max_cpu_id = s.split('-')[-1].strip()
LOG.info("Max cpu id:{} [{}]".format(max_cpu_id, s.strip()))
return max_cpu_id
return ""
def get_total_mem():
""" Get total memory size """
with open('/proc/meminfo') as f:
s = f.read()
for line in s.split("\n"):
if line.startswith("MemTotal:"):
mem_total = line.split()[1]
LOG.info("MemTotal:[{}]".format(mem_total))
return mem_total
return ""
def get_disk_size(disk):
""" Get the disk size """
disk_size = ""
try:
disk_size = subprocess.check_output(
['lsblk', '--nodeps', '--output', 'SIZE',
'--noheadings', '--bytes', disk])
except Exception as e:
LOG.exception(e)
LOG.error("Failed to get disk size [{}]".format(disk))
raise CloneFail("Failed to get disk size")
return disk_size.strip()
def create_ini_file(clone_archive_dir, iso_name):
"""Create clone ini file."""
interfaces = ""
my_hostname = utils.get_controller_hostname()
macs = sysinv_api.get_mac_addresses(my_hostname)
for intf in macs.keys():
interfaces += intf + " "
disk_paths = ""
for _, _, files in os.walk('/dev/disk/by-path'):
for f in files:
if f.startswith("pci-") and "part" not in f and "usb" not in f:
disk_size = get_disk_size('/dev/disk/by-path/' + f)
disk_paths += f + "#" + disk_size + " "
break # no need to go into sub-dirs.
LOG.info("create ini: {} {}".format(macs, files))
with open(os.path.join(clone_archive_dir, CLONE_ISO_INI), 'w') as f:
f.write('[clone_iso]\n')
f.write('name=' + iso_name + '\n')
f.write('host=' + my_hostname + '\n')
f.write('created_at=' + time.strftime("%Y-%m-%d %H:%M:%S %Z")
+ '\n')
f.write('interfaces=' + interfaces + '\n')
f.write('disks=' + disk_paths + '\n')
f.write('cpus=' + get_online_cpus() + '\n')
f.write('mem=' + get_total_mem() + '\n')
LOG.info("create ini: ({}) ({})".format(interfaces, disk_paths))
def create_iso(iso_name, archive_dir):
""" Create iso image. This is modelled after
the cgcs-root/build-tools/build-iso tool. """
try:
controller_0 = sysinv_api.get_host_data('controller-0')
except Exception as e:
e_log = "Failed to retrieve controller-0 inventory details."
LOG.exception(e_log)
raise CloneFail(e_log)
iso_dir = os.path.join(archive_dir, 'isolinux')
clone_archive_dir = os.path.join(iso_dir, CLONE_ARCHIVE_DIR)
output = None
tmpdir = None
total_steps = 6
step = 1
print ("\nCreating ISO:")
# Add the correct kick-start file to the image
ks_file = "controller_ks.cfg"
if utils.get_system_type() == si_const.TIS_AIO_BUILD:
if si_const.LOWLATENCY in tsconfig.subfunctions:
ks_file = "smallsystem_lowlatency_ks.cfg"
else:
ks_file = "smallsystem_ks.cfg"
try:
# prepare the iso files
images_dir = os.path.join(iso_dir, 'images')
os.mkdir(images_dir, 0644)
pxe_dir = os.path.join('/pxeboot',
'rel-' + tsconfig.SW_VERSION)
os.symlink(pxe_dir + '/installer-bzImage',
iso_dir + '/vmlinuz')
os.symlink(pxe_dir + '/installer-initrd',
iso_dir + '/initrd.img')
utils.progress(total_steps, step, 'preparing files', 'DONE')
step += 1
feed_dir = os.path.join('/www', 'pages', 'feed',
'rel-' + tsconfig.SW_VERSION)
os.symlink(feed_dir + '/Packages', iso_dir + '/Packages')
os.symlink(feed_dir + '/repodata', iso_dir + '/repodata')
os.symlink(feed_dir + '/LiveOS', iso_dir + '/LiveOS')
shutil.copy2(feed_dir + '/isolinux.cfg', iso_dir)
update_bootloader_default(iso_dir + '/isolinux.cfg', controller_0)
shutil.copyfile('/usr/share/syslinux/isolinux.bin',
iso_dir + '/isolinux.bin')
os.symlink('/usr/share/syslinux/vesamenu.c32',
iso_dir + '/vesamenu.c32')
for filename in glob.glob(os.path.join(feed_dir, '*ks.cfg')):
shutil.copy(os.path.join(feed_dir, filename), iso_dir)
utils.progress(total_steps, step, 'preparing files', 'DONE')
step += 1
efiboot_dir = os.path.join(iso_dir, 'EFI', 'BOOT')
os.makedirs(efiboot_dir, 0644)
l_efi_dir = os.path.join('/boot', 'efi', 'EFI')
shutil.copy2(l_efi_dir + '/BOOT/BOOTX64.EFI', efiboot_dir)
shutil.copy2(l_efi_dir + '/centos/MokManager.efi', efiboot_dir)
shutil.copy2(l_efi_dir + '/centos/grubx64.efi', efiboot_dir)
shutil.copy2('/pxeboot/EFI/grub.cfg', efiboot_dir)
update_bootloader_default(efiboot_dir + '/grub.cfg', controller_0)
shutil.copytree(l_efi_dir + '/centos/fonts',
efiboot_dir + '/fonts')
# copy EFI boot image and update the grub.cfg file
efi_img = images_dir + '/efiboot.img'
shutil.copy2(pxe_dir + '/efiboot.img', efi_img)
tmpdir = tempfile.mkdtemp(dir=archive_dir)
output = subprocess.check_output(
["mount", "-t", "vfat", "-o", "loop",
efi_img, tmpdir],
stderr=subprocess.STDOUT)
# replace the grub.cfg file with the updated file
efi_grub_f = os.path.join(tmpdir, 'EFI', 'BOOT', 'grub.cfg')
os.remove(efi_grub_f)
shutil.copy2(efiboot_dir + '/grub.cfg', efi_grub_f)
subprocess.call(['umount', tmpdir])
shutil.rmtree(tmpdir, ignore_errors=True)
tmpdir = None
epoch_time = "%.9f" % time.time()
disc_info = [epoch_time, tsconfig.SW_VERSION, "x86_64"]
with open(iso_dir + '/.discinfo', 'w') as f:
f.write('\n'.join(disc_info))
# copy the latest install_clone executable
shutil.copy2('/usr/bin/install_clone', iso_dir)
subprocess.check_output("cat /pxeboot/post_clone_iso_ks.cfg >> " +
iso_dir + "/" + ks_file, shell=True)
utils.progress(total_steps, step, 'preparing files', 'DONE')
step += 1
# copy patches
iso_patches_dir = os.path.join(iso_dir, 'patches')
iso_patch_repo_dir = os.path.join(iso_patches_dir, 'repodata')
iso_patch_pkgs_dir = os.path.join(iso_patches_dir, 'Packages')
iso_patch_metadata_dir = os.path.join(iso_patches_dir, 'metadata')
iso_patch_applied_dir = os.path.join(iso_patch_metadata_dir, 'applied')
iso_patch_committed_dir = os.path.join(iso_patch_metadata_dir,
'committed')
os.mkdir(iso_patches_dir, 0755)
os.mkdir(iso_patch_repo_dir, 0755)
os.mkdir(iso_patch_pkgs_dir, 0755)
os.mkdir(iso_patch_metadata_dir, 0755)
os.mkdir(iso_patch_applied_dir, 0755)
os.mkdir(iso_patch_committed_dir, 0755)
repodata = '/www/pages/updates/rel-%s/repodata/' % tsconfig.SW_VERSION
pkgsdir = '/www/pages/updates/rel-%s/Packages/' % tsconfig.SW_VERSION
patch_applied_dir = '/opt/patching/metadata/applied/'
patch_committed_dir = '/opt/patching/metadata/committed/'
subprocess.check_call(['rsync', '-a', repodata,
'%s/' % iso_patch_repo_dir])
if os.path.exists(pkgsdir):
subprocess.check_call(['rsync', '-a', pkgsdir,
'%s/' % iso_patch_pkgs_dir])
if os.path.exists(patch_applied_dir):
subprocess.check_call(['rsync', '-a', patch_applied_dir,
'%s/' % iso_patch_applied_dir])
if os.path.exists(patch_committed_dir):
subprocess.check_call(['rsync', '-a', patch_committed_dir,
'%s/' % iso_patch_committed_dir])
utils.progress(total_steps, step, 'preparing files', 'DONE')
step += 1
create_ini_file(clone_archive_dir, iso_name)
os.chmod(iso_dir + '/isolinux.bin', 0664)
iso_file = os.path.join(archive_dir, iso_name + ".iso")
output = subprocess.check_output(
["nice", "mkisofs",
"-o", iso_file, "-R", "-D",
"-A", "oe_iso_boot", "-V", "oe_iso_boot",
"-f", "-quiet",
"-b", "isolinux.bin", "-c", "boot.cat", "-no-emul-boot",
"-boot-load-size", "4", "-boot-info-table",
"-eltorito-alt-boot", "-e", "images/efiboot.img",
"-no-emul-boot",
iso_dir],
stderr=subprocess.STDOUT)
LOG.info("{} created: [{}]".format(iso_file, output))
utils.progress(total_steps, step, 'iso created', 'DONE')
step += 1
output = subprocess.check_output(
["nice", "isohybrid",
"--uefi",
iso_file],
stderr=subprocess.STDOUT)
LOG.debug("isohybrid: {}".format(output))
output = subprocess.check_output(
["nice", "implantisomd5",
iso_file],
stderr=subprocess.STDOUT)
LOG.debug("implantisomd5: {}".format(output))
utils.progress(total_steps, step, 'checksum implanted', 'DONE')
print("Cloned iso image created: {}".format(iso_file))
except Exception as e:
LOG.exception(e)
e_log = "ISO creation ({}) failed".format(iso_name)
if output:
e_log += ' [' + output + ']'
LOG.error(e_log)
raise CloneFail("ISO creation failed.")
finally:
if tmpdir:
subprocess.call(['umount', tmpdir], stderr=DEVNULL)
shutil.rmtree(tmpdir, ignore_errors=True)
def find_and_replace_in_file(target, find, replace):
""" Find and replace a string in a file. """
found = None
try:
for line in fileinput.FileInput(target, inplace=1):
if find in line:
# look for "find" string within word boundaries
fpat = r'\b' + find + r'\b'
line = re.sub(fpat, replace, line)
found = True
print line,
except Exception as e:
LOG.error("Failed to replace [{}] with [{}] in [{}]: {}"
.format(find, replace, target, str(e)))
found = None
finally:
fileinput.close()
return found
def find_and_replace(target_list, find, replace):
""" Find and replace a string in all files in a directory. """
found = False
file_list = []
for target in target_list:
if os.path.isfile(target):
if find_and_replace_in_file(target, find, replace):
found = True
file_list.append(target)
elif os.path.isdir(target):
try:
output = subprocess.check_output(
['grep', '-rl', find, target])
if output:
for line in output.split('\n'):
if line and find_and_replace_in_file(
line, find, replace):
found = True
file_list.append(line)
except Exception: