279 Commits

Author SHA1 Message Date
Dmitry Tantsur
5fa258b708 Fix "logs" inspection collector when logs contain non-ascii symbols
Somehow it didn't pop earlier. Updated tests to contain some creepy
russian letters :)

Closes-Bug: #1517913
Change-Id: I4c6712ea1e813d1f0f0d0aedaccfa1187526e0ec
2015-12-08 14:32:16 +01:00
Jenkins
1720759fa7 Merge "Support Linux-IO in addition to tgtd" 2015-12-08 04:52:42 +00:00
Dmitry Tantsur
2fc6ce22f8 pyudev exception has changed for from_device_file
Now pyudev raises DeviceNotFoundByFileError which does not inherit
from EnvironmentError, so our 'except' block in hardware.py no longer
catch the exception. It broke unit tests, but it can also potentially
break the deploy.

This patch updates hardware.py to catch both old a new exceptions.

Change-Id: Iaefd6089f6f766a241054d8e132b2f3098c8130d
Closes-Bug: #1522756
2015-12-07 16:47:29 +01:00
Dmitry Tantsur
c474a5ac6c Support Linux-IO in addition to tgtd
The iSCSI extension now tries to use Linux-IO first (via rtslib)
and falls back to tgtd if Linux-IO can't be used (e.g. in the CoreOS-based
image which uses containers).

Change-Id: I9cc7a30d9c93c445a66d183146e9260c2b096d33
Closes-Bug: #1504562
2015-11-30 18:38:03 +01:00
Jenkins
4ba04475ec Merge "Add support for streaming raw images directly onto the disk" 2015-11-21 19:17:23 +00:00
Jenkins
a2a71105c4 Merge "Refactor the image download and checksum computation bits" 2015-11-19 11:28:25 +00:00
Jenkins
b53bd28a20 Merge "Create partition at max msdos limit for disks > 2TB" 2015-11-18 20:42:21 +00:00
Lucas Alvares Gomes
e320bb8942 Add support for streaming raw images directly onto the disk
This patch adds support for streaming a raw image directly onto the disk,
that means no more time spent writing the image to a tmpfs partition prior
to copying it to the disk. Checksum computation is also done as the image
is being streamed. Streaming raw images is disabled by default, however
this behavior can be enabled by passing a key called "stream_raw_images"
with the value of True to the prepare_image() command of IPA.

For non-raw images this may not be possible, not sure about all image
file formats, but common types such as qcow2 requires random access to
the image file in order to be converted to raw.

Closes-Bug: #1505685
Change-Id: Iddf67907bc9b54bbd3065a97064cb5a3602cfe18
2015-11-18 11:19:40 +00:00
Lucas Alvares Gomes
65053b7737 Refactor the image download and checksum computation bits
Prior to this patch downloading and computing the checksum of the image
were done in different stages, after the download the file would need
to be re-read and the checksum was computed. This patch is changing it
by creating a ImageDownload class which computes the checksum at the same
time the image is being downloaded, this saves time and also make the
code more portable.

Related-Bug: #1505685
Change-Id: I71f9f2bd9a62a6a6cc474d0ae519591cea6381d6
2015-11-18 11:19:38 +00:00
Lucas Alvares Gomes
c21409e98b Follow up patch for da9c3b0adc67efa916fc534d975823c0a45948a1
This patch is a follow up patch fixing some nits left by the review
da9c3b0adc67efa916fc534d975823c0a45948a1, this patch adds the
wwn_with_extension and wwn_vendor_extension root device hints to the
"serializable_fields" list attribute of the BlockDevice class and fixes
some tests.

Change-Id: I6039be535988319276f9ac355c80997d34328ce8
2015-11-18 09:56:09 +00:00
Yolanda Robla
a01c4c90eb Create partition at max msdos limit for disks > 2TB
Currently there is a problem on partition creation for MBR
case. If disk exceeds 2TB, partition creation fails because
we are hitting max msdos limit. Detect total disk size, and if
it exceeds that limit, create the partition at the end of legal
limits.

Fixes-Bug: #1517077
Change-Id: I11dd3f11eaa6af764151b442768d10289ced6d3f
2015-11-18 10:48:46 +01:00
Jenkins
39bdbdd357 Merge "determine tgtd ready status through tgtadm" 2015-11-17 23:40:08 +00:00
Jenkins
5dd57838d1 Merge "Support proxies for image download" 2015-11-17 23:01:43 +00:00
Vladyslav Drok
54c901e8d5 Support proxies for image download
This change adds proxies parameter to requests.get method when
downloading images, and also sets no_proxy environment variable,
as requests does not support such parameter directly.

Implements: blueprint agent-image-proxy

Change-Id: I000ff9d0d4f5bde4766d085dc5a1c6cea1949bc5
2015-11-17 11:12:12 -08:00
Lucas Alvares Gomes
da9c3b0adc Extend root device hints for different types of WWN
This patch is extending the root device hints to also look at
ID_WWN_WITH_EXTENSION and ID_WWN_VENDOR_EXTENSION from udev.

Prior to this patch the IPA ramdisk only cared about ID_WWN but in some
systems in some platforms with a RAID controller, this ID can be same
even if they are different disks (see bug 1516641).

Closes-Bug: #1516641
Change-Id: Ic3e9a1111dfcc99702190c173562a0dccf5f94c4
2015-11-16 14:58:24 +00:00
Jenkins
2bce5f6065 Merge "Use oslo.log instead of original logging" 2015-11-02 17:44:19 +00:00
Jenkins
96e21042d8 Merge "Replace deprecated LOG.warn with LOG.warning" 2015-11-02 17:23:39 +00:00
ZhiQiang Fan
9e75ba5460 Use oslo.log instead of original logging
We are using oslo.log now, but some of the modules still use logging.
We should use oslo.log to keep consistency, besides, oslo.log can
provide fine wrapper for OpenStack projects.

Change-Id: Ibe57e503b88b39e284a9e4b11a1886cd4e8d4ccf
2015-10-24 03:22:36 -06:00
Jenkins
5a962e4ac7 Merge "Use system random where applicable" 2015-10-23 00:07:32 +00:00
Jenkins
64bdf16b68 Merge "Refactor backoff looping call" 2015-10-23 00:07:12 +00:00
Zhenguo Niu
18d5d6aba3 Replace deprecated LOG.warn with LOG.warning
Change-Id: Ib3d566f6e608ee453659e15cabcf8e9332aedc52
Closes-Bug: #1508442
2015-10-22 14:42:57 +08:00
dparalen
e51ccbe7c3 avoid duplicate text in ISCSIError message
The ISCSIError class defines a class-level message attribute with
value: "Error starting iSCSI target". This attribute is further
processed in RESTError.__init__ method, the ISCSIError super-class, to
create an Exception message concatenating self.message with provided
details argument.  However, the ISCISError.__init__ method provides a
details attribute prefixed with the same text to the super(ISCSIError,
self).__init__ method.  As a result, the text appears twice:
"ISCSIError: Error starting iSCSI target: Error starting iSCSI target:
ISCSI daemon didn't initialize. Failed with exit code 107. stdout: .
stderr: tgtadm: failed to send request hdr to tgt daemon, Transport
endpoint is not connected"

The patch purpose is to remove the details prefix to avoid duplicate
text in the exception text while honouring ISCSIError.message.

Change-Id: I9e1434ae17da5112527a841ac069ed2285566cca
2015-10-20 08:10:56 +02:00
dparalen
fb920f41ed determine tgtd ready status through tgtadm
The intention of the iscsi extension _wait_for_iscsi_daemon function
is to wait till the tgtd process boots completely before running any
iscsi commands through the tgtadm command line utility.  The tgtd
process utilises a unix socket for communication with the user and the
_wait_for_iscsi_daemon function asserts the tgtd process status based
on the socket presence.  The socket name/path to use is not specified
by iscsi extension configuration but rather falls-back to a
(system-wide) default value[1].  Since tgtd version 1.0.60, this path
changed from /var/run/tgtd.ipc_abstract_namespace to
/var/run/tgtd/socket[2] and now causes a false-negative while waiting
for the tgtd process to boot[3].

This patch follows implementation suggestion from the bug report[3] to
avoid relying on the socket presence but rather to check the tgtd
status by trying to communicate through the tgtadm tool.

[1] https://github.com/openstack/ironic-python-agent/blob/
5bbb9ded082f2cfde5e8877e9f33294eb93a5fb7/
ironic_python_agent/extensions/iscsi.py#L47 [2]
https://github.com/fujita/tgt/commit/
d1aa4dcfd691c9409dbad5db49ce6754ce9c1b5a
[3] https://bugs.launchpad.net/ironic-python-agent/+bug/1505923

Closes-Bug: #1505923
Change-Id: If3284e0b441fe6c2da507640c08a4bbefd9dad2b
2015-10-20 04:52:59 +02:00
Jenkins
90b4d15b41 Merge "Make calling arguments easier to understand" 2015-10-16 15:46:57 +00:00
Naohiro Tamura
f013333949 Add param docstring into the normalize func
This patch adds a missing param docstring into the normalize function
in util.py.

Change-Id: I91ecc0970fc386163657ab521d8cdd898694a426
Partial-bug: 1367915
2015-10-14 11:20:39 +09:00
John L. Villalovos
06d34ae5d7 Make calling arguments easier to understand
Use keyword= when doing the call so easier for reader of code to know
what all the arguments are.

Change-Id: I4d781b533b15c6855a3d8ad95bb17bf9d153f1eb
2015-10-13 13:26:18 -07:00
Jenkins
b6c2eb6ca6 Merge "Ensure all methods in utils.py have docstrings" 2015-10-13 16:13:05 +00:00
Jenkins
ef57379342 Merge "Add LOG.debug() if requested device type not found" 2015-10-12 08:54:00 +00:00
Jay Faulkner
6131b2e405 Ensure all methods in utils.py have docstrings
Previous some of these methods were not well documented. Hopefully they
now are.

Change-Id: If73987a2dd234b71a1c2af9b764becc34aee4496
Partial-bug: 1367915
2015-10-09 16:22:06 -07:00
Jenkins
4a248f3ea0 Merge "Use mount -t sysfs to avoid host /sys dependencies" 2015-10-08 09:54:43 +00:00
John L. Villalovos
deb50ac5a8 Add LOG.debug() if requested device type not found
This is a follow-up patch for commit
3af9ab36bfae3a369fdb3d2b6d02ac803c39ee17

The review requested that a LOG.debug() message be added.

Change-Id: I36fbd4269c948812f4bee66d0130150afd0c0279
2015-10-07 17:07:23 -07:00
Jenkins
2c0ebdea09 Merge "Adds more functional tests for commands" 2015-10-07 09:23:18 +00:00
Dan Prince
ebd7b076a5 Use mount -t sysfs to avoid host /sys dependencies
This patch updates the _install_grub2 function in image.py
so that we use 'mount -t sysfs' instead of bind mounting
the hosts /sys into the chroot.

This resolves issues which can occur with unmounting
the chroot mounts where 'target is busy' errors
may occur when you are debugging things with a serial
console.

Change-Id: I5c9a6546d048cbf54695329d9744fda55127bd30
Closes-bug: #1503385
2015-10-07 03:29:18 +00:00
Jenkins
aa908205c6 Merge "Refactor list_all_block_devices & add block_type param" 2015-10-07 02:42:07 +00:00
Dan Prince
5bbb9ded08 Fix log formatting error in iscsi.py
This fixes a ValueError exception that would occur if
log debug was enabled in IPA.

Change-Id: Id95f291dd77d69e7d4dd4f945c6c151db7a0646b
Closes-bug: #1503354
2015-10-06 22:47:54 +00:00
Mario Villaplana
4a7b954d14 Adds more functional tests for commands
This adds additional functional tests for the commands API.
It also restructures the current functional test to run in
sequence as a monolithic test to preserve IPA state and test
order between test runs. The time to wait for IPA to start
was increased due to an intermittent race condition that
occurs with the larger test suite.

Partial-Bug: 1492036
Change-Id: Iff9b41fb531d34225d702b9bfd39826e3c7195ad
2015-10-06 15:36:27 +00:00
John L. Villalovos
dcbba2b121 Enforce all flake8 rules except E129
Bring ironic-python-agent in line with the other ironic projects.

Stop ignoring all E12* errors except E129
Stop ignoring E711

Change-Id: Icb9bc198473d1b5e807c20869eb2af7f4d7ac360
2015-10-02 10:01:00 -07:00
Dmitry Tantsur
9d6b0864e3 Add "logs" and "extra-hardware" inspection collectors
This is a port of downstream inspector ramdisk plugins we found helpful.
* logs - sends journald logs with inspection data.
* extra-hardware - uses hardware-detect utility to collect bigger
  hardware inventory and to run benchmarks.

Change-Id: If05402606c45185d618279eef46e68c51209f82b
2015-10-01 18:25:30 +02:00
Jenkins
378197caee Merge "Make the erase_devices clean step abortable" 2015-09-26 22:25:17 +00:00
Lucas Alvares Gomes
cd70f514d6 Make the erase_devices clean step abortable
This patches updates the get_clean_steps() method to make the
erase_devices step abortable. Erasing devices is something that can be
cancelled without damaging the machine.

When a clean step is aborted the provision state of the Ironic node
will go to CLEANFAIL state. The operator can then do what is needed to
fix the problem (i.e network booting issues) and restart the cleaning
later on.

Partial-Bug: #1455825
Change-Id: Ic181ac3712810c6f6925e8b627ee79e77ecf4d83
2015-09-26 19:01:26 +00:00
John L. Villalovos
3af9ab36bf Refactor list_all_block_devices & add block_type param
Put the columns to retrieve from lsblk into a list so that future
modifications to columns will require fewer code changes.

Also add a 'block_type' parameter which defaults to 'disk'. To make the
function more flexible if callers wanted a different block type.

Update and add unit tests

Change-Id: If06460e13a5b56dc8d6efca9ff5b58ac6ba1f357
2015-09-24 15:31:13 -07:00
Jenkins
7dba81f200 Merge "Fixes config drive creation failure in UEFI boot mode" 2015-09-24 04:42:48 +00:00
Shivanand Tendulker
2d341c7f0d Fixes config drive creation failure in UEFI boot mode
This fix enables creation of config drive for UEFI only whole disk
images.
It will not work with the hybrid images that support booting in BIOS
and UEFI boot mode.

Change-Id: Ib4dd8c082a50e1dbaf0df91477b062716cb780ff
Closes-Bug: #1486887
Depends-On: I81400305f166d62aa4612aab54602abb8178b64c
2015-09-23 23:14:46 +00:00
Naohiro Tamura
cae39ae14d Fix default ironic api port number
This patch corrects the default ironic api port number to 6385 from
6835. The default value of 'ipa-api-url' is not used in production
environment since conductor and bare metal node have its own IP
address, but could be used in development environment.
Therefor this patch fixes this trivial error.

- default=APARAMS.get('ipa-api-url', 'http://127.0.0.1:6835'),
+ default=APARAMS.get('ipa-api-url', 'http://127.0.0.1:6385'),

Change-Id: Idb85e95b354a68111f94e8cddd5c1b6227e676ef
2015-09-23 14:11:29 +09:00
Jenkins
19d21eb999 Merge "Add more info to checksum exception" 2015-09-22 13:46:48 +00:00
Josh Gachnang
e17129dbad Add more info to checksum exception
If an image cannot be downloaded for some reason, it is helpful for
operators to have the image path, checksum, and calculated checksums
available easily from the API.

Change-Id: I6a2fb46726245cebd730b5c51d4f25f8465f1658
2015-09-21 15:54:02 -07:00
Dmitry Tantsur
b569e37d06 Expose serial, wwn and vendor on the BlockDevice object
Currently we only use these disk properties for root device hints.
However, they'll be really useful for inspector, especially for also
implementing root device hints.

Change-Id: I48aa6b6d2d198d16f2f8e387970f7230066cf8a2
2015-09-21 13:17:20 +02:00
Jenkins
d34f5d857a Merge "Create a SerializableComparable class" 2015-09-18 23:47:11 +00:00
Dmitry Tantsur
3b70647358 inspection: prepare for future deprecations
1. cleanly separate deprecated and non-deprecated properties
2. add root disk to inspection data, so that we can have a proper
   fallback when root device hints are not given.

Change-Id: Ie19b82ff2a914873ff4b2395b02643e086b934b1
2015-09-16 14:26:57 +02:00
Jenkins
3cd59fac63 Merge "Follow-up to inspection patch 096830414b" 2015-09-16 12:17:57 +00:00