'str' type in python2 is 'bytes' type in python3,
when use python3, we will get a prefix 'b':
sudo ip netns exec b'qrouter-39cc2b45-a27c-49c5-94a6-26443a49ac63' ip
neigh
--------------------------------------------------------------------------
*** Failed to run 'sudo ip netns exec
b'qrouter-39cc2b45-a27c-49c5-94a6-26443a49ac63' ip neigh': Command 'sudo
ip netns exec b'qrouter-39cc2b45-a27c-49c5-94a6-26443a49ac63' ip neigh'
returned non-zero exit status 1.
The message above is raised by running tools/worlddump.py with python3.
Change-Id: Ic254af86fa27729839f00c0ad4a5bbbc9e545a09
After Python 2 is getting unsupported, new distros
like CentOS 8 and RHEL8 have stopped providing 'python'
package forcing user to decide which alternative to
use by installing 'python2' or 'python3.x' package
and then setting python alternative.
This change is intended to make using python3 command as
much as possible and use it as default 'python' alternative
where needed.
The final goals motivating this change are:
- stop using python2 as much as possible
- help adding support for CentOS 8 and RHEL8
Change-Id: I1e90db987c0bfa6206c211e066be03ea8738ad3f
Some distros no longer ship brctl, iproute2 should be used
in its place. The linuxbridge agent plugin script was still
using it, as was worlddump, which generates this warning on
a failure:
Running devstack worlddump.py
/bin/sh: 1: brctl: not found
Conditionalizing worlddump based on whether brctl is installed
to make this go away.
Change-Id: Iafbf4038bab08c261d45d117b12d4629ba32d65e
All credit for figuring this out goes to frickler (and that was the hard
bit so thank you!). The worlddump files were not being collected because
they weren't in our log collection list. Add worlddump to this list
so that we collect these files.
One thing that makes this slightly complicated is the worlddump files
are named with a timestamp and we can't have globs in our collection
list. To address this we create a copy of the file with a -latest.txt
suffix. This gives us a deterministic file name for log collection
without using globs.
Note we do not use a symlink here because some jobs gzip their log files
(breaking symlinks) and others do not. This makes it painful to always
have a valid link. Not having a valid link can break log collection.
Hardlinks may be another option but simply making a copy is easier to
manage as you don't have to worry about links preexisting and the
dumpfiles are not that large.
Change-Id: I96ae5f5290546ad25ca434c1106c01354d2d053c
The bridge-utils package has been deprecated for some time now [1] and
'brctl' does not exist on some more recent distros like Fedora 28.
Replace references to brctl with the proper ip commands.
Calls to "brctl show" are not being replaced with calls to "bridge link"
because the output format is very different and in testing some bridges
were not listed. So the simpler method of consulting /sys/class/net is
used.
In worlddump.py we try running both because failures are handled
gracefully by _dump_cmd(), as well as "ip link show type bridge" for
additional info.
[1] https://lwn.net/Articles/703776/ for example
Change-Id: Ie4c8ad6ce4a09c38023c9e4ec7834c249403145f
Partial-Bug: #1801919
Worlddumping on a system without net-tools package gets an error showing
"arp: not found". As iproute2 can also show arp tables, we use it
instead.
Change-Id: I0cd83e6d14959dc5a1147c487b11f27fb92aa20a
This adds a flag and basic config for enabling coredumps for libvirt.
Partial-Bug: 1643911
Co-Authored-By: Matthew Booth <mbooth@redhat.com>
Change-Id: If7cd54e804a5a389a0d82a325b58f5b41b8ef0db
Sometimes when doing worlddump would see a command line like this:
sudo ip netns exec (id: ip addr
This would cause an error to be seen in console.log:
2017-02-07 00:03:03.659570 | /bin/sh: 1: Syntax error: "(" unexpected
This is caused by there sometimes being extra data returned from the
'ip netns' command [1]. For example it might look like:
qrouter-0805fd7d-c493-4fa6-82ca-1c6c9b23cd9e (id: 1)
qdhcp-bb2cc6ae-2ae8-474f-adda-a94059b872b5 (id: 0)
[1] https://lwn.net/Articles/629715/
Change-Id: Icece442023125ef55696b8d92a975d37e358b1b4
Closes-Bug: 1653969
Commit e7361775c112e32ea517eddc344641897d273d25 changed the code to
use Python 3 style print function, but when doing 'print()' in Python
2.7 it would print '()':
>>> print()
()
Import the __future__ print function so that a blank line will be
printed as expected. This will now work the same in Python 2 & 3.
Change-Id: I61742e107278f2327c18c9ab0de52d1914f16c97
Those reports may be helpful when debugging neutron gate issues.
pgrep is backwards compatible with old Solaris tools, which means it
does not match with commands that are longer than 15 characters. To
avoid that for neutron agent names which are longer than that, we need
to pass -f argument to match against the full cmdline.
Also killall instead of kill + pgrep in a subshell.
Change-Id: I9b3801e927c0e80443ed76e38cd8e3618e888e49
When devstack fails, some or all bridges may not exist.
This change allows an only existing bridge to executes ovs-ofctl command.
And fix duplicate ofp version specified in protocol option of ovs-ofctl.
Change-Id: Ied01de727ca9b867ce87db358f72ae44838b63af
Currently ovs-ofctl command is executed for only default ofp version
(OpenFlow10).
Some Neutron's plugin uses OpenFlow13 and in that case ovs-ofctl fails.
This chage allows us to get ovs info for all ofp versions supported by ovs.
And adds dump by dump-ports and dump-ports-desc.
Change-Id: I2d3c42835a5ad0f5ebf540e8127762f466347c9c
If a command we trigger fails for some reason, it's worth logging
details about the failure (like return code).
Change-Id: Ib19aa474eccdd11e138a4f55e125935b621bca05
Skip with a notification that a command was not found when trying
to run a dump that relies on optional external command.
Otherwise we produce noise in the error output that is misleading.
Change-Id: I0e3d99da8c54d2912463ecba7c1783864a7c7fc7
Closes-Bug: #1548833
Closes-Bug: #1506873
It makes it a bit easier to read the output since each new command is
now visually separated from the output of the previous one.
Change-Id: If441c61bb6f13f85f771dd31609b10d3dd1ee93c
Neutron hugely relies on namespaces, so we should try to dump
internal IP stack state for non-root network namespaces.
Change-Id: Ib980d22fbf3c6b680473754fa2b1684c2ef91b72
The function guru_meditation_report() currently uses the User-defined
signal SIGUSR1 to kill a Nova Compute process so that a Guru Meditation
Report is generated.
Testing locally, in a DevStack instance, manually attempting to kill a
Nova compute process [kill -s USR1 `pgrep nova-compute`] does not result
in process being terminated, and no error report generated.
It turns out[1] that SIGUSR1 is used by Apache 'mod_wsgi'.
Using the signal SIGUSR2 resolves this issue (i.e. 'nova-compute'
process is terminated, and the Guru Meditation Report is generated).
So, use USR2, instead of USR1.
Corresponding oslo.reports related commit[2].
[1] https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIRestrictSignal
[2] 45b1c02d113051d147e54ef921ce8e94135542d8 -- guru_meditation_report:
Use SIGUSR2 instead of SIGUSR1
[3] Original DevStack commit that brought in this change --
2ebe993b25462919e8aeeb896c9f91b6be7aa573
Change-Id: I8a7eaf71b83edca3c80074d6bf2d471e3db6142b
I'm still at a loss about why guests stop being pingable in grenade,
so lets get ourselves some ebtables output as well.
Change-Id: I4e40eff6d0b1ef194e43b151a83206fbd50deb66
We're worlddumping at success points in grenade, and it would be much
handier to explain when that happens via a symbolic name in the
filename. Add a --name option to worlddump to allow it.
Change-Id: I644200fe08e404dc7ca2006478ae4e11ca020672
Unconditionally running this can lead to confusing failure output from
kill as the pgrep matches nothing when nova-compute isn't yet running.
Change-Id: I37cb84fe8e0b393f49b8907af16a3e44f82c46a6
Nova-compute is hanging in the multinode test, and its difficult to
figure out why. So trigger a guru meditation report for nova-compute in
worlddump so we can see what nova-compute is doing when it is hung.
Having a hung nova-compute causes tempest to fail and
I035fe8e3333034e44b403ed0f986220ab5b0e57a runs worlddump whenever
tempest fails.
Bug 1462305 is one of the last issues left before the multinode job is
stable enough to gate on, and this patch should make it much easier to
debug.
Change-Id: I87d7536b5992c47b8082684cc662f953113fd1a8
Related-Bug: #1462305
This adds potentially helpful networking info to the world dump.
It also refactors some of the output mechanisms into reusable
functions for cleanliness in the code.
Change-Id: I39f95bd487c152925f8fadd1799149db35cffd52
If we fail during devstack / grenade runs, it would be nice to have
the map of iptables that are currently active as well. This makes it
handy to start figuring out what's going on when test servers don't
ping.
Change-Id: Ia31736ef2cb0221586d30c089473dfdc1db90e23
when we exit poorly, start building a tool for doing a capture
as to why. This tool is expected to be consumed directly from
grenade as well.
Change-Id: Ia1bc1c2e259587035ca7252baceec25fa464cb82
