Sometimes when doing worlddump would see a command line like this:
sudo ip netns exec (id: ip addr
This would cause an error to be seen in console.log:
2017-02-07 00:03:03.659570 | /bin/sh: 1: Syntax error: "(" unexpected
This is caused by there sometimes being extra data returned from the
'ip netns' command [1]. For example it might look like:
qrouter-0805fd7d-c493-4fa6-82ca-1c6c9b23cd9e (id: 1)
qdhcp-bb2cc6ae-2ae8-474f-adda-a94059b872b5 (id: 0)
[1] https://lwn.net/Articles/629715/
Change-Id: Icece442023125ef55696b8d92a975d37e358b1b4
Closes-Bug: 1653969
Commit e7361775c112e32ea517eddc344641897d273d25 changed the code to
use Python 3 style print function, but when doing 'print()' in Python
2.7 it would print '()':
>>> print()
()
Import the __future__ print function so that a blank line will be
printed as expected. This will now work the same in Python 2 & 3.
Change-Id: I61742e107278f2327c18c9ab0de52d1914f16c97
Those reports may be helpful when debugging neutron gate issues.
pgrep is backwards compatible with old Solaris tools, which means it
does not match with commands that are longer than 15 characters. To
avoid that for neutron agent names which are longer than that, we need
to pass -f argument to match against the full cmdline.
Also killall instead of kill + pgrep in a subshell.
Change-Id: I9b3801e927c0e80443ed76e38cd8e3618e888e49
When devstack fails, some or all bridges may not exist.
This change allows an only existing bridge to executes ovs-ofctl command.
And fix duplicate ofp version specified in protocol option of ovs-ofctl.
Change-Id: Ied01de727ca9b867ce87db358f72ae44838b63af
Currently ovs-ofctl command is executed for only default ofp version
(OpenFlow10).
Some Neutron's plugin uses OpenFlow13 and in that case ovs-ofctl fails.
This chage allows us to get ovs info for all ofp versions supported by ovs.
And adds dump by dump-ports and dump-ports-desc.
Change-Id: I2d3c42835a5ad0f5ebf540e8127762f466347c9c
If a command we trigger fails for some reason, it's worth logging
details about the failure (like return code).
Change-Id: Ib19aa474eccdd11e138a4f55e125935b621bca05
Skip with a notification that a command was not found when trying
to run a dump that relies on optional external command.
Otherwise we produce noise in the error output that is misleading.
Change-Id: I0e3d99da8c54d2912463ecba7c1783864a7c7fc7
Closes-Bug: #1548833
Closes-Bug: #1506873
It makes it a bit easier to read the output since each new command is
now visually separated from the output of the previous one.
Change-Id: If441c61bb6f13f85f771dd31609b10d3dd1ee93c
Neutron hugely relies on namespaces, so we should try to dump
internal IP stack state for non-root network namespaces.
Change-Id: Ib980d22fbf3c6b680473754fa2b1684c2ef91b72
The function guru_meditation_report() currently uses the User-defined
signal SIGUSR1 to kill a Nova Compute process so that a Guru Meditation
Report is generated.
Testing locally, in a DevStack instance, manually attempting to kill a
Nova compute process [kill -s USR1 `pgrep nova-compute`] does not result
in process being terminated, and no error report generated.
It turns out[1] that SIGUSR1 is used by Apache 'mod_wsgi'.
Using the signal SIGUSR2 resolves this issue (i.e. 'nova-compute'
process is terminated, and the Guru Meditation Report is generated).
So, use USR2, instead of USR1.
Corresponding oslo.reports related commit[2].
[1] https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIRestrictSignal
[2] 45b1c02d113051d147e54ef921ce8e94135542d8 -- guru_meditation_report:
Use SIGUSR2 instead of SIGUSR1
[3] Original DevStack commit that brought in this change --
2ebe993b25462919e8aeeb896c9f91b6be7aa573
Change-Id: I8a7eaf71b83edca3c80074d6bf2d471e3db6142b
I'm still at a loss about why guests stop being pingable in grenade,
so lets get ourselves some ebtables output as well.
Change-Id: I4e40eff6d0b1ef194e43b151a83206fbd50deb66
We're worlddumping at success points in grenade, and it would be much
handier to explain when that happens via a symbolic name in the
filename. Add a --name option to worlddump to allow it.
Change-Id: I644200fe08e404dc7ca2006478ae4e11ca020672
Unconditionally running this can lead to confusing failure output from
kill as the pgrep matches nothing when nova-compute isn't yet running.
Change-Id: I37cb84fe8e0b393f49b8907af16a3e44f82c46a6
Nova-compute is hanging in the multinode test, and its difficult to
figure out why. So trigger a guru meditation report for nova-compute in
worlddump so we can see what nova-compute is doing when it is hung.
Having a hung nova-compute causes tempest to fail and
I035fe8e3333034e44b403ed0f986220ab5b0e57a runs worlddump whenever
tempest fails.
Bug 1462305 is one of the last issues left before the multinode job is
stable enough to gate on, and this patch should make it much easier to
debug.
Change-Id: I87d7536b5992c47b8082684cc662f953113fd1a8
Related-Bug: #1462305
This adds potentially helpful networking info to the world dump.
It also refactors some of the output mechanisms into reusable
functions for cleanliness in the code.
Change-Id: I39f95bd487c152925f8fadd1799149db35cffd52
If we fail during devstack / grenade runs, it would be nice to have
the map of iptables that are currently active as well. This makes it
handy to start figuring out what's going on when test servers don't
ping.
Change-Id: Ia31736ef2cb0221586d30c089473dfdc1db90e23
when we exit poorly, start building a tool for doing a capture
as to why. This tool is expected to be consumed directly from
grenade as well.
Change-Id: Ia1bc1c2e259587035ca7252baceec25fa464cb82
