openstack
/
devstack-gate
Code Issues Proposed changes
Run DevStack in the gate
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1902 Commits
1 Branch
Branch: master
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from 'master'
${ noResults }
devstack-gate/multinode_setup_info.txt

multinode_setup_info.txt 9.8KB
Raw Permalink Blame History

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149 
  1. The basic requirement we have for running multinode openstack tests is that

  2. tempest must be able to ssh and ping the nested VMs booted by openstack and

  3. these nested VMs need to be able to talk to each other. This is due to how

  4. the tests are run by tempest.

  5. 

  6. We run devstack-gate on multiple public clouds. In order to meet the above

  7. requirement we need some control over l2 and l3 networking in the test envs,

  8. but not all of our clouds provide this control. To work around this we setup

  9. overlay networks across the VMs using software bridges and tunnels between

  10. the hosts. This provides routing for the floating IP network between tempest

  11. and VMs and between VMs themselves.

  12. 

  13. To map against a real deployment the overlay networks would be the networking

  14. provided by your datacenter for OpenStack and the existing eth0 for each test

  15. node is a management interface or ILO. We just have to set up our own

  16. datacenter networking because we are running in clouds.

  17. 

  18. Some useful IP ranges:

  19. 172.24.4.0/23 This is our "public" IP range. Test nodes get IPs in the first

  20. half of this subnet.

  21. 172.24.5.0/24 This is our floating IP range. VMs get assigned floating IPs

  22. from this range. The test nodes know how to "route" to these VMs due to the

  23. interfaces on 172.24.4.0/23.

  24. 

  25. Now to network solution specifics. Nova network and neutron are different

  26. enough that they deserve their own specific documentation below.

  27. 

  28. Nova Network

  29. ============

  30. 

  31.            Subnode1                      Primary Node                   Subnode2

  32. +--------------------------+   +--------------------------+   +--------------------------+

  33. |                          |   |                          |   |                          |

  34. |                          |   |                          |   |                          |

  35. |                          |   |                          |   |                          |

  36. |172.24.4.2/23             |   |172.24.4.1/23             |   |172.24.4.3/23             |

  37. |+------+      +--------+  |   |+-------+      +-------+  |   |+-------+      +-------+  |

  38. ||br_pub|      | br_flat|  |   ||br_pub |      |br_flat|  |   ||br_pub |      |br_flat|  |

  39. |+--+---+      +---+----+  |   |+---+---+      +---+---+  |   |+---+---+      +---+---+  |

  40. |   |              |       |   |    |              |      |   |    |              |      |

  41. |   |              +------------------vxlan-tunnel-+-----------------vxlan-tunnel-+      |

  42. |   |                      |   |    |                     |   |    |                     |

  43. |   +--------vxlan-tunnel-----------+--------vxlan-tunnel----------+                     |

  44. |                          |   |                          |   |                          |

  45. +--------------------------+   +--------------------------+   +--------------------------+

  46. 

  47. In addition to the requirement for floating IP connectivity above nova net

  48. also requires that the the private network for the VMs be shared so that the

  49. nested VMs can get access to nova services like dhcp and metadata. While not

  50. strictly necessary when using nova net multihost we support both and this

  51. allows nested VM to nested VM communication over private IP.

  52. 

  53. In this setup we have two soft bridges on the primary node (the controller).

  54. The br_flat bridge handles the l2 traffic for the VMs private interfaces.

  55. The br_pub bridge is where floating IPs are configured and allows for test

  56. node to nested VM communication and nested VM to nested VM communication.

  57. We cannot share the l2 bridge for separate l3 communication because nova net

  58. uses ebtables to prevent public IPs from talking to private IPs and we lose

  59. packets on a shared bridge as a result.

  60. 

  61. This is what it all looks like after you run devstack and boot some nodes.

  62. 

  63.            Subnode1                      Primary Node                   Subnode2

  64. +--------------------------+   +--------------------------+   +--------------------------+

  65. | +--+         +-----+     |   | +--+          +-----+    |   | +--+          +-----+    |

  66. | |vm|---------|br100|     |   | |vm|----------|br100|    |   | |vm|----------|br100|    |

  67. | +--+         +-----+     |   | +--+          +-----+    |   | +--+          +-----+    |

  68. |   |              |       |   |   |               |      |   |   |               |      |

  69. |172.25.5.1/24     |       |   |172.25.5.2/24      |      |   |172.25.5.3/24      |      |

  70. |172.24.4.2/23     |       |   |172.24.4.1/23      |      |   |172.24.4.3/23      |      |

  71. |+------+      +--------+  |   |+-------+      +-------+  |   |+-------+      +-------+  |

  72. ||br_pub|      | br_flat|  |   ||br_pub |      |br_flat|  |   ||br_pub |      |br_flat|  |

  73. |+--+---+      +---+----+  |   |+---+---+      +---+---+  |   |+---+---+      +---+---+  |

  74. |   |              |       |   |    |              |      |   |    |              |      |

  75. |   |              +------------------vxlan-tunnel-+-----------------vxlan-tunnel-+      |

  76. |   |                      |   |    |                     |   |    |                     |

  77. |   +--------vxlan-tunnel-----------+--------vxlan-tunnel----------+                     |

  78. |                          |   |                          |   |                          |

  79. +--------------------------+   +--------------------------+   +--------------------------+

  80. 

  81. Neutron

  82. =======

  83. 

  84. Neutron is a bit different and comes in two flavors. The base case is

  85. neutron without DVR. In this case all of the l3 networking runs on the

  86. primary node. The other case is with DVR where each test node handles

  87. l3 for the nested VMs running on that test node.

  88. 

  89. For the non DVR case we don't need to do anything special. Devstack and

  90. neutron setup br-int between the nodes for us and all public floating

  91. IP traffic is backhauled over br-int to the primary node where br-ex

  92. exists. br-ex is created on the primary node as it is on the single node

  93. tests and all tempest to floating IP and nested VM to nested VM communication

  94. happens here.

  95. 

  96.            Subnode1                      Primary Node                   Subnode2

  97. +--------------------------+   +--------------------------+   +--------------------------+

  98. |                          |   |                          |   |                          |

  99. |                          |   |                          |   |                          |

  100. |                          |   |                          |   |                          |

  101. |172.24.4.2/23             |   |172.24.4.1/23             |   |172.24.4.3/23             |

  102. |+------+                  |   |+-------+                 |   |+-------+                 |

  103. ||br-ex |                  |   ||br-ex  |                 |   ||br-ex  |                 |

  104. |+--+---+                  |   |+---+---+                 |   |+---+---+                 |

  105. |   |                      |   |    |                     |   |    |                     |

  106. |   |                      |   |    |                     |   |    |                     |

  107. |   +--------vxlan-tunnel-----------+--------vxlan-tunnel----------+                     |

  108. |                          |   |                          |   |                          |

  109. +--------------------------+   +--------------------------+   +--------------------------+

  110. 

  111. The DVR case is a bit more complicated. Devstack and neutron still configure

  112. br-int for us so we don't need two overlay networks like with nova net, but

  113. do need an overlay for floating IP public networking due to our original

  114. requirements. If floating IPs are configured on arbitrary test nodes we need

  115. to know how to get packets to them.

  116. 

  117. Neutron uses br-ex for the floating IP network; unfortunately, Devstack and

  118. neutron do not configure br-ex except for in the trivial detached from

  119. everything case described earlier. This means we have to configure br-ex

  120. ourselves and the simplest way to do that is to just make br-ex the overlay

  121. itself. Doing this allows neutron to work properly with nested VMs talking

  122. to nested VMs and it also allows the test nodes to talk to VMs over br-ex as

  123. well.

  124. 

  125. This is what it all looks like after you run devstack and boot some nodes.

  126. 

  127.            Subnode1                      Primary Node                   Subnode2

  128. +--------------------------+   +--------------------------+   +--------------------------+

  129. |                +------+  |   |                +------+  |   |                +------+  |

  130. |                |br-tun|--------tunnel---------|br-tun|--------tunnel---------|br-tun|  |

  131. |                +------+  |   |                +------+  |   |                +------+  |

  132. |                |br-int|  |   |                |br-int|  |   |                |br-int|  |

  133. |                +------+  |   |                +------+  |   |                +------+  |

  134. |                     |    |   |                     |    |   |                     |    |

  135. |172.24.4.2/23       +--+  |   |172.24.4.1/23       +--+  |   |172.24.4.3/23       +--+  |

  136. |172.24.5.1/24--NAT--|vm|  |   |172.24.5.2/24--NAT--|vm|  |   |172.24.5.3/24--NAT--|vm|  |

  137. |+------+            +--+  |   |+-------+           +--+  |   |+-------+           +--+  |

  138. ||br-ex |                  |   ||br-ex  |                 |   ||br-ex  |                 |

  139. |+--+---+                  |   |+---+---+                 |   |+---+---+                 |

  140. |   |                      |   |    |                     |   |    |                     |

  141. |   |                      |   |    |                     |   |    |                     |

  142. |   +--------vxlan-tunnel-----------+--------vxlan-tunnel----------+                     |

  143. |                          |   |                          |   |                          |

  144. +--------------------------+   +--------------------------+   +--------------------------+

  145. 

  146. When DVR is enabled, agent_mode in l3_agent.ini for the primary node will be set to "dvr"

  147. and "dvr_snat" for any remaining subnodes. DVR HA jobs need 3 node setup with this

  148. configuration, where "dvr_snat" represents the network node with centralized SNAT,

  149. and "dvr" represents compute nodes. There should be at least 2 "dvr_snat" nodes.