![joey5678](/assets/img/avatar_default.png)
Add the function to sync the flavor's extra_spces between cascading and cascaded when instance is launched or rebooted. Change-Id: I1e00aaae1660a58d2462896ea141bf240d81c932
14 KiB
Tricircle
Tricircle is a project for Openstack cascading solution, including the source code of Nova Proxy, Cinder Proxy, Neutron L2/L3 Proxy, Glance sync manager and Ceilometer Proxy(not implemented yet).
The project name "Tricircle" comes from a fractal. See the blog "OpenStack cascading and fractal" for more information.
Important to know
- Only about 15k code lines developed for OpenStack cascading.
- The source code now based on Juno is for PoC only. Refactory will be done constantly to reach OpenStack acceptance standard.
- The Neutron cascading using the feature of provider network. But horizon doen't support provider network very well. So you have to use Neutron CLI to create a network.
- Support L2 networking(VxLAN) across cascaded OpenStack, but only p2p remote host IP tunneling supported now.L2 networking through L2GW to reduce population traffic and simplify networking topology will be developed in the near future.
- The L3 networking across casacaded OpenStack will set up tunneling network for piggy data path, useing GRE tunneling over extra_route to brige the router in different cascaded OpenStack.Therefore, the loca L2 network (VLAN,VxLAN) in one cascaded OpenStack can reach L2 network(VLAN,VxLAN) located in another cascaded OpenStack.
- Glance cascading using Glance V2 API. Only CLI/pythonclient support V2 API, the Horizon doesn't support that version. So image management should be done through CLI, and using V2 only. Otherwise, the glance cascading cannot work properly.
- Glance cascading is not used by default, eg, useing global Glance by default. If Glance cascading is required, configuration is required.
Key modules
-
Nova proxy
The hypervisor driver for Nova running on Nova-Compute node. Transfer the VM operation to cascaded Nova. Also responsible for attach volume and network to the VM in the cascaded OpenStack.
-
Cinder proxy
The Cinder-Volume driver for Cinder running on Cinder-Volume node.. Transfer the volume operation to cascaded Cinder.
-
Neuton proxy
Including L2 proxy and L3 proxy, Similar role like OVS-Agent/L3-Agent. Finish L2/L3-networking in the cascaded OpenStack, including cross OpenStack networking.
-
Glance sync
Synchronize image among the cascading and policy determined Cascaded OpenStacks
Patches required
-
Juno-Patches
Pacthes for OpenStack Juno version, including patches for cascading level and cacscaded level.
Feature Supported
-
Nova cascading Launch/Reboot/Terminate/Resize/Rescue/Pause/Un-pause/Suspend/Resume/VNC Console/Attach Volume/Detach Volume/Snapshot/KeyPair/Flavor
-
Cinder cascading Create Volume/Delete Volume/Attach Volume/Detach Volume/Extend Volume/Create Snapshot/Delete Snapshot/List Snapshots/Create Volume from Snapshot/Create Volume from Image/Create Volume from Volume (Clone)/Create Image from Volume
-
Neutron cascading Network/Subnet/Port/Router. Including L2/L3 networking across cascaded OpenStacks
-
Glance cascading Only support V2 api. Create Image/Delete Image/List Image/Update Image/Upload Image/Patch Location/VM Snapshot/Image Synchronization
Known Issues
- Use "admin" role to experience these feature first, multi-tenancy has not been tested well.
- Launch VM only support "boot from image", "boot from volume", "boot from snapshot"
- Flavor only support new created flavor synchronized to the cascaded OpenStack, does not support flavor update synchronization to cascaded OpenStack yet.
Installation without Glance cascading
-
Prerequisites
- the minimal installation requires three OpenStack Juno installated to experience across cascaded OpenStacks L2/L3 function. The minimal setup needs four nodes, see the following picture:
-
the cascading OpenStack needs two node, Node1 and Node 2. Add Node1 to AZ1, Node2 to AZ2 in the cascading OpenStack for both Nova and Cinder.
-
It's recommended to name the cascading Openstack region to "Cascading" or "Region1"
-
Node1 is all-in-one OpenStack installation with KeyStone and Glance, Node1 also function as Nova-Compute/Cinder-Volume/Neutron OVS-Agent/L3-Agent node, and will be replaced to be the proxy node for AZ1.
-
Node2 is general Nova-Compute node with Cinder-Volume, Neutron OVS-Agent/L3-Agent function installed. And will be replaced to be the proxy node for AZ2
-
the all-in-one cascaded OpenStack installed in Node3 function as the AZ1. Node3 will also function as the Nova-Compute/Cinder-Volume/Neutron OVS-Agent/L3-Agent in order to be able to create VMs/Volume/Networking in this AZ1. Glance is only required to be installed if Glance cascading needed. Add Node3 to AZ1 in the cascaded OpenStack both for Nova and Cinder. It's recommended to name the cascaded Openstack region for Node3 to "AZ1"
-
the all-in-one cascaded OpenStack installed in Node4 function as the AZ2. Node3 will also function as the Nova-Compute/Cinder-Volume/Neutron OVS-Agent/L3-Agent in order to be able to create VMs/Volume/Networking in this AZ2. Glance is only required to be installed if Glance cascading needed.Add Node4 to AZ2 in the cascaded OpenStack both for Nova and Cinder.It's recommended to name the cascaded Openstack region for Node4 to "AZ2"
Make sure the time of these four nodes are synchronized. Because the Nova Proxy/Cinder Proxy/Neutron L2/L3 Proxy will query the cascaded OpenStack using timestamp, incorrect time will lead to VM/Volume/Port status synchronization not work properly.
Register all services endpoint in the global shared KeyStone.
Make sure the 3 OpenStack can work independently before cascading introduced, eg. you can boot VM with network, create volume and attach volume in each OpenStack. After verify that 3 OpenStack can work independently, clean all created resources VM/Volume/Network.
After all OpenStack installation is ready, it's time to install Juno pathces both for cascading OpenStack and cascaded OpenStack, and then replace the Nova-Compute/Cinder-Volume/Neutron OVS-Agent/L3-Agent to Nova Proxy / Cinder Proxy / Neutron l2/l3 Proxy.
-
Juno pachtes installation step by step
- Node1
-
Patches for Nova - instance_mapping_uuid_patch
This patch is to make the Nova proxy being able to translate the cascading level VM's uuid to cascadede level VM's uuid
Navigate to the folder
cd ./tricircle/juno-patches/nova/instance_mapping_uuid_patch
follow README.md instruction to install the patch
-
Patches for Cinder - Volume/SnapShot/Backup UUID mapping patch
This patch is to make the Cinder proxy being able to translate the cascading level (Volume/Snapshot/backup)'s uuid to cascaded level (Volume/Snapshot/backup)'s uuid
Navigate to the folder
cd ./tricircle/juno-patches/cinder/uuid-mapping-patch
follow README.md instruction to install the patch
-
Patches for Neutron - neutron_cascading_l3_patch
This patch is to enable cross cascaded OpenStack L3 routing over extra route.The mapping between cascaded OpenStack and it's onlink external network which is used for GRE tunneling data path
Navigate to the folder
cd ./tricircle/juno-patches/neutron/neutron_cascading_l3_patch
follow README.md instruction to install the patch
- Node3
-
Patches for Cinder - timestamp-query-patch
This patch is to make the cascaded Cinder being able to execute query with timestamp filter, but not to return all objects.
Navigate to the folder
cd ./tricircle/juno-patches/cinder/timestamp-query-patch
follow README.md instruction to install the patch
-
Patches for Neutron - neutron_cascaded_l3_patch
This patch is to enable cross cascaded OpenStack L3 routing over extra route..
Navigate to the folder
cd ./tricircle/juno-patches/neutron/neutron_cascaded_l3_patch
follow README.md instruction to install the patch
- Node4
-
Patches for Cinder - timestamp-query-patch
This patch is to make the cascaded Cinder being able to execute query with timestamp filter, but not to return all objects.
Navigate to the folder
cd ./tricircle/juno-patches/cinder/timestamp-query-patch
follow README.md instruction to install the patch
-
Patches for Neutron - neutron_cascaded_l3_patch
This patch is to enable cross cascaded OpenStack L3 routing over extra route..
Navigate to the folder
cd ./tricircle/juno-patches/neutron/neutron_cascaded_l3_patch
follow README.md instruction to install the patch
- Proxy installation step by step
- Node1
-
Nova proxy
Navigate to the folder
cd ./tricircle/novaproxy
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
-
Cinder proxy
Navigate to the folder
cd ./tricircle/cinderproxy
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
-
L2 proxy
Navigate to the folder
cd ./tricircle/neutronproxy/l2-proxy
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
-
L3 proxy
Navigate to the folder
cd ./tricircle/neutronproxy/l3-proxy
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
- Node2
-
Nova proxy
Navigate to the folder
cd ./tricircle/novaproxy
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
Navigate to the folder
cd ./tricircle/icehouse-patches/nova/instance_mapping_uuid_patch/nova/objects cp instance.py $python_installation_path/site-packages/nova/objects/
This file is a patch for instance UUID mapping used in the proxy nodes.
-
Cinder proxy
Navigate to the folder
cd ./tricircle/cinderproxy
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
Navigate to the folder
cd ./tricircle/icehouse-patches/cinder/uuid-mapping-patch/cinder/db/sqlalchemy cp models.py $python_installation_path/site-packages/cinder/db/sqlalchemy
This file is a patch for instance UUID mapping used in the proxy nodes.
-
L2 proxy
Navigate to the folder
cd ./tricircle/neutronproxy/l2-proxy
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
-
L3 proxy
Navigate to the folder
cd ./tricircle/neutronproxy/l3-proxy
follow README.md instruction to install the proxy. Please change the configuration value in the install.sh according to your environment setting
Upgrade to Glance cascading
-
Prerequisites
- To experience the glance cascading feature, you can simply upgrade the current installation with several step, see the following picture:
- Node1
-
Patches for Glance - glance_location_patch
This patch is to make the glance being able to handle http url location. The patch also insert the sync manager to the chain of responsibility.
Navigate to the folder
cd ./tricircle/juno-patches/glance/glance_location_patch
follow README.md instruction to install the patch
-
Patches for Glance - glance_store_patch
This patch is to make the glance being able to handle http url location.
Navigate to the folder
cd ./tricircle/juno-patches/glance_store/glance_store_patch
follow README.md instruction to install the patch
-
Sync Manager
Navigate to the folder
cd ./tricircle/glancesync
modify the storage scheme configuration for cascading and cascaded level
vi ./tricircle/glancesync/etc/glance/glance_store.yaml
follow README.md instruction to install the sync manager. Please change the configuration value in the install.sh according to your environment setting, espeically for configuration: sync_enabled=True sync_server_port=9595 sync_server_host=127.0.0.1
- Node3
-
Glance Installation
Please install Glance in the Node3 as the casacded Glance. Register the service endpoint in the KeyStone. Change the glance endpoint in nova.conf and cinder.conf to the Glance located in Node3
- Node4
-
Glance Installation
Please install Glance in the Node4 as the casacded Glance. Register the service endpoint in the KeyStone Change the glance endpoint in nova.conf and cinder.conf to the Glance located in Node4
- Configuration
-
Change Nova proxy configuration on Node1, setting the "cascaded_glance_flag" to True and add "cascaded_glance_url" of Node3 configurantion according to Nova-proxy README.MD instruction
-
Change Cinder proxy configuration on Node1, setting the "glance_cascading_flag" to True and add "cascaded_glance_url" of Node3 configurantion according to Nova-proxy README.MD instruction
-
Change Nova proxy configuration on Node2, setting the "cascaded_glance_flag" to True and add "cascaded_glance_url" of Node4 configurantion according to Nova-proxy README.MD instruction
-
Change Cinder proxy configuration on Node2, setting the "glance_cascading_flag" to True and add "cascaded_glance_url" of Node4 configurantion according to Nova-proxy README.MD instruction
- Experience Glance cascading
- Restart all related service
- Use Glance V2 api to create Image, Upload Image or patch location for Image. Image should be able to sync to distributed Glance if sync_enabled is setting to True
- Sync image only during first time usage but not uploading or patch location can works by just modify the option 'sync_strategy=nova' in /etc/glance-sync.conf file and restart the glance sync service.
- Create VM/Volume/etc from Horizon