.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode Convention for heading levels in Neutron devref: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 (Avoid deeper levels because they do not render well.) ============================= Kubernetes API Watcher Design ============================= This documentation describes the `Kuberenetes API`_ watcher daemon component, **Raven**, of Kuryr. What is Raven ------------- Raven is a daemon watches the internal Kubernetes (K8s) state through its API server and receives the changes with the event notifications. Raven then translate the state changes of K8s into requests against Neutron API and constructs the virtual network topology on Neutron. Raven must act as the centralized component for the translations due to the constraints come from the concurrent deployments of the pods on worker nodes. Unless it's centralized, each plugin on each work node would make requests against Neutron API and it would lead the conflicts of the requests due to the race condition because of the lack of the lock or the serialization mechanisms for the requests against Neutron API. Raven doesn't take care of the bindings between the virtual ports and the physical interfaces on worker nodes. It is the responsibility of Kuryr CNI_ plugin for K8s and it shall recognize which Neutron port should be bound to the physical interface associated with the pod to be deployed. So Raven focuses only on building the virtual network topology translated from the events of the internal state changes of K8s through its API server. Goal ~~~~ Through Raven the changes to K8s API server are translated into the appropriate requests against Neutron API and we can make sure their logical networking states are synchronized and consistent when the pods are deployed. Translation Mapping ------------------- The detailed specification of the translation mappings is described in another document, :doc:`../specs/kuryr_k8s_integration`. In this document we touch what to be translated briefly. The main focus of Raven is the following resources. * Namespace * Pod * Service (Optional) Namespaces are translated into the networking basis, Neutron networks and subnets for the cluster and the service with the explicitly predefined values in the configuration file, or implicitly specified by the environment variables, e.g., ``FLANNEL_NET=172.16.0.0/16`` as specified `in the deployment phase`_. Raven also creates Neutron routers for connecting the cluster subnets and the service subnets. When each namespace is created, a cluster network that contains a cluster subnet, a service network that contains a service subnet, and a router that connects the cluster subnet and the service subnet are created through Neutron API. The apiserver ensures namespaces are created before pods are creted. Pods contain the information required by creating Neutron ports. If pods are associated with the specific namespace, the ports are created and associated with the subnets for the namespace. In this case Raven also creates a LBaaS member for each port translated from the pod coordinating with the service translation. Although it's optional, Raven can emulate kube-proxy_. This is for the network controller that leverages isolated datapath from ``docker0`` bridge such as Open vSwitch datapath. Services contain the information for the emulation. Raven maps kube-proxy to Neutron load balancers with VIPs. For "externalIPs" type K8s service, Raven associates a floating IP with a load balancer for enabling the pubilc accesses. ================= ============= Kubernetes Neutron ================= ============= Namespace Network (Cluster subnet) (Subnet) Pod Port LBaaS Member Service LBaaS Pool LBaaS VIP (FloatingIP) ================= ============= .. _k8s-api-behaviour: K8s API behaviour ----------------- We look at the responses from the pod endpoints as an exmple. The following behaviour is based on the 1.2.0 release, which is the latest one as of March 17th, 2016. :: $ ./kubectl.sh version Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"clean"} Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"clean"} Regular requests ~~~~~~~~~~~~~~~~ If there's no pod, the K8s API server returns the following JSON response that has the empty list for the ``"items"`` property. :: $ curl -X GET -i http://127.0.0.1:8080/api/v1/pods HTTP/1.1 200 OK Content-Type: application/json Date: Tue, 15 Mar 2016 07:15:46 GMT Content-Length: 145 { "kind": "PodList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/pods", "resourceVersion": "227806" }, "items": [] } We deploy a pod as follow. :: $ ./kubectl.sh run --image=nginx nginx-app --port=80 replicationcontroller "nginx-app" created Then the response from the API server contains the pod information in ``"items"`` property of the JSON response. :: $ curl -X GET -i http://127.0.0.1:8080/api/v1/pods HTTP/1.1 200 OK Content-Type: application/json Date: Tue, 15 Mar 2016 08:18:25 GMT Transfer-Encoding: chunked { "kind": "PodList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/pods", "resourceVersion": "228211" }, "items": [ { "metadata": { "name": "nginx-app-o0kvl", "generateName": "nginx-app-", "namespace": "default", "selfLink": "/api/v1/namespaces/default/pods/nginx-app-o0kvl", "uid": "090cc0c8-ea84-11e5-8c79-42010af00003", "resourceVersion": "228094", "creationTimestamp": "2016-03-15T08:00:51Z", "labels": { "run": "nginx-app" }, "annotations": { "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"nginx-app\",\"uid\":\"090bfb57-ea84-11e5-8c79-42010af00003\",\"apiVersion\":\"v1\",\"resourceVersion\":\"228081\"}}\n" } }, "spec": { "volumes": [ { "name": "default-token-wpfjn", "secret": { "secretName": "default-token-wpfjn" } } ], "containers": [ { "name": "nginx-app", "image": "nginx", "ports": [ { "containerPort": 80, "protocol": "TCP" } ], "resources": {}, "volumeMounts": [ { "name": "default-token-wpfjn", "readOnly": true, "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount" } ], "terminationMessagePath": "/dev/termination-log", "imagePullPolicy": "Always" } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "serviceAccountName": "default", "serviceAccount": "default", "nodeName": "10.240.0.4", "securityContext": {} }, "status": { "phase": "Running", "conditions": [ { "type": "Ready", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2016-03-15T08:00:52Z" } ], "hostIP": "10.240.0.4", "podIP": "172.16.49.2", "startTime": "2016-03-15T08:00:51Z", "containerStatuses": [ { "name": "nginx-app", "state": { "running": { "startedAt": "2016-03-15T08:00:52Z" } }, "lastState": {}, "ready": true, "restartCount": 0, "image": "nginx", "imageID": "docker://sha256:af4b3d7d5401624ed3a747dc20f88e2b5e92e0ee9954aab8f1b5724d7edeca5e", "containerID": "docker://b97168314ad58404dbce7cb94291db7a976d2cb824b39e5864bf4bdaf27af255" } ] } } ] } We get the current snapshot of the requested resources with the regular requests against the K8s API server. Requests with ``watch=true`` query string ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ K8s provides the "watch" capability for the endpoints with ``/watch/`` prefix for the specific resource name, i.e., ``/api/v1/watch/pods``, or ``watch=true`` query string. If there's no pod, we get only the response header and the connection is kept open. :: $ curl -X GET -i http://127.0.0.1:8080/api/v1/pods?watch=true HTTP/1.1 200 OK Transfer-Encoding: chunked Date: Tue, 15 Mar 2016 08:00:09 GMT Content-Type: text/plain; charset=utf-8 Transfer-Encoding: chunked We create a pod as we did for the case without the ``watch=true`` query string. :: $ ./kubectl.sh run --image=nginx nginx-app --port=80 replicationcontroller "nginx-app" created Then we observe the JSON data corresponds to the event is given by each line. The event type is given in ``"type"`` property of the JSON data, i.e., ``"ADDED"``, ``"MODIFIED"`` and ``"DELETED"``. :: $ curl -X GET -i http://127.0.0.1:8080/api/v1/pods?watch=true HTTP/1.1 200 OK Transfer-Encoding: chunked Date: Tue, 15 Mar 2016 08:00:09 GMT Content-Type: text/plain; charset=utf-8 Transfer-Encoding: chunked {"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"nginx-app-o0kvl","generateName":"nginx-app-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/nginx-app-o0kvl","uid":"090cc0c8-ea84-11e5-8c79-42010af00003","resourceVersion":"228082","creationTimestamp":"2016-03-15T08:00:51Z","labels":{"run":"nginx-app"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"nginx-app\",\"uid\":\"090bfb57-ea84-11e5-8c79-42010af00003\",\"apiVersion\":\"v1\",\"resourceVersion\":\"228081\"}}\n"}},"spec":{"volumes":[{"name":"default-token-wpfjn","secret":{"secretName":"default-token-wpfjn"}}],"containers":[{"name":"nginx-app","image":"nginx","ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-wpfjn","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{}},"status":{"phase":"Pending"}}} {"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"nginx-app-o0kvl","generateName":"nginx-app-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/nginx-app-o0kvl","uid":"090cc0c8-ea84-11e5-8c79-42010af00003","resourceVersion":"228084","creationTimestamp":"2016-03-15T08:00:51Z","labels":{"run":"nginx-app"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"nginx-app\",\"uid\":\"090bfb57-ea84-11e5-8c79-42010af00003\",\"apiVersion\":\"v1\",\"resourceVersion\":\"228081\"}}\n"}},"spec":{"volumes":[{"name":"default-token-wpfjn","secret":{"secretName":"default-token-wpfjn"}}],"containers":[{"name":"nginx-app","image":"nginx","ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-wpfjn","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"10.240.0.4","securityContext":{}},"status":{"phase":"Pending"}}} {"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"nginx-app-o0kvl","generateName":"nginx-app-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/nginx-app-o0kvl","uid":"090cc0c8-ea84-11e5-8c79-42010af00003","resourceVersion":"228088","creationTimestamp":"2016-03-15T08:00:51Z","labels":{"run":"nginx-app"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"nginx-app\",\"uid\":\"090bfb57-ea84-11e5-8c79-42010af00003\",\"apiVersion\":\"v1\",\"resourceVersion\":\"228081\"}}\n"}},"spec":{"volumes":[{"name":"default-token-wpfjn","secret":{"secretName":"default-token-wpfjn"}}],"containers":[{"name":"nginx-app","image":"nginx","ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-wpfjn","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"10.240.0.4","securityContext":{}},"status":{"phase":"Pending","conditions":[{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2016-03-15T08:00:51Z","reason":"ContainersNotReady","message":"containers with unready status: [nginx-app]"}],"hostIP":"10.240.0.4","startTime":"2016-03-15T08:00:51Z","containerStatuses":[{"name":"nginx-app","state":{"waiting":{"reason":"ContainerCreating","message":"Image: nginx is ready, container is creating"}},"lastState":{},"ready":false,"restartCount":0,"image":"nginx","imageID":""}]}}} {"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"nginx-app-o0kvl","generateName":"nginx-app-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/nginx-app-o0kvl","uid":"090cc0c8-ea84-11e5-8c79-42010af00003","resourceVersion":"228094","creationTimestamp":"2016-03-15T08:00:51Z","labels":{"run":"nginx-app"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"nginx-app\",\"uid\":\"090bfb57-ea84-11e5-8c79-42010af00003\",\"apiVersion\":\"v1\",\"resourceVersion\":\"228081\"}}\n"}},"spec":{"volumes":[{"name":"default-token-wpfjn","secret":{"secretName":"default-token-wpfjn"}}],"containers":[{"name":"nginx-app","image":"nginx","ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-wpfjn","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"10.240.0.4","securityContext":{}},"status":{"phase":"Running","conditions":[{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2016-03-15T08:00:52Z"}],"hostIP":"10.240.0.4","podIP":"172.16.49.2","startTime":"2016-03-15T08:00:51Z","containerStatuses":[{"name":"nginx-app","state":{"running":{"startedAt":"2016-03-15T08:00:52Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"nginx","imageID":"docker://sha256:af4b3d7d5401624ed3a747dc20f88e2b5e92e0ee9954aab8f1b5724d7edeca5e","containerID":"docker://b97168314ad58404dbce7cb94291db7a976d2cb824b39e5864bf4bdaf27af255"}]}}} Raven Technical Design Overview ------------------------------- Problem Statement ~~~~~~~~~~~~~~~~~ To conform to the I/O bound requirement described in :ref:`k8s-api-behaviour`, the multiplexed concurrent network I/O is demanded. eventlet_ is used in various OpenStack projects for this purpose as well as other libraries such as Twisted_, Tornado_ and gevent_. However, it has problems as described in "`What's wrong with eventlet?`_" on the OpenStack wiki page. asyncio and Python 3 by default ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ asyncio_ was introduced as a standard asynchronous I/O library in Python 3.4. Its event loop and coroutines provide the mechanism to multiplex network I/O in the asynchronous fashion. Compared with eventlet, we can explicitly mark the I/O operations asynchronous with ``yield from`` or ``await`` introduced in Python 3.5. Trollius_ is a port of asyncio to Python 2.x. However `Trollius documentation`_ is describing a list of problems and even promoting the migration to Python 3 with asyncio. Kuryr is still a quite young project in OpenStack Neutron big tent. In addition to that, since it's a container related project it should be able to be run inside a container. So do Raven. Therefore we take a path to support for only Python 3 and drop Python 2. With asyncio we can achieve concurrent networking I/O operations required by watchers watch multiple endpoints and translate their responses into requests against Neutorn and K8s API server. Watchers ~~~~~~~~ A watcher can be represented as a pair of an API endpoint and a function used for the translation essentially. That is, the pair of what is translated and how it is. The API endpoint URI is associated with the stream of the event notifications and the translation function maps each event coming from the apiserver into another form such as the request against Neutron API server. Watchers can be considered as concerns and reactions. They should be decouped from the actual task dispatcher and their consumers. A single or multiple watchers can be mixed into the single class that leverages them, i.e., Raven, or even mutliple classes leverage them can have the same concern and the same reaction. The watchers can be able to be mixed into the single entity of the watcher user but they should work independently. For instance, ``AliceWatcher`` does its work and knows nothing about other watchers such as ``BobWatcher``. They don't work together depending on one or each. A minimum watcher can be defined as follow. .. code-block:: python from kuryr.raven import watchers class SomeWatcher(watchers.K8sApiWatcher): WATCH_ENDPOINT = '/' def translate(self, deserialized_json): pass The watcher is defined in the declarative way and ideally doesn't care when it is called and by whom. However, it needs to recongnize the context such as the event type and behave appropriately according to the situation. Raven ~~~~~ Raven acts as a daemon and it should be able to be started or stopped by operators. It delegates the actual watch tasks to the watchers and dispatch them with the single JSON response corresponds to each endpoint on which the watcher has its concern. Hence, Raven holds one or multiple watchers, opens connections for each endpoint, makes HTTP requests, gets HTTP responses and parses every event notification and dispatches the translate methods of the watchers routed based on their corresponding endpoints. To register the watchers to Raven or any class, ``register_watchers`` decorator is used. It simply inserts the watchers into the dictionary in the class, ``WATCH_ENDPOINTS_AND_CALLBACKS`` and it's up to the class how use the registerd watchers. The classes passed to ``register_watchers`` are defined in the configuration file and you can specify only what you need. In the case of Raven, it starts the event loop, open connections for each registered watcher and keeps feeding the notified events to the translate methods of the watchers. Raven is a service implements ``oslo_service.service.Service``. When ``start`` method is called, it starts the event loop and delegatations of the watch tasks. If ``SIGINT`` or ``SIGTERM`` signal is sent to Raven, it cancells all watch tasks, closes connections and stops immediately. Otherwise Raven lets watchers keep watching the API endpionts until the API server sends EOF strings. When ``stop`` is called, it cancells all watch tasks, closes connections and stops as well. Ideally, the translate method can be a pure function that doesn't depend on the user of the watcher. However, the translation gets involved in requests against Neutron and possibly the K8s API server. And it depends on the Neutron client that shall be shared among the watchers. Hence, Raven calls the translate methods of the watchers binding itself to ``self``. That is, Raven can propagate its contexts to the watchers and in this way watchers can share the same contexts. However, it's responsibilty of the writer of the watchers to track which variables are defined in Raven and what they are. Appendix A: JSON response from the apiserver for each resource -------------------------------------------------------------- Namespace ~~~~~~~~~ :: /api/v1/namespaces?watch=true ADDED +++++ :: { "type": "ADDED", "object": { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "test", "selfLink": "/api/v1/namespaces/test", "uid": "f094ea6b-06c2-11e6-8128-42010af00003", "resourceVersion": "497821", "creationTimestamp": "2016-04-20T06:41:41Z" }, "spec": { "finalizers": [ "kubernetes" ] }, "status": { "phase": "Active" } } } MODIFIED ++++++++ :: { "type": "MODIFIED", "object": { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "test", "selfLink": "/api/v1/namespaces/test", "uid": "f094ea6b-06c2-11e6-8128-42010af00003", "resourceVersion": "519095", "creationTimestamp": "2016-04-20T06:41:41Z", "deletionTimestamp": "2016-04-21T08:47:53Z" }, "spec": { "finalizers": [ "kubernetes" ] }, "status": { "phase": "Terminating" } } } DELETED +++++++ :: { "type": "DELETED", "object": { "kind": "Namespace", "apiVersion": "v1", "metadata": { "name": "test", "selfLink": "/api/v1/namespaces/test", "uid": "f094ea6b-06c2-11e6-8128-42010af00003", "resourceVersion": "519099", "creationTimestamp": "2016-04-20T06:41:41Z", "deletionTimestamp": "2016-04-21T08:47:53Z" }, "spec": {}, "status": { "phase": "Terminating" } } } Pod ~~~ :: /api/v1/pods?watch=true ADDED +++++ :: { "type": "ADDED", "object": { "kind": "Pod", "apiVersion": "v1", "metadata": { "name": "my-nginx-y67ky", "generateName": "my-nginx-", "namespace": "default", "selfLink": "/api/v1/namespaces/default/pods/my-nginx-y67ky", "uid": "d42b0bb2-dc4e-11e5-8c79-42010af00003", "resourceVersion": "63355", "creationTimestamp": "2016-02-26T06:04:42Z", "labels": { "run": "my-nginx" }, "annotations": { "kubernetes.io/created-by": { "kind": "SerializedReference", "apiVersion": "v1", "reference": { "kind": "ReplicationController", "namespace": "default", "name": "my-nginx", "uid": "d42a4ee1-dc4e-11e5-8c79-42010af00003", "apiVersion": "v1", "resourceVersion": "63348" } } } }, "spec": { "volumes": [ { "name": "default-token-wpfjn", "secret": { "secretName": "default-token-wpfjn" } } ], "containers": [ { "name": "my-nginx", "image": "nginx", "ports": [ { "containerPort": 80, "protocol": "TCP" } ], "resources": {}, "volumeMounts": [ { "name": "default-token-wpfjn", "readOnly": true, "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount" } ], "terminationMessagePath": "/dev/termination-log", "imagePullPolicy": "Always" } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "serviceAccountName": "default", "serviceAccount": "default", "nodeName": "10.240.0.4", "securityContext": {} }, "status": { "phase": "Pending", "conditions": [ { "type": "Ready", "status": "False", "lastProbeTime": null, "lastTransitionTime": "2016-02-26T06:04:43Z", "reason": "ContainersNotReady", "message": "containers with unready status: [my-nginx]" } ], "hostIP": "10.240.0.4", "startTime": "2016-02-26T06:04:43Z", "containerStatuses": [ { "name": "my-nginx", "state": { "waiting": { "reason": "ContainerCreating", "message": "Image: nginx is ready, container is creating" } }, "lastState": {}, "ready": false, "restartCount": 0, "image": "nginx", "imageID": "" } ] } } } MODIFIED ~~~~~~~~ :: { "type": "MODIFIED", "object": { "kind": "Pod", "apiVersion": "v1", "metadata": { "name": "my-nginx-y67ky", "generateName": "my-nginx-", "namespace": "default", "selfLink": "/api/v1/namespaces/default/pods/my-nginx-y67ky", "uid": "d42b0bb2-dc4e-11e5-8c79-42010af00003", "resourceVersion": "63425", "creationTimestamp": "2016-02-26T06:04:42Z", "deletionTimestamp": "2016-02-26T06:06:16Z", "deletionGracePeriodSeconds": 30, "labels": { "run": "my-nginx" }, "annotations": { "kubernetes.io/created-by": { "kind": "SerializedReference", "apiVersion": "v1", "reference": { "kind": "ReplicationController", "namespace": "default", "name": "my-nginx", "uid": "d42a4ee1-dc4e-11e5-8c79-42010af00003", "apiVersion": "v1", "resourceVersion": "63348" } } } }, "spec": { "volumes": [ { "name": "default-token-wpfjn", "secret": { "secretName": "default-token-wpfjn" } } ], "containers": [ { "name": "my-nginx", "image": "nginx", "ports": [ { "containerPort": 80, "protocol": "TCP" } ], "resources": {}, "volumeMounts": [ { "name": "default-token-wpfjn", "readOnly": true, "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount" } ], "terminationMessagePath": "/dev/termination-log", "imagePullPolicy": "Always" } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "serviceAccountName": "default", "serviceAccount": "default", "nodeName": "10.240.0.4", "securityContext": {} }, "status": { "phase": "Pending", "conditions": [ { "type": "Ready", "status": "False", "lastProbeTime": null, "lastTransitionTime": "2016-02-26T06:04:43Z", "reason": "ContainersNotReady", "message": "containers with unready status: [my-nginx]" } ], "hostIP": "10.240.0.4", "startTime": "2016-02-26T06:04:43Z", "containerStatuses": [ { "name": "my-nginx", "state": { "waiting": { "reason": "ContainerCreating", "message": "Image: nginx is ready, container is creating" } }, "lastState": {}, "ready": false, "restartCount": 0, "image": "nginx", "imageID": "" } ] } } } DELETED +++++++ :: { "type": "DELETED", "object": { "kind": "Pod", "apiVersion": "v1", "metadata": { "name": "my-nginx-y67ky", "generateName": "my-nginx-", "namespace": "default", "selfLink": "/api/v1/namespaces/default/pods/my-nginx-y67ky", "uid": "d42b0bb2-dc4e-11e5-8c79-42010af00003", "resourceVersion": "63431", "creationTimestamp": "2016-02-26T06:04:42Z", "deletionTimestamp": "2016-02-26T06:05:46Z", "deletionGracePeriodSeconds": 0, "labels": { "run": "my-nginx" }, "annotations": { "kubernetes.io/created-by": { "kind": "SerializedReference", "apiVersion": "v1", "reference": { "kind": "ReplicationController", "namespace": "default", "name": "my-nginx", "uid": "d42a4ee1-dc4e-11e5-8c79-42010af00003", "apiVersion": "v1", "resourceVersion": "63348" } } } }, "spec": { "volumes": [ { "name": "default-token-wpfjn", "secret": { "secretName": "default-token-wpfjn" } } ], "containers": [ { "name": "my-nginx", "image": "nginx", "ports": [ { "containerPort": 80, "protocol": "TCP" } ], "resources": {}, "volumeMounts": [ { "name": "default-token-wpfjn", "readOnly": true, "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount" } ], "terminationMessagePath": "/dev/termination-log", "imagePullPolicy": "Always" } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "serviceAccountName": "default", "serviceAccount": "default", "nodeName": "10.240.0.4", "securityContext": {} }, "status": { "phase": "Pending", "conditions": [ { "type": "Ready", "status": "False", "lastProbeTime": null, "lastTransitionTime": "2016-02-26T06:04:43Z", "reason": "ContainersNotReady", "message": "containers with unready status: [my-nginx]" } ], "hostIP": "10.240.0.4", "startTime": "2016-02-26T06:04:43Z", "containerStatuses": [ { "name": "my-nginx", "state": { "waiting": { "reason": "ContainerCreating", "message": "Image: nginx is ready, container is creating" } }, "lastState": {}, "ready": false, "restartCount": 0, "image": "nginx", "imageID": "" } ] } } } Service ~~~~~~~ :: /api/v1/services?watch=true ADDED +++++ :: { "type": "ADDED", "object": { "kind": "Service", "apiVersion": "v1", "metadata": { "name": "redis-master", "namespace": "default", "selfLink": "/api/v1/namespaces/default/services/redis-master", "uid": "7aecfdac-d54c-11e5-8cc5-42010af00002", "resourceVersion": "2074", "creationTimestamp": "2016-02-17T08:00:16Z", "labels": { "app": "redis", "role": "master", "tier": "backend" } }, "spec": { "ports": [ { "protocol": "TCP", "port": 6379, "targetPort": 6379 } ], "selector": { "app": "redis", "role": "master", "tier": "backend" }, "clusterIP": "10.0.0.102", "type": "ClusterIP", "sessionAffinity": "None" }, "status": { "loadBalancer": {} } } } MODIFIED ++++++++ The event could not be observed. DELETED +++++++ :: { "type": "DELETED", "object": { "kind": "Service", "apiVersion": "v1", "metadata": { "name": "redis-master", "namespace": "default", "selfLink": "/api/v1/namespaces/default/services/redis-master", "uid": "7aecfdac-d54c-11e5-8cc5-42010af00002", "resourceVersion": "2806", "creationTimestamp": "2016-02-17T08:00:16Z", "labels": { "app": "redis", "role": "master", "tier": "backend" } }, "spec": { "ports": [ { "protocol": "TCP", "port": 6379, "targetPort": 6379 } ], "selector": { "app": "redis", "role": "master", "tier": "backend" }, "clusterIP": "10.0.0.102", "type": "ClusterIP", "sessionAffinity": "None" }, "status": { "loadBalancer": {} } } } .. _`Kubernetes API`: http://kubernetes.io/docs/api/ .. _CNI: https://github.com/appc/cni .. _`in the deployment phase`: https://github.com/kubernetes/kubernetes/search?utf8=%E2%9C%93&q=FLANNEL_NET .. _kube-proxy: http://kubernetes.io/docs/user-guide/services/#virtual-ips-and-service-proxies .. _eventlet: http://eventlet.net/ .. _Twisted: https://twistedmatrix.com/trac/ .. _Tornado: http://tornadoweb.org/ .. _gevent: http://www.gevent.org/ .. _`What's wrong with eventlet?`: https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio#What.27s_wrong_with_eventlet.3F .. _asyncio: https://www.python.org/dev/peps/pep-3156/ .. _Trollius: http://trollius.readthedocs.org/ .. _`Trollius documentation`: http://trollius.readthedocs.org/deprecated.html