8c2433a2c4
This adds methods to allow us to save and restore spans using ZooKeeper data. Additionally, we subclass the tracing.Span class so that we can transparently handle timestamps which are stored as floating point numbers rather than integer nanoseconds. To exercise the new features, emit spans for QueueItems and BuildSets. Because most of our higher-level (parent) spans may start on one host and end on another, we save the full information about the span in ZK and restore it whenever we do anything with it, including starting child spans. This works well for starting a Build span given a BuildSet, since both objects are used by the executor client and so the span information for both is available. However, there are cases where we would like to have child spans and we do not have the full information of the parent, such as any children of the Build span on the executor. We could duplicate all the information of the Build span in ZK and send it along with the build request, but we really only need a few bits of info to start a remote child span. In OpenTelemetry, this is called trace propogation, and there are some tools for this which assume that the implicit trace context is being used and formats information for an HTTP header. We could use those methods, but this change adds a simpler API that is well suited to our typical json-serialization method of propogation. To use it, we will add a small extra dictionary to build and merge requests. This should serialize to about 104 bytes. So that we can transparantly handle upgrades from having no saved state for spans and span context in our ZK data, have our tracing API return a NonRecordingSpan when we try to restore from a None value. This code uses tracing.INVALID_SPAN or tracing.INVALID_SPAN_CONTEXT which are suitable constants. They are sufficiently real for the purpose of context managers, etc. The only down side is that any child spans of these will be real, actual reported spans, so in these cases, we will emit what we intend to be child spans as actual parent traces. Since this should only happen when a user first enables tracing on an already existing system, that seems like a reasonable trade-off. As new objects are populated, the spans will be emitted as expected. The trade off here is that our operational code can be much simpler as we can avoid null value checks and any confusion regarding context managers. In particular, we can just assume that tracing spans and contexts are always valid. Change-Id: If55b06572b5e95f8c21611b2a3c23f7fd224a547
59 lines
1.9 KiB
Python
59 lines
1.9 KiB
Python
# Copyright 2022 Acme Gating, LLC
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
# not use this file except in compliance with the License. You may obtain
|
|
# a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
# License for the specific language governing permissions and limitations
|
|
# under the License.
|
|
from concurrent import futures
|
|
|
|
import fixtures
|
|
import grpc
|
|
from opentelemetry import trace
|
|
from opentelemetry.proto.collector.trace.v1.trace_service_pb2_grpc import (
|
|
TraceServiceServicer,
|
|
add_TraceServiceServicer_to_server
|
|
)
|
|
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import (
|
|
ExportTraceServiceResponse,
|
|
)
|
|
|
|
|
|
class TraceServer(TraceServiceServicer):
|
|
def __init__(self, fixture):
|
|
super().__init__()
|
|
self.fixture = fixture
|
|
|
|
def Export(self, request, context):
|
|
self.fixture.requests.append(request)
|
|
return ExportTraceServiceResponse()
|
|
|
|
|
|
class OTLPFixture(fixtures.Fixture):
|
|
def __init__(self):
|
|
super().__init__()
|
|
self.requests = []
|
|
self.executor = futures.ThreadPoolExecutor(
|
|
thread_name_prefix='OTLPFixture',
|
|
max_workers=10)
|
|
self.server = grpc.server(self.executor)
|
|
add_TraceServiceServicer_to_server(TraceServer(self), self.server)
|
|
self.port = self.server.add_insecure_port('[::]:0')
|
|
# Reset global tracer provider
|
|
trace._TRACER_PROVIDER_SET_ONCE = trace.Once()
|
|
trace._TRACER_PROVIDER = None
|
|
|
|
def _setUp(self):
|
|
self.server.start()
|
|
|
|
def _cleanup(self):
|
|
self.server.stop()
|
|
self.server.wait_for_termination()
|
|
self.executor.shutdown()
|