Skip to content
Draft
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions docs/export-unsampled-spans.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Export Unsampled Spans Feature

The OpenTelemetry C++ SDK supports an opt-in feature to export unsampled but recording spans from trace processors. This allows collectors to see the full request volume, which is useful for calculating accurate metrics and performing tail-based sampling.

## Overview

By default, the C++ SDK only exports sampled spans (`Decision::RECORD_AND_SAMPLE`). With this feature enabled, spans with `Decision::RECORD_ONLY` (unsampled but recording) will also be exported.

## Usage

### BatchSpanProcessor

```cpp
#include "opentelemetry/sdk/trace/batch_span_processor.h"
#include "opentelemetry/sdk/trace/batch_span_processor_options.h"

// Configure BatchSpanProcessor to export unsampled spans
opentelemetry::sdk::trace::BatchSpanProcessorOptions options;
options.export_unsampled_spans = true; // Default is false

auto processor = std::make_unique<opentelemetry::sdk::trace::BatchSpanProcessor>(
std::move(exporter), options);
```

### SimpleSpanProcessor

```cpp
#include "opentelemetry/sdk/trace/simple_processor.h"
#include "opentelemetry/sdk/trace/simple_processor_options.h"

// Configure SimpleSpanProcessor to export unsampled spans
opentelemetry::sdk::trace::SimpleSpanProcessorOptions options;
options.export_unsampled_spans = true; // Default is false

auto processor = std::make_unique<opentelemetry::sdk::trace::SimpleSpanProcessor>(
std::move(exporter), options);
```

## Behavior

When `export_unsampled_spans` is:

- **false** (default): Only sampled spans are exported
- **true**: Both sampled and unsampled recording spans are exported

## Sampling Decisions

The feature respects the sampling decisions made during span creation:

- `Decision::DROP`: Spans are not recorded and never reach the processor (not affected)
- `Decision::RECORD_ONLY`: Spans are recorded but not sampled by default (affected by this feature)
- `Decision::RECORD_AND_SAMPLE`: Spans are recorded and sampled (always exported)

## Backward Compatibility

This feature maintains full backward compatibility:

- Default behavior is unchanged (only sampled spans are exported)
- Existing constructors continue to work as before
- Test spans with invalid contexts are always exported for compatibility

## Example

See [examples/unsampled_spans_demo.cc](../examples/unsampled_spans_demo.cc) for a complete working example demonstrating the feature.
137 changes: 137 additions & 0 deletions examples/unsampled_spans_demo.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

// Example demonstrating export_unsampled_spans feature
// This example shows how to configure span processors to export unsampled spans

#include <iostream>
#include <memory>

#include "opentelemetry/sdk/trace/batch_span_processor.h"
#include "opentelemetry/sdk/trace/batch_span_processor_options.h"
#include "opentelemetry/sdk/trace/simple_processor.h"
#include "opentelemetry/sdk/trace/simple_processor_options.h"
#include "opentelemetry/sdk/trace/span_data.h"
#include "opentelemetry/trace/span_context.h"
#include "opentelemetry/trace/trace_flags.h"

namespace trace_sdk = opentelemetry::sdk::trace;
namespace trace_api = opentelemetry::trace;

// Simple mock exporter for demonstration
class MockExporter : public trace_sdk::SpanExporter
{
public:
explicit MockExporter(const std::string &name) : name_(name) {}

std::unique_ptr<trace_sdk::Recordable> MakeRecordable() noexcept override
{
return std::make_unique<trace_sdk::SpanData>();
}

opentelemetry::sdk::common::ExportResult Export(
const opentelemetry::nostd::span<std::unique_ptr<trace_sdk::Recordable>>
&recordables) noexcept override
{
std::cout << name_ << " exported " << recordables.size() << " spans" << std::endl;
return opentelemetry::sdk::common::ExportResult::kSuccess;
}

bool ForceFlush(std::chrono::microseconds) noexcept override { return true; }
bool Shutdown(std::chrono::microseconds) noexcept override { return true; }

private:
std::string name_;
};

// Create a test span with specific sampling status
std::unique_ptr<trace_sdk::SpanData> CreateTestSpan(bool sampled)
{
auto span = std::make_unique<trace_sdk::SpanData>();
span->SetName("test_span");

// Set up valid context with proper sampling
trace_api::TraceFlags flags(sampled ? trace_api::TraceFlags::kIsSampled : 0);
uint8_t trace_id_bytes[16] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16};
uint8_t span_id_bytes[8] = {1, 2, 3, 4, 5, 6, 7, 8};

trace_api::TraceId trace_id{opentelemetry::nostd::span<const uint8_t, 16>(trace_id_bytes)};
trace_api::SpanId span_id{opentelemetry::nostd::span<const uint8_t, 8>(span_id_bytes)};

trace_api::SpanContext context(trace_id, span_id, flags, false);
span->SetIdentity(context, trace_api::SpanId());

return span;
}

int main()
{
std::cout << "OpenTelemetry C++ Export Unsampled Spans Demo\n" << std::endl;

// Example 1: BatchSpanProcessor without export_unsampled_spans (default behavior)
{
std::cout << "=== Example 1: BatchSpanProcessor (default behavior) ===" << std::endl;

trace_sdk::BatchSpanProcessorOptions options;
// export_unsampled_spans is false by default

auto processor = std::make_unique<trace_sdk::BatchSpanProcessor>(
std::make_unique<MockExporter>("BatchProcessor-Default"), options);

// Create one sampled and one unsampled span
auto sampled_span = CreateTestSpan(true);
auto unsampled_span = CreateTestSpan(false);

processor->OnEnd(std::unique_ptr<trace_sdk::Recordable>(sampled_span.release()));
processor->OnEnd(std::unique_ptr<trace_sdk::Recordable>(unsampled_span.release()));

processor->ForceFlush();
std::cout << "Expected: Only 1 span exported (sampled span only)\n" << std::endl;
}

// Example 2: BatchSpanProcessor with export_unsampled_spans enabled
{
std::cout << "=== Example 2: BatchSpanProcessor (export_unsampled_spans = true) ==="
<< std::endl;

trace_sdk::BatchSpanProcessorOptions options;
options.export_unsampled_spans = true; // Enable exporting unsampled spans

auto processor = std::make_unique<trace_sdk::BatchSpanProcessor>(
std::make_unique<MockExporter>("BatchProcessor-WithUnsampled"), options);

// Create one sampled and one unsampled span
auto sampled_span = CreateTestSpan(true);
auto unsampled_span = CreateTestSpan(false);

processor->OnEnd(std::unique_ptr<trace_sdk::Recordable>(sampled_span.release()));
processor->OnEnd(std::unique_ptr<trace_sdk::Recordable>(unsampled_span.release()));

processor->ForceFlush();
std::cout << "Expected: 2 spans exported (both sampled and unsampled)\n" << std::endl;
}

// Example 3: SimpleSpanProcessor with export_unsampled_spans enabled
{
std::cout << "=== Example 3: SimpleSpanProcessor (export_unsampled_spans = true) ==="
<< std::endl;

trace_sdk::SimpleSpanProcessorOptions options;
options.export_unsampled_spans = true; // Enable exporting unsampled spans

auto processor = std::make_unique<trace_sdk::SimpleSpanProcessor>(
std::make_unique<MockExporter>("SimpleProcessor-WithUnsampled"), options);

// Create one sampled and one unsampled span
auto sampled_span = CreateTestSpan(true);
auto unsampled_span = CreateTestSpan(false);

processor->OnEnd(std::unique_ptr<trace_sdk::Recordable>(sampled_span.release()));
processor->OnEnd(std::unique_ptr<trace_sdk::Recordable>(unsampled_span.release()));

std::cout << "Expected: 2 separate exports (one for each span)\n" << std::endl;
}

std::cout << "Demo completed!" << std::endl;
return 0;
}
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ class BatchSpanProcessor : public SpanProcessor
const size_t max_queue_size_;
const std::chrono::milliseconds schedule_delay_millis_;
const size_t max_export_batch_size_;
const bool export_unsampled_spans_;

/* The buffer/queue to which the ended spans are added */
opentelemetry::sdk::common::CircularBuffer<Recordable> buffer_;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@ struct BatchSpanProcessorOptions
* equal to max_queue_size.
*/
size_t max_export_batch_size = 512;

/**
* Whether to export unsampled but recording spans.
* By default, only sampled spans (Decision::RECORD_AND_SAMPLE) are exported.
* When set to true, unsampled recording spans (Decision::RECORD_ONLY) are also exported.
*/
bool export_unsampled_spans = false;
};

} // namespace trace
Expand Down
31 changes: 30 additions & 1 deletion sdk/include/opentelemetry/sdk/trace/simple_processor.h
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
#include "opentelemetry/sdk/trace/exporter.h"
#include "opentelemetry/sdk/trace/processor.h"
#include "opentelemetry/sdk/trace/recordable.h"
#include "opentelemetry/sdk/trace/simple_processor_options.h"
#include "opentelemetry/sdk/trace/span_data.h"
#include "opentelemetry/trace/span_context.h"
#include "opentelemetry/version.h"

Expand All @@ -40,7 +42,17 @@ class SimpleSpanProcessor : public SpanProcessor
* @param exporter the exporter used by the span processor
*/
explicit SimpleSpanProcessor(std::unique_ptr<SpanExporter> &&exporter) noexcept
: exporter_(std::move(exporter))
: exporter_(std::move(exporter)), export_unsampled_spans_(false)
{}

/**
* Initialize a simple span processor with options.
* @param exporter the exporter used by the span processor
* @param options the processor options
*/
explicit SimpleSpanProcessor(std::unique_ptr<SpanExporter> &&exporter,
const SimpleSpanProcessorOptions &options) noexcept
: exporter_(std::move(exporter)), export_unsampled_spans_(options.export_unsampled_spans)
{}

std::unique_ptr<Recordable> MakeRecordable() noexcept override
Expand All @@ -54,6 +66,22 @@ class SimpleSpanProcessor : public SpanProcessor

void OnEnd(std::unique_ptr<Recordable> &&span) noexcept override
{
// Check if we should export this span based on sampling status
auto *span_data = static_cast<SpanData *>(span.get());
const auto &span_context = span_data->GetSpanContext();

// For backward compatibility: always export spans with invalid context (e.g., test spans)
// For valid contexts: export sampled spans or unsampled spans if export_unsampled_spans is
// enabled
bool should_export =
!span_context.IsValid() || span_context.IsSampled() || export_unsampled_spans_;

if (!should_export)
{
// Drop unsampled spans if export_unsampled_spans is not enabled
return;
}

nostd::span<std::unique_ptr<Recordable>> batch(&span, 1);
const std::lock_guard<opentelemetry::common::SpinLockMutex> locked(lock_);
if (exporter_->Export(batch) == sdk::common::ExportResult::kFailure)
Expand Down Expand Up @@ -89,6 +117,7 @@ class SimpleSpanProcessor : public SpanProcessor

private:
std::unique_ptr<SpanExporter> exporter_;
const bool export_unsampled_spans_;
opentelemetry::common::SpinLockMutex lock_;
#if defined(__cpp_lib_atomic_value_initialization) && \
__cpp_lib_atomic_value_initialization >= 201911L
Expand Down
30 changes: 30 additions & 0 deletions sdk/include/opentelemetry/sdk/trace/simple_processor_options.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

#pragma once

#include "opentelemetry/version.h"

OPENTELEMETRY_BEGIN_NAMESPACE
namespace sdk
{

namespace trace
{

/**
* Struct to hold simple SpanProcessor options.
*/
struct SimpleSpanProcessorOptions
{
/**
* Whether to export unsampled but recording spans.
* By default, only sampled spans (Decision::RECORD_AND_SAMPLE) are exported.
* When set to true, unsampled recording spans (Decision::RECORD_ONLY) are also exported.
*/
bool export_unsampled_spans = false;
};

} // namespace trace
} // namespace sdk
OPENTELEMETRY_END_NAMESPACE
19 changes: 19 additions & 0 deletions sdk/src/trace/batch_span_processor.cc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
#include "opentelemetry/sdk/trace/exporter.h"
#include "opentelemetry/sdk/trace/processor.h"
#include "opentelemetry/sdk/trace/recordable.h"
#include "opentelemetry/sdk/trace/span_data.h"
#include "opentelemetry/version.h"

#ifdef ENABLE_THREAD_INSTRUMENTATION_PREVIEW
Expand All @@ -47,6 +48,7 @@ BatchSpanProcessor::BatchSpanProcessor(std::unique_ptr<SpanExporter> &&exporter,
max_queue_size_(options.max_queue_size),
schedule_delay_millis_(options.schedule_delay_millis),
max_export_batch_size_(options.max_export_batch_size),
export_unsampled_spans_(options.export_unsampled_spans),
buffer_(max_queue_size_),
synchronization_data_(std::make_shared<SynchronizationData>()),
worker_thread_instrumentation_(nullptr),
Expand All @@ -63,6 +65,7 @@ BatchSpanProcessor::BatchSpanProcessor(std::unique_ptr<SpanExporter> &&exporter,
max_queue_size_(options.max_queue_size),
schedule_delay_millis_(options.schedule_delay_millis),
max_export_batch_size_(options.max_export_batch_size),
export_unsampled_spans_(options.export_unsampled_spans),
buffer_(max_queue_size_),
synchronization_data_(std::make_shared<SynchronizationData>()),
worker_thread_instrumentation_(runtime_options.thread_instrumentation),
Expand All @@ -89,6 +92,22 @@ void BatchSpanProcessor::OnEnd(std::unique_ptr<Recordable> &&span) noexcept
return;
}

// Check if we should export this span based on sampling status
auto *span_data = static_cast<SpanData *>(span.get());
const auto &span_context = span_data->GetSpanContext();

// For backward compatibility: always export spans with invalid context (e.g., test spans)
// For valid contexts: export sampled spans or unsampled spans if export_unsampled_spans is
// enabled
bool should_export =
!span_context.IsValid() || span_context.IsSampled() || export_unsampled_spans_;

if (!should_export)
{
// Drop unsampled spans if export_unsampled_spans is not enabled
return;
}

Comment on lines +95 to +110
Copy link

@agent-adam agent-adam Aug 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn’t look like there was an IsSampled() check before this PR — so were we effectively exporting all recording spans (sampled + unsampled) to the exporter before this change?

Copy link
Member

@lalitb lalitb Sep 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so, this is what I see from the code:
What the Spec Says:

  • RECORD_ONLY spans (IsRecording=true, Sampled=false) should go to Span Processors but NOT to Span Exporters
  • Only RECORD_AND_SAMPLE spans should reach exporters

What the C++ SDK Does:

  • RECORD_ONLY spans create real spans with recordables
  • These spans go through the processor pipeline via OnEnd()
  • The processors (like BatchSpanProcessor) pass them to exporters via exporter_->Export()

I believe we can use this PR to fix that too. @copilot please see if we are compliant with specs as indicated above with this PR.

if (buffer_.Add(std::move(span)) == false)
{
OTEL_INTERNAL_LOG_WARN("BatchSpanProcessor queue is full - dropping span.");
Expand Down
3 changes: 2 additions & 1 deletion sdk/test/trace/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ foreach(
parent_sampler_test
trace_id_ratio_sampler_test
batch_span_processor_test
tracer_config_test)
tracer_config_test
unsampled_span_processor_test)
add_executable(${testname} "${testname}.cc")
target_link_libraries(
${testname}
Expand Down
Loading
Loading