Skip to content

Cosmos DB partial integration testing #7752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Cosmos DB partial integration testing #7752

wants to merge 9 commits into from

Conversation

@pablomartinezbernardo
Copy link
Contributor

@pablomartinezbernardo pablomartinezbernardo commented Nov 3, 2025 •
edited
Loading

Summary of changes

Integration test Cosmos DB instrumentation with mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview.

Due to incomplete support in the emulator, only item queries can be currently tested.

Reason for change

No integration tests exist for Cosmos DB. This PR changes it using the under development Linux-based emulator. Compatibility is limited, but it supports items operations, which are arguably used the most. The emulator will also help development of future Cosmos DB support.

Implementation details

Keep current test to keep supporting full manual testing, while doing specific testing for CI.

@pr-commenter
Copy link

pr-commenter bot commented Nov 3, 2025 •
edited
Loading

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7752 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.887
  • 13 benchmarks have fewer allocations
  • 7 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472 6.06 KB 6.12 KB 55 B 0.91%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net6.0 5.51 KB 5.55 KB 35 B 0.64%

Fewer allocations 🎉 in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1 5.68 KB 5.62 KB -58 B -1.02%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.7μs 58.8ns 338ns 0 0 0 5.51 KB
master StartStopWithChild netcoreapp3.1 14.3μs 71.8ns 337ns 0 0 0 5.68 KB
master StartStopWithChild net472 22μs 124ns 792ns 1 0.334 0.111 6.06 KB
#7752 StartStopWithChild net6.0 10.3μs 28.5ns 103ns 0 0 0 5.55 KB
#7752 StartStopWithChild netcoreapp3.1 13.4μs 70.1ns 336ns 0 0 0 5.62 KB
#7752 StartStopWithChild net472 21.9μs 121ns 744ns 0.965 0.322 0 6.12 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 3.35 KB 3.31 KB -46 B -1.37%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 952μs 280ns 1.08μs 0 0 0 2.72 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 170ns 660ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.2ms 77ns 278ns 0 0 0 3.35 KB
#7752 WriteAndFlushEnrichedTraces net6.0 934μs 63.6ns 246ns 0 0 0 2.7 KB
#7752 WriteAndFlushEnrichedTraces netcoreapp3.1 1.04ms 242ns 939ns 0 0 0 2.7 KB
#7752 WriteAndFlushEnrichedTraces net472 1.19ms 193ns 746ns 0 0 0 3.31 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Unknown 🤷 Fewer allocations 🎉

Fewer allocations 🎉 in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net6.0 1.22 KB 0 b -1.22 KB -100.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑netcoreapp3.1 1.2 KB 0 b -1.2 KB -100.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 1.23 KB 0 b -1.23 KB -100.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0 4.72 KB 0 b -4.72 KB -100.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑netcoreapp3.1 4.62 KB 0 b -4.62 KB -100.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472 4.74 KB 0 b -4.74 KB -100.00%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 1.09μs 6.29ns 50.3ns 0 0 0 1.22 KB
master AllCycleSimpleBody netcoreapp3.1 1.45μs 8.23ns 57ns 0 0 0 1.2 KB
master AllCycleSimpleBody net472 1.05μs 0.276ns 1.03ns 0.191 0 0 1.23 KB
master AllCycleMoreComplexBody net6.0 6.99μs 38.6ns 222ns 0 0 0 4.72 KB
master AllCycleMoreComplexBody netcoreapp3.1 8.98μs 22.8ns 88.3ns 0 0 0 4.62 KB
master AllCycleMoreComplexBody net472 7.64μs 6.04ns 23.4ns 0.729 0 0 4.74 KB
master ObjectExtractorSimpleBody net6.0 334ns 0.361ns 1.4ns 0 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 400ns 1.93ns 8.2ns 0 0 0 272 B
master ObjectExtractorSimpleBody net472 296ns 0.0186ns 0.0644ns 0.0434 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 6.33μs 4.04ns 15.6ns 0 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 7.72μs 36.4ns 150ns 0 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 6.71μs 2.01ns 7.52ns 0.574 0 0 3.8 KB
#7752 AllCycleSimpleBody net6.0 N/A N/A N/A NaN NaN NaN 0 b
#7752 AllCycleSimpleBody netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#7752 AllCycleSimpleBody net472 N/A N/A N/A NaN NaN NaN 0 b
#7752 AllCycleMoreComplexBody net6.0 N/A N/A N/A NaN NaN NaN 0 b
#7752 AllCycleMoreComplexBody netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#7752 AllCycleMoreComplexBody net472 N/A N/A N/A NaN NaN NaN 0 b
#7752 ObjectExtractorSimpleBody net6.0 315ns 1.73ns 9.81ns 0 0 0 280 B
#7752 ObjectExtractorSimpleBody netcoreapp3.1 397ns 2.2ns 14.3ns 0 0 0 272 B
#7752 ObjectExtractorSimpleBody net472 296ns 0.0241ns 0.0868ns 0.0433 0 0 281 B
#7752 ObjectExtractorMoreComplexBody net6.0 6.32μs 22.5ns 87.2ns 0 0 0 3.78 KB
#7752 ObjectExtractorMoreComplexBody netcoreapp3.1 7.78μs 30.6ns 114ns 0 0 0 3.69 KB
#7752 ObjectExtractorMoreComplexBody net472 6.61μs 1.03ns 3.86ns 0.596 0 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 77.4μs 32.7ns 118ns 0 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 97.8μs 36.1ns 140ns 0 0 0 32.4 KB
master EncodeArgs net472 110μs 12.1ns 46.8ns 4.96 0 0 32.51 KB
master EncodeLegacyArgs net6.0 146μs 194ns 752ns 0 0 0 2.15 KB
master EncodeLegacyArgs netcoreapp3.1 197μs 168ns 650ns 0 0 0 2.15 KB
master EncodeLegacyArgs net472 263μs 212ns 735ns 0 0 0 2.16 KB
#7752 EncodeArgs net6.0 77.3μs 134ns 518ns 0 0 0 32.4 KB
#7752 EncodeArgs netcoreapp3.1 97.5μs 298ns 1.15μs 0 0 0 32.4 KB
#7752 EncodeArgs net472 110μs 52ns 202ns 4.93 0 0 32.5 KB
#7752 EncodeLegacyArgs net6.0 139μs 12.8ns 49.4ns 0 0 0 2.15 KB
#7752 EncodeLegacyArgs netcoreapp3.1 200μs 193ns 749ns 0 0 0 2.14 KB
#7752 EncodeLegacyArgs net472 263μs 23.2ns 89.8ns 0 0 0 2.17 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #7752

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑netcoreapp3.1 2.076 412,767.50 856,794.48
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑netcoreapp3.1 1.716 298,978.50 512,951.41

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 406μs 111ns 400ns 0 0 0 4.55 KB
master RunWafRealisticBenchmark netcoreapp3.1 413μs 229ns 885ns 0 0 0 4.48 KB
master RunWafRealisticBenchmark net472 447μs 101ns 391ns 0 0 0 4.66 KB
master RunWafRealisticBenchmarkWithAttack net6.0 294μs 118ns 457ns 0 0 0 2.24 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 299μs 161ns 556ns 0 0 0 2.22 KB
master RunWafRealisticBenchmarkWithAttack net472 319μs 49.9ns 180ns 0 0 0 2.28 KB
#7752 RunWafRealisticBenchmark net6.0 396μs 132ns 513ns 0 0 0 4.55 KB
#7752 RunWafRealisticBenchmark netcoreapp3.1 790μs 14.8μs 148μs 0 0 0 4.48 KB
#7752 RunWafRealisticBenchmark net472 426μs 27.9ns 108ns 0 0 0 4.66 KB
#7752 RunWafRealisticBenchmarkWithAttack net6.0 285μs 49.5ns 192ns 0 0 0 2.24 KB
#7752 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 509μs 2.96μs 27.1μs 0 0 0 2.22 KB
#7752 RunWafRealisticBenchmarkWithAttack net472 315μs 22.1ns 85.4ns 0 0 0 2.29 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 62.2μs 87.6ns 303ns 0 0 0 14.52 KB
master SendRequest netcoreapp3.1 71.8μs 76.2ns 285ns 0 0 0 17.42 KB
master SendRequest net472 0.0175ns 0.00341ns 0.0132ns 0 0 0 0 b
#7752 SendRequest net6.0 61.2μs 55.4ns 207ns 0 0 0 14.52 KB
#7752 SendRequest netcoreapp3.1 72.6μs 343ns 1.61μs 0 0 0 17.42 KB
#7752 SendRequest net472 0.0111ns 0.00327ns 0.0127ns 0 0 0 0 b
Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑netcoreapp3.1 1 B 20 B 19 B 1,900.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0 2 B 4 B 2 B 100.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0 2 B 3 B 1 B 50.00%

Fewer allocations 🎉 in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net472 73 B 0 b -73 B -100.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net472 47 B 0 b -47 B -100.00%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master OriginalCharSlice net6.0 1.86ms 3.83μs 14.9μs 0 0 0 640 KB
master OriginalCharSlice netcoreapp3.1 2.05ms 4.47μs 16.1μs 0 0 0 640 KB
master OriginalCharSlice net472 2.69ms 2.18μs 8.46μs 100 0 0 641.95 KB
master OptimizedCharSlice net6.0 1.44ms 1.36μs 5.28μs 0 0 0 2 B
master OptimizedCharSlice netcoreapp3.1 1.65ms 264ns 989ns 0 0 0 1 B
master OptimizedCharSlice net472 1.9ms 298ns 1.12μs 0 0 0 73 B
master OptimizedCharSliceWithPool net6.0 820μs 66.1ns 256ns 0 0 0 2 B
master OptimizedCharSliceWithPool netcoreapp3.1 811μs 61.5ns 238ns 0 0 0 1 B
master OptimizedCharSliceWithPool net472 1.19ms 98.4ns 381ns 0 0 0 47 B
#7752 OriginalCharSlice net6.0 1.9ms 1.49μs 5.58μs 0 0 0 640.01 KB
#7752 OriginalCharSlice netcoreapp3.1 2.17ms 7.11μs 26.6μs 0 0 0 640 KB
#7752 OriginalCharSlice net472 2.66ms 116ns 434ns 100 0 0 641.95 KB
#7752 OptimizedCharSlice net6.0 1.34ms 421ns 1.63μs 0 0 0 4 B
#7752 OptimizedCharSlice netcoreapp3.1 1.66ms 432ns 1.67μs 0 0 0 1 B
#7752 OptimizedCharSlice net472 2.03ms 723ns 2.8μs 0 0 0 0 b
#7752 OptimizedCharSliceWithPool net6.0 798μs 94.8ns 355ns 0 0 0 3 B
#7752 OptimizedCharSliceWithPool netcoreapp3.1 840μs 88.4ns 342ns 0 0 0 20 B
#7752 OptimizedCharSliceWithPool net472 1.13ms 137ns 531ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.76 KB 42.22 KB 458 B 1.10%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 747μs 1.15μs 4.45μs 0 0 0 41.57 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 772μs 5.49μs 54.9μs 0 0 0 41.76 KB
master WriteAndFlushEnrichedTraces net472 931μs 4.28μs 16.6μs 8.33 0 0 55.98 KB
#7752 WriteAndFlushEnrichedTraces net6.0 713μs 622ns 2.33μs 0 0 0 41.75 KB
#7752 WriteAndFlushEnrichedTraces netcoreapp3.1 694μs 909ns 3.4μs 0 0 0 42.22 KB
#7752 WriteAndFlushEnrichedTraces net472 893μs 3.88μs 15μs 8.33 0 0 55.88 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.9μs 9.66ns 44.3ns 0 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 2.62μs 12.6ns 53.6ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.87μs 3.83ns 14.3ns 0.156 0.0142 0 987 B
#7752 ExecuteNonQuery net6.0 1.94μs 2.14ns 8.31ns 0 0 0 1.02 KB
#7752 ExecuteNonQuery netcoreapp3.1 2.65μs 12.4ns 48ns 0 0 0 1.02 KB
#7752 ExecuteNonQuery net472 2.95μs 4.86ns 18.8ns 0.148 0.0148 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.73μs 1.46ns 5.65ns 0 0 0 1.03 KB
master CallElasticsearch netcoreapp3.1 2.18μs 11.1ns 49.8ns 0 0 0 1.03 KB
master CallElasticsearch net472 3.53μs 1.9ns 6.84ns 0.158 0 0 1.04 KB
master CallElasticsearchAsync net6.0 1.84μs 7.43ns 28.8ns 0 0 0 1.01 KB
master CallElasticsearchAsync netcoreapp3.1 2.36μs 8.34ns 32.3ns 0 0 0 1.08 KB
master CallElasticsearchAsync net472 3.63μs 0.776ns 2.8ns 0.163 0 0 1.1 KB
#7752 CallElasticsearch net6.0 1.71μs 8.43ns 37.7ns 0 0 0 1.03 KB
#7752 CallElasticsearch netcoreapp3.1 2.24μs 11ns 45.2ns 0 0 0 1.03 KB
#7752 CallElasticsearch net472 3.53μs 3.14ns 12.2ns 0.16 0 0 1.04 KB
#7752 CallElasticsearchAsync net6.0 1.85μs 1.3ns 5.04ns 0 0 0 1.01 KB
#7752 CallElasticsearchAsync netcoreapp3.1 2.39μs 11.9ns 54.6ns 0 0 0 1.08 KB
#7752 CallElasticsearchAsync net472 3.67μs 6.11ns 22ns 0.166 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.87μs 4.01ns 15.5ns 0 0 0 952 B
master ExecuteAsync netcoreapp3.1 2.51μs 8.35ns 32.3ns 0 0 0 952 B
master ExecuteAsync net472 2.58μs 1.31ns 4.89ns 0.141 0 0 915 B
#7752 ExecuteAsync net6.0 1.85μs 5.73ns 22.2ns 0 0 0 952 B
#7752 ExecuteAsync netcoreapp3.1 2.46μs 9.42ns 36.5ns 0 0 0 952 B
#7752 ExecuteAsync net472 2.62μs 1.95ns 7.54ns 0.144 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 7.19μs 3.43ns 12.4ns 0 0 0 2.36 KB
master SendAsync netcoreapp3.1 8.67μs 19.4ns 75ns 0 0 0 2.9 KB
master SendAsync net472 12.5μs 8.63ns 33.4ns 0.497 0 0 3.18 KB
#7752 SendAsync net6.0 7.03μs 8.12ns 31.4ns 0 0 0 2.36 KB
#7752 SendAsync netcoreapp3.1 8.33μs 11ns 41ns 0 0 0 2.9 KB
#7752 SendAsync net472 11.9μs 11.8ns 45.6ns 0.474 0 0 3.18 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472 275.24 KB 286.72 KB 11.48 KB 4.17%

Fewer allocations 🎉 in #7752

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0 45.32 KB 43.77 KB -1.55 KB -3.42%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 274.41 KB 256.64 KB -17.77 KB -6.48%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 275.95 KB 255.45 KB -20.5 KB -7.43%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 44.9μs 314ns 2.93μs 0 0 0 45.32 KB
master StringConcatBenchmark netcoreapp3.1 51.9μs 506ns 4.83μs 0 0 0 42.9 KB
master StringConcatBenchmark net472 56.5μs 270ns 1.05μs 0 0 0 57.34 KB
master StringConcatAspectBenchmark net6.0 470μs 1.99μs 7.18μs 0 0 0 274.41 KB
master StringConcatAspectBenchmark netcoreapp3.1 468μs 5.37μs 52.3μs 0 0 0 275.95 KB
master StringConcatAspectBenchmark net472 417μs 2.31μs 14.4μs 0 0 0 275.24 KB
#7752 StringConcatBenchmark net6.0 42.8μs 236ns 1.87μs 0 0 0 43.77 KB
#7752 StringConcatBenchmark netcoreapp3.1 50.9μs 300ns 2.65μs 0 0 0 42.73 KB
#7752 StringConcatBenchmark net472 57.4μs 261ns 1.05μs 0 0 0 57.34 KB
#7752 StringConcatAspectBenchmark net6.0 447μs 2.16μs 8.63μs 0 0 0 256.64 KB
#7752 StringConcatAspectBenchmark netcoreapp3.1 514μs 1.85μs 7.17μs 0 0 0 255.45 KB
#7752 StringConcatAspectBenchmark net472 403μs 2.05μs 10.1μs 0 0 0 286.72 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.67μs 13.5ns 58.8ns 0 0 0 1.7 KB
master EnrichedLog netcoreapp3.1 3.6μs 17.5ns 74.4ns 0 0 0 1.7 KB
master EnrichedLog net472 4.02μs 4.38ns 16.4ns 0.241 0 0 1.64 KB
#7752 EnrichedLog net6.0 2.65μs 12.4ns 49.7ns 0 0 0 1.7 KB
#7752 EnrichedLog netcoreapp3.1 3.66μs 11.5ns 43.2ns 0 0 0 1.7 KB
#7752 EnrichedLog net472 4.06μs 3.42ns 13.2ns 0.242 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 121μs 64.4ns 241ns 0 0 0 4.31 KB
master EnrichedLog netcoreapp3.1 128μs 244ns 944ns 0 0 0 4.31 KB
master EnrichedLog net472 167μs 105ns 406ns 0 0 0 4.52 KB
#7752 EnrichedLog net6.0 123μs 71.7ns 248ns 0 0 0 4.31 KB
#7752 EnrichedLog netcoreapp3.1 129μs 78.5ns 283ns 0 0 0 4.31 KB
#7752 EnrichedLog net472 167μs 113ns 436ns 0 0 0 4.52 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 5.13μs 7.62ns 28.5ns 0 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 6.77μs 16.5ns 63.7ns 0 0 0 2.26 KB
master EnrichedLog net472 7.49μs 5.68ns 22ns 0.299 0 0 2.08 KB
#7752 EnrichedLog net6.0 5.16μs 20.7ns 77.6ns 0 0 0 2.26 KB
#7752 EnrichedLog netcoreapp3.1 6.81μs 15.5ns 60.1ns 0 0 0 2.26 KB
#7752 EnrichedLog net472 7.6μs 11.3ns 43.9ns 0.303 0 0 2.08 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.93μs 10ns 49.1ns 0 0 0 1.2 KB
master SendReceive netcoreapp3.1 2.65μs 12.4ns 48.2ns 0 0 0 1.2 KB
master SendReceive net472 3.05μs 1.37ns 5.12ns 0.182 0 0 1.2 KB
#7752 SendReceive net6.0 2.05μs 9.85ns 40.6ns 0 0 0 1.2 KB
#7752 SendReceive netcoreapp3.1 2.56μs 12.7ns 58.3ns 0 0 0 1.2 KB
#7752 SendReceive net472 2.95μs 2ns 7.76ns 0.178 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.25μs 16.5ns 59.5ns 0 0 0 1.58 KB
master EnrichedLog netcoreapp3.1 5.64μs 11.1ns 42.9ns 0 0 0 1.63 KB
master EnrichedLog net472 6.55μs 5.53ns 21.4ns 0.293 0 0 2.03 KB
#7752 EnrichedLog net6.0 4.21μs 3.41ns 12.3ns 0 0 0 1.58 KB
#7752 EnrichedLog netcoreapp3.1 5.54μs 18.8ns 72.7ns 0 0 0 1.63 KB
#7752 EnrichedLog net472 7.13μs 7.2ns 27.9ns 0.321 0 0 2.03 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 785ns 0.322ns 1.25ns 0 0 0 576 B
master StartFinishSpan netcoreapp3.1 966ns 4.8ns 22.5ns 0 0 0 576 B
master StartFinishSpan net472 920ns 0.57ns 2.21ns 0.0878 0 0 578 B
master StartFinishScope net6.0 924ns 1.26ns 4.86ns 0 0 0 696 B
master StartFinishScope netcoreapp3.1 1.24μs 5.83ns 22.6ns 0 0 0 696 B
master StartFinishScope net472 1.11μs 1.01ns 3.63ns 0.101 0 0 658 B
#7752 StartFinishSpan net6.0 761ns 4.09ns 22.4ns 0 0 0 576 B
#7752 StartFinishSpan netcoreapp3.1 997ns 5.13ns 23ns 0 0 0 576 B
#7752 StartFinishSpan net472 923ns 0.905ns 3.39ns 0.0878 0 0 578 B
#7752 StartFinishScope net6.0 918ns 4.38ns 18.1ns 0 0 0 696 B
#7752 StartFinishScope netcoreapp3.1 1.23μs 6.2ns 26.3ns 0 0 0 696 B
#7752 StartFinishScope net472 1.14μs 0.833ns 3.23ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.07μs 5.35ns 22.7ns 0 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 1.42μs 4.5ns 17.4ns 0 0 0 696 B
master RunOnMethodBegin net472 1.51μs 1.53ns 5.71ns 0.0981 0 0 658 B
#7752 RunOnMethodBegin net6.0 1.05μs 0.689ns 2.58ns 0 0 0 696 B
#7752 RunOnMethodBegin netcoreapp3.1 1.46μs 6.2ns 24ns 0 0 0 696 B
#7752 RunOnMethodBegin net472 1.5μs 1.07ns 4.16ns 0.0976 0 0 658 B

@dd-trace-dotnet-ci-bot
Copy link

dd-trace-dotnet-ci-bot bot commented Nov 3, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (7752) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration74.48 ± (74.31 - 74.88) ms73.69 ± (73.69 - 74.29) ms-1.1%✅
.NET Framework 4.8 - Bailout
duration78.90 ± (78.67 - 79.39) ms79.03 ± (79.03 - 79.84) ms+0.2%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1103.59 ± (1107.85 - 1120.05) ms1099.32 ± (1101.45 - 1110.34) ms-0.4%✅
.NET Core 3.1 - Baseline
process.internal_duration_ms22.96 ± (22.88 - 23.03) ms23.00 ± (22.93 - 23.07) ms+0.2%✅⬆️
process.time_to_main_ms86.93 ± (86.55 - 87.30) ms87.21 ± (86.84 - 87.57) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.91 ± (10.91 - 10.92) MB10.89 ± (10.88 - 10.89) MB-0.2%✅
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms22.82 ± (22.75 - 22.88) ms23.08 ± (23.01 - 23.15) ms+1.2%✅⬆️
process.time_to_main_ms89.15 ± (88.80 - 89.50) ms88.73 ± (88.39 - 89.06) ms-0.5%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.96 ± (10.96 - 10.97) MB10.94 ± (10.94 - 10.94) MB-0.2%✅
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms217.99 ± (216.69 - 219.29) ms221.49 ± (220.14 - 222.83) ms+1.6%✅⬆️
process.time_to_main_ms544.29 ± (543.13 - 545.46) ms547.81 ± (546.68 - 548.94) ms+0.6%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed52.71 ± (52.69 - 52.73) MB52.48 ± (52.46 - 52.50) MB-0.4%✅
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.5%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms21.61 ± (21.55 - 21.67) ms21.75 ± (21.69 - 21.81) ms+0.6%✅⬆️
process.time_to_main_ms75.35 ± (75.03 - 75.67) ms75.92 ± (75.63 - 76.22) ms+0.8%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.61 ± (10.61 - 10.61) MB10.60 ± (10.60 - 10.60) MB-0.1%✅
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 6 - Bailout
process.internal_duration_ms21.43 ± (21.38 - 21.47) ms21.66 ± (21.60 - 21.72) ms+1.1%✅⬆️
process.time_to_main_ms75.78 ± (75.47 - 76.10) ms76.92 ± (76.61 - 77.23) ms+1.5%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.66 ± (10.65 - 10.66) MB10.65 ± (10.65 - 10.65) MB-0.1%✅
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms205.79 ± (204.17 - 207.41) ms207.54 ± (206.06 - 209.01) ms+0.8%✅⬆️
process.time_to_main_ms511.24 ± (510.10 - 512.38) ms509.46 ± (508.45 - 510.48) ms-0.3%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed51.78 ± (51.74 - 51.82) MB51.65 ± (51.63 - 51.67) MB-0.3%✅
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms19.77 ± (19.70 - 19.83) ms19.74 ± (19.68 - 19.79) ms-0.2%✅
process.time_to_main_ms73.96 ± (73.64 - 74.28) ms74.73 ± (74.46 - 75.01) ms+1.0%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.66 ± (7.65 - 7.66) MB7.64 ± (7.64 - 7.65) MB-0.2%✅
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 8 - Bailout
process.internal_duration_ms19.68 ± (19.63 - 19.73) ms19.73 ± (19.68 - 19.78) ms+0.3%✅⬆️
process.time_to_main_ms74.67 ± (74.39 - 74.95) ms75.92 ± (75.63 - 76.20) ms+1.7%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.71 ± (7.70 - 7.71) MB7.69 ± (7.68 - 7.70) MB-0.2%✅
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms189.44 ± (188.40 - 190.47) ms193.61 ± (192.54 - 194.68) ms+2.2%✅⬆️
process.time_to_main_ms488.45 ± (487.28 - 489.62) ms489.77 ± (488.54 - 491.00) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed39.12 ± (39.08 - 39.15) MB38.89 ± (38.85 - 38.94) MB-0.6%✅
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)+0.0%✅⬆️

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration192.56 ± (192.48 - 193.30) ms194.00 ± (193.81 - 194.54) ms+0.7%✅⬆️
.NET Framework 4.8 - Bailout
duration195.81 ± (195.58 - 196.13) ms197.30 ± (197.14 - 197.78) ms+0.8%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1152.62 ± (1155.92 - 1164.24) ms1159.75 ± (1168.66 - 1179.53) ms+0.6%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms187.37 ± (187.04 - 187.69) ms187.86 ± (187.57 - 188.16) ms+0.3%✅⬆️
process.time_to_main_ms80.51 ± (80.31 - 80.72) ms80.46 ± (80.25 - 80.68) ms-0.1%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.03 ± (16.01 - 16.06) MB16.02 ± (15.99 - 16.04) MB-0.1%✅
runtime.dotnet.threads.count20 ± (19 - 20)20 ± (19 - 20)-0.0%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms187.00 ± (186.63 - 187.36) ms187.73 ± (187.31 - 188.14) ms+0.4%✅⬆️
process.time_to_main_ms81.90 ± (81.73 - 82.06) ms81.82 ± (81.65 - 81.98) ms-0.1%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.11 ± (16.09 - 16.14) MB16.09 ± (16.06 - 16.12) MB-0.2%✅
runtime.dotnet.threads.count21 ± (21 - 21)21 ± (21 - 21)+0.3%✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms396.00 ± (393.63 - 398.37) ms387.79 ± (385.63 - 389.96) ms-2.1%✅
process.time_to_main_ms519.83 ± (519.17 - 520.50) ms512.98 ± (512.34 - 513.62) ms-1.3%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed63.41 ± (63.28 - 63.55) MB62.32 ± (62.17 - 62.48) MB-1.7%✅
runtime.dotnet.threads.count29 ± (29 - 30)29 ± (29 - 29)-0.1%✅
.NET 6 - Baseline
process.internal_duration_ms191.20 ± (190.84 - 191.56) ms192.59 ± (192.16 - 193.03) ms+0.7%✅⬆️
process.time_to_main_ms69.64 ± (69.47 - 69.81) ms70.07 ± (69.86 - 70.28) ms+0.6%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed15.94 ± (15.79 - 16.08) MB16.09 ± (15.97 - 16.21) MB+1.0%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)19 ± (19 - 19)+3.3%✅⬆️
.NET 6 - Bailout
process.internal_duration_ms190.48 ± (190.15 - 190.81) ms191.83 ± (191.50 - 192.17) ms+0.7%✅⬆️
process.time_to_main_ms70.60 ± (70.48 - 70.71) ms71.04 ± (70.91 - 71.17) ms+0.6%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed15.85 ± (15.69 - 16.01) MB16.22 ± (16.14 - 16.29) MB+2.3%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)20 ± (20 - 20)+4.6%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms409.46 ± (406.42 - 412.50) ms411.36 ± (408.66 - 414.06) ms+0.5%✅⬆️
process.time_to_main_ms486.65 ± (486.07 - 487.24) ms482.59 ± (481.93 - 483.24) ms-0.8%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed61.93 ± (61.77 - 62.09) MB62.38 ± (62.26 - 62.50) MB+0.7%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 30)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms189.08 ± (188.85 - 189.31) ms190.01 ± (189.73 - 190.29) ms+0.5%✅⬆️
process.time_to_main_ms69.06 ± (68.90 - 69.21) ms69.25 ± (69.10 - 69.40) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.74 ± (11.71 - 11.77) MB11.75 ± (11.72 - 11.78) MB+0.1%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 18)+0.5%✅⬆️
.NET 8 - Bailout
process.internal_duration_ms188.67 ± (188.33 - 189.00) ms189.65 ± (189.39 - 189.91) ms+0.5%✅⬆️
process.time_to_main_ms70.33 ± (70.20 - 70.46) ms70.39 ± (70.28 - 70.50) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.66 ± (11.58 - 11.75) MB11.78 ± (11.73 - 11.83) MB+1.0%✅⬆️
runtime.dotnet.threads.count19 ± (18 - 19)19 ± (19 - 19)+1.9%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms357.31 ± (355.99 - 358.64) ms358.50 ± (356.99 - 360.00) ms+0.3%✅⬆️
process.time_to_main_ms463.88 ± (463.35 - 464.40) ms456.77 ± (456.13 - 457.41) ms-1.5%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed50.50 ± (50.45 - 50.54) MB50.32 ± (50.29 - 50.35) MB-0.4%✅
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)+1.1%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (74ms)  : 70, 78
    master - mean (75ms)  : 70, 79

    section Bailout
    This PR (7752) - mean (79ms)  : 73, 86
    master - mean (79ms)  : 73, 85

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (1,106ms)  : 1042, 1169
    master - mean (1,114ms)  : 1019, 1208
Loading
FakeDbCommand (.NET Core 3.1) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (118ms)  : 111, 124
    master - mean (117ms)  : 111, 124

    section Bailout
    This PR (7752) - mean (119ms)  : 113, 126
    master - mean (120ms)  : 112, 128

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (810ms)  : 776, 845
    master - mean (801ms)  : 766, 835
Loading
FakeDbCommand (.NET 6) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (105ms)  : 99, 110
    master - mean (104ms)  : 97, 111

    section Bailout
    This PR (7752) - mean (106ms)  : 99, 112
    master - mean (104ms)  : 96, 112

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (757ms)  : 699, 814
    master - mean (757ms)  : 696, 817
Loading
FakeDbCommand (.NET 8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (103ms)  : 98, 108
    master - mean (102ms)  : 95, 109

    section Bailout
    This PR (7752) - mean (104ms)  : 98, 110
    master - mean (103ms)  : 96, 110

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (721ms)  : 686, 757
    master - mean (717ms)  : 681, 753
Loading
HttpMessageHandler (.NET Framework 4.8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (194ms)  : 190, 198
    master - mean (193ms)  : 189, 197

    section Bailout
    This PR (7752) - mean (197ms)  : 194, 201
    master - mean (196ms)  : 193, 198

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (1,174ms)  : 1091, 1258
    master - mean (1,160ms)  : 1100, 1220
Loading
HttpMessageHandler (.NET Core 3.1) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (277ms)  : 272, 281
    master - mean (276ms)  : 272, 281

    section Bailout
    This PR (7752) - mean (278ms)  : 273, 283
    master - mean (277ms)  : 272, 281

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (938ms)  : 891, 985
    master - mean (951ms)  : 907, 995
Loading
HttpMessageHandler (.NET 6) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (271ms)  : 264, 278
    master - mean (269ms)  : 264, 275

    section Bailout
    This PR (7752) - mean (271ms)  : 264, 278
    master - mean (269ms)  : 265, 273

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (926ms)  : 883, 968
    master - mean (925ms)  : 876, 975
Loading
HttpMessageHandler (.NET 8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (7752) - mean (269ms)  : 266, 272
    master - mean (268ms)  : 264, 272

    section Bailout
    This PR (7752) - mean (270ms)  : 266, 273
    master - mean (269ms)  : 264, 273

    section CallTarget+Inlining+NGEN
    This PR (7752) - mean (846ms)  : 828, 864
    master - mean (853ms)  : 836, 869
Loading

@datadog-official

This comment has been minimized.

Sign in to view
pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/build/_build/Build.Steps.cs
var filter = (string.IsNullOrWhiteSpace(Filter), IsWin) switch
{
(false, _) => $"({Filter}){dockerFilter}{armFilter}",
(false, _) => $"({Filter})&(SkipInCI!=True){dockerFilter}{armFilter}",
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After adding IntegrationTestFilter: DockerGroup=$(dockerGroup)
 to the jobs, we started falling through this case, which ignored SkipInCI

pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/build/PackageVersionsGeneratorDefinitions.json
"SampleProjectName": "Samples.CosmosDb",
"NugetPackageSearchName": "Microsoft.Azure.Cosmos",
"MinVersion": "3.6.0",
"MinVersion": "3.12.0",
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to use HttpClientFactory when constructing the SDK client for use with the emulator, which has a self-signed certificate that .net fails to validate by default. 3.12.0 is the first version that supports HttpClientFactory

pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/test/test-applications/integrations/Samples.CosmosDb/Program.cs
Comment on lines 107 to 109

var user = await database.CreateUserAsync("user");
Console.WriteLine($"{DateTime.Now:o}: Created user: {user.Resource.Id}\n");
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Emulator doesn't support creating users

pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/test/test-applications/integrations/Samples.CosmosDb/Program.cs Outdated
ApplicationName = "CosmosDBDotnetQuickstart",
RequestTimeout = TimeSpan.FromMinutes(10),
OpenTcpConnectionTimeout = TimeSpan.FromMinutes(1),
ConnectionMode = ConnectionMode.Gateway,
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Emulator only supports gateway mode

pablomartinezbernardo
pablomartinezbernardo commented Nov 4, 2025
View reviewed changes
tracer/test/test-applications/integrations/Samples.CosmosDb/Program.cs Outdated
RequestTimeout = TimeSpan.FromMinutes(10),
OpenTcpConnectionTimeout = TimeSpan.FromMinutes(1),
ConnectionMode = ConnectionMode.Gateway,
LimitToEndpoint = true,
Copy link
Contributor Author

@pablomartinezbernardo pablomartinezbernardo Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.cosmosclientoptions.limittoendpoint?view=azure-dotnet

Otherwise the SDK will attempt to connect to 127.0.0.1 after the first request

@pablomartinezbernardo pablomartinezbernardo changed the title Run cosmos tests on emulator Cosmos DB partial integration testing Nov 4, 2025
@pablomartinezbernardo pablomartinezbernardo marked this pull request as ready for review November 4, 2025 13:40
@pablomartinezbernardo pablomartinezbernardo requested review from a team as code owners November 4, 2025 13:40
bouwkast
bouwkast reviewed Nov 4, 2025
View reviewed changes
Copy link
Collaborator

@bouwkast bouwkast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure how I feel about this, one reason why I closed #6683 was because the docker container couldn't run all of the necessary commands.

I think this is running into the same / similar issue where only a small subset of the functionality of CosmosDb works within the docker container.

I think we should maybe consider to go back to running the Cosmos Emulator (Windows) in CI as opposed to this

@pablomartinezbernardo
Copy link
Contributor Author

pablomartinezbernardo commented Nov 4, 2025

I think this is running into the same / similar issue where only a small subset of the functionality of CosmosDb works within the docker container.

It is a subset, but I would argue against it being small. The only opeartions that we are missing are queries over databases, containers and users. Operations on items, which are the most common scenarios, are supported. Some complex operations on items are not supported, but that becomes irrelevant, as testing the simpler cases completely tests our instrumentation.

I think we should maybe consider to go back to running the Cosmos Emulator (Windows) in CI as opposed to this

To defend your case... it is not only a Windows emulator, it can be run on Linux. Just not on arm64. It also looks like it will be heavier than the linux-based one:

docker images mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator                                                                                                                                                                                           
REPOSITORY                                               TAG             IMAGE ID       CREATED        SIZE
mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator   latest          bbf28177ceb0   2 months ago   3.06GB
mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator   vnext-preview   efe42b6f3e32   3 months ago   900MB

It's a valid option as well, the con is that there's no way to run it on arm machines (Not even with QEMU), which means

  • No local testing for mac users
  • No arm CI, as we are running dependencies on arm as well

Let's hear more opinions about this!

@bouwkast
Copy link
Collaborator

bouwkast commented Nov 4, 2025

This is the emulator that I was mentioning (non-Docker) https://learn.microsoft.com/en-us/azure/cosmos-db/emulator-release-notes - but yeah your limitations still apply.

We used to run this in CI, but it proved to be too flakey.

Could we do something where we split between the old way of testing and the new way of testing ->

{
    "IntegrationName": "CosmosDb",
    "SampleProjectName": "Samples.CosmosDb",
    "NugetPackageSearchName": "Microsoft.Azure.Cosmos",
    "MinVersion": "3.6.0",
    "MaxVersionExclusive": "4.0.0",
    "SpecificVersions": [
      "3.6.*",
      "3.*.*"
    ],
    "VersionConditions": [{
      "MinVersion": "3.29.0",
      "ExcludeTargetFrameworks": ["netcoreapp2.1", "netcoreapp3.0"]
    }]
  },
  {
    "IntegrationName": "CosmosDb_Emulator",
    "SampleProjectName": "Samples.CosmosDb",
    "NugetPackageSearchName": "Microsoft.Azure.Cosmos",
    "MinVersion": "3.12.0",
    "MaxVersionExclusive": "4.0.0",
    "SpecificVersions": [
      "3.12.*",
      "3.*.*"
    ],
    "VersionConditions": [{
      "MinVersion": "3.29.0",
      "ExcludeTargetFrameworks": ["netcoreapp2.1", "netcoreapp3.0"]
    }]

bouwkast
bouwkast reviewed Nov 4, 2025
View reviewed changes
docker-compose.yml
Comment on lines +68 to +71
cosmosdb-emulator_arm64:
image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
command: ["--protocol", "https"]

Copy link
Collaborator

@bouwkast bouwkast Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will require a VM update after merging to not slow down CI for too long - I'd think that we could probably do so around .NET 10 GA release

@pablomartinezbernardo pablomartinezbernardo marked this pull request as draft November 7, 2025 12:58
@pablomartinezbernardo pablomartinezbernardo marked this pull request as ready for review November 7, 2025 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bouwkast bouwkast Awaiting requested review from bouwkast

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pablomartinezbernardo @bouwkast