[Metrics SDK] Performance improvement in measurement processing

[lalitb]

This PR attempts to improve the measurement processing throughput - with benchmark as the time taken by SDK to process 1 million measurements with multiple dimensions and high cardinality.

The change is to remove the unnecessary memory copy operations done for each input measurement. The measurement is now copied to OrderedAttributeMap only if it is not already existing in the AttributesHashMap.

Existing flow:
instrument::Add(value, attributes) -> Copy attributes to OrderedAttributeMap -> Calculate hash for attributes in OrderedAttributeMap -> Add attributes to AttributesHashMap if not existing (by comparing hash calculated in step 3) -> Aggregate(value)

New flow:
instrument::Add(value, attributes) -> Calculate hash for attributes in KeyValueIterable -> Copy attributes to OrderedAttributeMap, and then add to AttributesHashMap if not existing (by comparing hash calculated in step 2) -> Aggregate(value)

The processing time for 1 million measurements is reduced from 3.7 secs to 1.9 secs ( on CPU: Intel(R) Core(TM) i9-10900 CPU @ 2.80GHz, 20 processors, Memory: 32GB)

The test results are for below scenario:

Number of dimensions/attributes : 3 Cardinality of each dimension : 10 Total possible dimensions: 10 * 10 * 10 = 1000 Total number of measurements : 1 million 

(The benchmark code is added as part of this PR - measurements_benchmark.cc:

Benchmark with this PR changes:

Running ./measurements_benchmark Run on (20 X 2808.01 MHz CPU s) CPU Caches: L1 Data 32K (x10) L1 Instruction 32K (x10) L2 Unified 256K (x10) L3 Unified 20480K (x1) Load Average: 8.23, 3.09, 1.54 -------------------------------------------------------------- Benchmark Time CPU Iterations -------------------------------------------------------------- BM_MeasurementsTest 1988578400 ns 1988543700 ns 1 

Benchmark with existing code:

2023-02-20 22:23:53 Running ./measurements_benchmark Run on (20 X 2808.01 MHz CPU s) CPU Caches: L1 Data 32K (x10) L1 Instruction 32K (x10) L2 Unified 256K (x10) L3 Unified 20480K (x1) Load Average: 2.84, 2.51, 1.45 -------------------------------------------------------------- Benchmark Time CPU Iterations -------------------------------------------------------------- BM_MeasurementsTest 3794032100 ns 3794023000 ns 1 

Changes

Please provide a brief description of the changes here.

For significant contributions please make sure you have completed the following items:

• CHANGELOG.md updated for non-trivial changes
• Unit tests have been added
• Changes in public API reviewed

[codecov]

Codecov Report

Merging #1993 (7d7755b) into main (649829f) will increase coverage by 0.01%.
The diff coverage is 89.42%.

Additional details and impacted files

@@ Coverage Diff @@ ## main #1993 +/- ## ========================================== + Coverage 87.32% 87.32% +0.01%  ========================================== Files 166 166 Lines 4673 4723 +50 ========================================== + Hits 4080 4124 +44  - Misses 593 599 +6 

Impacted Files	Coverage Δ
...ntelemetry/sdk/metrics/view/attributes_processor.h	90.00% <50.00%> (-10.00%)	⬇️
...clude/opentelemetry/sdk/common/attributemap_hash.h	83.88% <84.22%> (+7.41%)	⬆️
sdk/src/metrics/state/temporal_metric_storage.cc	98.31% <87.50%> (+0.06%)	⬆️
...ntelemetry/sdk/metrics/state/sync_metric_storage.h	86.37% <88.89%> (-1.13%)	⬇️
...entelemetry/sdk/metrics/state/attributes_hashmap.h	95.24% <96.67%> (-0.59%)	⬇️
...telemetry/sdk/metrics/state/async_metric_storage.h	86.49% <100.00%> (+0.38%)	⬆️

[lalitb]

This is ready for review now. @ThomsonTan , @esigo please help in review this PR. We can enable exemplar once this is merged, as exemplar has it's own performance cost.

[lalitb]

Added few more changes to reduce the lock contention (by removing AttributeHashMap hash calculation outside of mutex lock). Gained few more seconds, and better scaling with threads:

Number of threads : 1

$ ./measurements_benchmark 2023-02-24 23:09:48 Running ./measurements_benchmark Run on (20 X 2808.01 MHz CPU s) CPU Caches: L1 Data 32K (x10) L1 Instruction 32K (x10) L2 Unified 256K (x10) L3 Unified 20480K (x1) Load Average: 0.33, 1.90, 1.20 -------------------------------------------------------------- Benchmark Time CPU Iterations -------------------------------------------------------------- BM_MeasurementsTest 1582235960 ns 104430 ns 10 

Number of threads: 5

$ ./measurements_benchmark 2023-02-24 23:15:19 Running ./measurements_benchmark Run on (20 X 2808.01 MHz CPU s) CPU Caches: L1 Data 32K (x10) L1 Instruction 32K (x10) L2 Unified 256K (x10) L3 Unified 20480K (x1) Load Average: 0.20, 0.70, 0.87 -------------------------------------------------------------- Benchmark Time CPU Iterations -------------------------------------------------------------- BM_MeasurementsTest 991755670 ns 306610 ns 10 

Number of threads: 19

$ ./measurements_benchmark 2023-02-24 23:18:25 Running ./measurements_benchmark Run on (20 X 2808.01 MHz CPU s) CPU Caches: L1 Data 32K (x10) L1 Instruction 32K (x10) L2 Unified 256K (x10) L3 Unified 20480K (x1) Load Average: 0.65, 0.67, 0.81 -------------------------------------------------------------- Benchmark Time CPU Iterations -------------------------------------------------------------- BM_MeasurementsTest 926599920 ns 878030 ns 10 

So, 1.5 secs to process 1 million measurements from single thread, .98 secs to process 1 million measurements from 5 threads, .92 secs to process 1 million measurements from 19 threads. This is on machine with 20 processors. So, while the library scales well from 1 to 5 threads, it reaches its performance limit beyond that. As valgrind/callgrind cpu profiling result shows, the limit is reached for the synchronized access to std::unordered_map used inside AttributesHashMap across all these threads. I think we can live with current performance for now unless there are any cross-platform lock-free hashmap data structure we can easily used in our code.

[lalitb]

The results are more promising while building the application in Release mode (similar results on Windows and Linux).
So to process 1 million measurements:
Time taken (1 thread) : ~130 ms
Time taken (5 threads): ~147 ms
Time taken (19 threads): ~165 ms

Number of threads : 1

$ ./measurements_benchmark 2023-02-28 23:32:31 Running ./measurements_benchmark Run on (20 X 2808.01 MHz CPU s) CPU Caches: L1 Data 32K (x10) L1 Instruction 32K (x10) L2 Unified 256K (x10) L3 Unified 20480K (x1) Load Average: 0.16, 0.98, 0.88 -------------------------------------------------------------- Benchmark Time CPU Iterations -------------------------------------------------------------- BM_MeasurementsTest 133399288 ns 132125 ns 100 

Number of threads: 5

$ ./measurements_benchmark 2023-02-28 23:36:06 Running ./measurements_benchmark Run on (20 X 2808.01 MHz CPU s) CPU Caches: L1 Data 32K (x10) L1 Instruction 32K (x10) L2 Unified 256K (x10) L3 Unified 20480K (x1) Load Average: 0.62, 0.71, 0.78 -------------------------------------------------------------- Benchmark Time CPU Iterations -------------------------------------------------------------- BM_MeasurementsTest 147010309 ns 359316 ns 100 

Number of threads: 19

$ ./measurements_benchmark 2023-02-28 23:37:36 Running ./measurements_benchmark Run on (20 X 2808.01 MHz CPU s) CPU Caches: L1 Data 32K (x10) L1 Instruction 32K (x10) L2 Unified 256K (x10) L3 Unified 20480K (x1) Load Average: 0.33, 0.60, 0.73 -------------------------------------------------------------- Benchmark Time CPU Iterations -------------------------------------------------------------- BM_MeasurementsTest 165881775 ns 1205977 ns 100 

[esigo]

LGTM
Thanks for the PR :)
sorry for the delay