Skip to content

Commit 00dc240

Browse files
ehsannasjimit-j-shahgcf-owl-bot[bot]
authored
feat: Tracing using OpenTelemetry (#1728)
* feat: Add FirestoreOpenTelemetryOptions to FirestoreOptions. (#1531) * feat: Add FirestoreOpenTelemetryOptions to FirestoreOptions. * Address code review feedback. * feat: Add com.google.cloud.firestore.telemetry package. (#1533) * feat: Add FirestoreOpenTelemetryOptions to FirestoreOptions. * feat: Add com.google.cloud.firestore.telemetry package. * Address code review feedback. * Factor out the otel version in pom.xml. * fix: Remove OpenCensus tracing code. (#1589) * feat: Add FirestoreOpenTelemetryOptions to FirestoreOptions. * feat: Add com.google.cloud.firestore.telemetry package. * fix: Remove OpenCensus tracing code. * feat: tracing for aggregate queries, bulkwriter, partition queries, a… (#1590) * feat: Add FirestoreOpenTelemetryOptions to FirestoreOptions. * feat: Add com.google.cloud.firestore.telemetry package. * fix: Remove OpenCensus tracing code. * feat: tracing for aggregate queries, bulkwriter, partition queries, and listDocuments. * Address code review feedback. * Address feedback. * don't use wildcard imports. * feat: trace instrumentation for DocumentReference methods. (#1591) * feat: Add FirestoreOpenTelemetryOptions to FirestoreOptions. * feat: Add com.google.cloud.firestore.telemetry package. * fix: Remove OpenCensus tracing code. * feat: tracing for aggregate queries, bulkwriter, partition queries, and listDocuments. * feat: trace instrumentation for DocumentReference methods. * feat: trace instrumentation for queries and transactions. (#1592) * feat: Add FirestoreOpenTelemetryOptions to FirestoreOptions. * feat: Add com.google.cloud.firestore.telemetry package. * fix: Remove OpenCensus tracing code. * feat: tracing for aggregate queries, bulkwriter, partition queries, and listDocuments. * feat: trace instrumentation for DocumentReference methods. * feat: trace instrumentation for queries and transactions. * test: Adding first e2e client-tracing test w/ Custom Root Span (#1621) * test: Adding first e2e client-tracing test w/ Custom Root Span * Roll back E2E tests commit. * Address feedback. * Address feedback (better event log message). * Address feedback. --------- Co-authored-by: Jimit J Shah <57637300+jimit-j-shah@users.noreply.github.com> * test: End-to-End Integration Test for Client-side Tracing in Firestore Java Server SDK using OpenTelemetry SDK and Cloud Trace Exporter against Cloud Trace. (#1635) * Adding first e2e client-tracing test w/ Custom Root Span * test: Adding first e2e client-tracing test w/ Custom Root Span * Fixing test dependencies and use default GCP testing project. Fixing * Fixing test dependencies and use default GCP testing project. * Fixing formatting * Add aggregationQueryGet Test * Add bulkWriterCommitTrace Test * Fixing running multiple-tests * Add partitionQuery Test * Add collectionListDocumentsTrace Test * Add docRef*Trace Tests * Add docRefUpdate*Trace and docRefDelete*Trace Tests * Fixing Trace fetching using retries for missing or incomplete traces due to eventual consistency of Cloud Trace * Add get/query Trace Tests * Add Transaction test * Added TraceContainer to be able to test transaction test-cases * test: Adding Transaction tests * test: Adding Transaction tests * test: Adding TestParameterInjector to run the test for global and non-global opentelemetry SDK instances * test: formatting and cleanup * test: Adding first e2e client-tracing test w/ Custom Root Span * test: Add aggregationQueryGet Test * test: Add bulkWriterCommitTrace Test and fixed running multiple-tests * test: Add partitionQuery Test * test: Add collectionListDocumentsTrace Test * test: Add docRefUpdate*Trace and docRefDelete*Trace Tests and fixed Trace fetching using retries for missing or incomplete traces due to eventual consistency of Cloud Trace * test: Add get/query Trace Tests * test: Added Transaction tests using TraceContainer to verify traces for Transaction ops (BeginTransaction, Rollback etc) * test: Adding TestParameterInjector to run the test for global and non-global opentelemetry SDK instances * test: Formatting and cleanup * test: review comments * test: fixing dfs to handle case where the compareTo callstack may be shorter than the trace callstack - don't need to throw an exception in that case * test: Consolidating verification methods * test: review comments * fix: Make telemetry-related fields transient. (#1638) * fix: Rename 'enabled' to 'tracingEnabled'. (#1639) * fix: Rename 'enabled' to 'tracingEnabled'. In the future, FirestoreOpenTelemetryOptions will support enabling/disabling Logging and Metrics as well. So we should use a better name for this field. * address feedback. * fix: Minor improvement to the ITE2ETracingTest. (#1637) * fix: Minor improvement to the ITE2ETracingTest. * revert the numExpectedSpans change. * feat: Add 'isTransactional' attribute. (#1657) * fix: Necessary test improvements for CI environments. (#1673) * fix: Necessary test improvements for CI environments. * Address feedback. * feat: Disable the tracing feature and remove public APIs. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Add the Firestore SDK version to the attributes. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Update the "test" dependency versions. * Address feedback related to attributes. * Add 'project_id' attribute. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Revert a736fcc ("Disable the tracing feature and remove public APIs"). * GlobalOtel reset for test must happen in `before`, not `after`. * Address feedback. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Jimit J Shah <57637300+jimit-j-shah@users.noreply.github.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
1 parent 05a6f73 commit 00dc240

33 files changed

+4388
-591
lines changed

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,13 +57,13 @@ implementation 'com.google.cloud:google-cloud-firestore'
5757
If you are using Gradle without BOM, add this to your dependencies:
5858

5959
```Groovy
60-
implementation 'com.google.cloud:google-cloud-firestore:3.22.0'
60+
implementation 'com.google.cloud:google-cloud-firestore:3.23.1'
6161
```
6262

6363
If you are using SBT, add this to your dependencies:
6464

6565
```Scala
66-
libraryDependencies += "com.google.cloud" % "google-cloud-firestore" % "3.22.0"
66+
libraryDependencies += "com.google.cloud" % "google-cloud-firestore" % "3.23.1"
6767
```
6868
<!-- {x-version-update-end} -->
6969

@@ -222,7 +222,7 @@ Java is a registered trademark of Oracle and/or its affiliates.
222222
[kokoro-badge-link-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-firestore/java11.html
223223
[stability-image]: https://img.shields.io/badge/stability-stable-green
224224
[maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/google-cloud-firestore.svg
225-
[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-firestore/3.22.0
225+
[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-firestore/3.23.1
226226
[authentication]: https://github.com/googleapis/google-cloud-java#authentication
227227
[auth-scopes]: https://developers.google.com/identity/protocols/oauth2/scopes
228228
[predefined-iam-roles]: https://cloud.google.com/iam/docs/understanding-roles#predefined_roles

‎google-cloud-firestore/pom.xml‎

Lines changed: 81 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
</parent>
1717
<properties>
1818
<site.installationModule>google-cloud-firestore</site.installationModule>
19+
<opentelemetry.version>1.38.0</opentelemetry.version>
1920
</properties>
2021
<dependencies>
2122
<dependency>
@@ -39,10 +40,6 @@
3940
<artifactId>grpc-google-cloud-firestore-v1</artifactId>
4041
<scope>test</scope>
4142
</dependency>
42-
<dependency>
43-
<groupId>io.opencensus</groupId>
44-
<artifactId>opencensus-contrib-grpc-util</artifactId>
45-
</dependency>
4643
<dependency>
4744
<groupId>com.google.code.findbugs</groupId>
4845
<artifactId>jsr305</artifactId>
@@ -91,10 +88,6 @@
9188
<groupId>io.grpc</groupId>
9289
<artifactId>grpc-stub</artifactId>
9390
</dependency>
94-
<dependency>
95-
<groupId>io.opencensus</groupId>
96-
<artifactId>opencensus-api</artifactId>
97-
</dependency>
9891
<dependency>
9992
<groupId>com.google.auth</groupId>
10093
<artifactId>google-auth-library-credentials</artifactId>
@@ -113,6 +106,20 @@
113106
<artifactId>protobuf-java-util</artifactId>
114107
</dependency>
115108

109+
<!-- OpenTelemetry -->
110+
<dependency>
111+
<groupId>io.opentelemetry</groupId>
112+
<artifactId>opentelemetry-api</artifactId>
113+
</dependency>
114+
<dependency>
115+
<groupId>io.opentelemetry</groupId>
116+
<artifactId>opentelemetry-context</artifactId>
117+
</dependency>
118+
<dependency>
119+
<groupId>io.opentelemetry.instrumentation</groupId>
120+
<artifactId>opentelemetry-grpc-1.6</artifactId>
121+
</dependency>
122+
<!-- END OpenTelemetry -->
116123

117124
<!-- Test dependencies -->
118125
<dependency>
@@ -173,6 +180,72 @@
173180
<version>3.15.0</version>
174181
<scope>test</scope>
175182
</dependency>
183+
<!-- OpenTelemetry -->
184+
<dependency>
185+
<groupId>io.opentelemetry</groupId>
186+
<artifactId>opentelemetry-sdk</artifactId>
187+
<version>${opentelemetry.version}</version>
188+
<scope>test</scope>
189+
</dependency>
190+
<dependency>
191+
<groupId>io.opentelemetry</groupId>
192+
<artifactId>opentelemetry-sdk-testing</artifactId>
193+
<version>${opentelemetry.version}</version>
194+
<scope>test</scope>
195+
</dependency>
196+
<dependency>
197+
<groupId>io.opentelemetry</groupId>
198+
<artifactId>opentelemetry-semconv</artifactId>
199+
<version>1.30.1-alpha</version>
200+
<scope>test</scope>
201+
</dependency>
202+
<dependency>
203+
<groupId>io.opentelemetry</groupId>
204+
<artifactId>opentelemetry-sdk-trace</artifactId>
205+
<version>${opentelemetry.version}</version>
206+
<scope>test</scope>
207+
</dependency>
208+
<dependency>
209+
<groupId>io.opentelemetry</groupId>
210+
<artifactId>opentelemetry-sdk-common</artifactId>
211+
<version>${opentelemetry.version}</version>
212+
<scope>test</scope>
213+
</dependency>
214+
<dependency>
215+
<groupId>com.google.cloud.opentelemetry</groupId>
216+
<artifactId>exporter-trace</artifactId>
217+
<version>0.15.0</version>
218+
<scope>test</scope>
219+
</dependency>
220+
<!-- END OpenTelemetry -->
221+
<!-- Cloud Ops -->
222+
<dependency>
223+
<groupId>com.google.api.grpc</groupId>
224+
<artifactId>proto-google-cloud-trace-v1</artifactId>
225+
<version>1.3.0</version>
226+
<scope>test</scope>
227+
</dependency>
228+
<dependency>
229+
<groupId>com.google.cloud.opentelemetry</groupId>
230+
<artifactId>exporter-trace</artifactId>
231+
<version>0.15.0</version>
232+
<scope>test</scope>
233+
</dependency>
234+
<!-- END OpenTelemetry -->
235+
<!-- Cloud Ops -->
236+
<dependency>
237+
<groupId>com.google.api.grpc</groupId>
238+
<artifactId>proto-google-cloud-trace-v1</artifactId>
239+
<version>1.3.0</version>
240+
<scope>test</scope>
241+
</dependency>
242+
<dependency>
243+
<groupId>com.google.cloud</groupId>
244+
<artifactId>google-cloud-trace</artifactId>
245+
<version>1.3.0</version>
246+
<scope>test</scope>
247+
</dependency>
248+
<!-- END Cloud Ops -->
176249
</dependencies>
177250

178251
<reporting>

‎google-cloud-firestore/src/main/java/com/google/cloud/firestore/AggregateQuery.java‎

Lines changed: 77 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@
1616

1717
package com.google.cloud.firestore;
1818

19+
import static com.google.cloud.firestore.telemetry.TraceUtil.ATTRIBUTE_KEY_ATTEMPT;
20+
import static com.google.cloud.firestore.telemetry.TraceUtil.SPAN_NAME_RUN_AGGREGATION_QUERY;
21+
1922
import com.google.api.core.ApiFuture;
2023
import com.google.api.core.InternalExtensionOnly;
2124
import com.google.api.core.SettableApiFuture;
@@ -24,6 +27,8 @@
2427
import com.google.api.gax.rpc.StatusCode;
2528
import com.google.api.gax.rpc.StreamController;
2629
import com.google.cloud.Timestamp;
30+
import com.google.cloud.firestore.telemetry.TraceUtil;
31+
import com.google.cloud.firestore.telemetry.TraceUtil.Scope;
2732
import com.google.cloud.firestore.v1.FirestoreSettings;
2833
import com.google.common.collect.ImmutableMap;
2934
import com.google.firestore.v1.RunAggregationQueryRequest;
@@ -35,6 +40,7 @@
3540
import com.google.firestore.v1.Value;
3641
import com.google.protobuf.ByteString;
3742
import java.util.ArrayList;
43+
import java.util.Collections;
3844
import java.util.HashMap;
3945
import java.util.HashSet;
4046
import java.util.List;
@@ -59,6 +65,11 @@ public class AggregateQuery {
5965
this.aliasMap = new HashMap<>();
6066
}
6167

68+
@Nonnull
69+
private TraceUtil getTraceUtil() {
70+
return query.getFirestore().getOptions().getTraceUtil();
71+
}
72+
6273
/** Returns the query whose aggregations will be calculated by this object. */
6374
@Nonnull
6475
public Query getQuery() {
@@ -85,34 +96,57 @@ public ApiFuture<AggregateQuerySnapshot> get() {
8596
*/
8697
@Nonnull
8798
public ApiFuture<ExplainResults<AggregateQuerySnapshot>> explain(ExplainOptions options) {
88-
AggregateQueryExplainResponseDeliverer responseDeliverer =
89-
new AggregateQueryExplainResponseDeliverer(
90-
/* transactionId= */ null,
91-
/* readTime= */ null,
92-
/* startTimeNanos= */ query.rpcContext.getClock().nanoTime(),
93-
/* explainOptions= */ options);
94-
runQuery(responseDeliverer);
95-
return responseDeliverer.getFuture();
99+
TraceUtil.Span span = getTraceUtil().startSpan(TraceUtil.SPAN_NAME_AGGREGATION_QUERY_GET);
100+
try (Scope ignored = span.makeCurrent()) {
101+
AggregateQueryExplainResponseDeliverer responseDeliverer =
102+
new AggregateQueryExplainResponseDeliverer(
103+
/* transactionId= */ null,
104+
/* readTime= */ null,
105+
/* startTimeNanos= */ query.rpcContext.getClock().nanoTime(),
106+
/* explainOptions= */ options);
107+
runQuery(responseDeliverer, /* attempt */ 0);
108+
ApiFuture<ExplainResults<AggregateQuerySnapshot>> result = responseDeliverer.getFuture();
109+
span.endAtFuture(result);
110+
return result;
111+
} catch (Exception error) {
112+
span.end(error);
113+
throw error;
114+
}
96115
}
97116

98117
@Nonnull
99118
ApiFuture<AggregateQuerySnapshot> get(
100119
@Nullable final ByteString transactionId, @Nullable com.google.protobuf.Timestamp readTime) {
101-
AggregateQueryResponseDeliverer responseDeliverer =
102-
new AggregateQueryResponseDeliverer(
103-
transactionId, readTime, /* startTimeNanos= */ query.rpcContext.getClock().nanoTime());
104-
runQuery(responseDeliverer);
105-
return responseDeliverer.getFuture();
120+
TraceUtil.Span span =
121+
getTraceUtil()
122+
.startSpan(
123+
transactionId == null
124+
? TraceUtil.SPAN_NAME_AGGREGATION_QUERY_GET
125+
: TraceUtil.SPAN_NAME_TRANSACTION_GET_AGGREGATION_QUERY);
126+
try (Scope ignored = span.makeCurrent()) {
127+
AggregateQueryResponseDeliverer responseDeliverer =
128+
new AggregateQueryResponseDeliverer(
129+
transactionId,
130+
readTime,
131+
/* startTimeNanos= */ query.rpcContext.getClock().nanoTime());
132+
runQuery(responseDeliverer, /* attempt= */ 0);
133+
ApiFuture<AggregateQuerySnapshot> result = responseDeliverer.getFuture();
134+
span.endAtFuture(result);
135+
return result;
136+
} catch (Exception error) {
137+
span.end(error);
138+
throw error;
139+
}
106140
}
107141

108-
private <T> void runQuery(ResponseDeliverer<T> responseDeliverer) {
142+
private <T> void runQuery(ResponseDeliverer<T> responseDeliverer, int attempt) {
109143
RunAggregationQueryRequest request =
110144
toProto(
111145
responseDeliverer.getTransactionId(),
112146
responseDeliverer.getReadTime(),
113147
responseDeliverer.getExplainOptions());
114148
AggregateQueryResponseObserver<T> responseObserver =
115-
new AggregateQueryResponseObserver<T>(responseDeliverer);
149+
new AggregateQueryResponseObserver<T>(responseDeliverer, attempt);
116150
ServerStreamingCallable<RunAggregationQueryRequest, RunAggregationQueryResponse> callable =
117151
query.rpcContext.getClient().runAggregationQueryCallable();
118152
query.rpcContext.streamRequest(request, responseObserver, callable);
@@ -249,20 +283,34 @@ private final class AggregateQueryResponseObserver<T>
249283
private Timestamp readTime = Timestamp.MAX_VALUE;
250284
@Nullable private Map<String, Value> aggregateFieldsMap = null;
251285
@Nullable private ExplainMetrics metrics = null;
286+
private int attempt;
252287

253-
AggregateQueryResponseObserver(ResponseDeliverer<T> responseDeliverer) {
288+
AggregateQueryResponseObserver(ResponseDeliverer<T> responseDeliverer, int attempt) {
254289
this.responseDeliverer = responseDeliverer;
290+
this.attempt = attempt;
291+
}
292+
293+
Map<String, Object> getAttemptAttributes() {
294+
return Collections.singletonMap(ATTRIBUTE_KEY_ATTEMPT, attempt);
255295
}
256296

257297
private boolean isExplainQuery() {
258298
return this.responseDeliverer.getExplainOptions() != null;
259299
}
260300

261301
@Override
262-
public void onStart(StreamController streamController) {}
302+
public void onStart(StreamController streamController) {
303+
getTraceUtil()
304+
.currentSpan()
305+
.addEvent(SPAN_NAME_RUN_AGGREGATION_QUERY + " Stream started.", getAttemptAttributes());
306+
}
263307

264308
@Override
265309
public void onResponse(RunAggregationQueryResponse response) {
310+
getTraceUtil()
311+
.currentSpan()
312+
.addEvent(
313+
SPAN_NAME_RUN_AGGREGATION_QUERY + " Response Received.", getAttemptAttributes());
266314
if (response.hasReadTime()) {
267315
readTime = Timestamp.fromProto(response.getReadTime());
268316
}
@@ -288,8 +336,19 @@ public void onResponse(RunAggregationQueryResponse response) {
288336
@Override
289337
public void onError(Throwable throwable) {
290338
if (shouldRetry(throwable)) {
291-
runQuery(responseDeliverer);
339+
getTraceUtil()
340+
.currentSpan()
341+
.addEvent(
342+
SPAN_NAME_RUN_AGGREGATION_QUERY + ": Retryable Error",
343+
Collections.singletonMap("error.message", throwable.getMessage()));
344+
345+
runQuery(responseDeliverer, attempt + 1);
292346
} else {
347+
getTraceUtil()
348+
.currentSpan()
349+
.addEvent(
350+
SPAN_NAME_RUN_AGGREGATION_QUERY + ": Error",
351+
Collections.singletonMap("error.message", throwable.getMessage()));
293352
responseDeliverer.deliverError(throwable);
294353
}
295354
}

‎google-cloud-firestore/src/main/java/com/google/cloud/firestore/BulkCommitBatch.java‎

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -21,13 +21,10 @@
2121
import com.google.api.gax.rpc.ApiException;
2222
import com.google.cloud.Timestamp;
2323
import com.google.common.base.Preconditions;
24-
import com.google.common.collect.ImmutableMap;
2524
import com.google.common.util.concurrent.MoreExecutors;
2625
import com.google.firestore.v1.BatchWriteRequest;
2726
import com.google.firestore.v1.BatchWriteResponse;
2827
import io.grpc.Status;
29-
import io.opencensus.trace.AttributeValue;
30-
import io.opencensus.trace.Tracing;
3128
import java.util.ArrayList;
3229
import java.util.List;
3330
import java.util.Set;
@@ -69,18 +66,10 @@ ApiFuture<WriteResult> wrapResult(int writeIndex) {
6966
* <p>The writes in the batch are not applied atomically and can be applied out of order.
7067
*/
7168
ApiFuture<Void> bulkCommit() {
72-
7369
// Follows same thread safety logic as `UpdateBuilder::commit`.
7470
committed = true;
7571
BatchWriteRequest request = buildBatchWriteRequest();
7672

77-
Tracing.getTracer()
78-
.getCurrentSpan()
79-
.addAnnotation(
80-
TraceUtil.SPAN_NAME_BATCHWRITE,
81-
ImmutableMap.of(
82-
"numDocuments", AttributeValue.longAttributeValue(request.getWritesCount())));
83-
8473
ApiFuture<BatchWriteResponse> response =
8574
processExceptions(
8675
firestore.sendRequest(request, firestore.getClient().batchWriteCallable()));

0 commit comments

Comments
 (0)