add migration section, update trade-offs and minitations, clean up

Author: Liudmila Molkova

oteps/4333-recording-exceptions-on-logs.md

@@ -6,6 +6,7 @@
 - [Guidance](#guidance)
  * [Details](#details)
 - [API changes](#api-changes)
+- [Migrating instrumentations](#migrating-instrumentations)
 - [Examples](#examples)
  * [Logging errors from client library in a user application](#logging-errors-from-client-library-in-a-user-application)
  * [Logging errors inside the natively instrumented Library](#logging-errors-inside-the-natively-instrumented-library)
@@ -19,7 +20,7 @@
 
 <!-- tocstop -->
 
-This OTEP provides guidance on how to record errors using OpenTelemetry Logs
+This OTEP provides guidance on how to record errors using OpenTelemetry Logs,
 focusing on minimizing duplication and providing context to reduce the noise.
 
 In the long term, errors recorded on logs **will replace span events**
@@ -35,7 +36,7 @@ In the long term, errors recorded on logs **will replace span events**
 
 ## Motivation
 
-Today OTel supports recording *exceptions* using span events available through the Trace API. Outside the OTel world,
+Today, OTel supports recording *exceptions* using span events available through the Trace API. Outside the OTel world,
 *errors* are usually recorded by user apps and libraries by using logging libraries,
 and may be recorded as OTel logs via a logging bridge.
 
@@ -48,20 +49,20 @@ Using logs to record errors has the following advantages over using span events:
 
 Recording errors is essential for troubleshooting, but regardless of how they are recorded, they could be noisy:
 
-- distributed applications experience transient errors at the rate proportional to their scale and
- errors in logs could be misleading - individual occurrences of transient errors
+- distributed applications experience transient errors at a rate proportional to their scale, and
+ errors in logs could be misleading. Individual occurrences of transient errors
  are not necessarily indicative of a problem.
-- exception stack traces can be huge. The corresponding attribute value can frequently reach several KBs resulting in high costs
+- exception stack traces can be huge. The corresponding attribute value can frequently reach several KBs, resulting in high costs
  associated with ingesting and storing them. It's also common to log errors multiple times
- as they bubble up leading to duplication and aggravating the verbosity problem.
+ as they bubble up, leading to duplication and aggravating the verbosity problem.
 - severity depends on the context and, in the general case, is not known at the time the error
  occurs since errors are frequently handled (suppressed, retried, ignored) by the caller.
 
 In this OTEP, we'll provide guidance around recording errors that minimizes duplication,
 allows reducing noise with configuration, and allows capturing errors in the
 absence of a recorded span.
 
-This guidance applies to general-purpose instrumentations including natively
+This guidance applies to general-purpose instrumentations, including natively
 instrumented libraries.
 
 Application developers should consider following it as a starting point, but
@@ -75,7 +76,7 @@ Instrumentations SHOULD record error information along with relevant context as
 a log record with appropriate severity.
 
 Instrumentations SHOULD set severity to `Error` or higher only when the log describes a
-problem affecting application functionality, availability, performance, security or
+problem affecting application functionality, availability, performance, security, or
 another aspect that is important for the given type of application.
 
 When instrumentation records an exception, it SHOULD provide
@@ -109,14 +110,14 @@ be to record exception stack traces when logging exceptions at `Error` or higher
  severity not higher than `Info`.
 
  Such errors can be used to control application logic and have a minor impact, if any,
- on application functionality, availability, or performance (beyond performance hit introduced
- if exception is used to control application logic).
+ on application functionality, availability, or performance (beyond the performance hit introduced
+ if an exception is used to control application logic).
 
  Examples:
 
  - an error is returned when checking optional dependency or resource existence.
- - an exception is thrown on the server when client disconnects before reading
- full response from the server
+ - an exception is thrown on the server when the client disconnects before reading
+ the full response from the server.
 
  - Errors that are expected to be retried or handled by the caller or another
  layer of the component SHOULD be recorded with severity not higher than `Warn`.
@@ -127,11 +128,11 @@ be to record exception stack traces when logging exceptions at `Error` or higher
 
  Examples:
 
- - an attempt to connect to the required remote dependency times out
- - a remote dependency returns 401 "Unauthorized" response code
- - writing data to a file results in an IO exception
- - a remote dependency returned 503 "Service Unavailable" response for 5 times in a row,
- retry attempts are exhausted and the corresponding operation has failed.
+ - an attempt to connect to the required remote dependency times out.
+ - a remote dependency returns a 401 "Unauthorized" response code.
+ - writing data to a file results in an IO exception.
+ - a remote dependency returned a 503 "Service Unavailable" response for 5 times in a row,
+ retry attempts are exhausted, and the corresponding operation has failed.
 
  - Unhandled (by the application code) errors that don't result in application
  shutdown SHOULD be recorded with severity `Error`
@@ -148,8 +149,8 @@ be to record exception stack traces when logging exceptions at `Error` or higher
  Note: some frameworks use exceptions as a communication mechanism when a request fails. For example,
  Spring users can throw a [ResponseStatusException](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/server/ResponseStatusException.html)
  exception to return an unsuccessful status code. Such exceptions represent errors already handled by the application code.
- Application code, in this case, is expected to log this at appropriate severity.
- General-purpose instrumentation MAY record such errors, but at severity not higher than `Warn`.
+ Application code, in this case, is expected to log this at the appropriate severity.
+ General-purpose instrumentation MAY record such errors, but at a severity not higher than `Warn`.
 
  - Errors that result in application shutdown SHOULD be recorded with severity `Fatal`.
 
@@ -169,33 +170,53 @@ be to record exception stack traces when logging exceptions at `Error` or higher
  records stack trace conditionally.
 
 8. Instrumentation libraries that record exceptions using span events SHOULD gracefully migrate
- to log-based exceptions offering it as an opt-in feature first and then switching to log-based exceptions
+ to log-based exceptions, offering it as an opt-in feature first and then switching to log-based exceptions
  in the next major version update.
 
 ## API changes
 
 > [!NOTE]
 >
-> It should not be an instrumentation concern to decide whether exception stack trace
+> It should not be an instrumentation concern to decide whether an exception stack trace
 > should be recorded or not.
 >
-> A natively instrumented library may write logs providing exception instance
+> A natively instrumented library may write logs providing an exception instance
 > through a log bridge and not be aware of this guidance.
 >
 > It also may be desirable by some vendors/apps to record all exception details at all levels.
 
-OTel Logs API SHOULD provide methods that enrich log record with exception details such as
-`setException(exception)` and similar to [RecordException](../specification/trace/api.md#record-exception) method on span.
+The OTel Logs API SHOULD provide methods that enrich log records with exception details such as
+`setException(exception)` and similar to the [RecordException](../specification/trace/api.md#record-exception) method on span.
 
-OTel SDK, based on the log severity and configuration, SHOULD record exception details fully or partially.
+The OTel SDK, based on the log severity and configuration, SHOULD record exception details fully or partially.
 
 The signature of the method is to be determined by each language
-and can be overloaded as appropriate including ability to customize stack trace
+and can be overloaded as appropriate, including the ability to customize stack trace
 collection.
 
 It MUST be possible to efficiently set exception information on a log record based on configuration
 and without using the `setException` method.
 
+## Migrating instrumentations
+
+> [!NOTE]
+> New instrumentations or existing ones that do not record exceptions on span events SHOULD
+> NOT start recording exceptions on span events. They SHOULD NOT implement the migration plan
+> described below.
+>
+> This section covers migration recommendations for existing instrumentations that already
+> report exceptions using span events.
+
+We will define a configuration option to let users choose if they want instrumentations to record exceptions
+on span events or logs.
+
+Specific instrumentation SHOULD default to recording exceptions on span events in its current major version
+and record them on logs only when the user opts-in.
+
+In the next major version, this instrumentation SHOULD stop recording exceptions on span events.
+
+This is a simplified version of [stability opt-in migration](https://github.com/open-telemetry/semantic-conventions/blob/727700406f9e6cc3f4e4680a81c4c28f2eb71569/docs/http/README.md?plain=1#L13-L37) used in semantic conventions.
+
 ## Examples
 
 ### Logging errors from client library in a user application
@@ -236,7 +257,7 @@ try {
 ### Logging errors inside the natively instrumented Library
 
 It's a common practice to record errors using logging libraries. Client libraries that are natively instrumented with OpenTelemetry should
-leverage OTel Events/Logs API for their exception logging purposes.
+leverage the OTel Events/Logs API for their exception logging purposes.
 
 ```java
 public class StorageClient {
@@ -250,9 +271,9 @@ public class StorageClient {
  }
 
  logger.logRecordBuilder()
- // In general we don't know if it's an error - we expect caller
- // to handle it and decide. So this is warning (at most).
- // If exception thrown below remains unhandled, it'd be logged by the global handler.
+ // In general we don't know if it's an error - we expect the caller
+ // to handle it and decide. So this is a warning (at most).
+ // If the exception thrown below remains unhandled, it'd be logged by the global handler.
  .setSeverity(Severity.WARN)
  .addAttribute(AttributeKey.stringKey("com.example.content.id"), contentId)
  .addAttribute(AttributeKey.stringKey("http.response.status_code"), response.statusCode())
@@ -268,7 +289,7 @@ public class StorageClient {
 }
 ```
 
-Network level errors are part of normal life, we should consider using low severity for them
+Network-level errors are part of normal life; we should consider using low severity for them.
 
 ```java
 public class NetworkClient {
@@ -281,7 +302,7 @@ public class NetworkClient {
  } catch (SocketException ex) {
  logger.logRecordBuilder()
  // we'll retry it, so it's info or lower.
- // we'll write a warn for overall operation if retries are exhausted.
+ // we'll write a warn for the overall operation if retries are exhausted.
  .setSeverity(Severity.INFO)
  .addAttribute("connection.id", this.getId())
  .addException(ex)
@@ -318,9 +339,9 @@ this example:
 ```java
 MessageContext context = retrieveNext();
 try {
- processMessage.accept(context)
+ processMessage.accept(context);
 } catch (Throwable t) {
- // This natively instrumented library may use OTel log API or another logging library such as slf4j.
+ // This natively instrumented library may use the OTel log API or another logging library such as slf4j.
  // Here we use Error severity since this exception was not handled by the application code.
  logger.atError()
  .addKeyValuePair("messaging.message.id", context.getMessageId())
@@ -336,7 +357,7 @@ span.
 
 #### Instrumentation library
 
-In this example we leverage Spring Kafka `RecordInterceptor` extensibility point that allows to
+In this example, we leverage the Spring Kafka `RecordInterceptor` extensibility point that allows us to
 listen to exceptions that remained unhandled.
 
 ```java
@@ -366,27 +387,23 @@ See [corresponding Java (tracing) instrumentation](https://github.com/open-telem
 1. Switching from recording exceptions as span events to log records is a breaking change
  for any component following existing [exception guidance](/specification/trace/exceptions.md).
 
- **Mitigation:**
- - OpenTelemetry API and/or SDK in the future may provide opt-in span events -> log-based events conversion,
- but that's not enough - instrumentations will have to change their behavior to report errors
- as logs with appropriate severity.
- - We should provide opt-in mechanism for existing instrumentations to switch to logs.
-
 2. Recording exceptions as log-based events would result in UX degradation for users
  leveraging trace-only backends such as Jaeger.
 
- **Mitigation:**
- - OpenTelemetry API and/or SDK may provide span events -> log events conversion.
- See also [Event vision OTEP](./0265-event-vision.md#relationship-to-span-events).
+**Mitigation:**
+
+In addition to the plan outlined in the [Migration](#migrating-instrumentations) section, we
+should provide opt-in [log <-> span events conversion](https://github.com/open-telemetry/opentelemetry-specification/issues/4393)
+following [Event vision OTEP](./0265-event-vision.md#relationship-to-span-events).
 
 ## Prior art and alternatives
 
 Alternatives:
 
 1. Deduplicate exception info by marking exception instances as logged.
- This can potentially mitigate the problem for existing application when it logs exceptions extensively.
- We should still provide optimal guidance for the greenfield applications and libraries,
- covering wider problem of recording errors.
+ This can potentially mitigate the problem for existing applications when they log exceptions extensively.
+ We should still provide optimal guidance for greenfield applications and libraries,
+ covering the wider problem of recording errors.
 
 ## Open questions