OTEP: Recording exceptions as log based events

[lmolkova]

Related to open-telemetry/semantic-conventions#1536

Changes

Recording exceptions as span events is problematic since it

• ties recording exceptions to tracing/sampling
• duplicates exceptions recorded by instrumented libraries on logs
• does not leverage log features such as typical log filtering based on severity

This OTEP provides guidance on how to record exceptions using OpenTelemetry logs focusing on minimizing duplication and providing context to reduce the noise.

If accepted, the follow-up spec changes are expected to replace existing (stable) documents:

• https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/exceptions.md
• https://github.com/open-telemetry/semantic-conventions/blob/main/docs/exceptions/exceptions-spans.md

---

• Related issues Document generic approach for span status (code + description) and exception event when instrumented code throws semantic-conventions#1536
• Related OTEP(s) #
• Links to the prototypes (when adding or changing features)
• CHANGELOG.md file updated for non-trivial changes
• spec-compliance-matrix.md updated if necessary

[pellared]

I think this is a related issue:

• Define whether instrumentations can emit non-event log records #4234

[carlosalberto]

A small doubt:

If this instrumentation supports tracing, it should capture the error in the scope of the processing span.

Although (I think) it's not called out, I'm understanding exceptions should now be explicitly reported as both 1) Span.Event and 2) Log/Event? i.e. coding wise you should do this:

currentSpan.recordException(e);
logger.logRecordBuilder
    .addException(e);

Is this the case?

[jsuereth]

Overall I'm very supportive. Just some nits and one mitigation I'd like to see called out/addressed.

[jsuereth]

What's an example of this? I'm struggling to understand when this would be used.

[jsuereth]

How do you know an error is transient when writing instrumentation? I think you mean errors that you KNOW the application will attempt to handle / retry, right?

I'd suggest rewording (or defining the meaning of transient).

[jsuereth]

I like the goal of the taxonomy, but think we need to crisp up the language around Info/Warning

[jsuereth]

Two things:

1. SHOULD is normative, so please capatilize (and I think IS a normative statement here).
2. This may not be language neutral, so I think SHOULD is the right guidance here. For example, in Rust, stack traces are something you can opt-in on an error. They leave some details to libraries (see Rust Backtrace/Source capabilities on https://docs.rs/thiserror/latest/thiserror/ e.g. or C++ [prior to 23] https://github.com/jeremy-rifkin/cpptrace).
   Additionally, in some highly green-thread/async APIs, I've seen custom stack traces created (e.g. Scala's ZIO where they try to preserve logical stack when physical stack is a confusing mess of work-stealing green-threads. We should allow these to interact with exception reporting in OTEL in some fashion.

I agree with the sentiment, I'd expand the wording though to allow languages like Rust/C++ (and Java ecosystem) to provide stack trace compatibility with their library ecosystem.

[jsuereth]

I think we should also call out the fact that now we have two channels of exporting/batching/recording information of exceptions and Traces. In this new world, you may see a trace before an exception or vice versa, and one may be dropped where the other is not.

We probably need some other mitigatioin should that requiring knowledge of an exception event under a Span is no longer needed (e.g. more aggressively using Span.status and attributes around "transient failures" as we discussed in Semconv SIG.

[pellared]

We should also call out that that it would be helpful for metrics instrumentation libraries (that do not produce spans at all, like runtime instrumentation) so they could report exceptions (errors).