fix(tracing): don't export provider gen_ai spans nested in a Langfuse generation

[zikrulihsan]

What does this PR do?

Fixes langfuse/langfuse#18978

Since anthropic 1.14, the Anthropic SDK emits its own anthropic.messages.create span when a global tracer provider exists. With the LangChain CallbackHandler, this span is a direct child of the ChatAnthropic generation, and the default filter exports it because it has gen_ai.* attributes. So every Claude call shows up as two generations, and token/cost dashboards show about 2x.

With this change, when the default export filter is used, LangfuseSpanProcessor drops third-party gen_ai.* spans whose direct parent is an open Langfuse generation. The parent's type is checked when the child ends, because langfuse.observation.type is set after the span starts.

See the image below:

Notes:

• Provider spans without a Langfuse generation parent are still exported (e.g. using the Anthropic SDK directly).
• If should_export_span is set, nothing changes. Users who want both spans can pass should_export_span=is_default_export_span.
• Behavior change: this applies to any third-party gen_ai.* span directly under a Langfuse generation, not only Anthropic. If someone relies on an instrumentor child span for usage/cost under a manual generation, they would lose it. They can keep the old behavior with should_export_span=is_default_export_span. I can narrow it (e.g. only the Anthropic scope) if you prefer.
• Workaround until this is released: ANTHROPIC_OPEN_TELEMETRY=false.
• The cache-write double count in the issue seems already fixed server-side in fix(otel): normalize official cache-write usage langfuse#17129 (the reporter is on server 4.3.1).
• Limitation: this fixes invoke and stream, not ainvoke / astream. In async runs, LangChain calls the sync CallbackHandler in an executor with a copied context, so the generation is not the OTel parent of the provider call, and the Anthropic span becomes its own trace. Setting run_inline = True on the handler fixes this in my local test, but it was added in fix(langchain): maintain async context in tool executions #1380 and removed in feat!: prepare v4 release #1486, so I left it out of this PR.

cc @hassiebp since you're assigned to the issue and own most of span_processor.py. Happy to change the approach if you prefer another one.

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor
• Documentation update
• Tooling, CI, or repo maintenance

Verification

List the main commands you ran:

uv run --frozen ruff format .
uv run --frozen ruff check .
uv run --frozen mypy langfuse --no-error-summary
uv run --frozen pytest tests/unit

• New tests in tests/unit/test_generation_child_span_filter.py. The two "drops" tests fail without the fix.

• tests/unit: all pass except 18 errors in tests/unit/test_prompt.py ("Langfuse client is not initialized"), which also happen on main without this change in my local setup.

• End to end with anthropic 1.15.0 + langchain-anthropic 1.7.4 against a local fake Anthropic API and OTLP endpoint:

  Mode	Before	After
  invoke	2 spans	1 span
  stream	2 spans	1 span
  ainvoke / astream	2 spans	2 spans (not fixed, see Limitation in Notes)

  A direct anthropic.Anthropic().messages.create(...) call is still exported.

• Same runs against a local self-hosted Langfuse server (v4.59.0, official docker-compose.yml), counting stored GENERATION observations through /api/public/v2/observations: invoke and stream went from 2 generations / 230 total tokens to 1 generation / 115 tokens. ainvoke and astream stay at 2 generations in 2 separate traces.

Checklist

• I self-reviewed the diff using code_review.md.
• I added or updated tests for behavior changes.
• I updated docs, examples, or .env.template if needed.
• I did not hand-edit generated files; if generated files changed, I used the upstream regeneration path.
• I did not commit secrets or credentials.

Confidence Score: 5/5

The PR appears safe to merge within its stated scope.

Summary

The default filter now drops third-party gen_ai.* spans directly beneath an open Langfuse generation. It reads the parent's current observation type when the child ends.

• Nested provider spans no longer appear as a second generation.

Acknowledged scope from zikrulihsan: suppression intentionally covers all providers, including calls under manual generations. Async LangChain calls remain outside this fix because their provider spans do not share the generation parent.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Provider span ends] --> B{Default filter and gen_ai attributes?}
  B -->|No| E[Apply configured export filter]
  B -->|Yes| C{Direct parent is an open Langfuse generation?}
  C -->|No| E
  C -->|Yes| D[Drop duplicate provider span]

Loading

_{Reviews (1) · Last reviewed commit: "fix(tracing): don't export provider gen_..." · Reviewed by Greptile}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[CLAassistant]

All committers have signed the CLA.

[shashb27]

Verified this against a repro I'd built earlier (anthropic 1.15 with a mock transport and an in-memory exporter): the double-export reproduces on main, and with this change the nested span drops while a root-level provider span still exports — that second part is the case worth protecting, since setups using anthropic's native OTel as their only instrumentation rely on that span as their only generation record. One edge case worth considering: a provider span nested under a non-generation Langfuse span (agent/span wrappers) would still double-log — might be fine to leave, just flagging it.