Skip to content

perf(litellm): avoid serializing streaming chunks - #944

Merged
Abhijeet Prasad (AbhiPrasad) merged 2 commits into
mainfrom
abhi/raw-litellm-streaming-98203204
Oct 16, 2026
Merged

Abhijeet Prasad (AbhiPrasad) merged 2 commits into
mainfrom
abhi/raw-litellm-streaming-98203204

Conversation

@AbhiPrasad

@AbhiPrasad Abhijeet Prasad (AbhiPrasad) commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Retain raw LiteLLM completion chunks while snapshotting only aggregation fields before yielding, avoiding per-chunk model serialization and insulating logged output from consumer mutations.
  • Build fresh function dictionaries while aggregating tool-call deltas.
  • Append Responses annotations as raw values for serialization at the logging boundary.

Logged shape

Raw Pydantic annotations are serialized by core deep-copy with exclude_none=True, so None fields may be omitted compared with the previous conversion path. _parse_metrics_from_usage remains unchanged.

Validation

  • VCR-backed sync/async completion streaming test, including mutation of yielded chunks — 4 passed.
  • VCR-backed sync/async Responses streaming tests — 2 passed.
  • mise exec -- pre-commit run --files py/src/braintrust/integrations/litellm/tracing.py py/src/braintrust/integrations/litellm/test_litellm.py — passed.
  • mise exec -- nox -s pylint — passed.
  • Existing cassettes reused; none re-recorded or changed.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T00:15:23.656818Z 433e076 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bc1ee3e38f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment thread py/src/braintrust/integrations/litellm/tracing.py Outdated
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit 0e52647 into main Oct 16, 2026
191 of 195 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the abhi/raw-litellm-streaming-98203204 branch October 16, 2026 00:32
@AbhiPrasad

Copy link
Copy Markdown
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant