Skip to content

fix(oaistream): replay ReasoningContent when converting messages - #4365

Open
20csm2k25-netizen wants to merge 1 commit into
docker:mainfrom
20csm2k25-netizen:fix/reasoning-content-oaistream
Open

20csm2k25-netizen wants to merge 1 commit into
docker:mainfrom
20csm2k25-netizen:fix/reasoning-content-oaistream

Conversation

@20csm2k25-netizen

Copy link
Copy Markdown

Fixes #4363

Root cause

convertMessagesWithCaps in pkg/model/provider/oaistream/messages.go built the outgoing assistant message param from Content, FunctionCall, and ToolCalls only. msg.ReasoningContent was never read, so reasoning captured from OpenAI-compatible custom providers (e.g. Qwen via llama.cpp) was silently dropped when the conversation history was replayed on the next request.

Fix

openai-go's typed ChatCompletionAssistantMessageParam has no field for reasoning_content — it's a non-standard, provider-specific extension (see openai/openai-go#558). This PR attaches the stored reasoning via SetExtraFields when msg.ReasoningContent is non-empty, so it's a no-op for every model that doesn't produce reasoning.

Open question for maintainers

The issue notes: "We should at least have an option to preserve reasoning when defining models against this provider, or have it set by default." This PR makes it default-on for the OpenAI-compatible path. Given the reporter's own finding that replaying reasoning inflated a request from ~120K to ~277K tokens, happy to add a per-model opt-out (e.g. via the existing modelinfo.CapsOverride mechanism from #2741) if that's preferred instead.

Testing

Added repro_issue4363_test.go covering: reasoning is carried onto the replayed message, no field is added when reasoning is absent, and reasoning survives alongside tool calls. Full package test suite passes.

Reasoning captured from OpenAI-compatible custom providers (e.g. Qwen
via llama.cpp) was stored on chat.Message.ReasoningContent but never
read back when converting the conversation for the next request.
convertMessagesWithCaps built the outgoing assistant message from
Content, FunctionCall, and ToolCalls only, so reasoning was silently
dropped on replay.

This meant a reasoning model could never see its own prior reasoning
after a tool call, forcing it to reconstruct that analysis from
scratch on every subsequent turn, inflating reasoning tokens and
latency in long-running agent workflows.

openai-go's typed ChatCompletionAssistantMessageParam has no field
for reasoning_content, since it is a non-standard, provider-specific
extension (see openai/openai-go#558), so the stored reasoning is
attached via SetExtraFields instead.

Fixes docker#4363

Signed-off-by: Ahsan Ali <20csm2k25@gmail.com>
@20csm2k25-netizen
20csm2k25-netizen requested a review from a team as a code owner September 19, 2026 19:07
@aheritier aheritier added area/providers For features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.) area/providers/openai For features/issues/fixes related to the usage of OpenAI models kind/fix PR fixes a bug (maps to fix:). Use on PRs only. status/needs-signed-commits Some commits in the PR are signed with a valid SSH/GPG key labels Sep 19, 2026
@aheritier

Copy link
Copy Markdown
Collaborator

👋 Some commits in this PR are not signed and verified by GitHub. Please sign your commits with a GPG or SSH key registered in your GitHub account, then force-push.

Commits that are not verified: 630ad45

See GitHub's guide on signing commits for setup instructions. I've added status/needs-signed-commits; it will be removed automatically once every commit in this PR carries a valid GitHub-verified signature.

@MysticalMount

MysticalMount commented Sep 20, 2026

Copy link
Copy Markdown

Hi - original issue reporter here. Thanks for picking this up. A couple of things stood out to me on this PR:

  • gofmt hasn’t been run on the test file, which makes the diff a little difficult to read.
  • In my local testing, simply restoring ReasoningContent on replay wasn’t sufficient. The token estimation used for compaction also needed to account for the restored reasoning tokens; otherwise the compaction threshold was substantially underestimated once reasoning started being replayed.

I hit this immediately with a long-running Qwen session: restoring the reasoning caused the effective context size to jump significantly, so without the corresponding accounting change the compaction behaviour became incorrect.

Happy to share the relevant part of my local patch if useful.

@MysticalMount

Copy link
Copy Markdown

A clarification after investigating further: the compaction problem appears to be a transition edge case for existing sessions, rather than something this PR needs to address.

My session had accumulated stored reasoning while it was still omitted from requests. Enabling replay suddenly added that history to the next prompt, while provider-reported usage still reflected the smaller pre-fix requests. That caused the jump from roughly 71K to 277K tokens.

For new sessions where reasoning is replayed from the beginning, usage should progressively reflect that context, so the focused change here looks like the correct scope for #4363. The stale-usage and llama.cpp overflow-recovery cases can be handled separately.

@aheritier aheritier added status/needs-signed-commits Some commits in the PR are signed with a valid SSH/GPG key and removed status/needs-signed-commits Some commits in the PR are signed with a valid SSH/GPG key labels Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/providers/openai For features/issues/fixes related to the usage of OpenAI models area/providers For features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.) kind/fix PR fixes a bug (maps to fix:). Use on PRs only. status/needs-signed-commits Some commits in the PR are signed with a valid SSH/GPG key

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI custom provider does not send reasoning from previous turns

3 participants