fix(oaistream): replay ReasoningContent when converting messages - #4365
20csm2k25-netizen wants to merge 1 commit into
Conversation
Reasoning captured from OpenAI-compatible custom providers (e.g. Qwen via llama.cpp) was stored on chat.Message.ReasoningContent but never read back when converting the conversation for the next request. convertMessagesWithCaps built the outgoing assistant message from Content, FunctionCall, and ToolCalls only, so reasoning was silently dropped on replay. This meant a reasoning model could never see its own prior reasoning after a tool call, forcing it to reconstruct that analysis from scratch on every subsequent turn, inflating reasoning tokens and latency in long-running agent workflows. openai-go's typed ChatCompletionAssistantMessageParam has no field for reasoning_content, since it is a non-standard, provider-specific extension (see openai/openai-go#558), so the stored reasoning is attached via SetExtraFields instead. Fixes docker#4363 Signed-off-by: Ahsan Ali <20csm2k25@gmail.com>
|
👋 Some commits in this PR are not signed and verified by GitHub. Please sign your commits with a GPG or SSH key registered in your GitHub account, then force-push. Commits that are not verified: See GitHub's guide on signing commits for setup instructions. I've added |
|
Hi - original issue reporter here. Thanks for picking this up. A couple of things stood out to me on this PR:
I hit this immediately with a long-running Qwen session: restoring the reasoning caused the effective context size to jump significantly, so without the corresponding accounting change the compaction behaviour became incorrect. Happy to share the relevant part of my local patch if useful. |
|
A clarification after investigating further: the compaction problem appears to be a transition edge case for existing sessions, rather than something this PR needs to address. My session had accumulated stored reasoning while it was still omitted from requests. Enabling replay suddenly added that history to the next prompt, while provider-reported usage still reflected the smaller pre-fix requests. That caused the jump from roughly 71K to 277K tokens. For new sessions where reasoning is replayed from the beginning, usage should progressively reflect that context, so the focused change here looks like the correct scope for #4363. The stale-usage and llama.cpp overflow-recovery cases can be handled separately. |
Fixes #4363
Root cause
convertMessagesWithCapsinpkg/model/provider/oaistream/messages.gobuilt the outgoing assistant message param fromContent,FunctionCall, andToolCallsonly.msg.ReasoningContentwas never read, so reasoning captured from OpenAI-compatible custom providers (e.g. Qwen via llama.cpp) was silently dropped when the conversation history was replayed on the next request.Fix
openai-go's typedChatCompletionAssistantMessageParamhas no field forreasoning_content— it's a non-standard, provider-specific extension (see openai/openai-go#558). This PR attaches the stored reasoning viaSetExtraFieldswhenmsg.ReasoningContentis non-empty, so it's a no-op for every model that doesn't produce reasoning.Open question for maintainers
The issue notes: "We should at least have an option to preserve reasoning when defining models against this provider, or have it set by default." This PR makes it default-on for the OpenAI-compatible path. Given the reporter's own finding that replaying reasoning inflated a request from ~120K to ~277K tokens, happy to add a per-model opt-out (e.g. via the existing
modelinfo.CapsOverridemechanism from #2741) if that's preferred instead.Testing
Added
repro_issue4363_test.gocovering: reasoning is carried onto the replayed message, no field is added when reasoning is absent, and reasoning survives alongside tool calls. Full package test suite passes.