Skip to content

[DOCS] Chat example says role-only stream chunks validate, but the model rejects them #147

Closed

Description

@inference-gateway-maintainer

Summary

The Streaming Chat Completion snippet in examples/chat/README.md says "The first chunk carries only role (no content) and validates fine." The code contradicts this. The generated ChatCompletionStreamResponseDelta declares content: str with no default (inference_gateway/models.py:2408), and ChatCompletionStreamChoice.finish_reason is a required FinishReason whose literal does not allow null (inference_gateway/models.py:2479 and :800-803). The repo's own strict xfail test test_stream_response_delta_allows_missing_content (tests/test_client.py:487-511) asserts that ChatCompletionStreamResponseDelta.model_validate({"role": "assistant"}) raises ValidationError. The snippet only catches json.JSONDecodeError, so a role-only chunk, or any chunk with "finish_reason": null, raises an uncaught pydantic.ValidationError out of the loop. examples/chat/main.py, which the README tells users to run, uses the same except json.JSONDecodeError-only pattern. The root README.md "Streaming Content" section already handles this correctly: it catches (json.JSONDecodeError, ValidationError) and skips the chunk.

Acceptance Criteria

  • examples/chat/README.md: remove the comment that a role-only first chunk "validates fine". CreateChatCompletionStreamResponse.model_validate raises ValidationError for a delta without content and for a choice whose finish_reason is null.
  • examples/chat/README.md: catch pydantic.ValidationError alongside json.JSONDecodeError in the streaming snippet and skip the chunk, matching the root README.md "Streaming Content" example.
  • examples/chat/main.py: apply the same ValidationError handling so python main.py does not crash on a chunk the strict model rejects.

Activity

  1. edenreich commented on Oct 4, 2026

    @edenreich
    Contributor

    @claude can you work on this? use /ponytail:ponytail

  2. inference-gateway-maintainer commented on Oct 4, 2026

    @inference-gateway-maintainer
    ContributorAuthor

    Claude finished @edenreich's task in 1m 14s —— View job • docs/issue-147-20261004-1152 • [Create PR ➔](https://github.com/inference-gateway/python-sdk/compare/main...docs/issue-147-20261004-1152?quick_pull=1&title=Issue%20%23147%3A%20Changes%20from%20Claude&body=This%20PR%20addresses%20issue%20%23147%0A%0AGenerated%20with%20%5BClaude%20Code%5D(https%3A%2F%2Fclaude.ai%2Fcode)


    Done - PR #150

    • Gather context (read example + root README.md pattern)
    • Understand request (docs/example fix)
    • examples/chat/README.md: removed the "validates fine" claim, now notes the model rejects a delta without content and a null finish_reason; snippet catches (json.JSONDecodeError, ValidationError) and continues
    • examples/chat/main.py: added a ValidationError handler so the script skips rejected chunks instead of crashing
    • Commit + open PR
    • Lint/test - not run: pip install is blocked in this sandbox, so black/isort/mypy/pytest were unavailable. Add Bash(pip install:*) to --allowedTools if you want them run here. CI will cover it; no library code changed.

    Import order in the README snippet was aligned with the root README.md ("Streaming Content"). No SDK behavior change, so no [DOCS] ticket and no downstream repo impact.

    PR: #150
    | Branch

  3. moved this from QA to Done in 2026on Oct 4, 2026
  4. inference-gateway-releaser commented on Oct 21, 2026

    @inference-gateway-releaser
    Contributor

    🎉 This issue has been resolved in version 0.22.0 🎉

    The release is available on GitHub release

    Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationreleased

    Type

    Projects

    • Status
      Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions