Repository navigation
feat(integrations): support multimodal decision spans - #941
Abhijeet Prasad (AbhiPrasad) wants to merge 4 commits into
Conversation
Preserve box answers and image associations across TypeSafe, OpenAI Decisions, and LiteLLM Decisions normalization. Capture nested image assets without changing provider requests, and reuse a shared media walker with per-call image caching. Test Jev 1.14 boxes and images with TypeSafe SDK 0.7.5 while retaining 0.7.4 coverage. Add focused regressions for named answers, nullable confidence, and attachment conversion fallback.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 62978193b0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| _log_and_end_span( | ||
| span, | ||
| output=_question_span_output(answers), | ||
| output=_question_span_output(answers, span_input), |
There was a problem hiding this comment.
When callers use TypeSafe's documented custom response_model pattern where response.answers is a Pydantic model rather than a Mapping (TypeSafe documentation), _answer_payload is skipped and this call passes the opaque model to _question_span_output, which returns without examining its box answers. Consequently, box normalization and the new single-image image_path inference do not run for these successful multimodal requests; convert model containers with _try_to_dict before checking for a mapping.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf3a10292b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
Reserve named answer matches before pairing the remaining questions and answers, so reordering across unnamed questions cannot duplicate an answer ID. Prefer longer TypeSafe asset references in embedded text, and let media transformers capture serialized image mappings before recursive traversal. Cover the regressions with focused normalization tests and the existing TypeSafe asset-reference recording.
This branch has not been deployed
1 participant
TypeSafe Jev 1.14 adds box answers and native image assets. Existing instrumentation drops box coordinates and logs native images as SDK internals. Capture both using the updated decision-span spec, and align the shared normalization used by OpenAI and LiteLLM Decisions.
Span flow
flowchart LR TS["TypeSafe system_one"] --> A["Provider adapters"] OA["OpenAI / LiteLLM Decisions"] --> A A --> IN["input: state + questions<br/>Nested media becomes attachments"] A --> OUT["output: answers<br/>Preserve IDs, box edges, confidence"] IN --> S["One decision span"] OUT --> S IN -. "One image: fill missing image_path" .-> OUTBehavior
decisionspan. Preserve fractional edges and confidence, including zero, null, and absence.image_pathwith an input-rooted, escaped JSON Pointer, such as/state/photos~1~0/0.Imageobjects, including subclasses, in state, instructions, and criteria. Resolve TypeSafe asset references while preserving text order.