Skip to content

feat(endode-extstore-refs): external storage references are now encoded and decoded. - #1946

Open
cconstable wants to merge 7 commits into
mainfrom
extstore-encode-refs
Open

cconstable wants to merge 7 commits into
mainfrom
extstore-encode-refs

Conversation

@cconstable

@cconstable cconstable commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

External storage reference payloads are now encoded with the configured payload codec. Every reference carries a __temporal_external_storage_reference metadata marker so the SDK can decode.

  • Outbound: codec encode → store → codec encode the reference → add the marker.
  • Inbound: marked payloads are codec-decoded → retrieved → codec-decoded.

References written by earlier versions, which are unencoded and unmarked, are still read as before.

Why

Reference payloads contain driver claim data, such as bucket names and object keys. That data was written to workflow history in plain text, even when a codec was configured to encrypt everything else.

Checklist

  • added tests
  • check client flow

@cconstable
cconstable requested a review from a team as a code owner October 8, 2026 18:08
@cconstable
cconstable marked this pull request as draft October 8, 2026 18:15
@cconstable
cconstable marked this pull request as ready for review October 13, 2026 18:04
@tconley1428

Copy link
Copy Markdown
Contributor

I don't see any issues, but I'll let @jmaeagle99 review

@jmaeagle99 jmaeagle99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments. And we should consider a change log entry that may be a breaking change to clearly describe the behavior here. The problem is if an open workflow is processed on a worker with this change and then further processed on a worker with a prior SDK version, that will likely fail, due to references being encoded.

Comment thread temporalio/converter/_data_converter.py Outdated
Comment thread temporalio/converter/_data_converter.py Outdated
Comment thread temporalio/converter/_data_converter.py
Comment thread tests/worker/test_extstore.py
Comment thread temporalio/worker/_nexus.py
Comment thread temporalio/converter/_extstore.py
Comment thread tests/test_extstore_client.py
Comment thread CHANGELOG.md Outdated
Comment thread tests/worker/test_extstore.py Outdated
Comment thread tests/test_extstore.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants