Concurrent writes to a remote-signing catalog go out unsigned and fail 403 #3896

Activity

1. VivekMalipatel commented on Sep 16, 2026

   

   VivekMalipatel

   on Sep 16, 2026

   More actions

   We hit this in production against a remote-signing REST catalog (Lakekeeper) with an S3-compatible store, and tracked it to a specific interaction between two caches. Reads are affected as well as writes. Sharing the mechanism, the evidence, and — more usefully — why the obvious one-line fix is not safe in general.

   Versions: pyiceberg 0.12.0, botocore 1.43.75, s3fs 2026.7.0, fsspec 2026.7.0, aiobotocore 3.9.1, Python 3.11.

   Mechanism

   pyiceberg/io/fsspec.py::_s3 installs the signer by unregister-then-register, and sets signature_version=UNSIGNED:

   fs = S3FileSystem(**s3_fs_kwargs)
   for event_name, event_function in register_events.items():
       fs.s3.meta.events.unregister(event_name, unique_id=1925)
       fs.s3.meta.events.register_last(event_name, event_function, unique_id=1925)

   Two caches with different scopes collide:

   • FsspecFileIO.get_fs is memoised per thread (threading.local + lru_cache), so every worker in the scan pool runs _s3 itself.
   • fsspec's _Cached metaclass caches S3FileSystem per process, so all of those workers get the same instance and mutate the same fs.s3.meta.events emitter.

   Between the unregister and the register_last there is no before-sign.s3 handler. Because signing is UNSIGNED, a request that serialises in that gap is not signed by anything and goes out with no Authorization header at all — it is not a bad signature, it is an unauthenticated request. The store then answers with whatever it does for anonymous access; for us that is 403 AccessDenied, which is why this reads as a permissions problem and sends you looking at the catalog, the credentials, and clock skew.

   Short-lived jobs are hit hardest: the registration storm and the first object requests are the same event, so the first file of a scan was the victim nearly every time, while a long-lived process that warms its pool once never failed.

   Evidence

   Wrapping AioEndpoint._send to pair each request with its response:

   • failing request carried only User-Agent, amz-sdk-invocation-id, amz-sdk-request, and the signer had never seen its URL
   • a request that succeeded seconds earlier on the same connection carried Authorization, x-amz-date, x-amz-content-sha256, and the signer had seen its URL

   A packet capture confirmed the failing request's response body is byte-identical to that of a deliberately unauthenticated request, and distinct from both a duplicated-header and a wrong-secret response.

   Interleaved A/B on one scan, n=50: 9/25 failures with 6 workers, 0/25 with 1 worker, and every failure carried at least one unsigned request while no success did. In production, across short-lived jobs, 23 of 47 attempts (49%) failed before and 0 of 10 after.

   Workaround

   Setting max workers to 1 for short-lived readers removes it entirely. We deliberately did not add a 403 retry, since that would mask this class of bug.

   Why deleting the unregister is tempting but unsafe

   botocore/hooks.py::_register_section early-returns for a unique_id it already holds:

   if unique_id in self._unique_id_handlers:
       # We've already registered a handler using this unique_id
       # so we don't need to register it again.
       ...
       return

   So re-registering the same unique_id is a no-op, and it is natural to conclude the unregister is redundant and can simply be dropped — closing the window while keeping concurrency.

   That breaks multi-table processes. With remote signing, the signer is per table, not per catalog: s3.signer.endpoint embeds the table id, and S3V4RestSigner.__call__ reads self.properties, so each instance is bound to the table it was constructed for. Three tables in one of our processes give three distinct endpoints:

   v1/signer/<warehouse-id>/tabular-id/<table-A-id>/v1/aws/s3/sign
   v1/signer/<warehouse-id>/tabular-id/<table-B-id>/v1/aws/s3/sign
   v1/signer/<warehouse-id>/tabular-id/<table-C-id>/v1/aws/s3/sign
   

   A job that reads table A and then table B shares one process-cached S3FileSystem. Drop the unregister and botocore's early-return keeps table A's signer installed and silently skips table B's registration — so table B's object requests are signed against table A's signer path. Against a catalog that authorises per table location that is either a 403 for a new reason or, worse, an authorisation that should not have been granted. The unregister is not vestigial; it is what swaps the per-table signer on a shared client.

   So the root cause is narrower than "the unregister": a per-table signer installed on a process-cached client and mutated under concurrency.

   Suggested direction

   Make multiple signers able to coexist instead of swapping one in place — for example scope each signer to its own table's object prefix so it no-ops for requests outside it, and register it under a unique_id derived from the table id, never unregistering. No gap, no swap, concurrency preserved. Giving each table its own S3FileSystem instance (bypassing the fsspec instance cache) would also work, at the cost of more clients per process.

   Whatever the fix, a regression test that reads two different tables concurrently in one process and asserts each request was signed against its own table's endpoint would be worth having: a test that only checks "no 403 under concurrency" passes for the unsafe patch above.