Skip to content

Fix receiver write admission stalls and RPC buffer allocation failures - #18918

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-receiver-memory-pressure
Open

Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-receiver-memory-pressure

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Large tablet transfers can encounter avoidable receiver stalls in two independent memory-control paths:

  • When write memory is rejected, an insert holding the region write lock can wait for a different TsFileProcessor in the same region to flush. The asynchronous flush submitter needs that lock too. Mark the current memtable for flushing so the writer can finish and release the lock. Admission still checks the existing hard memtable limit, and normal flush selection still prefers the largest memtables. This also handles a first tablet whose memory has been reserved before the memtable contains data.
  • RPC buffers skip shrinking for requests at or below their initial capacity, including flush resets, retaining oversized arrays and their pool reservations. Include these requests in the existing shrink policy while preserving the initial capacity and shrink cooldown. If optional 1.5x growth does not fit, reserve only the required frame size before waiting or failing; successful fallback does not increment allocation-failure metrics.

Validation

  • Regression tests reproduced the admission and RPC allocation failures before the fixes.
  • English full-reactor build and 133 selected tests passed: 103 Service-RPC tests, 4 SystemInfo tests, and 26 TsFileProcessor tests.
  • Chinese full-reactor build and 44 targeted tests passed: AutoResizingBufferTest, SystemInfoTest, and TsFileProcessorTest. Both locale builds covered all 53 reactor modules, including main and test compilation, with Checkstyle and Spotless checks.
  • The broader Chinese Service-RPC run exposed two existing English-only message assertions in TElasticFramedTransportTest.testSingularSize and DeepCopyRpcTransportFactoryTest.testIndependentMaxFrameSizeConfigurations. The buffer tests passed in that run.
  • The original full historical-sync workload has not been rerun; throughput improvement still needs workload verification.

This PR has:

  • been self-reviewed.
  • added comments explaining the intent of the changes.
  • added regression tests covering memory admission, hard-limit rejection, frame integrity, capacity reuse between connections, and allocation accounting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant