Status: accepted. Canonical slice: ClusterRouting; parent NativeCqrs. Decision: ADR-082; connection-owned execution: ADR-125; cluster concept: ADR-136. Related slices: ClusterReplication, ClientApi, Authorization, ResourceExecution and StorageRecovery.
The final-response .NET SDK and official MCP adapters consume one native ManagedCode.Communication CQRS stream from the client's Orleans connection grain. The connection grain invokes the logical partition grain, which calls the silo-local ZoneTree Grain Service on its own or another node through ordinary Orleans calls; a write is acknowledged after all KeyLoad:ReplicaCount copies confirm durable local apply. The current final JSON remains the external contract. Public progress streaming and resumable long operations are outside this contract. Frontend: N/A, no UI is involved.
flowchart LR
Caller[SDK or MCP caller] --> Identity[Persisted identity and signed request]
Identity --> Connection[Connection grain stream]
Connection --> Started[Native Started chunk]
Started --> Partition[Logical partition grain]
Partition --> Owner[Silo local ZoneTree Grain Service]
Owner --> Terminal[Completed or Failed terminal]
Terminal --> Drain[Bounded validation and EOF]
Drain --> Response[Final SDK and MCP response]
| Requirement | Measurable acceptance | Automated evidence |
|---|---|---|
| REQ-CRS-001: one versioned native response stream | AC-CRS-001: genuine native Orleans calls execute the signed read and command through the client's connection grain and the existing logical partition grain. The interface/method aliases and the generated progress record round-trip with the native Communication converter. There is one native stream entry and one operation dispatcher. | RequestCqrsRoutingTests.AcCrs001NativeRequestRoutesAStableBatchThroughItsCommandGrain; Aspire SDK/official MCP replicated-cluster operations |
| REQ-CRS-002: the signed request is verified before admission | AC-CRS-002: only the exact current signed purpose and envelope are admitted. An unsupported or stale purpose, tampered signature, wrong actor key, or malformed or trailing payload fails closed before identity-context publication, dispatch or effects. No conversion or alternate reader exists. | GrainRequestEnvelopeWireTests.NativeSignatureWithUnsupportedPurposeIsInvalidated; GrainRequestCodecTests.WrongActorKeyAndTamperedSignatureAreDenied and SignedMalformedOrTrailingPayloadIsDenied |
| REQ-CRS-003: the producer has a small truthful well-formed lifecycle | AC-CRS-003: lazy execution verifies the exact signed scope and request GUID before Started. Valid routing emits Started then one terminal; an early rejection emits one Failed. Sequences are exactly 1/2 or 1, and no percentage, work count, authorization success or commit progress is invented. Domain rejection, a genuine unexpected exception, cancellation and early disposal settle the actual native producer, a healthy following operation succeeds, and no identity or history crosses between concurrent operations. | RequestCqrsBoundaryTests.AcCrs003RealNativeProducerSettlesOnCancellation; RequestCqrsRoutingTests.AcCrs003RevokedCommandFailsSafelyAndFollowingRequestSucceeds; RequestCqrsFatalSettlementTests; RequestCqrsProducerNotificationSettlementTests |
| REQ-CRS-004: a stream cannot bypass resource or failure/privacy admission | AC-CRS-004: exact and excess chunk count, native encoded bytes, aggregate bytes, payload/detail bounds, sequence, kind, field shape, duplicate terminal, trailing chunk and missing terminal inputs fail closed. The consumer reaches actual EOF and disposes without materializing a chunk list. Native byte measurement uses the actual silo serializer without retaining another encoded payload. Controlled private exception text, data and stack never reach the public result or logs. | RequestCqrsBoundaryTests AcCrs004 cases; RequestCqrsPublicFatalMiddlewareTests.AcCrs003004HttpBoundaryPreservesNativeFatalIdentityAndAllowsNextRequest; RequestCqrsProbeCodecTests; RequestCqrsProbeRecordFileTests |
| REQ-CRS-005: persisted receipts retain write-outcome authority across interruption | AC-CRS-005: interruption before and after native submit preserves the stable command ID and persisted receipt. A caller without a validated final response observes UnknownWriteOutcome for an interrupted write; a same-ID retry with a fresh request GUID returns the canonical receipt without repeating effects. Cancellation is never proof of rollback. Read cancellation remains distinct. | OrleansRpcFailureTests.AcCrs005CallerCancellationPreservesWriteUncertaintyAndDistinctReadCancellation and AcCrs005UnvalidatedStreamFailuresDoNotExposeArbitraryExceptionDetails; RequestCqrsPhaseFaultTests (four SDK/official MCP cases); MultiLaneReceiveBoundaryWholeFlowTests |
| REQ-CRS-006: native Orleans RequestContext propagates bounded identity and request state without becoming authorization authority | AC-CRS-006: genuine native serialization and first/later pulls carry exactly the server-authenticated persisted subject and matching request/command GUIDs. Missing, extra, mismatched, unauthenticated and forged context fail before capability execution. Authentication carries no principal. Current persisted grants, expiry and revocation are checked again at execution. Concurrent streams, cancellation, failure and early disposal restore both exact prior context values; a healthy following request has no inherited identity. Native activation migration and restart reconstruct request state from a fresh signed request, never a retained activation or context cache. | RequestCqrsBoundaryTests AcCrs006 cases; RequestCqrsRoutingTests.AcCrs006MissingAndForgedContextFailBeforeCommandEffects; RequestCqrsAuthorityFaultTests.SdkWriteHeldAcrossPersistedRevocationIsUnauthorizedWithoutEffects and OfficialMcpWriteHeldAcrossPersistedRevocationIsUnauthorizedWithoutEffects |
| REQ-CRS-WORK-001: silo-local request work is bounded and joined | AC-CRS-WORK-001: see Bounded request work. Producer and total caps, duplicate identity, lazy admission, a held native producer finally, a paused consumer, repeated drain/disposal, a throwing cancellation callback and rejection followed by activation settlement are exercised with real native producers. Registry counts alone do not establish settlement. | NativeRequestWorkOwnerTests (three AcCrsWork001 cases); NativeRequestWorkKernelTests (four AcCrsWork001 cases) |
| REQ-CRS-JOIN-001: native producer cancellation always joins the original producer | AC-CRS-JOIN-001: a real Create producer with a throwing registered cancellation callback remains joined by early enumerator disposal; disposal cannot complete while that producer's finally is still active, and the callback failure is observable afterward. Healthy subsequent creation, ordinary cancellation and fatal-aggregate cases keep passing. | Owning ManagedCode.Communication regressions; KeyLoad consumes only the published, centrally pinned package |
| REQ-CRS-DIAG-001: MCP transport rejection diagnostics stay bounded and private | AC-CRS-DIAG-001: through the actual Aspire ResourceLoggerService, the fixed transport rejection message yields only defined stage/category records, at most 32 per node; oversized, numeric-enum, unknown-value and trailing-content lines are excluded. The JSON holds only version, wave, node and closed stage/category fields. Subscriptions are admitted before startup and joined before evidence is written; early stream completion and caller cancellation publish no success evidence. | RequestCqrsClusterDiagnosticsTests |
| REQ-CRS-DIAG-002: a real malformed MCP request is rejected without disturbing healthy callers | AC-CRS-DIAG-002: in a healthy Aspire cluster, one fixed malformed MCP transport request with a genuine persisted credential returns typed Validation and publishes one closed McpTransportGuard warning (BodyMethodMismatch / ToolsCall), captured as exactly that record; a following SDK and official MCP operation succeeds. Credentials, headers, targets, body and exception text never enter the artifact. |
RequestCqrsClusterMcpGuardEvidenceTests.AcCrsDiag002RealGuardWarningIsCapturedBetweenHealthySdkAndMcpCalls |
These criteria refine AC-NCQRS-001..004 in NativeCqrs.
IConnectionGrain.ExecuteStreamAsync is the single request entry. Its aliases and interface version come from the named GrainRoutingProtocol constants; the alias generation and the Orleans interface version are separate identifiers. The method returns native IAsyncEnumerable<CqrsStreamChunk<GrainRequestProgress, GrainOperationReply>> and accepts the exact signed string and a real CancellationToken. ICommandPartitionGrain keeps its Task leaf method as the canonical bounded operation, with a distinct capability interface version constant.
The generated GrainRequestProgress Started payload carries the request GUID. It means only that the verified signed request has begun routing; it is not authorization, successful execution or an acknowledgement. No user content, principal, token, command payload, row identity, resource name, percentage or estimate is carried in Started.
GrainOperationReply keeps its existing alias and field IDs. Successful terminal data is its exact bounded native GrainValue payload. A failure becomes a native failed Result with a controlled Problem, validated by GrainRequestStreamProblem: exactly one ErrorCode extension naming a defined KeyLoad error (urn:keyload:error:<name>), a title equal to that name, the existing status for that code, a positive SafeDetail of at most 4096 characters, and a null Instance. Arbitrary extensions, ErrorType, contradictory success/Problem, a failed Value, extra progress or trailing terminal fields are rejected. GrainReplyFactory and the consumer share this closed schema; generic FromException details are never accepted.
KeyLoad.Clustering references the centrally pinned ManagedCode.Orleans.Identity.Core package. It reuses OrleansIdentityConstants.USER_CLAIMS, the OrleansIdentityContext APIs and the native claims surrogate converters registered on the real silo serializer, proven by native round trips. KeyLoad does not use the package's HTTP context publisher or server authorization filter; the signed request and persisted database authorization remain the boundary, and no substitute authorization dispatcher exists.
The published ClaimsPrincipal has exactly one ClaimsIdentity with authentication type KeyLoad.PersistedPrincipal.v1 and exactly one NameIdentifier claim equal to the authenticated persisted principal ID (at most 256 UTF-8 bytes). No roles, grants, policy epochs, tenant, API credential, signed token, payload or user content are context fields. Context is correlation metadata; a matching subject grants nothing.
The generated GrainRequestContextState carries the request GUID, the stable command GUID (empty for a read) and the owning connection GUID under the fixed RequestContext key KeyLoad-RequestState. Before the stream is pulled, the native encodings are admitted with the same non-retaining counter as chunks: principal at most 4096 bytes and state at most 256 bytes.
GrainRequestIdentityScope publishes only these two keys using the running silo's serializers and restores their exact prior values, including absence, in finally. It covers the complete Orleans dispatch, bounded drain, actual EOF and awaited disposal. Credential authentication uses the same scope with no principal and clears a prior principal only inside that scope. Other RequestContext entries are left untouched, and an inherited HttpContext.User or caller-supplied principal is never an input.
At lazy stream start the grain verifies the signed envelope and actor GUID first (GrainIdentityContext.Validate), then requires the context state and principal shape to match before Started. Authenticate is the only null-principal operation and requires absent USER_CLAIMS. The capability boundary repeats the context match and reloads the active persisted principal immediately before execution; an active principal without grants is denied, and a revoked or expired principal is unauthenticated. Identity never appears in progress, results or logs.
Native Orleans owns enumeration cancellation and disposal. Cleanup may release an already-owned producer after authority expires or is revoked, but it never executes another capability, exposes data or acknowledges a write.
The producer is the actual CqrsStream.Create capacity-one channel with native Orleans enumeration at batch size 1. The handler verifies the signed scope, request key and cancellation, writes Started, then awaits the authorized capability with its real token and Orleans synchronization context and returns exactly one terminal Result. There is no eager side effect before enumeration, no background Task.Run worker and no alternate command executor.
Deactivation runs in the outer iterator finally, after child enumeration disposal has cancelled and joined the actual producer; disposal never races an active MoveNext. A timeout wrapper that abandons work is not cleanup evidence. Persisted receipts keep their authority when the requesting connection ends. GrainRoutingOptions RequestLifetime and ExecutionLifetime (default one minute each, at most two minutes) bound the signed request and the linked execution deadline shared by native pulls, validation and the consumer.
Shape: at most two chunks, Started (sequence 1) plus Completed or Failed (sequence 2), or an early Failed (sequence 1). Started has only the matching request GUID. A completed reply has positive payload bytes within MaximumReplyBytes and no Error or SafeDetail; a failed terminal has only the closed Problem above.
Validated GrainRoutingOptions ceilings: Started at most 8192 native bytes; failed terminal at most 65536; successful reply payload at most 16777216; successful terminal at most 16842752; aggregate at most 16850944. Count, sequence and byte arithmetic is checked. The caller retains only the current chunk and one bounded terminal reply. These are serialization admission limits, not measured process RSS.
GrainNativeByteCounter measures with the silo's registered Serializer<CqrsStreamChunk<...>> through a lazy non-retaining counting destination: scratch grows only on request up to MaximumScratchBytes (ceiling 1048576), admission counts actual Advance bytes rather than reservation hints, and cancellation is checked at entry and each buffer boundary. Native silo message-body admission is at least the successful chunk bound plus the existing 65536-byte response-envelope allowance; native transport still enforces its own final bound.
The consumer validates every field and the increasing sequence, records one terminal, then requires actual EOF. It does not repair invalid output, stop at the first terminal or forward unknown Problem fields. A duplicate, trailing, missing or oversized stream fails the operation; disposal is awaited on every exit and cleanup failures stay observable.
At the producer boundary, handled domain/native failures and other nonfatal failures map through GrainReplyFactory before returning Result, keeping safe structured diagnostics without raw exception objects, messages, data, tokens or query bodies. At the consumer boundary, a transport, cancellation or protocol failure before a validated final response is OwnershipLost or Cancelled for reads and UnknownWriteOutcome for writes. The stable command ID is preserved and nothing is re-dispatched automatically. A valid persisted receipt stays authoritative even if its delivery is interrupted.
A reply-encoding failure after a native submit has returned a committed value is tagged with the closed ReplyEncoding stage and classified as UnknownWriteOutcome, including a multi-lane parent's final encoding. Definitive canonical failures (including quota and validation) and read failures keep their exact codes.
Native Create and Normalize never convert OutOfMemoryException, StackOverflowException or AccessViolationException into failed chunks. The owning CqrsRuntimeFailures.FindFatal preserves direct and nested AggregateException fatal identity; KeyLoad does not copy that algorithm. NativeCqrsStreamLifetime joins producer disposal before activation settlement without masking primary or cleanup failures. The public MCP dispatcher and HTTP error middleware never turn a fatal runtime failure into an ordinary Problem or tool result; ordinary failure mapping and caller cancellation are unchanged.
A defect in native producer cancellation or join belongs to the ManagedCode.Communication repository. It is repaired there with complete regressions and a canonical release, and KeyLoad integrates only the published package. No consumer copy, local package or project reference is allowed.
NativeRequestWorkOwner is silo-local. It tracks at most GrainRoutingOptions.MaximumRequestProducers request-producer frames (default 64) and MaximumTotalFrames producer, read and command frames in total (default 128). An entry holds only the request GUID, a closed work kind and its lease identity, never activation context, request content, principal, credentials or open storage. Duplicate live GUID/kind admission and capacity exhaustion fail with ResourceExhausted; admission after shutdown fails with OwnershipLost; an empty GUID or undefined kind fails Validation before admission and consumes no capacity. Limits apply to internal and public callers alike.
A lazy unenumerated stream admits nothing. The first enumeration admits one RequestProducer lease and links the owner token into the execution token. Rejected admission still settles the activation and rethrows its original typed error, with no synthesized Started/Failed chunk. A consumer paused after a chunk must resume or dispose; cancellation alone cannot release its frame. Read work is admitted after the signed read and context checks; command work after route validation. Each lease is released exactly once, after native child disposal and activation settlement or the original capability finally.
Closing the owner is atomic: it cancels its token and joins every admitted frame even if a cancellation callback throws. Every DrainAsync caller observes the same original task, and caller cancellation never replaces it. DisposeAsync joins that drain before disposing its token source, preserving original errors and fatal priority. During native silo shutdown the node closes this admission and joins every local frame before the silo-local ZoneTree Grain Service closes its stores (NativeRequestDrainLifecycle). A finite observation deadline reports failure without releasing an unfinished frame. The stream adds no persisted data or public authorization fields.
Interruption criteria (AC-CRS-003..006) use an optional IGrainRequestPhaseObserver that is absent in ordinary registration; the ordinary path allocates no phase record or registry. It is enabled only by an explicit ephemeral test AppHost setting that fails closed on unknown or inconsistent settings (RequestCqrsProbeAppHostAdmissionTests), and no HTTP, MCP or Orleans fault endpoint exists. Closed phases are RequestStarted, AuthorizationReload (immediately before the persisted principal reload), BeforeSubmit and SubmitReturned, plus a synchronous ProducerDisposed notification after the producer has settled. An arm selects one principal and one command or read kind, claims the first matching request, and either holds it (joined on release, cancellation or host shutdown, with a finite ceiling) or throws one controlled nonfatal error with private canaries. Private control records are at most 8192 bytes (8192 accepted, 8193 rejected), counts and total bytes per node are bounded, and records never enter canonical storage or carry principals, payloads or credentials.
| Criterion | Caller observation and final oracle |
|---|---|
| AC-CRS-003/005 before submit | After the BeforeSubmit marker, a separate authorized read shows the unchanged entity; then the original caller is cancelled. The SDK returns UnknownWriteOutcome; official MCP returns a server-generated Problem or its actual cancellation/transport failure, never a fabricated tool response. A same-command retry with a fresh signed request commits exactly one effect. |
| AC-CRS-005 after submit | After SubmitReturned and an independent read of the committed entity, the caller is interrupted. Retrying the identical command through SDK and official MCP returns the complete committed receipt with exactly one revision advance. |
| AC-CRS-006 current authority | Held at AuthorizationReload, the principal is revoked through a separate administrator command whose persisted revocation is verified. The released request is Unauthenticated with no document, revision, outcome or outbox effect; a fresh authorized command then succeeds and the revoked principal stays rejected. |
| AC-CRS-003/004 privacy and settlement | A controlled ordinary error on an actual request leaves no private message, data or stack canary in public output or logs; producer disposal is observed and a healthy fresh request has no inherited identity. |
After the owned cluster has shut down and every owner lock is released, a test-only RecoveryProcess child opens each node's store through the guarded current-format open and reads only the scoped StoredOutcome and its locator: the held command is absent on every node and a positive control is present on every configured copy. Malformed input or a mismatched store fails closed with no receipt, and the child never emits paths, principals, commands, content or exception text.
Qualification: these criteria close only through passing Aspire replicated-cluster runs on the delivered source with the real .NET SDK and official MCP client; authored fixtures and local runs do not close them.