Skip to content

PC-5206: wait out Integration Service, never re-send timed-out activity - #1963

Draft
al3xanndru wants to merge 1 commit into
mainfrom
wabbit-dri/PC-5206_1
Draft

al3xanndru wants to merge 1 commit into
mainfrom
wabbit-dri/PC-5206_1

Conversation

@al3xanndru

@al3xanndru al3xanndru commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Problem

PC-5206 (Bug, Minor): an agent's Integration Service tool run fails although Integration Service still answers.

ConnectionsService.invoke_activity / invoke_activity_async (the only caller is the agent IS tool, uipath_langchain/agent/tools/integration_tool.py) use the SDK's default httpx timeout of 30 s (common/_http_config.py). They also use BaseService.request's retry, which re-sends on any TimeoutException, up to MAX_RETRY_ATTEMPTS = 5 (common/retry.py::is_retryable_platform_exception). Integration Service waits up to about 120 s on the vendor behind an activity. When the data source is slow:

  • the client hangs up at 30 s and POSTs the same non-idempotent activity again, up to 5 times;
  • IS runs every abandoned copy, so a saturated source gets up to 5 duplicate heavy requests;
  • after the fifth timeout the run faults with "did not complete in time… usually a transient issue", although IS answered every time.

Evidence (DRI archive, cause is-tool-30s-client-timeout-retries-post-while-is-working):

Alert Env Tool What the client saw What IS did
SRE-679256, job a1a0c713-a321-4339-af24-1c54ffa62bc8 stg Run KQL Query 5 × httpx.ReadTimeout at 30.0 s 200 after 55–85 s on all 5 attempts (time spent in the customer's ADX)
SRE-684905, job f49af9f4-… stg Run KQL Query timeouts 400 at ~120 s (Kusto timeout) on 9 of 10 attempts; the retries multiplied the load
SRE-684735, job 88f2a28d-1d69-430d-9850-72bb14cf7b01 prd Web Reader 5 × ReadTimeout at 30 s 200 after 90.9–91.5 s on all 5 attempts

Root cause

The client's per-attempt budget (30 s) is shorter than the service's (~120 s). The retry policy also treats a read timeout, where the request was delivered and is still being processed, like a connect failure, where nothing was sent, and re-sends a non-idempotent POST.

Change

packages/uipath-platform/src/uipath/platform/connections/_connections_service.py only:

  • Timeout. Activity invocations pass timeout=Timeout(30.0, read=150.0). This follows the per-request override pattern already used by PiiDetectionService. The read timeout outlasts IS's ~120 s vendor budget, so the client gets IS's real answer: the 200, or IS's own 4xx/5xx. Connect, write and pool timeouts stay at 30 s.
  • Retry. Activity invocations use BaseService.request(_async).retry_with(retry=…), which keeps BaseService's wait and stop strategies. The only change is a predicate that returns False for httpx.ReadTimeout and defers to is_retryable_platform_exception for everything else. Connect errors, ConnectTimeout, PoolTimeout, WriteTimeout and the retryable statuses (408/429/502/503/504/524) are retried exactly as before.
  • The worst-case wall time per tool call is about the same as before: 1 × 150 s, against 5 × 30 s plus backoff. A slow vendor now gets one request instead of five.
  • uipath-platform 0.2.41 → 0.2.42 (pyproject and both uv.lock entries, as the previous platform fixes did).

BaseService, common/retry.py and the global 30 s default are unchanged, so no other service's behaviour changes.

Tests

New in tests/services/test_connections_service.py::TestConnectorActivityInvocation:

Test Checks
test_invoke_activity_waits_longer_than_integration_service The activity request carries read 150 s and connect 30 s; the connection lookup keeps 30 s.
test_invoke_activity_read_timeout_not_retried (sync) A ReadTimeout is raised after exactly one POST.
test_invoke_activity_async_read_timeout_not_retried (async) The same, plus the 150 s read timeout.
test_invoke_activity_async_connect_error_still_retried A ConnectTimeout is still retried, and the second attempt succeeds.
test_invoke_activity_retryable_status_still_retried A 429 is still retried.

Results:

  • Without the source change: the three timeout tests fail, and the two "still retried" tests pass.
  • pytest tests (uipath-platform): 1747 passed, 11 skipped (credential-bound integration tests).
  • Checks: ruff check ., ruff format --check . and mypy src tests (233 files) are clean.

The sandbox could not reach PyPI, so these ran on Python 3.12 from an existing venv (ruff 0.15.12), with this checkout's src first on the path, not with uv sync. CI should re-run them on 3.11.

Risks / open questions

  • A ReadTimeout now fails on the first attempt. A transient network stall after the request was sent is no longer retried. This is deliberate for non-idempotent activities, but it also applies to GET activities. With a 150 s budget that is longer than IS's own, a retry would rarely help.
  • 150 s is derived from telemetry: IS's daily max for the vendor call is 120.1–120.5 s, Semantic Proxy cancels at ~100 s, and Web Reader answered at 91 s. If IS has a documented activity timeout, align to it.
  • Retryable statuses still retry. A 502 or 504 from IS after the vendor already ran (for example the Semantic Proxy 502 at ~100 s) is still retried, as before. Whether activity POSTs should retry 5xx at all is a wider policy question for the SDK owners.
  • retry_with and # type: ignore[attr-defined]. The change relies on tenacity's retry_with on the decorated BaseService.request(_async) (tenacity ≥ 9, as pinned). tenacity's stubs type the decorated function as the original callable, hence the two ignores.
  • Out of scope, uipath-langchain-python (not ours). integration_tool.py still turns a timeout into a fatal System AgentRuntimeError that says "usually a transient issue; please retry". With this change it fires only when IS takes more than 150 s. Rewording it, or returning it to the LLM as a tool error, is for that repo's maintainers.

Drafted by the wabbit-dri draft-fixes loop for PC-5206 (draft #1), for a human to review: not ready to merge as is.

🤖 Generated with Claude Code

…ivities

ConnectionsService.invoke_activity(_async) used the SDK's default 30 s
httpx timeout and BaseService's retry, which re-sends a request on any
TimeoutException up to 5 times. Integration Service waits up to ~120 s on
the vendor behind an activity, so a slow data source made the client hang
up and POST the same non-idempotent activity 5 times while IS ran every
copy and answered 200.

Activity calls now get a 150 s read timeout (connect/write/pool stay 30 s)
and do not retry a ReadTimeout. Connect failures, pool and write timeouts
and retryable statuses are still retried as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-integrations labels Oct 29, 2026
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-integrations test:uipath-langchain Triggers tests in the uipath-langchain-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant