Skip to content

CH-300 feat: Upgrade Argo to 3.3.10 (to allow self-report progress) - #886

Closed
afonsobspinto wants to merge 4 commits into
developfrom
feature/CH-300
Closed

afonsobspinto wants to merge 4 commits into
developfrom
feature/CH-300

Conversation

@afonsobspinto

@afonsobspinto afonsobspinto commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Closes https://metacell.atlassian.net/browse/CH-300

Upgrade Argo Workflows from 3.1.15 to 3.3.10, retaining PNS and the existing CloudHarness SDK/API contract. Include a small controller-only upstream backport so self-reported progress reaches Workflow status while long tasks are still running.

Changes

  • Update server, executor and the seven official CRDs to 3.3.10.
  • Build the official 3.3.10 controller at pinned commit b19870d737a14b21d86f6267642a63dd14e5acd5 with the TaskResult create/update event handlers from upstream Argo 3.5.8. The controller reports v3.3.10-ch.1; no newer controller behavior or executor is adopted.
  • Use CloudHarness's ordinary Argo application-image build and reuse the existing controller Deployment template. No extra application Deployment, listener or progress publisher is added. Registry pull Secrets and explicit controller image overrides are preserved.
  • Grant the executor Role native TaskResult create/patch permissions. The controller retains TaskResult list/watch/cleanup access.
  • Retain PNS and default pod naming v1 for CloudHarness logs. Existing extraEnv entries, including explicit pod-name overrides, remain intact.
  • Remove the unsupported pod-worker argument and keep chart/build regression checks.

Miscellaneous changes already included: upgrade the scanner-runner Ubuntu version and fix the pre-existing syntax/formatting failures required for CI.

Why the controller backport is needed

Stock 3.3.10's executor publishes progress to WorkflowTaskResults, but its controller does not enqueue the Workflow when those records change. Long tasks can therefore show stale Workflow progress until another event or the roughly twenty-minute reconciliation. Short tests that finish their pod can hide the problem.

Relying on legacy pod-annotation fallback is insufficient: CloudHarness's workflow account already has broader permissions that allow TaskResult writes. The backport fixes the native path through Argo's existing rate-limited work queue. NeuroGlass and other consumers can keep reading normal Workflow status.

Self-reporting progress

Tasks overwrite ARGO_PROGRESS_FILE with completed/total, for example:

echo "0/4" > "$ARGO_PROGRESS_FILE"
# Process the first batch.
echo "1/4" > "$ARGO_PROGRESS_FILE"
# Continue after each batch, ending with "4/4".

By default the executor reads the file every 3 seconds, publishes changed counters approximately every minute, and the patched controller queues Workflow reconciliation (default delay 10 seconds). Consumers use the existing CloudHarness workflow API. API load and application caching can add delay; this is not a maximum-latency guarantee.

Build/deploy Argo through the normal CloudHarness pipeline so the controller uses the built image. An explicit override to the stock upstream controller removes the backport. No files under libraries/ change.

Validation

  • The real informer/watch regression fails against unpatched 3.3.10 and passes with the backport.
  • Helm lint, chart regression tests and the combined CloudHarness/MNP chart generation pass, including image overrides and registry credentials.
  • A running task using the default intervals propagated a new TaskResult into Workflow status in 9.8 seconds after publication, without restarting its pod.
  • The active skeleton ingestion propagated its next real update in 10.2 seconds after publication. Its pod UID, container IDs, start times and restart counts were unchanged.
  • A finite workflow exposed preparation, byte download, import, maintenance, cache and verification ratios while running, then completed its file output parameter and onExit handler successfully. That separate stage fixture used 5-second publication; the latency measurements above used the default one-minute interval.
  • NeuroGlass's 18 existing progress-reader tests pass unchanged.

@afonsobspinto afonsobspinto changed the title CH-300 feat: Upgrade Argo to 3.7.18 CH-300 feat: Upgrade Argo to 3.3.10 with PNS Oct 23, 2026
@afonsobspinto afonsobspinto changed the title CH-300 feat: Upgrade Argo to 3.3.10 with PNS CH-300 feat: Upgrade Argo to 3.3.10 (to allow self-report progress) Oct 23, 2026
@afonsobspinto

Copy link
Copy Markdown
Member Author

Dropped for #887

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant