diff --git a/docs/getting-started/self-hosted.md b/docs/getting-started/self-hosted.md index abcd25667..119f4d6f6 100644 --- a/docs/getting-started/self-hosted.md +++ b/docs/getting-started/self-hosted.md @@ -18,6 +18,8 @@ The Session brings its own model provider; the installation default never applie The installer brings its own pinned Node.js and Harness versions (listed in [`scripts/build-native-installer.mjs`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/scripts/build-native-installer.mjs)) and leaves other installations of those tools untouched. On a platform without a matching installer, the command fails. +The download is about 320 MiB. At its peak the installation needs about 2 GiB free on the disk that holds the Runtime home and the installation directory with one Harness selected (archive 316 MiB, unpacked installer 907 MiB, installed files about 560 MiB), and about 2.2 GiB with all three; the unpacked copy is removed when the command ends and the archive when the installation completes. The finished installation takes about 560 MiB with one Harness and about 940 MiB with all three. + The machine needs: - HTTP or HTTPS access to Core, and to the release download host unless Core carries an offline copy of the installers; @@ -69,7 +71,7 @@ The installer reports three results: | **Daemon connection** | Core confirmed the daemon's authenticated connection | | **Model configuration** | Not checked; the first Turn uses the Session's model provider | -Downloads retry temporary network failures up to three attempts and show progress in a terminal. Disk space is checked before downloading, extracting and copying components. If a download or installation is interrupted, rerun the command: it clears unfinished temporary copies while preserving completed components, credentials and the workspace. Download staging lives in `native-download` under the Runtime home; its small `download.lock` file remains for concurrency control. An active download or installation is never cleared by another run. If the command expires, copy a fresh one from the Session. +The installer checks the grant before it downloads anything, so an expired command costs no download. The archive download resumes after a reset, fails only when it stalls (no data for 60 seconds) and gives up after five attempts; rerun the command to continue where it stopped. A verified archive is kept in `native-download` under the Runtime home until the installation completes, so a rerun after a failure needs no new download; it is checked against Core's checksum on every run and discarded if it fails. Disk space is checked before downloading, extracting and copying components. If a download or installation is interrupted, rerun the command: it clears unfinished temporary copies while preserving completed components, credentials and the workspace. The small `download.lock` file in `native-download` remains for concurrency control. An active download or installation is never cleared by another run. If the command expires, copy a fresh one from the Session. If the connection is not confirmed within 45 seconds, the installer prints the path of the daemon's log. The daemon keeps reconnecting. Fix the cause and run the install command again with the same installation directory, copying a fresh one from Web if it has expired: completed components and the credential are kept and a running daemon is reused. Do not remove the workspace or the Session to retry. diff --git a/docs/zh/getting-started/self-hosted.md b/docs/zh/getting-started/self-hosted.md index 7d2f216ec..767902e00 100644 --- a/docs/zh/getting-started/self-hosted.md +++ b/docs/zh/getting-started/self-hosted.md @@ -1,7 +1,7 @@ --- title: "自托管执行器" source: docs/getting-started/self-hosted.md -source_hash: 8603c51679351eefcaa3977c90af7658e1e86d62be7980baf42f1d72faedf044 +source_hash: 1084921d73014c1c48c6b4c2df5fa356544ed3b046bf0bc90cb9dce85b7188f7 --- `self_hosted` Session 在应用拥有的机器上运行：工作站、虚拟机或你管理的沙箱。应用通过 `/v1` 创建 Session，并获得安装 `oac-daemon`、启动它并连接 Core 的命令。Web 在 Session 页面展示同一命令；Web 是可选的。Core 不创建、停止或回收这台机器。 @@ -20,6 +20,8 @@ Session 自带模型提供商；安装默认模型不适用（[原因](../../../ 安装程序自带固定版本的 Node.js 和 Harness（列于 [`scripts/build-native-installer.mjs`](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/scripts/build-native-installer.mjs)），不修改这些工具的其他安装。在没有匹配安装程序的平台上，命令会失败。 +下载约 320 MiB。安装高峰时，Runtime 主目录和安装目录所在磁盘需要约 2 GiB 可用空间（只选一个 Harness：归档 316 MiB、解包后的安装程序 907 MiB、安装后的文件约 560 MiB），三个 Harness 全选约 2.2 GiB；解包副本在命令结束时删除，归档在安装完成时删除。安装完成后占用约 560 MiB（一个 Harness）至约 940 MiB（全部三个）。 + 机器需要： - 通过 HTTP 或 HTTPS 访问 Core，以及访问发布下载主机；如果 Core 已有安装程序的离线副本，则无需后者； @@ -71,7 +73,7 @@ Session 自带模型提供商；安装默认模型不适用（[原因](../../../ | **Daemon connection** | Core 已确认守护进程的认证连接 | | **Model configuration** | 未检查；第一个 Turn 使用 Session 的模型提供商 | -临时网络故障会重试下载，最多尝试三次，并在终端展示进度。下载、解压和复制组件前会检查磁盘空间。如果下载或安装被中断，重新执行命令：它清理未完成的临时副本，同时保留已完成的组件、凭据和工作区。下载暂存位于 Runtime 主目录的 `native-download` 中；小型 `download.lock` 文件保留用于并发控制。其他运行不会清理仍在进行的下载或安装。命令过期时，从 Session 复制新命令。 +安装程序在下载任何内容之前先检查授权，因此过期的命令不会白下载。归档下载在连接重置后会续传，只在传输停滞（60 秒没有数据）时失败，最多尝试五次；重新执行命令即可从中断处继续。校验通过的归档保存在 Runtime 主目录的 `native-download` 中，直到安装完成，因此失败后重跑无需重新下载；每次运行都会对照 Core 提供的校验和检查它，未通过则丢弃。下载、解压和复制组件前会检查磁盘空间。如果下载或安装被中断，重新执行命令：它清理未完成的临时副本，同时保留已完成的组件、凭据和工作区。`native-download` 中的小型 `download.lock` 文件保留用于并发控制。其他运行不会清理仍在进行的下载或安装。命令过期时，从 Session 复制新命令。 如果 45 秒内未确认连接，安装程序输出守护进程日志路径。守护进程继续重连。修复原因后，以同一安装目录重新执行安装命令；如果命令已过期，从 Web 复制新的。已完成的组件和凭据会保留，已运行的守护进程会复用。不要删除工作区或 Session 来重试。 diff --git a/services/core/internal/nativeinstaller/assets/bootstrap.ps1 b/services/core/internal/nativeinstaller/assets/bootstrap.ps1 index d09ca9464..4d10dff6e 100644 --- a/services/core/internal/nativeinstaller/assets/bootstrap.ps1 +++ b/services/core/internal/nativeinstaller/assets/bootstrap.ps1 @@ -47,13 +47,19 @@ function Show-Bytes([string]$Label, [long]$Bytes, [long]$Total) { $percent = if ($Total -gt 0) { [Math]::Min(100, [int](100.0 * $Bytes / $Total)) } else { -1 } Write-Progress -Activity $Label -Status ("{0:N1} MiB" -f ($Bytes / 1MB)) -PercentComplete $percent } +# Resume after resets and fail only when a transfer stalls, with no total time cap. A partial file is kept in the download directory for the next run. function Get-NativeFile([string]$Uri, [string]$Destination, [long]$Limit) { - for ($attempt=1; $attempt -le 3; $attempt++) { - $response = $null; $inputStream = $null; $outputStream = $null; $retryable = $true + for ($attempt=1; $attempt -le 5; $attempt++) { + $request = $null; $response = $null; $inputStream = $null; $outputStream = $null; $retryable = $true try { + [long]$offset = 0 + if (Test-Path -LiteralPath $Destination) { $offset = (Get-Item -LiteralPath $Destination).Length } $address = [Uri]$Uri for ($redirect=0; $redirect -le 5; $redirect++) { - $response = $client.GetAsync($address, [Net.Http.HttpCompletionOption]::ResponseHeadersRead).GetAwaiter().GetResult() + $request = [Net.Http.HttpRequestMessage]::new([Net.Http.HttpMethod]::Get, $address) + if ($offset -gt 0) { $request.Headers.Range = [Net.Http.Headers.RangeHeaderValue]::new($offset, $null) } + $response = $client.SendAsync($request, [Net.Http.HttpCompletionOption]::ResponseHeadersRead).GetAwaiter().GetResult() + $request.Dispose(); $request = $null $status = [int]$response.StatusCode if ($status -notin @(301,302,303,307,308)) { break } if ($redirect -eq 5 -or !$response.Headers.Location) { $retryable=$false; throw 'Installer download has too many or invalid redirects.' } @@ -61,17 +67,21 @@ function Get-NativeFile([string]$Uri, [string]$Destination, [long]$Limit) { if ($next.Scheme -ne 'https' -or $next.UserInfo) { $retryable=$false; throw 'Installer download redirect must use HTTPS without credentials.' } $response.Dispose(); $response=$null; $address=$next } - if ($status -ne 200) { + if ($status -eq 416) { Remove-Item -LiteralPath $Destination -Force; throw 'The server cannot resume the earlier download; starting it again.' } + if ($status -notin @(200,206)) { $retryable = $status -in @(408,429,500,502,503,504) if ($status -eq 404) { throw 'This Core has no qualified installer for this platform.' } throw "Installer download failed (HTTP $status). Check Core and the download host." } + # A 200 answer ignores the Range header, so start the file again. + $mode = [IO.FileMode]::Append + if ($status -eq 200) { $mode = [IO.FileMode]::Create; $offset = 0 } $total = $response.Content.Headers.ContentLength if ($null -eq $total) { $total = 0 } $retryable=$false - if ($total -gt $Limit) { throw 'Installer download exceeds the available space or metadata size limit.' } + if ($offset + $total -gt $Limit) { throw 'Installer download exceeds the available space or metadata size limit.' } Assert-Space $total - $outputStream = [IO.File]::Open($Destination, [IO.FileMode]::Create, [IO.FileAccess]::Write, [IO.FileShare]::None) + $outputStream = [IO.File]::Open($Destination, $mode, [IO.FileAccess]::Write, [IO.FileShare]::None) $retryable=$true $inputStream = $response.Content.ReadAsStreamAsync().GetAwaiter().GetResult() $buffer = New-Object byte[] 65536 @@ -79,7 +89,6 @@ function Get-NativeFile([string]$Uri, [string]$Destination, [long]$Limit) { $clock = [Diagnostics.Stopwatch]::StartNew() [long]$lastProgress=0 while ($true) { - if ($clock.Elapsed.TotalSeconds -gt 1200) { throw 'Installer download timed out; retry with a fresh command if it expired.' } $cancel = [Threading.CancellationTokenSource]::new() try { $cancel.CancelAfter(60000) @@ -90,10 +99,10 @@ function Get-NativeFile([string]$Uri, [string]$Destination, [long]$Limit) { if ($read -eq 0) { break } $received += $read $retryable=$false - if ($received -gt $Limit) { throw 'Installer download exceeds the available space or metadata size limit.' } + if ($offset + $received -gt $Limit) { throw 'Installer download exceeds the available space or metadata size limit.' } $outputStream.Write($buffer,0,$read) $retryable=$true - if ($clock.ElapsedMilliseconds - $lastProgress -ge 200) { Show-Bytes 'Downloading installer' $received $total; $lastProgress=$clock.ElapsedMilliseconds } + if ($clock.ElapsedMilliseconds - $lastProgress -ge 200) { Show-Bytes 'Downloading installer' ($offset + $received) ($offset + $total); $lastProgress=$clock.ElapsedMilliseconds } } if ($total -gt 0 -and $received -ne $total) { throw 'Installer download was interrupted.' } $retryable=$false @@ -101,17 +110,31 @@ function Get-NativeFile([string]$Uri, [string]$Destination, [long]$Limit) { return } catch { if (!$retryable) { throw } - if ($attempt -eq 3) { throw 'Installer download failed after three attempts. Check network, proxy and certificates, then retry.' } - Write-Host "Download interrupted; retrying ($attempt/2)..." + if ($attempt -eq 5) { throw 'Installer download failed after five attempts. Check the network to Core and GitHub and any proxy settings, then rerun the command; downloaded data is kept and resumed.' } + Write-Host "Download interrupted; retrying ($attempt/4)..." Start-Sleep -Seconds $attempt } finally { if ($inputStream) { $inputStream.Dispose() } if ($outputStream) { $outputStream.Dispose() } + if ($request) { $request.Dispose() } if ($response) { $response.Dispose() } if ($terminal) { Write-Progress -Activity 'Downloading installer' -Completed } } } } +function Get-ArchiveHash([string]$Path) { (Get-FileHash -Algorithm SHA256 -LiteralPath $Path).Hash.ToLowerInvariant() } +# Reject a rejected grant before any large download. Only a 401 stops here; any other answer is reported by the installer. +function Test-Grant { + if (!$env:OAC_INSTALL_GRANT) { return } + $rejected = $false + try { + $message = [Net.Http.HttpRequestMessage]::new([Net.Http.HttpMethod]::Post, ($Base -replace '/install/[^/]+$', '/installation')) + $message.Headers.Authorization = [Net.Http.Headers.AuthenticationHeaderValue]::new('Bearer', $env:OAC_INSTALL_GRANT) + $answer = $client.SendAsync($message).GetAwaiter().GetResult() + $rejected = [int]$answer.StatusCode -eq 401 + } catch { } finally { if ($message) { $message.Dispose() }; if ($answer) { $answer.Dispose() } } + if ($rejected) { throw 'Installation authorization rejected or expired; copy a fresh command from the Session.' } +} try { Assert-OrdinaryPath $cache [IO.Directory]::CreateDirectory($cache) | Out-Null @@ -136,17 +159,29 @@ try { $handler.AllowAutoRedirect = $false $client = [Net.Http.HttpClient]::new($handler) $client.Timeout = [TimeSpan]::FromSeconds(15) + Test-Grant Assert-Space 0 Write-Host 'Downloading the installer matched to Core...' $checksum = Join-Path $work 'checksum' Get-NativeFile "$Base/windows-$architecture.sha256" $checksum 1024 $expected = [IO.File]::ReadAllText($checksum).Trim() if ($expected -cnotmatch '^[0-9a-f]{64}$') { throw 'Core returned an invalid installer checksum.' } - $archive = Join-Path $work 'bundle.tar.gz' - $available = [OacNativeDownloadSpace]::Available($cache) - Get-NativeFile "$Base/windows-$architecture.tar.gz" $archive ($available-64MB) - Write-Host 'Verifying the installer archive...' - if ((Get-FileHash -Algorithm SHA256 $archive).Hash.ToLowerInvariant() -ne $expected) { throw 'Installer checksum mismatch; download again.' } + # The verified archive outlives this run so a later failure, such as a wrong option, needs no new download; it is removed once the installation completes. + $archive = Join-Path $cache "$expected.tar.gz" + Get-ChildItem -LiteralPath $cache -Filter '*.tar.gz*' -File | Where-Object { $_.Name -ne "$expected.tar.gz" -and $_.Name -ne "$expected.tar.gz.part" } | Remove-Item -Force + if (Test-Path -LiteralPath $archive) { + Write-Host 'Verifying the installer archive from an earlier run...' + if ((Get-ArchiveHash $archive) -ne $expected) { Remove-Item -LiteralPath $archive -Force; Write-Host 'The earlier archive failed verification and was discarded.' } + } + if (!(Test-Path -LiteralPath $archive)) { + $part = "$archive.part" + if ((Test-Path -LiteralPath $part) -and (Get-Item -LiteralPath $part).Length -gt 0) { Write-Host 'Resuming the earlier download...' } + $available = [OacNativeDownloadSpace]::Available($cache) + Get-NativeFile "$Base/windows-$architecture.tar.gz" $part ($available-64MB) + Write-Host 'Verifying the installer archive...' + if ((Get-ArchiveHash $part) -ne $expected) { Remove-Item -LiteralPath $part -Force; throw 'Installer checksum mismatch; the download was discarded. Rerun the command to download it again.' } + Move-Item -LiteralPath $part -Destination $archive + } Write-Host 'Checking extraction space...' $file = [IO.File]::OpenRead($archive) $gzip = [IO.Compression.GZipStream]::new($file,[IO.Compression.CompressionMode]::Decompress) @@ -165,6 +200,7 @@ try { Write-Host 'Starting installation...' & (Join-Path $bundle 'oac-daemon.exe') install --onboard-url $endpoint @InstallArguments if ($LASTEXITCODE -ne 0) { throw 'Installation or connection failed; follow the installer guidance and retry.' } + Remove-Item -LiteralPath $archive -Force } finally { if ($client) { $client.Dispose() } if ($lock) { diff --git a/services/core/internal/nativeinstaller/assets/bootstrap.sh b/services/core/internal/nativeinstaller/assets/bootstrap.sh index 311782863..d4ed9a10e 100644 --- a/services/core/internal/nativeinstaller/assets/bootstrap.sh +++ b/services/core/internal/nativeinstaller/assets/bootstrap.sh @@ -51,46 +51,75 @@ space() { } progress=(-sS) if [ -t 2 ]; then progress=(--progress-bar --show-error); fi +# Resume after resets and fail only when a transfer stalls, with no total time cap. A partial file is kept in the download directory for the next run. fetch() { local url=$1 destination=$2 limit=$3 status code attempt - for attempt in 1 2 3; do + for attempt in 1 2 3 4 5; do code=0 - run curl "${progress[@]}" --fail --location --proto-redir =https --connect-timeout 15 --max-time 1200 --speed-time 60 --speed-limit 1024 --max-filesize "$limit" --write-out '%{http_code}' "$url" -o "$destination" >"$work/http-status" || code=$? + run curl "${progress[@]}" --fail --location --proto-redir =https --connect-timeout 15 --speed-time 60 --speed-limit 1024 --continue-at - --max-filesize "$limit" --write-out '%{http_code}' "$url" -o "$destination" >"$work/http-status" || code=$? status=$(cat "$work/http-status") [ "$code" -ne 0 ] || return 0 case "$status" in 404) fail 'This Core has no qualified installer for this platform.';; esac case "$code" in 23) fail 'Cannot write the installer download. Check disk space, quota and directory permissions.';; 63) fail 'Installer download exceeds the available space or metadata size limit.';; + 33) rm -f "$destination";; # The server cannot resume; start the file again. esac case "$code:$status" in - 6:*|7:*|18:*|28:*|52:*|55:*|56:*|22:408|22:429|22:500|22:502|22:503|22:504) - if [ "$attempt" -lt 3 ]; then printf 'Download interrupted; retrying (%s/2)...\n' "$attempt" >&2; run sleep "$attempt"; continue; fi;; + 6:*|7:*|16:*|18:*|28:*|33:*|35:*|52:*|55:*|56:*|92:*|22:408|22:429|22:500|22:502|22:503|22:504) + if [ "$attempt" -lt 5 ]; then printf 'Download interrupted; retrying (%s/4)...\n' "$attempt" >&2; run sleep "$attempt"; continue; fi;; esac - fail "Installer download failed (HTTP ${status:-unknown}, curl $code). Check network, proxy and certificates, then retry." + fail "Installer download failed (HTTP ${status:-unknown}, curl $code). Check the network to Core and GitHub (set HTTPS_PROXY if a proxy is required), then rerun the command; downloaded data is kept and resumed." done } -printf 'Downloading the installer matched to Core...\n' +# Reject a rejected grant before any large download. Only a 401 stops here; any other answer is reported by the installer. +check_grant() { + [ -n "${OAC_INSTALL_GRANT:-}" ] || return 0 + local status + printf 'header = "Authorization: Bearer %s"\n' "$OAC_INSTALL_GRANT" | run curl -sS --config - --request POST --connect-timeout 15 --max-time 30 --write-out '%{http_code}' --output /dev/null "${base%/install/*}/installation" >"$work/http-status" 2>/dev/null || : + status=$(cat "$work/http-status") + [ "$status" != 401 ] || fail 'Installation authorization rejected or expired; copy a fresh command from the Session.' +} +verified() { + run "${hash[@]}" "$1" >"$work/digest" || return 1 + local actual rest + read -r actual rest <"$work/digest" + [ "$actual" = "$expected" ] +} space 0 +check_grant +printf 'Downloading the installer matched to Core...\n' fetch "$base/$os-$arch.sha256" "$work/checksum" 1024 expected=$(cat "$work/checksum") [[ "$expected" =~ ^[0-9a-f]{64}$ ]] || fail 'Core returned an invalid installer checksum.' -available=$(df -Pk "$work" | awk 'NR==2 {print $4}') -fetch "$base/$os-$arch.tar.gz" "$work/bundle.tar.gz" "$((available * 1024 - 67108864))" -printf 'Verifying the installer archive...\n' -run "${hash[@]}" "$work/bundle.tar.gz" >"$work/digest" -read -r actual rest <"$work/digest" -[ "$actual" = "$expected" ] || fail 'Installer checksum mismatch; download again.' +# The verified archive outlives this run so a later failure, such as a wrong option, needs no new download; it is removed once the installation completes. +archive=$cache/$expected.tar.gz +for stale in "$cache"/*.tar.gz "$cache"/*.tar.gz.part; do + case "$stale" in "$archive"|"$archive.part") ;; *) rm -f "$stale";; esac +done +if [ -f "$archive" ]; then + printf 'Verifying the installer archive from an earlier run...\n' + verified "$archive" || { rm -f "$archive"; printf 'The earlier archive failed verification and was discarded.\n' >&2; } +fi +if [ ! -f "$archive" ]; then + [ ! -s "$archive.part" ] || printf 'Resuming the earlier download...\n' + available=$(df -Pk "$work" | awk 'NR==2 {print $4}') + fetch "$base/$os-$arch.tar.gz" "$archive.part" "$((available * 1024 - 67108864))" + printf 'Verifying the installer archive...\n' + verified "$archive.part" || { rm -f "$archive.part"; fail 'Installer checksum mismatch; the download was discarded. Rerun the command to download it again.'; } + mv "$archive.part" "$archive" +fi printf 'Checking extraction space...\n' # Count the verified tar stream without storing another copy, including its headers and padding. -run bash -o pipefail -c 'gzip -dc "$1" | wc -c' -- "$work/bundle.tar.gz" >"$work/unpacked-size" || fail 'Installer archive is incomplete or invalid.' +run bash -o pipefail -c 'gzip -dc "$1" | wc -c' -- "$archive" >"$work/unpacked-size" || { rm -f "$archive"; fail 'Installer archive is incomplete or invalid; it was discarded. Rerun the command to download it again.'; } unpacked=$(cat "$work/unpacked-size") space "$unpacked" printf 'Extracting the installer...\n' mkdir "$work/bundle" -run tar -xzf "$work/bundle.tar.gz" -C "$work/bundle" || fail 'Installer extraction failed. Check disk space, quota and filesystem permissions.' +run tar -xzf "$archive" -C "$work/bundle" || fail 'Installer extraction failed. Check disk space, quota and filesystem permissions.' [ -x "$work/bundle/oac-daemon" ] || fail 'Cannot start the native installer. Check executable permissions and use a Runtime home that permits execution.' if run "$work/bundle/oac-daemon" install --onboard-url "${base%/install/*}/installation" "$@"; then + rm -f "$archive" exit 0 else code=$? diff --git a/services/core/internal/nativeinstaller/bootstrap_interrupt_unix_test.go b/services/core/internal/nativeinstaller/bootstrap_interrupt_unix_test.go index 6455c6234..f66121445 100644 --- a/services/core/internal/nativeinstaller/bootstrap_interrupt_unix_test.go +++ b/services/core/internal/nativeinstaller/bootstrap_interrupt_unix_test.go @@ -24,6 +24,10 @@ func TestBootstrapParentDeathKeepsActiveDownloadLocked(t *testing.T) { release := make(chan struct{}) var downloads atomic.Int32 server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.HasSuffix(r.URL.Path, "/installation") { + w.WriteHeader(404) + return + } if strings.HasSuffix(r.URL.Path, ".sha256") { fmt.Fprintf(w, "%x\n", sha256.Sum256(archive)) return @@ -36,7 +40,7 @@ func TestBootstrapParentDeathKeepsActiveDownloadLocked(t *testing.T) { <-release return } - w.Write(archive) + http.ServeContent(w, r, "", time.Time{}, bytes.NewReader(archive)) })) defer server.Close() defer close(release) @@ -63,8 +67,8 @@ func TestBootstrapParentDeathKeepsActiveDownloadLocked(t *testing.T) { if err == nil || !bytes.Contains(retry, []byte("Another native download")) { t.Fatalf("active download was not protected: %v %s", err, retry) } - if _, err := os.Stat(filepath.Join(home, "native-download", "staging", "bundle.tar.gz")); err != nil { - t.Fatal("active staging removed") + if _, err := os.Stat(filepath.Join(home, "native-download", fmt.Sprintf("%x.tar.gz.part", sha256.Sum256(archive)))); err != nil { + t.Fatal("active download removed") } syscall.Kill(-command.Process.Pid, syscall.SIGKILL) _ = command.Wait() diff --git a/services/core/internal/nativeinstaller/bootstrap_recovery_test.go b/services/core/internal/nativeinstaller/bootstrap_recovery_test.go index d9e1a7165..4e2bc23c6 100644 --- a/services/core/internal/nativeinstaller/bootstrap_recovery_test.go +++ b/services/core/internal/nativeinstaller/bootstrap_recovery_test.go @@ -15,14 +15,19 @@ import ( "strings" "sync/atomic" "testing" + "time" ) func bootstrapFixtureArchive(t *testing.T) []byte { + return bootstrapScriptArchive(t, "#!/bin/sh\nexit 0\n") +} + +func bootstrapScriptArchive(t *testing.T, script string) []byte { t.Helper() var buffer bytes.Buffer z := gzip.NewWriter(&buffer) tarball := tar.NewWriter(z) - data := []byte("#!/bin/sh\nexit 0\n") + data := []byte(script) if err := tarball.WriteHeader(&tar.Header{Name: "oac-daemon", Mode: 0700, Size: int64(len(data))}); err != nil { t.Fatal(err) } @@ -61,6 +66,10 @@ func TestBootstrapRecovery(t *testing.T) { sum = strings.Repeat("0", 64) } server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.HasSuffix(r.URL.Path, "/installation") { + w.WriteHeader(404) + return + } if strings.HasSuffix(r.URL.Path, ".sha256") { request := metadata.Add(1) if scenario == "missing" { @@ -80,7 +89,7 @@ func TestBootstrapRecovery(t *testing.T) { w.Write(archive[:len(archive)/2]) return } - w.Write(archive) + http.ServeContent(w, r, "", time.Time{}, bytes.NewReader(archive)) })) defer server.Close() home := filepath.Join(t.TempDir(), "Runtime home with spaces") diff --git a/services/core/internal/nativeinstaller/bootstrap_resume_unix_test.go b/services/core/internal/nativeinstaller/bootstrap_resume_unix_test.go new file mode 100644 index 000000000..e5deefd7c --- /dev/null +++ b/services/core/internal/nativeinstaller/bootstrap_resume_unix_test.go @@ -0,0 +1,145 @@ +//go:build unix + +package nativeinstaller + +import ( + "bytes" + "crypto/sha256" + "fmt" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "strings" + "sync/atomic" + "testing" + "time" +) + +// resumeServer serves the installation endpoint and a ranged archive. +type resumeServer struct { + *httptest.Server + archive []byte + downloads, checks atomic.Int32 + ranges chan string + grantStatus int + stallFirstDownload bool +} + +func newResumeServer(t *testing.T, archive []byte) *resumeServer { + t.Helper() + s := &resumeServer{archive: archive, ranges: make(chan string, 8), grantStatus: 200} + s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case strings.HasSuffix(r.URL.Path, "/installation"): + s.checks.Add(1) + if r.Header.Get("Authorization") != "Bearer fixture-grant" { + t.Error("grant missing from the installation check") + } + w.WriteHeader(s.grantStatus) + fmt.Fprint(w, `{"harness":"codex"}`) + case strings.HasSuffix(r.URL.Path, ".sha256"): + fmt.Fprintf(w, "%s\n", sha256Hex(archive)) + default: + request := s.downloads.Add(1) + s.ranges <- r.Header.Get("Range") + if s.stallFirstDownload && request == 1 { + w.Header().Set("Content-Length", fmt.Sprint(len(archive))) + w.Write(archive[:len(archive)/2]) + w.(http.Flusher).Flush() + <-r.Context().Done() + return + } + http.ServeContent(w, r, "", time.Time{}, bytes.NewReader(archive)) + } + })) + t.Cleanup(s.Close) + return s +} + +func (s *resumeServer) command(t *testing.T, home string, args ...string) *exec.Cmd { + command := exec.Command("bash", append([]string{"assets/bootstrap.sh", s.URL + "/api/v1/agent-daemon/install/test"}, args...)...) + command.Env = append(os.Environ(), "OAC_RUNTIME_HOME="+home, GrantEnv+"=fixture-grant", "NO_PROXY=127.0.0.1", "no_proxy=127.0.0.1") + return command +} + +func TestBootstrapResumesStalledDownload(t *testing.T) { + curl, err := exec.LookPath("curl") + if err != nil { + t.Skip("curl unavailable") + } + archive := bootstrapFixtureArchive(t) + server := newResumeServer(t, archive) + server.stallFirstDownload = true + // Shorten only the stall window; the script's own retry and resume logic runs. + bin := t.TempDir() + if err = os.WriteFile(filepath.Join(bin, "curl"), []byte("#!/bin/sh\nexec "+shellQuote(curl)+" \"$@\" --speed-time 2\n"), 0700); err != nil { + t.Fatal(err) + } + command := server.command(t, t.TempDir()) + command.Env = append(command.Env, "PATH="+bin+string(os.PathListSeparator)+os.Getenv("PATH")) + if output, err := command.CombinedOutput(); err != nil { + t.Fatalf("%v: %s", err, output) + } + if first, second := <-server.ranges, <-server.ranges; first != "" || !strings.HasPrefix(second, "bytes=") { + t.Fatalf("second request did not resume: %q then %q", first, second) + } +} + +func TestBootstrapKeepsVerifiedArchiveUntilInstallCompletes(t *testing.T) { + // The first install attempt fails after the download; the rerun needs no new archive and removes it on success. + archive := bootstrapScriptArchive(t, "#!/bin/sh\n[ -e \"$OAC_RESUME_OK\" ]\n") + server := newResumeServer(t, archive) + home, ok := t.TempDir(), filepath.Join(t.TempDir(), "ok") + cached := filepath.Join(home, "native-download", sha256Hex(archive)+".tar.gz") + command := server.command(t, home) + command.Env = append(command.Env, "OAC_RESUME_OK="+ok) + if output, err := command.CombinedOutput(); err == nil { + t.Fatalf("expected the fixture install to fail: %s", output) + } + if _, err := os.Stat(cached); err != nil { + t.Fatal("verified archive was not kept") + } + if err := os.WriteFile(ok, nil, 0600); err != nil { + t.Fatal(err) + } + command = server.command(t, home) + command.Env = append(command.Env, "OAC_RESUME_OK="+ok) + if output, err := command.CombinedOutput(); err != nil { + t.Fatalf("%v: %s", err, output) + } + if server.downloads.Load() != 1 { + t.Fatalf("archive downloaded %d times", server.downloads.Load()) + } + if _, err := os.Stat(cached); !os.IsNotExist(err) { + t.Fatal("archive kept after a completed installation") + } +} + +func TestBootstrapDiscardsCachedArchiveThatFailsVerification(t *testing.T) { + archive := bootstrapFixtureArchive(t) + server := newResumeServer(t, archive) + home := t.TempDir() + cached := filepath.Join(home, "native-download", sha256Hex(archive)+".tar.gz") + if err := os.MkdirAll(filepath.Dir(cached), 0700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(cached, []byte("corrupt"), 0600); err != nil { + t.Fatal(err) + } + if output, err := server.command(t, home).CombinedOutput(); err != nil || server.downloads.Load() != 1 { + t.Fatalf("%v downloads=%d: %s", err, server.downloads.Load(), output) + } +} + +func TestBootstrapChecksGrantBeforeDownloading(t *testing.T) { + server := newResumeServer(t, bootstrapFixtureArchive(t)) + server.grantStatus = 401 + output, err := server.command(t, t.TempDir()).CombinedOutput() + if err == nil || !strings.Contains(string(output), "authorization rejected or expired") || server.downloads.Load() != 0 || strings.Contains(string(output), "fixture-grant") { + t.Fatalf("%v downloads=%d: %s", err, server.downloads.Load(), output) + } +} + +func sha256Hex(data []byte) string { return fmt.Sprintf("%x", sha256.Sum256(data)) } diff --git a/services/core/internal/nativeinstaller/bootstrap_test.go b/services/core/internal/nativeinstaller/bootstrap_test.go index eb61bf6ad..4eda8217f 100644 --- a/services/core/internal/nativeinstaller/bootstrap_test.go +++ b/services/core/internal/nativeinstaller/bootstrap_test.go @@ -67,6 +67,10 @@ func TestBootstrapDownloadsVerifiedPlatformAcrossHTTPSRedirect(t *testing.T) { for _, valid := range []bool{true, false} { t.Run(fmt.Sprint(valid), func(t *testing.T) { core := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.HasSuffix(r.URL.Path, "/installation") { + w.WriteHeader(404) + return + } if strings.HasSuffix(r.URL.Path, ".sha256") { checksum := hex.EncodeToString(hash[:]) if !valid {