Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions app/en/operate/deploy/helm/_meta.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import type { MetaRecord } from "nextra";

export const meta: MetaRecord = {
"internal-networking": {
title: "Internal networking",
},
"ipv6-dual-stack": {
title: "IPv6 and dual-stack",
},
"container-registry": {
title: "Use a different container registry",
},
};

export default meta;
134 changes: 134 additions & 0 deletions app/en/operate/deploy/helm/container-registry/page.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
---
title: "Use a different container registry"
description: "Pull the Helm-installed Arcade platform's images from your own registry mirror, including for air-gapped installs"
---

import { Callout } from "nextra/components";

# Use a different container registry

This guide covers pulling every image the [Helm-installed Arcade platform](/operate/deploy/helm) needs from a registry you run, instead of Docker Hub.

## When you need it

- Your cluster can't reach Docker Hub, or your policy requires images from an approved registry.
- You run an air-gapped install.
- You mirror images to avoid Docker Hub rate limits.

## Values overlay

There's no single registry prefix for the whole chart: set each image's repository to its path in your registry. Copy the images at the tags your chart version uses (see the values reference on [Artifact Hub](https://artifacthub.io/packages/helm/arcade/arcade)).

```yaml
dockerRegistry:
enabled: true
existingSecret: arcade-registry-pull # a kubernetes.io/dockerconfigjson Secret

engine:
image:
repository: registry.example.com/arcadedev/engine
coordinator:
image:
repository: registry.example.com/arcadedev/coordinator
dashboard:
image:
repository: registry.example.com/arcadedev/dashboard
experienceApi:
image:
repository: registry.example.com/arcadedev/experience-api
workerDefaults:
image:
repository: registry.example.com/arcadedev/worker
rbac:
openfga:
image:
repository: registry.example.com/arcadedev/openfga
redis:
image:
repository: registry.example.com/redis
postgres:
image: registry.example.com/postgres:17-alpine

supportImages:
busybox:
repository: registry.example.com/busybox
curl:
repository: registry.example.com/curlimages/curl
postgres:
repository: registry.example.com/postgres
tag: 17-alpine

# Only if Arcade Deploy is enabled
deployments:
image:
repository: arcadedev/runner
registry:
server: registry.example.com
imagePullSecrets: [arcade-registry-pull]
```

### Pull credentials

`dockerRegistry` controls the pull Secret on the Arcade pods:

- `dockerRegistry.enabled` (default `true`) adds an `imagePullSecrets` entry to the Arcade component pods (Arcade Engine, coordinator, dashboard, Experience API, OpenFGA, workers, and the chart's Jobs). Leave it on for the private `arcadedev/*` images or a registry that needs credentials.
- `dockerRegistry.existingSecret` names a Secret you create in the release namespace, of type `kubernetes.io/dockerconfigjson`. This is the usual choice for a mirror.
- Without `existingSecret`, the chart generates a Secret named `dockerRegistry.secretName` from `dockerRegistry.server`, `dockerRegistry.username`, and `dockerRegistry.token` (a Docker Hub access token).

<Callout type="warning">
The bundled PostgreSQL and Redis StatefulSets don't get a pull Secret. If your mirror requires authentication, they can't pull from it. Mirror `postgres.image` and `redis.image` to a path your nodes can pull without credentials (for example, through node-level registry credentials), or use external PostgreSQL and Redis.
</Callout>

### Support images

The init containers use `supportImages.busybox` (`generate-env-file`), `supportImages.curl` (`wait-for-workers`, which must be a real curl image), and `supportImages.postgres` (`wait-for-db`, which needs the `psql` client). Each has `repository`, `tag`, and `pullPolicy`. An empty `supportImages.postgres.repository` falls back to `postgres.image`, so a mirrored `postgres.image` already covers it.

### The Arcade Deploy runner image

If you enable [Arcade Deploy](/build/arcade-deploy), the Arcade Engine starts every deployed server from the runner image in the workers namespace:

- The image is `deployments.registry.server` joined with `deployments.image.repository` (default `arcadedev/runner`). With `deployments.registry.server` empty, the repository is used as-is. Its tag is `deployments.image.tag`, or the chart's app version when that's empty.
- Deployed server pods use the Secrets listed in `deployments.registry.imagePullSecrets`. These Secrets must exist in the workers namespace (`deployments.namespace`), not the release namespace.
- When `dockerRegistry.enabled` is on and `dockerRegistry.existingSecret` is empty, the chart also adds `dockerRegistry.secretName` to that list. A Secret you name in `dockerRegistry.existingSecret` isn't added, so list it in `deployments.registry.imagePullSecrets` and create a copy in the workers namespace.

## Air-gapped installs

1. Copy every image above, at your chart version's tags, into your registry. Include the Arcade Deploy runner image if you use Arcade Deploy, and any images your workers list overrides.
2. Copy the chart itself (`oci://public.ecr.aws/s5i6x9d1/charts/arcade`) into an OCI registry you can reach, and install from there.
3. Create the pull Secret in the release namespace, and in the workers namespace for Arcade Deploy.
4. Install with the overlay above.

Image pulls are only part of an air-gapped install: the identity provider, OAuth providers, and any MCP server dependencies also need to be reachable from inside your network.

## Verify it

```bash
# Every image comes from your registry
kubectl -n arcade get pods -o jsonpath='{range .items[*]}{range .spec.initContainers[*]}{.image}{"\n"}{end}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u

# Pods pulled successfully
kubectl -n arcade get pods
kubectl -n arcade get events --field-selector reason=Failed
```

For Arcade Deploy, deploy a test server and run the same image check in the workers namespace (`arcade-workers` by default).

## Common failures

| Symptom | Likely cause and fix |
| --- | --- |
| `ImagePullBackOff` on one component | Its `image.repository` still points at Docker Hub, or the tag isn't mirrored. Check the pod's image and mirror the missing tag. |
| `ImagePullBackOff` on init containers | `supportImages.*` still point at Docker Hub. Override them. |
| `401` or `unauthorized` pulling PostgreSQL or Redis | The bundled StatefulSets have no pull Secret. See the warning under [Pull credentials](#pull-credentials). |
| `401` or `unauthorized` pulling | The pull Secret is missing, in the wrong namespace, or not `kubernetes.io/dockerconfigjson`. Check `dockerRegistry.existingSecret`. |
| Deployed servers fail to pull the runner image | The Secret isn't in the workers namespace, or isn't listed in `deployments.registry.imagePullSecrets`. |
| The runner image path is doubled, for example `registry.example.com/registry.example.com/...` | `deployments.image.repository` includes the registry host while `deployments.registry.server` is also set. Set the host in one place. |

## What you own and what the chart handles

| You | The chart |
| --- | --- |
| Running the registry, and mirroring each image at the right tag | Using the repository and tag you set for every image |
| Creating pull Secrets in the release namespace and the workers namespace | Adding `imagePullSecrets` to Arcade pods when `dockerRegistry.enabled` |
| Mirroring the chart for air-gapped installs | Generating a pull Secret from `dockerRegistry.username` and `dockerRegistry.token` when you don't bring one |
| Keeping mirrors current on each upgrade | Building the runner image path from `deployments.registry.server` and passing pull Secrets to deployed servers |
112 changes: 112 additions & 0 deletions app/en/operate/deploy/helm/internal-networking/page.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
title: "Internal networking"
description: "Run a Helm-installed Arcade platform with no public ingress, behind an internal load balancer or Gateway API, and let it reach MCP servers on your internal network"
---

import { Callout } from "nextra/components";

# Internal networking

This guide covers running the [Helm-installed Arcade platform](/operate/deploy/helm) entirely inside your network: no public ingress, traffic through an internal load balancer or a Gateway API implementation you run, and MCP servers on internal addresses that the Arcade Engine is allowed to reach.

## When you need it

- Arcade must not be reachable from the internet, only from your corporate network or VPN.
- Your cluster already exposes internal services through a Gateway API implementation or an internal ingress controller.
- MCP servers or workers that Arcade calls live on private addresses (for example `10.0.0.0/8` or `*.corp.internal`).

## Values overlay

The chart doesn't create a load balancer itself. Every Arcade Service is `ClusterIP` by default (`engine.service.type`, `coordinator.service.type`, `dashboard.service.type`, `experienceApi.service.type`), and traffic enters through routes the chart renders for your own Gateway or ingress controller. Making that entry point internal is a property of the Gateway or ingress controller you attach to.

### Gateway API (recommended)

The chart renders `HTTPRoute`s, never the `Gateway`; you run that. When the AWS Load Balancer Controller's `TargetGroupConfiguration` CRD (`gateway.k8s.aws/v1beta1`) is installed, the chart also renders one per service to set its health check path. Point the routes at an internal Gateway, one whose listener is backed by an internal load balancer in your environment:

```yaml
gateway:
enabled: true
parentRefs:
- name: arcade-internal-gw
namespace: gateway-system
sectionName: https
hostname: arcade.corp.example.com
protocol: https

ingress:
enabled: false
```

A service's route renders only when it has parentRefs: its own `gateway.<service>.parentRefs` if set, otherwise the shared `gateway.parentRefs`. With both empty, that service gets no route. Services get hostnames from `gateway.subdomains.*` (`api`, `coordinator`, `dashboard`, `experienceApi`) under `gateway.hostname`. To attach one service to a different Gateway, set its own `gateway.<service>.parentRefs` (`engine`, `coordinator`, `dashboard`, `experienceApi`), which replaces the shared list for that service. A service with its own `parentRefs` gets a route even when the shared list is empty. If your controller only serves `v1beta1` routes, set `gateway.apiVersion`.

### Legacy Ingress

The `ingress.*` path is deprecated in favor of `gateway.*`, but still works. Use the private ingress class, the one backed by your internal load balancer:

```yaml
ingress:
enabled: true
type: private
privateIngressClass: nginx-internal
hostname: arcade.corp.example.com
privateAnnotations:
cert-manager.io/cluster-issuer: selfsigned-ca-issuer
nginx.ingress.kubernetes.io/ssl-redirect: "true"
```

`ingress.type: private` (the default) selects `ingress.privateIngressClass` and `ingress.privateAnnotations`. With `ingress.selfSignedIssuer.enabled` (on by default), the chart also creates a self-signed ClusterIssuer named by `ingress.selfSignedIssuer.name`; replace it with your internal CA's issuer if you have one.

### Let the Arcade Engine reach internal MCP servers

By default the Arcade Engine refuses to dial private and internal addresses from user-created worker and MCP server URLs. List the destinations it may reach in `engine.ssrfAllowlist`:

```yaml
engine:
ssrfAllowlist:
- https://tools.corp.example.com:8443 # exact scheme + host [+ port]
- "*.corp.internal" # any subdomain of the suffix
- 10.20.0.0/16 # CIDR or bare IP, matched against the dialed IP
```

An empty list (the default) keeps the private-address block. `"*"` turns the check off for every destination; avoid it outside evaluation. Two related lists use the same entry shapes: `engine.authSsrfAllowlist` for tenant-created OAuth provider endpoints (empty by default), and `engine.pluginSsrfAllowlist` for plugin webhook hooks.

<Callout type="warning">
`engine.pluginSsrfAllowlist` defaults to `["*"]`, so plugin hooks can reach any internal address, cloud metadata included. On an internal network, narrow it to the hook endpoints you use, or set it to `[]` if no hook needs an internal address.
</Callout>

### Trust your internal proxy's forwarded headers

If TLS ends at your load balancer or Gateway, the coordinator needs to trust its `X-Forwarded-*` headers to build `https://` sign-in redirects. Set `coordinator.proxyHeaders.forwardedAllowIps` to the pod or Service CIDR of your ingress controller or Gateway data plane (comma-separated); the default trusts only `127.0.0.1`.

## Verify it

```bash
# Routes attached to your Gateway, and accepted by it
kubectl -n arcade get httproute
kubectl -n arcade describe httproute

# No public ingress or LoadBalancer Services from the chart
kubectl -n arcade get ingress,svc
```

Then, from a machine inside your network, open the dashboard hostname and sign in. From outside the network, the hostname shouldn't resolve or connect.

## Common failures

| Symptom | Likely cause and fix |
| --- | --- |
| No `HTTPRoute` for a service | `gateway.enabled` is `false`, or neither `gateway.parentRefs` nor that service's `gateway.<service>.parentRefs` is set. |
| Routes exist but don't serve traffic | The Gateway doesn't accept them. Check the route's `status.parents` conditions, the listener's `sectionName`, and whether the Gateway allows routes from the release namespace. |
| Sign-in redirects to `http://` or loops | TLS ends upstream and the coordinator doesn't trust the proxy. Set `coordinator.proxyHeaders.forwardedAllowIps`. |
| A worker or MCP server on a private address fails to connect | The Arcade Engine's private-address block refused it. Add the host, wildcard, or CIDR to `engine.ssrfAllowlist`. For an OAuth provider endpoint, use `engine.authSsrfAllowlist`. |
| Connections to an allowed internal address still time out with `networkPolicy.enabled: true` | The chart's egress policy for Arcade pods doesn't include that range. Add it to `networkPolicy.egressCIDRs`. |

## What you own and what the chart handles

| You | The chart |
| --- | --- |
| The `Gateway` or ingress controller, and the internal load balancer behind it | `HTTPRoute`s (or Ingress resources) for each Arcade service |
| Internal DNS records for the hostnames | `ClusterIP` Services for every component |
| TLS certificates or the issuer for your internal CA | A self-signed ClusterIssuer when `ingress.selfSignedIssuer.enabled` |
| Firewalls and network reachability between your network and the cluster | The Arcade Engine's private-address block and the allowlists you configure |
| Choosing which internal destinations Arcade may reach | Optional NetworkPolicies for Arcade pods (`networkPolicy.*`) |
Loading
Loading