Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 14 additions & 13 deletions .github/workflows/approve-override.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,25 +48,26 @@ jobs:
role-to-assume: arn:aws:iam::${{ env.AWS_ACCOUNT_ID }}:role/javabin-ci-override-approver
aws-region: ${{ env.AWS_REGION }}

# The gate verifies this job's GitHub OIDC token (workflow, repo and
# override-approval environment) and records its actor as approver.
# Inputs go through env, never into the script text.
- name: Sign override via gate Lambda
run: |
aws lambda invoke \
--function-name javabin-apply-gate \
--payload "$(jq -n \
--arg action sign \
--arg plan_key "${{ inputs.plan_key }}" \
--arg approved_by "${{ github.actor }}" \
--arg reason "${{ inputs.reason }}" \
'{action: $action, plan_key: $plan_key, approved_by: $approved_by, reason: $reason}')" \
--cli-binary-format raw-in-base64-out \
/dev/stdout
env:
PLAN_KEY: ${{ inputs.plan_key }}
REASON: ${{ inputs.reason }}
run: sh scripts/invoke-override-sign.sh

- name: Notify Slack
env:
SSM_WEBHOOK_PARAM: /javabin/slack/platform-override-alerts-webhook
run: sh scripts/notify-slack.sh "Risk Override Approved" "*Repo:* ${{ inputs.repo }}\n*Plan:* \`${{ inputs.plan_key }}\`\n*By:* ${{ github.actor }}\n*Reason:* ${{ inputs.reason }}"
REPO: ${{ inputs.repo }}
PLAN_KEY: ${{ inputs.plan_key }}
REASON: ${{ inputs.reason }}
run: sh scripts/notify-slack.sh "Risk Override Approved" "*Repo:* ${REPO}\n*Plan:* \`${PLAN_KEY}\`\n*By:* ${GITHUB_ACTOR}\n*Reason:* ${REASON}"

- name: Retrigger failed apply
env:
GH_TOKEN: ${{ github.token }}
run: gh run rerun ${{ inputs.run_id }} --repo ${{ inputs.repo }} --failed
RUN_ID: ${{ inputs.run_id }}
REPO: ${{ inputs.repo }}
run: gh run rerun "$RUN_ID" --repo "$REPO" --failed
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ terraform/state/
| `password-set` | Self-service password-set for new hero accounts (Function URL) |
| `budget-enforcer` | Scales ECS services to zero when team exceeds 200% budget |
| `resource-tagger` | EventBridge-triggered, auto-tags created-by + commit on new resources |
| `ci-broker` | Validates repo ∈ app.yaml team, vends short-lived team role credentials |
| `ci-broker` | Verifies the job's GitHub OIDC token (repo from signed claims), checks repo ∈ app.yaml team, vends short-lived team role credentials |
| `app-auth-sync` | Creates/deletes Cognito app clients requested by app.yaml `auth:` (SSM request → client + credentials) |

### Scripts
Expand All @@ -179,6 +179,7 @@ terraform/state/
| `scripts/registry.py` | Module sources + boilerplate templates for generate-modules.py |
| `scripts/resolve-tool-version.sh` | CI: resolve node/java version from the app repo's version files |
| `scripts/sync-registered-teams.py` | CI: GitHub org teams → `registered-teams.auto.tfvars` (needs App token) |
| `scripts/github-oidc-token.sh` | CI: mint a GitHub OIDC token for the broker / apply gate |
| `scripts/provision-teams.py` | CI: fetch team YAMLs from registry, invoke team-provisioner Lambda |
| `scripts/provision-groups.py` | CI: resolve members.yaml + access.yaml, invoke team-provisioner Lambda |
| `scripts/sync-members.py` | CI: sync approved heroes from Google Sheets into members.yaml |
Expand All @@ -202,6 +203,7 @@ terraform/state/
| SSM params (platform) | `/javabin/{namespace}/{name}` |
| SSM params (app) | `/javabin/apps/{team}/{service}/{name}` |
| ECS cluster | `javabin-platform` |
| ECS execution roles | `javabin-ecs-exec-{team}` (platform-owned, reads only `/javabin/apps/{team}/*`) |
| ECR repos | `{team}-{service}` |
| SNS topics | `javabin-{alerts,security,budget-enforcement}` |
| Log groups | `/ecs/{team}/{service}` |
Expand Down Expand Up @@ -287,7 +289,7 @@ The SA JSON key is at `/javabin/platform/google-admin-sa`, the impersonation tar
### Known Issues
- **ECS deploy stabilization**: platform-test-app task registers but service fails health check
- **Cognito**: pools deployed; internal pool's `GoogleSAML` IdP was created outside Terraform
- **CI broker trusts the payload repo name**: code running inside an app job (e.g. a Terraform `external` data source in tf-plan) can mint a new OIDC token for the broker role and name another repo. Fix: verify a GitHub OIDC token in the Lambda instead of a payload field
- **App databases share one network path**: service-rds allows ingress from the shared ECS tasks SG, so every team's tasks can reach every team's DB port (password still required). Fix: per-app security groups
- **Team provisioner Lambda**: All sync functions working (Google/GitHub/Budget/Cognito/Identity Center). Password-set flow deployed.
- **`registered_app_repos` manually managed**: Being replaced with team-scoped IAM roles (repo→team resolved via GitHub API at runtime)
- **Cost allocation tags pending activation**: `repo`, `created-by`, `commit` tags need activation in Billing console (requires billing data to appear first)
Expand Down
22 changes: 21 additions & 1 deletion docs/apply-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,10 @@ tf-plan → uploads plan + plan-output.txt to S3
plan-review → LLM reviews plan → uploads risk.json to S3
tf-apply → assumes javabin-ci-apply-gate (lightweight OIDC role)
→ downloads plan from S3, verifies SHA256
→ invokes gate Lambda: check(plan_key, repo_name)
→ invokes gate Lambda: check(plan_key, team, id_token)
→ Lambda verifies the GitHub OIDC token (aud javabin-apply-gate,
a javaBin/platform workflow on main) and takes the repo from its
signed `repository` claim; plan_key must be under plans/javaBin/<repo>/
→ Lambda reads risk.json from S3
→ LOW/MEDIUM: assumes app CI role via STS, returns temp credentials
→ HIGH: checks for override.json, verifies HMAC signature
Expand All @@ -29,6 +32,23 @@ tf-apply → assumes javabin-ci-apply-gate (lightweight OIDC role)
→ tf-apply uses temp credentials for terraform init + apply
```

## Caller identity

The gate and broker roles can be assumed from any javaBin repo's platform
workflow job, and code inside that job (e.g. a Terraform `external` data
source) can invoke the Lambdas directly. So nothing identity-related is taken
from the payload:

- `check` — repo comes from the job's GitHub OIDC token, the team must be the
app.yaml team the repo belongs to, and the plan must be the repo's own.
- `sign` — only a token from `approve-override.yml` in javaBin/platform,
running in the board-reviewed `override-approval` environment, is accepted.
The recorded approver is the token's `actor`: whoever dispatched the workflow (a board member must still approve the environment for it to run). Without this, any app's tf-apply job
could sign its own HIGH-risk override.

Tokens are minted with `scripts/github-oidc-token.sh <audience>` and verified
in `terraform/lambda-src/shared/github_oidc.py` (RS256 against GitHub's JWKS).

## IAM Roles

| Role | Purpose | Permissions |
Expand Down
1 change: 1 addition & 0 deletions scripts/generate-modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,7 @@ def src(mod):
blocks.append(module_block("platform", src("platform-data"), {
"project": PROJECT,
"domain": DOMAIN,
"team": team,
}))

# ECR (always)
Expand Down
22 changes: 22 additions & 0 deletions scripts/github-oidc-token.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/bin/sh
# Print a GitHub Actions OIDC token for the given audience.
#
# Usage: github-oidc-token.sh <audience>
#
# The job needs `permissions: id-token: write`. The platform Lambdas
# (ci-broker, apply-gate) verify the token and read the repo from its signed
# claims, so a job can only ever act as its own repo.

set -e

AUDIENCE="${1:?Usage: github-oidc-token.sh <audience>}"
: "${ACTIONS_ID_TOKEN_REQUEST_URL:?id-token: write permission is required}"

RESPONSE=$(curl -sSf -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=${AUDIENCE}")
TOKEN=$(echo "$RESPONSE" | jq -r '.value // empty')
if [ -z "$TOKEN" ]; then
echo "ERROR: no OIDC token returned for audience ${AUDIENCE}" >&2
exit 1
fi
echo "$TOKEN"
5 changes: 4 additions & 1 deletion scripts/invoke-apply-gate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,16 @@ echo "Requesting apply credentials from gate Lambda..."
# Build session name from GitHub context for alert attribution
GATE_SESSION="${GITHUB_ACTOR:-unknown}-$(echo "${GITHUB_SHA:-00000000}" | cut -c1-8)-${GITHUB_RUN_ID:-0}"

ID_TOKEN=$(sh "$(dirname "$0")/github-oidc-token.sh" "${PROJECT:-javabin}-apply-gate")

PAYLOAD=$(jq -n \
--arg action "check" \
--arg plan_key "$PLAN_KEY" \
--arg repo_name "$REPO_NAME" \
--arg session_name "$GATE_SESSION" \
--arg team "$TEAM" \
'{action: $action, plan_key: $plan_key, repo_name: $repo_name, session_name: $session_name, team: $team}')
--arg id_token "$ID_TOKEN" \
'{action: $action, plan_key: $plan_key, repo_name: $repo_name, session_name: $session_name, team: $team, id_token: $id_token}')

RESPONSE=$(aws lambda invoke \
--function-name "$LAMBDA_NAME" \
Expand Down
7 changes: 5 additions & 2 deletions scripts/invoke-ci-broker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
# Usage: invoke-ci-broker.sh <action>
# action: "plan" or "deploy"
#
# Env: GITHUB_REPOSITORY (org/repo)
# Env: GITHUB_REPOSITORY (org/repo, informational — the broker takes the
# repo from the job's OIDC token; needs id-token: write)
# APP_TEAM (optional) — team from app.yaml; the broker verifies the repo
# belongs to it instead of picking the repo's first GitHub team
# Output: Exports AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
Expand All @@ -14,13 +15,15 @@ set -e

ACTION="${1:?Usage: invoke-ci-broker.sh <plan|deploy>}"
REPO="${GITHUB_REPOSITORY##*/}"
ID_TOKEN=$(sh "$(dirname "$0")/github-oidc-token.sh" "${PROJECT:-javabin}-ci-broker")

echo "Requesting ${ACTION} credentials for ${REPO}..."

RESPONSE=$(aws lambda invoke \
--function-name "${PROJECT:-javabin}-ci-broker" \
--payload "$(jq -nc --arg repo "$REPO" --arg action "$ACTION" --arg session_name "${SESSION_NAME:-}" --arg team "${APP_TEAM:-}" \
'{repo: $repo, action: $action, session_name: $session_name, team: $team}')" \
--arg id_token "$ID_TOKEN" \
'{repo: $repo, action: $action, session_name: $session_name, team: $team, id_token: $id_token}')" \
--cli-binary-format raw-in-base64-out \
/tmp/broker-response.json 2>&1)

Expand Down
30 changes: 30 additions & 0 deletions scripts/invoke-override-sign.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/sh
# Ask the apply-gate Lambda to sign a risk override for a plan.
#
# Usage: invoke-override-sign.sh
# approved_by is still sent for the previously deployed gate, which reads
# it; the current gate ignores it and uses the token's actor.
#
# Env: PLAN_KEY, REASON; runs in approve-override.yml (the gate checks the
# OIDC token comes from that workflow and the override-approval
# environment, and records the token's actor as approver).

set -e

DIR=$(dirname "$0")
ID_TOKEN=$(sh "$DIR/github-oidc-token.sh" "${PROJECT:-javabin}-apply-gate")

aws lambda invoke \
--function-name "${PROJECT:-javabin}-apply-gate" \
--payload "$(jq -nc --arg plan_key "$PLAN_KEY" --arg reason "$REASON" --arg id_token "$ID_TOKEN" \
--arg approved_by "${GITHUB_ACTOR:-}" \
'{action: "sign", plan_key: $plan_key, reason: $reason, id_token: $id_token, approved_by: $approved_by}')" \
--cli-binary-format raw-in-base64-out \
/tmp/override-response.json >/dev/null

BODY=$(jq -r '.body' /tmp/override-response.json)
if [ "$(echo "$BODY" | jq -r '.signed // false')" != "true" ]; then
echo "ERROR: override not signed: $(echo "$BODY" | jq -r '.error // "unknown error"')"
exit 1
fi
echo "Override signed for $PLAN_KEY"
38 changes: 34 additions & 4 deletions terraform/lambda-src/apply_gate/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@

import boto3
from shared.github import resolve_team
from shared.github_oidc import TokenError, require_workflow
from shared.github_oidc import repo_name as oidc_repo_name
from shared.github_oidc import verify as verify_oidc

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
Expand All @@ -32,6 +35,8 @@
)
PLAN_BUCKET = os.environ.get("PLAN_BUCKET", "")
PROJECT = os.environ.get("PROJECT", "javabin")
GITHUB_ORG = os.environ.get("GITHUB_ORG", "javaBin")
OIDC_AUDIENCE = os.environ.get("OIDC_AUDIENCE", f"{PROJECT}-apply-gate")
CREDENTIAL_DURATION = int(os.environ.get("CREDENTIAL_DURATION", "900")) # 15 min

# Cache signing key across invocations
Expand Down Expand Up @@ -87,8 +92,17 @@ def action_check(event):
Output: {approved, risk_level, credentials?, reason?}
"""
plan_key = event["plan_key"]
repo_name = event["repo_name"]
team = event.get("team") or None
# Repo from the signed GitHub OIDC token, not the payload (see ci_broker)
try:
repo_name = oidc_repo_name(verify_oidc(event.get("id_token"), OIDC_AUDIENCE))
except TokenError as e:
logger.warning("Rejected gate call: %s", e)
return {"approved": False, "reason": f"Invalid GitHub OIDC token: {e}"}
# Plans are stored under plans/<org>/<repo>/...; never apply another
# repo's (possibly LOW-risk or overridden) plan.
if not plan_key.startswith(f"plans/{GITHUB_ORG}/{repo_name}/"):
return {"approved": False, "reason": f"Plan {plan_key} does not belong to {repo_name}"}
session_name = event.get("session_name")
prefix = _plan_prefix(plan_key)

Expand Down Expand Up @@ -153,11 +167,26 @@ def action_check(event):
def action_sign(event):
"""Create a signed override token and write it to S3.

Input: {plan_key, approved_by, reason}
Input: {plan_key, reason, id_token}
Output: {signed: true, plan_key}

Only the approve-override workflow in javaBin/platform, running behind
the board-reviewed override-approval environment, may sign. Without this
check any app's tf-apply job could invoke the gate and approve its own
HIGH-risk plan. The approver is the token's actor, not a payload field.
"""
plan_key = event["plan_key"]
approved_by = event["approved_by"]
try:
claims = require_workflow(
verify_oidc(event.get("id_token"), OIDC_AUDIENCE),
"approve-override.yml",
repository=f"{GITHUB_ORG}/platform",
environment="override-approval",
)
except TokenError as e:
logger.warning("Rejected override signing: %s", e)
return {"signed": False, "error": f"Invalid GitHub OIDC token: {e}"}
approved_by = claims["actor"]
reason = event.get("reason", "")
prefix = _plan_prefix(plan_key)

Expand Down Expand Up @@ -245,7 +274,8 @@ def _issue_credentials(repo_name, requested_team=None, session_name=None):


def handler(event, context):
logger.info("Apply gate invoked: %s", json.dumps(event, default=str)[:500])
logged = {k: v for k, v in event.items() if k != "id_token"}
logger.info("Apply gate invoked: %s", json.dumps(logged, default=str)[:500])

action = event.get("action", "check")

Expand Down
14 changes: 11 additions & 3 deletions terraform/lambda-src/ci_broker/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

import boto3
from shared.github import resolve_team
from shared.github_oidc import TokenError, repo_name, verify

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
Expand Down Expand Up @@ -57,13 +58,20 @@ def _assume_role(role_arn, session_name, duration):
}


OIDC_AUDIENCE = os.environ.get("OIDC_AUDIENCE", f"{PROJECT}-ci-broker")


def handler(event, context):
repo = event.get("repo", "")
action = event.get("action", "plan")
requested_team = event.get("team") or None

if not repo:
return {"error": "repo is required", "approved": False}
# The repo comes from a GitHub OIDC token, never from the payload: code
# running inside an app job can invoke this Lambda and name any repo.
try:
repo = repo_name(verify(event.get("id_token"), OIDC_AUDIENCE))
except TokenError as e:
logger.warning("Rejected broker call: %s", e)
return {"error": f"Invalid GitHub OIDC token: {e}", "approved": False}

if action not in ROLE_MAP:
return {"error": f"Invalid action: {action}. Must be plan or deploy", "approved": False}
Expand Down
Loading
Loading