diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..cd3af69 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,146 @@ +# Changelog + +All notable changes to cloudexit are recorded here. The format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and cloudexit uses +[Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +The public contract is the `cloudexit` command, `python -m cloudexit` and +`cloudexit.cli:main`. Everything under `cloudexit.*` is internal and may move in +any 2.x release. + +## [2.0.0] - unreleased + +Upgrading from 1.x? Follow the [migration guide](./docs/migration-2.0.md). + +### Breaking + +- `config.py` is no longer read. Settings come from flags, `CLOUDEXIT_*` + environment variables or `~/.config/cloudexit/config.toml`. Under + `python -m cloudexit` the current directory is first on `sys.path`, so a + `config.py` there used to be imported and executed at start-up; it no longer + is. When one is present and nothing else supplies a host or a key, the run + logs a single warning saying where to move the values. The file is never + imported, opened or parsed. +- The `HOST` and `KEY` environment variables are no longer read. Use + `CLOUDEXIT_HOST` and `CLOUDEXIT_API_KEY_FILE` (or `CLOUDEXIT_API_KEY`). The + same single warning covers them; their values are never read or logged. +- A run that relied on either source now runs offline, or fails with a + configuration error under `--upload required`. +- Platform uploads use API v2. A 1.x platform key does not work; create a new + assessment key on the assessment page. +- The sync document is `raw_data/sync.json` in the shape defined by + [`docs/api-v2/`](./docs/api-v2/), replacing 1.x `payload.json`. +- Exit codes 10–15 are reserved for upload outcomes and 130 for Ctrl+C, so a + pipeline that treated any non-zero exit as a collection failure needs + updating. See + [Exit codes](https://cloudexit.escapecloud.io/reference/exit-codes.html). +- The catalogue dataset lives in the user data directory and updates itself. A + `./datasets/data.db` from 1.x is no longer picked up implicitly; install it + with `cloudexit catalogue import datasets/data.db`. +- The package is `cloudexit/`. 1.x `core.*` and `utils.*` imports are gone. +- There is no `egress.html`. The egress section is part of `index.html`, with + `egress.pdf` as its printable form. + +### Added + +- `--upload auto|required|off` (also `CLOUDEXIT_UPLOAD` and `upload` in the + config file), so a pipeline can insist that an upload happens rather than + falling back to an offline run. +- A config file with profiles: `[default]` and `[profiles.]`, selected + with `--config-profile` or `CLOUDEXIT_PROFILE`. Template: + [`config.example.toml`](./config.example.toml). +- Key files: `--api-key-file`, `CLOUDEXIT_API_KEY_FILE` and `api_key_file`, with + a warning when the file is readable by group or others. +- `cloudexit status` — the resolved settings with their sources, the TLS chain, + the key (kind, organisation, assessment, scopes, expiry) and the platform's + own `/meta`, plus the spool and a catalogue comparison. `--output json` prints + the same as one JSON object. +- `cloudexit config path` and `cloudexit config show` — where settings are + looked up, and every resolved value with its source. Keys are masked. +- `cloudexit catalogue status|update|import` — inspect, download or install the + catalogue dataset, including on an air-gapped machine (`--sha256` verifies the + file). `--catalogue FILE` uses a file for one run without installing it, and + `CLOUDEXIT_CATALOGUE_URL` points the download at a mirror. +- `cloudexit upload` — upload sync documents, or folders of them, for example + from an air-gapped machine. With an account key it takes `--assessment-id`, or + `--name` and `--exit-strategy`. +- A spool for uploads that could not be delivered, with + `cloudexit spool list|flush|purge`. Retries honour `Retry-After`, carry one + idempotency key per sync so a repeat is never charged twice, and stay within + `--upload-budget` seconds. `--no-flush` leaves earlier syncs alone and + `--spool-on-error` also keeps syncs the platform refused. +- `--emit-payload DIR` — write a second copy of the sync document for review or + for a later upload. +- `--strict` — fail the run when the platform reports warnings, after the report + has been written. +- `--output json` — one JSON object on stdout at the end of a run, with the step + output on stderr. +- `--egress-names plain|hashed` — send egress resource names as they are, or as + HMAC-SHA256 hashes with a per-scope salt that stays on your machine. +- `--ca-bundle` and `CLOUDEXIT_CA_BUNDLE` for private certificate authorities; + `REQUESTS_CA_BUNDLE`, `SSL_CERT_FILE`, `HTTPS_PROXY` and `NO_PROXY` are + honoured. Certificate verification cannot be turned off. +- `--timeout` and `--upload-budget` for slow links. +- Exit strategy 2, Hybrid Cloud Adoption, can be chosen in the interactive + prompt and with `ESC_EXIT_STRATEGY`, matching the JSON config, the catalogue + and the reports. +- This changelog, and the [2.0 migration guide](./docs/migration-2.0.md). + +### Changed + +- `index.html` is one self-contained file. Fonts, icons and styles are bundled + and nothing is fetched from the network, so the report opens on an air-gapped + machine and can be attached to an email. +- `report.pdf` is the Basic Executive Summary, set in IBM Plex, opening with a + cover and a document information page that record the cloudexit version, the + run ID, the catalogue version and hash, and either the platform link or the + statement that the data was self-collected. The same facts are in the PDF + metadata and in a `provenance` object in the JSON report. Both PDFs end with + the Legal Notices. +- The local report is always the Basic report, computed on your machine. What a + platform computes stays on the platform; after an upload the local report + records only the link. +- A failed upload still writes the report, with a short banner, and the run + exits with the upload code once everything else has finished. Collection + failures take precedence over upload codes. +- Platform messages name the host rather than a product, an offline run is + reported as a normal mode instead of a warning, and the assessment type is + only asked when an upload will happen. +- `-vv` no longer writes request signing details or credentials to the console + or to `run.log`: botocore's signing and credential loggers stay quiet, and a + filter on both handlers redacts `Authorization` values, credentials, session + tokens and signatures. +- Egress figures that leave unsized resources out are marked with a `+`, and an + estimate where nothing could be sized shows `–` rather than a zero. +- A half-configured platform (a host without a key, or the reverse) is a + configuration error instead of a silent offline run. +- Loggers are named after their module, so `run.log` says which part of the CLI + wrote each line. +- Dependencies refreshed; `requires-python` is `>=3.12`. +- Installation is from a release tag with `pip install .`. cloudexit is not + published on PyPI. +- Data Landscape & Egress Estimation and the Infrastructure-as-Code State Scan + are no longer marked alpha. +- The platform, catalogue, upload and report reference moved from the README to + the [documentation site](https://cloudexit.escapecloud.io), so it is written + once. The README keeps a short pointer and a links table. + +### Removed + +- `config.py` and the `HOST` / `KEY` environment variables as settings sources + (see Breaking). +- The end-of-run promotional block in the terminal, and the "Next Steps" page of + the egress PDF. The terminal states what happened and where the outputs are. +- The two-second artificial pause behind the progress spinner. +- `egress.html` as a separate file, and the shared icon assets that were fetched + per run. + +### Deprecated + +- Boolean `upload` values in the config file and the environment. `true` and + `false` still map to `auto` and `off`, but write the mode instead. + +## Earlier releases + +1.x release notes are on the +[releases page](https://github.com/escapecloud/cloudexit/releases). diff --git a/README.md b/README.md index 19b2f83..42eed00 100644 --- a/README.md +++ b/README.md @@ -38,23 +38,25 @@ It is the open-source foundation of the [EscapeCloud](https://escapecloud.io) ec **Prerequisites:** Python 3.12+, and either an AWS CLI profile or an Azure CLI session. ```bash -git clone git@github.com:escapecloud/cloudexit.git +git clone --branch v2.0.0 https://github.com/escapecloud/cloudexit.git cd cloudexit -python -m venv .venv && source .venv/bin/activate -pip install -r requirements.txt +python3 -m venv .venv && source .venv/bin/activate +pip install . ``` +That installs the `cloudexit` command. To run from the checkout instead, use `pip install -r requirements.txt` and `python3 main.py`. cloudexit is not published on PyPI. + Upgrading from 1.x? Install into a new virtual environment, not over the old one. Run your first assessment: ```bash # AWS — using your existing CLI profile -python main.py aws --profile default +python3 main.py aws --profile default # Azure — using your Azure CLI session az login -python main.py azure --cli +python3 main.py azure --cli ``` Reports are written locally as HTML, PDF, and JSON. Full credential and permissions setup is covered in the [documentation](https://cloudexit.escapecloud.io). @@ -68,7 +70,7 @@ cloudexit produces a structured exit readiness report covering: - **Risk Assessment** — rule-based evaluation of exit feasibility across each identified service - **Alternative Technologies** — open-source and portable replacements for each identified cloud service, with additional details such as company ownership and location -![Report](./docs/images/Report.png) +The [documentation](https://cloudexit.escapecloud.io/reports/overview.html) shows an example report. ## Cloud Providers @@ -89,170 +91,61 @@ The CLI supports multiple input modes for both AWS and Azure. | Mode | Command | |---|---| -| Interactive (manual credentials) | `python main.py aws` | -| AWS CLI profile | `python main.py aws --profile PROFILE` | -| JSON config file | `python main.py aws --config config.json` | -| Terraform/OpenTofu state file | `python main.py aws --tfstate infra.tfstate` | -| Non-interactive (env vars, CI) | `python main.py aws --non-interactive` | +| Interactive (manual credentials) | `python3 main.py aws` | +| AWS CLI profile | `python3 main.py aws --profile PROFILE` | +| JSON config file | `python3 main.py aws --config config.json` | +| Terraform/OpenTofu state file | `python3 main.py aws --tfstate infra.tfstate` | +| Non-interactive (env vars, CI) | `python3 main.py aws --non-interactive` | **Azure** | Mode | Command | |---|---| -| Interactive (service principal) | `python main.py azure` | -| Azure CLI session | `python main.py azure --cli` | -| JSON config file | `python main.py azure --config config.json` | -| Terraform/OpenTofu state file | `python main.py azure --tfstate infra.tfstate` | -| Non-interactive (env vars, CI) | `python main.py azure --non-interactive` | +| Interactive (service principal) | `python3 main.py azure` | +| Azure CLI session | `python3 main.py azure --cli` | +| JSON config file | `python3 main.py azure --config config.json` | +| Terraform/OpenTofu state file | `python3 main.py azure --tfstate infra.tfstate` | +| Non-interactive (env vars, CI) | `python3 main.py azure --non-interactive` | See the [configuration reference](https://cloudexit.escapecloud.io/config/config-schema.html) for required permissions and config file format. -Want to see how a regulatory-aligned report looks (DORA / FINMA / UK PRA)? Run with `--dry-run` and send the output `raw_data/sync.json` to request_report@escapecloud.io — we'll generate a sample you can share with your risk or compliance team. - -## Platform settings - -Connecting to a platform is optional. Without a host and an API key cloudexit runs fully offline. Each value is resolved separately, highest precedence first: - -1. **Flags**: `--host`, `--api-key-file`, `--config-profile`, `--assessment-id`, `--ca-bundle`, `--timeout`, `--upload` -2. **Environment**: `CLOUDEXIT_HOST`, `CLOUDEXIT_API_KEY_FILE`, `CLOUDEXIT_API_KEY`, `CLOUDEXIT_PROFILE`, `CLOUDEXIT_ASSESSMENT_ID`, `CLOUDEXIT_UPLOAD`, `CLOUDEXIT_CA_BUNDLE`, `CLOUDEXIT_TIMEOUT` -3. **Config file**: `~/.config/cloudexit/config.toml` (`$XDG_CONFIG_HOME/cloudexit/config.toml`, or `%APPDATA%\cloudexit\config.toml` on Windows), with a `[default]` table and optional `[profiles.]` tables. See [`config.example.toml`](./config.example.toml). -4. **Deprecated**: the `HOST` and `KEY` environment variables (only when both are set) and `config.py`. Both still work and print a warning. - -Prefer a key file (`--api-key-file`, `CLOUDEXIT_API_KEY_FILE` or `api_key_file`) over a key in plain text, and keep it readable by you only. The CLI warns when the config file or a key file is readable by group or others. `http://` hosts are rejected, except `localhost` and `127.0.0.1`. - -```bash -cloudexit config path # where the config file is looked up -cloudexit config show # every resolved value and its source (keys are masked) -cloudexit status # check the settings, the TLS chain, the key and the platform -``` - -### Upload modes - -`--upload` (also `CLOUDEXIT_UPLOAD` and `upload` in the config file) decides what a run does with a configured platform: - -| Mode | What happens | -| --- | --- | -| `auto` (default) | Uploads when a host and an API key are configured, and runs offline when neither is. If the upload fails, the report is still written and the run exits with the upload code. A host without a key, or a key without a host, is a configuration error (exit 2), never a quiet offline run. | -| `required` | Exits 2 before anything is collected when no host and key are configured. | -| `off` | Never contacts the platform, even with a key configured. The run is the local Basic report ("Forcing Basic Assessment: uploads are off."). | - -`--dry-run` is `--upload=off` plus the sync document in `raw_data/sync.json`; it keeps the assessment depth you asked for, so the document can be uploaded later with `cloudexit upload`. It cannot be combined with `--upload=auto` or `--upload=required` on the command line. "Forcing Basic Assessment" appears only with `--upload=off` or when nothing is configured; a configured platform keeps the depth you asked for. +## Connecting to a platform -`cloudexit status` shows where each setting comes from (keys masked), checks the TLS connection (which CA bundle was used and, on failure, which certificate was not trusted), and asks the platform for its `/meta` (versions, limits, catalogue version, whether this cloudexit version is accepted) and `/whoami` (organisation, key kind, assessment, scopes, expiry, with a warning from 14 days before it). It also lists the spooled syncs and compares the installed catalogue with the platform's. `--output json` prints the same as one JSON object. +Connecting to a platform is optional; without a host and an API key cloudexit runs fully offline. Create an **assessment key** on the assessment's page, then set `CLOUDEXIT_HOST` and `CLOUDEXIT_API_KEY_FILE` — a key file readable by you only, rather than a key in plain text. In CI add `--upload required`, so a missing key fails the job before anything is collected instead of quietly producing an offline report. `cloudexit status` checks the settings, the TLS chain and the key; `cloudexit config show` says where each setting came from. -| Exit code | Meaning | -| --- | --- | -| 0 | Everything checked is fine, or no platform is configured. | -| 11 | A host without a key (or the reverse), or a key the platform does not accept. | -| 12 | The TLS check failed or the platform is unreachable. | -| 15 | The platform does not accept this cloudexit version or the sync document version. | - -## Catalogue dataset - -cloudexit assesses against a catalogue dataset that it keeps up to date. It is stored in your data directory (`~/.local/share/cloudexit/catalogue/`, `$XDG_DATA_HOME/cloudexit/catalogue/`, or `%LOCALAPPDATA%\cloudexit\catalogue\` on Windows) together with the previous version. Each run checks for a newer catalogue first. If that check fails, the run continues with the installed catalogue and says so; without any catalogue it stops. - -```bash -cloudexit catalogue status # installed and latest catalogue (and the platform's, when a key is configured) -cloudexit catalogue update # download the latest (or --version V) -cloudexit catalogue import FILE.db.gz # install a file on an air-gapped machine (--sha256 to verify) -cloudexit aws --tfstate infra.tfstate --catalogue FILE.db.gz # use a file for one run, without installing it -``` - -Set `CLOUDEXIT_CATALOGUE_URL` to download from a mirror instead of the default location. A `./datasets/data.db` from an earlier version is not picked up on its own; install it on purpose with `cloudexit catalogue import datasets/data.db`. A catalogue that holds a trigger or a view is refused, wherever it comes from. - -## Sync document - -Every run writes `raw_data/sync.json` next to the report: the run's result in the shape defined by the API v2 definition in [`docs/api-v2/`](./docs/api-v2/) (a vendored copy; do not edit it here). It holds the scope (an AWS account and region, an Azure subscription, or a Terraform state), the resource inventory, the monthly costs and, when `--egress` ran, the egress estimate. It never contains credentials and is built without a network connection. - -```bash -cloudexit aws --config config.json --emit-payload out/ # also write out/.json -cloudexit aws --config config.json --egress --egress-names hashed # send egress names as salted hashes -``` +> **Changed in 2.0:** `config.py` and the `HOST` / `KEY` environment variables are no longer read. See [Migrating from 1.x](./docs/migration-2.0.md). -With `--egress-names hashed` each egress resource name is replaced by an HMAC-SHA256 with a random per-scope salt that stays on your machine (`~/.local/share/cloudexit/salts/`), so the same name hashes the same way in later runs of the same scope. +## Documentation -## The report - -Every run writes the same three files: `report.pdf`, `index.html` and `raw_data/assessment_result.json`. The PDF is the Basic Executive Summary, set in IBM Plex (SIL Open Font License, in `cloudexit/assets/fonts/`). It opens with a cover and a document information page that record where the data came from: the cloudexit version, the run ID, the data catalogue version and hash, and either a link to the platform's reports or the statement that the data was self-collected. The same facts are in the PDF metadata and in a `provenance` object in the JSON report. The last page says what the full assessment on a platform adds. - -## Reports and uploads - -The local report is always the Basic report, computed on your machine, whether or not a platform is configured. Anything the platform computes stays on the platform: after an upload the local report only records the link (also stored in `raw_data/upload.json`). If an upload fails, the report is still written, with a short "Not uploaded: " banner, and the run exits with an upload code after everything else has finished. Problems in the collection stages take precedence over upload codes. - -`--strict` also fails the run when the platform reports warnings (an outdated catalogue, unknown resource types, a downgraded assessment tier, a deprecated cloudexit version, and so on). The report is written first, then the run exits 10; an upload failure keeps its own code. - -`--output json` makes stdout carry exactly one JSON object when the run ends, and sends the banner and the step output to stderr: - -```json -{ - "exit_code": 0, - "report_dir": "/work/reports/20261102_101500", - "sync_json": "/work/reports/20261102_101500/raw_data/sync.json", - "upload": { - "status": "uploaded", - "reason": null, - "assessment_id": "…", - "sync_id": "…", - "platform_sync_id": "…", - "url": "https://…", - "warnings": [{"code": "collector_catalogue_outdated", "message": "…"}], - "error": null - }, - "results": {"…": "what the platform returned, never shown in the local report"}, - "catalogue": {"version": "2026-11-02", "outdated": false} -} -``` - -`upload.status` is `uploaded`, `spooled`, `skipped` (`reason` is `offline`, `off` or `dry_run`) or `failed`; `error` holds `code`, `message` and `request_id`. `results` is `null` unless the platform answered. The summary never contains the API key. - -| Exit code | Meaning | -| --- | --- | -| 10 | The platform rejected the data, or it is beyond a limit the platform announces. Retrying the same data will not help. | -| 11 | The API key is missing, mistyped, expired, revoked, not allowed to do this, or belongs to another region. | -| 12 | The platform is unreachable, busy or in maintenance. A spooled sync is delivered on the next run, and that run then makes a new assessment of its own: see [Retries and the spool](#retries-and-the-spool). | -| 13 | Credits or a quota ran out. | -| 14 | The assessment named by `--assessment-id` is missing, deleted, archived, or does not match the data. | -| 15 | The platform does not accept this cloudexit version or the sync document version. | - -Before anything is sent the key is checked on your machine (prefix, length and checksum), and the platform's own description of itself (`/meta`, which needs no key) must match the key: a regional ExitCloud key is never sent to a platform that reports another region or product. Host names, addresses and ports do not matter. The key never appears in output or logs. On failure the run prints the platform's error code, its message and a request ID to quote when asking for help. - -### Retries and the spool - -A failed upload is retried when it may pass: connection errors, timeouts, HTTP 408, 425, 429, 500, 502, 503 and 504, and any error the platform marks as retryable. The wait doubles from 1 second up to 60 seconds with random jitter, `Retry-After` is honoured, and every attempt carries the same idempotency key (the sync ID), so a repeat never stores or charges a second time. All attempts together stay within `--upload-budget` seconds (default 300, `CLOUDEXIT_UPLOAD_BUDGET`). Final failures (exit codes 10, 11, 13, 14 and 15) are not retried. - -When the budget runs out on retryable failures the sync is saved to a local spool and the run exits 12 ("spooled; run `cloudexit spool flush`"). `--spool-on-error` also spools syncs the platform refused, for a later manual retry. The spool is `$XDG_STATE_HOME/cloudexit/spool//` (`~/.local/state/...`, or `CLOUDEXIT_SPOOL_DIR`): the sync document and a small file with the attempts and the last error, readable by you only, never holding the API key. The host id is a hash, so the host name is not in the path. Each spooled sync also records a fingerprint of the API key in use when it was spooled (a hash, never the key). - -```bash -cloudexit spool list # id, host id, age, attempts, last error -cloudexit spool flush [--host URL] # upload the spooled syncs, oldest first -cloudexit spool purge [--older-than 30d] [--yes] -cloudexit upload raw_data/sync.json # upload sync documents (or folders of them), for example from an air-gapped machine -``` - -Every run with a platform first delivers the spooled syncs of the same host and the same API key (oldest first, within 60 seconds); `--no-flush` skips that, and a failing flush never blocks the new run. Syncs spooled with another API key are never delivered with this one: they stay in the spool, the run says so, and `cloudexit spool list` marks them (`cloudexit spool purge` removes them). A spooled sync with no recorded key is treated the same way. - -Running again after exit 12 does two things: it delivers the spooled sync under its own sync ID, which the platform stores and charges once however often it is sent, and it then makes a new assessment, which is a new sync with a new sync ID and is charged like any other. Use `--no-flush` to leave the spool alone, or `cloudexit spool flush` to deliver it without a new assessment. A spooled sync older than the platform's `max_collection_age_days` is reported as "will be rejected" and kept until you purge it. For an account key, `upload` needs `--assessment-id`, or `--name` and `--exit-strategy`, to say where the sync goes. - -Connections always verify the platform's certificate. For a private certificate authority use `--ca-bundle` or `CLOUDEXIT_CA_BUNDLE`; otherwise `REQUESTS_CA_BUNDLE` and `SSL_CERT_FILE` are honoured. There is no option to turn verification off. `HTTPS_PROXY` and `NO_PROXY` are honoured. Requests give up after 10 seconds to connect and 60 seconds to read (`--timeout` sets both). The platform's capabilities are cached for five minutes in `~/.cache/cloudexit/` (`$XDG_CACHE_HOME/cloudexit/`, or `%LOCALAPPDATA%\cloudexit\cache\` on Windows). +| Topic | Page | +|---|---| +| Settings, precedence, config file, key files | [Platform settings →](https://cloudexit.escapecloud.io/config/platform-settings.html) | +| Keys, upload modes, the spool, what gets uploaded | [Connected Mode →](https://cloudexit.escapecloud.io/connected-mode/upload-modes.html) | +| Running in CI/CD | [Non-Interactive Mode →](https://cloudexit.escapecloud.io/non-interactive/overview.html) | +| The catalogue dataset | [Catalogue →](https://cloudexit.escapecloud.io/catalogue/overview.html) | +| Exit codes | [Exit codes →](https://cloudexit.escapecloud.io/reference/exit-codes.html) | +| Troubleshooting | [Troubleshooting →](https://cloudexit.escapecloud.io/troubleshooting/common-errors.html) | +| Migrating from 1.x | [docs/migration-2.0.md](./docs/migration-2.0.md) | +| Release history | [CHANGELOG.md](./CHANGELOG.md) | -## Data Landscape & Egress Estimation (alpha) +## Data Landscape & Egress Estimation Add `--egress` to any assessment to measure how much data lives in the assessed scope and estimate the one-time internet egress fee for moving it out — based on the provider's tiered list prices, with no additional permissions required. ```bash -python main.py azure --cli --egress -python main.py aws --profile PROFILE --egress +python3 main.py azure --cli --egress +python3 main.py aws --profile PROFILE --egress ``` See the [egress reference](https://cloudexit.escapecloud.io/egress/overview.html) for details. -## Infrastructure-as-Code State Scan (alpha) +## Infrastructure-as-Code State Scan Instead of connecting to a cloud account, cloudexit can build the assessment from a local **Terraform / OpenTofu state file** — no credentials, no API calls, nothing leaves your machine. ```bash -python main.py aws --tfstate infra.tfstate -python main.py azure --tfstate infra.tfstate +python3 main.py aws --tfstate infra.tfstate +python3 main.py azure --tfstate infra.tfstate ``` If your state lives in a remote backend (S3, Azure Storage, Terraform Cloud, …), export it first: @@ -281,6 +174,8 @@ cloudexit runs headlessly in CI pipelines via `--non-interactive` and environmen [View on GitHub Marketplace →](https://github.com/marketplace/actions/github-action-for-cloudexit) +The Action is at `v1`; a 2.0 version is coming. Until then, pass the 2.0 flags to the CLI directly, as below. + In a pipeline that must upload, add `--upload=required` so a missing key fails the job before anything is collected, `--strict` to fail on platform warnings, and `--output json` to read the result from stdout. ## Ecosystem @@ -308,4 +203,6 @@ The public contract is the `cloudexit` command, `python -m cloudexit` and `cloud © 2024–2026 EscapeCloud. cloudexit is licensed under the **GNU Affero General Public License v3 (AGPL-3.0)**. See [LICENSE](./LICENSE) for details. +A commercial licence is available through the EscapeCloud enterprise package for organizations that cannot use AGPL-licensed software; see [escapecloud.io](https://escapecloud.io). + EscapeCloud, cloudexit and the EscapeCloud logo are trademarks. The AGPL-3.0 licence does not grant any right to use them; modified versions must not use them in a way that suggests they are official. diff --git a/config/aws_example.json b/config/aws_example.json index 1647581..01df95c 100644 --- a/config/aws_example.json +++ b/config/aws_example.json @@ -3,7 +3,7 @@ "exitStrategy": 3, "assessmentType": 1, "providerDetails":{ - "accessKey":"AKIAXASFHMTLKD6YQLHA", + "accessKey":"AKIAIOSFODNN7EXAMPLE", "secretKey":"", "region":"eu-central-1" } diff --git a/config/azure_example.json b/config/azure_example.json index 6c72a09..7081b78 100644 --- a/config/azure_example.json +++ b/config/azure_example.json @@ -3,9 +3,9 @@ "exitStrategy": 3, "assessmentType": 1, "providerDetails":{ - "clientId":"57344955-1579-4058-8604-5bb8724002de", + "clientId":"00000000-0000-0000-0000-000000000000", "clientSecret":"", - "tenantId":"38997009-9dad-42b2-b187-53f1cb71560e", + "tenantId":"00000000-0000-0000-0000-000000000000", "subscriptionId":"", "resourceGroupName":"" } diff --git a/docs/images/AWS_Config.png b/docs/images/AWS_Config.png deleted file mode 100644 index ef10ed9..0000000 Binary files a/docs/images/AWS_Config.png and /dev/null differ diff --git a/docs/images/AWS_Profile.png b/docs/images/AWS_Profile.png deleted file mode 100644 index 59a4e10..0000000 Binary files a/docs/images/AWS_Profile.png and /dev/null differ diff --git a/docs/images/Azure_CLI.png b/docs/images/Azure_CLI.png deleted file mode 100644 index d3d7c3d..0000000 Binary files a/docs/images/Azure_CLI.png and /dev/null differ diff --git a/docs/images/Azure_Config.png b/docs/images/Azure_Config.png deleted file mode 100644 index 10d0839..0000000 Binary files a/docs/images/Azure_Config.png and /dev/null differ diff --git a/docs/images/Main.png b/docs/images/Main.png deleted file mode 100644 index 5488d50..0000000 Binary files a/docs/images/Main.png and /dev/null differ diff --git a/docs/images/Report.png b/docs/images/Report.png deleted file mode 100644 index 5c6daf9..0000000 Binary files a/docs/images/Report.png and /dev/null differ diff --git a/docs/migration-2.0.md b/docs/migration-2.0.md new file mode 100644 index 0000000..bf74561 --- /dev/null +++ b/docs/migration-2.0.md @@ -0,0 +1,254 @@ +# Migrating from 1.x to 2.0 + +2.0 changes where settings come from, how uploads work and what a run writes. +Nothing about your cloud credentials or the required read-only permissions +changes. A run with no platform configured still works fully offline. + +Read [Before you start](#before-you-start), then work through the steps that +apply to you. The full list of changes is in [CHANGELOG.md](../CHANGELOG.md). + +## Before you start + +- 2.0 needs Python 3.12 or newer. +- Nothing is read from `config.py` or from the `HOST` / `KEY` environment + variables any more. If you use either, do [step 2](#2-move-your-settings) + before your first 2.0 run. +- If you upload to a platform, you need a new API key: see + [step 5](#5-get-a-new-platform-key). + +## 1. Install into a new virtual environment + +**Recommended.** 1.x and 2.0 pin different dependency versions, and `pip` leaves +the old ones in place when it upgrades, which can leave a partially upgraded +Azure SDK behind. + +```bash +git clone --branch v2.0.0 https://github.com/escapecloud/cloudexit.git cloudexit-2.0 +cd cloudexit-2.0 +python3 -m venv .venv && source .venv/bin/activate +pip install . +cloudexit --version +``` + +To run from the checkout without installing, use +`pip install -r requirements.txt` and `python3 main.py` instead. + +### Upgrading in place + +If you upgrade over an existing environment and a run stops with + +``` +The Azure SDK in this environment is incomplete (cannot import azure.mgmt..). +``` + +reinstall the package the message names, pinned as this installation asks for +it: + +```bash +pip install --force-reinstall --no-deps 'azure-mgmt-resource==' +``` + +The message always names the exact package and version. If several are affected, +a new virtual environment is faster than repairing them one at a time. + +## 2. Move your settings + +`config.py` is **removed** in 2.0, not deprecated: it is never imported, opened +or parsed. The same is true of the `HOST` and `KEY` environment variables. When +one of them is still present and nothing else supplies a host or a key, the run +logs one warning telling you where to move the values — their contents are never +read or logged. + +A 1.x `config.py` looked like this: + +```python +HOST = "platform.example.com" +KEY = "your-platform-key" +``` + +Pick one of the two replacements. + +**A config file** (`cloudexit config path` prints the location — +`~/.config/cloudexit/config.toml` on Linux and macOS, +`%APPDATA%\cloudexit\config.toml` on Windows): + +```toml +[default] +host = "platform.example.com" +api_key_file = "~/.config/cloudexit/api-key" +upload = "auto" +``` + +Keep the key in its own file rather than in the config file: + +```bash +mkdir -p ~/.config/cloudexit +install -m 600 /dev/null ~/.config/cloudexit/api-key +printf '%s' 'YOUR-NEW-KEY' > ~/.config/cloudexit/api-key +chmod 600 ~/.config/cloudexit/config.toml +``` + +**Environment variables**, which suit CI better: + +```bash +export CLOUDEXIT_HOST=platform.example.com +export CLOUDEXIT_API_KEY_FILE=/run/secrets/cloudexit-key +# or, when a file is not possible: +export CLOUDEXIT_API_KEY=… +``` + +Precedence is flags, then `CLOUDEXIT_*`, then the config file. Check the result +without running an assessment: + +```bash +cloudexit config show # every value and where it came from; keys are masked +``` + +Then delete the old `config.py` and unset `HOST` and `KEY`, so the warning stops +and nothing is left lying around with a key in it. + +> `HOST` and `KEY` are generic names that other tools also set. In 1.x, having +> both set was enough to make cloudexit treat them as a platform and upload to +> it. That is the main reason they are gone. + +## 3. Review your exit codes + +1.x used a small range of exit codes. 2.0 reserves 10–15 for upload outcomes and +130 for Ctrl+C, and the local report is always written before an upload code is +returned. + +| Code | Meaning | +| --- | --- | +| 0 | Success. | +| 1 | An unexpected error. | +| 2 | A configuration problem, including a half-configured platform. | +| 3–9 | Credentials, permissions, inventory, costs, risk, report, egress. | +| 10–15 | Upload outcomes: rejected, key, unreachable, quota, target, unsupported. | +| 130 | Interrupted with Ctrl+C. | + +The full table is in the +[documentation](https://cloudexit.escapecloud.io/reference/exit-codes.html). Two +consequences for pipelines: + +- A job that treated any non-zero exit as "the assessment failed" now also fails + on an upload problem, even though the report exists. Decide per code which + ones should fail the job. +- A job that must upload should pass `--upload required`, so a missing key fails + it before any cloud API is called rather than producing a quiet offline report. + +## 4. Expect the Basic report locally + +The local report is always the **Basic** report, computed on your machine, +whether or not a platform is configured. Scoring and the richer reports live on +the platform; after an upload the local report records the link to them and +nothing else. + +Each run writes: + +| File | What it is | +| --- | --- | +| `index.html` | One self-contained HTML report. Fonts, icons and styles are bundled and nothing is fetched from the network. | +| `report.pdf` | The Basic Executive Summary. | +| `egress.pdf` | Written when `--egress` ran. | +| `raw_data/sync.json` | The sync document (see [step 6](#6-payloadjson-becomes-syncjson)). | +| `raw_data/assessment_result.json` | The local results as JSON. | +| `raw_data/upload.json` | Written when an upload was attempted. | +| `run.log` | The run's log. | + +There is no separate `egress.html` any more: the Egress & Data Landscape section +is part of `index.html`, and `egress.pdf` is its printable form. If you archived +`egress.html`, archive `index.html` instead. + +## 5. Get a new platform key + +Uploads use API v2, and 1.x keys do not work with it. Create an **assessment +key** on the assessment's page in the platform and use it for every run of that +scope — one key per assessment. + +An **account key** is the other kind: it can create assessments, so +`cloudexit upload` with one needs `--assessment-id`, or `--name` and +`--exit-strategy`, to say where a sync belongs. + +Check a key before you rely on it: + +```bash +cloudexit status # key kind, organisation, assessment, scopes, expiry, TLS chain +``` + +`cloudexit status` warns from 14 days before a key expires. If your platform +uses a private certificate authority, point `--ca-bundle` or +`CLOUDEXIT_CA_BUNDLE` at the bundle; verification cannot be turned off. + +## 6. `payload.json` becomes `sync.json` + +1.x wrote `payload.json`. 2.0 writes `raw_data/sync.json` in the shape defined by +the vendored API v2 definition in [`docs/api-v2/`](./api-v2/). It holds the +scope, the resource inventory, the monthly costs and, when `--egress` ran, the +egress estimate. It never contains credentials and is built with no network +connection, so you can read it before anything is uploaded. + +```bash +cloudexit aws --config config.json --dry-run # local report plus sync.json, no upload +cloudexit aws --config config.json --emit-payload out/ # also write out/.json +cloudexit upload raw_data/sync.json # upload it later, or from another machine +``` + +If you had a script that read `payload.json`, point it at `raw_data/sync.json` +and check the field names against the definition in `docs/api-v2/` — the shape +is not the 1.x one. + +## 7. Stop updating the dataset by hand + +In 1.x the assessment dataset was a file in the checkout, under `datasets/`, that +you refreshed by pulling the repository. + +2.0 keeps a **catalogue** in your user data directory and checks for a newer one +at the start of each run, using a signed manifest. If the check fails, the run +continues with the installed catalogue and says so. The previous version is kept +alongside the current one. + +| Where | Path | +| --- | --- | +| Linux | `$XDG_DATA_HOME/cloudexit/catalogue/`, else `~/.local/share/cloudexit/catalogue/` | +| macOS | `~/.local/share/cloudexit/catalogue/` | +| Windows | `%LOCALAPPDATA%\cloudexit\catalogue\` | + +```bash +cloudexit catalogue status # installed and latest +cloudexit catalogue update # download the latest, or --version V +cloudexit catalogue import FILE.db.gz # install a file, with --sha256 to verify +``` + +Your old `datasets/data.db` is **not** picked up automatically. If you want to +keep using it, install it on purpose: + +```bash +cloudexit catalogue import datasets/data.db +``` + +On a machine with no outbound access, either import a file as above or pass +`--catalogue FILE` to use one for a single run without installing it. Set +`CLOUDEXIT_CATALOGUE_URL` to download from a mirror of your own. + +## 8. Run it once and compare + +```bash +cloudexit config show +cloudexit status # only meaningful with a platform configured +cloudexit aws --profile default --dry-run +``` + +`--dry-run` makes a complete local report and the sync document without +contacting a platform, which is the safest first 2.0 run. When the report looks +right, drop `--dry-run`. + +## Still stuck? + +- [Troubleshooting](https://cloudexit.escapecloud.io/troubleshooting/common-errors.html) + covers TLS, proxies, timeouts and the common exit codes. +- `-v` adds the interaction narrative and `-vv` adds debug output, including + third-party libraries. Neither prints credentials, and `run.log` in the report + folder holds the same lines. +- [Open an issue](https://github.com/escapecloud/cloudexit/issues) with the + command you ran, the exit code, and the request ID if the platform returned + one. diff --git a/publiccode.yml b/publiccode.yml index c076146..2b0c465 100644 --- a/publiccode.yml +++ b/publiccode.yml @@ -1,4 +1,4 @@ -publiccodeYmlVersion: "0.4.0" +publiccodeYmlVersion: "0.8.0" name: cloudexit url: https://github.com/escapecloud/cloudexit landingURL: https://escapecloud.io @@ -8,19 +8,29 @@ platforms: - windows categories: - cloud-management -developmentStatus: beta +developmentStatus: stable softwareType: standalone/backend +softwareVersion: 2.0.0 description: en: shortDescription: Open-source tool for cloud exit assessments - longDescription: Open-source tool for cloud exit assessments that helps - organizations evaluate risks, dependencies, and alternative strategies - before leaving a cloud provider. + longDescription: Open-source command-line tool for cloud exit assessments + that helps organizations evaluate risks, dependencies, and alternative + strategies before leaving a cloud provider. It inspects an AWS or Azure + scope with read-only credentials, or a Terraform/OpenTofu state file, and + writes a self-contained HTML report and a PDF executive summary on the + machine that ran it. It can also estimate the one-time data egress fee of + moving the data out, at the provider's list prices. It runs fully offline + and needs no account; results can optionally be uploaded to a platform. features: - cloud exit assessment - risk management - alternative technology analysis + - resource and cost inventory + - data egress estimation + - infrastructure-as-code state scan + - offline operation legal: license: AGPL-3.0-only diff --git a/tests/test_docs.py b/tests/test_docs.py new file mode 100644 index 0000000..9254ff7 --- /dev/null +++ b/tests/test_docs.py @@ -0,0 +1,206 @@ +import argparse +import re +import unittest +from unittest.mock import patch + +from cloudexit.cli.parser import parse_arguments +from cloudexit.settings import ENV_VARS +from tests.support.paths import ROOT + +MARKDOWN = ("README.md", "CHANGELOG.md", "docs/migration-2.0.md") +INVOCATION = re.compile( + r"(?", ">>", "|", "&&", "2>&1", "") + + +class Captured(Exception): + pass + + +def built_parser(): + # parse_arguments() builds the parser and parses in one go, so take the + # parser it would have used rather than changing the CLI for a test + holder = [] + + def capture(self, *args, **kwargs): + holder.append(self) + raise Captured + + with patch.object(argparse.ArgumentParser, "parse_args", capture): + try: + parse_arguments() + except Captured: + pass + return holder[0] + + +def surface(parser, flags, commands, path=()): + for action in parser._actions: + flags.update(action.option_strings) + if isinstance(action, argparse._SubParsersAction): + for name, subparser in action.choices.items(): + commands.add((*path, name)) + surface(subparser, flags, commands, (*path, name)) + + +def text_of(name): + return (ROOT / name).read_text(encoding="utf-8") + + +def code_fragments(name): + body = text_of(name).replace("\\\n", " ") + yield from re.findall(r"```[a-z]*\n(.*?)```", body, re.S) + yield from re.findall(r"`([^`\n]+)`", body) + + +def invocations(): + for name in MARKDOWN: + for fragment in code_fragments(name): + for match in INVOCATION.finditer(fragment): + tokens = [] + for token in match.group(1).split(): + token = token.strip("[]`'\",") + if token in STOP or token.startswith(">"): + break + tokens.append(token) + if tokens: + yield name, tokens + + +class DocumentedCommandTests(unittest.TestCase): + def setUp(self): + self.flags, self.commands = set(), set() + surface(built_parser(), self.flags, self.commands) + + def test_the_parser_surface_was_read(self): + self.assertIn(("aws",), self.commands) + self.assertIn(("spool", "flush"), self.commands) + self.assertIn("--upload", self.flags) + + def test_every_documented_flag_exists(self): + unknown = { + f"{name}: {token}" + for name, tokens in invocations() + for token in tokens + if token.startswith("--") and token not in self.flags + } + self.assertEqual(sorted(unknown), []) + + def test_every_documented_subcommand_exists(self): + unknown = set() + for name, tokens in invocations(): + words = [token for token in tokens if not token.startswith("-")] + if words and (words[0],) not in self.commands: + unknown.add(f"{name}: {words[0]}") + self.assertEqual(sorted(unknown), []) + + def test_every_documented_environment_variable_exists(self): + known = set(ENV_VARS.values()) + unknown = { + f"{name}: {found}" + for name in MARKDOWN + for found in re.findall(r"CLOUDEXIT_[A-Z_]+", text_of(name)) + if found not in known + } + self.assertEqual(sorted(unknown), []) + + +def slug(heading): + text = re.sub(r"[^\w\s-]", "", heading.strip().lower()) + return re.sub(r"\s+", "-", text.strip()) + + +def anchors(name): + found, fenced = set(), False + for line in text_of(name).splitlines(): + if line.startswith("```"): + fenced = not fenced + elif not fenced and line.startswith("#"): + found.add(slug(line.lstrip("#"))) + return found + + +class MarkdownLinkTests(unittest.TestCase): + LINK = re.compile(r"\[[^\]]*\]\(([^)\s]+)\)") + + def links(self): + for name in MARKDOWN: + for target in self.LINK.findall(text_of(name)): + if not target.startswith(("http://", "https://", "mailto:", "#")): + yield name, target + elif target.startswith("#"): + yield name, target + + def test_every_relative_link_points_at_something_in_the_repository(self): + missing = [] + for name, target in self.links(): + path = target.split("#", 1)[0] + if path and not ((ROOT / name).parent / path).exists(): + missing.append(f"{name} -> {target}") + self.assertEqual(missing, []) + + def test_every_anchor_exists_in_the_file_it_points_at(self): + missing = [] + for name, target in self.links(): + path, _, anchor = target.partition("#") + if not anchor: + continue + if path: + resolved = ((ROOT / name).parent / path).resolve() + if resolved.suffix != ".md" or not resolved.is_file(): + continue + target_name = resolved.relative_to(ROOT).as_posix() + else: + target_name = name + if anchor not in anchors(target_name): + missing.append(f"{name} -> {target}") + self.assertEqual(missing, []) + + +class ReadmeStateTests(unittest.TestCase): + def test_nothing_is_marked_alpha_any_more(self): + for name in MARKDOWN: + with self.subTest(name): + self.assertNotIn("(alpha)", text_of(name)) + + def test_the_moved_reference_block_is_not_in_the_readme(self): + body = text_of("README.md") + for heading in ( + "## Platform settings", + "## Catalogue dataset", + "## Sync document", + "## Reports and uploads", + "### Retries and the spool", + ): + with self.subTest(heading): + self.assertNotIn(heading, body) + + def test_the_readme_points_at_the_documentation_and_the_migration_guide(self): + body = text_of("README.md") + self.assertIn("## Documentation", body) + self.assertIn("./docs/migration-2.0.md", body) + self.assertIn("./CHANGELOG.md", body) + + +class ExampleConfigTests(unittest.TestCase): + PLACEHOLDERS = { + "config/aws_example.json": "AKIAIOSFODNN7EXAMPLE", + "config/azure_example.json": "00000000-0000-0000-0000-000000000000", + } + + def test_the_examples_carry_obvious_placeholders(self): + for name, expected in self.PLACEHOLDERS.items(): + with self.subTest(name): + self.assertIn(expected, text_of(name)) + + def test_no_example_config_holds_a_secret(self): + for name in self.PLACEHOLDERS: + with self.subTest(name): + body = text_of(name) + self.assertNotRegex(body, r'"secretKey"\s*:\s*"[^"]') + self.assertNotRegex(body, r'"clientSecret"\s*:\s*"[^"]') + + +if __name__ == "__main__": + unittest.main()