Skip to content

operations.md

Latest commit

 

History

History
1077 lines (910 loc) · 80.9 KB

operations.md

File metadata and controls

1077 lines (910 loc) · 80.9 KB
Raw

Operations & Maintenance

Command reference for pithead, the CLI that manages the stack. Run ./pithead help for the same list. The commands that only mean something on the appliance image are grouped separately, below.

Command reference

Command Description
./pithead setup First-time setup (interactive): dependency check, config, Tor provisioning, kernel optimization, and start. --skip-optimize skips kernel/GRUB tuning; --skip-deps skips the dependency check/install.
./pithead apply Preview and apply config.json changes. Warns before disruptive ones and recreates only what changed. -y / --yes skips the prompt.
./pithead up Start the stack.
./pithead down Stop the stack.
./pithead restart [tor|monerod] Restart the stack, or one container. restart tor rebuilds circuits when Tor clearnet egress is stuck (#424) — every Tor circuit drops and rebuilds, mining onions included, and a local monerod restarts right after tor is healthy again so it re-dials its peers (#972). restart monerod re-dials peers on its own when the node reports not synchronized after a Tor restart. See Tor egress broken while mining works.
./pithead upgrade Re-render the generated config, then pull (release bundle) or rebuild (source checkout) the images and restart. Run after downloading a newer bundle or a git pull.
./pithead logs [service] Follow logs for all containers, or a single service (e.g. logs p2pool).
./pithead status Show container status and health-check every expected service. Warns about anything down/unhealthy and exits non-zero if so (handy for cron/monitoring). Profile-aware, and treats a stopped p2pool/xmrig-proxy as intentional during a node-down failover or while the miner is held until the chains sync.
./pithead test-alert Send one marked test message through every configured Telegram, webhook, and ntfy sink. Prints one result per sink without exposing URLs or tokens and exits non-zero if any configured sink fails. Healthchecks is reported as excluded because a ping would move the dead-man switch.
./pithead doctor Read-only diagnostics: deps, Docker, AVX2, HugePages, RAM/disk, .env/onion state, and container status — plus runtime checks: the Tor container is actually running while the mining stack runs (a loud FAIL when it's down — the privacy backbone is dead), the Tor-egress firewall rules are actually installed (a reboot silently drops them while the containers auto-restart), something is listening on the stratum port (p2pool.stratum_port, default 3333, and whether that port sits on a public IP), the dashboard app answers behind its container, a clearnet request through Tor's SOCKS succeeds (a failing Tor guard breaks Healthchecks/Telegram/XvB while mining still works — fix with ./pithead restart tor, or set tor.auto_heal: true with a dashboard login and dashboard.control.enabled: true to automate it), a local monerod reports synchronized from its own RPC (a node stranded at 0 peers by a Tor restart keeps a green healthcheck while mining sits on a stale tip — fix with ./pithead restart monerod, #972), and, on the appliance, Caddy has no wildcard or public-address dashboard listener and the served certificate covers every name Caddy answers on without expiring within 30 days (fix either with ./pithead apply; an unreadable certificate file warns instead of failing, so a read hiccup can't reboot-loop the box). A paste-able health report.
./pithead backup Save config.json, .env, Caddyfile, the Tor onion keys, and the dashboard's database (your hashrate history & settings) to a timestamped, passphrase-encrypted archive under backups/ (checks free space first; stops a running stack, except caddy, for a clean copy, then restarts everything). If startup fails with unhealthy Tor and its circuit history has the saturated signature, backup discards only Tor's state file before retrying; onion keys remain. Other startup failures retain their original error and diagnostics. --with-chains also includes the blockchain data; --no-encrypt writes a plaintext tar.gz; -y / --yes skips the prompts (low free space and stopping the stack).
./pithead restore <archive> Restore configuration, data, and validated generated secrets from an encrypted or plaintext backup; regenerate .env, Caddyfile, and the masked dashboard configuration from the configuration (asks before overwriting; fixes Tor key ownership). -y / --yes skips the prompt.
./pithead reset-dashboard DESTRUCTIVE. Wipes and recreates the dashboard and P2Pool data. -y / --yes skips the prompt.
./pithead rotate-secrets Regenerate the stack's internal credentials after a suspected leak: the local Monero RPC password, the stratum access-password (only when p2pool.stratum_password is "auto"), and the xmrig-proxy control-API token. Recreates the affected containers. -y / --yes skips the prompt. See Rotating the internal secrets.
./pithead onion-client-key Print the Tor client-auth line for the dashboard onion. This is the client private key, deliberately kept out of status — add it to your Tor client's ClientOnionAuthDir. With the config editor on, the dashboard header's Show client key button gets the same line without a shell, which is how an appliance operator gets it. See Remote access over Tor.
./pithead rotate-dashboard-onion Mint a new dashboard onion address and client-auth keypair, retiring the old one. Run after a leaked address or key.
./pithead control-run-pending Drain the dashboard's control-request spool once. Fired by the pithead-control systemd path unit; run it by hand only when debugging the control channel. See Editing config from the dashboard.
./pithead egress-status Check the Tor-only egress firewall the way doctor does and write the verdict to data/control/results/egress-status.json for the dashboard. Read-only. Fired every two minutes by pithead-egress.timer; up, apply and upgrade install the timer, and network.tor_egress_firewall: false or uninstall removes it.
./pithead lan-guard-check Check the live LAN-only source rule. If it is missing, remove the node start marker and stop the nodes that publish LAN ports; if a stop cannot be verified, try restoring the rule on all three fixed node ports and keep reporting failure until the stop is verified. Fired every two minutes by pithead-lan.timer; up, apply and upgrade install the timer while a LAN port is configured or still published by a node. Run ./pithead up after fixing the firewall to restore access.
./pithead render Regenerate every derived file (.env, the Caddyfile, service configs, host units) from config.json without touching containers. The appliance runs this every boot; run it by hand after replacing the program under an existing config.
./pithead support-bundle Collect a chmod 600 diagnostics tarball for a bug report: host facts, doctor in prose and JSON, a masked config, a redacted .env, and the last 200 log lines per container with launch-line credentials, wallet addresses and the service onion scrubbed — as is any Monero address or onion written anywhere else in the log text. Read-only, and nothing leaves the box — review it, then share it.
./pithead config-reset DESTRUCTIVE. Clear the configuration and reopen the setup wizard, keeping every data directory — chains, wallets, Tor onion keys and dashboard history all stay, so reconfiguring costs no resync. Type-to-confirm unless -y / --yes. If Compose shutdown fails, print its error and warn that containers may still be running, then continue the wipe (exit 0 on completion).
./pithead uninstall DESTRUCTIVE. The clean exit: removes everything pithead put on this host, including the egress firewall rules with their pithead-egress.service boot unit and pithead-egress.timer check, the LAN-only source rule and its pithead-lan.timer check, and deletes NO data, on any flag. Prints the three-column inventory below, resolved for this box, and the exact command to delete the rest. Refuses in a pithead-vX.Y.Z directory that current does not point at, because every version directory drives the same live stack: run it in the live directory, and delete an old version directory by hand. Type-to-confirm unless -y / --yes.
./pithead version Print the installed stack version on one line (also -V / --version). Offline; no update check. doctor repeats it in its header.
./pithead help Show all commands.

Service names for logs match the containers: p2pool, xmrig-proxy, tor, dashboard, docker-proxy, docker-control, and caddy always; monerod only with monero.mode: local, tari only with tari.mode: local, and wallet-rpc / tari-wallet only when the matching view_key turns payout confirmation on. A service that isn't running has no logs to follow.

On the appliance dashboard, Advanced → Service diagnostics → Run health check runs the same read-only doctor report once and groups its existing checks under each service and the machine. Open Recent log under a service to fetch its last 200 redacted lines on demand. A dashboard wait limit means the request may still be queued or running on the host; it does not by itself show that the control runner is stuck. The wallet-rpc and tari-wallet checks still appear, but their logs stay in the owner-only support bundle because their ordinary output can carry wallet material the browser log redactor cannot safely recognize.

Appliance-only commands

These exist on every install but only do something on the appliance image, where the boot path drives most of them for you:

Command Description
./pithead os-update BUNDLE Install an OS update bundle into the spare A/B slot (rauc install). Refuses a bundle older than the running OS — a signed downgrade re-opens fixed holes — and refuses one below the /data migration floor outright. --allow-downgrade overrides the first, never the second. A bundle that declares data_migration is refused when a local Tari node's data volume lacks free space of data.mdb's size plus 5 GiB, the room its migration needs for the compacted copy; the refusal names the volume, the size needed and the size free, the same check upgrade makes (#2645). A floor above the running OS is a migrating update that fell back before its migration ran, not migrated data: the next boot of a fixed slot restores the floor, and the refusal says so and names the open route (the floor version or newer) rather than a reset. On a debug build (SSH baked in) installing a non-debug bundle, it warns first: that install removes the SSH channel you're driving it over. -y / --yes skips the prompt. On success it says the update is written to the spare slot, that this machine keeps running the current version until it reboots, that the reboot boots the new slot and commits it only if the stack comes up healthy (otherwise the next boot falls back), and the exact reboot command. --reboot reboots to finish the update once installed — it asks first unless -y is also given.
./pithead factory-reset DESTRUCTIVE, appliance only. Erase the whole data partition back to a blank machine — chains, wallets, Tor keys and settings all go — then reboot into the setup wizard. The resync that follows costs days, so reach for config-reset first. Type-to-confirm unless -y.
./pithead firstboot-wizard Browser-first setup for an unconfigured install: serves a token-gated form on http://<this-host>/, validates the answers host-side, then runs setup. A pre-seeded config.json skips the form; --cli runs the terminal wizard instead. The one-time token prints to the console, and five wrong tries mint a fresh one.
./pithead load-images Load the baked container-image archives from /opt/pithead/images into the engine when their content changed since the last load. The boot path runs this every boot, so a reinstall or update that ships new images converges with no wizard involvement.
./pithead local-miner Converge this machine's RigForge worker to what the machine is: on a coordinator it follows local_miner.enabled (RigForge setup in appliance mode when on, the miner stopped when off); on a rig it follows rig.json. Run every boot; a no-op outside the appliance.
./pithead render-quadlet Render Podman Quadlet units — the appliance runtime — from a rendered .env. Defaults to --env .env and --out ./quadlet. The os/quadlet/ fixtures pin its output byte for byte.

With a node running elsewhere

monero.mode: remote and tari.mode: remote leave that node's container out of the stack, and the commands above follow suit:

  • logs and restart have nothing to address for that node — its chain runs on the other host, so read its logs there.
  • doctor drops it from every check it owns: its data directory leaves the disk budget, its inbound onion reports "not needed" rather than missing, and the Monero sync check skips when there is no local monerod to ask.
  • status is profile-aware, so a container that mode turned off is not reported as down.
  • Switching a node between local and remote takes effect on ./pithead apply. Going remote stops and removes that node's container first, so the switch can't leave the old one running against a config that no longer points at it. Going local mints the inbound onion that mode never provisioned.

Chaining commands

Run several commands in one invocation — each step in order, left to right:

./pithead apply upgrade      # apply config changes, then upgrade
./pithead upgrade status     # upgrade, then health-check the result

Chainable commands: apply, up, down, restart, upgrade, status, doctor, backup. The whole chain is validated before anything runs; a nonsense chain is rejected with nothing executed. Rejected: a command that isn't chainable (setup, logs, restore, reset-dashboard, and the info commands run alone), a duplicated command, more than one of up/down/restart in the same chain (up down contradicts itself), and down anywhere but last (the steps after it would act on a stopped stack).

A chain fails fast: the first step that exits non-zero stops it, pithead reports which step failed and what did and didn't run, and the failing step's exit code becomes the chain's. Flags don't chain — apply -y upgrade is a single apply invocation (and apply rejects the stray argument), so run flagged commands separately.

Two commands at once

Commands that change the stack take a lock, so a second one waits instead of running alongside the first. uninstall joins that window before its first destructive step. Without it a backup — which stops the stack to take a consistent archive — could remove a container out from under a setup or an apply that was still using it.

The waiting command says what it is waiting for:

[WARNING] Another pithead operation is in progress (pid=4821 verb=backup since=2026-08-24T14:42:32Z) — waiting up to 300s for it to finish.

It then runs as soon as the first command finishes. If the wait runs out it stops without changing anything, and you can re-run it once the other command is done. Set PITHEAD_LOCK_TIMEOUT (seconds) to wait longer or give up sooner.

It only names a command it can show is still running. Where it cannot — the line was left behind by a command that was killed, or the lock is held by something that is not pithead at all — it says holder unrecorded in place of the name rather than reporting one that may have gone. The wait is the same either way: it is the lock that decides it, never the line.

The lock covers the parts of a command that change things, not the whole command. A backup takes it after its prompts, so a passphrase you have not typed yet holds nothing up, and the first-boot wizard holds it only while it deploys. os-update takes it just before it writes the spare slot, so a confirmation it is still waiting on holds nothing up either, and rotate-secrets takes it after its own confirmation and before it writes the copies of your old secrets, so a rotation that ends up waiting has not yet changed a credential. A one-click upgrade from the dashboard takes it before it writes over the install, so one that arrives while another command is running comes back refused with nothing changed, instead of overwriting the install underneath it. Read-only commands — status, doctor and logs among them — never take it and never wait.

The dashboard takes the same lock for each container start or stop it sends through its control proxy. This keeps the sync gate, node-down failover, clearnet-to-Tor transition and Tor self-heal from changing a container while a CLI mutation is recreating the stack. The dashboard receives a read-only bind mount of the same inode: it can hold the advisory lock, but it cannot alter the holder record.

The lock covers one stack, not one directory. A bundle install keeps each release in its own pithead-vX.Y.Z directory beside the one before it (see The deploy-box layout), and every one of those directories drives the same containers and the same data — including the fresh directory a one-click upgrade creates and runs from. They share a single .pithead.lock in the directory that holds them all. A plain pithead/ checkout has no siblings and keeps its lock in the checkout. PITHEAD_LOCK_FILE overrides the path.

The lock belongs to the running process rather than to the file: if a command or dashboard request ends, the lock is released with it. A leftover .pithead.lock after a crash is an ordinary file, not a stale lock, and there is nothing to clean up by hand. Do not delete or replace it while the stack is running: the CLI and dashboard must keep locking the same inode. The line inside it can outlive the command that wrote it, which is why the next command checks it before quoting it back to you. Where the file cannot be opened at all — a directory only root can write, a read-only mount — pithead says so and runs anyway rather than refusing every command that changes the stack. Dashboard container control instead fails closed if its read-only lock mount cannot be opened. A compromised dashboard can hold the lock and make a CLI mutation time out, so the lock protects operation ordering rather than availability.

Tab completion

pithead-completion.bash (in the repo root and the release bundle) completes subcommands for ./pithead <TAB>, service names for ./pithead logs <TAB>, and tor or monerod for ./pithead restart <TAB>.

bash — add to ~/.bashrc:

source /path/to/pithead/pithead-completion.bash

zsh — add to ~/.zshrc (bashcompinit needs compinit loaded first):

autoload -U +X compinit && compinit
autoload -U +X bashcompinit && bashcompinit
source /path/to/pithead/pithead-completion.bash

Service-name completion reads docker-compose.yml next to the pithead you're completing, so it works from any checkout or bundle directory.

Day-to-day

Check status and watch logs:

./pithead status
./pithead logs                 # everything
./pithead logs p2pool          # one service

status prints the usual compose table, then a per-service health check: a green ✓ for each running (and healthy) service, and a ⚠/✗ for anything unhealthy, restarting, stopped, or missing. It compares two container-engine snapshots three seconds apart. An explicit restarting state fails without requiring a metadata change, including while Docker is backing off with an unchanged restart count and start time; an increased restart count or changed start time catches loops between point-in-time Podman or Docker reads. An old, unchanged restart count on a running service is history and does not fail the check. A miner deliberately created/exited/stopped by the sync gate or node failover is the sole stopped exception during normal operation. A pending appliance data migration also withholds its chain services until the slot commits. Restarting or unhealthy services always make status exit non-zero; doctor applies the same observed-restart check to every Compose service, including Caddy. The appliance boot health gate runs both commands with the internal PITHEAD_BOOT_GATE=1, which skips the three-second restart window so its 10-second poll keeps the 2.0.0 pace. Like 2.0.0, the gate does not look for restart loops between reads; do not set the variable in a shell. Every container carries its own healthcheck — including the dashboard, Caddy, xmrig-proxy and the two Docker-socket proxies — so a ✓ usually means the service answered its probe, not merely that a process exists. xmrig-proxy is the one exception: its healthcheck script ships in the same image the control API runs in, and if a running image ever predates its own healthcheck script (an appliance whose compose was rendered ahead of its pinned images), the check reports ✓ without ever dialing the API — a caveat, not a bug, since it only trades a permanent false ✗ for a rare false ✓ on a release you're already about to update past. A service whose check hasn't passed yet shows as starting, which is normal for a minute after a start or upgrade. The Monero payout wallet stays healthy while it scans, on its first run and while it catches up after each restart, but only while its scan marker is less than 24 hours old; the marker is cleared once the wallet reaches monerod's tip. After that, a silent wallet is unhealthy; PAYOUT_SCAN_GRACE_SEC defaults to 86400 seconds in the wallet-rpc environment. The Tari payout wallet probes gRPC GetVersion instead of process liveness. Its first-scan marker survives restarts and allows at most 24 hours of silent gRPC (PAYOUT_SCAN_GRACE_SEC, default 86400 seconds); a successful probe clears the marker, so later silence is unhealthy. This container grace never hides dashboard wallet reachability. It exits non-zero when something needs attention, so you can wire it into a cron/monitoring check. A stopped p2pool/xmrig-proxy is reported as intentional, not an error: the dashboard stops it either to fail workers over a node-down outage or while the miner is held until the required chains finish their initial sync. While the dashboard's global sync gate (/api/state.syncing) is on, status reads /api/state and prints each chain's progress inline — a line per chain with the percent and the number of blocks left, so you don't have to open the dashboard to see how far off release is. A Tari row is never listed while TARI_REQUIRED=false. No ETA is shown: block rate isn't sampled, so the blocks-remaining count is the honest figure. The lines are skipped when the gate is off, or when the dashboard app isn't answering yet. When the gate is off and a miner is stopped anyway, the reason reads "the dashboard reports no chain sync in progress — check the dashboard and the container logs" instead of claiming a sync hold.

Start / stop / restart:

./pithead up
./pithead down
./pithead restart

Change a setting: edit config.json, then run ./pithead apply. See Configuration › Changing settings later.

Preview without applying: ./pithead apply --dry-run prints the change rows and stops — nothing is written or recreated. --porcelain makes the output machine-readable (FLAG<TAB>KEY<TAB>MESSAGE per row); the dashboard control runner uses exactly this.

Changing the payout wallet: apply asks you to type the last 8 characters of the new address (a bare y is not enough for the one change that redirects every future reward; apply -y skips the prompt for automation). Changing both the Monero and Tari addresses in one apply prompts once per address. The dashboard also watches the wallet p2pool actually mines to: any change raises a wallet_changed Telegram alert and a 72-hour top-bar banner — including a change you made yourself. Treat that pair as confirmation; if you didn't change it, your rewards are being redirected — check config.json and re-apply the correct address.

Reboot resilience: every service runs with restart: unless-stopped, so the stack restarts after a reboot or power loss, provided the Docker daemon starts at boot. Ubuntu's packaged Docker enables this by default; a custom/rootless install (or setup --skip-deps) may leave it disabled. ./pithead doctor checks this and warns if Docker isn't boot-enabled. Fix it with sudo systemctl enable --now docker.

The Tor-egress firewall lives in the kernel, so a reboot clears it while the containers restart. up, apply and upgrade install pithead-egress.service, which Docker's own start pulls in and waits for: it puts the rules back into DOCKER-USER before any container starts. doctor warns when the rules are live but the unit is not enabled, and FAILs when the rules are missing. The dashboard learns the same state from pithead-egress.timer, which runs pithead egress-status every two minutes, and alerts when the rules go missing while the stack runs. See Privacy › Enforced fail-closed.

Editing config from the dashboard

With dashboard.control.enabled: true (default off; needs a dashboard.auth.password), the dashboard gains a Configuration view that stages config changes for a host-side runner. apply (and setup/upgrade) install two systemd units when the flag is on, and remove them when it is off:

  • pithead-control.path — watches ./data/control/requests/ for request files.
  • pithead-control.service — a root oneshot running pithead control-run-pending from the install directory. Fixed command, no parameters from the container. It retries on failure (Restart=on-failure, 15s) with systemd's own start-limit disabled, so a request that lands while the stack momentarily isn't fully set up — recovering from a setup fault, mid-apply — gets picked up once it is, instead of leaving the runner wedged until a reboot (#2219). A stack that never finishes setup keeps retrying every 15s rather than wedging — check ./pithead doctor or systemctl status pithead-control.service if requests never seem to land.

The unit names are global to the host, so removal is ownership-checked: a checkout with the flag off only removes units whose ExecStart points at itself, comparing physical paths so the current symlink and the versioned directory it targets count as the same checkout. Another checkout on the same box (an e2e harness, a bundle smoke test) therefore cannot delete the live stack's runner and strand its queued requests. That physical-path comparison also decides whether apply needs to touch the runner at all: an apply whose config did not change re-provisions only when the installed units genuinely differ (a stale checkout path, a container-engine change, a missing hardening field) — never on a routine, unchanged apply, however the checkout was reached (current symlink or its versioned directory).

When re-provisioning is needed, apply holds the shared mutation lock while it stops pithead-control.path, waits up to 30 seconds for a request the runner has already claimed to write its result, rewrites the units and enables the path unit again. The runner itself never takes the lock, so a request that changes nothing on the stack (a preview, a diagnostic) is never delayed by a pithead command running in a shell, and never delays one. A request that changes the stack (a commit, a lifecycle verb, an upgrade) takes the lock inside its own handler like any shell command. If one is in flight when apply re-provisions, it is waiting for the lock apply holds, so the 30-second wait runs out, apply finishes, and the request then runs. None of these calls stops a runner that is working a request: on systemd 255 the running service finishes and writes its result. A request still sitting in requests/ is untouched, and pithead-control.path fires for it as soon as the path unit is enabled again. A first install, and the pithead render that runs on every appliance boot, have no runner to drain and take no lock.

Installation is ownership-checked the same way: when the units already name a different install that still exists on disk, apply refuses to overwrite them and names the owning directory — a sibling checkout turning the flag on can no longer repoint the live stack's runner at itself, which would leave the live dashboard's requests sitting unread with no error on either side. Units whose install directory is gone are adopted, a unit this tool did not write is left untouched, and a successful one-click upgrade takes the units over as part of moving current (the previous version's directory stays on disk as the rollback, so the upgrade is the one takeover that must not be refused). For the remaining deliberate cases — migrating an install by hand, or repairing after a failed upgrade — setting PITHEAD_STEAL_CONTROL_UNITS=1 forces the takeover.

The runner dispatches a fixed set of actions and rejects everything else: apply --dry-run --porcelain (preview), apply -y (commit), a release upgrade — the dashboard's Upgrade button, for which the runner re-derives the target from the GitHub release API itself and refuses any other version — the two worker actions (worker-apply, worker-upgrade), which forward a config change or an upgrade to a rig's own control API rather than touching this host, and the restart / apply lifecycle pair. An unrecognized action is rejected and audited, never run.

The spool lives under ./data/control/: requests/ (the only directory the dashboard container can write), staged/ (host-only), and results/ + audit/ + masked/ (container read-only). masked/config.json is a host-rendered copy of the config with every set secret replaced by a sentinel (#440); the Configuration form prefills from it, and the raw config.json is never mounted into the container. It is re-rendered on every setup/apply/upgrade and on every runner pass. audit/control.log records one JSON line per handled request: timestamp, logged-in dashboard user, action, outcome, and the names of the changed settings (never their values). A commit's audit line is written before its result is published. The container cannot rewrite the log. The writer trims it to its newest 2000 entries once it passes 512 KiB, so it never grows unbounded.

To disable the channel, set dashboard.control.enabled: false and run ./pithead apply: the units are disabled and removed, the routes disappear from the dashboard (404), and the spool directory is left in place. To inspect or drain the queue by hand, run ./pithead control-run-pending. A commit keeps the previous config at config.json.bak-control; a failed apply names it in the result and retries container recreation on the next apply, same as the CLI.

Watching for intruders

Caddy writes a JSON access log — one line per dashboard request, with timestamp, HTTP status, method, path, and the authenticated user — to ./data/caddy-logs/access.log (#349). It is always on. Caddy redacts credential headers by default, so no password material lands in it, and Caddy's native rolling caps it at 4 MiB per file, the current file plus two rolled ones (~12 MiB worst case). The dashboard reads the active file and at most two timestamped access-*.log or access-*.log.gz generations. Each plain-file tail and each gzip decompressed prefix is capped at 4 MiB; compressed input is also capped at 4 MiB per generation. One summary reads at most 12 MiB of JSON and 8 MiB of compressed input. Oversized files, missing or corrupt generations, and Caddy retention can omit records from the last-24-hour count. The dashboard's Configuration view surfaces these bounded records: recent accesses and a failed-login count.

How to read it, given the dashboard is reachable over a Tor onion (#343): every request arrives via the local Tor daemon, so there is no source IP to trace or block — the attack signal is the rate and pattern of failures, not identity.

  • A burst of 401s means someone who can already reach the dashboard — they have the onion address, and the client-auth key if dashboard.onion.client_auth is on (the default) — is guessing the password. Rotate it: set a new dashboard.auth.password in config.json and run ./pithead apply. Check control.log for an onion-client-key entry you did not make: that is the one place a client key can be handed out while the stack is running.
  • Unexplained traffic on a client-auth-off onion means the address itself has leaked (it is online-guessable in that mode). Rotate the address: ./pithead rotate-dashboard-onion mints a fresh onion and client key; the old ones stop working immediately (details).

Config tampering shows in the other log: ./data/control/audit/control.log (above) names every change staged through the dashboard and who committed it. A change you didn't make is the same rotate-now signal. Secret rotation beyond the dashboard password is tracked in #378.

control.log only sees requests that went through the dashboard. The dashboard's own poll loop additionally watches for three changes it did NOT make and appends them to the same audit trail (#530, #1551, details): a config.json edit with no matching commit (host-edit), a worker's control API reporting a change the dashboard never sent (rig-edit), and a rig whose config revision moved with nothing recording the change at all (rig-drift) — the hand-edit underneath RigForge that reports no change to notice. All three name only what changed — settings, a worker, a revision, never a secret value — and all three are the same rotate-now signal as an unexplained control.log entry.

Unlike control.log, which the writer trims to bound its size, the audit trail served by the dashboard persists to its own database — mirrored control.log rows plus the three out-of-band kinds above — so the Security panel's range presets, date fields and search look back further than the log's own trimmed window. Because rig-edit and rig-drift both read off the unauthenticated worker feed, they share one rate cap per worker (#724): a rig reporting a fresh change_id, or a fresh revision, every poll can add only a bounded number of rows per hour between them before the rest are dropped behind a single rate-limited marker. They share one budget rather than holding one each, since two would double what that device can make permanent. That cap bounds how many rows arrive and not how large each one is, so every audit row's own identifier is length-capped and whitelisted where it is written (#1561) — the one field the trail's sanitizer used to skip, and the one a rig picks the input to. The parts a rig chooses are escaped before that identifier is assembled (#1566), so no rig can land a row on another rig's identifier — which used to drop the second detection rather than record it. Between these bounds, a rig holding one worker name is bounded in how many rows it adds, how large each one is, and whose rows it can displace. The cap is keyed on the name the device presents and a device chooses that freely, so how many names may hold a live window at once is bounded as well (#1695): past that ceiling a name not already holding a window is refused, which is what stops a device rotating names from drawing a fresh budget per name. A ceiling on names rather than one overall row budget, because an overall budget would be spent on every other rig's behalf; a name that already holds a window keeps it. It does NOT leave every established rig alone: holding a window means having produced an out-of-band detection within the last hour, so a rig whose changes all go through the dashboard holds none, and during a flood its first detection is refused and dropped. Read the protection as "a name already being audited keeps its budget", and expect that residual. It surfaces the same way the per-worker cap does, as one rate-limited marker for the episode that names no rotated name. A genuine occasional rig change still records normally; only a flood is capped, and the cap is visible — the marker names which detection tipped it, and the dashboard logs a warning.

Archiving the XvB winners feed

scripts/watch/xvb-winners-archive.sh saves a dated snapshot of the public XvB winners feed (https://xmrvsbeast.com/p2pool/winners_recent_full_pub.txt) — the full round schedule with per-round prize hashrates and qualifier counts — before the feed's ~45-day rolling window drops the oldest rounds. The fetch runs inside the dashboard container, so it rides the stack's Tor SOCKS like every other XvB call; the feed is public and carries no wallet. The stack must be running.

This is optional. The reason to run it: the dashboard's XvB earnings estimate rests on a measured delivery band — the fraction of the advertised prize that actually arrives on-chain, measured once in the XvB delivery study (Jun–Aug 2026: point 0.33, 95% CI 0.28–0.39). XvB can change its payout behaviour at any time, and the winners feed is the raw material for re-running that measurement; without an archive, anything older than ~45 days is gone.

Run it daily from cron on the deploy box, and give it an archive directory outside the version directories:

10 0 * * * $HOME/mining/current/scripts/watch/xvb-winners-archive.sh $HOME/mining/xvb-winners-archive

Snapshots land there as winners-YYYYMMDD.txt (UTC date). The directory argument matters: without it the script defaults to xvb-winners-archive/ inside the stack directory, and the deploy layout above rotates old version directories away on upgrade — taking any archive inside them along. The explicit directory survives every upgrade. Each snapshot is ~90 KB, so a daily archive grows about 3 MB a month.

Updating the stack

The update path depends on how you installed (see Getting Started).

From the dashboard: with dashboard.control.enabled: true, a release install can run this whole sequence from the browser — see Dashboard › Upgrading from the dashboard. The steps below are what that button performs, and the only path for source checkouts.

Release bundle (the default): from the install directory, re-download the latest bundle over it, then upgrade. upgrade pulls the new published images:

curl -fsSL https://github.com/p2pool-starter-stack/pithead/releases/latest/download/pithead.tar.gz | tar xz --strip-components=1
./pithead upgrade

Source checkout: pull the latest code, then upgrade. upgrade rebuilds the first-party images locally and pulls only the pinned third-party images (Tari, Caddy, the socket proxies) that are not on the host. After the build, it recreates any container whose image ID differs from its Compose declaration and fails if the IDs still differ. setup and up fetch missing third-party images the same way:

git pull
./pithead upgrade

Either way, upgrade re-renders the generated config (.env, the Caddyfile, and the Tari config) for the new release before pulling/rebuilding, so a release that changes a config template or adds an .env var takes effect, not just the new image. Data directories and config.json are untouched, so blockchain sync and settings survive an upgrade.

An upgrade to a new Tari major version, such as 5.x to 6.x, migrates the node database on its first start by writing a compacted copy beside the old one. Before it starts or recreates any container, upgrade compares the major version of the existing tari container with the one the new release starts. When the major goes up, upgrade checks free space on the volume that holds Tari's data.mdb against the file's current size plus 5 GiB. That is a conservative bound, since the compacted copy is smaller than the original. If there is less, upgrade stops, names the volume, the size needed and the size free, and leaves the running containers as they were. Free space on that volume, or move tari.data_dir to a larger one, and run upgrade again. If no tari container exists (for example after ./pithead down), upgrade cannot tell which version wrote the database, so a shortfall is a warning instead.

On a release install with the release public key on disk (cosign.pub, shipped in every signed bundle), upgrade verifies each image's cosign signature before pulling and aborts on any failure. There is nothing to install for this: the verifier runs as a digest-pinned container, so Docker — already a prerequisite — is the only thing it needs. The first upgrade fetches that image once and never mentions it again. See Releasing › Verifying a release for what is checked.

Verification fails closed. A bad signature, a stripped pithead.tar.gz.sig, or an image the bundle does not pin by digest each abort the upgrade before anything is pulled, and the stack keeps running the version it was on. The one and only way to end up unverified is an install older than the first signed release, which carries no cosign.pub at all — that case warns loudly and proceeds.

If your install is still on v1.18.1 or v1.19.0, the upgrade refuses with "cosign is not installed on the host". Those versions verify with a host binary rather than the container, and an upgrade is driven by the code the box is already running — so the fix has to happen on the box once, before it can move forward. See Releasing › Upgrading an install older than v1.19.1.

Run ./pithead version to see what is currently installed before and after an upgrade.

Switching a source checkout to release images

A cloned repo builds :dev images locally and shows a dev · branch @ commit version badge. To run the published images instead (clean version badge, working update-checker, no local build), convert the install in place. config.json, .env, the Tor onion keys, and data directories are preserved:

./pithead backup -y          # safety snapshot: config.json, .env, onion keys, dashboard db (chains excluded)
# overlay the published release files — leaves config.json, .env, .git, and your data dirs untouched:
curl -fsSL https://github.com/p2pool-starter-stack/pithead/releases/latest/download/pithead.tar.gz | tar xz --strip-components=1
rm -f build/*/Dockerfile dashboard/Dockerfile   # remove the image Dockerfiles → pithead switches from build to pull
./pithead upgrade            # re-render config, pull :vX.Y.Z, recreate

pithead chooses build-vs-pull by whether dashboard/Dockerfile is present (the one Dockerfile is_source_checkout keys off). Deleting the Dockerfiles flips it from building :dev locally to pulling the published :vX.Y.Z. To go back to building from source, git checkout vX.Y.Z (or git pull) restores the Dockerfiles, then ./pithead upgrade rebuilds locally. upgrade refuses to start on a non-primary Monero payout address, so confirm yours is a 4…/95-char address first (see Configuration).

Moving the install? Data directories are stored as absolute paths in .env, so relocating or copying the stack to a different path (or running a second checkout) points it at a different, empty data/: a full re-sync, and the dashboard history is orphaned. If you move the install, move its data/ with it, or set absolute data_dir paths in config.json and run ./pithead apply. ./pithead up and ./pithead doctor now warn when a data directory named in .env is missing.

What uninstall removes

uninstall tears down and removes everything pithead put on this machine. It deletes no data, on any flag — then prints where the data is and the exact command that removes it, if you want it gone.

Removed:

item what
containers + networks the pithead compose project, mining_net, proxy_net
images every ref from docker compose config --images
named volumes caddy_data, wallet_data, tari_wallet_db — pithead's, not yours: the wallet volumes are view-only wallets that rebuild from the view keys in the kept config.json, and caddy_data is ACME state Caddy re-issues. On startup, tari-wallet repairs a root-owned volume before running as uid 1000. Uninstall removes the Tari wallet volume even when its payout profile is disabled, after checking its Compose ownership labels. If Docker cannot list, inspect, or remove that volume, uninstall stops before deleting .env so the operator can retry.
systemd units pithead-control.path / .service, this checkout's only
firewall the Tor-egress rules this checkout installed, and their pithead-egress.service boot unit; the LAN-only source rule, its pithead-lan-guard.service boot unit, pithead-lan-hold.service, and pithead-lan.timer check
rendered files .env, Caddyfile, build/tari/config.toml, .pithead-first-run-done
derived state dirs data/control/ (control spool + audit trail), data/clearnet-state/, data/caddy-logs/, data/proxy-tls/ (the stratum TLS keypair), and data/tari-wallet-secret.env (the Tari view-key secret) — each removed individually by path, never rm -rf data/
version symlink <parent>/current, only when it points at this checkout

Kept — yours, never touched: the Monero, Tari, P2Pool, Tor, and dashboard data dirs; config.json; backups/.

A derived directory is removed only at the path setup gives it. If .env names it anywhere else, or at, above or inside a kept path, uninstall leaves it in place with a warning. After uninstall, ./pithead setup re-provisions from the kept config.json and data dirs. The chains are reused rather than re-synced, and the kept Tor data gives back the same onion addresses. Secrets that lived only in .env or in a removed directory are generated anew: the proxy token, an auto stratum password and the stratum TLS keypair. Rigs that use the generated password or pin the TLS fingerprint need the new values.

Left behind — installed by setup, shared with the machine, not removed:

item why it stays to remove it by hand
apt packages jq, openssl, docker.io, docker-compose-v2 other software on the box may use them sudo apt-get remove <pkgs>
GRUB HugePages cmdline reverting needs update-grub and a reboot the verb must not trigger remove /etc/default/grub.d/zz-pithead-hugepages.cfg, restore /etc/default/grub.bak if an older setup edited the defaults, sudo update-grub, reboot
runtime HugePages pool resets on reboot anyway sudo sysctl -w vm.nr_hugepages=0

The deploy-box layout

A box that installs each release into its own directory keeps code and data apart. This is the canonical layout — one parent directory holds the version dirs, a current symlink, and a shared data root:

~/mining/
├── current -> pithead-v1.5.0      # the live install — maintained by setup/upgrade
├── pithead-v1.5.0/                # code + config.json + generated config (.env, Caddyfile)
├── pithead-v1.4.0/                # ONE previous version dir, kept for rollback
└── data/                          # shared data root — survives every upgrade
    ├── monero/  tari/  p2pool/  tor/
    └── dashboard/                 # the dashboard database lives here too (see below)
  • current — ./pithead setup and ./pithead upgrade update it (ln -sfn) whenever the install directory is named pithead-vX.Y.Z; the dashboard's one-click upgrade runs the same upgrade and moves it too. Only a successful upgrade moves the pointer, so current always names the version that last came up. It is informational — the containers mount the absolute paths in .env — but it makes the live install discoverable without docker inspect. Any other directory name (a source checkout, a plain pithead/ extract) leaves the symlink alone.
  • Version dirs — keep current's target plus one older dir for rollback; delete anything older by hand. ./pithead uninstall refuses in a version dir that current does not name, because it would stop the live stack. Each release lands in a fresh dir: extract the bundle, copy config.json and .env from the previous dir, run ./pithead upgrade. Rollback is the same two steps from the older dir. The single-directory overlay under Updating the stack also works; the per-version layout is what a long-lived box converges to. The dashboard's one-click upgrade follows the same discipline on this layout: it extracts the new release into a fresh pithead-v<new>/ beside the running one, seeds config.json, .env, and the control spool, and leaves the previous dir intact for rollback. Only when a data directory resolves inside the install dir (the pre-shared-root default) does it extract in place instead — a dir swap would strand that data — and says so in the journal.
  • Shared data root — point monero.data_dir, tari.data_dir, p2pool.data_dir, and tor.data_dir at absolute paths under one parent (here ~/mining/data). When all four share a parent, the dashboard database defaults there as well (<root>/dashboard) instead of inside the version dir — no data moves with the code. Installs that pre-date this carry the dashboard data at the old in-install default (./data/dashboard); the first upgrade (or apply) moves it to the shared root automatically, stops the dashboard for the move, and verifies the database arrived. That automatic migration leaves an explicit dashboard.data_dir alone. When you confirm a change to that setting, apply instead copies and verifies the live database at the new path; a non-empty target refuses rather than guessing which database is live.
  • Config archives — ./pithead backup writes under backups/ inside the dir that ran it. Before deleting an old version dir, keep any backups/ archives you still want.

Backups

State lives in the data directories (by default under ./data/, or wherever each *.data_dir points; see Configuration › Data directories):

  • config.json: settings. chmod 600; keep a copy off-host.
  • config.json.bak-upgrade-* / .env.bak-upgrade-*: pre-upgrade copies the dashboard's one-click upgrade keeps before an in-place extraction (#637). The newest three pairs are kept; older ones are pruned automatically. The .env copies carry secrets — handle them like .env itself.
  • data/tor/: onion service keys retain the same addresses across a rebuild. Restore discards Tor's disposable circuit state, so it builds fresh circuits on the next start.
  • data/monero/, data/tari/: the blockchains. Large; backing them up saves a re-sync, but they re-download from the network if lost.
  • data/dashboard/: the dashboard database (hashrate history and settings). Small and irreplaceable — it does not re-sync — so it is part of the default backup.

Stop the stack (./pithead down) before copying data directories by hand, so files are consistent.

backup / restore

Instead of copying files by hand, run:

./pithead backup

This prompts for a passphrase, then writes a timestamped, encrypted tar.gz.enc under backups/ holding the irreplaceable state: config.json, .env (secrets), the Caddyfile (if present), the Tor onion keys, and the dashboard database (hashrate history and settings). Blockchains are excluded (they re-sync), so the archive is small. The archive is chmod 600, and pithead prints its path when done. Before touching anything, backup checks that config.json and .env are still there as regular files, then checks free space and prompts if it looks tight — either check can refuse before the running stack is stopped for an archive that would fail anyway. On a source checkout, backups/ is git-ignored — the archive carries .env and the onion private keys, and secret scanners can't see inside a tarball.

Encryption is openssl enc -aes-256-cbc -pbkdf2 with 600,000 PBKDF2 iterations (openssl is already a stack dependency; AEAD modes like GCM are not available through openssl enc, so CBC is the portable choice). The tar stream pipes straight into openssl — no plaintext archive ever touches the disk — and the passphrase passes over a file descriptor, never the command line. chmod 600 protects the archive on this disk only; the encryption is what protects it once it is copied off-box. The archive is unreadable without the passphrase, onion keys included, so store the passphrase somewhere other than this host.

Losing the passphrase loses the backup — there is no recovery!

For unattended runs (cron), set PITHEAD_BACKUP_PASSPHRASE in the environment. An unattended run (--yes) with no passphrase set refuses rather than downgrading — a job whose passphrase line is typo'd away must fail loudly, not archive your onion keys in plaintext while reporting success. To write a plaintext archive on purpose, pass --no-encrypt (an empty passphrase at the interactive prompt does the same, with a warning).

If the stack is running, backup stops it (except caddy, the reverse proxy, which stays up — none of its own state is in the archive) for a consistent copy and restarts everything when done. A failed backup (disk full mid-archive, for example) removes the partial archive and still restarts the stack before reporting the error. If startup fails, it prints Tor's health-check history and last 40 container log lines before retrying, while retaining the original startup error. Failed Tor health checks record bootstrap progress and tag, or a cookie, control-query, authentication, or invalid-reply failure. Raw control replies and relay summaries are omitted. Pass -y / --yes to skip both prompts (low-space warning, stop-the-stack question).

Include the blockchains (larger, slower) with:

./pithead backup --with-chains   # also include the blockchain data

To recover a DIY installation, run from the original installation directory. After config-reset, no setup or manual .env is needed: the reset retains only the trusted data paths in an owner-only .restore-paths record. Restore removes that record after success. On a fresh installation, default data directories need no prior configuration. For custom data directories without a reset record, configure the original paths in config.json and run ./pithead render before restoring; an archive cannot authorize new destination paths.

On the appliance, config-reset reboots into the setup wizard. After a successful provisioned boot, the reset records the current good slot and exact image source. When the wizard container starts, firstboot restores that same slot's good state on each configless boot, including a reboot while setup is still open. The record remains until a provisioned boot passes its health gate, so reset, CLI restore and setup followed by a normal reboot keep the intended slot. An unverified slot keeps its normal fallback; the next provisioned boot still requires the stack health gate. If you restore from the shell instead of the wizard page, firstboot detects a configuration that appears after its entry check, including while it prepares the page. It closes the page and frees ports 80 and 443 for Caddy; run ./pithead up after that. A retry round that opened with a retained configuration does not automatically detect its replacement by a CLI restore.

Copy the archive back and run:

./pithead down                       # stop the stack first so files restore cleanly
./pithead restore backups/pithead-backup-YYYYmmdd-HHMMSS.tar.gz.enc
./pithead up

restore detects the format from the archive itself — encrypted backups ask for the passphrase (or read PITHEAD_BACKUP_PASSPHRASE), and plaintext archives from earlier releases restore unchanged, no flag needed. A wrong passphrase, or a corrupt or truncated archive of either format, fails before anything on disk is touched. restore also refuses while services are running, restarting, or paused. With .env absent, restore checks the engine's project labels directly; down stops and removes those containers without interpolating Compose, keeping data and named volumes. Networks remain until a configured Compose shutdown. The pinned pithead project and this directory's legacy project are checked; legacy containers must also carry this directory's working-directory label. It stages the archive privately, accepts only the configured files and data directories, rejects redirected destinations, and clamps restored secrets to owner-only modes before committing them. .env, Caddyfile, and the masked dashboard configuration are regenerated from validated config.json. The next up serves the restored configuration and energy cost without an intervening apply or another restart; the validated proxy token, wallet RPC and database passwords, and Tor identities are retained exactly from the archived environment. The dashboard login hash and fingerprint are retained with them while the fingerprint matches dashboard.auth.password and the hash is well-formed bcrypt; otherwise restore hashes the configured password again. An archive is trusted as far as its own config.json: whoever can edit it can change the login, so keep backups private and encrypted. --yes skips the overwrite prompt, not these checks. Restore fixes Tor key ownership so the onion address returns unchanged, and restores hashrate history and dashboard settings — including the sync gate's own released/held state, since this is the same-box recovery door: the machine's chains have not gone anywhere. Restore discards Tor's circuit state, including from older archives; Tor rebuilds that history on startup without changing the onion keys.

Restore collision rules

The restore door determines which copy wins when both the archive and destination have a file at the same path. ./pithead restore is an explicit recovery command on an already deployed box, so the archive replaces the destination's file. The appliance wizard's restore-at-setup path instead preserves the destination's file in data/monero, data/tari, and data/p2pool: a wipe=keep install may already have synced chain data, and replacing it would force a resync. It still adds files that only the archive has. The wizard replaces its Tor and dashboard directories normally.

NOTE: The archive stores the source box's absolute paths, and restore puts every file back exactly where it came from, after validating the destinations. A different installation path is refused rather than redirected into the old directory tree. Recreate the original installation path and configure any custom data paths before restoring.

NOTE: A same-machine restore preserves the dashboard TLS certificate and key when the settled names are unchanged. On the appliance, a certificate covering the current canonical names also survives temporary address loss and dashboard password changes. A genuinely new, uncovered name requires a new certificate; verify its fingerprint before trusting it.

If your browser warns after a DIY restore, verify and trust the installation's root CA. Accepting only the current certificate can produce another warning after renewal.

Rotating the internal secrets

.env and config.json carry three credentials that are generated once and otherwise live forever. If either file may have leaked — a backup left the box, a .env was pasted into a bug report, an operator with access left — rotate them:

./pithead rotate-secrets

The command previews exactly what your config rotates, asks for confirmation (-y / --yes skips it), regenerates the secrets, re-renders .env, and recreates the containers that consume them (a brief restart; chain data and dashboard history are untouched):

  • Monero node RPC password (monero.node_password) — rotated in local mode only; internal to the stack, no follow-up needed. With monero.mode: "remote" the credential belongs to the remote node and is skipped.
  • Stratum access-password (PROXY_STRATUM_PASSWORD) — rotated only when p2pool.stratum_password is "auto". The new value is printed at the end. Every rig is rejected until its stratum pass is updated to it — see Workers › Authentication. A literal password lives in config.json: change it there and run ./pithead apply. When stratum auth is off there is nothing to rotate.
  • xmrig-proxy control-API token (PROXY_AUTH_TOKEN) — always rotated; internal to the stack, no follow-up needed.

Before changing anything, rotate-secrets saves owner-only copies of config.json and .env as config.json.bak-<timestamp> and .env.bak-<timestamp>. They hold the OLD secrets: delete them once the stack is confirmed healthy. If the container recreate fails, the new values are already committed — fix the cause and run ./pithead apply to retry the recreate, or restore the two copies and run ./pithead apply to roll back.

The dashboard's .onion address and client-auth key have their own command, ./pithead rotate-dashboard-onion — see Configuration › Remote access over Tor.

Troubleshooting

The dashboard is stuck on Sync Mode. A chain is still downloading. Confirm steady progress:

./pithead logs monerod
./pithead logs tari

If a node is stalled (no new blocks over a long period), restart it with ./pithead restart. To skip the wait, point the stack at an existing synced blockchain or a remote node. See Configuration › Reusing an existing node.

Tari shows high memory use. Most of it is reclaimable disk cache, not a leak. Tari runs under an auto-sized memory limit (tari.mem_limit) that caps a runaway from affecting the rest of the stack. Change it only if Tari restarts repeatedly (raise it) or you need RAM for other apps (lower it), then run ./pithead apply. This covers the bundled node only: with tari.mode: remote no Tari container runs here, the limit is never applied, and that node's memory is the business of the host running it.

Browser warns "your connection is not private." With dashboard.secure: true, DIY installs use Caddy's local certificate authority. Verify and trust its root CA to avoid repeated warnings after certificate renewal. To use plain HTTP, set dashboard.secure: false and run ./pithead apply.

Workers don't show up in the dashboard. Check that each rig points at YOUR_STACK_IP:3333 and that port 3333 is reachable from the worker (firewall on the stack host?). See Connecting Miners.

Hashrate reads zero or the chart is blank. Stats take about a minute to populate after a worker connects. Confirm the worker is hashing (./pithead logs xmrig-proxy).

P2Pool can't connect to a remote Monero node. The node must be set up for mining: ZMQ publishing enabled (zmq-pub) and its RPC reachable by P2Pool. Public "open node" endpoints don't qualify; use a node you run and control. It also has to sit in a private range — the Tor-egress firewall drops the dial otherwise. See Configuration › Connecting to a remote Monero node.

Tari isn't merge-mining against a remote node. The base node must publish its gRPC where this stack can reach it: a stock Minotari node binds 18142 to localhost only. If that node is another Pithead stack, the switch is tari.grpc_lan_access: true on it. Check ./pithead logs p2pool for the merge-mine bridge line — with Tor on, p2pool bridges the gRPC through loopback socat to keep it off the SOCKS proxy, so a failure shows up there rather than as a Tor error. Same private-range requirement as Monero. See Configuration › Remote Tari node.

HugePages shows as disabled / low. Persistent HugePages require a GRUB change and a reboot. Re-run ./pithead setup (without --skip-optimize). Setup writes its own /etc/default/grub.d/zz-pithead-hugepages.cfg, preserves user drop-ins and console arguments, and verifies the generated kernel entries. If verification fails, check later GRUB drop-ins and the generated /boot/grub/grub.cfg, then re-run setup. Reboot only after verification succeeds; confirm HugePages_Total in /proc/meminfo afterwards. An unchanged setup with those flags already in the running kernel does not request another reboot. Setup automatically repairs its managed drop-in and recognizable legacy Pithead reservations. Other reservations, including differently ordered legacy flags or a plural THP typo, require interactive confirmation; setup preserves the main defaults rather than rewriting the typo there.

Tor egress broken while mining works. Healthchecks reports the host down and Telegram commands go quiet, yet workers keep hashing. Tor can bootstrap to 100% and then sit on a failing guard: new circuits time out, so every feature that exits Tor to the clearnet — Healthchecks pings, the Telegram bot, XvB stats — breaks at once, while mining rides established onion circuits and keeps working. ./pithead doctor confirms it: the Tor clearnet-egress check WARNs while everything else reads healthy. Fix:

./pithead restart tor

The restart rebuilds circuits through the retained guards; all Tor circuits drop and rebuild. p2pool re-peers on its own within minutes; monerod does not — it keeps its dead SOCKS connections and can sit at 0 peers for hours while its healthcheck stays green (#972) — so a local monerod is restarted right after tor is healthy again and re-dials in about a minute. Re-run ./pithead doctor to confirm egress recovered. To enable bounded automatic recovery, set tor.auto_heal: true in config.json and run ./pithead apply. Its circuit refreshes and state recovery go through the host control runner, and a refresh must be confirmed before the final restart, so the heal also needs dashboard.control.enabled: true (the appliance sets it only when a dashboard login exists); without it the heal only probes and logs warnings, and sends a recovery note when egress returns. Doctor reports this configuration as INFO and names the login and control prerequisite or manual Tor restart. The dashboard probes every five minutes with a new SOCKS circuit per request. A failed request is corroborated against a second target before it counts toward the 15-minute outage window. The host control runner permits NEWNYM at most twice per 24 hours, 30 minutes apart; continued failure after accepted refreshes permits one final action. With saturated circuit history, the dashboard submits tor-recover; the host runs the existing recovery gates, including the persistent six-hour cooldown, and backs up state before resetting guards. A refused or failed recovery is logged and alerted without a fallback restart. Otherwise it performs one Tor container restart, which also re-dials a running local Monero once Tor's start is confirmed. A failed or timed-out stop/start response is an uncertain mutation: the restart attempt and cooldown remain spent, including when both responses are unconfirmed. The stop request waits up to two minutes (15 seconds of grace, then the kill), and the dashboard issues the start even when the stop times out, because an API stop is not undone by the restart policy. An unconfirmed stop is given up to 30 seconds to settle before the start, so a start cannot be answered "already running" by a Tor that is still going down, and an unconfirmed start is retried up to three times, five seconds apart (#3032). A confirmed start after an unconfirmed stop still re-dials a running local Monero; a stopped or remote node stays untouched. Logs record each control result and retain the distinction between a confirmed restart, a confirmed start with an unconfirmed stop, and an unconfirmed restart in the recovery note. A rejected NEWNYM does not advance to that restart: the dashboard warns and retries after 30 minutes while the outage persists. Once the host's daily refresh budget is spent, it cannot take another automatic action until the host accepts a request. Each step and its probe evidence is logged. Two consecutive successful probes confirm recovery and carry the targets, circuits, duration and preceding action into the Telegram note; the action is not credited as the cause of recovery. On the first heal round, 15 minutes into sustained failure, the monitor asks the host for a read-only reading of Tor's circuit-history state, then refreshes it on later rounds. A saturated history (abandoned count and total build times both at 1000, no CircuitBuildTimeBin) is logged on each such round, sent once per outage to the alert sinks (failed delivery retries on later rounds), and names ./pithead tor-recover; ./pithead doctor reports it as a FAIL, also shown in the Tor section of Service diagnostics after a health check. The doctor check does not require an outage record or auto-heal: saturated history plus live egress failure is enough for a FAIL. The action budget is three per outage; after that the monitor warns until egress recovers. A Tor start the monitor did not cause (an operator, ./pithead or Docker restart) opens a fresh outage window and resets the clock, budget and cooldown, so a Tor that restarts more often than every 15 minutes never reaches an action or the warning. DIY installs stay opt-in; appliances enable tor.auto_heal when the key is absent. A confirmed state reset sends one alert per outage explaining the saturated history and guard reset.

Saturated Tor circuit history while chains stop advancing or egress stays down. A completed bootstrap, a failed clearnet probe, or unavailable chain RPC alone cannot authorize a state reset. If Tor repeatedly reports invalid circuit build timing, run ./pithead tor-recover check. A refused precondition prints its warning and exits 1 without the unexpected-abort diagnostic; no recovery is applied. This read-only check validates the live Tor data mount and the saturated history signature. When local Monero RPC answers, it requires peerless, stalled Monero across three minutes (0 outgoing peers at one height, whatever synchronized says: monerod keeps its last value after losing every peer), with real outgoing peer counts read through the authenticated in-container admin helper. When RPC is unavailable, it instead requires two cookie-authenticated Tor observations three minutes apart: bootstrap 95% at circuit_create, no established circuit, and the same running Tor instance. At least two invalid circuit-timing warnings must appear in the last 200 log lines from that interval; unreadable diagnostics refuse recovery. A third class covers a synchronized Monero behind dead egress: two failed heal rounds from the same sustained outage, recorded by the host at least 15 minutes apart, plus a live check that both clearnet targets still fail through Tor's SOCKS. The healer requests these readings from the first round after 15 minutes of corroborated failure. Recovery clears the outage evidence; the persistent 24-hour NEWNYM budget remains separate. After the automatic action budget is spent, read-only observations continue every 30 minutes. The host retains the last two spaced rounds. Evidence older than one hour since the last observation is refused. A Tor whose egress answers is refused. It uses sudo for read-only access to Tor-owned state and identity keys.

./pithead tor-recover apply rechecks the evidence under the mutation lock, verifies onion identity keys, backs up only Tor's state, and restarts Tor. It re-dials a running local Monero after the actual restart. If Compose left Monero Created behind unhealthy Tor, recovery starts that existing node once Tor is healthy. The command verifies Tor health and Monero peers and records the attempt in the control audit; this connectivity result does not prove chain sync. Restore the remaining baseline services through the caller's normal workflow and independently verify baseline health and authenticated Monero and Tari sync before admitting new work.

The backup remains for inspection. A persistent six-hour cooldown includes failed attempts. The command refuses an ambiguous or symlinked Tor mount, an active mutation, ordinary warnings, or an advancing chain. It never removes onion keys, wallets, configuration or chain data. If verification fails, the command reports failure and leaves the backup for diagnosis; inspect Tor and Monero before retrying. This is an explicit operator operation, not an automatic watchdog. A trusted CI caller must hold its fleet reservation and exclude active consumers through recovery, baseline restoration and independent sync verification. Pithead enforces local mutation exclusion; it does not acquire or attest fleet reservations.

Monero node out of sync after a Tor restart. Anything that restarts or recreates the tor container outside the stack's own operations — a Docker daemon restart, a manual docker restart tor — kills every SOCKS connection, and monerod keeps the dead sockets: 0 in / 0 out peers, get_info reporting synchronized: false, height creeping on occasional lucky dials, every healthcheck green. Detection: the dashboard alerts through the node_down toggle once the node has been out of sync for 10 minutes (only after it was in sync at least once, so initial sync never trips it), and ./pithead doctor (Monero sync check) WARNs on the live state. Fix:

./pithead restart monerod

Stack operations don't need the manual step: compose restarts monerod automatically whenever it restarts or recreates tor (up, apply, upgrade, restart tor), and the final tor.auto_heal container restart does the same. NEWNYM does not restart monerod. (#972)

Local node only. With monero.mode: remote there is no monerod here to restart and doctor's Monero sync check skips, so a stranded node is the remote host's problem to detect and fix.

Monero node isolated or not advancing. A local monerod can keep synchronized: true and a green RPC healthcheck while it has no outgoing peers, or while its tip stops moving because every peer sits on the same stale block (#2499). The XMR Network card shows Node Health, Peers and Height Moved; the tick means the node is at the tip with peers. The published RPC is restricted, and a restricted get_info answers 0 for every peer count, so the counts come from a second, unrestricted RPC that monerod binds to its own container's loopback on an unpublished port (it needs the same login and is not reachable from the host, the LAN or other containers). The container's healthcheck reads it and prints one line that the dashboard reads from docker inspect (State.Health.Log); ./pithead doctor and ./pithead tor-recover read it with docker exec. A reading that is missing, older than 90 seconds, from before the container's current start, with an end before its start, or malformed shows as "peers not visible", never as 0 peers and never as green; the container healthcheck fails until the local reading returns. A restart resets the peerless and height clocks even when it falls between dashboard polls. A stalled height still turns the card red without peer visibility. The container healthcheck also fails after 30 minutes without a rise past its best height, even with outgoing peers; its monotonic height clock resets on a new container run or an unavailable height reading. A syncing local node has no at-tip verdict; a node marked stale by the sync monitor is red. Doctor omits the last-block age when the RPC timestamp is zero, absent, malformed or outside its positive-integer bound. An unavailable tip time does not imply a stale node; peer and initial-sync warnings still apply. The chain verdict turns red, with the numbers, when monerod has had 0 outgoing peers for 10 minutes (NODE_STALE_AFTER_SEC) or its height has not moved for 30 minutes (Monero blocks arrive about every 2). A red local verdict adds a “Monero chain unhealthy” header badge with the reason and recovery advice. It also fails ./pithead doctor (the Monero sync check prints the peer counts and last-block age) and adds a monero chain line to ./pithead status. Each condition sends a node_down-toggle alert and a recovery note. A measured height stall alerts and recovers even while peer counts are unavailable; only the peerless edge needs visible counts. Remote and unreachable nodes have no chain verdict. The container's docker inspect health becomes unhealthy after 600 seconds without outgoing peers (MONERO_HEALTH_PEERLESS_SEC) or 30 minutes without height progress; it still prints its peer counts, so the card keeps reading them. The stack only reports: it restarts nothing. Fix:

./pithead restart monerod

The clocks start when the dashboard first sees the node and again whenever it stops answering, so a dashboard restart or a monerod restart never reads as a stall; a node that does not answer shows no verdict here, because the node-down alert already covers it. With monero.mode: remote the peers of the remote node are not visible to this stack, so the card says so and gives no verdict.

Tari node stuck or forked. A running Tari node can stop following the chain while every healthcheck stays green: the process lives, its gRPC answers and P2Pool's merge-mine channel reads READY, so every merge-mined Tari block is built on a stale tip (#2464). The dashboard judges the node on three signals: its tip unchanged for 30 minutes, 0 peer connections for 10 minutes, and its height more than 50 blocks behind a public explorer fetched through Tor once an hour (tari.explorer_url). One signal turns the Tari status amber with the reason; two, or explorer lag on its own, turn it red, fail ./pithead doctor, add a line to ./pithead status and send an alert. The dashboard only reports: it restarts nothing and P2Pool keeps merge-mining against the node, so Tari work stays wasted until you recover the node. Monero mining is not affected. Recover in this order:

  1. Restart the node. Do not restart it while it is migrating its database after an upgrade (its gRPC answers only once the migration finishes, #2593):

    ./pithead restart tari

    A restart clears the node's list of rejected blocks. That list is what locked out the canonical chain in #2465: the node banned every peer that served it. Catch-up then takes minutes, and the status returns to green once the tip moves again with peers connected and the explorer lag is gone.

  2. Still red after a restart or two: likely a chain fork or an upgrade required. A node on the wrong side of a hard fork rejoins the same dead branch after every restart. Check the Tari release notes for a required upgrade first. A node that followed a dead branch past a fork height has to be rewound below it before it can sync; #2618 covers that rewind for the 350,000 fork.

  3. Resync from scratch as the last resort: stop the node, move its chain data away (tari.data_dir, ./data/tari by default) and start it again. Over Tor this takes days. tari.clearnet_initial_sync: true cuts it to hours at the cost of exposing the host's IP to Tari peers while it runs (see Privacy).

The dashboard data looks broken and you want a clean slate. ./pithead reset-dashboard wipes and recreates the dashboard and P2Pool data. This is destructive: it drops P2Pool sidechain state and dashboard history (blockchains and wallets are unaffected). Targets come from DASHBOARD_DATA_DIR and P2POOL_DATA_DIR in the live .env, including absolute paths with spaces. Directory values containing CR or LF are refused before containers stop or data is removed, including encoded line breaks in quoted .env values. Pass -y / --yes to skip the confirmation prompt.

For developers

  • Run the test suites locally (mirrors CI): make test, or individually make test-dashboard, make test-stack, make test-compose, make lint.
  • Dashboard development: see dashboard/README.md for the package layout, local dev setup, and the hermetic test suite.