diff --git a/AGENTS.md b/AGENTS.md index 6cd396d5..a77dd5c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,7 @@ So "done" always includes a documentation reconciliation step. Before considerin 1. **Grep for what you touched** across docs — the feature name, the module/class/function, the dataset, the spec, the notebook. The fast version: `grep -rn "" --include="*.md" .` (and check notebook markdown cells). Don't rely on memory for where something is mentioned. 2. **Reconcile every hit.** If a doc calls something "planned", "deferred", "not yet wired in", a "seam", or "out of scope" and you just made it real, update that wording. If a doc lists files, notebooks, predictors, specs, or data sources and you added or removed one, fix the list. If you changed a default, a metric, or a command, fix it everywhere it appears. -3. **Update the layered docs together**, not just the nearest one: the use-case README (most detail), the reference-implementations table in the root `README.md`, the method catalog (`aieng-forecasting/aieng/forecasting/methods/README.md`) when you touch a reusable predictor, and `planning-docs/roadmap.md` when something moves from "extension idea" to "shipped". +3. **Update the layered docs together**, not just the nearest one: the use-case README (most detail), the contents table in the root `README.md`, the method catalog (`aieng-forecasting/aieng/forecasting/methods/README.md`) when you touch a reusable predictor, and `planning-docs/roadmap.md` when something moves from "extension idea" to "shipped". Concrete example: integrating Canada's Food Price Report PDFs into the food-price LLM-Process prompt is **not done when the code runs** — it is done when `implementations/food_price_forecasting/README.md` (which currently frames report→prompt wiring as a deferred extension) and the "Reports as predictor context" entry in `planning-docs/roadmap.md` no longer describe it as future work. Shipping the code while those still say "deferred" is the regression the reviewer should catch. @@ -31,6 +31,7 @@ The older planning log, backlog, project charter, and technical-design files und Project shape to keep in mind: - The core library `aieng.forecasting` owns stable infrastructure; reusable predictors live in `aieng.forecasting.methods`; use-case material lives in `implementations//`. +- Strategy guides live under `guides/` (onboard a dataset, create an experiment, customize an agent, audit a result). - YAML specs are co-located under `implementations//specs/`. - Reference implementations: Getting Started, Food Price Forecasting, Energy/Oil (stateless capability track plus an adaptive learning agent), BoC Rate Decisions (quantitative path, cutoff-aware press-release ingestion, and a reasoning-alignment evaluator), and S&P 500 (in active development). - Energy/oil's older information-session notebooks are archived under `playground/energy_case_study/`. @@ -38,7 +39,7 @@ Project shape to keep in mind: ### README files -Search the repo for `README.md` files (excluding `.venv/`) to find every README — there is one at the root, one per package (`aieng-forecasting/`, `implementations/`), the method catalog under `aieng-forecasting/aieng/forecasting/methods/`, and one per use case under `implementations//`. These are the primary user surface and the first thing a new contributor reads; the reconciliation rule above applies to all of them. Keep them accurate and production-quality: describe what the code does and what you can build from it, with no internal program, scheduling, or ownership framing. +Search the repo for `README.md` files (excluding `.venv/`) to find every README — there is one at the root, one per package (`aieng-forecasting/`, `implementations/`), the method catalog under `aieng-forecasting/aieng/forecasting/methods/`, one per use case under `implementations//`, and the strategy-guide index under `guides/`. These are the primary user surface and the first thing a new contributor reads; the reconciliation rule above applies to all of them. Keep them accurate and production-quality: describe what the code does and what you can build from it, with no internal program, scheduling, or ownership framing. --- diff --git a/README.md b/README.md index c2c9164c..325c0a27 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,19 @@ A foundation for building, evaluating, and comparing forecasting systems — conventional numerical models, LLM Processes, and agentic forecasters — on real economic, financial, and event-prediction tasks. -The repository pairs a small, stable core library with a set of self-contained reference implementations. The library gives you cutoff-safe data handling, a single `Predictor` interface, and a backtest/evaluation harness. Each reference implementation is a worked example of a different forecasting problem and the techniques that suit it. Start from whichever one is closest to what you want to build. +## Contents + +The repo has two layers. A small core library (`aieng.forecasting`) owns cutoff-safe data handling, a shared `Predictor` interface, reusable methods, and the backtest/evaluation harness. Self-contained **reference implementations** under [`implementations/`](implementations/) apply those methods to real forecasting problems — pick the one closest to what you want to build; each directory has its own README. + +| # | Implementation | Use case | Methods | +| --- | --- | --- | --- | +| 0 | [Getting started](implementations/getting_started/) | Canada CPI gasoline, one month ahead — the smallest end-to-end loop | Naive last-value, AutoARIMA; CRPS via `backtest()` / `evaluate()` | +| 1 | [S&P 500](implementations/sp500_forecasting/) | Daily index returns under a leak-safe macro/market covariate panel (1 / 5 / 21 business-day horizons) | Naive, ETS, Kalman, AutoARIMA, linear regression, LightGBM; covariate-aware LLM-Process | +| 2 | [Food price forecasting](implementations/food_price_forecasting/) | Multivariate Canadian food CPI in the style of Canada's Food Price Report (nine sub-indices, 12-month trajectory, avg/avg YoY) | Naive last-value, AutoARIMA; report-grounded LLM-Process (quantile grid and sampled trajectory) | +| 3 | [Energy / oil](implementations/energy_oil_forecasting/) | Daily WTI crude under regime-breaking news (continuous trajectory, binary up-shock, scenario analysis) | Prophet, LLM-Process, news-grounded agent, code-executing agent, adaptive (curriculum-trained) agent | +| 4 | [BoC rate decisions](implementations/boc_rate_decisions/) | Will the Bank of Canada cut, hold, or hike at its next meeting? (ordered categorical; binary cut-vs-not special case) | Climatological frequency, multinomial logistic, categorical LLM-Process, analyst agent; LLM-as-judge reasoning alignment | + +Also in this README: [Setup](#setup) · [Core concepts](#core-concepts) · [Repository layout](#repository-layout) · [Documentation](#documentation) > **👉 First time here? Run the environment check.** After `uv sync` (see [Setup](#setup)), open [`implementations/getting_started/00_environment_check.ipynb`](implementations/getting_started/00_environment_check.ipynb) and run it top to bottom. It's a self-guided preflight that verifies every capability — proxy LLM inference, Langfuse, E2B code execution, StatCan/FRED data access, and an end-to-end mini backtest — and tells you exactly what to fix when something isn't set up. **Do this before anything else.** @@ -23,18 +35,11 @@ Every method can be used in one of two modes, and the distinction runs through t ## Reference implementations -Each is independent and self-contained — pick the one that matches the problem you care about, and read that directory's `README.md` for the full walkthrough. They are numbered in a recommended order that mirrors the bootcamp progression — conventional numerical methods → LLM Processes → agents → agentic evaluation — but any one stands on its own, so jump straight to the problem you care about. - -**Start here → #0 [`getting_started/`](implementations/getting_started/)** — one CPI series, one month ahead. The smallest end-to-end loop: a `Predictor`, a `BacktestSpec` and `EvalSpec`, naive + AutoARIMA baselines, CRPS scoring. The place to learn the evaluation framework before picking a domain below. Also includes [`99_repo_concierge.ipynb`](implementations/getting_started/99_repo_concierge.ipynb) — a lite-model repo guide for “how does this codebase work?” questions (`uv run adk run implementations/getting_started/concierge_agent` from the repo root). +Use cases, methods, and links are in the [contents](#contents) table above. Each implementation is independent — pick the problem you care about and read that directory's `README.md` for the full walkthrough. They are numbered in a recommended order that mirrors the bootcamp progression — conventional numerical methods → LLM Processes → agents → agentic evaluation — but any one stands on its own. -| # | Implementation | The problem | Concepts & techniques it demonstrates | -| --- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | [`sp500_forecasting/`](implementations/sp500_forecasting/) | S&P 500 returns under a macro/market covariate panel. | A head-to-head of conventional numerical methods (naive, ETS, Kalman, AutoARIMA, linear regression, LightGBM) plus a covariate-aware LLM-Process, all reading the same leak-safe covariate panel. Cumulative-return targets at 1/5/21-business-day horizons, CRPS + direction metrics, config-driven specs. | -| 2 | [`food_price_forecasting/`](implementations/food_price_forecasting/) | A multivariate food-CPI trajectory, in the style of Canada's Food Price Report. | Nine correlated sub-indices, a 12-step trajectory, a domain metric (avg/avg YoY), baselines vs LLM-Process predictors, leakage-aware backtests, and cached artifacts for fast iteration. | -| 3 | [`energy_oil_forecasting/`](implementations/energy_oil_forecasting/) | Daily WTI crude-oil price under regime-breaking news. | A capability progression — Prophet → LLM-Process → news-grounded agent → code-executing agent — plus an adaptive agent that learns a strategy from data and is scored before vs after. Continuous trajectories, a binary up-shock task, and interactive scenario analysis. | -| 4 | [`boc_rate_decisions/`](implementations/boc_rate_decisions/) | Will the Bank of Canada cut, hold, or hike at its next meeting? | Discrete-event forecasting: ordered-categorical outcomes on an irregular calendar, RPS scoring and one-vs-rest calibration (instead of CRPS), a binary (Brier) special case, cutoff-aware document ingestion, and an LLM-as-judge that scores an agent's reasoning against the official rationale. | +**Start here → #0 [`getting_started/`](implementations/getting_started/)** if the evaluation loop is new to you. That directory also includes [`99_repo_concierge.ipynb`](implementations/getting_started/99_repo_concierge.ipynb) — a lite-model repo guide for “how does this codebase work?” questions (`uv run adk run implementations/getting_started/concierge_agent` from the repo root). -**Not sure where to start building?** Each of the four domain implementations above ends with a `99_starter_agent.ipynb` — a fresh, hackable **starter agent** (a `starter_agent/` module) with toggleable news search and code execution, two lightweight tool-usage skills, an interactive cell, and one scored forecast. It's the consistent "continue from here" entry point for taking any reference use case in an agentic direction, and a quick end-to-end test of that use case's agent stack. +**Not sure where to start building?** Each of the four domain implementations (#1–#4) ends with a `99_starter_agent.ipynb` — a fresh, hackable **starter agent** (a `starter_agent/` module) with toggleable news search and code execution, two lightweight tool-usage skills, an interactive cell, and one scored forecast. It's the consistent "continue from here" entry point for taking any reference use case in an agentic direction, and a quick end-to-end test of that use case's agent stack. ## Time Series Data sources @@ -67,6 +72,7 @@ On Coder workspaces, bootcamp keys (`OPENAI_*`, `E2B_*`, `LANGFUSE_*`) live in y ```text aieng-forecasting/ # Installable library: import as aieng.forecasting implementations/ # Self-contained reference implementations + co-located specs +guides/ # Step-by-step strategy guides for common build-phase tasks scripts/ # Data-fetch scripts + E2B template builder tests/ # Onboarding integration tests (not run in CI) planning-docs/ # Architecture notes and the extension/roadmap catalog @@ -197,6 +203,8 @@ uv run pre-commit run --all-files ## Documentation - Per-implementation READMEs under [`implementations/`](implementations/) — the primary user surface. +- [`guides/`](guides/) — self-contained, step-by-step strategy guides for the most common build-phase tasks: onboarding a dataset, creating an experiment, customizing an agent's strategy, and auditing a result before you believe it. +- [Architecture atlas](https://vectorinstitute.github.io/agentic-forecasting/architecture-atlas.html) ([source](docs/architecture-atlas.html)) — a self-contained visual atlas of the system architecture: the loop, the temporal fence, predictor families, the harness, agent anatomy, and how each reference implementation instantiates them. - [`aieng-forecasting/README.md`](aieng-forecasting/README.md) and [`aieng-forecasting/aieng/forecasting/methods/README.md`](aieng-forecasting/aieng/forecasting/methods/README.md) — the library and the method catalog. - [`planning-docs/roadmap.md`](planning-docs/roadmap.md) — architecture principles and extension ideas. diff --git a/docs/.nojekyll b/docs/.nojekyll new file mode 100644 index 00000000..e69de29b diff --git a/docs/architecture-atlas.html b/docs/architecture-atlas.html new file mode 100644 index 00000000..b34abb71 --- /dev/null +++ b/docs/architecture-atlas.html @@ -0,0 +1,858 @@ + + + +Agentic Forecasting Atlas + + + + +
+
+
Vector Institute · agentic-forecasting · bootcamp atlas
+

Agentic Forecasting Atlas

+

The system architecture of the bootcamp repo, read as one continuous story: how a series and its context get fenced behind a cutoff, how predictors — from naive baselines to tool-wielding agents — consume them, how the harness scores the results and judges the reasoning, and how five reference implementations put the same machinery to work.

+
+ data layer + predictors & agents + evaluation harness + temporal fence +
+
+
+ + + + + + + + as_of + the information cutoff + history the predictor may see + + + + + + + + + + + context — news · documents — fenced by the same cutoff + quantile fan · scored later by CRPS + + a rationale you can judge +
+ +
+ +
+ +
+
+
01System architecture
+

One loop, two layers

+
+

Every experiment in this repo — a two-line naive baseline on gasoline CPI, a curriculum-trained oil analyst — walks the same loop. Data enters through adapters, one fetch() per source; there is nothing special about the built-in StatCan, FRED, and yfinance adapters, and nothing stops you from wrapping a CSV, an API, or a database the same way. Registered on a DataService, the data is fenced behind a cutoff, handed to predictors as one fresh context per forecast origin, and comes out the other side as scored, cached, comparable results.

+

The two layers matter: the core library aieng.forecasting owns everything drawn solid below. You author the dashed pieces — a dataset, a spec, a predictor lineup — and the loop does the rest.

+
+ +
+ + + + + + + + + Data sources + StatCan · FRED + yfinance + + your source + CSV · API · DB · files + + + fetch + + + Adapters + one fetch() per source + timestamp · value · + released_at + cached once under data/ + + + register + + + DataService + series_id → frame + metadata + + CutoffEnforcer + released_at ≤ as_of + + + context(as_of) + + + ForecastContext + get_series(id) + get_documents(src) + everything a predictor sees, + scoped to one origin date + + + + one fresh context per origin + + + + Spec — specs/*.yaml + task · target_series_id · horizons + window · stride · warmup · spec_id + + + Predictor lineup — in code + baselines · Darts · LLM-Process · agents + one shared interface (§03) + + + + origins + predict + + + backtest() / evaluate() + for each origin: + 1 build the cutoff-scoped context + 2 predictor.predict(task, context) + 3 resolve outcomes from later data + 4 score — CRPS · Brier · RPS + retries per origin · skips, never fabricates + + + + + BacktestResult + predictions · scores · mean_score + + + data/predictions/<spec_id>/…yaml + cached — crash-safe, instant re-runs + + + leaderboard · analysis · traces + mean ± SE · per-horizon · rationales + +
Dashed boxes are authored per use case; solid boxes ship in the core library. The red enforcer is the subject of the next section.
+
+
aieng-forecasting/aieng/forecasting/{data, evaluation, methods} · guides/01–02 walk this loop end to end
+
That red box in the middle is the whole reason the loop can be trusted → 02 The temporal fence
+
+
+ +
+
+
02The load-bearing idea
+

The temporal fence

+
+

A backtest is only honest if every forecast is made from what was knowable at its origin — and "knowable" is not the same as "existed". A market close exists the moment the bell rings but publishes the next morning; official statistics trail their reference month by weeks. The repo encodes this as data, not discipline: every row carries a released_at stamp alongside its timestamp (absent the stamp, the timestamp stands in), and predictors can only see through the fence it defines. The same fence covers context — news snapshots and documents are cutoff-scoped exactly like series rows.

+
+
+
+ + + as_of + ← visible in ForecastContext + the future → + + + spot close + released_at = t + + + + + + market close + released_at = t+1 bday + + + + + exists, but publishes tomorrow → blocked + + + official stats + released_at = t+21 days + + + + + publication lag → blocked + +
Filled = visible through context(as_of). Hollow red = recorded but not yet released — the stamp, not the timestamp, decides.
+
+
+
+

How it's enforced

+
    +
  • Predictors receive a ForecastContext, never the raw DataService — through that path, series leakage is structurally impossible.
    • +
  • Every series read routes through the CutoffEnforcer — keep rows where released_at ≤ as_of, falling back to timestamp; document reads are filtered by publication date the same way.
    • +
  • Honest released_at stamps are your responsibility when onboarding data — the one leak the library can't catch for you.
    • +
+
+
+

The exception: agents leak

+

An agent can leak through its tools, not the database — a web search, or any tool that trusts a model-supplied cutoff. The harness pushes back — a seeded cutoff plus an independent verifier (§05) — but the filter sometimes fails. Agent backtests are optimistic by default; audit traces.

+

More tool freedom, less information control — that's structural. It pushes evaluation toward fenced, pre-cached context (less agency) or live forecasting, where nothing exists to leak.

+
+
+
+
data/cutoff.py · data/context.py · data/service.py · guide 01 — choosing released_at honestly
+
With the data fenced, the question becomes who consumes it → 03 The predictors
+
+
+ +
+
+
03Methods
+

One interface, four families

+
+

Everything that forecasts implements the same two-member contract: a predictor_id and a predict(task, context) that returns one prediction per horizon. That is the whole trick that makes the leaderboard possible — a naive baseline, a gradient-boosted model, a single structured LLM call, and a multi-turn agent all answer the same question from the same fenced context, so their scores mean the same thing.

+

The four families below are less a taxonomy than an escalation of how much machinery sits between the context and the answer. Keep the left two honest and strong: an agent is only as interesting as its margin over the best baseline you can field.

+
+ +
+
Predictor
+ predictor_id: str  ·  predict(task, context) → list[Prediction] +
+
+
+

Baselines

+
LastValueHistoricalFrequencyCategoricalFrequency
+

The calibration floor. Anything that loses to naive is a finding.

+
+
+

Numerical

+
AutoARIMA · ETS · KalmanLinearRegression · LightGBMProphet (impl-local)
+

Darts-backed (Prophet aside), covariate-aware, probabilistic via sampling.

+
+
+

LLM Processes

+
QuantileGridSampledTrajectoryBinary / Categorical prob.
+

One structured LLM call — series + metadata in, distribution out. No tools.

+
+
+

Agentic

+
AgentPredictor= AgentConfig + prompt  builder + output schema
+

A multi-turn ADK agent with tools, wrapped to honor the same contract (§05).

+
+
+
+ The task's payload type fixes the metric: + continuous → CRPS + binary → Brier + categorical → RPS + — lower is better, everywhere. +
+
evaluation/predictor.py · methods/{baselines, numerical, llm_processes, agentic} · LastValuePredictor is the annotated reference implementation
+
A shared interface makes methods comparable; the harness makes the comparison fair → 04 The harness
+
+
+ +
+
+
04Evaluation harness
+

Backtest to develop, evaluate to commit

+
+

Specs are experiment design, kept in YAML and out of code: a task, a window of forecast origins, a stride, a warmup — or, for irregular calendars like BoC's meeting dates, an explicit origin_dates list. Because the lineup and the spec are independent, the same predictors run against a two-origin smoke spec, a full development backtest, and a protected evaluation — without edits.

+

The two loops answer different questions. backtest() is the open loop you run freely while developing and tuning. evaluate() is for the answer you commit to — and tuning against it destroys the answer, which is why it is budgeted, tracked, and never cached.

+
+
+
+ + + 2025 + 2026 + + + warmup + ≥ 250 rows + + + development backtest + open loop · cached · run freely + + + + + + + + + + origins — every stride-th step + + + + + + h = 5 · 10 · 21 resolve + against later observations + + disjoint + + + protected eval + held-out · budgeted · uncached + + + max_runs + end trails the data + by ≥ max(horizons) + +
Every origin: at least warmup rows behind it, all horizons resolvable ahead of it — or it is silently skipped.
+
+
+
+ + + + + + + + +
backtest()evaluate()
purposedevelop & tunethe committed answer
runsunlimitedmax_runs budget, tracked on disk
cachingcached & resumablenever — spend stays visible
windowhistorical dev windowheld-out, often post-LLM-cutoff
+
+
+

Two protection styles in the repo

+

Enforced — EvalSpec + max_runs (getting_started, S&P 500, BoC). Convention — energy's notebook 06: disjoint 2026 window, run-guards defaulting to committed artifacts, checksummed agent state.

+
+
+
+
evaluation/{backtest, eval, artifacts}.py · implementations/*/specs/*.yaml · the results cache is keyed by ids, never by spec contents — bump spec_id when the window changes
+
The harness treats an agent like any other predictor. Inside, it is anything but → 05 The agent
+
+
+ +
+
+
05The agent stack
+

Anatomy of an analyst agent

+
+

Identity (what the agent is) and role (its job in one experiment) are deliberately separate objects. The identity — persona, toolbelt, skills, model — is an AgentConfig; the role — what payload it sees per origin and what structured forecast it must return — lives on the AgentPredictor that wraps it. One identity can play many tasks, and rival strategies can play the same task fairly.

+

The capabilities column is where the fence from §02 gets stress-tested: the search tool runs a grounded sub-agent whose brief passes through an independent verifier before the analyst ever sees it. It reduces leakage; it does not eliminate it.

+
+ +
+
+

Identity — AgentConfig

+
namekeys the predictor_id
+
instructionpersona + supplements
+
modelLITE / ADVANCED
+
context_retrievalsearch sub-agent
+
code_executionE2B sandbox
+
function_toolse.g. run_forecast
+
extra_toolse.g. skill-mutation tools
+
skills_dirsSKILL.md playbooks
+

Composed from a toolbelt in the energy starter agent — one ToolSpec per capability, folded onto the config; the other starters set the same fields directly. Adding a tool is one line.

+
+
+

Capabilities — build_adk_agent → tools

+
+ + + + + + + search_web + query, cutoff + + + grounded search + sub-agent · own + instruction + + brief + + verifier LLM + independent model, + strips post-cutoff + + ≤ 3 attempts + + + filtered brief + + [SEARCH_VERIFICATION + _FAILED] + +
+
run_code — E2B cloud sandbox. Fresh sandbox per call: prompt for self-contained scripts, batch-job style.
+
run_forecast — AutoARIMA behind a fixed, auditable interface. Statistics without code generation.
+
SkillToolset — SKILL.md playbooks, listed up front, loaded on demand. The adaptive agent's learned strategy is a mutable skill, edited through dedicated extra_tools.
+
+
+

Role — AgentPredictor

+
prompt_buildertask + context → payload
+
output_schemastructured forecast → Prediction
+

The payload carries as_of, horizons, quantile levels, and compressed history — wrap the builder to inject anything else: pre-cached news, engineered features, prior forecasts.

+
+
__as_of__ — the harness seeds the session cutoff, overriding whatever the model passes to search. Necessary, not sufficient: verification can fail, so agent backtests stay optimistic (§02).
+
Langfuse — with tracing configured, every prediction carries its trace URL. Read one full trace before trusting a score; guide 04 turns that habit into a method.
+
+
+
+
methods/agentic/{agent_factory, predictor}.py · analyst_agent (energy, BoC) · starter_agent (#1–#4) · guide 03 — every customization lever, with a worked change for each
+
That is the architecture in full. Here is how five real problems put it to work → 06 The implementations
+
+
+ +
+
+
06Reference implementations
+

Five instantiations of one architecture

+
+

Same loop, same interface, same fence — different data, payloads, and machinery switched on. The matrix reads horizontally as "what does this implementation exercise" and vertically as "where do I find a working example of this component".

+
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#  ImplementationTarget · payloadCovariatesLLM-ProcessAgent predictorsCode execAdaptive skillsBudgeted evalDocs / reportsLLM judgeIrregular origins
0  getting_started
the smallest end-to-end loop		CA gasoline CPI, 1 mo
continuous		—————●———
1  sp500_forecasting
leak-safe covariate discipline		daily log returns, 1/5/21 bd
continuous		●●○○—●———
2  food_price
report-grounded prompting		food CPI: aggregate + 8 sub-indices, 6–17 mo
continuous ×9		—●○○——●——
3  energy_oil
the agentic staircase		daily WTI, 5/10/21 bd
continuous + binary		●●●●●conv.●——
4  boc_rate_decisions
judged categorical reasoning		cut / hold / hike per meeting
categorical		●●●○—●●●●
+
+

+ ● built into the curriculum  ·  ○ available as an option or starter-agent toggle  ·  — not used  ·  conv. protected by convention, not max_runs (notebook 06). + Docs/reports: food = CFPR reports via DocumentStore; energy = pre-cached news snapshots (the adaptive curriculum); BoC's press releases serve as the judge's reference text. #1–#4 each end with a hackable 99_starter_agent; #0 adds a repo-concierge Q&A agent. +

+

What each one teaches

+
+

The numbering is not arbitrary: it mirrors the bootcamp progression — conventional methods, then LLM processes, then agents, then agentic evaluation. Each step assumes the last and adds one idea.

+
+ +
+ + + + + 0 + getting_started + the loop itself: + backtest → evaluate, CRPS, + naive vs AutoARIMA + + 1 + sp500 + + discipline: + leak-safe covariate panel, + six numerical methods, + budgeted protected eval + + 2 + food_price + + context: + nine targets at once, + expert reports grounding + the LLM-Process prompt + + 3 + energy_oil + + agency: + news → code → tools, + then an adaptive agent that + trains its own strategy skill + + 4 + boc_rate_decisions + + judgment: + categorical decisions on an + irregular meeting calendar, + LLM-judged rationales + + + every step scores against the same harness + +
Each implementation stands alone — start at the problem closest to yours. The staircase is a reading order, not a dependency graph.
+
+
implementations/{getting_started, sp500_forecasting, food_price_forecasting, energy_oil_forecasting, boc_rate_decisions} · each README is the full walkthrough · not sure where to start? 99_starter_agent.ipynb
+
Which brings it back to you → 07 Build your own
+
+
+ +
+
+
07Extending the foundation
+

Build your own forecaster

+
+

The architecture exists so that a new forecaster is always the same three moves — and each move has a step-by-step guide in the repo.

+
+ +
+
+
1 · Onboard data
+

Any source works — a CSV, an API, a database, scraped files. Map it onto timestamp · value · released_at, choose the release stamp honestly, register it on a DataService.

+ guides/01-onboard-a-dataset.md +
+
+
2 · Declare an experiment
+

A task + window in YAML, a lineup in code, cached_multi_backtest, a leaderboard — with strong baselines in every lineup: the naive floor plus the best statistical model you can field. An agent is only as interesting as its margin over them. Hold out your eval window before you tune.

+ guides/02-create-an-experiment.md +
+
+
3 · Shape the agent
+

Persona, toolbelt, search brief, skills — every lever that changes how the agent thinks, each with a worked change, scored against the baselines.

+ guides/03-customize-agent-strategy.md +
+
+
+ The shape of a new forecaster is always the same: implement Predictor, declare a spec, and run backtest() / evaluate() against the baselines — then audit the result before you believe it (guides/04-audit-your-results.md). Everything else in this atlas exists to keep that comparison honest. +
+
+
+ +
+ +
+
+ github: VectorInstitute/agentic-forecasting + fin. +
+
+ + diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 00000000..5c38b0d0 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,6 @@ + + + + +Agentic Forecasting Atlas +

Architecture atlas

diff --git a/guides/01-onboard-a-dataset.md b/guides/01-onboard-a-dataset.md new file mode 100644 index 00000000..ac3239fc --- /dev/null +++ b/guides/01-onboard-a-dataset.md @@ -0,0 +1,199 @@ +# Guide 1 — Onboarding a new time series dataset + +**By the end of this guide** you will have taken a plain CSV file and registered it as a first-class series in the repo's data layer: cutoff-safe, discoverable by id, and usable by every predictor and the backtest harness. Everything here runs offline — no API keys. + +**Prerequisites:** `uv sync --dev` from the repo root. That's it. + +--- + +## The mental model + +Three objects stand between a raw file and an honest backtest: + +1. **An adapter** produces a DataFrame in the canonical schema. The contract is [`BaseAdapter`](../aieng-forecasting/aieng/forecasting/data/adapters/base.py) — one method, `fetch() -> pd.DataFrame`. The repo ships adapters for StatCan, FRED, and yfinance; for your own file you'll use [`StaticFrameAdapter`](../aieng-forecasting/aieng/forecasting/data/features.py), which just wraps a frame you've already prepared. **There is no CSV adapter** — reading the file and shaping the frame is your job, and it's four lines of pandas. +2. **A `DataService`** holds registered series in memory, keyed by a `series_id` string, each with metadata. +3. **A `ForecastContext`** is what predictors actually receive. It is produced by `service.context(as_of=...)` and is *cutoff-scoped*: `context.get_series(series_id)` can only return observations that were knowable at `as_of`. This is the mechanism that keeps backtests honest, and it works automatically once your data is registered correctly. + +The canonical schema is a tidy, three-column frame (default `RangeIndex`, **not** a DatetimeIndex): + +| column | dtype | meaning | +| --- | --- | --- | +| `timestamp` | `datetime64[ns]`, tz-naive | when the observation refers to | +| `value` | `float64` | the observation | +| `released_at` | `datetime64[ns]`, tz-naive | when the observation became publicly knowable | + +`series_id` is *not* a column — it's the key you register under. Rows are sorted ascending by `timestamp`. Everything must be timezone-naive; the cutoff machinery raises if you compare tz-aware and tz-naive stamps. + +--- + +## The sample dataset + +Imagine you have a CSV of daily commodity prices — here, a synthetic "Harbourview harbor diesel spot price" series committed at [`guides/assets/harbourview_diesel_spot.csv`](assets/harbourview_diesel_spot.csv) so you can run every step verbatim: + +```text +date,price_usd +2023-06-01,91.92 +2023-06-02,90.13 +2023-06-05,91.41 +2023-06-06,91.83 +2023-06-07,91.37 +2023-06-08,90.48 +2023-06-09,89.18 +2023-06-12,89.96 +... +``` + +804 business-day rows, 2023-06-01 through 2026-06-30, with one deliberate regime shock in February 2026 (the series is generated, seeded, and clearly synthetic — but shaped like real data: business-day gaps, drift, a shock). Your real CSV will have different column names and quirks; the steps below are exactly the same. + +--- + +## Step 1 — Map your columns onto the canonical schema + +Two decisions happen here, and the second one matters more than it looks. + +**Rename** your columns to `timestamp` and `value`. **Then decide `released_at`**: when did each observation actually become knowable? [`canonical_three_col`](../aieng-forecasting/aieng/forecasting/data/features.py) requires all three columns to be present — it will not invent `released_at` for you, and rows where it is missing get dropped. That's deliberate: it forces you to make the publication-lag call explicitly. + +```python +import pandas as pd + +from aieng.forecasting.data.features import canonical_three_col + +raw = pd.read_csv("guides/assets/harbourview_diesel_spot.csv") +frame = raw.rename(columns={"date": "timestamp", "price_usd": "value"}) + +# A same-day close is knowable at end of day: released_at = timestamp. +frame["released_at"] = frame["timestamp"] + +frame = canonical_three_col(frame) # coerce dtypes, strip tz, sort, drop NaNs +``` + +> **Choosing `released_at` honestly.** `released_at = timestamp` means a forecast origin on date *D* can see *D*'s own close. For market close prices, the repo's yfinance adapter is stricter: it stamps `released_at = timestamp + BDay(1)`, because at the moment a session opens you don't yet know its close. For official statistics the gap is much bigger — StatCan CPI publishes about three weeks after the reference month. If your series has a publication lag and you stamp `released_at = timestamp`, every backtest on it is quietly optimistic. When in doubt, model the lag: `frame["released_at"] = frame["timestamp"] + pd.offsets.BDay(1)` or `+ pd.Timedelta(days=21)`. + +## Step 2 — Register the series + +```python +from aieng.forecasting.data import DataService, SeriesMetadata +from aieng.forecasting.data.features import StaticFrameAdapter + +SERIES_ID = "harbourview_diesel_spot" + +service = DataService() +service.register( + SERIES_ID, + StaticFrameAdapter(frame), + SeriesMetadata( + series_id=SERIES_ID, + description="Harbourview harbor diesel spot price, daily close (synthetic sample data)", + source="local CSV (guides/assets/harbourview_diesel_spot.csv)", + units="USD per barrel", + frequency="B", + ), +) +print(service.summary()) +``` + +Two fields deserve care: + +- **`frequency`** is a pandas offset alias (`"B"` business-daily, `"D"` calendar-daily, `"MS"` month-start). It must match the grid your timestamps actually sit on — the backtest harness generates forecast origins and resolves outcomes by *exact* timestamp arithmetic on this frequency (more on that below). +- **`description` / `source` / `units` are injected verbatim into LLM prompts** by the LLM-process and agent predictors. They are not decorative. "Harbourview harbor diesel spot price, daily close" gives the model something to reason with; "my data" does not. + +## Step 3 — Verify the cutoff discipline + +This is the check that proves your series is wired correctly: + +```python +ctx = service.context(as_of=pd.Timestamp("2025-06-02")) +visible = ctx.get_series(SERIES_ID) +print(len(visible), visible["timestamp"].max()) # 523 rows, last = 2025-06-02 + +full = service.get_series(SERIES_ID, as_of=pd.Timestamp("2026-12-31")) +assert len(full) > len(visible) # the future exists — predictors just can't see it +``` + +A predictor handed `ctx` physically cannot reach observations released after `2025-06-02`. You never write leak-prevention code yourself — you get it by registering data with honest `released_at` stamps. + +## Step 4 — Package it as a module + +The pattern every implementation follows is a `data.py` with a **module-level series-id constant** and a **`build_*_service()` factory** ([energy's version](../implementations/energy_oil_forecasting/data.py) is the model). The constant is the single source of truth shared by your code, your notebooks, and your experiment specs (guide 2): + +```python +# implementations//data.py +"""Data service for the Harbourview diesel sample series.""" + +from pathlib import Path + +import pandas as pd +from aieng.forecasting.data import DataService, SeriesMetadata +from aieng.forecasting.data.features import StaticFrameAdapter, canonical_three_col + +HARBOURVIEW_SERIES_ID = "harbourview_diesel_spot" + +_REPO_ROOT = Path(__file__).resolve().parents[2] +_CSV_PATH = _REPO_ROOT / "guides" / "assets" / "harbourview_diesel_spot.csv" + + +def build_harbourview_service(csv_path: Path | None = None) -> DataService: + """Register the Harbourview diesel series on a fresh DataService.""" + raw = pd.read_csv(csv_path or _CSV_PATH) + frame = raw.rename(columns={"date": "timestamp", "price_usd": "value"}) + frame["released_at"] = frame["timestamp"] + service = DataService() + service.register( + HARBOURVIEW_SERIES_ID, + StaticFrameAdapter(canonical_three_col(frame)), + SeriesMetadata( + series_id=HARBOURVIEW_SERIES_ID, + description="Harbourview harbor diesel spot price, daily close (synthetic sample data)", + source="local CSV (guides/assets/harbourview_diesel_spot.csv)", + units="USD per barrel", + frequency="B", + ), + ) + return service +``` + +Note the path idiom: resolve from `__file__`, not from the current working directory. Notebooks run from their own directories, and CWD-relative paths are the number-one cause of "works in the notebook, breaks in the script" (and of duplicated data caches — the repo's `.gitignore` carries scars from this). + +If your data comes from a live source rather than a fixed file, also write a one-shot fetch script under `scripts/` that warms a local cache — [`scripts/fetch_wti.py`](../scripts/fetch_wti.py) (43 lines) is the smallest template, and [`scripts/fetch_fred.py`](../scripts/fetch_fred.py) shows the multi-series catalogue version. Data files live under the repo-root `data/` directory and are **never committed** (gitignored); fetch once, then everything runs offline. + +--- + +## Verify it worked + +Run the whole thing: + +```bash +uv run python -c " +import pandas as pd +from aieng.forecasting.data import DataService, SeriesMetadata +from aieng.forecasting.data.features import StaticFrameAdapter, canonical_three_col + +raw = pd.read_csv('guides/assets/harbourview_diesel_spot.csv') +frame = raw.rename(columns={'date': 'timestamp', 'price_usd': 'value'}) +frame['released_at'] = frame['timestamp'] +service = DataService() +service.register('harbourview_diesel_spot', StaticFrameAdapter(canonical_three_col(frame)), + SeriesMetadata(series_id='harbourview_diesel_spot', + description='Harbourview diesel spot, daily close (synthetic)', + source='local CSV', units='USD per barrel', frequency='B')) +ctx = service.context(as_of=pd.Timestamp('2025-06-02')) +s = ctx.get_series('harbourview_diesel_spot') +assert s['timestamp'].max() <= pd.Timestamp('2025-06-02') +print('onboarded:', len(s), 'rows visible as of 2025-06-02 — cutoff enforced') +" +``` + +Expected output: `onboarded: 523 rows visible as of 2025-06-02 — cutoff enforced`. + +--- + +## Gotchas that will actually bite you + +- **Timestamps must land exactly on the frequency grid.** The harness computes origins as `pd.date_range(start, end, freq=frequency)` and resolves a horizon-`h` forecast by looking up the row at exactly `as_of + offset * h`. Off-grid stamps (weekend rows in a `"B"` series, mid-month stamps in an `"MS"` series) don't error — origins silently fail to resolve and get skipped. Helper: [`drop_weekend_timestamp_rows`](../aieng-forecasting/aieng/forecasting/data/features.py) for daily series with stray Sat/Sun rows. +- **History length vs. warmup.** Backtest specs declare a `warmup` (minimum visible rows per origin; the energy specs use 250 ≈ one trading year). Origins with less history are silently skipped; if *every* origin is skipped you get `ValueError: No predictions were scored`. Bring at least `warmup + your backtest window + max(horizons)` worth of rows. +- **Gaps are fine at the data layer — handled at the model boundary.** The store makes no regularity guarantee. The Darts-based predictors fill missing dates at conversion time; if you need explicit control, see `business_daily_ffill` and friends in [`features.py`](../aieng-forecasting/aieng/forecasting/data/features.py). +- **Everything tz-naive.** Strip timezones on the way in (`canonical_three_col` does this for you) and pass naive `as_of` stamps. + +## Where to go next + +Your series is now indistinguishable, to the rest of the repo, from WTI or CPI. **[Guide 2](02-create-an-experiment.md)** builds a full experiment on it: a backtest spec, a predictor lineup, and a leaderboard — still with zero API keys. diff --git a/guides/02-create-an-experiment.md b/guides/02-create-an-experiment.md new file mode 100644 index 00000000..7c57c9c4 --- /dev/null +++ b/guides/02-create-an-experiment.md @@ -0,0 +1,171 @@ +# Guide 2 — Creating a new experiment + +**By the end of this guide** you will have a complete, repeatable experiment on the dataset from [guide 1](01-onboard-a-dataset.md): a forecasting task and backtest window declared in YAML, a predictor lineup selected in code, cached backtest runs, and a scored leaderboard — plus a clear picture of how the *protected evaluation* differs from the development backtest. Everything except the optional LLM predictors runs offline. + +**Prerequisites:** guide 1 (or any registered series of your own). + +--- + +## The mental model + +An experiment is four objects, each with one job: + +1. **`ForecastingTask`** — *what* to forecast: target series, horizons, frequency, and payload type (`continuous`, `binary`, or `categorical`). The payload type fixes the metric: continuous → CRPS, binary → Brier, categorical → RPS. You don't choose a metric; you choose a task. +2. **A spec** — *when and how often*: the window of forecast origins, stride, and warmup. Authored as YAML in `implementations//specs/`, loaded into a pydantic model. Specs are experiment design; keeping them out of code means the same predictor lineup can run against a smoke spec, a development backtest, and a protected eval without edits. +3. **`Predictor`s** — *how* to answer. Selected and configured in code. +4. **`backtest()` / `evaluate()`** — the loop: for each origin, build a cutoff-scoped context, call each predictor, resolve outcomes against later data, score. + +## Step 1 — Write the spec + +Create `specs/harbourview_backtest_2025h1.yaml` (anywhere works for a first run; convention is `implementations//specs/`): + +```yaml +spec_id: harbourview_backtest_2025h1 + +description: >- + Development backtest for the Harbourview diesel sample series: 13 + fortnightly origins over the first half of 2025. + +tasks: + - task_id: harbourview_diesel_price_forecast + target_series_id: harbourview_diesel_spot + horizons: [5, 10] + frequency: B + description: >- + Harbourview harbor diesel spot price (USD/bbl, synthetic sample series), + projected 5 and 10 business days ahead. + +start: "2025-01-06" +end: "2025-06-30" +stride: 10 +warmup: 250 +``` + +Reading it like the harness does: + +- **Origins** are `pd.date_range(start, end, freq=frequency)[::stride]` — here, every 10th business day from Jan 6, giving 13 origins. For irregular calendars (e.g. central-bank announcement dates), replace `start`/`end`/`stride` with an explicit `origin_dates:` list — see the [BoC specs](../implementations/boc_rate_decisions/specs/) for the pattern. +- **`warmup: 250`** requires ~one trading year of visible history at each origin, or the origin is skipped. +- **`target_series_id` must exactly match the id you registered** in guide 1. This string is the joint between your data module and your spec — which is why the convention is a shared module-level constant. +- **`end` must trail your data by at least `max(horizons)`** — an origin's 10-day-ahead forecast can only be scored if the outcome exists. The [energy eval spec](../implementations/energy_oil_forecasting/specs/energy_oil_eval.yaml) documents this discipline in its header comment. +- **`spec_id` names the experiment** — it keys the results cache (see the gotchas). One task here; add more entries under `tasks:` and every predictor runs all of them (they must share a frequency). + +Load and sanity-check it: + +```python +import yaml + +from aieng.forecasting.evaluation import MultiTargetBacktestSpec, describe_spec + +with open("specs/harbourview_backtest_2025h1.yaml") as f: + spec = MultiTargetBacktestSpec.model_validate(yaml.safe_load(f)) + +print(describe_spec(spec, service)) # human-readable summary incl. series metadata +print(len(spec.specs()[0].origins())) # 13 +``` + +(Single-task experiments can use plain `BacktestSpec` without `spec_id`/`tasks:` — see [`getting_started/specs/`](../implementations/getting_started/specs/) — but the multi-target form is what the domain implementations use, and it costs nothing.) + +## Step 2 — Select predictors in code + +The convention (from [notebook 04](../implementations/energy_oil_forecasting/04_systematic_backtest_eval.ipynb)) is a small registry with **lazy factories** and an **`enabled` flag** — so an expensive predictor is only constructed when it's actually in the lineup, and turning one off is a one-character edit: + +```python +from dataclasses import dataclass +from typing import Callable + +from aieng.forecasting.methods import DartsAutoARIMAPredictor, LastValuePredictor + + +@dataclass +class PredictorEntry: + name: str + factory: Callable[[], object] + enabled: bool = True + + +REGISTRY = [ + PredictorEntry("Naive (Last Value)", LastValuePredictor), + PredictorEntry("AutoARIMA", DartsAutoARIMAPredictor), + # PredictorEntry("LightGBM", lambda: DartsLightGBMPredictor(lags=21), enabled=False), +] +PREDICTORS = {e.name: e.factory() for e in REGISTRY if e.enabled} +``` + +Always include `LastValuePredictor`: it emits a degenerate zero-spread forecast and is the floor every method must beat — if something loses to it, that's a finding. + +The full off-the-shelf catalogue lives in [`aieng.forecasting.methods`](../aieng-forecasting/aieng/forecasting/methods/README.md): naive/frequency baselines, five Darts numerical predictors, four LLM-process predictors, and the agentic `AgentPredictor`. The LLM and agent predictors slot into this same registry — they just need proxy credentials and money, so leave them out of your first run. + +### Writing your own predictor + +A predictor is a subclass of the two-member ABC — a `predictor_id` property and `predict(task, context) -> list[Prediction]`, one `Prediction` per horizon: + +```python +from aieng.forecasting.evaluation import Prediction, Predictor +``` + +[`LastValuePredictor`](../aieng-forecasting/aieng/forecasting/methods/baselines/naive.py) is the annotated reference implementation — its source is a commented walkthrough of the contract. Two things it demonstrates: predictors read data **only** through `context.get_series(...)` (which is cutoff-scoped — leakage is structurally impossible), and quantile payloads use the shared `STANDARD_QUANTILES` grid. Your predictor does *not* need to live in the core package — energy's `ProphetPredictor` lives in the implementation directory, and yours can live next to your notebook. + +Choose `predictor_id` carefully: it is the leaderboard key **and** the results-cache filename. Two configurations that produce the same id will silently clobber each other's artifacts — when you parameterize a predictor, fold the parameters into the id (the LLM predictors' `variant_tag` and the Darts `_cov` suffix exist for exactly this). + +## Step 3 — Run it + +```python +from aieng.forecasting.evaluation import cached_multi_backtest + +results = {} +for name, predictor in PREDICTORS.items(): + results[name] = cached_multi_backtest(predictor, spec, service) +``` + +Use `cached_multi_backtest` (not raw `multi_backtest`): it writes each task's result to `data/predictions//__.yaml` as it completes, so a crash preserves finished work and a re-run is instant. First run on the sample data: about two seconds. Per-origin failures are retried twice, then that origin is skipped rather than killing the run. + +## Step 4 — Read the leaderboard + +Each entry in `results` maps `task_id -> BacktestResult` — a serializable record carrying the spec, every scored `Prediction`, per-prediction scores, and `mean_score`: + +```python +import pandas as pd + +rows = [ + {"predictor": name, "task": task_id, "metric": r.metric, + "mean_score": round(r.mean_score, 3), "n_scored": len(r.scores), + "skipped_origins": r.skipped_origins} + for name, by_task in results.items() + for task_id, r in by_task.items() +] +print(pd.DataFrame(rows).sort_values("mean_score").to_string(index=False)) +``` + +Expected output on the sample data: + +```text + predictor task metric mean_score n_scored skipped_origins + AutoARIMA harbourview_diesel_price_forecast crps 1.922 26 0 +Naive (Last Value) harbourview_diesel_price_forecast crps 2.598 26 0 +``` + +Check `n_scored` and `skipped_origins` before believing `mean_score`: 13 origins × 2 horizons = 26 means everything resolved. And with a handful of origins, rankings sit inside the noise — energy's [`analysis.py`](../implementations/energy_oil_forecasting/analysis.py) has `leaderboard_with_uncertainty` (mean ± standard error) plus MAE/coverage helpers worth borrowing once you care about the answer rather than the pipeline. + +## Step 5 — Understand the protected eval before you need it + +The development backtest above is an **open loop**: run it as often as you like, tune freely. A protected evaluation answers a different question — *how good is the thing you already committed to?* — and tuning against it destroys the answer. The repo protects eval windows two ways; know both: + +**The enforced way: `EvalSpec` + `evaluate()`.** An eval spec adds `max_runs` (a run budget) to the same fields you wrote above. `evaluate()` / `multi_evaluate()` check the budget against a persistent on-disk counter before running, raise `EvalBudgetExceededError` when it's spent, record which run number produced each result — and **never cache**, precisely so that budget spend stays visible. One `multi_evaluate` call counts as one run regardless of task count. See [`sp500_eval_2026.yaml`](../implementations/sp500_forecasting/specs/sp500_eval_2026.yaml) (`max_runs: 5`) and [`cpi_gasoline_eval_2025.yaml`](../implementations/getting_started/specs/cpi_gasoline_eval_2025.yaml), which carries a runnable snippet in its header. + +**The conventions way: energy's [notebook 06](../implementations/energy_oil_forecasting/06_protected_eval.ipynb).** No `max_runs` — protection comes from discipline: a 2026 eval window fully disjoint from the 2025 development backtest (and past the LLM's training cutoff, so results can't be memorization); predictors selected on 2025 data alone; `RUN_EVAL = False` run-guards that default to loading committed artifacts; and checksums proving the adaptive agent's strategy wasn't mutated between "before" and "after" scoring. + +For your own experiment the minimum viable discipline is: **hold out the most recent slice of your data now** (write the eval spec today, before you start tuning), develop only against the backtest window, and touch the eval spec a small, pre-declared number of times. + +--- + +## Gotchas that will actually bite you + +- **The results cache is keyed by `(spec_id, predictor_id, task_id)` — not by contents.** Edit the spec's window, or the underlying data, while keeping `spec_id`, and `cached_multi_backtest` happily returns stale results. Bump `spec_id` or pass `force_refresh=True`. +- **The cache directory is CWD-relative** (`data/predictions/` by default). A notebook in `implementations/foo/` and a script at the repo root will maintain two separate caches. Pass `store_dir=` explicitly if that matters. +- **A task that fails all retries is *omitted* from the returned dict** with only a warning log. Check `results[name].keys()` when a lineup includes flaky (network-dependent) predictors. +- **Cost scales as predictors × origins × tasks.** Before adding LLM predictors, make a 2-origin smoke spec (copy your spec, shrink the window, suffix the `spec_id` — see [`energy_oil_smoke.yaml`](../implementations/energy_oil_forecasting/specs/energy_oil_smoke.yaml)) and treat its output as a pipeline check, never as evidence. +- **Spec-validation tests are cheap insurance**: a tiny pytest that loads your YAMLs and asserts window/lead invariants — pattern at [`implementations/tests/boc_rate_decisions/test_specs.py`](../implementations/tests/boc_rate_decisions/test_specs.py). + +## Where to go next + +Add an agentic predictor to `REGISTRY` and the same spec scores it against your baselines — that's the whole point of the shared interface. **[Guide 3](03-customize-agent-strategy.md)** shows what an agent is made of and every lever you have to change how it forecasts. diff --git a/guides/03-customize-agent-strategy.md b/guides/03-customize-agent-strategy.md new file mode 100644 index 00000000..7793edfb --- /dev/null +++ b/guides/03-customize-agent-strategy.md @@ -0,0 +1,148 @@ +# Guide 3 — Customizing an analyst agent's strategy + +**By the end of this guide** you will know every lever that changes how an analyst agent behaves — persona, toolbelt, skills, the web-search strategy, output contract, run parameters — and you will have a worked change for each. The point: even the simple starter agent (data + web search) supports *many* genuinely different strategies, and most of them are prompt, skill, or one-line config changes, not new infrastructure. + +**Prerequisites:** a working agent environment (run [`00_environment_check.ipynb`](../implementations/getting_started/00_environment_check.ipynb)). This guide anchors on the energy/WTI agent stack; the sibling implementations share the same template, so everything transfers. + +--- + +## The mental model: identity vs. role + +One object defines what an agent *is*: [`AgentConfig`](../aieng-forecasting/aieng/forecasting/methods/agentic/agent_factory.py) — name, model, system instruction, capabilities (search / code execution / function tools), and skill directories. Two more objects define its *role in an experiment*: a **prompt builder** (serializes the task + cutoff-scoped data into the user payload) and an **output schema** (the structured forecast it must return). [`AgentPredictor`](../aieng-forecasting/aieng/forecasting/methods/agentic/predictor.py) marries the two, and its `predict(task, context)` slots straight into the guide-2 harness. + +That split is deliberate and worth internalizing: **the same identity can play several roles** (energy's notebook 03 runs one agent config against trajectory, binary-shock, and scenario tasks), and **the same role can be played by different identities** (which is how you A/B two strategies fairly). + +Your edit surface is the **starter agent** — [`starter_agent/`](../implementations/energy_oil_forecasting/starter_agent/) plus [`99_starter_agent.ipynb`](../implementations/energy_oil_forecasting/99_starter_agent.ipynb) — a small, hackable template built for exactly this. The **analyst agent** ([`analyst_agent/agent.py`](../implementations/energy_oil_forecasting/analyst_agent/agent.py)) is the finished four-level example to study: `basic` (no tools) → `news` (search) → `code_exec` (search + sandbox + skills) → `tool` (search + a fixed AutoARIMA function tool). + +## The levers + +| Lever | What it changes | Where | +| --- | --- | --- | +| 1. Persona / instruction | How the agent reasons and frames its analysis | `starter_agent/agent.py` → `_build_starter_instruction()` | +| 2. Toolbelt | What the agent *can do* | the `tools=[...]` list in the notebook; `starter_agent/tools.py` | +| 3. Search strategy | What the search sub-agent looks for and returns | `ContextRetrievalConfig.instruction` + the `research-playbook` skill | +| 4. Skills | Playbooks the agent loads on demand | `starter_agent/skills/*/SKILL.md` | +| 5. Run parameters | Model, token budget, temperature | factory kwargs (`model=`, `max_output_tokens=`, …) | +| 6. Prompt builder / output schema | What the agent sees per-origin, and what it must return | wrapper classes (lever 6 below) | + +Worked change for each, below. + +--- + +## Lever 1 — The persona + +Edit [`_build_starter_instruction()`](../implementations/energy_oil_forecasting/starter_agent/agent.py). It is deliberately short — identity and conduct only: + +> *"You are a WTI crude oil market analyst — fluent in supply/demand fundamentals, OPEC+ policy, geopolitical and shipping-lane risk, and price dynamics…"* + +Make it a contrarian who must argue against the consensus before forecasting; a risk manager who reasons in scenarios and widens intervals under ambiguity; a pure technician who ignores narratives. Persona changes are the cheapest strategy changes you have, and they measurably move calibration and interval width. + +Two rules, both learned the hard way in this repo: + +- **Never reference a tool the config doesn't attach.** The analyst agent's instruction is composed as a base plus per-capability *supplements* (`_CONTEXT_RETRIEVAL_SUPPLEMENT`, `_CODE_EXEC_SKILLS_SUPPLEMENT`) appended only by the factories that actually wire the tool. Before that refactor, the no-tool `basic` config told the model to call `search_web` — which it didn't have — producing silent empty turns. If you add prompt text about a capability, gate it on the capability. +- **Don't restate what ADK injects.** The framework already puts every attached tool's and skill's name + description into the system prompt. The starter persona contains no tool mechanics and no output schema for exactly this reason (the schema rides in the *user payload* — see lever 6). + +## Lever 2 — The toolbelt + +The starter agent's capabilities are a plain list you compose in the notebook — each entry is a [`ToolSpec`](../implementations/energy_oil_forecasting/starter_agent/tools.py) bundling a config fragment, an optional playbook skill, and an optional instruction supplement: + +```python +from energy_oil_forecasting.starter_agent import tools, build_starter_agent_config + +toolbelt = [ + tools.news_search(), # cutoff-aware Google Search (proxy-only, no extra key) + tools.arima_forecast(), # AutoARIMA behind a fixed `run_forecast` tool + # tools.code_sandbox(), # E2B Python sandbox (needs E2B_API_KEY, slower) +] +config = build_starter_agent_config(tools=toolbelt) +``` + +Adding or removing a capability is one line, and `build_starter_agent_config` folds each spec into the right `AgentConfig` field. The three shipped tools already span an interesting design axis — *open-ended* (code sandbox: maximum flexibility, minimum auditability) versus *fixed-interface* (`arima_forecast`: the agent can invoke statistics but not write them). Deciding where your agent sits on that axis **is** a strategy decision. + +To add your own tool, write a factory returning a `ToolSpec` — the fold keeps working. Which brings us to the most interesting worked example: + +## Lever 3 — The search strategy (the deep one) + +When the agent calls `search_web`, a bounded **sub-agent** — one grounded-search LLM call with its own system instruction — runs the query and returns a 3–5 paragraph markdown brief plus up to five source URLs. So "how my agent uses web search" decomposes into three independently editable layers: + +1. **What the analyst asks for** — query guidance in the `research-playbook` skill (lever 4) or an instruction supplement. +2. **What the search sub-agent looks for and reports** — `ContextRetrievalConfig.instruction`. *This is the big one*: the analyst never sees raw search results, only this sub-agent's brief. Its instruction currently says "cover price level and trend, OPEC+ supply, geopolitical risk, SPR/policy, analyst targets." Change the brief, change what your agent knows. +3. **Enforcement machinery** — pass `cutoff_date` and an independent verifier model checks the brief for post-cutoff leakage, rewrites or rejects it (returning a `[SEARCH_VERIFICATION_FAILED]` sentinel after 3 failed attempts). The harness overrides the agent-supplied cutoff with the true origin date, so a backtested agent can't leak by "forgetting" the argument. Knobs (verifier model, attempts, confidence threshold, `enforce_cutoff=False` for live forecasting) live on `ContextRetrievalConfig`. + +Worked change — an **inventory-first** search strategy, as a drop-in `ToolSpec` factory (put it next to your notebook or in `tools.py`): + +```python +from aieng.forecasting.methods.agentic.agent_factory import ContextRetrievalConfig +from energy_oil_forecasting.starter_agent.tools import ToolSpec, news_search + +INVENTORY_FIRST_BRIEF = """\ +You are an oil-market intelligence specialist with web search. + +Prioritise, in order: (1) EIA/API inventory data and refinery utilisation, +(2) physical market signals (crack spreads, freight rates, floating storage), +(3) OPEC+ policy, (4) macro demand. Report analyst price targets only if tied +to one of the above. 3-5 paragraphs, every claim grounded in a retrieved +result. When a cutoff date is specified, never report events after it.\ +""" + + +def inventory_first_search() -> ToolSpec: + base = news_search() # inherit the default model + playbook skill + return ToolSpec( + label="inventory_first_search", + context_retrieval=ContextRetrievalConfig( + enabled=True, + instruction=INVENTORY_FIRST_BRIEF, + search_model=base.context_retrieval.search_model, + ), + skill_dir=base.skill_dir, + ) + + +config = build_starter_agent_config(tools=[inventory_first_search()]) +``` + +Same agent, same tools, same cost — but every forecast is now grounded in physical-market evidence instead of headline narrative. Pair this with a matching edit to the playbook skill's query guidance and you have a coherent, testable strategy. (Keep the default's leakage-hygiene language — the "judge recency from substance, not bylines; don't fill gaps from memory" paragraph in [`tools.py`](../implementations/energy_oil_forecasting/starter_agent/tools.py) exists because those failure modes were observed.) + +## Lever 4 — Skills + +Skills are directories with a `SKILL.md` (and optional `references/`) that the agent lists and loads *on demand* — cheap standing knowledge that doesn't bloat the system prompt. The starter agent ships three playbooks; [`research-playbook/SKILL.md`](../implementations/energy_oil_forecasting/starter_agent/skills/research-playbook/SKILL.md) even has a "Domain focus (edit this for your use case)" section that is explicitly yours to rewrite: which signals matter, which queries pay off, which sources to trust. + +Worked change: add a `references/high-signal-queries.md` to the research playbook with five dated example searches that worked, and cite it from the SKILL body. Rules of the road are in [`docs/adk-skills-guide.md`](../docs/adk-skills-guide.md) — the short version: keep `SKILL.md` minimal; if a skill has no scripts, *say so* in the prompt (the analyst's does: "These skills have NO scripts. Do not call `run_skill_script`"); and don't tell the model it can execute code unless code execution is actually enabled. + +For where this lever ends up at full power, study the [adaptive agent](../implementations/energy_oil_forecasting/adaptive_agent/): its entire learned strategy is a *mutable* skill (`wti-strategy` → `wti-strategy-trained`) that the agent itself rewrites under evidence governance — notebooks 05–06. + +## Lever 5 — Run parameters + +`model=` (constants from `aieng.forecasting.models` — `LITE_MODEL` for iteration, `ADVANCED_MODEL` for quality runs; never hardcode model strings), `max_output_tokens`, temperature. One trap: **code execution requires `max_output_tokens=16_384`** — the 4k default can't hold a complete script, and the failure mode is confusing empty-argument retries, not a clean error. `tools.code_sandbox()` sets this for you; remember it if you wire `CodeExecutionConfig` by hand. Also know that `run_code` gets a **fresh sandbox per call** — no variables or files survive between calls, so prompt for self-contained scripts, batch-job style. + +## Lever 6 — Prompt builder and output schema + +What the agent sees at forecast time is produced by a prompt builder — any callable `(*, task, context) -> str`. The starter's [`_StarterForecastPromptBuilder`](../implementations/energy_oil_forecasting/starter_agent/agent.py) demonstrates the **wrapper pattern**: take the stock builder's JSON payload, `json.loads` it, add keys, re-dump. That's the seam for injecting anything you want the agent to condition on — pre-computed statistics, regime labels, cached news briefs — keyed to the origin date. (Whatever you inject, [guide 4](04-audit-your-results.md)'s payload audit is how you confirm the agent actually received it.) + +--- + +## Close the loop: measure it + +A strategy change you haven't scored is a vibe. Both arms drop straight into guide 2's harness: + +```python +from energy_oil_forecasting.starter_agent import build_starter_agent_predictor + +PREDICTORS = { + "starter (default search)": build_starter_agent_predictor( + build_starter_agent_config(tools=[tools.news_search()])), + "starter (inventory-first)": build_starter_agent_predictor( + build_starter_agent_config(tools=[inventory_first_search()])), +} +``` + +Three disciplines: + +- **Smoke first.** Run both against a 2-origin smoke spec before any full window — agent runs cost real money (a single forecast can be a dozen LLM calls once search + verification are counted). +- **Distinguish your variants' `predictor_id`s.** Cached results and the leaderboard are keyed by it, and two starter configs currently produce the *same* id (it embeds only the agent name, model, and modality). Rename one — `config.model_copy(update={"name": "wti_starter_inventory_first"})` — so the ids diverge, or your arms will overwrite each other's artifacts. +- **Read a full trace before trusting a score.** Langfuse tracing is on automatically when configured; every `Prediction` carries its trace URL in `metadata`. Open one end-to-end — payload in, searches, brief, rationale out — and check the agent actually did what your prompt asked. Strategy changes fail silently more often than they fail loudly. + +## Where to go next + +[`99_starter_agent.ipynb`](../implementations/energy_oil_forecasting/99_starter_agent.ipynb) §4 is a six-step "make it yours" ladder that mirrors these levers interactively. Then **[Guide 4](04-audit-your-results.md)** closes the series with the other half of "measure it": auditing the results — payloads, traces, per-origin decomposition, the noise floor — before you believe them. diff --git a/guides/04-audit-your-results.md b/guides/04-audit-your-results.md new file mode 100644 index 00000000..980b3e52 --- /dev/null +++ b/guides/04-audit-your-results.md @@ -0,0 +1,351 @@ +# Guide 4 — Auditing a result before you believe it + +**By the end of this guide** you will have taken a finished backtest — the kind [guide 2](02-create-an-experiment.md) leaves you with — and interrogated it at four altitudes: what went *into* the models, what the model *said*, where the score *comes from*, and whether the ranking survives the *noise floor*. You will end with a claim you can defend in a writeup, and a short list of claims you now know you can't make. Everything runs offline on the guide-1 sample series — no API keys (the one agent-facing step reads artifacts already committed to the repo). + +This is the last guide in the series because it's the last thing you do on a build day — and the first thing anyone reviewing your writeup will do to it. The gap between "my number is lower" and "my method works" is where projects lose their credibility, and it's crossed with about thirty lines of code. + +**Prerequisites:** guide 2's mental model (spec, registry, `cached_multi_backtest`, leaderboard). Guide 3 helps for the trace-audit section but isn't required. + +--- + +## The mental model + +A leaderboard is a claim, not a finding. Four questions stand between the two, ordered by how cheaply they can invalidate everything downstream: + +1. **Did the inputs make sense?** A broken payload — leaked future rows, vacuous context, wrong dates — invalidates the whole run, and it's the only failure you can catch *before* spending money. +2. **Did the model do what you designed?** Agents fail silently far more often than loudly: ignoring a tool, ignoring the context you built, searching for the wrong thing. +3. **Where does the score come from?** A mean over origins and horizons hides regime breaks, decisive horizons, and single origins that carry the ranking. +4. **Could it be noise?** With a few dozen scored points, rankings routinely sit inside the noise — and the honest test is *paired*, not two separate error bars. + +Each audit is a few lines. The expensive thing is remembering to run them before the writeup, not after someone asks. + +## Setup — a year of backtest in one block + +Guide 2 ended with 13 origins on the first half of 2025. The obvious next move on a build day is *more data*: extend the window to a full year, through May 2026 — which, unbeknownst to the leaderboard, spans the sample series' February 2026 regime break. (Real data does this to you too; the sample series just guarantees it.) + +Rebuild the service (guide 1, condensed) and write `specs/harbourview_backtest_1y.yaml`: + +```python +import pandas as pd + +from aieng.forecasting.data import DataService, SeriesMetadata +from aieng.forecasting.data.features import StaticFrameAdapter, canonical_three_col + +SERIES_ID = "harbourview_diesel_spot" + +raw = pd.read_csv("guides/assets/harbourview_diesel_spot.csv") +frame = raw.rename(columns={"date": "timestamp", "price_usd": "value"}) +frame["released_at"] = frame["timestamp"] + +service = DataService() +service.register( + SERIES_ID, + StaticFrameAdapter(canonical_three_col(frame)), + SeriesMetadata( + series_id=SERIES_ID, + description="Harbourview harbor diesel spot price, daily close (synthetic sample data)", + source="local CSV (guides/assets/harbourview_diesel_spot.csv)", + units="USD per barrel", + frequency="B", + ), +) +``` + +```yaml +spec_id: harbourview_backtest_1y + +description: >- + Development backtest for the Harbourview diesel sample series: ~23 + fortnightly origins from July 2025 through May 2026, spanning the + February 2026 regime break. + +tasks: + - task_id: harbourview_diesel_price_forecast + target_series_id: harbourview_diesel_spot + horizons: [5, 10] + frequency: B + description: >- + Harbourview harbor diesel spot price (USD/bbl, synthetic sample series), + projected 5 and 10 business days ahead. + +start: "2025-07-07" +end: "2026-05-29" +stride: 10 +warmup: 250 +``` + +Run guide 2's lineup against it: + +```python +import yaml + +from aieng.forecasting.evaluation import MultiTargetBacktestSpec, cached_multi_backtest +from aieng.forecasting.methods import DartsAutoARIMAPredictor, LastValuePredictor + +with open("specs/harbourview_backtest_1y.yaml") as f: + spec = MultiTargetBacktestSpec.model_validate(yaml.safe_load(f)) + +PREDICTORS = { + "Naive (Last Value)": LastValuePredictor(), + "AutoARIMA": DartsAutoARIMAPredictor(), +} + +results = {name: cached_multi_backtest(p, spec, service) for name, p in PREDICTORS.items()} + +for name, by_task in results.items(): + for task_id, r in by_task.items(): + print(f"{name}: mean {r.metric} {r.mean_score:.3f} over {len(r.scores)} scored points " + f"({r.skipped_origins} origins skipped)") +``` + +```text +Naive (Last Value): mean crps 2.917 over 48 scored points (0 origins skipped) +AutoARIMA: mean crps 2.188 over 48 scored points (0 origins skipped) +``` + +AutoARIMA beats the naive floor by 25%. Ship it? Not yet. (But do read `n_scored` and `skipped_origins` first, exactly as guide 2 taught — 24 origins × 2 horizons = 48 means every origin resolved. A silently shrunken *n* invalidates every audit below.) + +## Audit 1 — read what went in + +The cheapest audit, and the only one that works *before* a paid run. Two layers. + +**What could the predictors see?** One context probe per experiment: + +```python +origins = spec.specs()[0].origins() +ctx = service.context(as_of=origins[0]) +visible = ctx.get_series(SERIES_ID) +print(f"origin {origins[0].date()}: {len(visible)} rows visible, " + f"last timestamp {visible['timestamp'].max().date()}") +``` + +```text +origin 2025-07-07: 548 rows visible, last timestamp 2025-07-07 +``` + +The last visible row lands exactly on the origin — the cutoff discipline from guide 1, verified at the experiment's own first origin. If the last timestamp trails the origin by weeks, your `released_at` stamps (or your frequency grid) are wrong, and every score below is an answer to a different question than you think. + +**What would an LLM arm actually be sent?** LLM and agent predictors don't see the DataFrame — they see a string a prompt builder serialized from it. Print one, and *read it*: + +```python +from energy_oil_forecasting.analyst_agent import WtiPriceForecastPromptBuilder + +task = spec.specs()[0].task +payload = WtiPriceForecastPromptBuilder()(task=task, context=service.context(as_of=origins[0])) +print(payload[:400]) +``` + +```text +{ + "task": "harbourview_diesel_price_forecast", + "as_of": "2025-07-07", + "horizons": [ + 5, + 10 + ], + ... + "target_summary": { + "last_close_usd_bbl": 99.6, + "last_date": "2025-07-07", + ... +``` + +Check three things: `last_date` equals the origin, the history ends there, and nothing in the payload postdates it. And notice what reading buys you that the leaderboard never will: this reused WTI builder labels the field `last_close_usd_bbl` — harmless for a diesel series priced in USD/bbl, but exactly the kind of mislabel (wrong units, wrong series description, empty context field) you only ever catch by looking. **The rule: one payload, read end to end, per arm, before any paid run.** Guide 3 spent a paragraph on this; a broken payload discovered after a 24-origin agent run is money already gone. + +## Audit 2 — read what the model said + +For numerical baselines there is nothing to read. For LLM and agent arms there is — every `Prediction` they emit carries `metadata` with the model's free-text `rationale` and, when Langfuse tracing is configured, a `langfuse_trace_url`. The repo commits real agent artifacts, so you can practice this audit without spending anything: + +```python +from pathlib import Path + +from aieng.forecasting.evaluation import BacktestResult + +artifact = Path( + "implementations/energy_oil_forecasting/data/predictions/energy_oil_eval/" + "agent_predictor_wti_analyst_news_gemini-3.5-flash_continuous__wti_oil_price_forecast.yaml" +) +agent_result = BacktestResult.model_validate(yaml.safe_load(artifact.read_text())) + +seen: set[str] = set() +for pred in agent_result.predictions: + day = str(pred.as_of.date()) + if day in seen: + continue # the rationale repeats across horizons; one per origin is enough + seen.add(day) + meta = pred.metadata or {} + print(f"--- {day} | trace: {meta.get('langfuse_trace_url', '(none)')}") + print(f" {meta.get('rationale', '(no rationale)')[:220]}") + if len(seen) == 2: + break +``` + +```text +--- 2026-02-02 | trace: https://us.cloud.langfuse.com/project/.../traces/bd47122c5ed5... + As of February 2, 2026, the crude oil market balances supportive geopolitical + risk premiums with structural bearishness from projected 2026 global surpluses. ... +--- 2026-02-09 | trace: https://us.cloud.langfuse.com/project/.../traces/f93e2686ff62... + As of February 9, 2026, WTI crude oil is trading near $63.55, supported by a + significant geopolitical risk premium ($4–$10/bbl) ... +``` + +The rationale is the cheap read; the trace is the full record. Open **at least one full trace per method** in Langfuse — payload in, every tool call, the search brief that came back, rationale out — and check, minimally: + +- **Did the agent use what you gave it?** A rationale that cites specifics from the payload and search brief ("trading near $63.55", "$4–$10/bbl risk premium") is doing what you designed. A rationale that would read the same at any origin — generic trend talk, no numbers — means your context is being ignored, and your A/B is comparing decoration, not strategy. +- **Did the tools behave?** Search-enabled agents can return a `[SEARCH_VERIFICATION_FAILED]` sentinel when the cutoff verifier gave up (guide 3, lever 3). An agent forecasting from sentinel briefs all window isn't the strategy you meant to test. +- **Is a null result real?** "News didn't help" and "the news briefs were vacuous" produce identical leaderboards. Only reading the briefs distinguishes them — audit the artifact before you trust the null. + +For a whole lineup at once, [`extract_agent_rationales`](../implementations/energy_oil_forecasting/analysis.py) flattens every agent prediction's rationale and trace link into one DataFrame — the raw material for reading origin by origin. + +## Audit 3 — decompose the mean + +Energy's [`analysis.py`](../implementations/energy_oil_forecasting/analysis.py) ships domain-agnostic helpers for exactly this; borrow them. `predictions_to_frame` explodes results into one tidy row per scored prediction — point, quantiles, actual, CRPS, error, interval width, coverage: + +```python +from energy_oil_forecasting.analysis import ( + leaderboard_with_uncertainty, + per_horizon_crps, + predictions_to_frame, +) + +pf = predictions_to_frame(results, service) +print(per_horizon_crps(pf).round(2)) +``` + +```text + h=5d h=10d All +predictor +AutoARIMA 1.77 2.61 2.19 +Naive (Last Value) 2.33 3.51 2.92 +``` + +The lead holds at both horizons — good. (When it doesn't, you've learned the ranking is decided by one horizon, which changes what you claim.) Now the axis that actually hides things — **origins**: + +```python +by_origin = pf.pivot_table(index="as_of", columns="predictor", values="crps", aggfunc="mean") +print(by_origin.loc["2026-01-01":].round(2).to_string()) # the 2026 slice; drop .loc for all 24 +``` + +```text +predictor AutoARIMA Naive (Last Value) +as_of +2026-01-05 0.87 0.69 +2026-01-19 1.44 2.33 +2026-02-02 5.06 6.50 +2026-02-16 2.09 3.30 +2026-03-02 2.78 4.11 +2026-03-16 4.34 6.21 +2026-03-30 9.09 10.86 +2026-04-13 2.35 3.61 +2026-04-27 1.20 1.40 +2026-05-11 0.96 0.55 +2026-05-25 1.14 1.63 +``` + +Both methods blow up around the February break and the April slide — errors of 5–11 against a typical 1–2. How concentrated is the damage? + +```python +total = pf.groupby("as_of")["crps"].sum().sort_values(ascending=False) +share = total.head(3).sum() / total.sum() +print(f"worst 3 of {len(total)} origins carry {share:.0%} of all CRPS") +``` + +```text +worst 3 of 24 origins carry 35% of all CRPS +``` + +A third of the entire score lives in three fortnights. Split the window at the break and look at *calibration*, not just error: + +```python +import numpy as np + +pf["period"] = np.where(pf["as_of"] < pd.Timestamp("2026-02-01"), "before break", "after break") +split = pf.groupby(["predictor", "period"])[["crps", "abs_error", "inside80", "width80"]].mean() +print(split.round(2)) +``` + +```text + crps abs_error inside80 width80 +predictor period +AutoARIMA after break 3.22 4.26 0.50 5.75 + before break 1.57 2.12 0.77 5.29 +Naive (Last Value) after break 4.24 4.24 0.00 0.00 + before break 2.12 2.12 0.00 0.00 +``` + +This table is the guide's punchline. Before the break, AutoARIMA's 80% interval covers 77% of outcomes — honest. After the break it covers **50%**: the intervals barely widened (5.29 → 5.75) while the errors doubled. "Mean CRPS 2.19" was hiding a model that is *reasonably calibrated in the regime it was fit on and overconfident the moment the regime changes* — which, for a forecasting system someone might act on, is the single most important sentence in your writeup. (Ignore the naive row's coverage: `LastValuePredictor` emits a degenerate zero-width interval by design, so its `inside80` is trivially 0 — compare it on CRPS only.) + +## Audit 4 — the noise floor + +Is the 2.19-vs-2.92 ranking even real? First pass, mean ± standard error: + +```python +print(leaderboard_with_uncertainty(pf).round(3)) +``` + +```text + mean_crps se n family +predictor +AutoARIMA 2.188 0.309 48 Numerical ML +Naive (Last Value) 2.917 0.403 48 Baseline +``` + +The gap (0.73) is about the size of the two SEs combined — suggestive, not conclusive. But those error bars overstate the noise: both methods were scored on the **same origins**, so the volatile fortnights inflate both means together. The honest test is *paired* — difference the two methods point by point, then ask whether the differences are consistently signed: + +```python +paired = pf.pivot_table(index=["as_of", "horizon"], columns="predictor", values="crps") +diff = paired["AutoARIMA"] - paired["Naive (Last Value)"] +win_rate = (diff < 0).mean() +print(f"AutoARIMA wins {win_rate:.0%} of {len(diff)} paired points; " + f"mean diff {diff.mean():+.2f} (negative = AutoARIMA better), SE {diff.sem():.2f}") +``` + +```text +AutoARIMA wins 77% of 48 paired points; mean diff -0.73 (negative = AutoARIMA better), SE 0.12 +``` + +Paired, the picture sharpens dramatically: the mean advantage is six times its standard error, and AutoARIMA wins three points out of four. One more cheap robustness probe — does a single origin decide it? + +```python +worst = pf.groupby("as_of")["crps"].sum().idxmax() +trimmed = pf[pf["as_of"] != worst] +print(f"dropping {worst.date()}:") +print(trimmed.groupby("predictor")["crps"].mean().round(2).to_string()) +``` + +```text +dropping 2026-03-30: +AutoARIMA 1.89 +Naive (Last Value) 2.57 +``` + +Ranking unchanged. Note what just happened: the audit *confirmed* the headline claim. Auditing isn't debunking — it's how you find out which of your claims survive, and earn the right to state the survivors plainly. + +## What you can now write + +Compare the claim you'd have written at the leaderboard stage with the one the audits license: + +> *Before:* "AutoARIMA beat the naive baseline (CRPS 2.19 vs 2.92)." +> +> *After:* "AutoARIMA beat the last-value baseline by 0.73 mean CRPS on 24 fortnightly origins over one year — a paired advantage six times its standard error, winning 77% of scored points, robust to dropping the worst origin, and holding at both horizons. However, both methods degrade sharply at the February 2026 regime break (three origins carry 35% of total CRPS), and AutoARIMA's 80% interval covers only 50% of post-break outcomes against 77% before — the model is overconfident precisely when the regime changes." + +Same experiment, same numbers — but the second paragraph is a finding, states its own evidence, and hands the reader the limitation before they find it themselves. Three disciplines complete it: + +- **Write the qualitative pass before the scores.** Read payloads, rationales, and per-origin tables and write down what you see *before* looking at who won — conclusions formed after seeing the ranking have a way of explaining whatever the ranking says. +- **The audits above are development hygiene — the eval window stays untouched.** Everything here ran on the open-loop development backtest. Your held-out eval spec (guide 2, step 5) gets touched the small, pre-declared number of times you committed to, *after* the audits have settled what you're claiming. +- **Declare what you didn't test.** One year, one series, two methods, no protected eval yet: saying so costs a sentence and buys credibility for everything else. + +--- + +## Gotchas that will actually bite you + +- **You can audit a stale artifact with total confidence.** The results cache is keyed by `(spec_id, predictor_id, task_id)`, not contents — edit the spec's window while keeping its `spec_id` and every audit above will authoritatively describe a run that no longer exists. Bump `spec_id` when the experiment changes (guide 2's first gotcha, now with sharper teeth). +- **Check `n_scored` and `skipped_origins` before any other number.** Warmup skips, unresolvable horizons, and failed retries all shrink *n* silently, and every statistic in this guide degrades quietly as it shrinks. +- **Degenerate baselines poison calibration comparisons.** Zero-width intervals make coverage and width columns meaningless for `LastValuePredictor`; compare it on the proper score (CRPS) only. +- **Coverage on small *n* is noisy.** 67% coverage on 48 points is a weak signal on its own; 77%-vs-50% *split at a known break* is a pattern. Slice coverage by something meaningful before reading it as miscalibration. +- **Smoke runs are pipeline checks, never evidence.** The 2-origin smoke spec exists to catch wiring errors cheaply; nothing computed from it belongs in a writeup (guide 2 said it; it bears repeating next to real statistics). +- **A mean CRPS averages across horizons in target units.** When horizons differ in scale or difficulty, the "All" column is dominated by the hardest one — `per_horizon_crps` exists so you notice. + +## Where to go next + +This is the last guide in the series. Energy's [notebook 04](../implementations/energy_oil_forecasting/04_systematic_backtest_eval.ipynb) is this audit at full scale — the same helpers over a nine-predictor lineup with agent arms — and [notebook 06](../implementations/energy_oil_forecasting/06_protected_eval.ipynb) shows the endgame: a protected eval window, run-budget discipline, and results published with their audits attached. The next artifact you audit should be your own build-phase project. diff --git a/guides/README.md b/guides/README.md new file mode 100644 index 00000000..38dd2bd0 --- /dev/null +++ b/guides/README.md @@ -0,0 +1,18 @@ +# Bootcamp strategy guides + +Step-by-step guides for the tasks you are most likely to take on during the build phase. Each one is self-contained: it states what you will have at the end, walks there in numbered steps with runnable code, and ends with a verification check and extension ideas. Every code snippet in these guides has been run against the repo as written. + +These are **guides, not reference implementations**. The reference implementations under [`implementations/`](../implementations/) show finished forecasting systems; these guides show you the *moves* — how to bring in data, stand up an experiment, and reshape an agent — using only machinery that already exists in the repo. + +| # | Guide | You will end up with | +| --- | --- | --- | +| 1 | [Onboarding a new time series dataset](01-onboard-a-dataset.md) | A CSV of your own, registered as a cutoff-safe series that every predictor and the backtest harness can use | +| 2 | [Creating a new experiment](02-create-an-experiment.md) | A backtest + evaluation setup on that series: a YAML spec, a predictor lineup in code, and a scored leaderboard | +| 3 | [Customizing an analyst agent's strategy](03-customize-agent-strategy.md) | A map of every lever that changes how an agent behaves — persona, toolbelt, skills, search strategy — with a worked change for each | +| 4 | [Auditing a result before you believe it](04-audit-your-results.md) | A finished backtest interrogated at four altitudes — inputs, model behaviour, score decomposition, noise floor — ending in a claim you can defend in a writeup | + +**Recommended order.** Guides 1 and 2 chain: guide 1 onboards a small sample dataset (committed at [`assets/harbourview_diesel_spot.csv`](assets/harbourview_diesel_spot.csv)) and guide 2 runs a full experiment on it. Both run entirely offline — no API keys needed. Guide 3 is a breadth tour of the agent customization surface. Guide 4 closes the series with the discipline every build-phase writeup leans on: auditing a finished result — payloads, traces, per-origin decomposition, the noise floor — before you believe it. It, too, runs entirely offline. + +**Prerequisites.** A working environment (`uv sync --dev` from the repo root — see the [main README](../README.md#setup)). Guides 1, 2, and 4 need nothing else. Guide 3 exercises the agent stack, so run [`00_environment_check.ipynb`](../implementations/getting_started/00_environment_check.ipynb) first if you haven't. + +**Conventions.** Guide 3 anchors on the [energy / WTI implementation](../implementations/energy_oil_forecasting/) — the most complete agent stack in the repo — and guide 4 borrows its analysis helpers and committed artifacts; every pattern they teach transfers to the other implementations, which share the same starter-agent template. Paths in code are shown relative to the repo root; run snippets from the repo root unless a step says otherwise. diff --git a/guides/assets/harbourview_diesel_spot.csv b/guides/assets/harbourview_diesel_spot.csv new file mode 100644 index 00000000..38c9288a --- /dev/null +++ b/guides/assets/harbourview_diesel_spot.csv @@ -0,0 +1,805 @@ +date,price_usd +2023-06-01,91.92 +2023-06-02,90.13 +2023-06-05,91.41 +2023-06-06,91.83 +2023-06-07,91.37 +2023-06-08,90.48 +2023-06-09,89.18 +2023-06-12,89.96 +2023-06-13,90.01 +2023-06-14,89.36 +2023-06-15,87.34 +2023-06-16,87.42 +2023-06-19,85.87 +2023-06-20,85.92 +2023-06-21,85.27 +2023-06-22,84.86 +2023-06-23,85.19 +2023-06-26,87.46 +2023-06-27,87.33 +2023-06-28,87.55 +2023-06-29,85.32 +2023-06-30,84.94 +2023-07-03,85.16 +2023-07-04,85.42 +2023-07-05,85.87 +2023-07-06,85.7 +2023-07-07,86.2 +2023-07-10,85.19 +2023-07-11,85.2 +2023-07-12,85.08 +2023-07-13,85.16 +2023-07-14,85.6 +2023-07-17,85.99 +2023-07-18,86.84 +2023-07-19,88.63 +2023-07-20,87.82 +2023-07-21,86.05 +2023-07-24,89.29 +2023-07-25,87.59 +2023-07-26,86.58 +2023-07-27,87.19 +2023-07-28,88.36 +2023-07-31,89.09 +2023-08-01,89.01 +2023-08-02,89.14 +2023-08-03,89.22 +2023-08-04,90.78 +2023-08-07,90.85 +2023-08-08,89.57 +2023-08-09,89.68 +2023-08-10,88.98 +2023-08-11,90.45 +2023-08-14,91.26 +2023-08-15,90.5 +2023-08-16,91.18 +2023-08-17,91.47 +2023-08-18,94.39 +2023-08-21,94.55 +2023-08-22,95.6 +2023-08-23,95.08 +2023-08-24,93.72 +2023-08-25,92.38 +2023-08-28,91.06 +2023-08-29,91.85 +2023-08-30,91.73 +2023-08-31,91.43 +2023-09-01,91.33 +2023-09-04,91.02 +2023-09-05,89.84 +2023-09-06,90.43 +2023-09-07,90.28 +2023-09-08,89.77 +2023-09-11,90.78 +2023-09-12,90.32 +2023-09-13,91.31 +2023-09-14,91.57 +2023-09-15,92.67 +2023-09-18,92.42 +2023-09-19,92.64 +2023-09-20,91.68 +2023-09-21,89.99 +2023-09-22,90.86 +2023-09-25,90.68 +2023-09-26,90.9 +2023-09-27,92.11 +2023-09-28,89.91 +2023-09-29,90.67 +2023-10-02,90.76 +2023-10-03,90.56 +2023-10-04,89.65 +2023-10-05,89.08 +2023-10-06,90.45 +2023-10-09,91.23 +2023-10-10,91.99 +2023-10-11,92.1 +2023-10-12,91.27 +2023-10-13,91.52 +2023-10-16,93.18 +2023-10-17,93.81 +2023-10-18,95.44 +2023-10-19,95.5 +2023-10-20,96.71 +2023-10-23,95.72 +2023-10-24,94.77 +2023-10-25,94.95 +2023-10-26,95.24 +2023-10-27,97.29 +2023-10-30,97.32 +2023-10-31,98.94 +2023-11-01,99.22 +2023-11-02,96.53 +2023-11-03,96.38 +2023-11-06,95.94 +2023-11-07,96.06 +2023-11-08,95.44 +2023-11-09,96.65 +2023-11-10,95.85 +2023-11-13,96.55 +2023-11-14,95.92 +2023-11-15,95.03 +2023-11-16,93.19 +2023-11-17,93.38 +2023-11-20,91.79 +2023-11-21,91.3 +2023-11-22,92.62 +2023-11-23,92.11 +2023-11-24,92.51 +2023-11-27,92.79 +2023-11-28,92.31 +2023-11-29,91.93 +2023-11-30,91.69 +2023-12-01,92.32 +2023-12-04,92.88 +2023-12-05,90.77 +2023-12-06,89.75 +2023-12-07,88.8 +2023-12-08,87.8 +2023-12-11,90.2 +2023-12-12,92.0 +2023-12-13,91.74 +2023-12-14,92.47 +2023-12-15,91.92 +2023-12-18,92.69 +2023-12-19,92.75 +2023-12-20,92.86 +2023-12-21,94.14 +2023-12-22,94.26 +2023-12-25,95.43 +2023-12-26,96.78 +2023-12-27,97.07 +2023-12-28,97.3 +2023-12-29,97.41 +2024-01-01,98.63 +2024-01-02,99.91 +2024-01-03,100.0 +2024-01-04,102.15 +2024-01-05,102.48 +2024-01-08,102.31 +2024-01-09,101.61 +2024-01-10,101.55 +2024-01-11,102.6 +2024-01-12,101.74 +2024-01-15,103.41 +2024-01-16,102.05 +2024-01-17,101.36 +2024-01-18,101.54 +2024-01-19,100.55 +2024-01-22,102.14 +2024-01-23,100.49 +2024-01-24,99.43 +2024-01-25,98.23 +2024-01-26,95.09 +2024-01-29,95.33 +2024-01-30,96.08 +2024-01-31,95.26 +2024-02-01,95.24 +2024-02-02,95.77 +2024-02-05,97.62 +2024-02-06,98.09 +2024-02-07,96.96 +2024-02-08,95.14 +2024-02-09,95.07 +2024-02-12,94.72 +2024-02-13,93.77 +2024-02-14,92.98 +2024-02-15,92.88 +2024-02-16,91.67 +2024-02-19,91.35 +2024-02-20,91.52 +2024-02-21,91.24 +2024-02-22,92.89 +2024-02-23,92.03 +2024-02-26,92.83 +2024-02-27,92.67 +2024-02-28,92.34 +2024-02-29,92.73 +2024-03-01,93.66 +2024-03-04,93.45 +2024-03-05,94.61 +2024-03-06,95.37 +2024-03-07,97.75 +2024-03-08,99.93 +2024-03-11,98.21 +2024-03-12,97.63 +2024-03-13,96.93 +2024-03-14,97.61 +2024-03-15,97.47 +2024-03-18,100.31 +2024-03-19,101.88 +2024-03-20,100.8 +2024-03-21,100.29 +2024-03-22,100.52 +2024-03-25,100.53 +2024-03-26,101.51 +2024-03-27,102.09 +2024-03-28,100.13 +2024-03-29,99.82 +2024-04-01,101.31 +2024-04-02,103.82 +2024-04-03,104.12 +2024-04-04,103.28 +2024-04-05,102.62 +2024-04-08,104.41 +2024-04-09,102.62 +2024-04-10,101.16 +2024-04-11,102.94 +2024-04-12,103.98 +2024-04-15,103.38 +2024-04-16,102.19 +2024-04-17,104.48 +2024-04-18,103.63 +2024-04-19,104.53 +2024-04-22,105.49 +2024-04-23,104.07 +2024-04-24,105.18 +2024-04-25,103.31 +2024-04-26,104.81 +2024-04-29,104.05 +2024-04-30,102.66 +2024-05-01,101.97 +2024-05-02,101.02 +2024-05-03,102.91 +2024-05-06,103.2 +2024-05-07,103.62 +2024-05-08,104.96 +2024-05-09,104.63 +2024-05-10,103.97 +2024-05-13,103.17 +2024-05-14,101.05 +2024-05-15,100.17 +2024-05-16,100.63 +2024-05-17,100.94 +2024-05-20,102.45 +2024-05-21,104.81 +2024-05-22,104.89 +2024-05-23,104.23 +2024-05-24,104.72 +2024-05-27,104.0 +2024-05-28,105.48 +2024-05-29,104.44 +2024-05-30,104.26 +2024-05-31,102.87 +2024-06-03,103.19 +2024-06-04,105.15 +2024-06-05,104.73 +2024-06-06,102.02 +2024-06-07,103.18 +2024-06-10,104.37 +2024-06-11,106.72 +2024-06-12,106.29 +2024-06-13,108.18 +2024-06-14,107.52 +2024-06-17,107.76 +2024-06-18,107.7 +2024-06-19,107.52 +2024-06-20,107.9 +2024-06-21,108.34 +2024-06-24,108.51 +2024-06-25,107.74 +2024-06-26,106.64 +2024-06-27,109.18 +2024-06-28,105.66 +2024-07-01,106.04 +2024-07-02,106.86 +2024-07-03,105.16 +2024-07-04,104.5 +2024-07-05,104.04 +2024-07-08,103.5 +2024-07-09,102.73 +2024-07-10,103.07 +2024-07-11,101.17 +2024-07-12,101.8 +2024-07-15,100.8 +2024-07-16,102.77 +2024-07-17,102.41 +2024-07-18,103.51 +2024-07-19,103.87 +2024-07-22,105.34 +2024-07-23,105.48 +2024-07-24,105.96 +2024-07-25,105.65 +2024-07-26,104.93 +2024-07-29,103.98 +2024-07-30,101.47 +2024-07-31,102.37 +2024-08-01,103.28 +2024-08-02,104.51 +2024-08-05,104.64 +2024-08-06,105.85 +2024-08-07,104.69 +2024-08-08,103.32 +2024-08-09,102.0 +2024-08-12,103.5 +2024-08-13,104.07 +2024-08-14,104.87 +2024-08-15,105.2 +2024-08-16,105.14 +2024-08-19,106.29 +2024-08-20,107.23 +2024-08-21,107.03 +2024-08-22,108.12 +2024-08-23,108.5 +2024-08-26,108.97 +2024-08-27,108.63 +2024-08-28,106.92 +2024-08-29,106.35 +2024-08-30,106.0 +2024-09-02,103.7 +2024-09-03,104.11 +2024-09-04,102.71 +2024-09-05,101.77 +2024-09-06,102.72 +2024-09-09,106.32 +2024-09-10,106.91 +2024-09-11,105.29 +2024-09-12,103.22 +2024-09-13,103.28 +2024-09-16,103.88 +2024-09-17,105.04 +2024-09-18,106.85 +2024-09-19,107.49 +2024-09-20,105.32 +2024-09-23,103.7 +2024-09-24,105.52 +2024-09-25,104.45 +2024-09-26,105.63 +2024-09-27,104.53 +2024-09-30,104.51 +2024-10-01,103.55 +2024-10-02,103.48 +2024-10-03,101.34 +2024-10-04,103.42 +2024-10-07,104.49 +2024-10-08,102.96 +2024-10-09,102.08 +2024-10-10,101.08 +2024-10-11,100.2 +2024-10-14,100.92 +2024-10-15,100.47 +2024-10-16,100.43 +2024-10-17,100.57 +2024-10-18,100.64 +2024-10-21,101.85 +2024-10-22,102.25 +2024-10-23,102.19 +2024-10-24,100.63 +2024-10-25,102.06 +2024-10-28,104.06 +2024-10-29,102.57 +2024-10-30,101.87 +2024-10-31,101.61 +2024-11-01,102.5 +2024-11-04,102.04 +2024-11-05,102.62 +2024-11-06,104.31 +2024-11-07,105.53 +2024-11-08,103.79 +2024-11-11,102.27 +2024-11-12,102.63 +2024-11-13,103.67 +2024-11-14,102.24 +2024-11-15,101.85 +2024-11-18,102.01 +2024-11-19,101.84 +2024-11-20,102.78 +2024-11-21,102.04 +2024-11-22,101.46 +2024-11-25,100.57 +2024-11-26,100.91 +2024-11-27,101.49 +2024-11-28,100.42 +2024-11-29,100.95 +2024-12-02,100.44 +2024-12-03,101.27 +2024-12-04,102.9 +2024-12-05,102.44 +2024-12-06,103.28 +2024-12-09,105.12 +2024-12-10,106.6 +2024-12-11,107.82 +2024-12-12,108.2 +2024-12-13,108.15 +2024-12-16,107.64 +2024-12-17,106.41 +2024-12-18,106.75 +2024-12-19,106.13 +2024-12-20,104.65 +2024-12-23,105.25 +2024-12-24,105.82 +2024-12-25,106.05 +2024-12-26,104.94 +2024-12-27,103.74 +2024-12-30,103.03 +2024-12-31,105.52 +2025-01-01,104.26 +2025-01-02,103.29 +2025-01-03,102.13 +2025-01-06,102.36 +2025-01-07,102.26 +2025-01-08,103.19 +2025-01-09,102.83 +2025-01-10,101.64 +2025-01-13,102.1 +2025-01-14,102.32 +2025-01-15,102.3 +2025-01-16,102.81 +2025-01-17,103.85 +2025-01-20,105.21 +2025-01-21,104.9 +2025-01-22,103.89 +2025-01-23,102.55 +2025-01-24,101.38 +2025-01-27,101.89 +2025-01-28,101.12 +2025-01-29,102.57 +2025-01-30,102.72 +2025-01-31,102.38 +2025-02-03,103.7 +2025-02-04,102.52 +2025-02-05,103.33 +2025-02-06,101.7 +2025-02-07,103.55 +2025-02-10,102.57 +2025-02-11,100.42 +2025-02-12,99.63 +2025-02-13,98.52 +2025-02-14,99.01 +2025-02-17,101.38 +2025-02-18,103.3 +2025-02-19,101.89 +2025-02-20,101.64 +2025-02-21,102.22 +2025-02-24,102.7 +2025-02-25,104.72 +2025-02-26,105.67 +2025-02-27,103.74 +2025-02-28,103.5 +2025-03-03,103.0 +2025-03-04,103.73 +2025-03-05,102.74 +2025-03-06,102.49 +2025-03-07,102.46 +2025-03-10,103.88 +2025-03-11,103.11 +2025-03-12,102.46 +2025-03-13,102.38 +2025-03-14,102.78 +2025-03-17,103.3 +2025-03-18,105.71 +2025-03-19,105.89 +2025-03-20,105.59 +2025-03-21,107.7 +2025-03-24,108.54 +2025-03-25,110.88 +2025-03-26,109.99 +2025-03-27,110.9 +2025-03-28,111.66 +2025-03-31,114.4 +2025-04-01,115.94 +2025-04-02,116.78 +2025-04-03,118.54 +2025-04-04,116.95 +2025-04-07,116.27 +2025-04-08,115.92 +2025-04-09,114.22 +2025-04-10,113.75 +2025-04-11,113.5 +2025-04-14,113.72 +2025-04-15,115.54 +2025-04-16,114.68 +2025-04-17,115.85 +2025-04-18,116.51 +2025-04-21,114.72 +2025-04-22,115.75 +2025-04-23,116.75 +2025-04-24,114.07 +2025-04-25,116.5 +2025-04-28,118.28 +2025-04-29,117.29 +2025-04-30,117.65 +2025-05-01,116.65 +2025-05-02,114.6 +2025-05-05,115.83 +2025-05-06,115.29 +2025-05-07,115.94 +2025-05-08,115.91 +2025-05-09,116.42 +2025-05-12,116.82 +2025-05-13,114.8 +2025-05-14,113.78 +2025-05-15,114.31 +2025-05-16,113.59 +2025-05-19,112.8 +2025-05-20,111.47 +2025-05-21,110.3 +2025-05-22,109.6 +2025-05-23,110.46 +2025-05-26,108.84 +2025-05-27,111.94 +2025-05-28,111.24 +2025-05-29,110.47 +2025-05-30,109.74 +2025-06-02,109.15 +2025-06-03,108.6 +2025-06-04,108.39 +2025-06-05,105.88 +2025-06-06,103.46 +2025-06-09,102.73 +2025-06-10,101.3 +2025-06-11,102.12 +2025-06-12,102.99 +2025-06-13,102.96 +2025-06-16,103.67 +2025-06-17,103.71 +2025-06-18,102.21 +2025-06-19,101.58 +2025-06-20,101.26 +2025-06-23,100.97 +2025-06-24,100.07 +2025-06-25,101.21 +2025-06-26,100.75 +2025-06-27,101.6 +2025-06-30,102.15 +2025-07-01,100.53 +2025-07-02,99.87 +2025-07-03,97.73 +2025-07-04,97.65 +2025-07-07,99.6 +2025-07-08,98.27 +2025-07-09,99.74 +2025-07-10,98.18 +2025-07-11,97.53 +2025-07-14,97.52 +2025-07-15,97.22 +2025-07-16,95.82 +2025-07-17,96.16 +2025-07-18,95.27 +2025-07-21,94.81 +2025-07-22,94.41 +2025-07-23,94.31 +2025-07-24,94.05 +2025-07-25,93.04 +2025-07-28,93.06 +2025-07-29,92.44 +2025-07-30,92.56 +2025-07-31,92.18 +2025-08-01,91.34 +2025-08-04,90.42 +2025-08-05,90.67 +2025-08-06,89.13 +2025-08-07,88.58 +2025-08-08,89.22 +2025-08-11,89.47 +2025-08-12,89.37 +2025-08-13,89.56 +2025-08-14,89.64 +2025-08-15,89.15 +2025-08-18,89.42 +2025-08-19,88.3 +2025-08-20,87.56 +2025-08-21,86.11 +2025-08-22,84.83 +2025-08-25,83.5 +2025-08-26,83.76 +2025-08-27,82.93 +2025-08-28,82.5 +2025-08-29,82.21 +2025-09-01,81.71 +2025-09-02,82.85 +2025-09-03,83.27 +2025-09-04,83.67 +2025-09-05,82.68 +2025-09-08,82.12 +2025-09-09,80.07 +2025-09-10,81.39 +2025-09-11,82.55 +2025-09-12,81.92 +2025-09-15,82.41 +2025-09-16,83.45 +2025-09-17,83.22 +2025-09-18,82.16 +2025-09-19,83.08 +2025-09-22,82.02 +2025-09-23,81.42 +2025-09-24,81.77 +2025-09-25,81.02 +2025-09-26,80.86 +2025-09-29,81.43 +2025-09-30,81.8 +2025-10-01,80.59 +2025-10-02,81.11 +2025-10-03,82.34 +2025-10-06,81.41 +2025-10-07,82.01 +2025-10-08,83.11 +2025-10-09,82.39 +2025-10-10,82.95 +2025-10-13,83.6 +2025-10-14,83.94 +2025-10-15,82.65 +2025-10-16,81.45 +2025-10-17,83.55 +2025-10-20,85.44 +2025-10-21,85.69 +2025-10-22,86.33 +2025-10-23,84.91 +2025-10-24,86.81 +2025-10-27,87.28 +2025-10-28,86.52 +2025-10-29,86.26 +2025-10-30,85.03 +2025-10-31,86.63 +2025-11-03,85.61 +2025-11-04,86.34 +2025-11-05,86.39 +2025-11-06,85.44 +2025-11-07,86.04 +2025-11-10,87.62 +2025-11-11,86.87 +2025-11-12,87.94 +2025-11-13,88.69 +2025-11-14,88.45 +2025-11-17,88.43 +2025-11-18,88.92 +2025-11-19,88.85 +2025-11-20,89.05 +2025-11-21,87.18 +2025-11-24,86.51 +2025-11-25,86.25 +2025-11-26,88.37 +2025-11-27,87.3 +2025-11-28,87.41 +2025-12-01,84.9 +2025-12-02,85.1 +2025-12-03,84.99 +2025-12-04,84.09 +2025-12-05,83.22 +2025-12-08,83.53 +2025-12-09,84.13 +2025-12-10,83.52 +2025-12-11,83.26 +2025-12-12,82.44 +2025-12-15,82.02 +2025-12-16,82.72 +2025-12-17,82.97 +2025-12-18,82.6 +2025-12-19,83.15 +2025-12-22,82.28 +2025-12-23,82.58 +2025-12-24,82.65 +2025-12-25,83.9 +2025-12-26,84.66 +2025-12-29,85.37 +2025-12-30,87.54 +2025-12-31,86.92 +2026-01-01,86.93 +2026-01-02,87.7 +2026-01-05,86.76 +2026-01-06,86.6 +2026-01-07,86.47 +2026-01-08,86.49 +2026-01-09,86.14 +2026-01-12,87.27 +2026-01-13,86.06 +2026-01-14,86.82 +2026-01-15,86.91 +2026-01-16,86.88 +2026-01-19,87.64 +2026-01-20,86.92 +2026-01-21,87.02 +2026-01-22,86.92 +2026-01-23,87.27 +2026-01-26,86.11 +2026-01-27,86.94 +2026-01-28,86.82 +2026-01-29,85.98 +2026-01-30,85.28 +2026-02-02,84.52 +2026-02-03,85.85 +2026-02-04,86.25 +2026-02-05,85.87 +2026-02-06,86.29 +2026-02-09,87.05 +2026-02-10,94.53 +2026-02-11,94.79 +2026-02-12,93.68 +2026-02-13,94.97 +2026-02-16,94.98 +2026-02-17,93.35 +2026-02-18,92.75 +2026-02-19,90.1 +2026-02-20,90.23 +2026-02-23,90.74 +2026-02-24,98.1 +2026-02-25,98.21 +2026-02-26,94.13 +2026-02-27,94.06 +2026-03-02,92.62 +2026-03-03,90.62 +2026-03-04,94.24 +2026-03-05,95.08 +2026-03-06,97.38 +2026-03-09,98.17 +2026-03-10,98.18 +2026-03-11,95.83 +2026-03-12,93.96 +2026-03-13,93.68 +2026-03-16,95.29 +2026-03-17,96.34 +2026-03-18,94.48 +2026-03-19,93.67 +2026-03-20,92.3 +2026-03-23,91.07 +2026-03-24,90.02 +2026-03-25,89.85 +2026-03-26,90.7 +2026-03-27,91.31 +2026-03-30,87.09 +2026-03-31,88.44 +2026-04-01,85.84 +2026-04-02,83.03 +2026-04-03,77.15 +2026-04-06,76.15 +2026-04-07,75.46 +2026-04-08,75.55 +2026-04-09,75.66 +2026-04-10,77.67 +2026-04-13,76.31 +2026-04-14,75.42 +2026-04-15,77.24 +2026-04-16,76.73 +2026-04-17,74.81 +2026-04-20,74.44 +2026-04-21,72.94 +2026-04-22,72.37 +2026-04-23,71.53 +2026-04-24,70.76 +2026-04-27,70.97 +2026-04-28,70.78 +2026-04-29,71.15 +2026-04-30,72.3 +2026-05-01,72.4 +2026-05-04,70.11 +2026-05-05,71.57 +2026-05-06,69.95 +2026-05-07,69.39 +2026-05-08,69.14 +2026-05-11,69.03 +2026-05-12,69.64 +2026-05-13,69.64 +2026-05-14,69.44 +2026-05-15,69.08 +2026-05-18,68.46 +2026-05-19,69.02 +2026-05-20,68.99 +2026-05-21,70.14 +2026-05-22,69.87 +2026-05-25,69.57 +2026-05-26,70.1 +2026-05-27,70.23 +2026-05-28,69.59 +2026-05-29,69.09 +2026-06-01,68.64 +2026-06-02,68.43 +2026-06-03,68.6 +2026-06-04,66.99 +2026-06-05,67.21 +2026-06-08,67.24 +2026-06-09,66.78 +2026-06-10,68.57 +2026-06-11,68.12 +2026-06-12,68.71 +2026-06-15,69.68 +2026-06-16,70.19 +2026-06-17,70.33 +2026-06-18,71.44 +2026-06-19,71.65 +2026-06-22,70.5 +2026-06-23,69.34 +2026-06-24,68.4 +2026-06-25,68.28 +2026-06-26,68.54 +2026-06-29,68.56 +2026-06-30,69.01 diff --git a/implementations/README.md b/implementations/README.md index cbe7a512..7f861a2e 100644 --- a/implementations/README.md +++ b/implementations/README.md @@ -28,7 +28,7 @@ implementations/ `-- pyproject.toml # local workspace packaging ``` -YAML backtest and eval specs live under each use case in `specs/`. Each directory is independent; see its `README.md` for the walkthrough. +YAML backtest and eval specs live under each use case in `specs/`. Each directory is independent; see its `README.md` for the walkthrough. For the build-phase moves — onboarding data, standing up an experiment, customizing an agent, auditing a result — see [`guides/`](../guides/). Every domain use case (all except `getting_started`) also ships a `starter_agent/` module and a `99_starter_agent.ipynb` — a fresh, hackable **starter agent** that is the consistent "build your own" entry point for that use case (toggleable news search + code execution, two lightweight tool-usage skills, an interactive cell, and one scored forecast).