feat: release history via get_release_changes

CHANGELOG.md

@@ -29,16 +29,27 @@ Refactors, CI, and formatting land in the git history, not here.
   `analysis_result_id` values, as it already did for `collection_id`.
 - `get_attribute_values` / `GET /v3/attributes/{attribute}/values` return `null_count`, the
   number of series with no value, which `values` does not list.
+- `GET /v3/releases/changes?version=N` and the MCP tool `get_release_changes(version)` describe
+  what changed in IDC release N (`24` or `v24`; default the served release) against N-1: series
+  added, revised and removed with TB and patients affected, new, updated and removed collections,
+  and analysis results with changed series. Any past release is covered, computed from the served
+  release's tables; v1 and v2 carry a `warnings` entry, since v1 is only partly recorded
+  ([#40](https://github.com/ImagingDataCommons/IDC-REST-MCP/issues/40)).
+- MCP prompt `whats_new` (optional `version`) asks for a summary of a release.
+- `GET /v3/tables` / `list_tables` give each table a few `notable_columns`, such as
+  `series_init_idc_version` on `index`.
 
 ### Changed
 
-- **`idc-index` 0.13.3 / `idc-index-data` 25.0.4** (still IDC v25): a missing value in
+- **`idc-index` 0.13.4 / `idc-index-data` 25.0.5** (still IDC v25): a missing value in
   `index.PatientAge`, `index.SeriesDescription`, `pt_index.SeriesType`,
   `collections_index.supporting_data` or `clinical_index.column_label` is now NULL, never an empty
   string, so SQL that tested `= ''` must test `IS NULL`. `volume_geometry_index` adds
   `origin_patient`, `ImageOrientationPatient` and `slice_spacing_mm`. `GET /v3/version` and
-  `get_idc_version` report `idc_index_data_version` `25.0.4`; collection, series, and size totals
-  are unchanged.
+  `get_idc_version` report `idc_index_data_version` `25.0.5`; collection, series, and size totals
+  are unchanged. `prior_versions_index` now has a table description and column descriptions,
+  which say it is not a record of what is new: a row also ends when a version only moved storage
+  bucket.
 
 ### Fixed
 

dev/deployment.md

@@ -160,7 +160,7 @@ python3 dev/smoke_openapi_examples.py "$URL"
 The image installs dependencies from [uv.lock](../uv.lock) (see [Dockerfile](../Dockerfile)), so
 it bakes exactly the `idc-index-data` the lock records and a rebuild of the same commit is
 deterministic. To move IDC versions, bump the exact `idc-index` pin in
-[pyproject.toml](../pyproject.toml) (e.g. `idc-index==0.13.3`, which pulls a specific
+[pyproject.toml](../pyproject.toml) (e.g. `idc-index==0.13.4`, which pulls a specific
 `idc-index-data`), run `uv lock` and commit both files — CI's `uv sync --locked` fails if the lock
 is stale — then **rebuild and redeploy** (steps 2–3). The running version is always reported at
 `/v3/version`.

docs/user-guide.md

@@ -60,7 +60,7 @@ understanding, because picking the right one makes everything else easy:
 
 | Surface | Answers | REST | MCP tools |
 |---|---|---|---|
-| **Discovery** | "What exists? What can I filter on?" | `GET /v3/version`, `/v3/stats`, `/v3/collections`, `/v3/collections/{id}`, `/v3/analysis_results`, `/v3/attributes`, `/v3/attributes/{attr}/values` | `get_idc_version`, `get_stats`, `list_collections`, `get_collection`, `list_analysis_results`, `list_attributes`, `get_attribute_values` |
+| **Discovery** | "What exists? What can I filter on? What changed in a release?" | `GET /v3/version`, `/v3/releases/changes`, `/v3/stats`, `/v3/collections`, `/v3/collections/{id}`, `/v3/analysis_results`, `/v3/attributes`, `/v3/attributes/{attr}/values` | `get_idc_version`, `get_release_changes`, `get_stats`, `list_collections`, `get_collection`, `list_analysis_results`, `list_attributes`, `get_attribute_values` |
 | **Cohort** | "How big is *my* selection, and what's in it?" | `POST /v3/cohort/counts`, `POST /v3/cohort/manifest` | `build_cohort` |
 | **Retrieval** | "Give me the download links" | `POST /v3/cohort/manifest.txt` | `get_cohort_urls` |
 | **SQL** | "Run my custom query" + schema | `GET /v3/tables`, `/v3/tables/{table}`, `POST /v3/sql` | `list_tables`, `get_table_schema`, `run_sql` |
@@ -154,7 +154,7 @@ per-collection, keyed by `collection_id`).
 | `index` | one row per **series** — the main table |
 | `collections_index` | one row per collection (curated metadata) |
 | `analysis_results_index` | one row per analysis result |
-| `version_metadata_index` / `prior_versions_index` | IDC release versions / removed series |
+| `version_metadata_index` / `prior_versions_index` | IDC release dates / series versions no longer served (not a list of what is new) — see [Release history](#release-history) |
 | `seg_index`, `ann_index`, `ann_group_index`, `rtstruct_index` | segmentations / annotations / RT structures: **what was segmented** (`SegmentedPropertyType_CodeMeanings` — `BodyPartExamined` reflects the source acquisition, not this) and the **reference** to the image series they derive from (`segmented_SeriesInstanceUID` / `referenced_SeriesInstanceUID`) |
 | `ct_index`, `mr_index`, `pt_index` | per-modality acquisition parameters (slice thickness, kVp, TE/TR, injected dose…) |
 | `sm_index`, `sm_instance_index` | slide-microscopy (pathology) series / instance metadata |
@@ -238,6 +238,46 @@ WHERE i.collection_id = 'nlst' AND i.Modality = 'CT'
 > `IDC_API_INCLUDE_INDICES=all` includes it; the clinical tools return a clear "not included"
 > error otherwise).
 
+### Release history
+
+The server serves **one** IDC data release (see `GET /v3/version` / `get_idc_version`), but that
+release carries the history of every earlier one, so "what's new in v24" or "what changed since
+v20" are answerable without external release notes.
+
+`GET /v3/releases/changes?version=N` / `get_release_changes(version=N)` diffs release N against
+N-1: series **added / revised / removed** (with TB and patients affected), collections that are
+**new / updated / removed**, and the analysis results with changed series. `version` takes `24` or
+`v24`; omit it for the served release. A series that moved to another collection counts as removed
+from one and added to the other in the per-collection rows, but not in the totals. Results for v1
+and v2 carry a `warnings` entry: series versions served in v1 and replaced later were never
+recorded, so those two diffs are incomplete.
+
+For per-series detail of the series still served, query `index` with SQL:
+`series_init_idc_version` is the release a series first appeared in, and
+`series_revised_idc_version` the release its current content dates from. For example, the series
+of a collection that changed since v20:
+
+```sql
+SELECT SeriesInstanceUID, series_init_idc_version, series_revised_idc_version
+FROM index
+WHERE collection_id = 'nlst' AND series_revised_idc_version > 20
+```
+
+`prior_versions_index` lists the series versions IDC no longer serves, mainly so manifests from
+earlier releases can still be resolved. It is **not** a record of what is new: it never lists
+added series, and a row also ends when a version only moved storage bucket, so counting rows by
+`max_idc_version` overstates revisions (for v20 it would count 11,928, of which 11,338 were
+bucket moves). `get_release_changes` accounts for this.
+
+Removed series, and earlier versions of revised ones, are only in `prior_versions_index`. To
+compare two releases series by series, build each release from both tables (an `index` row is in
+release R when `series_revised_idc_version <= R`, a `prior_versions_index` row when
+`min_idc_version <= R <= max_idc_version`), then compare which series are present and, for those
+in both, their `crdc_series_uuid`.
+
+The diff tells you *what* changed, not *why*; for the narrative, see the
+[IDC release notes](https://learn.canceridc.dev/data/data-release-notes).
+
 ---
 
 ## 2. Using the REST API
@@ -253,13 +293,14 @@ uv run idc-api          # http://127.0.0.1:8000  — Swagger UI at /v3/docs
 | Method & path | Purpose |
 |---|---|
 | `GET /v3/version` | IDC data release served (e.g. `v25`) + pinned index version, **and** this server's own software version (`api_version`, plus `build` if the deploy stamped one) |
+| `GET /v3/releases/changes?version=` | What changed in an IDC release vs. the previous one: series added / revised / removed, new / updated / removed collections, analysis results (default: the served release) |
 | `GET /v3/stats` | Headline totals (collections, patients, studies, series, size_TB) |
 | `GET /v3/collections` | List collections (datasets) |
 | `GET /v3/collections/{id}` | Collection detail: counts, modalities, license breakdown |
 | `GET /v3/analysis_results` | Derived datasets (segmentations/annotations) |
 | `GET /v3/attributes` | Filterable attributes (name, type, term/range, categorical) |
 | `GET /v3/attributes/{attr}/values?limit=` | Distinct values + counts for an attribute, plus a `note` caveat when one applies (e.g. `BodyPartExamined` ≠ segmented anatomy) |
-| `GET /v3/tables` | Tables available to SQL |
+| `GET /v3/tables` | Tables available to SQL, each with a few `notable_columns` |
 | `GET /v3/tables/{table}` | Column schema for a table |
 | `GET /v3/clinical/tables?collection_id=` | Per-collection clinical tables (optionally one collection) |
 | `GET /v3/clinical/tables/{table}` | Clinical table columns + human-readable labels |
@@ -311,6 +352,8 @@ For `analysis_result_id` that is all of the original data.
 
 ```bash
 curl -s localhost:8000/v3/version                 # data release + this server's build
+curl -s localhost:8000/v3/releases/changes       # what's new in the served release
+curl -s 'localhost:8000/v3/releases/changes?version=23'  # …or in any earlier one
 curl -s localhost:8000/v3/stats                   # headline totals
 curl -s localhost:8000/v3/collections             # list datasets
 curl -s localhost:8000/v3/collections/nlst        # one collection's detail
@@ -409,13 +452,16 @@ uv run idc-mcp --http --host 0.0.0.0 --port 8080     # hosted/shared
 
 - **Discovery:** `get_idc_version`, `get_stats`, `list_collections`, `get_collection`,
   `list_analysis_results`, `list_attributes`, `get_attribute_values`
+- **Release history:** `get_release_changes`
 - **Schema (for SQL):** `list_tables`, `get_table_schema`
 - **Clinical data:** `list_clinical_tables`, `get_clinical_table_schema`, `get_clinical_table`
 - **Cohort / query:** `build_cohort`, `run_sql`
 - **Retrieval & side tools:** `get_cohort_urls`, `get_viewer_url`,
   `get_citations`, `get_licenses`
 - **Resources:** `idc://guide` (data model + recommended workflow), `idc://tables`,
   `idc://schema/{table}`
+- **Prompts:** `whats_new` (optional `version`) — a ready-made "summarize what's new in this
+  release" request for clients that show MCP prompts
 
 Tool descriptions are prescriptive about *when* to call each one, and the server ships an
 `idc://guide` resource with the same conceptual model as this document — so a capable agent can

pyproject.toml

@@ -11,12 +11,12 @@ keywords = ["IDC", "DICOM", "cancer imaging", "MCP", "REST", "idc-index"]
 dependencies = [
     # Query engine + data.
     # idc-index is pinned EXACTLY, unlike every other dependency here, because it transitively
-    # pins idc-index-data (0.13.3 -> 25.0.4) — which *is* the IDC data release this server
+    # pins idc-index-data (0.13.4 -> 25.0.5) — which *is* the IDC data release this server
     # advertises at /v3/version and documents in the changelog. The image installs from uv.lock,
     # so the lock alone already fixes it; the exact pin keeps the served IDC release explicit
     # here and stops a routine `uv lock --upgrade` from silently moving it. Bump this pin (then
     # `uv lock`) to move IDC versions — see dev/deployment.md § Updating for a new IDC release.
-    "idc-index==0.13.3",
+    "idc-index==0.13.4",
     "duckdb>=1.5.5",
     "pyarrow>=25.0.0",
     # Shared models / settings

src/idc_api/core/context.py

@@ -13,6 +13,7 @@
     LicenseService,
     ManifestService,
     QueryService,
+    ReleaseService,
     ViewerService,
 )
 
@@ -22,6 +23,7 @@ def __init__(self, settings: Settings | None = None):
         self.settings = settings or get_settings()
         self.backend = DuckDBBackend(self.settings)
         self.discovery = DiscoveryService(self.backend)
+        self.releases = ReleaseService(self.backend)
         self.cohort = CohortService(self.backend, self.settings)
         self.manifest = ManifestService(self.backend, self.settings)
         self.query = QueryService(self.backend, self.settings)

src/idc_api/core/models.py

@@ -39,6 +39,80 @@ class Stats(BaseModel):
     size_TB: float
 
 
+class CollectionChange(BaseModel):
+    collection_id: str
+    status: Literal["new", "removed", "updated"] = Field(
+        ...,
+        description="'new' (no series in the previous release), 'removed' (no series in this "
+        "release), or 'updated' (present in both, with series added/revised/removed). A series "
+        "that moved to another collection counts as removed from one and added to the other.",
+    )
+    series_added: int
+    series_revised: int
+    series_removed: int
+    size_TB_added: float
+    size_TB_revised: float
+    size_TB_removed: float
+    patients_affected: int = Field(
+        ..., description="Distinct patients with at least one added, revised, or removed series."
+    )
+
+
+class AnalysisResultChange(BaseModel):
+    analysis_result_id: str
+    is_new: bool = Field(
+        ..., description="True if this analysis result had no series in the previous release."
+    )
+    series_added: int
+    series_revised: int
+    series_removed: int
+
+
+class ReleaseChanges(BaseModel):
+    """What changed in one IDC release relative to the one before it."""
+
+    idc_version: str = Field(..., description="The release described, e.g. 'v24'.")
+    release_date: str | None = None
+    previous_version: str | None = Field(None, description="The release compared against.")
+    previous_release_date: str | None = None
+    current_version: str = Field(..., description="The release this server serves.")
+    series_added: int = Field(..., description="Series new to IDC in this release.")
+    series_revised: int = Field(
+        ...,
+        description="Series in both releases whose content changed (a new crdc_series_uuid).",
+    )
+    series_removed: int = Field(..., description="Series IDC no longer serves in this release.")
+    size_TB_added: float
+    size_TB_revised: float
+    size_TB_removed: float
+    patients_affected: int = Field(
+        ...,
+        description="Distinct patients (PatientID within a collection) with at least one added, "
+        "revised, or removed series.",
+    )
+    new_collections: list[str] = Field(
+        default_factory=list, description="Collections that first appeared in this release."
+    )
+    removed_collections: list[str] = Field(
+        default_factory=list, description="Collections with no series left in this release."
+    )
+    collections: list[CollectionChange] = Field(
+        default_factory=list, description="Per-collection change, largest additions first."
+    )
+    analysis_results: list[AnalysisResultChange] = Field(
+        default_factory=list,
+        description="Analysis results with series added, revised, or removed in this release. A "
+        "series no longer served today cannot be attributed to its analysis result, so for an "
+        "older release these counts omit series that were later removed.",
+    )
+    warnings: list[str] = Field(
+        default_factory=list,
+        description="Caveats about how complete this release's numbers are. Empty when the "
+        "served data records the release in full.",
+    )
+    note: str = ""
+
+
 class CollectionSummary(BaseModel):
     collection_id: str
     collection_name: str | None = None
@@ -125,6 +199,11 @@ class TableInfo(BaseModel):
     name: str
     description: str = ""
     column_count: int
+    notable_columns: list[str] = Field(
+        default_factory=list,
+        description="A few columns worth knowing about before writing SQL (not the full list — "
+        "get_table_schema has every column with its description).",
+    )
 
 
 class TableList(BaseModel):

src/idc_api/core/schema.py

@@ -193,6 +193,33 @@ def _column_type(c: dict, struct_types: dict[str, str] | None = None) -> str:
     ),
 }
 
+# A handful of columns per table surfaced by list_tables, so a caller learns they exist without
+# having to call get_table_schema first (a caller that guesses the obvious columns right never
+# does). Not a full listing; names absent from a table's schema are dropped.
+NOTABLE_COLUMNS: dict[str, list[str]] = {
+    "index": [
+        "collection_id",
+        "analysis_result_id",
+        "PatientID",
+        "SeriesInstanceUID",
+        "Modality",
+        "BodyPartExamined",
+        "SeriesDescription",
+        "license_short_name",
+        "series_size_MB",
+        "series_init_idc_version",
+        "series_revised_idc_version",
+    ],
+    "version_metadata_index": ["idc_version", "version_timestamp"],
+    "prior_versions_index": [
+        "SeriesInstanceUID",
+        "crdc_series_uuid",
+        "min_idc_version",
+        "max_idc_version",
+    ],
+    "seg_index": ["segmented_SeriesInstanceUID", "SegmentedPropertyType_CodeMeanings"],
+}
+
 
 @lru_cache(maxsize=None)
 def table_schema(table: str) -> dict:

src/idc_api/core/services/__init__.py

@@ -11,6 +11,7 @@
 from .licenses import LicenseService
 from .manifest import ManifestService
 from .query import QueryService
+from .releases import ReleaseService
 from .viewer import ViewerService
 
 __all__ = [
@@ -21,5 +22,6 @@
     "LicenseService",
     "ManifestService",
     "QueryService",
+    "ReleaseService",
     "ViewerService",
 ]

src/idc_api/core/services/_common.py

@@ -0,0 +1,19 @@
+"""Small helpers shared by several services, so every endpoint reports the same thing the same
+way."""
+
+from __future__ import annotations
+
+import idc_index_data
+
+_MB_PER_TB = 1_000_000
+
+
+def mb_to_tb(mb: float) -> float:
+    """Megabytes (as ``series_size_MB`` sums are) to the ``size_TB`` every response reports."""
+    return round(mb / _MB_PER_TB, 3)
+
+
+def served_idc_release() -> int:
+    """The IDC data release this server serves: the major version of the pinned
+    ``idc-index-data`` (25.0.5 → 25)."""
+    return int(idc_index_data.__version__.split(".")[0])

src/idc_api/core/services/cohort.py

@@ -12,8 +12,8 @@
     SeriesManifestRow,
 )
 from .manifest import ManifestService
+from ._common import mb_to_tb
 
-_MB_PER_TB = 1_000_000
 
 _ROW_COLUMNS = [
     "collection_id",
@@ -55,7 +55,7 @@ def counts(self, filters: CohortFilters) -> CohortCounts:
             studies=row["studies"],
             series=row["series"],
             instances=int(row["instances"]),
-            size_TB=round(row["size_mb"] / _MB_PER_TB, 3),
+            size_TB=mb_to_tb(row["size_mb"]),
             # Echo the effective filter so a caller can tell an empty cohort apart from a
             # dropped filter without guessing from the magnitude of the numbers.
             filters_applied=f.applied,