Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -591,6 +591,23 @@ def greeting_pipeline(who: In[str], cfg) -> Out[str]:

`In[T]` parameters become runtime graph inputs. A single `-> Out[T]` return exposes one graph output; use `@pipeline(output_name=...)` to name that output. For multiple outputs, define a frozen dataclass subclass of `Outputs` with `Out[T]` fields and return an instance. A pipeline that accepts a `cfg` parameter reads `config.yaml` (or the path passed via `@pipeline(config="...")`) at compile time, with `--override key=value` values overlaid by the compile command.

###### Input defaults

A parameter default becomes the input's `default`, which the schema carries as a string, so `threshold: In[int] = 5` emits `default: '5'`. The same rules apply to `graph_input(default=...)`:

| Default | Emitted | Note |
| --- | --- | --- |
| `"x"` | `x` | Unchanged, always accepted |
| `5`, `-1` | `'5'`, `'-1'` | |
| `1.5` | `'1.5'` | `In[float] = 1` emits `'1'`, not `'1.0'` |
| `True` / `False` | `'True'` / `'False'` | Python spelling, as `@task` component inputs use; the generated container parses it case-insensitively |
| `{"b": 1, "a": 2}`, `[2, 1]` | `'{"a": 2, "b": 1}'`, `'[2, 1]'` | JSON with sorted keys, so the document is reproducible |
| `None` | `null` | Schema-valid; `graph_input` instead rejects an explicit `default=None` — omit the argument |

Anything else — an enum, a dataclass, a `datetime`, a `Path` — raises `InvalidInputDefaultError` (a `CompileError`) naming the input and the type, never the value. Render it yourself and pass the string. A non-finite float (`nan`, `inf`) is rejected for the same reason: JSON has no spelling for it.

The default's type must also agree with the declared one. `In[int] = 1.5`, `In[str] = 5`, and `In[int] = True` (a `bool` is a Python `int`, but not a Tangle `Integer`) are all refused. A `graph_input` type is a Tangle type string describing the wire form, so it additionally accepts an already-rendered string: `graph_input("n_samples", "Integer", default="-1")` emits exactly `-1`, which is how every default in the corpus is written. Types with no scalar Python equivalent, such as `Json`, are not type-checked.

##### Environment-selected pipeline config (`_select`, `_env`)

A pipeline `config.yaml` — root or child, including one broadcast with `propagate_config=True` — is resolved by the same loader as `--config` files (see [`_select`](#environment-selected-configs-_select) and [`_env`](#environment-variable-values-_env)), with one shape rule: the selected document, and every case and `default` branch, must be a mapping. `_defaults`/`configs` have no special meaning in a pipeline config; they are ordinary keys.
Expand Down Expand Up @@ -664,7 +681,8 @@ def daily_pulse() -> Out[str]:
- **A name is declared once.** Duplicates are rejected, including a collision with an `In[...]` parameter or with the name the return annotation contributes (`Out[T]`'s `output_name`, or an `Outputs` field).
- **Outputs wire to a handle, never a constant** — a task output or a graph input, the same rule the `Outputs` return path enforces. Returning a value *and* declaring outputs is supported: the returned output is appended after the declared ones.
- **Layout comes along**: `position=(x, y)` writes the same canonical `editor.position` annotation as `.with_position(...)`, which is how a graph input gets a position in the editor. `annotations={...}` is validated under the same caller policy as `pipeline_annotations`.
- **Validated up front, without echoing values.** A non-string `default` (the schema's `InputSpec.default` is a string), a non-string name/type/description, a non-boolean `optional`, a constant output, or a call outside a `@pipeline` body raises `InvalidGraphIoError` (a `CompileError`) naming the field, never the value.
- **`default` follows the [input-default rules](#input-defaults)** — scalars and JSON containers are rendered for you, an already-rendered string is passed through untouched, and the type must agree with the declared Tangle type.
- **Validated up front, without echoing values.** A non-string name/type/description, a non-boolean `optional`, a constant output, or a call outside a `@pipeline` body raises `InvalidGraphIoError` (a `CompileError`) naming the field, never the value.

The signature route remains the recommended default: it is typed, it is checked by your IDE, and it reads better. Reach for these when porting existing YAML or when the shape genuinely needs it.

Expand Down
2 changes: 1 addition & 1 deletion packages/tangle-cli/src/tangle_cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@
try:
__version__ = metadata_version("tangle-cli")
except PackageNotFoundError:
__version__ = "0.1.24"
__version__ = "0.1.25"

__all__ = ["TangleDynamicDiscoveryClient", "__version__"]
109 changes: 109 additions & 0 deletions packages/tangle-cli/src/tangle_cli/input_defaults.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
"""Serialize a Python graph-input default into the string the schema wants.

``InputSpec.default`` is ``string | null``, so every default a pipeline
declares from Python has to be rendered as text. Callers inject ``error_cls``
and a field label; diagnostics name the field and the type, never the value.
"""
from __future__ import annotations

import json
import math
from typing import Any

# Types a default may be written as. Anything else is refused rather than
# guessed at: ``str(value)`` on an enum or a dataclass produces text no one
# meant to put in a pipeline document.
SUPPORTED_DEFAULT_TYPES = (str, bool, int, float, list, dict)

# Tangle type strings that pin a default to a Python type. Json and JsonArray
# are deliberately absent: their values are structural, not scalar.
_TANGLE_SCALARS: dict[str, type] = {
"String": str,
"Integer": int,
"Float": float,
"Boolean": bool,
}


def serialize_input_default(
value: Any,
*,
field: str,
declared_type: Any = None,
error_cls: type[Exception] = ValueError,
) -> str | None:
"""Return the YAML form of ``value``, or ``None`` to write no default.

``declared_type`` is a Python type or a Tangle type string; when it names
a builtin scalar the default's type must match it. A Tangle type string
additionally accepts an already-rendered ``str``, which is the wire form
and how every default in the corpus is written. Booleans render in Python
spelling (``"True"``), matching ``@task`` component inputs and the corpus;
containers render as JSON with sorted keys so the document is reproducible.
"""
if value is None:
return None
if not isinstance(value, SUPPORTED_DEFAULT_TYPES):
supported = ", ".join(t.__name__ for t in SUPPORTED_DEFAULT_TYPES)
raise error_cls(
f"{field} default must be one of {supported} or None; "
f"got {type(value).__name__}. Render it yourself and pass a string."
)
_check_declared_type(value, declared_type, field=field, error_cls=error_cls)
if isinstance(value, str):
return value
if isinstance(value, bool):
return str(value)
if isinstance(value, float):
if not math.isfinite(value):
raise error_cls(
f"{field} default must be a finite number; "
f"got a non-finite float. JSON has no spelling for it."
)
return str(value)
if isinstance(value, int):
return str(value)
try:
return json.dumps(value, sort_keys=True)
except (TypeError, ValueError) as exc:
raise error_cls(
f"{field} default must be JSON-serializable; "
f"{type(value).__name__} contains a value that is not."
) from exc


def _check_declared_type(
value: Any,
declared_type: Any,
*,
field: str,
error_cls: type[Exception],
) -> None:
expected = declared_type
if isinstance(declared_type, str):
expected = _TANGLE_SCALARS.get(declared_type)
# A Tangle type string describes the wire form, where every value is
# text. A pre-rendered default is therefore always in range, so
# ported YAML keeps emitting exactly the bytes it did before.
if isinstance(value, str):
return
if expected not in (str, bool, int, float):
return

actual = type(value)
# bool is a subclass of int, so an unguarded isinstance would let
# ``In[int] = True`` through and emit "True" for an Integer input.
if expected is bool:
ok = actual is bool
elif expected is float:
# A whole number written for a Float input is idiomatic and appears
# in the corpus ("0", "72"); it renders as-is, not as "0.0".
ok = actual is float or actual is int
else:
ok = actual is expected

if not ok:
raise error_cls(
f"{field} is declared {expected.__name__} but its default is "
f"{actual.__name__}. Declare the matching type or convert the default."
)
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@ class InvalidEditorLayoutError(CompileError):
"""


class InvalidInputDefaultError(CompileError):
"""Raised on an ``In[T]`` parameter default the schema cannot carry.

Covers an unsupported type, a non-finite float, and a default whose type
contradicts the declared ``T``. Messages name the input, never the value.
"""


class InvalidGraphIoError(CompileError):
"""Raised on an unusable ``graph_input()`` / ``graph_output()`` call.

Expand Down
19 changes: 16 additions & 3 deletions packages/tangle-cli/src/tangle_cli/python_pipeline/graph_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
from typing import Any

from tangle_cli.editor_layout import POSITION_ANNOTATION, position_annotation_value
from tangle_cli.input_defaults import serialize_input_default
from tangle_cli.schema_validation import CALLER_ANNOTATION_POLICY, check_annotations

from .errors import InvalidEditorLayoutError, InvalidGraphIoError
Expand Down Expand Up @@ -74,9 +75,21 @@ def graph_input(
if description is not None:
entry["description"] = _check_str("description", description)
if default is not _UNSET:
# InputSpec.default is a string in the schema; a non-string default
# would only fail later, against a schema path instead of this call.
entry["default"] = _check_str("default", default, hint="pass str(value)")
# InputSpec.default is a string in the schema, so the value is
# rendered here rather than failing later against a schema path.
# An explicit default=None is a mistake worth reporting: omit the
# argument to declare no default.
if default is None:
raise InvalidGraphIoError(
"default must not be None. Omit the argument to declare "
"an input with no default."
)
entry["default"] = serialize_input_default(
default,
field=f"graph input {name!r}",
declared_type=type,
error_cls=InvalidGraphIoError,
)
entry["optional"] = True if optional is None else _check_bool(optional)
elif optional is not None:
entry["optional"] = _check_bool(optional)
Expand Down
24 changes: 22 additions & 2 deletions packages/tangle-cli/src/tangle_cli/python_pipeline/trace.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@
from contextvars import ContextVar
from typing import TYPE_CHECKING, Any

from .errors import CompileError
from tangle_cli.input_defaults import serialize_input_default

from .errors import CompileError, InvalidInputDefaultError
from .graph import EdgeRef, GraphBuilder
from .placeholders import GraphInputPlaceholder, TaskOutputProxy
from .types import In, Out, Outputs
Expand Down Expand Up @@ -166,6 +168,20 @@ def declared_output_names(pipeline_fn: "PipelineFn") -> tuple[str, ...] | None:
return None


def _serialize_signature_default(value: Any, name: str, inner: Any) -> Any:
"""Return the YAML form of an ``In[T]`` parameter default.

``None`` is passed through: ``default: null`` is schema-valid, and
rewriting it would change existing documents for no gain.
"""
return serialize_input_default(
value,
field=f"input {name!r}",
declared_type=inner,
error_cls=InvalidInputDefaultError,
)


def _python_type_to_tangle_type(t: Any) -> str:
"""Map ``str``/``int``/etc. to Tangle type strings."""
if t is str:
Expand Down Expand Up @@ -243,7 +259,11 @@ def trace_pipeline(

has_default = param.default is not inspect.Parameter.empty
if has_default:
entry["default"] = param.default
# ``InputSpec.default`` is a string in the schema, so a bare
# ``In[int] = 5`` used to emit a schema-invalid ``default: 5``.
entry["default"] = _serialize_signature_default(
param.default, param_name, inner
)
entry["optional"] = True
builder.inputs.append(entry)

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "tangle-cli"
version = "0.1.24"
version = "0.1.25"
description = "CLI for Tangle, the open-source ML pipeline orchestration platform"
readme = "README.md"
authors = [
Expand Down
29 changes: 23 additions & 6 deletions tests/test_graph_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -224,10 +224,11 @@ def p() -> Out[str]:
]


def test_a_non_string_default_is_refused_without_echoing_it(tmp_path):
"""``InputSpec.default`` is a string in the pipeline schema; catching it
here names the field instead of failing later against a schema path."""
message = _expect(
def test_a_scalar_default_is_rendered_as_the_schema_string(tmp_path):
"""``InputSpec.default`` is a string in the pipeline schema, so the value
is rendered here rather than failing later against a schema path. Full
rules and refusals live in ``test_signature_defaults.py``."""
doc = _doc(_compile(
tmp_path,
'''
@pipeline("P")
Expand All @@ -236,11 +237,27 @@ def p() -> Out[str]:
run_greet = greet(greeting=value)
return run_greet
''',
"scalar_default",
))

assert doc["inputs"][0]["default"] == "41"


def test_an_unrenderable_default_is_refused_without_echoing_it(tmp_path):
message = _expect(
tmp_path,
'''
@pipeline("P")
def p() -> Out[str]:
value = graph_input("value", "Integer", default=4.1)
run_greet = greet(greeting=value)
return run_greet
''',
"bad_default",
)

assert "default" in message and "int" in message
assert "41" not in message
assert "default" in message and "float" in message
assert "4.1" not in message


# ---------------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion tests/test_packaging.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ def test_tangle_cli_wheel_supports_expert_no_deps_import_path_without_tangle_api
requires_dist = [line for line in metadata.splitlines() if line.startswith("Requires-Dist: ")]
assert not any(name.startswith("tangle_api/") for name in names)
assert "tangle_cli/openapi/openapi.json" not in names
assert "Version: 0.1.24" in metadata
assert "Version: 0.1.25" in metadata
assert "Requires-Dist: tangle-api==0.1.1" in requires_dist
assert not any("extra == 'native'" in line for line in requires_dist)
assert "Provides-Extra: native" in metadata
Expand Down
Loading
Loading