From 3700413818619abc69ee1d046707d294bef230b4 Mon Sep 17 00:00:00 2001 From: Yves Brissaud Date: Sat, 5 Sep 2026 00:28:38 +0200 Subject: [PATCH 1/9] hack/designs: adopt the module-max SDK interface Design and implementation plan for replacing the beta SDK-module interface (initModule, targetRuntime, the @generate hook, and currentModule.asSDK) with detectScope and generateScope, as specified by dagger/dagger#13992. Signed-off-by: Yves Brissaud --- .../2026-09-04-sdk-module-interface.md | 572 ++++++++++++++++++ 1 file changed, 572 insertions(+) create mode 100644 hack/designs/2026-09-04-sdk-module-interface.md diff --git a/hack/designs/2026-09-04-sdk-module-interface.md b/hack/designs/2026-09-04-sdk-module-interface.md new file mode 100644 index 0000000..020e647 --- /dev/null +++ b/hack/designs/2026-09-04-sdk-module-interface.md @@ -0,0 +1,572 @@ +# Adopt the module-max SDK interface + +Status: proposed +Date: 2026-09-04 + +## Reviewed baselines + +Every claim in this document was checked against these exact revisions. + +| What | Revision | +| --- | --- | +| This repository (`dagger/java-sdk`), base of the change | `be18cc2d64951628a79ae7da626ab2427b6a2436` | +| The engine change, `dagger/dagger#13992`, branch `sdk-ux-module-max` | `78c241b6ce5f950461c74811e768f92e946e2ca7` | +| The precedent, `dagger/python-sdk#25`, head | `c05426e0fbef5d758184667d62ddc406591b8192` (merged as `d8f8eca33c75c1113ba8412b3d3ad626a0c9b0ef`) | +| The released engine and CLI this repository's CI runs | `v1.0.0-beta.11` | + +`sdk-ux-module-max` is force-pushed regularly. Every reference to it below means +the commit in this table, not the branch head at the time of reading. + +## Problem + +`dagger/dagger#13992` changes the contract between the engine and an SDK module. +It ships no compatibility adapter. On an engine built from that change, this +repository's root module cannot serve as the Java SDK at all. + +Three removals break it: + +- `CurrentModule.asSDK` is gone. `JavaSdk.modules` selects it, so module + discovery has no source. +- The beta SDK-module interface — `initModule`, `targetRuntime`, and the + `@generate` hook — is gone. `JavaSdk.initModule`, `JavaSdk.targetRuntime`, and + `JavaSdk.generateAll` implement exactly that interface. +- `ModuleSource.generateLocalDependencies` is gone. `Mod.generateModule` selects + it to stage a module's local dependencies before code generation. + +The replacement is two required functions and one optional one, declared in +`core/sdkmodule/provider.go`: + +- `detectScope(ws: Workspace!): String!` returns the workspace-relative path of + the nearest scope that contains the workspace cwd, or `""` when there is none. +- `generateScope(ws: Workspace!, isModule: Boolean!, name: String!, clients: [ModuleSource!]!): Workspace!` + receives a workspace whose cwd is already the scope, and returns the complete + scope: the starter template and a module manifest when the scope is new, and + freshly generated bindings always. The `clients` list becomes the scope's + dependency set. +- `defaultModulePath(ws: Workspace!, name: String!): String!` is optional. See + Non-goals. + +The engine validates the names, order, and types of those arguments exactly, so +they are not negotiable. + +Registration moves with the interface. An SDK is recorded as +`[sdks.] module = ""` in `dagger.toml`, and each +managed scope as `[sdks..scopes.""]`. + +Three sibling SDKs have already adopted the same interface the same way: +`dagger/python-sdk#25` (merged), `dagger/go-sdk#37`, and `dagger/dang-sdk#13`. + +## Goals + +1. Implement `detectScope` and `generateScope`, and delete the beta interface + they replace. +2. Write a module's `dagger-module.toml` through the engine's manifest builder, + so dependency editing stays the engine's business. +3. Turn a module scope's client list into the module's dependency set, so the + generated Java bindings carry each client's types. +4. Re-register this SDK and the end-to-end fixtures under `[sdks.java]`. +5. Prove the result against a real engine built from `sdk-ux-module-max`, in CI. +6. Keep pre-1.0 `dagger.json` modules working: a module that already has a + config keeps the runtime it already names. +7. Leave the pull request's CI green, rather than merging it red as + `dagger/python-sdk#25` did. + +## Non-goals + +- **Standalone clients.** A scope with clients but no module (`isModule: false`) + is refused with an error. This SDK has no mechanism to serve a client outside + a Java module: every generated binding is vendored under a module's `sdk/` + directory and compiled by that module's `pom.xml`. `dagger/python-sdk#25` + refuses the same case for the same reason. Serving standalone clients needs a + separate design — where the generated code goes, what builds it, what depends + on it — not a branch in this change. +- **`defaultModulePath`.** The engine's own default for a `dagger module init` + with no `--path` is `/.dagger/modules/` + (`core/schema/workspace_sdk_module.go`). That is the same convention this + repository already used, so implementing the hook would only restate it. + `mod.dang` carries an unused private `defaultModulePath` helper from the beta + interface; it goes away with the rest of that interface. +- **A public single-module generate entry point.** `dagger generate` regenerates + the recorded scopes, and the engine narrows that set to the scopes containing + the caller's cwd, so running it inside a module regenerates that module. A + separate `dagger call java-sdk mod --path … generate` command shape would be + new public surface with its own CLI contract to document and test, and the + engine interface does not need it. `Mod` stays internal. +- **Manifest v2 and generated entrypoints** (`dagger/dagger#14038`). That is a + different engine change, prototyped separately in `dagger/java-sdk#19`. +- **The unified-clients redesign** (`dagger/java-sdk#17`), which replaces module + dependencies with generated clients throughout the Java SDK. This change + adopts one engine interface; it does not redesign the Java client model. +- **Keeping the SDK loadable on the released engine.** See "Alternatives + considered". +- **Changing what generation produces.** The vendored SDK sources, the generated + bindings, and the generated entrypoint keep their current layout and build. + +## Proposed approach + +### `detectScope` + +A Java module always has a `pom.xml` at its root: the starter template writes +one, and the module's build needs one. Nothing else in a generated module is a +project marker. The vendored SDK under `/sdk` is added to the module's +build as extra source roots and carries no `pom.xml` of its own; the optional +committed SDK jar under `/sdk/repo` is accompanied by a `*.pom` file, +which is not named `pom.xml` and so is not a marker either. + +So `detectScope` answers with the directory of the nearest `pom.xml` at or above +the workspace cwd, as a path relative to the workspace root, and with `""` when +there is none. + +This is the direct analogue of python-sdk's rule — the nearest `pyproject.toml` — +minus the correction python needs. A Python module's vendored client library is +itself an installable Python project with its own `pyproject.toml`, so +python-sdk must lift a hit inside `sdk/` back to the owning module. Java has no +such hit to lift. + +Consequences worth stating: + +- In a Maven multi-module project, the nearest `pom.xml` wins, so a Dagger + module nested inside an aggregator resolves to itself, not to the aggregator. +- A Gradle project has no `pom.xml`, so `detectScope` returns `""` and the + engine reports that client generation is unavailable there. This SDK builds + modules with Maven; that is the correct answer, and the README says so. + +### `generateScope` + +```mermaid +flowchart TD + A["generateScope(ws, isModule, name, clients)"] --> B{isModule} + B -- "no, and clients is empty" --> C["return ws unchanged"] + B -- "no, and clients is not empty" --> D["raise: standalone clients unsupported"] + B -- yes --> E{"scope already has\ndagger-module.toml\nor dagger.json?"} + E -- no --> F["overlay the starter template\n+ write dagger-module.toml\nfrom the manifest builder"] + E -- yes --> G["leave the scope's files alone"] + F --> H["replace the manifest's dependencies\nwith one entry per client"] + G --> H + H --> I{"existing module\nunder a skip marker?"} + I -- yes --> J["return without generating"] + I -- no --> K["vendor the SDK, generate the bindings\nand the entrypoint"] + J --> L["restore the scope cwd, return the workspace"] + K --> L +``` + +`dagger generate` never reaches the `isModule: false, clients empty` branch: the +engine's scope planner skips a scope that is neither a module nor a client +holder (`core/schema/workspace_sdk_generator.go`). The branch exists because +`generateScope` is also callable directly, which is how the end-to-end checks +drive it. + +Five properties of that flow are worth stating separately. + +**A new module is always generated.** The skip marker +(`.dagger-java-sdk-skip-generate`, at or above a module root) exists to keep +fixtures and vendored trees out of bulk regeneration. A module that was just +created has nothing to protect and everything to produce, so the marker only +holds for a scope that already had a config. + +**The manifest comes from the engine's builder.** `moduleManifest` builds and +serializes both manifest formats (`tomlFile`, `legacyJSONFile`) and edits +dependency entries (`withDependency`, `withoutDependency`). Dependency editing +is the part this repository would otherwise have to implement itself, and the +part it must not: rewriting an existing TOML manifest by hand means parsing and +re-emitting a format the engine owns. + +There is one wrinkle. The builder's runtime setters are one per builtin runtime +(`withLegacyJavaRuntime`, `withLegacyGoRuntime`, …) and they write the builtin +short name, so `withLegacyJavaRuntime` writes `source = "java"` — the engine's +own Java runtime. This SDK targets its own repository's build-and-package-only +runtime, `github.com/dagger/java-sdk/runtime`, for which the builder has no +setter. It does accept one path: `ModuleManifest.Validate` rejects a non-builtin +runtime only when the manifest was built from nothing, and accepts it when the +manifest was loaded from a config file. So a new module's manifest is built by +loading a seed `dagger-module.toml` that names the runtime, the module, and the +live engine version, then applying the client dependencies. An existing module's +manifest is loaded from the file the module already has. + +**Clients become dependencies.** In a module scope, the complete client set +replaces the module's dependency list: the manifest's dependencies are cleared +structurally with the builder's `withoutDependencies`, then one entry is added +per client — a git client by its ref as is, a local client by its path relative +to the module. Clearing by name would not do. `WithoutDependency` matches an +unnamed dependency on its *source*, and reading the recorded names means +selecting `ModuleSource.dependencies`, which resolves every one of them, so a +single stale or unreachable entry would fail generation instead of being +dropped. The entry is written back into the manifest file the module actually +has (`dagger-module.toml`, or the `dagger.json` of a pre-1.0 module that has +nothing else). Because the Java bindings are generated from the module's +introspection schema, and that schema includes its dependencies' types, this is +all it takes for a client's types to appear in the generated bindings. + +A manifest that already records exactly the requested clients is not rewritten, +so a hand-written one is not reformatted for nothing. That is decided by +comparing the builder's serialization of the loaded manifest against its +serialization of the configured one: both sides go through the same serializer, +so the comparison is formatting-neutral and never diffs against the bytes on +disk. + +**A module with dependencies and no clients loses those dependencies.** That is +the contrapositive of the rule above, it is deliberate, and it is the module-max +model: the client set *is* the dependency set. It is also a real migration +hazard, because the engine's own config migration records only `is-module` and +`name` on a scope and never seeds `clients` from an existing dependency list +(`core/workspace/migrate.go`). A module that has dependencies today therefore +needs each of them re-registered as a client before the first `dagger generate` +under the new interface. The README says so, and a fixture pins the behaviour. + +**Generation runs with the cwd at the workspace root.** The engine resolves a +module's local dependency to a workspace-root-relative path and then reads it +relative to `Workspace.cwd` (`ResolveDepToSource` in `core/modulesource.go`). +With the cwd at the scope, as it is on entry to `generateScope`, a dependency +`../../dep` of `mods/app` is looked up under `mods/app/../../dep` resolved from +`mods/app` — the wrong place. Moving the cwd to the workspace root for the work +and restoring the scope cwd on the result avoids it. The engine requires the +restore in any case: it rejects a `generateScope` result whose cwd is not the +scope. + +### What survives from `mod.dang` + +`Mod` holds everything that is not part of the engine interface: the Maven +codegen containers, the vendored SDK build, the entrypoint compilation, the skip +marker check, and the module-relative path arithmetic. None of it is touched by +#13992 and all of it is kept. + +Four changes are needed there: + +- `Mod.generateModule` selects the removed `ModuleSource.generateLocalDependencies` + to stage local dependencies before reading the module's introspection schema. + The engine now generates scopes in dependency order itself, so the staging + step is removed rather than replaced. +- `generateScope` must return a `Workspace`, not a `Changeset`. `Mod` gains + `generated: Workspace!` — the workspace with this module's generated files + merged in, mirroring `dagger/python-sdk#25` — and `generate: Changeset!` is + deleted rather than rewritten: `generateScope` applies the skip marker itself, + and nothing else called it. `Mod.path` and `Mod.hasMarker` go with it. +- `Mod` currently takes `ws` both as a constructor field and as an argument to + `generate` and `skipGenerate`. The two are always the same workspace at every + call site. The argument goes away, so `Mod` has one workspace. +- The unused `defaultModulePath` and `cleanModulePath` helpers, both left over + from the beta init contract, are deleted with it. + +`generateScope` constructs `Mod` directly from the scope the engine handed it. +Nothing needs to read the registered scope list, so this SDK never selects +`Workspace.sdk`. python-sdk does, because it keeps a public `mod` that resolves +a module by path; the corresponding fragility — the lookup keys on the SDK's +*install* name in `dagger.toml`, not on its SDK name — does not arise here. + +### Registration + +`dagger.toml` at the repository root gains: + +```toml +[modules.java-sdk] +source = "." +check.skip = ["*"] + +[sdks.java] +module = "java-sdk" +``` + +`[modules.java-sdk]` is required, not decorative: the engine rejects a +`[sdks.]` entry whose `module` is not an installed module. + +The end-to-end fixtures have their own nested workspace config, +`.dagger/modules/e2e/fixtures/dagger.toml`, which is where they are registered +today under `[modules.java-sdk.as-sdk]`. Each fixture moves to a +`[sdks.java.scopes.""]` block with `is-module = true` and the module's +`name`, both of which the engine requires for a module scope. + +Two registrations go away: + +- `[modules.sdk-sdk]` and the checks it contributes. `github.com/dagger/sdk-sdk` + validates the beta contract this change removes: it asserts that `initModule` + seeds files without writing config, that `dagger sdk install` writes an + `as-sdk` marker, that `dagger module deps list` works. Every one of those + statements is false after this change. `dagger/python-sdk#25` dropped the same + dependency. +- `[modules.dagger-dang-sdk.as-sdk]`, which registers this repository's own Dang + modules (the root module and `.dagger/modules/templates`) with the Dang SDK. + `as-sdk` is removed by #13992, and dang-sdk has not yet adopted the + replacement (`dagger/dang-sdk#13` is open), so there is no correct new form to + move this to. A stale `as-sdk` table would not fail — the engine's config + parser ignores unknown keys — but silently ignored configuration is worse than + no configuration. `[modules.dagger-dang-sdk]` itself stays installed. Nothing + is lost: Dang modules have no generated files to produce, and + `.dagger/modules/templates` keeps its own `@generate` hook, registered as an + ordinary module. + +## Testing + +### What the released engine can and cannot do + +Dang infers a whole program on each call into a module. `generateScope` selects +`moduleManifest`, which the released engine `v1.0.0-beta.11` does not have, so +on that engine every call into this module fails — not only the calls that reach +the manifest builder. + +A call into the module is the only thing that fails. This was measured, not +assumed. On `v1.0.0-beta.11`, in a scratch workspace with two Dang modules where +module `ok` depends on module `bad`, and `bad` has one function selecting +`moduleManifest`: + +- the workspace loads and `dagger check` enumerates every check; +- `ok:independent`, which does not touch `bad`, passes; +- `ok:touches-bad`, which selects one unrelated field of `bad`, fails with + `"moduleManifest" not found`; +- with `check.skip = ["*"]` on the module that owns a failing check, the run is + green. + +So `check.skip` is sufficient to keep the released-engine run green, and +`dagger call` is unaffected by it: skip patterns are read only by the `checks` +resolver. + +`dagger/python-sdk#25` did not use that. Its merged head +(`c05426e0fbef5d758184667d62ddc406591b8192`) carries 19 commit statuses, of +which 11 are red: every `e-2-e:*` check that calls the python-sdk module. It has +one green development-engine check, `engine-e-2-e:dev-sdk-check`, an +initialization smoke test. Its remaining new-interface checks were run by hand +in a development engine and are not covered by CI at all. + +### Two engines, two check sets + +| Where | Engine | What it covers | +| --- | --- | --- | +| `e-2-e:*` that do not call this module, `packager:*`, `templates:generate` | released, `v1.0.0-beta.11` | the SDK library build, its unit tests, the prebuilt assets, the templates | +| `engine-e-2-e:*` | built from `sdk-ux-module-max` at `78c241b6ce5f950461c74811e768f92e946e2ca7` | the whole `detectScope` / `generateScope` contract | + +`[modules.e2e] check.skip = ["*"]` keeps a released-engine `dagger check` from +attempting them. Every check the module has calls this SDK, so a wildcard says +exactly what a list of all of them would, and it also covers the next one +somebody adds. + +A new module, `.dagger/modules/engine-e2e`, depends on +`github.com/dagger/dagger/.dagger/modules/engine-dev` pinned to that same +commit, builds the engine from it, and runs it as a playground container with +this checkout mounted inside. + +`engine-e-2-e:dev-sdk-check` is the deliverable, and mirrors python-sdk's: + +1. `dagger sdk list` reports `java`. This proves the registration parses; it + loads no module, so it proves nothing more. +2. `dagger module init java --name … --path …` succeeds. This is the check that + proves the interface: it loads the SDK module, validates its function + signatures against `core/sdkmodule/provider.go`, and calls `generateScope`. + It then asserts the files that call produced — the manifest, the `pom.xml`, + the module class, and the generated bindings. +3. `dagger call` against the initialized module proves the generated module + builds with Maven and serves its API. + +A second check, `engine-e-2-e:sdk-contract-check`, replays the gated `e-2-e` +checks inside the same playground as ordinary `dagger call e-2-e ` +invocations, which the skip list does not suppress. This is coverage +python-sdk#25 does not have. It is also the expensive part: the playground +engine starts with cold Maven caches, and this SDK installs its jars under a +per-module Maven version on purpose, so two checks with different module names +share no build. + +That cost was measured rather than guessed, on a developer machine with a warm +outer engine. `engine-e-2-e:dev-sdk-check` takes 6m23s including building the +engine from source. `engine-e-2-e:sdk-contract-check`, replaying all six gated +checks, takes 7m16s on a first full pass and 3m29s once the vendored SDK build +is cached. Both fit, so `sdk-contract-check` replays all six rather than the +subset this design first proposed. CI starts colder than this and will be +slower; the figures bound the shape of the cost, not its exact value. + +Client handling stays inside Dang throughout: the checks call +`javaSdk.generateScope(...)` and diff the result, and never go through +`dagger module client add`. That CLI command is broken on `sdk-ux-module-max` at +`78c241b6ce5f950461c74811e768f92e946e2ca7` — it loses the workspace overlay on +reload and silently writes nothing, on every SDK — and the fault is in the CLI +(`internal/cmd/dagger/module_sdk.go`), not in any SDK's `generateScope`. +python-sdk's checks avoid it the same way. + +### Known gaps + +Two behaviours ship unchecked, deliberately: + +- `dependencySource`'s `GIT_SOURCE` arm. Recording a git client by its ref as is + is not exercised by any check: a git `ModuleSource` needs a real remote, which + no check here can produce hermetically. +- A full generation of a pre-1.0 `dagger.json` module. + `generate-scope-clients-check` drives the `loadJSON` / `legacyJSONFile` + manifest branch on the `generate/app` fixture, but that fixture is under the + skip marker, so the Maven half of the path — vendoring and generating into a + module whose config is `dagger.json` — is never run. + +### The engine pin + +Both the `engine-dev` dependency and the engine source are pinned to +`78c241b6ce5f950461c74811e768f92e946e2ca7`, so CI does not float with a branch +that force-pushes. Bumping the branch means bumping both, plus `dagger.lock`. + +## Alternatives considered + +**Keep the SDK loadable on the released engine.** `generateScope` could render +`dagger-module.toml` as a string instead of selecting `moduleManifest`, and the +module would keep working on `v1.0.0-beta.11`. Rejected: replacing the complete +dependency set of an *existing* manifest means parsing and rewriting TOML, which +Dang cannot do and which would put manifest editing back into this repository. +Seeding a manifest is a different matter — the seed for a *new* module is +hand-rendered TOML, because the builder has no setter for this SDK's runtime — +but a fixed three-key seed is not a TOML editor. The other three SDKs all take +the builder. + +**Regenerate the manifest from scratch instead of loading it.** `dagger/dang-sdk#13` +builds each manifest from a fresh builder and does not merge existing content, +for deterministic output. Rejected here: a Java module's manifest can carry +`include` paths and settings that this SDK did not write and has no business +dropping. + +**Migrate the Dang SDK registration to `[sdks.dang]` at the same time.** It +would keep this repository's Dang modules registered with an SDK. Rejected: it +names dang-sdk as the provider of an interface dang-sdk does not implement yet, +so it would fail on the very engine it is meant to serve. + +**Use `withLegacyJavaRuntime` and accept the engine's builtin `java` runtime.** +Rejected: it would silently move every newly created module off this +repository's runtime and onto the engine's, undoing the self-contained layout +that is the point of this SDK. + +**Run the released-engine CI with an explicit include list +(`dagger check packager:* templates:*`) instead of a skip list.** Rejected: this +repository has no CI configuration of its own, so the `dagger check` invocation +is not ours to change, and the measurement above shows a skip list is enough. + +## Affected components + +| Path | Change | +| --- | --- | +| `main.dang`, `main.dang.tmpl` | `detectScope`, `generateScope`; `initModule`, `targetRuntime`, `modules`, `generateAll` removed | +| `mod.dang` | `generated: Workspace!`; local-dependency staging removed; single workspace field; dead init helpers removed | +| `dagger.toml` | `[modules.java-sdk]`, `[sdks.java]`, `[modules.engine-e2e]`, `[modules.e2e] check.skip`; `[modules.sdk-sdk]` and the `as-sdk` block removed | +| `dagger.lock` | the `engine-dev` dependency closure | +| `.dagger/modules/e2e/fixtures/dagger.toml` | `[sdks.java]` with one scope per fixture | +| `.dagger/modules/e2e/fixtures/**` | a `pom.xml` per fixture module; a client fixture | +| `.dagger/modules/e2e/main.dang` | checks rewritten against the new interface | +| `.dagger/modules/engine-e2e/` | new: builds the branch engine and checks against it | +| `README.md` | new command shapes, the module-scope model, the dependency migration step | + +## Risks + +- **Existing modules lose their dependencies on the first generate.** Described + under "Clients become dependencies" above. Mitigated by a README migration + step and a fixture, not by code: re-deriving clients from an existing + dependency list is the engine's migration to make, not this SDK's. +- **The branch moves.** `sdk-ux-module-max` force-pushes. The pin makes CI + reproducible, but it also means the checks validate a commit, not the branch + head. A later engine change can break this SDK without CI noticing until the + pin is bumped. +- **Nested Java builds are slow.** The `engine-e2e` checks run Maven inside a + development engine inside the outer engine, with cold caches. Java code + generation is the heaviest operation this repository has. The measurement + under "Two engines, two check sets" says the current set fits. A check added + later that generates under a *new module name* pays for a whole vendored SDK + build of its own, because this SDK installs its jars under a per-module Maven + version on purpose; re-measure when one is added. +- **The seed-manifest path depends on a validation detail.** Loading a config + file is what lets a non-builtin runtime through `ModuleManifest.Validate`. If + the engine later rejects non-builtin runtimes outright, new Java modules can + no longer name `github.com/dagger/java-sdk/runtime`, and this SDK needs a + builder API for an arbitrary runtime source. That is worth raising on #13992 + independently of this change. +- **`dagger module client add` is broken on the branch.** Client handling is + therefore verified at the API level only. When the CLI is fixed, the + playground checks should drive it end to end. +- **Standalone clients are refused.** A user who runs `dagger module client add` + from a directory that is not a Java module gets an error rather than a + generated client. This matches python-sdk, and is the honest answer while the + Java SDK has nowhere to put such a client. + +## Implementation plan + +Four commits. Each one leaves the tree in a state that loads, and none of them +registers a module whose source does not yet exist. + +### 1. `java-sdk: implement the module-max SDK interface` + +The interface cutover is one commit because its parts cannot be separated: the +moment `main.dang` drops `initModule`, the end-to-end module that calls it stops +compiling, and the moment `main.dang` selects `moduleManifest`, the released +engine needs the skip list. + +`main.dang` and `main.dang.tmpl` — kept identical apart from the template +placeholder, as they are today: + +- Remove `targetRuntime`, `initModule`, `modules`, and `generateAll`. +- Keep the runtime source as a private `let`. It is still needed to seed a new + module's manifest; it is simply no longer an engine interface function. +- Add the `template: String! = "default"` constructor setting, so + `dagger module init java --template legacy` reaches the template selection + that `initModule`'s `template` argument used to carry. +- Add `detectScope(ws)`: `ws.findUp("pom.xml")`, trimmed to its directory and + normalized, `""` when absent. +- Add `generateScope(ws, isModule, name, clients)` per the flow above. +- Add private helpers: `moduleManifestFor` (seed or load), + `withClientDependencies`, `dependencySource`, `relativePath`, `normalizePath`, + `scopeHasFile`, `hasModuleConfig`, and `mod`. +- Keep `skipGenerateFilename`, `vendorSdkJar`, and `renderedTemplate`. + +`mod.dang`: + +- Add `generated: Workspace!`; rewrite `generate` as the skip-marker wrapper. +- Drop the `generateLocalDependencies` staging from `generateModule`. +- Drop the `ws` arguments from `generate` and `skipGenerate`, leaving the field. +- Anchor `skipGenerate`'s `findUp` at `"/" + rootPath`, so the marker is looked + up from the workspace root rather than from the cwd. +- Delete the unused `defaultModulePath` and `cleanModulePath` helpers. + +`.dagger/modules/e2e/main.dang`: + +| Check | Replaces | Asserts | +| --- | --- | --- | +| `detect-scope-check` | `modules-check`, `modules-cwd-check` | the nearest `pom.xml` wins; a nested directory resolves to its module; a directory with no `pom.xml` above it gives `""` | +| `generate-scope-init-check` | `init-check`, `init-existing-check`, part of `generate-cwd-check` | a config-less scope gets the template, a `dagger-module.toml` naming this repository's runtime, and generated bindings; existing files survive; the cwd is unchanged; regenerating the module it just created changes nothing | +| `generate-scope-clients-check` | new | a client is recorded as a dependency in the manifest the module has, `dagger-module.toml` or a pre-1.0 `dagger.json`, and removing it drops the entry again; a module with dependencies and no clients has them dropped; a new module generated with a client carries the client's type in its vendored bindings; a scope with no module is untouched; standalone clients raise | +| `generate-scope-skip-check` | new | the skip marker holds an existing module and does not hold a new one | +| `nullable-return-check` | itself | unchanged behaviour, driven through `generateScope` | +| `skip-generate-filename-check` | itself | unchanged | + +Fixtures: + +- `.dagger/modules/e2e/fixtures/dagger.toml` moves to `[sdks.java]` + scopes. +- Add `clients/dep`, a small Dang module used as a client. +- Add a `pom.xml` to the fixture modules `detect-scope-check` reads, so they are + Java scopes rather than config-only stubs. +- Keep `deps/app`, repurposed: it is the module with a recorded dependency and + no clients, and it pins the dependency-dropping behaviour. +- Keep `generate/app`, repurposed: it is the pre-1.0 module, and its `dagger.json` + records the same dependency, so the `loadJSON` / `legacyJSONFile` manifest + branch has a check. + +Root `dagger.toml`: add `[modules.java-sdk]`, `[sdks.java]`, and +`[modules.e2e] check.skip`; remove `[modules.sdk-sdk]` and the `as-sdk` block. + +### 2. `e2e: check the SDK against an engine built from sdk-ux-module-max` + +Add `.dagger/modules/engine-e2e`, its `[modules.engine-e2e]` registration, and +the regenerated `dagger.lock`. The `engine-dev` dependency and the engine source +both name `78c241b6ce5f950461c74811e768f92e946e2ca7`. + +### 3. `README: document the module-scope model` + +Rewrite the install, create, generate, and client sections around +`dagger module install`, `dagger module init java --name … --path …`, and +`dagger generate`. State that a pre-1.0 `dagger.json` module keeps the runtime it +already names, that an existing module's dependencies must be re-registered as +clients before the first generate, that a directory with no `pom.xml` is not a +Java scope, and that this SDK needs an engine with #13992. + +### 4. `hack/designs: archive the module-max SDK interface design` (after CI is green) + +Move this document to `hack/designs/done/`. Move +`hack/designs/2026-08-17-nullable-object-returns.md` there too: it is +implemented, it still reads as proposed, and it describes verification through +`generateAll` and `sdk-sdk:*`, both of which this change removes. A short note +records what replaced them. + +### Test strategy + +- `dagger check` on the released engine for the ungated checks, locally and in + CI. +- `dagger check engine-e-2-e:dev-sdk-check` locally before handing off; this + builds the branch engine and is the slow path. +- Measure `engine-e-2-e:sdk-contract-check` before committing to its contents. From d8d5de29cdff873703c5f6ecdc3f640063dc63b3 Mon Sep 17 00:00:00 2001 From: Yves Brissaud Date: Sat, 5 Sep 2026 00:29:10 +0200 Subject: [PATCH 2/9] java-sdk: implement the module-max SDK interface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit dagger/dagger#13992 replaces the beta SDK-module interface with detectScope and generateScope, and removes CurrentModule.asSDK, ModuleSource.generateLocalDependencies, initModule, targetRuntime and the @generate hook. It ships no compatibility adapter, so every public function of this module selected something that no longer exists. detectScope answers with the directory of the nearest pom.xml. Every Java module has one at its root, and the SDK vendored under a module carries none of its own, so the nearest hit is always the owning module — python-sdk has to lift a hit out of its vendored sdk/, Java does not. generateScope scaffolds a scope that has no config from the template and writes its dagger-module.toml through the engine's manifest builder, then generates it; a scope that already has a config is only generated. A new module is generated even under a generate skip marker, which only holds a module that already exists. The builder has one runtime setter per builtin runtime, and this SDK targets github.com/dagger/java-sdk/runtime, so the runtime is named by loading a seed file: ModuleManifest.Validate accepts a non-builtin runtime on a manifest loaded from a config file and rejects it on one built from nothing. In a module scope the client set becomes the module's dependency set. The manifest's dependencies are cleared structurally, with the builder's withoutDependencies, rather than by name: withoutDependency matches an unnamed dependency on its source, and reading the recorded names means resolving every one of them, so a single stale entry would fail generation instead of being dropped. A manifest that already records exactly the requested clients is left alone, decided by comparing the builder's rendering of both sides, so a hand-written one is never reformatted for nothing. That cuts both ways — a module that records dependencies and has no clients loses them — so the fixtures pin it, in both manifest formats. Standalone clients, in a scope without a module, are refused, as python-sdk refuses them: every generated binding lives under a module's sdk/ and is compiled by that module's pom.xml, so there is nowhere to put one. Mod keeps its Maven machinery and gains `generated`, the workspace with the module's vendored SDK and entrypoint merged in. Its local-dependency staging goes: the engine orders scope generation by dependency itself. `generate`, `path` and `hasMarker` go with the interface they served. The SDK registers under [sdks.java] with one scope per fixture, and the e2e checks move to the new interface. They call this module, so on the released engine every one of them fails; [modules.e2e] check.skip covers them. sdk-sdk goes with the contract it validates: it asserts initModule, the as-sdk marker and `dagger module deps list`, none of which survive. The dang-sdk as-sdk block goes too, with no replacement: dang-sdk has not adopted the new interface yet (dagger/dang-sdk#13), and this repository's Dang modules have no generated files to produce. Signed-off-by: Yves Brissaud --- .../fixtures/clients/dep/dagger-module.toml | 5 + .../e2e/fixtures/clients/dep/main.dang | 6 + .dagger/modules/e2e/fixtures/dagger.toml | 21 - .../e2e/fixtures/deps/app/dagger-module.toml | 9 + .../modules/e2e/fixtures/deps/app/dagger.json | 7 - .../e2e/fixtures/generate/app/dagger.json | 8 +- .../e2e/fixtures/init-existing/keep.txt | 1 - .../modules/e2e/fixtures/lookup/app/pom.xml | 10 + .dagger/modules/e2e/main.dang | 419 ++++++++++-------- dagger.toml | 47 +- main.dang | 277 +++++++++--- main.dang.tmpl | 277 +++++++++--- mod.dang | 112 +---- 13 files changed, 746 insertions(+), 453 deletions(-) create mode 100644 .dagger/modules/e2e/fixtures/clients/dep/dagger-module.toml create mode 100644 .dagger/modules/e2e/fixtures/clients/dep/main.dang delete mode 100644 .dagger/modules/e2e/fixtures/dagger.toml create mode 100644 .dagger/modules/e2e/fixtures/deps/app/dagger-module.toml delete mode 100644 .dagger/modules/e2e/fixtures/deps/app/dagger.json delete mode 100644 .dagger/modules/e2e/fixtures/init-existing/keep.txt create mode 100644 .dagger/modules/e2e/fixtures/lookup/app/pom.xml diff --git a/.dagger/modules/e2e/fixtures/clients/dep/dagger-module.toml b/.dagger/modules/e2e/fixtures/clients/dep/dagger-module.toml new file mode 100644 index 0000000..5f67e9a --- /dev/null +++ b/.dagger/modules/e2e/fixtures/clients/dep/dagger-module.toml @@ -0,0 +1,5 @@ +name = "client-dep" +engineVersion = "v1.0.0-0" + +[runtime] + source = "dang" diff --git a/.dagger/modules/e2e/fixtures/clients/dep/main.dang b/.dagger/modules/e2e/fixtures/clients/dep/main.dang new file mode 100644 index 0000000..50f1615 --- /dev/null +++ b/.dagger/modules/e2e/fixtures/clients/dep/main.dang @@ -0,0 +1,6 @@ +""" +A module the e2e checks add as a client. +""" +type ClientDep { + pub greeting: String! { "hello from the client dependency" } +} diff --git a/.dagger/modules/e2e/fixtures/dagger.toml b/.dagger/modules/e2e/fixtures/dagger.toml deleted file mode 100644 index bc02768..0000000 --- a/.dagger/modules/e2e/fixtures/dagger.toml +++ /dev/null @@ -1,21 +0,0 @@ -[modules.java-sdk] -source = "../../../.." -check.skip = ["*"] - -[modules.java-sdk.as-sdk] -name = "java" - -[[modules.java-sdk.as-sdk.modules]] -path = "generate/app" - -[[modules.java-sdk.as-sdk.modules]] -path = "lookup/app" - -[[modules.java-sdk.as-sdk.modules]] -path = "deps/app" - -[[modules.java-sdk.as-sdk.modules]] -path = "skip/app" - -[[modules.java-sdk.as-sdk.modules]] -path = "managed-toml/app" diff --git a/.dagger/modules/e2e/fixtures/deps/app/dagger-module.toml b/.dagger/modules/e2e/fixtures/deps/app/dagger-module.toml new file mode 100644 index 0000000..1cb2fda --- /dev/null +++ b/.dagger/modules/e2e/fixtures/deps/app/dagger-module.toml @@ -0,0 +1,9 @@ +name = "deps-app" +engineVersion = "v1.0.0-0" + +[runtime] + source = "java" + +[[dependencies]] + name = "client-dep" + source = "../../clients/dep" diff --git a/.dagger/modules/e2e/fixtures/deps/app/dagger.json b/.dagger/modules/e2e/fixtures/deps/app/dagger.json deleted file mode 100644 index adb4387..0000000 --- a/.dagger/modules/e2e/fixtures/deps/app/dagger.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "name": "deps-app", - "engineVersion": "latest", - "sdk": { - "source": "java" - } -} diff --git a/.dagger/modules/e2e/fixtures/generate/app/dagger.json b/.dagger/modules/e2e/fixtures/generate/app/dagger.json index 34d9f00..99cab32 100644 --- a/.dagger/modules/e2e/fixtures/generate/app/dagger.json +++ b/.dagger/modules/e2e/fixtures/generate/app/dagger.json @@ -3,5 +3,11 @@ "engineVersion": "latest", "sdk": { "source": "java" - } + }, + "dependencies": [ + { + "name": "client-dep", + "source": "../../clients/dep" + } + ] } diff --git a/.dagger/modules/e2e/fixtures/init-existing/keep.txt b/.dagger/modules/e2e/fixtures/init-existing/keep.txt deleted file mode 100644 index a7107ad..0000000 --- a/.dagger/modules/e2e/fixtures/init-existing/keep.txt +++ /dev/null @@ -1 +0,0 @@ -kept by init diff --git a/.dagger/modules/e2e/fixtures/lookup/app/pom.xml b/.dagger/modules/e2e/fixtures/lookup/app/pom.xml new file mode 100644 index 0000000..b300bb9 --- /dev/null +++ b/.dagger/modules/e2e/fixtures/lookup/app/pom.xml @@ -0,0 +1,10 @@ + + + + 4.0.0 + io.dagger.modules.lookupapp + lookup-app + 1.0-SNAPSHOT + diff --git a/.dagger/modules/e2e/main.dang b/.dagger/modules/e2e/main.dang index 06234e4..9ee25de 100644 --- a/.dagger/modules/e2e/main.dang +++ b/.dagger/modules/e2e/main.dang @@ -1,19 +1,44 @@ """ End-to-end checks for the Java SDK helper module. + +Every check here calls the java-sdk module, which needs an engine with +dagger/dagger#13992. They are listed in `[modules.e2e] check.skip` so a released +engine does not attempt them, and `engine-e-2-e:sdk-contract-check` runs them as +`dagger call` invocations inside an engine built from that change. """ type E2e { + let fixtureRoot: String! = ".dagger/modules/e2e/fixtures" + # Scratch scopes, created in memory. Outside fixtureRoot, which carries a + # generate skip marker. let outputRoot: String! = ".dagger/modules/e2e/out" - let fixtureRoot: String! = ".dagger/modules/e2e/fixtures" - let generateModulePath: String! = fixtureRoot + "/generate/app" + # The Java identifiers in nullableReturnSource — package + # io.dagger.modules.newapp, class NewApp — are derived from this name by hand + # and have to be changed with it. + let newModuleName: String! = "new-app" + let newModulePath: String! = outputRoot + "/" + newModuleName + let clientAppPath: String! = outputRoot + "/client-app" + + # managed-toml/app, deps/app, generate/app and skip/app are config-only stubs: + # no pom.xml and no sources. The fixture-wide skip marker is what keeps + # generateScope from trying to build them. + + # A scope with no config of its own under the fixture-wide skip marker: the + # marker must not hold a module that does not exist yet. + let markedModulePath: String! = fixtureRoot + "/new/app" let lookupModulePath: String! = fixtureRoot + "/lookup/app" let lookupNestedPath: String! = lookupModulePath + "/nested" - let depsModulePath: String! = fixtureRoot + "/deps/app" + # A module using a different SDK, with no pom.xml: never a Java scope. + let nonJavaModulePath: String! = fixtureRoot + "/lookup/not-java" let skipModulePath: String! = fixtureRoot + "/skip/app" # A CLI 1.0 managed module: configured by dagger-module.toml, not dagger.json. let managedTomlModulePath: String! = fixtureRoot + "/managed-toml/app" - # A module using a different SDK; this SDK must never manage it. - let nonJavaModulePath: String! = fixtureRoot + "/lookup/not-java" + # A pre-1.0 module: configured by dagger.json, which is also where its + # dependencies are recorded. + let legacyJSONModulePath: String! = fixtureRoot + "/generate/app" + # A module that records a dependency and has no clients. + let depsModulePath: String! = fixtureRoot + "/deps/app" + let clientDepPath: String! = fixtureRoot + "/clients/dep" """ Fail the current check when a condition is false. @@ -45,230 +70,240 @@ type E2e { } """ - New Java modules should target this repository's build/package-only runtime - rather than a builtin SDK name. + Assert that a string contains every substring. """ - targetRuntimeCheck: Void @check { - assert( - javaSdk.targetRuntime == "github.com/dagger/java-sdk/runtime", - "targetRuntime should be the in-repo build/package-only runtime", - ) + let assertContainsAll(value: String!, wants: [String!]!): Void { + wants.each { want => assertContains(value, want, "expected to contain: " + want) } null } """ - The generate skip marker filename should be stable. + Assert that a string does not contain a substring. """ - skipGenerateFilenameCheck: Void @check { - assert( - javaSdk.skipGenerateFilename == ".dagger-java-sdk-skip-generate", - "skip marker filename changed", - ) - null + let assertNotContains(value: String!, want: String!, message: String!): Void { + assert(value.contains(want) == false, message) } """ - initModule should seed only the SDK-owned template files: a two-pass pom - defaulting dagger.proc=none and the module sources, with the module name - substituted into paths and contents. Engine-owned files (dagger-module.toml, - the legacy dagger.json) are produced by the engine, not by initModule, and no - existing files are touched. + detectScope answers with the directory of the nearest pom.xml, as a path + relative to the workspace root: the module itself from inside it, the owning + module from a subdirectory, the owning module from inside its vendored SDK, + and nothing where no pom.xml is in scope. """ - initCheck(ws: Workspace!): Void @check { - let p = outputRoot + "/init-default" - let changes = javaSdk.initModule(ws, name: "init-default", path: p) - - assertAdded(changes, p + "/pom.xml") - assertAdded(changes, p + "/src/main/java/io/dagger/modules/initdefault/InitDefault.java") - assertAdded(changes, p + "/src/main/java/io/dagger/modules/initdefault/package-info.java") - + detectScopeCheck(ws: Workspace!): Void @check { assert( - contains(changes.addedPaths, p + "/dagger-module.toml") == false, - "initModule should not write the engine-owned dagger-module.toml", + javaSdk.detectScope(ws.withWorkdir(lookupModulePath)) == lookupModulePath, + "detectScope should answer with the module root itself", ) assert( - contains(changes.addedPaths, p + "/dagger.json") == false, - "initModule should not write the engine-owned dagger.json", + javaSdk.detectScope(ws.withWorkdir(lookupNestedPath)) == lookupModulePath, + "detectScope should find the module owning a nested path", ) - assert(changes.modifiedPaths.length == 0, "init unexpectedly modified existing files") - assert(changes.removedPaths.length == 0, "init unexpectedly removed files") - let pom = changes.layer.file(p + "/pom.xml").contents - assertContains(pom, "none", "pom should default dagger.proc to none") - assertContains(pom, "init-default", "pom artifactId should be the kebab-case module name") + # A module's vendored SDK has no pom.xml of its own, so the module answers + # for it; nothing has to lift a hit out of sdk/ the way python-sdk does. + let vendored = lookupModulePath + "/sdk/src/main/java" + let withVendoredSdk = ws.withNewFile("/" + vendored + "/.keep", "") + assert( + javaSdk.detectScope(withVendoredSdk.withWorkdir(vendored)) == lookupModulePath, + "detectScope should answer with the module owning a vendored sdk/", + ) - let module = changes.layer.file(p + "/src/main/java/io/dagger/modules/initdefault/InitDefault.java").contents - assertContains(module, "class InitDefault", "module class should be the camel-case module name") + assert( + javaSdk.detectScope(ws.withWorkdir(nonJavaModulePath)) == "", + "a module of another SDK, with no pom.xml, is not a Java scope", + ) + assert( + javaSdk.detectScope(ws.withWorkdir(".")) == "", + "the workspace root is not a Java scope", + ) null } """ - initModule onto a directory that already holds files should lay the template - over them and leave them in place: a user running init inside an existing - project must not lose their work. - """ - initExistingCheck(ws: Workspace!): Void @check { - let p = fixtureRoot + "/init-existing" - let changes = javaSdk.initModule(ws, name: "init-existing", path: p) - - assertAdded(changes, p + "/pom.xml") - assert(changes.removedPaths.length == 0, "init removed a file already in the target directory") - - null - } - - let testWS(ws: Workspace!): Workspace! { - ws.directory("/").withoutFile("dagger.toml").asWorkspace(cwd: ".dagger/modules/e2e/fixtures") - } + generateScope initializes a scope that has no config: the template, a + dagger-module.toml from the engine's manifest builder naming this repository's + runtime, and the generated bindings and entrypoint, all under the scope, with + the workspace cwd and the files already there untouched. + Generating the result again changes nothing: an existing module is generated + in place, its manifest is not rewritten, and generation is reproducible. """ - From the workspace root the whole workspace is in scope, so modules() should - return every Java SDK module this workspace manages — whether it is configured - by the legacy dagger.json or the CLI 1.0 dagger-module.toml — and nothing that - isn't managed by this SDK (e.g. a sibling module using another SDK). - """ - modulesCheck(ws: Workspace!): Void @check { - let pathRecords = javaSdk.modules(testWS(ws)).{{rootPath}} + generateScopeInitCheck(ws: Workspace!): Void @check { + let scoped = ws + .withNewFile("/" + newModulePath + "/keep.txt", "keep\n") + .withWorkdir(newModulePath) + let generated = javaSdk.generateScope(scoped, isModule: true, name: newModuleName, clients: []) + let changes = generated.withWorkdir(".").changes(scoped.withWorkdir(".")) + assert(generated.cwd == scoped.cwd, "generateScope must not change the workspace cwd") + + assertAdded(changes, newModulePath + "/dagger-module.toml") assert( - pathRecords.filter { r => r.rootPath == lookupModulePath }.length > 0, - "lookup Java module should be listed", - ) - assert( - pathRecords.filter { r => r.rootPath == skipModulePath }.length > 0, - "skip-marked Java module should still be listed (skip only affects generate)", - ) - assert( - pathRecords.filter { r => r.rootPath == generateModulePath }.length > 0, - "generate Java module should be listed", - ) - assert( - pathRecords.filter { r => r.rootPath == depsModulePath }.length > 0, - "deps Java module should be listed", + contains(changes.addedPaths, newModulePath + "/dagger.json") == false, + "initializing a scope must not write a dagger.json", ) + assertAdded(changes, newModulePath + "/pom.xml") + assertAdded(changes, newModulePath + "/src/main/java/io/dagger/modules/newapp/NewApp.java") + assertAdded(changes, newModulePath + "/sdk/src/main/java/io/dagger/client/Dagger.java") + assertAdded(changes, newModulePath + "/src/generated/java/io/dagger/gen/entrypoint/Entrypoint.java") assert( - pathRecords.filter { r => r.rootPath == managedTomlModulePath }.length > 0, - "a dagger-module.toml (CLI 1.0) managed module should be listed", + changes.modifiedPaths.length == 0, + "initializing a scope modified existing files: " + changes.modifiedPaths.join(", "), ) assert( - pathRecords.filter { r => r.rootPath == nonJavaModulePath }.length == 0, - "a module not managed by this SDK should be excluded from modules listing", + changes.removedPaths.length == 0, + "initializing a scope removed existing files: " + changes.removedPaths.join(", "), ) + assertContainsAll(changes.after.file(newModulePath + "/dagger-module.toml").contents, [ + "name = \"" + newModuleName + "\"", + "engineVersion = \"", + "[runtime]", + "source = \"github.com/dagger/java-sdk/runtime\"", + ]) + assertContainsAll(changes.after.file(newModulePath + "/pom.xml").contents, [ + "none", + "" + newModuleName + "", + ]) + + let again = javaSdk + .generateScope(generated, isModule: true, name: newModuleName, clients: []) + .withWorkdir(".") + .changes(generated.withWorkdir(".")) + assert(again.isEmpty, "regenerating an unchanged module should change nothing") + null } """ - Discovery is anchored at the client's cwd, not the workspace root. Re-anchoring - the workspace to a subdirectory scopes modules() to the managed modules in that - cone (walk-down) plus the nearest enclosing one (find-up), and excludes managed - modules that live outside it. + In a module scope the client set becomes the module's dependency set: a client + is recorded in the manifest the module actually has, dagger-module.toml or the + pre-1.0 dagger.json; a client that is no longer requested is dropped; and a + module that records a dependency and has no clients loses it. A scope without a + module is left alone, and standalone clients are refused rather than silently + skipped. Generating a new module with a client puts the client's types in that + module's bindings, which is what recording the dependency is for. """ - modulesCwdCheck(ws: Workspace!): Void @check { - # A stable snapshot of the workspace, re-anchorable at any cwd. It has to - # carry the fixture dagger.toml, which is where currentModule.asSDK reads the - # managed-module list from, and lookup/app/nested, the config-less - # subdirectory the find-up case is anchored in. - let root = testWS(ws).directory("/") - - # Walk-down: from fixtures/generate only generate/app is in the cone; the - # sibling managed modules live outside it and must be excluded. Its - # cwd-relative path is "app" — a directory beneath the cwd. - let fromGenerate = javaSdk.modules( - root.asWorkspace(cwd: fixtureRoot + "/generate"), - ) - let fromGenerateRoots = fromGenerate.{{rootPath}} + generateScopeClientsCheck(ws: Workspace!): Void @check { + let root = ws.withWorkdir(".") + let scoped = ws.withWorkdir(managedTomlModulePath) + let manifestPath = managedTomlModulePath + "/dagger-module.toml" + let client = ws.moduleSource("/" + clientDepPath) + + let untouched = javaSdk + .generateScope(scoped, isModule: false, name: "managed-toml-app", clients: []) + .withWorkdir(".") + .changes(root) + assert(untouched.isEmpty, "a scope without a module should not be generated") + + let refused = javaSdk + .generateScope(scoped, isModule: false, name: "managed-toml-app", clients: [client]) + .cwd rescue "raised" + assert(refused == "raised", "generateScope should refuse standalone clients") + + # Manifest contents are read from the returned workspace, not from the + # changeset: a changeset carries only the paths that changed, and a manifest + # that already records exactly the requested clients is deliberately left + # alone, so `after` would not have the file at all. + let withClient = javaSdk.generateScope(scoped, isModule: true, name: "managed-toml-app", clients: [client]) + let added = withClient.withWorkdir(".").changes(root) + assert(contains(added.modifiedPaths, manifestPath), "adding a client should record it in the manifest") + assertContainsAll(withClient.file("/" + manifestPath).contents, [ + "[[dependencies]]", + "name = \"client-dep\"", + "source = \"../../clients/dep\"", + ]) assert( - fromGenerateRoots.filter { r => r.rootPath == generateModulePath }.length > 0, - "cwd=generate: the managed module in the cone should be discovered", + contains(added.addedPaths, managedTomlModulePath + "/dagger.json") == false, + "adding a client should not add a manifest the module did not have", ) - assert( - fromGenerateRoots.filter { r => r.rootPath == lookupModulePath }.length == 0, - "cwd=generate: lookup/app is outside the cone and must be excluded", - ) - assert( - fromGenerateRoots.filter { r => r.rootPath == depsModulePath }.length == 0, - "cwd=generate: deps/app is outside the cone and must be excluded", - ) - assert( - fromGenerateRoots.filter { r => r.rootPath == skipModulePath }.length == 0, - "cwd=generate: skip/app is outside the cone and must be excluded", + + let withoutClient = javaSdk.generateScope(withClient, isModule: true, name: "managed-toml-app", clients: []) + assertNotContains( + withoutClient.file("/" + manifestPath).contents, + "client-dep", + "removing the last client should drop the dependency", ) - assert(fromGenerateRoots.length == 1, "cwd=generate: exactly one managed module is in the cone") - assert(fromGenerate.{{path}}.filter { r => r.path == "app" }.length > 0, "cwd=generate: the discovered module's path should be cwd-relative (app)") - - # Find-up: from inside lookup/app (a nested subdir with no config of its own) - # the enclosing managed module is discovered; siblings are not. Its - # cwd-relative path is ".." — an ancestor of the cwd. - let fromNested = javaSdk.modules(root.asWorkspace(cwd: lookupNestedPath)) - let fromNestedRoots = fromNested.{{rootPath}} - assert( - fromNestedRoots.filter { r => r.rootPath == lookupModulePath }.length > 0, - "cwd=lookup/app/nested: find-up should discover the enclosing lookup/app", + + # The client set is the whole dependency set, so a module that records a + # dependency and has no clients loses it. Migrating a module therefore means + # re-registering each of its dependencies as a client. + let dropped = javaSdk.generateScope(ws.withWorkdir(depsModulePath), isModule: true, name: "deps-app", clients: []) + assertNotContains( + dropped.file("/" + depsModulePath + "/dagger-module.toml").contents, + "client-dep", + "a module with no clients should keep no dependencies", ) + + # A pre-1.0 module records its dependencies in the dagger.json it already + # has, and is not given a dagger-module.toml alongside it. + let legacyScoped = ws.withWorkdir(legacyJSONModulePath) + let legacyManifestPath = legacyJSONModulePath + "/dagger.json" + let legacyWithClient = javaSdk.generateScope(legacyScoped, isModule: true, name: "generate-app", clients: [client]) + assertContainsAll(legacyWithClient.file("/" + legacyManifestPath).contents, [ + "\"name\": \"client-dep\"", + "\"source\": \"../../clients/dep\"", + ]) assert( - fromNestedRoots.filter { r => r.rootPath == generateModulePath }.length == 0, - "cwd=lookup/app/nested: generate/app is outside the cone", + contains(legacyWithClient.withWorkdir(".").changes(root).addedPaths, legacyJSONModulePath + "/dagger-module.toml") == false, + "a pre-1.0 module should not be given a dagger-module.toml", ) - assert(fromNestedRoots.length == 1, "cwd=lookup/app/nested: only the enclosing module should be discovered") - assert(fromNested.{{path}}.filter { r => r.path == ".." }.length > 0, "cwd=lookup/app/nested: the enclosing module's path should be cwd-relative (..)") - - # Root cwd: the whole workspace is in scope, so every managed module — and - # only the managed ones — is discovered, whether marked by dagger.json or - # dagger-module.toml. - # Using the fixture root where the test specific workspace is defined - let fromRoot = javaSdk.modules(root.asWorkspace(cwd: fixtureRoot)).{{rootPath}} - assert( - fromRoot.filter { r => r.rootPath == generateModulePath }.length > 0, - "cwd=/: generate/app should be listed", + + let legacyWithoutClient = javaSdk.generateScope(legacyScoped, isModule: true, name: "generate-app", clients: []) + assertNotContains( + legacyWithoutClient.file("/" + legacyManifestPath).contents, + "client-dep", + "a pre-1.0 module with no clients should keep no dependencies", ) - assert(fromRoot.filter { r => r.rootPath == lookupModulePath }.length > 0, "cwd=/: lookup/app should be listed") - assert(fromRoot.filter { r => r.rootPath == depsModulePath }.length > 0, "cwd=/: deps/app should be listed") - assert(fromRoot.filter { r => r.rootPath == skipModulePath }.length > 0, "cwd=/: skip/app should be listed") + + # The payoff: because the client is recorded as a dependency, the module's + # introspection schema carries its types and so do the generated bindings. + let clientApp = javaSdk + .generateScope(ws.withWorkdir(clientAppPath), isModule: true, name: "client-app", clients: [client]) + let bindingsPath = clientAppPath + "/sdk/src/generated/java/io/dagger/client" + let bindings = clientApp.directory("/" + bindingsPath).entries assert( - fromRoot.filter { r => r.rootPath == managedTomlModulePath }.length > 0, - "cwd=/: the dagger-module.toml managed module should be listed", + bindings.filter { entry => entry.contains("ClientDep") }.length > 0, + "no ClientDep binding under " + bindingsPath + ", only: " + bindings.join(", "), ) - assert(fromRoot.length == 5, "cwd=/: exactly the five managed modules should be listed") null } """ - Generation from a subdirectory must resolve managed module root paths from - the workspace root, not relative to the caller's cwd, and must return a - changeset whose paths are relative to that cwd — the form the engine applies - a generator's result in. + A skip marker at or above a module root makes generateScope return an existing + module as it is, and does not hold a module that is being created. """ - generateCwdCheck(ws: Workspace!): Void @check { - # Prepare a valid module in memory and remove the fixture-wide skip marker - # so this check can exercise generation without changing committed fixtures. - let initialized = javaSdk.initModule(ws, name: "generate-app", path: generateModulePath) - let root = testWS(ws) - .directory("/", exclude: [fixtureRoot + "/.dagger-java-sdk-skip-generate"]) - .withDirectory(".", initialized.layer) - - # Module-relative paths of the two artifacts generation stages. - let vendoredClient = "sdk/src/main/java/io/dagger/client/Dagger.java" - let entrypoint = "src/generated/java/io/dagger/gen/entrypoint/Entrypoint.java" - - # From the module's parent the module is one level down, so its changes are - # rooted at "app" — the same cwd-relative path modulesCwdCheck expects. - let fromParent = javaSdk.generateAll( - root.asWorkspace(cwd: fixtureRoot + "/generate"), - ) - assertAdded(fromParent, "app/" + vendoredClient) - assertAdded(fromParent, "app/" + entrypoint) + generateScopeSkipCheck(ws: Workspace!): Void @check { + let root = ws.withWorkdir(".") + + let existing = javaSdk + .generateScope(ws.withWorkdir(skipModulePath), isModule: true, name: "skip-app", clients: []) + .withWorkdir(".") + .changes(root) + assert(existing.isEmpty, "a skip-marked module should not be generated") + + let created = javaSdk + .generateScope(ws.withWorkdir(markedModulePath), isModule: true, name: newModuleName, clients: []) + .withWorkdir(".") + .changes(root) + assertAdded(created, markedModulePath + "/dagger-module.toml") + assertAdded(created, markedModulePath + "/pom.xml") + assertAdded(created, markedModulePath + "/src/generated/java/io/dagger/gen/entrypoint/Entrypoint.java") - # From the module itself the cwd is the module root, so its changes carry no - # prefix at all. - let fromModule = javaSdk.generateAll( - root.asWorkspace(cwd: generateModulePath), - ) - assertAdded(fromModule, vendoredClient) - assertAdded(fromModule, entrypoint) + null + } + """ + The generate skip marker filename should be stable. + """ + skipGenerateFilenameCheck: Void @check { + assert( + javaSdk.skipGenerateFilename == ".dagger-java-sdk-skip-generate", + "skip marker filename changed", + ) null } @@ -277,25 +312,19 @@ type E2e { return type — and, since generation compiles the entrypoint it produces, generating at all proves the entrypoint the annotation processor writes for an Optional return is valid Java. - - Reuses the generate fixture rather than adding a managed module, so the module - inventory the discovery checks assert stays as it is. """ nullableReturnCheck(ws: Workspace!): Void @check { - let modPath = generateModulePath - let initialized = javaSdk.initModule(ws, name: "generate-app", path: modPath) - let root = testWS(ws) - .directory("/", exclude: [fixtureRoot + "/.dagger-java-sdk-skip-generate"]) - .withDirectory(".", initialized.layer) - .withNewFile( - modPath + "/src/main/java/io/dagger/modules/generateapp/GenerateApp.java", - nullableReturnSource, - ) - - let changes = javaSdk.generateAll(root.asWorkspace(cwd: modPath)) - let entrypoint = changes - .layer - .file("src/generated/java/io/dagger/gen/entrypoint/Entrypoint.java") + # Same scope and module name as generateScopeInitCheck, so both checks share + # the vendored SDK build instead of each paying for one of their own. + let scoped = ws.withWorkdir(newModulePath) + let scaffolded = javaSdk.generateScope(scoped, isModule: true, name: newModuleName, clients: []) + let withSource = scaffolded.withNewFile( + "/" + newModulePath + "/src/main/java/io/dagger/modules/newapp/NewApp.java", + nullableReturnSource, + ) + let entrypoint = javaSdk + .generateScope(withSource, isModule: true, name: newModuleName, clients: []) + .file("/" + newModulePath + "/src/generated/java/io/dagger/gen/entrypoint/Entrypoint.java") .contents assertContains( @@ -314,7 +343,7 @@ type E2e { """A module whose only function returns a nullable object.""" let nullableReturnSource: String! { - "package io.dagger.modules.generateapp;\n" + "package io.dagger.modules.newapp;\n" + "\n" + "import static io.dagger.client.Dagger.dag;\n" + "\n" @@ -324,7 +353,7 @@ type E2e { + "import java.util.Optional;\n" + "\n" + "@Object\n" - + "public class GenerateApp {\n" + + "public class NewApp {\n" + " @Function\n" + " public Optional maybeDirectory(boolean found) {\n" + " if (!found) {\n" diff --git a/dagger.toml b/dagger.toml index a3b65d4..ed15b5a 100644 --- a/dagger.toml +++ b/dagger.toml @@ -1,5 +1,15 @@ +# Dagger workspace configuration +# Install modules with: dagger module install +# Example: +# dagger module install github.com/dagger/dagger/modules/wolfi + [modules.e2e] source = ".dagger/modules/e2e" +# Every check below calls the java-sdk module, which needs an engine with +# dagger/dagger#13992. On the released engine each of them fails with +# `"moduleManifest" not found`, so they run in the engine-e2e playground +# instead (engine-e-2-e:sdk-contract-check). +check.skip = ["*"] [modules.packager] source = ".dagger/modules/packager" @@ -7,17 +17,36 @@ source = ".dagger/modules/packager" [modules.dagger-dang-sdk] source = "github.com/dagger/dang-sdk" -[modules.dagger-dang-sdk.as-sdk] -name = "dang" - [modules.templates] source = ".dagger/modules/templates" -[[modules.dagger-dang-sdk.as-sdk.modules]] -path = ".dagger/modules/templates" +[modules.java-sdk] +source = "." +check.skip = ["*"] + +[sdks.java] +module = "java-sdk" + +# Inert for `dagger generate`: the released engine parses no [sdks.*] table at +# all, and on an engine with #13992 every scope below sits under the fixture-wide +# .dagger-java-sdk-skip-generate marker and already has a config, so generateScope +# reconciles the manifest and returns without generating. +[sdks.java.scopes.".dagger/modules/e2e/fixtures/generate/app"] +is-module = true +name = "generate-app" + +[sdks.java.scopes.".dagger/modules/e2e/fixtures/lookup/app"] +is-module = true +name = "lookup-app" + +[sdks.java.scopes.".dagger/modules/e2e/fixtures/deps/app"] +is-module = true +name = "deps-app" -[[modules.dagger-dang-sdk.as-sdk.modules]] -path = "." +[sdks.java.scopes.".dagger/modules/e2e/fixtures/skip/app"] +is-module = true +name = "skip-app" -[modules.sdk-sdk] -source = "github.com/dagger/sdk-sdk" +[sdks.java.scopes.".dagger/modules/e2e/fixtures/managed-toml/app"] +is-module = true +name = "managed-toml-app" diff --git a/main.dang b/main.dang index 061cc6a..d554c79 100644 --- a/main.dang +++ b/main.dang @@ -1,23 +1,23 @@ """ Manage Dagger modules that use the Java SDK (new self-contained code organisation). -Modules created by `init` use the self-contained layout: the Java SDK is vendored -as source and code is generated into the module (see `generate`), and the module -runtime is this repository's build/package-only runtime +Modules created by this SDK use the self-contained layout: the Java SDK is +vendored as source and code is generated into the module, and the module runtime +is this repository's build/package-only runtime (`github.com/dagger/java-sdk/runtime`). Because the generated files are committed, the runtime skips codegen at module load and just builds and packages the committed sources. """ type JavaSdk { """ - Runtime source written into the dagger-module.toml of new Java modules. + Marker filename that skips generate when found at or above a Java SDK module root. """ - targetRuntime: String! { "github.com/dagger/java-sdk/runtime" } + skipGenerateFilename: String! = ".dagger-java-sdk-skip-generate" """ - Marker filename that skips generate when found at or above a Java SDK module root. + Starter under templates/