bundle: convert the configuration to typed structs once after loading

.agents/rules/auto-generated-files.md

@@ -15,6 +15,9 @@ globs:
   - "internal/mocks/**/*.go"
   - "bundle/direct/dresources/configs/*.generated.yml"
   - "bundle/config/validate/*_walk.go"
+  - "bundle/config/walk/*_walk.go"
+  - "bundle/config/mutator/resourcemutator/defaults_walk.go"
+  - "bundle/config/resources_iterate.go"
   - "bundle/schema/jsonschema.json"
   - "python/databricks/bundles/version.py"
   - "python/databricks/bundles/*/__init__.py"
@@ -34,6 +37,9 @@ paths:
   - "internal/mocks/**/*.go"
   - "bundle/direct/dresources/configs/*.generated.yml"
   - "bundle/config/validate/*_walk.go"
+  - "bundle/config/walk/*_walk.go"
+  - "bundle/config/mutator/resourcemutator/defaults_walk.go"
+  - "bundle/config/resources_iterate.go"
   - "bundle/schema/jsonschema.json"
   - "python/databricks/bundles/version.py"
   - "python/databricks/bundles/*/__init__.py"

.agents/rules/style-guide-go.md

@@ -113,7 +113,7 @@ return fieldPaths
 
 ### Encoding
 
-**RULE: When mutating an API response by round-tripping JSON, decode with `json.Decoder` + `UseNumber()`.** The naive `json.Marshal` → `map[string]any` → `Marshal` path corrupts any int64 larger than 2^53 (it degrades to a float64 mantissa — e.g. a real `spark_context_id`) and alphabetizes object keys. `libs/dyn/jsonloader` preserves key order but also lacks `UseNumber`, so it shares the int64 hazard.
+**RULE: When mutating an API response by round-tripping JSON, decode with `json.Decoder` + `UseNumber()`.** The naive `json.Marshal` → `map[string]any` → `Marshal` path corrupts any int64 larger than 2^53 (it degrades to a float64 mantissa — e.g. a real `spark_context_id`) and alphabetizes object keys.
 
 **RULE: Be careful with `encoding/csv` `Writer.UseCRLF = true`.** It rewrites both record terminators AND embedded newlines inside quoted fields to `\r\n`, so tests for quoted multiline fields must expect `\r\n`, not just the line endings between rows.
 
@@ -191,7 +191,7 @@ func NewService(deps ServiceDeps) *Service { ... }
 
 ### Configuration patterns
 
-- Bundle config uses `dyn.Value` for dynamic typing
+- Bundle config is loaded, included and target-merged as a `dyn.Value`, then converted once to typed structs (`bundle/config.Root`) at the end of the load phase; after that, mutators use the structs, and `libs/structs/structvar` carries `${...}` references and source locations
 - Config loading supports includes, variable interpolation, and target overrides
 - Schema generation is automated from Go struct tags
 

.agents/skills/bump-sdk/SKILL.md

@@ -9,7 +9,7 @@ allowed-tools: Read, Edit, Write, Bash, Glob, Grep, WebFetch, AskUserQuestion
 
 The SDK version lives in `go.mod` (`github.com/databricks/databricks-sdk-go`) and the pinned spec SHA lives in `.codegen/_openapi_sha`.
 These two move as a pair; everything else in this skill is regenerated from them or is fallout you fix by hand.
-Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/config/validate/*_walk.go`, `bundle/direct/dresources/configs/*.generated.yml`, `python/databricks/bundles/**`); regenerate them.
+Do not hand-edit generated files (`.codegen/cli.json`, `cmd/workspace/*`, `cmd/account/*`, `bundle/schema/jsonschema.json`, `bundle/config/validate/*_walk.go`, `bundle/config/walk/*_walk.go`, `bundle/direct/dresources/configs/*.generated.yml`, `python/databricks/bundles/**`); regenerate them.
 
 The Python tasks (`pydabs-*`, and the `pydabs-codegen` step inside `generate-check`) all run through `uv`. If one fails because `uv` is missing or because the host's `python3` is too old (e.g. 3.9), install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`) rather than touching the system Python: `uv run` provisions the interpreter each package pins (`>=3.10`, and `==3.13.*` under `python/codegen/`) and downloads it if needed. Do not chase the system Python version.
 The acceptance goldens are the exception: they need CI's `uv` index, see step 4.

.github/workflows/python_push.yml

@@ -39,7 +39,7 @@ jobs:
             acceptance/bundle/python/ \
             go.mod \
             go.sum \
-            libs/dyn/dynvar/testdata/reference_vectors.json \
+            libs/structs/structvar/testdata/reference_vectors.json \
             python/ \
             tools/task/ \
           )

.nextchanges/bundles/7098-presence.md

@@ -0,0 +1 @@
+* `bundle validate` no longer treats a written zero value in a field that cannot record it (no `ForceSendFields`, e.g. `job_id: 0` in `run_job_task`) or an empty mapping (e.g. `presets: {}`) as set, and does not show the pipeline `development` deprecation warning for `development: true` under `mode: development`. ([#7098](https://github.com/databricks/cli/pull/7098))

.nextchanges/bundles/7098.md

@@ -0,0 +1 @@
+* Speed up bundle commands on large bundles: configuration is kept as typed structs instead of being converted to and from a dynamic tree (`bundle validate` and `bundle plan` with 1000 jobs are about 10x faster and use about 3.5x less memory). ([#7098](https://github.com/databricks/cli/pull/7098))

AGENTS.md

@@ -14,7 +14,7 @@ This is the Databricks CLI, a command-line interface for interacting with Databr
 
 **RULE: Keep each PR focused on one change.** If you notice an unrelated cleanup, bug fix, or refactor while making your primary change, leave it alone or put it in a separate PR. Reviewers consistently ask to split mixed PRs, especially when a dependency bump or schema diff rides along with a feature change.
 
-**RULE: Before adding a new helper, search the codebase for an existing one.** Common homes: `libs/` (shared utilities), `libs/databrickscfg/` (config), `libs/git/`, `libs/filer/`, `libs/cmdio/` (CLI I/O, spinners, prompts), `libs/env/` (env vars), `libs/testserver/`, `libs/structpath/` and `libs/dyn/` (path / dynamic values), `acceptance/bin/` (acceptance test helpers), `internal/mocks/` (generated mocks). A function that duplicates an existing name and signature in the same package is a compile error waiting to happen; grep before you name.
+**RULE: Before adding a new helper, search the codebase for an existing one.** Common homes: `libs/` (shared utilities), `libs/databrickscfg/` (config), `libs/git/`, `libs/filer/`, `libs/cmdio/` (CLI I/O, spinners, prompts), `libs/env/` (env vars), `libs/testserver/`, `libs/structs/structpath/` and `libs/structs/structvar/` (paths / typed config values), `acceptance/bin/` (acceptance test helpers), `internal/mocks/` (generated mocks). A function that duplicates an existing name and signature in the same package is a compile error waiting to happen; grep before you name.
 
 # Development Commands
 
@@ -73,7 +73,8 @@ GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true VISUAL=true GIT_PAGER=cat git rebase or
 - `bundle/phases/` - High-level deployment phases
 
 **libs/** - Shared libraries and utilities
-- `libs/dyn/` - Dynamic configuration value manipulation
+- `libs/dyn/` - Dynamic configuration value tree: loads `databricks.yml` and its includes and merges targets, before the configuration is converted to typed structs
+- `libs/structs/structvar/` - Typed configuration values with references and locations
 - `libs/filer/` - File system abstraction (local, DBFS, workspace)
 - `libs/auth/` - Databricks authentication handling
 - `libs/sync/` - File synchronization between local and remote

Taskfile.yml

@@ -1021,8 +1021,11 @@ tasks:
       - "{{.EMBED_SOURCES}}"
     generates:
       - bundle/config/validate/*_walk.go
+      - bundle/config/walk/*_walk.go
+      - bundle/config/mutator/resourcemutator/defaults_walk.go
+      - bundle/config/resources_iterate.go
     cmds:
-      - "sh -c 'go run ./bundle/internal/validation/. && gofmt -w -s ./bundle/config/validate/required_walk.go ./bundle/config/validate/enum_walk.go'"
+      - "go run ./bundle/internal/validation/."
 
   generate-direct:
     desc: Generate direct engine config (apitypes + resources)

acceptance/bundle/presets/trigger_pause_status_whole_schedule_reference/output.txt

@@ -1,11 +1,17 @@
 
->>> musterr [CLI] bundle validate
-Error: expected a map to index "pause_status", found string
+>>> [CLI] bundle validate
+Warning: required field "quartz_cron_expression" is not set
+  at resources.jobs.viaref.schedule
+  in databricks.yml:16:17
+
+Warning: required field "timezone_id" is not set
+  at resources.jobs.viaref.schedule
+  in databricks.yml:16:17
 
 Name: test-bundle
 Target: default
 Workspace:
   User: [USERNAME]
   Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
 
-Found 1 error
+Found 2 warnings

acceptance/bundle/presets/trigger_pause_status_whole_schedule_reference/script

@@ -1,3 +1,3 @@
 # trigger_pause_status needs to set pause_status inside the schedule, but the schedule of viaref is a reference to the whole
 # schedule of another job and is still unresolved at that point.
-trace musterr $CLI bundle validate
+trace $CLI bundle validate

acceptance/bundle/presets/trigger_pause_status_whole_schedule_reference/test.toml

@@ -1 +1 @@
-Badness = "trigger_pause_status fails with a confusing error when the schedule is a reference to the whole schedule of another job."
+Badness = "Required-field warnings are reported for a schedule that is a reference to a complete schedule."

acceptance/bundle/presets/trigger_pause_status_whole_trigger_reference/output.txt

@@ -1,11 +1,9 @@
 
->>> musterr [CLI] bundle validate
-Error: expected a map to index "pause_status", found string
-
+>>> [CLI] bundle validate
 Name: test-bundle
 Target: default
 Workspace:
   User: [USERNAME]
   Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
 
-Found 1 error
+Validation OK!

acceptance/bundle/presets/trigger_pause_status_whole_trigger_reference/script

@@ -1,2 +1,2 @@
 # Same as with a schedule: pause_status cannot be set inside a trigger that is a reference to the whole trigger of another job.
-trace musterr $CLI bundle validate
+trace $CLI bundle validate

acceptance/bundle/python/pipeline-development-deprecated/output.txt

@@ -1,22 +1,14 @@
 
 >>> uv run [UV_ARGS] -q [CLI] bundle validate
-Warning: The "development" property of pipelines is deprecated. Use "mode: development" in your databricks.yml instead. See https://docs.databricks.com/dev-tools/bundles/deployment-modes.
-  at resources.pipelines.with_development.development
-  in resources.py:7:1
-
 Name: my_project
 Target: dev
 Workspace:
   User: [USERNAME]
   Path: /Workspace/Users/[USERNAME]/.bundle/my_project/dev
 
-Found 1 warning
+Validation OK!
 
 >>> uv run [UV_ARGS] -q [CLI] bundle validate --output json
-Warning: The "development" property of pipelines is deprecated. Use "mode: development" in your databricks.yml instead. See https://docs.databricks.com/dev-tools/bundles/deployment-modes.
-  at resources.pipelines.with_development.development
-  in resources.py:7:1
-
 {
   "with_development": true,
   "without_development": true

acceptance/bundle/refschema/output/root.txt

@@ -1,4 +1,4 @@
-__locations	*dynloc.Locations	INPUT
+__locations	*loctable.Locations	INPUT
 __locations.files	[]string	INPUT
 __locations.files[*]	string	INPUT
 __locations.locations	map[string][][]int	INPUT

acceptance/bundle/strict_profile_match/default_profile/host/output.txt

@@ -18,7 +18,7 @@ Error: cannot resolve bundle auth configuration: resolve: strict profile matchin
 Name: strict-profile-match
 Target: classic_other_workspace
 Workspace:
-  Host: [DATABRICKS_URL]
+  Host: [DATABRICKS_URL]/?o=2222222222
 
 Found 1 error
 

acceptance/bundle/strict_profile_match/host/output.txt

@@ -22,7 +22,7 @@ Error: cannot resolve bundle auth configuration: resolve: strict profile matchin
 Name: strict-profile-match
 Target: classic_other_workspace
 Workspace:
-  Host: [DATABRICKS_URL]
+  Host: [DATABRICKS_URL]/?o=2222222222
 
 Found 1 error
 
@@ -48,7 +48,7 @@ Error: cannot resolve bundle auth configuration: resolve: strict profile matchin
 Name: strict-profile-match
 Target: custom_host_other_workspace
 Workspace:
-  Host: [CUSTOM_HOST]
+  Host: [CUSTOM_HOST]/?o=2222222222
 
 Found 1 error
 

acceptance/bundle/summary/modified_status/output.txt

@@ -200,4 +200,4 @@ All files and directories at the following location will be deleted: /Workspace/
 Destroy: 4 deleted
 
 >>> [CLI] bundle summary -o json
-{}
+null

acceptance/bundle/validate/app_resource_duplicate_name_location/output.txt

@@ -2,7 +2,7 @@
 >>> [CLI] bundle validate
 Warning: invalid value "INVALID" for enum field. Valid values are [CAN_MANAGE CAN_MANAGE_RUN CAN_VIEW IS_OWNER]
   at resources.apps.app.resources[0].job.permission
-  in databricks.yml:13:25
+  in databricks.yml:17:25
 
 Name: test-bundle
 Target: default
@@ -15,7 +15,7 @@ Found 1 warning
 >>> [CLI] bundle validate -o json
 Warning: invalid value "INVALID" for enum field. Valid values are [CAN_MANAGE CAN_MANAGE_RUN CAN_VIEW IS_OWNER]
   at resources.apps.app.resources[0].job.permission
-  in databricks.yml:13:25
+  in databricks.yml:17:25
 
 [
   {

acceptance/bundle/validate/app_resource_duplicate_name_location/script

@@ -1,4 +1,4 @@
-# A diagnostic on a field of a duplicated app resource entry points at the first entry when the duplicate's value is the same.
+# A diagnostic on a field of a duplicated app resource entry points at the entry that is kept (the later one).
 errcode trace $CLI bundle validate
 
 # The entries are merged by name, the later one wins.

acceptance/bundle/validate/includes/merge_failure_short_circuits_diagnostics/script

@@ -1,2 +1,2 @@
-# Includes after one that fails to merge do not report their diagnostics.
+# Includes after one that fails to merge still report their diagnostics.
 errcode trace $CLI bundle validate

acceptance/bundle/validate/nulls/resources_container/script

@@ -1,2 +1,2 @@
-# resources: null is rejected.
+# resources: null is accepted.
 errcode trace $CLI bundle validate

acceptance/bundle/validate/references/resolved_invalid_map_value/script

@@ -1,2 +1,2 @@
-# A map value whose reference resolves to the wrong type is dropped, with a warning at its path.
+# A map value whose reference resolves to the wrong type becomes an empty string, with a warning at its path.
 trace $CLI bundle validate -o json | jq .resources.jobs.j.tags

acceptance/bundle/validate/references/resolved_invalid_sequence_item/script

@@ -1,2 +1,2 @@
-# A list element whose reference resolves to the wrong type is dropped, with a warning at its path.
+# A list element whose reference resolves to the wrong type becomes an empty string, with a warning at its path.
 trace $CLI bundle validate -o json | jq .resources.jobs.j.email_notifications

acceptance/bundle/validate/references/resolved_invalid_sequence_items/databricks.yml

@@ -0,0 +1,15 @@
+bundle:
+  name: test-bundle
+
+variables:
+  bad:
+    type: complex
+    default:
+      x: 1
+
+resources:
+  jobs:
+    j:
+      name: test
+      email_notifications:
+        on_failure: ["a@example.test", "${var.bad}", "b@example.test", "${var.bad}", "${var.bad}", "c@example.test"]

acceptance/bundle/validate/references/resolved_invalid_sequence_items/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = false
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/validate/references/resolved_invalid_sequence_items/output.txt

@@ -0,0 +1,21 @@
+
+>>> [CLI] bundle validate -o json
+Warning: expected string, found map
+  at resources.jobs.j.email_notifications.on_failure[1]
+  in databricks.yml:15:40
+
+Warning: expected string, found map
+  at resources.jobs.j.email_notifications.on_failure[3]
+  in databricks.yml:15:72
+
+Warning: expected string, found map
+  at resources.jobs.j.email_notifications.on_failure[4]
+  in databricks.yml:15:86
+
+{
+  "on_failure": [
+    "a@example.test",
+    "b@example.test",
+    "c@example.test"
+  ]
+}

acceptance/bundle/validate/references/resolved_invalid_sequence_items/script

@@ -0,0 +1,2 @@
+# Several list elements whose reference resolves to the wrong type are dropped, the others keep their order.
+trace $CLI bundle validate -o json | jq .resources.jobs.j.email_notifications

acceptance/bundle/validate/references/to_omitted_struct_defaults/output.txt

@@ -2,18 +2,10 @@
 >>> [CLI] bundle validate -o json
 {
   "default": {
-    "actual_branch": "",
-    "branch": "",
-    "bundle_root_path": ".",
-    "commit": "",
-    "origin_url": ""
+    "bundle_root_path": "."
   },
   "type": "complex",
   "value": {
-    "actual_branch": "",
-    "branch": "",
-    "bundle_root_path": ".",
-    "commit": "",
-    "origin_url": ""
+    "bundle_root_path": "."
   }
 }

acceptance/bundle/validate/references/to_omitted_struct_defaults/script

@@ -1,3 +1,3 @@
 # A reference to a struct that is not set in the configuration (here outside a git repository)
-# resolves to the struct with its fields filled in, as empty values.
+# resolves to the fields that are set; unset fields are omitted.
 trace $CLI bundle validate -o json | jq .variables.git_info

acceptance/bundle/validate/required/databricks.yml

@@ -19,7 +19,7 @@ resources:
   jobs:
     my_job_1:
       tasks:
-        # job_id being 0 should not trigger a warning.
+        # job_id 0 is not set either: RunJobTask has no ForceSendFields to record it.
         - task_key: "task_key1"
           run_job_task:
             job_id: 0

acceptance/bundle/validate/required/output.txt

@@ -4,6 +4,10 @@ Warning: required field "catalog_name" is not set
   at resources.volumes.my_volume
   in databricks.yml:35:7
 
+Warning: required field "job_id" is not set
+  at resources.jobs.my_job_1.tasks[0].run_job_task
+  in databricks.yml:25:13
+
 Warning: required field "job_id" is not set
   at resources.jobs.my_job_1.tasks[1].run_job_task
   in databricks.yml:29:24
@@ -26,4 +30,4 @@ Workspace:
   User: [USERNAME]
   Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
 
-Found 5 warnings
+Found 6 warnings

acceptance/bundle/validate/targets/scalar_rewrite_diagnostic/script

@@ -1,2 +1,2 @@
-# A scalar targets section is an error.
+# A scalar targets section is reported with a warning at its location.
 errcode trace $CLI bundle validate

acceptance/bundle/validate/variables/complex_default_nested_timestamp/output.txt

@@ -1,10 +1,9 @@
 
 >>> [CLI] bundle validate
-Warn: unable to convert typed configuration to dynamic configuration: cannot convert struct field to dynamic type "time": src=time.Date(2020, time.January, 2, 0, 0, 0, 0, time.UTC) ref=time.Date(2020, time.January, 2, 0, 0, 0, 0, time.UTC)
-Error: exit error: cannot convert struct field to dynamic type "time": src=time.Date(2020, time.January, 2, 0, 0, 0, 0, time.UTC) ref=time.Date(2020, time.January, 2, 0, 0, 0, 0, time.UTC)
-
 Name: test-bundle
+Target: default
+Workspace:
+  User: [USERNAME]
+  Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
 
-Found 1 error
-
-Exit code: 1
+Validation OK!

acceptance/bundle/validate/variables/complex_unknown_field_location/output.txt

@@ -2,7 +2,6 @@
 >>> [CLI] bundle validate
 Warning: unknown field: misspelled_field
   at resources.jobs.j
-  in databricks.yml:9:7
 
 Name: test-bundle
 Target: default

acceptance/bundle/validate/variables/complex_unknown_field_location/test.toml

@@ -0,0 +1 @@
+Badness = "The unknown field warning has no location; it should point at the key in the variable default (databricks.yml:9:7)."

acceptance/bundle/validate/yaml_anchors/duplicate_merge_keys/output.txt

@@ -7,3 +7,11 @@ Error: duplicate YAML merge key ('<<') is not allowed; to merge multiple maps, u
 Found 1 error
 
 Exit code: 1
+
+>>> [CLI] bundle validate -t dev -o json
+Error: duplicate YAML merge key ('<<') is not allowed; to merge multiple maps, use a sequence: '<<: [*anchor1, *anchor2]'
+  in databricks.yml:18:7
+
+
+Exit code: 1
+null

acceptance/bundle/validate/yaml_anchors/duplicate_merge_keys/script

@@ -1,2 +1,4 @@
 # Two '<<' keys in target variables are an error.
 errcode trace $CLI bundle validate -t dev
+# What was merged into the variable:
+errcode trace $CLI bundle validate -t dev -o json | jq .variables.x