Fix terraform migration failing on refs to undeployed resources by denik · Pull Request #7157 · databricks/cli

denik · 2026-10-22T06:18:27Z

Changes

When migrating terraform state, a job in the state that references a resource not yet deployed (e.g. a new alert used by a new alert_task) failed with cannot resolve field ... not found in TF state. Such fields are now recorded as unset, so the first direct plan applies them.

This pull request and its description were written by Isaac.

A resource that is in the terraform state but references a resource that was never deployed (new in config) made the migration abort. Record such fields as unset so the first direct plan applies them. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

github-actions · 2026-10-22T06:31:28Z

Commit: 394ed4cd9 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0

test	branch	main (`e004bdf`)	latest (v1.21.0)
TestAccept/bundle/migrate/auto/new-resource-ref/DMS=	✅	❌	❌

TestAccept/bundle/migrate/auto/new-resource-ref/DMS=: branch ✅ | main (e004bdf) ❌ | latest (v1.21.0) ❌

main (e004bdf):

acceptance_test.go:1307: Diff:
        --- bundle/migrate/auto/new-resource-ref/output.txt
        +++ /tmp/TestAcceptbundlemigrateautonew-resource-refDMS=1289812391/002/output.txt
        @@ -5,171 +5,9 @@
         === Add a new alert and a job task that references it
         === Plan shows the new alert and the job update (the terraform state is migrated in memory)
         >>> [CLI] bundle plan -o json
        -Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
        -{
        -  "plan_version": [PLAN_VERSION],
        -  "cli_version": "[CLI_VERSION]",
        -  "lineage": "[UUID]",
        -  "serial": 3,
        -  "plan": {
        -    "resources.alerts.new_alert": {
        -      "action": "create",
        -      "new_state": {
        -        "value": {
        -          "display_name": "New alert",
        -          "evaluation": {
        -            "comparison_operator": "EQUAL",
        -            "notification": {
        -              "retrigger_seconds": 300
        -            },
        -            "source": {
        -              "name": "1"
        -            },
        -            "threshold": {
        -              "value": {
        -                "double_value": 1.3
        -              }
        -            }
        -          },
        -          "parent_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/resources",
        -          "query_text": "select 2",
        -          "schedule": {
        -            "pause_status": "UNPAUSED",
        -            "quartz_cron_schedule": "44 19 */1 * * ?",
        -            "timezone_id": "Europe/Amsterdam"
        -          },
        -          "warehouse_id": "[TEST_DEFAULT_WAREHOUSE_ID]"
        -        }
        -      }
        -    },
        -    "resources.jobs.test_job": {
        -      "depends_on": [
        -        {
        -          "node": "resources.alerts.new_alert",
        -          "label": "${resources.alerts.new_alert.evaluation.notification.retrigger_seconds}"
        -        },
        -        {
        -          "node": "resources.alerts.new_alert",
        -          "label": "${resources.alerts.new_alert.id}"
        -        }
        -      ],
        -      "action": "update",
        -      "new_state": {
        -        "value": {
        -          "deployment": {
        -            "kind": "BUNDLE",
        -            "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
        -          },
        -          "edit_mode": "UI_LOCKED",
        -          "format": "MULTI_TASK",
        -          "max_concurrent_runs": 1,
        -          "name": "Test Auto-Migrate Job",
        -          "queue": {
        -            "enabled": true
        -          },
        -          "tasks": [
        -            {
        -              "alert_task": {
        -                "alert_id": "${resources.alerts.new_alert.id}"
        -              },
        -              "task_key": "check"
        -            }
        -          ],
        -          "timeout_seconds": 300
        -        },
        -        "vars": {
        -          "tasks[0].alert_task.alert_id": "${resources.alerts.new_alert.id}"
        -        }
        -      },
        -      "remote_state": {
        -        "created_time": [UNIX_TIME_MILLIS],
        -        "creator_user_name": "[USERNAME]",
        -        "deployment": {
        -          "kind": "BUNDLE",
        -          "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/metadata.json"
        -        },
        -        "edit_mode": "UI_LOCKED",
        -        "email_notifications": {},
        -        "format": "MULTI_TASK",
        -        "job_id": [TEST_JOB_ID],
        -        "max_concurrent_runs": 1,
        -        "name": "Test Auto-Migrate Job",
        -        "queue": {
        -          "enabled": true
        -        },
        -        "run_as_user_name": "[USERNAME]",
        -        "timeout_seconds": 0,
        -        "webhook_notifications": {}
        -      },
        -      "changes": {
        -        "email_notifications": {
        -          "action": "skip",
        -          "reason": "empty",
        -          "remote": {}
        -        },
        -        "tasks": {
        -          "action": "update",
        -          "old": [
        -            {
        -              "alert_task": {
        -                "alert_id": ""
        -              },
        -              "task_key": "check"
        -            }
        -          ],
        -          "new": [
        -            {
        -              "alert_task": {
        -                "alert_id": "${resources.alerts.new_alert.id}"
        -              },
        -              "task_key": "check"
        -            }
        -          ]
        -        },
        -        "tasks[task_key='check'].alert_task.alert_id": {
        -          "action": "update",
        -          "old": "",
        -          "new": "${resources.alerts.new_alert.id}"
        -        },
        -        "timeout_seconds": {
        -          "action": "update",
        -          "new": 300,
        -          "remote": 0
        -        },
        -        "webhook_notifications": {
        -          "action": "skip",
        -          "reason": "empty",
        -          "remote": {}
        -        }
        -      }
        -    }
        -  }
        -}
        +Error: migrating Terraform state to the direct engine: converting terraform state to the direct engine: resources.jobs.test_job: cannot resolve field "tasks[0].alert_task.alert_id" (template "${resources.alerts.new_alert.id}"): jobs.test_job field tasks[0].alert_task.alert_id: method A: index 0 out of range (len 0); method B: cannot look up "resources.alerts.new_alert.id": databricks_alert_v2.new_alert not found in TF state
         
        -=== Deploy migrates to direct, creates the alert and updates the job
        ->>> [CLI] bundle deploy
        -Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
        -Migrated 1 resource to direct deployment engine.
        -Created alerts.new_alert
        -Updated jobs.test_job
        -Files: 1 uploaded, 0 deleted
        -Resources: 1 created, 1 changed, 0 deleted, 0 unchanged
        +The Terraform deployment engine was removed in Databricks CLI v1.20.0. Auto-migration of your bundle failed. Install Databricks CLI v1.19.x to deploy on Terraform, and please report this to dabs-feedback@databricks.com
         
        -=== The job update carries the new alert id and timeout
        ->>> print_requests.py //api/2.2/jobs/reset
        -json.body.new_settings.tasks[0].alert_task.alert_id = "[NEW_ALERT_ID]";
        -json.body.new_settings.timeout_seconds = 300;
         
        -=== Redeploy is a no-op
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
        -
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.alerts.new_alert
        -  delete resources.jobs.test_job
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
        -
        -Destroy: 2 deleted
        +Exit code: 1
        
FAIL

latest (v1.21.0): same output as main (e004bdf)

Co-authored-by: Isaac <no-reply@databricks.com>

eng-dev-ecosystem-bot · 2026-10-22T07:03:00Z

Integration test report

Commit: 394ed4c

Run: 39745541562

	Env	✅pass	🙈skip	Time
✅	aws linux-2core-8gb	281	16	4:18
✅	aws-windows-latest-4core-16gb	283	14	3:59
✅	azure linux-2core-8gb	280	16	4:08
✅	azure-windows-latest-4core-16gb	282	14	4:31
✅	gcp linux-2core-8gb	281	16	4:21
✅	gcp-windows-latest-4core-16gb	283	14	3:34

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
4:25	azure-windows-latest-4core-16gb	TestAccept
3:55	aws-windows-latest-4core-16gb	TestAccept
3:28	gcp-windows-latest-4core-16gb	TestAccept

A ref in a number field is already unset after conversion, so there is nothing to zero. Also skip refs that do not target a resource (for example variables) when deciding whether a target is undeployed. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

## Release v1.22.0 ### CLI * Fix `databricks aitools install` overwriting newer raw skills with an older pinned release. CLI 1.22.0 and newer now install the latest published skills and prevent automatic downgrades. ([#7069](#7069)) * Fix `auth login --timeout` not applying to the browser login, which waited for the OAuth redirect indefinitely. ([#7103](#7103)) * Remove the 50-workspace limit in `auth login` and support searching workspaces by name or ID. ([#7071](#7071)) * `databricks ssh connect` now fails right away with the workspace's reason when port forwarding to serverless compute is disabled, instead of waiting out the server start timeout. ([#7057](#7057)) ### AI Runtime * `air run` now spends less time before submitting: it fetches only the user name from SCIM. ([#7153](#7153)) * Align multi-image container name validation with backend services: allow hyphens and up to 32 characters, and reject uppercase letters and underscores. ([#7154](#7154)) ### Bundles * The `compute_id` deprecation warning for a target now shows the full path (`targets.<name>.compute_id`). ([#7048](#7048)) * Add `DATABRICKS_BUNDLE_STRICT_PROFILE_MATCH`, which requires a target's host and workspace ID to match exactly one profile in the config file before authenticating. Targets without a host or workspace ID are not affected. ([#7109](#7109)) * Zero-value overrides of `presets`, `lifecycle.prevent_destroy` and `deployment` settings in includes now replace non-zero base values, and `bundle validate -o json` shows explicit zero presets. ([#7117](#7117)) * `bundle deployment bind` and `unbind` now accept a resource as `type.key`, e.g. `jobs.foo`. ([#7128](#7128)) * Retry more transient server errors during `bundle deploy` and `bundle destroy`, including when releasing the deploy lock and reading state. ([#7129](#7129)) * Fix migration from Terraform failing when a deployed resource references a resource that was never deployed. ([#7157](#7157)) * Add `app-appkit` to bundle templates to simplify starting a Databricks Apps project. ([#7004](#7004), [#7007](#7007)) * Add support for the `features` resource type. ([#6952](#6952)) * Fix `bundle validate` warning that `sync.include` and `sync.exclude` patterns do not match any files when `sync.paths` includes a directory outside the bundle root. ([#7122](#7122)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#7083](#7083)) * Bump Go to 1.27.2. Release binaries now require macOS 13 Ventura or later. ([#7052](#7052)) * Update the Go SDK to [v0.189.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.189.0). ([#7138](#7138)) * Bump Go toolchain to 1.27.3. ([#7116](#7116))

denik and others added 2 commits October 22, 2026 08:18

Add changelog fragment

319b494

Co-authored-by: Isaac <no-reply@databricks.com>

github-actions Bot added the DABs DABs related issues label Oct 22, 2026

denik and others added 2 commits October 22, 2026 08:23

Add acceptance test for migration with a ref to a new resource

cec7e58

Co-authored-by: Isaac <no-reply@databricks.com>

Fix changelog fragment format

6de85e4

Co-authored-by: Isaac <no-reply@databricks.com>

acc: make new-resource-ref fixture readable with commented-out config

c42fff8

Co-authored-by: Isaac <no-reply@databricks.com>

denik and others added 3 commits October 22, 2026 09:57

acc: cover a number field reference in new-resource-ref

cbf7be8

Co-authored-by: Isaac <no-reply@databricks.com>

acc: record the JSON plan in the new resource ref test

394ed4c

Co-authored-by: Isaac <no-reply@databricks.com>

denik marked this pull request as ready for review October 22, 2026 08:09

denik requested review from a team as code owners October 22, 2026 08:09

denik requested review from janniklasrose and rugpanov October 22, 2026 08:09

janniklasrose approved these changes Oct 22, 2026

View reviewed changes

denik added this pull request to the merge queue Oct 22, 2026

Merged via the queue into main with commit be461d4 Oct 22, 2026
28 checks passed

denik deleted the denik/terraform-migration-error branch October 22, 2026 08:58

This was referenced Oct 22, 2026

Update Databricks CLI to v1.22.0 databricks/databricks-vscode#2269

Closed

Update Databricks CLI to v1.22.1 databricks/databricks-vscode#2270

Merged

Navigation Menu

Repository navigation

Fix terraform migration failing on refs to undeployed resources - #7157

Fix terraform migration failing on refs to undeployed resources#7157
denik merged 8 commits into
mainfrom
denik/terraform-migration-error

denik commented Oct 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

denik commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

github-actions Bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression test report

Uh oh!

eng-dev-ecosystem-bot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

denik commented Oct 22, 2026 •

edited

Loading

github-actions Bot commented Oct 22, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Oct 22, 2026 •

edited

Loading