Genie Space bundle resource requires its source table to exist at deploy time, blocking first-time bundle deploys #6750

Describe the issue

Deploying a Databricks Asset Bundle that declares a genie_spaces resource in YAML requires the underlying data source (schema/table) referenced in data_sources.tables to already exist in the target workspace at deploy time. This makes it impossible to deploy a bundle end-to-end on a fresh environment (e.g. a new feature-branch target) where the schema and tables are created by the same deployment and do not exist yet - the deployment fails outright.

This enforcement is inconsistent with the rest of the table's lifecycle: once a Genie Space has been created against a table, Unity Catalog does not block dropping that table even though a Genie Space still references it. The Genie Space is left pointing at a source that no longer exists (surfaced only as a UI warning on the agent page), which should not be possible if the CLI/platform is going to enforce the reverse direction (source-must-exist-before-agent-creation) so strictly.

We want CLI to be able to create the Genie Space even if the source doesn't exist at the time pf deployment so a clean deployment can happen when a Genie Space resource exist in the bundle or make these two enforcement points to be symmetric: if Databricks is going to require the source to exist before a Genie Space can reference it, it should equally block deleting/dropping that source while a Genie Space still references it.

Notably, databricks bundle destroy already handles this correctly: it tears down the Genie Space and the schema/tables together as part of the same bundle teardown, with no errors. It's only the two independent, out-of-band actions - bundle deploy (create) and DROP TABLE/DROP SCHEMA (delete) - that are inconsistent with each other.

Configuration

Sanitised relevant bundle resource configuration (real shape from our repo, genie_spaces resource type):

resources:
  genie_spaces:
    iris_genie_space:
      title: ${var.workflow_prefix}ABC Genie Space
      description: |
        ...
      warehouse_id: ${var.sp_warehouse_id}
      serialized_space:
        version: 2
        config:
          data_sources:
            tables:
              - identifier: ${var.catalog}.${var.schema_prefix}refined_source.bac_123
          instructions:
            text_instructions:
              - content:
                  - |-
                    ...

The bundle provisions an isolated Unity Catalog schema per Git branch/feature deployment (via catalog / schema_prefix bundle variables), and other resources in the same bundle (jobs/pipelines) create the schema and populate refined_source.bac_123 as part of the same databricks bundle deploy via a job_runs resource.

Steps to reproduce the behavior

1. Define a genie_spaces resource in a bundle's databricks.yml, with data_sources.tables referencing a table created by another resource (job/pipeline) in the same bundle.
2. Deploy the bundle for the first time against a brand-new target/branch where the schema and table do not yet exist (e.g. databricks bundle deploy -t feature_branch --var="branch_name=<new_branch>").
3. Observe the deployment fails because the Genie Space cannot be created/validated against a non-existent source table. (screenshot below)
4. As a separate step, deploy the bundle again (or redeploy just the Genie Space resource) after the schema/table has been created by a prior successful run — the Genie Space now deploys successfully.
5. Run DROP TABLE <catalog>.<schema_prefix>refined_source.bac_123 (or drop the schema) directly, while the Genie Space above still references it.
6. Observe the drop succeeds with no error/warning at drop time, even though a live Genie Space references the table. The Genie Space itself is not removed, disabled, or blocked from continuing to exist.

Expected Behavior

Enforcement of the source ↔ Genie Space coupling should be symmetric. Concretely, we'd like:

• Allow the genie_spaces bundle resource to deploy successfully even when the referenced source table does not yet exist, deferring the "source missing" state to the existing UI-level warning shown on the Genie Space page (i.e. the same warning that already appears today after a table is dropped post-creation).

If that referential-integrity enforcement isn't desired on the create side, then the correct fix is to relax the drop side instead, since databricks bundle destroy already proves the CLI is capable of managing both together:

• DROP TABLE / DROP SCHEMA to be blocked (or require an explicit override, similar to how Unity Catalog already blocks dropping objects that have dependent views/constraints) when a Genie Space still lists that table in data_sources.tables - mirroring the same strictness the CLI already applies at Genie Space creation time.

Either fix removes the current asymmetry; preferably creation of Genie Space without source's existence is needed.

Actual Behavior

• First-time deployment of a bundle containing a genie_spaces resource fails outright when the referenced table does not already exist, blocking a clean, single-pass bundle deploy for new branches/environments.
• DROP TABLE/DROP SCHEMA succeeds with no check for, or warning about, Genie Spaces that reference the object being dropped, even though the CLI enforces the opposite dependency direction at deploy time.
• databricks bundle destroy is the one place this is handled correctly - it removes the Genie Space and its source schema/tables together, with no ordering errors.

Why the usual workaround is unsuitable

The only workaround is to split the deployment into two manual passes: first deploy everything except the genie_spaces resource so the schema/table is created, then deploy again (or deploy the Genie Space resource in isolation) once the source exists. This defeats the purpose of a single declarative bundle deployment and is not viable for CI/CD pipelines that deploy an entire bundle atomically per branch (our case: jobs, schemas/tables + Genie Space are meant to be provisioned together for each feature branch). It also does nothing to address the reverse gap — nothing stops a later DROP TABLE from silently orphaning the Genie Space's reference.

OS and CLI version

• OS / CI runner: Linux
• Databricks CLI version: 1.15.0
• Still reproducible with latest CLI: yes

Is this a regression?

No

Debug Logs

Output from a databricks bundle deploy command where the bundle has a genie_space resource without the underlying data source in existence.

Below is the screenshot from a Genie Agent that exist after the underlying data source was dropped.

Additional Context

We provision an isolated Unity Catalog schema per Git branch/feature deployment (via bundle variables), and a Genie Space is intended to be created against that schema as part of the same bundle deployment. Because the Genie Space resource cannot be created before its source table exists, every new branch's first deployment fails, forcing us to either exclude the Genie Space resource from new-branch deployments or accept a broken first deploy. Since databricks bundle destroy already proves the platform can coordinate deletion of a Genie Space and its source together, we'd like that same coordination applied consistently - either by relaxing the create-time requirement to match how loosely the two are actually coupled after creation, or by blocking the drop of a referenced source.