Added 'databricks bundle plan' command

[andrewnester]

Changes

Made 'databricks bundle plan' command public (previously hidden)

Why

This command builds the bundle and displays the actions that will be performed on the deployed resources without making any changes.
It is useful for previewing changes before running bundle deploy.
The output of the command is high level list of actions and resources which are changed, for example

>>> databricks bundle plan

create job.my_project_job
delete job.sample_job
create pipeline.my_project_pipeline
delete pipeline.sample_etl

Plan: 2 to add, 0 to change, 2 to delete, 3 unchanged

Tests

Covered by existing tests

[denik]

quick question - why it is in PreRunE and not RunE?

[andrewnester]

Just to separate these two and run the check earlier

[eng-dev-ecosystem-bot]

Run: 17975134246

	Env	✅​pass	🙈​skip
✅​	aws linux	311	529
✅​	aws windows	312	528
✅​	aws-ucws linux	423	427
✅​	aws-ucws windows	424	426
✅​	azure linux	311	528
✅​	azure windows	312	527
✅​	azure-ucws linux	423	426
✅​	azure-ucws windows	424	425
✅​	gcp linux	310	530
✅​	gcp windows	311	529

[lennartkats-db]

What is the exact expected behavior of bundle plan?

Running it locally:

$ databricks bundle plan
Building python_artifact...
Uploading dist/my_project-0.0.1-py3-none-any.whl...
create jobs.my_project_job
delete jobs.sample_job
create pipelines.my_project_pipeline
delete pipelines.sample_etl

• It seems odd that it builds artifacts as a side effect (but maybe this is necessary?)
• It seems odd that it uploads artifacts as a side effect (edit: this appears to be fixed in Fix bundle plan to not create workspace objects or upload the files #3442)
• Should there not be a summary line, like in Terraform (Plan: 2 to add, 0 to change, 2 to destroy)?
• Should we not use the term "destroy" instead of "delete"? Since that is also the term used for the databricks bundle destroy?

We need to be certain of the interface before making it public

cc @denik

[denik]

Building an artifact is unfortunately required, because we don't know the final filename of the artifact and we need that filename in the resource configuration.

Uploading the artifact is certainly a bug and we should have acceptance test + fix for it. I think we should fix it before the release.

Summary is nice to have but could be a follow up.

Regarding delete/destroy, this is not new, this is what DABs always used in other places:

The following resources will be deleted:
  delete app myapp

[andrewnester]

@lennartkats-db are you sure you're using latest main? Not uploading artifacts as part of the plan was fixed here #3442

[lennartkats-db]

Cool, yes, I'm not using the latest main, good to see uploading is fixed.

It still seems to build the artifact though, but I suppose that might be necessary? It's quite unfortunate it prints a message about but if it is necessary then it should offer that transparency.

And delete is indeed used in other places. I also kind of like that word. But we should treat this moment as the time where we decide to cast that in
stone.

The summary I'd really want to have and seems easy to add.

One more observation: the output of the command should go to stdout, not stderr.

[denik]

Should we not use the term "destroy" instead of "delete"? Since that is also the term used for the databricks bundle destroy?

Deletion can happen as a result of "bundle deploy" as well. It's a more specific and recognizable term IMO. All APIs and SDKs call call this deletion, not destruction.

[lennartkats-db]

Cool. So how about

• We make sure output goes to stdout (this is the most crucial change)
• We add that summary line item
• We maybe do some minor formatting tweaks (to make the summary and llog line stand out + follow the two-space indent of existing warnings)

So output could look like

$ databricks bundle plan
Building python_artifact...  ### this would still go to stderr
Plan: 2 to add, 0 to change, 2 to delete
  create jobs.my_project_job
  delete jobs.sample_job
  create pipelines.my_project_pipeline
  delete pipelines.sample_etl

(I'd also be up for variations of this exact output, but I'm proposing example output to try close on this.)

[andrewnester]

Here's PR to improve the plan output #3546