Spark

Using Databricks connect.

!pip3 install --upgrade "databricks-connect==13.0.*"

Or X.Y.* to match your cluster version

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.remote(
  host       = "my_host",
  token      = "my_token",
  cluster_id = "my_cluster_id",
  
).getOrCreate()