Configure Databricks as a Source

In this article:

• Overview
• Prerequisites
• Step-by-Step Configuration
  • Step 1: Adding a Databricks Source in DvSum
  • Step 2: Configure Connection
  • Step 3: Select Database
  • Step 4: Save & Test Connection
  • Step 5: Scan the Data Source
• Authentication Options
  • Scenario 1: Authentication via Access Token
  • Scenario 2: Authentication via Client Secret
• Reviewing Scan Insights

Overview:

Azure Databricks is an optimized platform for Azure, offering tight integration with services like Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, and Power BI. It allows data storage in a unified, open lakehouse while consolidating analytics and AI workloads.

This article describes the process of configuring Databricks as a data source in DvSum, facilitating the integration of data cataloging and profiling. The steps outlined apply to both DvSum Data Insights (DI) and DvSum Data Quality (DQ), with minor platform-specific variations.

Adding Databricks source in DvSum:

Prerequisites:

Enabling Query History for Databricks 
Before configuring Databricks as a source, ensure that query history is enabled for your Databricks account. This is crucial for tracking data lineage and gaining insights into usage patterns. For more information, refer to the Enabling Query History for Data Sources article.
 

Cluster and Account Setup: For authentication of the Databricks Source, a user must have an account on the Azure Databricks portal on which a cluster is running attached to a database. On the Azure Databricks portal, go to the Compute tab and start your cluster if it is in the stop state.

Required Table Access for Catalog Scans

To support cataloging and schema discovery in Databricks—especially for use cases involving the ability to pick and choose catalogs—it is important to have read-only access to the following system tables:

• system.information_schema.catalogs

• system.information_schema.schemata

• system.information_schema.tables

• system.information_schema.columns

Without access to these tables, cataloging scans will not function as expected. Please ensure this access is in place before initiating scans.

Step-by-Step Configuration

Step 1: Adding a Databricks Source in DvSum

1. Navigate to Data Sources.
2. Click on Add Source.
3. In the modal, select Databricks.
4. Provide a source name and click Save.

Step 2: Configure Connection

Once the source is saved, you will be redirected to the connection settings detail page of the new Databricks source. First, enable the On-premise Web Service checkbox and select the appropriate SAWS (Secure Access Web Service) that is currently set up and running.

You can authenticate using either:

• Access Token

• Client Secret

Note: By Default the SAWS type will be cloud. For more information regarding Cloud SAWS, click here

Authentication Options

You can authenticate using either:

• Access Token
• Client Secret

Scenario 1: Authentication via Access Token

1. Enable the On-Premise Web Service checkbox.
2. Select the SAWS (Secure Access Web Service) that is set up and running.
3. Enter the following details:
   • Server Hostname
   • HTTP Path
   • Personal Access Token
4. Click Authenticate.

Scenario 2: Authentication using Client Secret

~Prerequisites for Configuring Azure Databricks (Service Principal Service):

Please refer to the article to configure Azure Databricks (Service Principal Service).

Under the Host Information section, select Client Secret.

Enter the following details:

•  

  • Server Hostname

  • HTTP Path

  • Azure Client Id

  • OAuth Secret

Click Authenticate.

Note: To optimize job performance and memory usage with OAuth Secret (a confidential key used to securely authenticate and authorize applications when integrating with external services), select the checkbox and enter the OAuth Secret value. This will automatically initiate and connect to the clusters prior to execution.

The OAuth Secret can be generated by the admin from the Service Principal's secret tab.

 

Step 3: Select Database

Once authenticated, the Database section will appear.

1. Select the appropriate Catalog Name from the dropdown.

2. (Optional) To restrict scanning to specific schemas, enable the Limit to specific schemas checkbox and choose the required schemas.

Step 4: Save & Test Connection

1. Scroll to the top and click Done.
2. Click Save.
3. Click Test Connection to validate the setup.

After that click the “Save” button. The source will get saved successfully and after that click on the “Test Connection” button. 

Note: When adding or editing a Data Source, if incorrect details or invalid credentials are entered, it will still allows to save the Data Source. However, authentication will fail, and the Data Source will be marked as unusable with a red icon. The Data Source will not be usable until the correct credentials are provided.

The connection is saved, but it cannot be used until valid authentication details are updated.
 

Step 5: Scan the Data Source

1. Navigate to Scan History.
2. Click Scan Now.
3. Wait for the job status to change to Completed.
4. Click on the Scan Name to view the Scan Summary

Reviewing Scan Insights

1. Go to the Dictionaries dropdown and select the Data Dictionary tab.
2. Click on Recently Refreshed to view newly discovered tables.
3. Click on table names to explore metadata details.