In this Article:

• Overview
• Smart Scan Compatible Sources
• What happens in Smart Scan
• How Smart Scan Works
• Smart Scan vs Full Scan Behavior
• Example Scenario
• Re-Profiling After Significant Data Changes 
• How to Handle This Case
• Performance Impact
• When to Use Smart Scan
• FAQs

Overview

Smart Scan is an optimized scanning option that reduces unnecessary profiling by detecting when metadata or schema changes actually require profiling. While a catalog scan always captures schema changes, profiling queries are executed only for tables that have changed since the previous scan.

By skipping expensive profiling queries when metadata remains unchanged, Smart Scan significantly reduces compute load on source systems and improves overall scan performance. This makes Smart Scan especially effective for large databases and environments with frequent or scheduled scans.

Smart Scan Compatible Sources

Smart Scan is supported on the following source types:

• Databricks
• Snowflake
• Oracle
• MySQL
• IBM DB2
• SQL Server
• Azure SQL
• Azure Synapse

How to run a Smart Scan 

Smart Scan evaluates whether profiling is necessary based on the schema. To initiate a Smart Scan, please refer to this image

Smart Scan option will be available only when profiling is selected as part of Scan job.

How Smart Scan Works

Step 1: Catalog Scan

• Connects to the source
• Identifies all tables and columns
• Detects schema level changes
• Updates the DvSum catalog

This step runs even when no changes are detected.

Step 2: Smart Profiling Logic

Databases provide timestamp of last modification to the schemas,. DvSum uses this field as CDC (change data capture) to detect any schema change.

For each table:

1. Compare existing modification timestamp with newly obtained from the database
2. If change is detected in the tables or columns then profiling queries will be executed on them only
3. If no change is detected in schema, then profiling will be skipped completely

This avoids running expensive profiling queries when schema has not changed.

Smart Scan vs Full Scan Behavior

Example Scenario

A data source contains 10 tables.

• First Smart Scan performs full cataloging and profiling
• Second Smart Scan detects no schema changes, hence profiling queries are skipped entirely
• Third Smart Scan detects change in one table only, as a result profiling queries will be executed on this table only

This results in significant time and compute savings, especially for large tables.

Re-Profiling After Significant Data Changes

Smart Scan determines whether profiling is required based solely on schema (metadata) changes detected using schema modification timestamps provided by the source database.

If a table’s data changes significantly — for example, large volumes of new rows are inserted, existing data is updated, or records are deleted — but the table schema itself remains unchanged, Smart Scan will not trigger re-profiling. In this scenario, column statistics and data distributions are not refreshed automatically.

This behavior is expected and intentional, as Smart Scan is designed to minimize unnecessary profiling and reduce load on source systems.

How to Handle This Case

If updated profiling results are required despite no schema change, you can:

• Run a full scan, or

• Run a profiling scan on a specific table or column

Performance Impact

Smart Scan significantly reduces scan duration and compute usage by skipping unnecessary profiling. Performance improvements are most noticeable when:

• Tables are large
• Scans are run frequently

When to Use Smart Scan

Use Smart Scan when:

• Data structure change infrequently
• Faster scan execution is desired
• Reduced load on the source system is important

FAQs

Is Smart Scan available for all sources?
No. Smart Scan is only supported on the compatible sources listed above.

Does Smart Scan skip cataloging?
No. Cataloging always runs to ensure schema accuracy.

What happens if table data changes but the schema does not change?
Smart Scan will skip profiling because it only triggers re-profiling based on schema (metadata) changes. If updated profiling results are required, a full scan or a targeted profiling scan should be used.