Rule Detail page

Avatar
DvSum Support
Follow

In this Article:

Overview:

Many features are added on the Rule detail page like offline and online execution capabilities, providing users with greater flexibility. The alert status of each rule is displayed as either "Healthy" or "Alerting" based on specified thresholds. Users can now easily edit rule definitions, descriptions, priorities, and scope, including window types and lookback days. Data aggregation into buckets enables detailed data quality checks. Various threshold types allow tracking and alerting based on exceptions. Rule notifications, action statuses, and scheduling options have been enhanced for improved rule management. The history tab provides execution charts and grids for comprehensive performance analysis. Overall, all these features enhance rule customization, monitoring, and analysis which will be explained in this article.

 

Initial State of a Newly Created Rule:

Before moving to functionalities let's go through when any rule is created. The "Data" tab and "History"
tab is disabled initially when the rule is created.
Here on the "Run Rule", the user can either run the rule online or offline. For more information regarding
Run Online & Run Offline, please visit the article "Run Online/Run Offline". Once the Rule is run, the "Data"
tab (if there are exceptions) & History tab gets enabled:
Now we will go through the major functionalities that have been implemented in the User Interface are
highlighted in the screenshot below for a quick review.

 

Key Functionalities:

The Rule Detail page has a significant number of features Let's explore these enhancements in detail:

Not Scheduled

A notification at the top of the page is shown which indicates whether the rule is scheduled. If the rule is not scheduled it will not run automatically.

Alert status

Alert status for the rule will be shown as  

  • Healthy

    - if the metric value is within specified threshold limits, then the alert status will be Healthy 

  • Alerting

    - If the metric value is not within specified threshold limits, then the status will be Alerting

Run Result

  • Passed

    - If the metric value has zero exceptions then the run result will be passed and the alert status will be healthy.

  • Failed

    - If the metric value is not within the defined threshold then the run result will fail and the alert status will be alerting applicable to Unique values, Freshness, Metric, and Count rule.

Run:

  • Online & Offline rules can be run directly from the Rule detail page
  • For Details please check the article Run Online/Offline

Edit Functionality: 

This button will allow the user to update the following: 

  • Overview

    • Rule Description: Users can update the description for the rule 
    • Priority: Users can update the priority for the rule 

  • Open Rule Definition: Users can directly open the rule definition page from here

Attributes

Users can add any tags to the rule

  • Scope 

    • Metric Time

      If the metric time field is selected on the table level. Selected field names will be inherited from the rule as well
      Otherwise, the user can also select/update the metric time field at the rule level as well

  • Users can set the window type: - Used to define what is the scope of data to be selected. 

    • All Data - By Default all the rules will window type as All data, which will consider all the data in the table during the  execution of the rule 

  • Data Max Time - In the case of incremental data, we can choose this option to run validation only on newly added data based on the timestamp available on the table 

  • Clock Time -  In the case of incremental data, we can choose this option to run validation only on newly added data based on the current timestamp available on the table 

Example:

There is a rule scheduled to run on 2024-01-06 at 11:00:00. This rule is applied to the "UPDATE_DT" column. Please review the sample data below, which produces different outputs based on the selected window type:

  • Users can set the Lookback days for Data max time & Clock time  
  • Users can optionally aggregate data into Buckets and DQ checks will be performed on each bucket. 
    • No bucket 
    • 1 Day  
    • 1 Hour 
  • Users can select available Slicer options

Alert Thresholds

Summary Metric Configuration

The Summary Metric section under Alert Thresholds allows users to define aggregated metrics that can be used for alert evaluation. This enables rule execution based on summarized values rather than only row-level conditions.

Purpose

Summary Metrics provide the ability to evaluate data using aggregation functions such as maximum, minimum, sum, or average. This helps in performing high-level data quality checks and defining alert conditions based on overall dataset behavior.

Configuration

To configure a Summary Metric:

  1. Navigate to Edit Settings → Alert Thresholds
  2. In the Summary Metric section, define the following:
  • Aggregation (Required)
    Select an aggregation function such as MAX, MIN, SUM, or AVG.
  • Column (Required)
    Select the metric column on which the aggregation will be applied (for example, Customer Account Balance).

Formatting Options

Users can enable formatting to control how metric values are displayed.

When Enable Formatting is selected, the following options are available:

  • Currency Symbol
    Define how the currency symbol is displayed:
    • Prefix (e.g., $100)
    • Suffix (e.g., 100$)
    • None
  • Thousand Separator
    Improves readability for large numbers. The following options are available:
    • Comma separator (e.g., 1,000,000)
    • Dot separator (e.g., 1.000.000)
  • Decimal Precision
    Allows users to define the number of decimal places to display.

If formatting is not enabled, the system uses raw numeric values for evaluation and display.

Adding Multiple Metrics

Users can add multiple Summary Metrics using the Add Field option. Each metric can be configured independently with its own aggregation, column, and formatting settings.

Threshold Configuration

The Threshold Configuration section defines how the system evaluates conditions to trigger alerts.

Alert Metric

The Alert Metric field determines which value the system uses to evaluate alert conditions.

Available options include:

  • Run Result
  • DQ Score
  • Summary Metrics (if configured)

Default Behavior

  • By default, the Alert Metric is set to Run Result
  • This means alerts are evaluated based on the rule execution output at the row level

Dependency on Summary Metrics

  • The Alert Metric field is partially dependent on Summary Metric configuration
  • If no Summary Metric is created:
    • Only default options such as Run Result and DQ Score are available
    • Summary Metric options remain unavailable
  • Once at least one Summary Metric is created:
    • The Alert Metric field is enabled to include Summary Metric options
    • Users can select any configured Summary Metric (e.g., MAX - Customer Acct Bal)

Threshold Type

The Threshold Type defines how alert conditions are evaluated for the selected Alert Metric.

Users can select from the following options:

No Threshold

  • No threshold condition is applied
  • The rule executes, but no alerts are triggered based on threshold evaluation
  • Useful when users want to monitor results without generating alerts

Number

  • Alerts are triggered based on fixed numeric values
  • Users can define:
    • Upper Bound
    • Lower Bound
    • Both
  • Example:
    Trigger an alert if the metric value is greater than or less than a specified number

Adaptive

  • Thresholds are automatically calculated based on historical data patterns
  • The system dynamically adjusts thresholds to detect anomalies
  • Useful for datasets where values fluctuate and fixed thresholds are not effective

Percentage Increase

  • Alerts are triggered based on percentage change in the metric value
  • Compares current execution results with previous runs
  • Useful for identifying sudden spikes or drops
  • Example:
    Trigger an alert if the value increases by more than a defined percentage compared to the previous run

Example: Using Summary Metric for Alerting

Use Case

A financial dataset contains a column Customer Account Balance, and the goal is to monitor unusually high account balances.

Configuration

  • Aggregation: MAX
  • Column: Customer Account Balance
  • Summary Metric Created: MAX – Customer Account Balance
  • Alert Metric: MAX – Customer Account Balance
  • Threshold Type: Number
  • Condition: Upper Bound = 100,000

How It Works

  • During rule execution, the system calculates the maximum value of the Customer Account Balance column
  • Suppose the data contains values:
    25,000; 40,000; 75,000; 120,000
  • The Summary Metric (MAX) = 120,000
  • This value is compared against the defined threshold (100,000)

Result

  • Since 120,000 > 100,000, the rule triggers an alert

Workflow

  • Assign a Rule to a User
    Assign the rule to a specific user for ownership and accountability.

  • Add/Update the Schedule for a Rule
    Select or create a scheduled job to automate rule execution.

  • Exception Configuration (New)

    • Attach Exception File: Includes exception details in the notification email.
    • Enable Exception Tracking: Allows tracking of new and existing exceptions across runs.

For a detailed explanation of Exception Configuration and how it impacts the Data tab, refer to the article: Configuring Exception Settings for Rules in DvSum.

Work Flow Actions:

Before moving to Workflow Actions, we need to add a Data Quality workflow enabled  Data domain to our table. On the "Data Domain" tab create a new domain or use an already existing one:

The "Data Quality Workflow" checkbox should be enabled on creating the new Data Domain:

Now on the "Overview" tab on the Table detail page, the above Data domain needs to be added:

Once we have added the Data Quality workflow Data Domain to our table we can then add any scheduled job from the drop-down

Workflow actions are going to appear at the top right

Actions dropdown will contain In Progress and Resolve

When the user marks the rule as In Progress, the status will be changed to "In Progress".

When the user resolves the rule, a Pop-up will be displayed with the Reason Code and Description

Workflow status is changed to resolved

One thing to note here is that the "Actions" button and its options on the top right show if the Rule has met the following conditions:

  1. The rule must have exceptions

  2. The rule must be executed

  3. The Table on which the Rule is created must have the Data domain added for which the Data Quality workflow is enabled

  4. The scheduler must be attached to the Rule

Just so you know, workflow actions will also show when ticketing is enabled on the rule. For more details on Ticketing, please take a look at the article JIRA Integration on DQ Rules.

The workflow status is Activity bar. Clicking on the comments icon opens the detailed activities.

On the Activity bar, clicking on the comments icon will open the detailed activities.

Tabs on Rule Detail Page:

On the Lower Section of the Rule detail page, the user will be able to see only the 3 tabs by default:

 

Users can click on Show More to view the other tabs available:

After clicking on the Show More tab, the user will see the following tabs and if the user wants to hide them they simply need to click on Hide Others

Data Tab:

Data View

The "Data" tab is available after the rule is executed, regardless of whether exceptions are generated. By default, exceptions are highlighted in red for the column(s) where the rule is applied. All exceptions, along with the corresponding data in the table, are displayed.

Users can filter the exceptions using the drop-down menu with the following options:

    • Current: Displays exceptions from the most recent execution.

    • 30 Days: Shows exceptions from the past 30 days.

    • 90 Days: Shows exceptions from the past 90 days.

    • All: Displays the metric value for all available data, up to a maximum of the last 120 days (selectable via the drop-down).

Note: Only the first 300 exceptions are displayed in the Grid on the Data tab. if needs to view more exceptions, must enable the “Include all Exceptions” checkbox in the Run Online dropdown before running the rule.

One thing is to be noted here that the "Data" tab will be shown for the following rules only:

  1. Data Format

  2. Blanks

  3. Value Range

  4. Uniqueness

  5. Ruleset

  6. Custom Query

  7. Orphan Keys

  8. Orphan Records

  9. Compare Schema

  10. Compare Metric

Note: The Data tab supports column-level filtering and allows copying specific rows or column values directly from the exception grid for easier analysis.

 

 

Pivot View.

The Data tab provides a detailed, interactive view of rule execution results, with a focus on exception data through the Pivot View.

The Pivot View in Data Quality Rules enables users to analyze exceptions by grouping and aggregating records directly within the application, helping identify patterns and derive insights.

For more information, refer to the sub-article “Pivot View Configuration in Data Quality Rules.”

 

History:

  •  

    • The history tab will show the execution history for the rule in the form of a chart

    • The History tab has 2 different views which are:

  • Chart View - 

    In this view, the data is shown to the user in the form of a chart

When the slicer is selected, the data in the History Chart will get distributed based on the Slicer's Field

 

  • Grid View -

    In this view, the same data is shown to the user in a tabular form

Users can view execution history for: 

  • Current: It will show the metric value for the current execution

  • 30 Days: It will show metric value for 30 days

  • 90 Days: It will show the metric value for 90 days

  • All: Displays the metric value for all available data, up to a maximum of the last 120 days.

Instructions:

Users can add any instructions.

Column Sequence:

On the "Column Sequence" tab, the users can set 3 different sequences that can be applied to the Rule:

  1. Suggested Sequence

  2. Table Specific Sequence

  3. Rule Specific Sequence

For more details on Column Sequence, there is a separate article written for "Column Sequence". 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk