Quality Check

Data Quality Rules

Overview

The Data Quality Rules feature helps ensure data consistency and reliability across database pipelines by validating source and destination data during pipeline execution.

Users can configure automated quality checks, define acceptable variance thresholds, schedule validations, receive notifications, and review execution history.

Navigation

Navigate to:

Database Pipeline → Pipeline Configuration → Quality Rules

The Quality Rules tab allows users to:

• Configure data quality checks

• Define thresholds

• Schedule executions

• Configure notifications

• View execution history

• Enable or disable rules

Features

1. Row Count Difference Validation

The system compares row counts between source and destination datasets and displays:

• Source row count

• Destination row count

• Difference count

• Variance percentage

This helps identify missing or inconsistent data during pipeline execution.

2. Threshold Configuration

Users can define acceptable variance limits between source and destination data.

Supported threshold types:

• Percentage-Based Threshold (Default: 10%)

• Absolute Value Threshold

If the variance exceeds the configured threshold, the quality check is marked as failed.

3. Scheduling and Notifications

Quality checks can be:

• Executed manually

• Scheduled at predefined intervals

Email notifications are triggered when:

• A scheduled quality check executes

• A mismatch exceeds the configured threshold

Multiple recipient email addresses can be configured from the Settings panel.

4. History and Audit Tracking

All quality check executions are stored in the History section.

The History page provides:

• Execution details

• Comparison statistics

• Validation results

• Failure messages

• Schema comparison details

Access History from:

Quality Rules → Actions → History

5. Enable or Disable Quality Checks

Users can enable or disable quality checks at any time from the Quality Rules section based on operational requirements.

Summary:

The Data Quality Rules feature provides automated validation, configurable thresholds, scheduled monitoring, notifications, and historical tracking to help maintain reliable and consistent data across database pipelines.