Quality Check
Data Quality Rules
Overview
The Data Quality Rules feature helps ensure data consistency and reliability across database pipelines by validating source and destination data during pipeline execution.
Users can configure automated quality checks, define acceptable variance thresholds, schedule validations, receive notifications, and review execution history.
Navigation
Navigate to:
Database Pipeline β Pipeline Configuration β Quality Rules
The Quality Rules tab allows users to:
Configure data quality checks
Define thresholds
Schedule executions
Configure notifications
View execution history
Enable or disable rules

Features
1. Row Count Difference Validation
The system compares row counts between source and destination datasets and displays:
Source row count
Destination row count
Difference count
Variance percentage
This helps identify missing or inconsistent data during pipeline execution.

2. Threshold Configuration
Users can define acceptable variance limits between source and destination data.
Supported threshold types:
Percentage-Based Threshold (Default: 10%)
Absolute Value Threshold
If the variance exceeds the configured threshold, the quality check is marked as failed.

3. Scheduling and Notifications
Quality checks can be:
Executed manually
Scheduled at predefined intervals
Email notifications are triggered when:
A scheduled quality check executes
A mismatch exceeds the configured threshold
Multiple recipient email addresses can be configured from the Settings panel.

4. History and Audit Tracking
All quality check executions are stored in the History section.
The History page provides:
Execution details
Comparison statistics
Validation results
Failure messages
Schema comparison details
Access History from:
Quality Rules β Actions β History

5. Enable or Disable Quality Checks
Users can enable or disable quality checks at any time from the Quality Rules section based on operational requirements.

Summary:
The Data Quality Rules feature provides automated validation, configurable thresholds, scheduled monitoring, notifications, and historical tracking to help maintain reliable and consistent data across database pipelines.
Last updated