Quality Check
Data Quality Rules
Section titled “Data Quality Rules”Overview
Section titled “Overview”The Data Quality Rules feature helps ensure data consistency and reliability across database pipelines by validating source and destination data during pipeline execution.
Users can configure automated quality checks, define acceptable variance thresholds, schedule validations, receive notifications, and review execution history.
Navigation
Section titled “Navigation”Navigate to:
Database Pipeline → Pipeline Configuration → Quality Rules
The Quality Rules tab allows users to:
- Configure data quality checks
- Define thresholds
- Schedule executions
- Configure notifications
- View execution history
- Enable or disable rules
.png)
Features
Section titled “Features”1. Row Count Difference Validation
Section titled “1. Row Count Difference Validation”The system compares row counts between source and destination datasets and displays:
- Source row count
- Destination row count
- Difference count
- Variance percentage
This helps identify missing or inconsistent data during pipeline execution.
.png)
2. Threshold Configuration
Section titled “2. Threshold Configuration”Users can define acceptable variance limits between source and destination data.
Supported threshold types:
- Percentage-Based Threshold (Default: 10%)
- Absolute Value Threshold
If the variance exceeds the configured threshold, the quality check is marked as failed.
.png)
3. Scheduling and Notifications
Section titled “3. Scheduling and Notifications”Quality checks can be:
- Executed manually
- Scheduled at predefined intervals
Email notifications are triggered when:
- A scheduled quality check executes
- A mismatch exceeds the configured threshold
Multiple recipient email addresses can be configured from the Settings panel.
.png)
4. History and Audit Tracking
Section titled “4. History and Audit Tracking”All quality check executions are stored in the History section.
The History page provides:
- Execution details
- Comparison statistics
- Validation results
- Failure messages
- Schema comparison details
Access History from:
Quality Rules → Actions → History
.png)
5. Enable or Disable Quality Checks
Section titled “5. Enable or Disable Quality Checks”Users can enable or disable quality checks at any time from the Quality Rules section based on operational requirements.
.png)
Summary:
Section titled “Summary:”The Data Quality Rules feature provides automated validation, configurable thresholds, scheduled monitoring, notifications, and historical tracking to help maintain reliable and consistent data across database pipelines.