Data Pipeline Monitoring: Detecting Anomalies in ETL Jobs with SQL,\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n

Create a complete data pipeline monitoring system using SQL for [COMPANY NAME]'s data warehouse with [NUMBER] tables and [NUMBER] daily ETL jobs.\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n**Monitor 1 — Data Freshness:**\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Query: last update timestamp per table vs. expected refresh time\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Alert when: table not updated within [X] hours of expected time\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- SQL: SELECT table_name, MAX(updated_at) FROM information_schema...\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n**Monitor 2 — Volume Anomalies:**\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Query: row count today vs. 7-day average and 30-day average\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Alert when: row count deviates >[X]% from average\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- SQL: Compare COUNT(*) with LAG() window function over 30 days\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Handle seasonality: compare with same day of week last week\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n**Monitor 3 — Data Quality Checks:**\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- NULL rate per critical column (alert if >[X]%)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Duplicate detection: COUNT(*) vs COUNT(DISTINCT primary_key)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Referential integrity: orphan records (FK with no matching PK)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Value range validation: negative prices, future dates, impossible values\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n**Monitor 4 — Schema Change Detection:**\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- New columns added or removed\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Data type changes (INT to VARCHAR)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Constraint violations after schema migration\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n**Monitor 5 — Pipeline Dependency Graph:**\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Table dependency map (which tables feed which)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Impact analysis: if table X fails, which dashboards break?\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Cascading failure detection\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n**Implementation:**\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- dbt test definitions (if using dbt)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Or standalone SQL scripts scheduled via [Airflow/Cron/GitHub Actions]\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Alert routing: Slack channel per severity\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n- Dashboard: monitoring status page with all checks\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\nWrite complete SQL for each monitor with alert thresholds.