Pipeline Dashboard
Real-time monitoring · AWS us-east-1 · Updated just now
Records / sec
6,284
Data Quality
99.4%
Avg Latency
2.8s
Pipeline SLA
99.8%
⬡
Live Data Flow — Production
Running
🌐
REST
APIs
APIs
1.2k/s
▶
ingest
🪣
Amazon
S3
S3
4.7 TB
▶
trigger
⚙️
AWS
Glue
Glue
3 jobs
▶
ETL
△
Delta
Lake
Lake
ACID ✓
▶
load
🔴
Redshift
DW
DW
8.2 TB
▶
serve
📊
Power
BI
BI
10 dashboards
Throughput — 24h Healthy
Pipeline Runs — 7d success / fail
Active Glue Jobs 3 running · 1 queued
| Job Name | Status | Duration | DPUs | Records | Source → Target | Last Run |
|---|
Pipeline Registry
All ETL/ELT workflows · production
| Pipeline | Type | Schedule | Status | Avg Duration | SLA | Last Success |
|---|
Data Quality
Validation rules · accuracy · completeness
Overall Score
99.4%
across 6 datasets
Quality Trend — 14d
Validation Rules
| Rule | Dataset | Type | Pass Rate | Failures | Status |
|---|
Redshift Cluster
ra3.4xlarge · 4 nodes · us-east-1
CPU Usage
42%
Storage
8.2 TB
Query/min
284
Connections
48
WLM Queues
Cluster Performance
Top Queries by Duration
| Query ID | User | Duration | Rows | Queue | Status |
|---|
Airflow DAGs
Apache Airflow 2.7 · MWAA · 12 active DAGs
DAG Run History — Last 10 runs
Pipeline Logs
CloudWatch · streaming
Cost Monitor
AWS monthly spend · data engineering resources
MTD Spend
$4,820
Projected
$6,100
Savings (IaC)
$1,840
Top Service
Redshift
Spend by Service
30-day Trend