Monitoring

You can monitor many metrics for each of your pipelines and blocks.

Soon, you’ll be able to monitor aggregate metrics across all pipelines and blocks.

Metrics

Here are the following metrics that are monitored for each pipeline and block:

• Pipeline runs completed successfully
• Pipeline runs failed
• Block runs completed successfully
• Block runs failed
• How long each block ran for
• CPU usage for Kubernetes block and pipeline runs in Mage Pro
• Memory usage for Kubernetes block and pipeline runs in Mage Pro
• Data validation success for each block (coming soon)
• Data validation failure for each block (coming soon)

For pipelines that run with the Kubernetes executor in Mage Pro, the pipeline monitoring dashboard can show historical peak CPU and memory usage, sample counts, and copied resource request and limit recommendations based on stored run metrics. See Monitoring with Prometheus to set up Prometheus as the preferred resource usage source.