This page explains pipeline logging in Cloud Data Fusion.
For each data pipeline run, Cloud Data Fusion generates logs to help you monitor data pipeline runs and debug issues. As soon as a pipeline run begins, you can view the logs and filter them by log level.
You can access the logs in the Cloud Data Fusion Studio on the Preview page and the Pipeline details page. To monitor the progress of a pipeline, click Logs while the pipeline is running.
Filter logs by level
When viewing the pipeline log messages, you can filter them by type, or log level:
- Error
- Warning
- Informational
- Debug
- Trace
To set the log level, go to the Cloud Data Fusion Studio's Preview page or Pipeline details page, and click Logs > Level. Select the types of messages you want to appear in the log.
View basic or advanced logs
For each pipeline run, you can view basic or advanced logs. Basic logs include messages about the stages of the pipeline. Advanced logs include all basic log messages, and information about cluster health and the execution engine.
To view advanced logs during a pipeline preview or run, go to the Cloud Data Fusion Studio's Preview page or Pipeline details page, and click Logs > View advanced logs.
To return to viewing basic logs, click Hide Advanced Logs.
View the most recent log messages
If you have a pipeline run with a large number of messages, you can go to the bottom of the page for the latest messages. Go to the Cloud Data Fusion Studio's Preview page or Pipeline details page, and click Logs > Scroll.
Downloading logs
For debugging issues, after a pipeline run completes, you can download a log file that includes all of the advanced log and stack trace messages.
To download the log file, go to the Cloud Data Fusion Studio's Preview page or Pipeline details page, and click Logs > Download all.
To view the raw log as plain text in a new browser tab, click Download all > View raw logs.
What's next
- Learn more about orchestrating pipelines.