Debug an Airflow DAG

Click-through Demo

See how an agent working in a workspace debugs Airflow DAGs quickly. By combining platform integrations and best-practice templates from the Knowledge Hub, the agent identifies issues, suggests fixes, and applies improvements automatically. Memory shared through the workspace boosts context and consistency for faster, smarter troubleshooting.

This walkthrough was recorded before Datamates became Workspaces, so it shows the old names. Today, the flow starts at Workspaces → New workspace in the web app.

Walkthrough

The above demo goes over the following steps to improve the debugging process. By creating a workspace, engineers can work in their IDE without content switching or pouring over airflow log files.

Creating a workspace: The presentation begins by showing how to create a workspace with the necessary integrations. The example uses an "Airflow tester" workspace configured with Memory, Airflow, GitHub, Databricks, and Jira integrations.

Adding Context from Knowledge Hub: Altimate's Knowledge Hub is introduced as a way to provide context to agents. It offers templates that users can fork and modify to fit their company's best practices. The "Airflow Cookbook" from the Knowledge Hub is used in this example to help debug Airflow DAGs. The URL of the knowledge document can be copied and provided to the coding assistant for context.

Debugging an Airflow DAG: A problematic Airflow DAG named asset1_producer is shown to have failed runs, indicated by red "X failed" statuses. The bug is identified as a ZeroDivisionError due to error_count being 0 in a success rate calculation.

AI-Assisted Triage and Fixes: The user prompts the coding assistant with the problem and provides the link to the "Airflow Cookbook" from the Knowledge Hub. The agent, using the workspace's memory and the provided context, identifies the ZeroDivisionError and suggests fixes based on the best practices outlined in the cookbook, such as proper error handling, input validation, numerical operations, structured logging, and exception handling. The agent then applies the recommended code changes.

Successful DAG Run and Memory Hub: After applying the fixes, the DAG runs successfully. Finally, the presentation highlights the Memory Hub, which stores memories shared across the workspace, leading to enhanced contextual understanding, increased efficiency, and reduced repetition.