feat: Adding DAG parsing profiler tool #13645
Conversation
product-auto-label bot added samples 
tiranux force-pushed the add-dag-parsing-profiler branch from a74ffb7 to 8552eb2 Compare 
| Here is the summary of changes. You are about to add 2 region tags.
This comment is generated by snippet-bot.
|
tiranux force-pushed the add-dag-parsing-profiler branch from 8552eb2 to 507bff9 Compare 
| When triggered, it offloads the resource-intensive DAG parsing process to a temporary, isolated Kubernetes Pod. Its primary goal is to detect **parsing latency issues** and identify heavy top-level code execution without impacting your environment's workload resources. As a byproduct of this analysis, it also validates DAG integrity and catches syntax errors. | ||
| | ||
| ## 🌟 Key Features | ||
| * **Isolated Execution:** Offloads parsing logic to a separate Pod, protecting the Scheduler from resource contention and crashes. |
There was a problem hiding this comment.
note, in Composer 3 dag processor is run separately from Scheduler, in a dedicated workload - let's add info about Composer 2/Composer 3 and when this tool is applicable
There was a problem hiding this comment.
Done, ommited the specific mention of the scheduler as refer to environment resources as it moving to the Kubernetes pod, isolates the workload of this analysis wether it is a Composer 2 or Composer 3.
There was a problem hiding this comment.
Added further clarification regarding the version for image setup (when no automatic detection).
2 participants
Description
Adding a new DAG tool to troubleshoot and profile DAGs parsing.
Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.
Checklist
nox -s py-3.9(see Test Environment Setup)nox -s lint(see Test Environment Setup)