I try to run a simple DAG that asynchronously submits beam python pipeline and the sensor that waits for the pipeline completion, as suggested in airflow/beam best practice.
According to the official airflow documentation, the Airflow operator should asynchronously submit the Beam job, where the wait_for_the_beam_job task is checking the status of the submit_beam_job and mark the sensor task as Done when the job finishes.
Unexpectedly, the beam is running synchronously. Have you any idea how can this issue be solved? Here is my minimal reproducible example: https://github.com/MichaelDuemcke/minimal-reproducible-example-airflow-waiting-beam
