Skip to main content
2 of 2
deleted 1 character in body
bunta
  • 702
  • 12
  • 26

ModuleNotFoundError - Airflow error while import Python file

I created a very simple DAG to execute a Python file using PythonOperator. I'm using docker image to run Airflow but it doesn't recognize a module where I have my .py file

The structure is like this:

main_dag.py plugins/__init__.py plugins/njtransit_scrapper.py plugins/sql_queries.py plugins/config/config.cfg 

cmd to run docker airflow image:

docker run -p 8080:8080 -v /My/Path/To/Dags:/usr/local/airflow/dags puckel/docker-airflow webserver 

I already tried airflow initdb and restarting the web server but it keeps showing the error ModuleNotFoundError: No module named 'plugins'

For the import statement I'm using:

from plugins import njtransit_scrapper 

This is my PythonOperator:

tweets_load = PythonOperator( task_id='Tweets_load', python_callable=njtransit_scrapper.main, dag=dag ) 

My njtransit_scrapper.py file is just a file that collects all tweets for a tweeter account and saves the result in a Postgres database.

If I remove the PythonOperator code and imports the code works fine. I already test almost everything but I'm not quite sure if this is a bug or something else.

It's possible that when I created a volume for the docker image, it's just importing the main dag and stopping there causing to not import the entire package?

bunta
  • 702
  • 12
  • 26