Update: Forget the 'best' way, any way to start with is good enough. I am new to this and figuring it out all by myself is taking too long. I don't even know where to start.
I am new to this and unsure where to start.
I have 5 ML models deployed as 5 microservices, each with its own Docker image. Together they form a pipeline, and the services act sequentially on the input data. Is there a way to orchestrate this using Dagster or something similar? Should I?
Currently I am activating each stage manually while developing it, but the time has come to do this in a better way.
One option is to choreograph it, where each service, when it successfully completes its steps, sends a message to the next service via the message broker to retrieve the intermediary data and continue the pipeline.
Other is to orchestrate it using Dagster or Airflow. There is also Ray and Dask and I am not sure what to do here.
I have heard software like Dagster can be good for both approaches, but I don't know how to execute this.
If I do execute this, can each service container still be scaled independently via Docker Swarm or K8s?
Some good material and suggestions will be most appreciated.
