- Notifications
You must be signed in to change notification settings - Fork 295
Add --prod_db flag option for --dbt diffs #761
Description
Is your feature request related to a problem? Please describe.
We have multiple prod databases that we frequently want to run data-diffs against. With the current options for setting the prod_database value (the two options outlined here in the docs), it's cumbersome to switch between prod_databases.
Describe the solution you'd like
Add a flag that can be passed in for --dbt diffs that allows us to define the prod_database data-diff should compare against.
Currently the prod_database to diff against can be defined in the two ways outlined here in the docs. This would be a third way to define the prod_database to diff against.
I'm envisioning the command being something like: data-diff --dbt --prod_db <my_prod_db_name>
Describe alternatives you've considered
The other ways to switch which prod database we diff against are:
- update the prod_database variable in the dbt_project.yml file each time
- run
dbt ls -t prodto produce a manifest.json file and then run data diff with the--stateflag (as outlined here)dbt ls -t prodtakes many minutes to run and makes for a frustrating development process for the data engineers on the team who are trying to use data-diff. Ultimately they end up skipping the data-diff if it's this slow
Additional context
Discussed with @vvkh in this Slack thread