Skip to content
This repository was archived by the owner on May 17, 2024. It is now read-only.
This repository was archived by the owner on May 17, 2024. It is now read-only.

Add --prod_db flag option for --dbt diffs #761

@mimoyer21

Description

@mimoyer21

Is your feature request related to a problem? Please describe.
We have multiple prod databases that we frequently want to run data-diffs against. With the current options for setting the prod_database value (the two options outlined here in the docs), it's cumbersome to switch between prod_databases.

Describe the solution you'd like
Add a flag that can be passed in for --dbt diffs that allows us to define the prod_database data-diff should compare against.
Currently the prod_database to diff against can be defined in the two ways outlined here in the docs. This would be a third way to define the prod_database to diff against.
I'm envisioning the command being something like: data-diff --dbt --prod_db <my_prod_db_name>

Describe alternatives you've considered
The other ways to switch which prod database we diff against are:

  • update the prod_database variable in the dbt_project.yml file each time
  • run dbt ls -t prod to produce a manifest.json file and then run data diff with the --state flag (as outlined here)
    • dbt ls -t prod takes many minutes to run and makes for a frustrating development process for the data engineers on the team who are trying to use data-diff. Ultimately they end up skipping the data-diff if it's this slow

Additional context
Discussed with @vvkh in this Slack thread

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions