Using multiple schemas in dbt Cloud based on postgresql instance

Question

I am using dbt Cloud to complete the tutorial to learn more about the tool. I'm using Postgresql because I don't have any access to the paid tools that are heavily supported.

I have orders and customers in the jaffle_shop schema and payment in the stripe schema. I have the following staging table for payment:

with payment as ( select orderid as order_id, amount from dbt.stripe.payment ) select * from payment

When I try to do the simple pull to test:

with payments as ( select * from {{ref('stg_payments')}} ) select * from payments

I get an error. I try the compile it keeps insisting on going back to the jaffle_shop default schema, even though I've been more specific about using stripe as above (compiled output below with wrong schema):

with payments as ( select * from "dbt"."jaffle_shop"."stg_payments" ) select * from payments limit 500 /* limit added automatically by dbt cloud */

Is there something I should do differently to make it go to the correct schema? Or is this a limit of dbt Cloud and Postgresql? Thank you.

tconbeer · Accepted Answer · 2022-06-29 19:39:54Z

You need to add a config to the stg_payments model to tell dbt to build that model in another schema. See the Docs for Custom Schemas

So in stg_payments.sql:

{{ config(schema='stripe') }} ...

Then, when you use {{ ref('stg_payments') }}, dbt will compile that to use the stripe custom schema. (However! note that dbt will prepend the target name to stripe to make the custom schema name, so dbt can create models in multiple dev and prod environments).

You can override this behavior (as noted in the docs), but I wouldn't recommend it, since you will lose the ability to develop in multiple environments.

Lastly, if you have a source table that is loaded into a specific schema (and not a model that is created/managed by dbt), you should reference that in your dbt models using the source() macro, instead of ref() -- the source config will allow you to specify the exact schema where the source data lives

Collectives™ on Stack Overflow

Using multiple schemas in dbt Cloud based on postgresql instance

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related