153

I'll need to invoke REFRESH MATERIALIZED VIEW on each change to the tables involved, right? I'm surprised to not find much discussion of this on the web.

How should I go about doing this?

I think the top half of the answer here is what I'm looking for: https://stackoverflow.com/a/23963969/168143

Are there any dangers to this? If updating the view fails, will the transaction on the invoking update, insert, etc. be rolled back? (this is what I want... I think)

4
  • I'm unclear what answer you are looking for which is not already covered in the post you linked to. Is your situation different in some fundamental way? Commented Apr 3, 2015 at 18:29
  • 2
    1) The answer did not satisfy the asker of that question 2) I am asking an at least somewhat different question. So I do not know if that answer perfectly applies to my question. Commented Apr 3, 2015 at 22:58
  • In Oracle you would use ‘ON COMMIT FAST REFRESH’ along with ‘CREATE MATERIALIZED VIEW LOG’. I wonder if there is something similar for PG. Commented Apr 20, 2021 at 22:34
  • @vaughan there is an extension that try to mimic that function on oracle see this stackoverflow.com/questions/29437650/… Commented Jan 15, 2023 at 6:35

2 Answers 2

389

I'll need to invoke REFRESH MATERIALIZED VIEW on each change to the tables involved, right?

Yes, PostgreSQL by itself will never call it automatically, you need to do it some way.

How should I go about doing this?

Many ways to achieve this. Before giving some examples, keep in mind that REFRESH MATERIALIZED VIEW command does block the view in AccessExclusive mode, so while it is working, you can't even do SELECT on the table.

Although, if you are in version 9.4 or newer, you can give it the CONCURRENTLY option:

REFRESH MATERIALIZED VIEW CONCURRENTLY my_mv; 

This will acquire an ExclusiveLock, and will not block SELECT queries, but may have a bigger overhead (depends on the amount of data changed, if few rows have changed, then it might be faster). Although you still can't run two REFRESH commands concurrently.

Refresh manually

It is an option to consider. Specially in cases of data loading or batch updates (e.g. a system that only loads tons of information/data after long periods of time) it is common to have operations at end to modify or process the data, so you can simple include a REFRESH operation in the end of it.

Scheduling the REFRESH operation

The first and widely used option is to use some scheduling system to invoke the refresh, for instance, you could configure the like in a cron job:

*/30 * * * * psql -d your_database -c "REFRESH MATERIALIZED VIEW CONCURRENTLY my_mv" 

And then your materialized view will be refreshed at each 30 minutes.

Considerations

This option is really good, specially with CONCURRENTLY option, but only if you can accept the data not being 100% up to date all the time. Keep in mind, that even with or without CONCURRENTLY, the REFRESH command does need to run the entire query, so you have to take the time needed to run the inner query before considering the time to schedule the REFRESH.

Refreshing with a trigger

Another option is to call the REFRESH MATERIALIZED VIEW in a trigger function, like this:

CREATE OR REPLACE FUNCTION tg_refresh_my_mv() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN REFRESH MATERIALIZED VIEW CONCURRENTLY my_mv; RETURN NULL; END; $$; 

Then, in any table that involves changes on the view, you do:

CREATE TRIGGER tg_refresh_my_mv AFTER INSERT OR UPDATE OR DELETE ON table_name FOR EACH STATEMENT EXECUTE PROCEDURE tg_refresh_my_mv(); 

Considerations

It has some critical pitfalls for performance and concurrency:

  1. Any INSERT/UPDATE/DELETE operation will have to execute the query (which is possible slow if you are considering MV);
  2. Even with CONCURRENTLY, one REFRESH still blocks another one, so any INSERT/UPDATE/DELETE on the involved tables will be serialized.

The only situation I can think that as a good idea is if the changes are really rare.

Refresh using LISTEN/NOTIFY

The problem with the previous option is that it is synchronous and impose a big overhead at each operation. To ameliorate that, you can use a trigger like before, but that only calls a NOTIFY operation:

CREATE OR REPLACE FUNCTION tg_refresh_my_mv() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN NOTIFY refresh_mv, 'my_mv'; RETURN NULL; END; $$; 

So then you can build an application that keep connected and uses LISTEN operation to identify the need to call REFRESH. One nice project that you can use to test this is pgsidekick, with this project you can use shell script to do LISTEN, so you can schedule the REFRESH as:

pglisten --listen=refresh_mv --print0 | xargs -0 -n1 -I? psql -d your_database -c "REFRESH MATERIALIZED VIEW CONCURRENTLY ?;" 

Or use pglater (also inside pgsidekick) to make sure you don't call REFRESH very often. For example, you can use the following trigger to make it REFRESH, but within 1 minute (60 seconds):

CREATE OR REPLACE FUNCTION tg_refresh_my_mv() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN NOTIFY refresh_mv, '60 REFRESH MATERIALIZED VIEW CONCURRENLTY my_mv'; RETURN NULL; END; $$; 

So it will not call REFRESH in less the 60 seconds apart, and also if you NOTIFY many times in less than 60 seconds, the REFRESH will be triggered only once.

Considerations

As the cron option, this one also is good only if you can bare with a little stale data, but this has the advantage that the REFRESH is called only when really needed, so you have less overhead, and also the data is updated more closer to when needed.

OBS: I haven't really tried the codes and examples yet, so if someone finds a mistake, typo or tries it and works (or not), please let me know.

Sign up to request clarification or add additional context in comments.

11 Comments

Epic answer! I see that updating on each operation won't be an option for me. I was hoping to use the materialized view as a fancy complicated index. But it doesn't go along with other expectations a programmer has about db usage. So I guess this is why this "auto update" functionality hasn't been built into the feature.
"if few rows have changed, then it might be faster" - no it won't. Postgres always rebuilds the entire MVIEW there is no incremental update, so the number changed rows does not affect the refresh time.
@a_horse_with_no_name, I think you are mistaken, if you use CONCURRENTLY it DELETE old/changed rows and INSERT new/changed rows, but it doesn't touch non-changed rows. See refresh_by_match_merge function in PG's source code. In any case it does execute the entire underlying query.
@MatheusOl a_horse_with_no_name is correct. The entire materialized view is recalculated when using concurrently. Then postgres will use the unique index on the materialized view to find which rows have changed and only update those rows from the recalculated values. This prevents blocking of the entire view, but it does not speed up calculation time.
@Wildcard Materialized views do allow indices, and CONCURRENTLY actually requires a UNIQUE index: This option is only allowed if there is at least one UNIQUE index on the materialized view which uses only column names and includes all rows; that is, it must not index on any expressions nor include a WHERE clause. postgresql.org/docs/current/static/…
|
35

Now there is a PostgreSQL extension to keep materialized views updated: pg_ivm.

It only computes and applies the incremental changes, rather than recomputing the contents fully as REFRESH MATERIALIZED VIEW does. It has 2 approaches, IMMEDIATE and DEFERRED:

  • For IMMEDIATE, the views are updated in the same transaction that its base table is modified.

  • For DEFERRED, the views are updated after the transaction is committed.

Version 1.0 has been released on 2022-04-28.

2 Comments

They dont support the foreign table based views, any way for that ?
ask a new questions or open an issue at their repo, there you could explain ur use case more throughly

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.