Most efficient way of bulk deleting rows from postgres

Question

I'm wondering what the most efficient way would be to delete large numbers of rows from PostgreSQL, this process would be part of a recurring task every day to bulk import data (a delta of insertions + deletions) into a table. There could be thousands, potentially millions of rows to delete.

I have a file of primary keys, one per line. The two options I was thinking of were along the lines of the below, but I don't know/understand enough of the internals of PostgreSQL to make an informed decision which would be best.

Execute a DELETE query for each row in the file, with a simple WHERE on primary key (or group the deletes in batches of n using an IN() clause)
Import the primary keys into a temporary table using the COPY command and then deleting from the main table using a join

Any suggestions will be much appreciated!

The same question has been answered in more detail here: stackoverflow.com/a/8290958 — Simon
– Simon, Commented Jun 22, 2015 at 14:21
In case anyone else doesn't know (like I did until recently), disabling or deleting triggers (which PostgreSQL uses internally for constraints too) can improve the performance of bulk deletions by several orders of magnitude. Given that the original question mentions recurring (daily) bulk imports, I imagine that was being done for the relevant tables already. — Kenny Evitt
– Kenny Evitt, Commented Dec 23, 2020 at 20:38

Glorfindel · Accepted Answer · 2022-05-26 20:16:24Z

Your second option is far cleaner and will perform well enough to make that worth it. Your alternative is to build gigantic queries which will be quite a pain to plan and execute. In general you are going to be better off letting PostgreSQL do the work here. In general, I have found updates on tens of thousands of rows in the manner you are describing to perform adequately, but there is one important thing to avoid doing.

The way to do it is to use a select and a join in your delete.

DELETE FROM foo WHERE id IN (select id from rows_to_delete);

Under no circumstances you should do as follows with a large table:

DELETE FROM foo WHERE id NOT IN (select id from rows_to_keep);

This will usually cause a nested loop anti-join which will make performance rather problematic. If you end up having to go that route, do this instead:

DELETE FROM foo WHERE id IN (select id from foo f LEFT JOIN rows_to_keep d on f.id = d.id WHERE d.id IS NULL);

PostgreSQL is usually pretty good at avoiding bad plans but there are still cases involving outer joins which can make a big difference between good and bad plans.

This is wandering a little more far afield, but I figure it is worth mentioning because of how easy it is to go from the IN to NOT IN and watch query performance tank.

That helped a lot, thanks! However I found that using "combining queries" is more efficient in this particular case. E.g. IN ( select id from foo except select id from rows_to_keep ) See postgresql.org/docs/9.4/static/queries-union.html — Ufos
– Ufos, Commented Sep 12, 2017 at 13:31

Niro · Accepted Answer · 2018-10-29 23:56:47Z

I came across this question because I had a similar problem. I am cleaning up a database that has 300M+ rows, the final database will only have around 30% of the original data. If you are facing a similar scenario, it is actually easier to insert in a new table and re-index instead of deleting.

Do something like

CREATE temp_foo as SELECT * FROM foo WHERE 1=2; INSERT INTO temp_foo (SELECT * FROM foo where foo.id IN (SELECT bar.id FROM BAR);

With proper indexing on foo and bar, you can avoid Seq scans.

Then you would have to re-index and rename the table.

Stack Exchange Network

Most efficient way of bulk deleting rows from postgres

2 Answers 2

Hot Network Questions

Most efficient way of bulk deleting rows from postgres

2 Answers 2

Related

Hot Network Questions