93

I have the following table of counters:

CREATE TABLE cache ( key text PRIMARY KEY, generation int ); 

I would like to increment one of the counters, or set it to zero if the corresponding row doesn't exist yet. Is there a way to do this without concurrency issues in standard SQL? The operation is sometimes part of a transaction, sometimes separate.

The SQL must run unmodified on SQLite, PostgreSQL and MySQL, if possible.

A search yielded several ideas which either suffer from concurrency issues, or are specific to a database:

  • Try to INSERT a new row, and UPDATE if there was an error. Unfortunately, the error on INSERT aborts the current transaction.

  • UPDATE the row, and if no rows were modified, INSERT a new row.

  • MySQL has an ON DUPLICATE KEY UPDATE clause.

EDIT: Thanks for all the great replies. It looks like Paul is right, and there's not a single, portable way of doing this. That's quite surprising to me, as it sounds like a very basic operation.

3

10 Answers 10

153

MySQL (and subsequently SQLite) also support the REPLACE INTO syntax:

REPLACE INTO my_table (pk_id, col1) VALUES (5, '123'); 

This automatically identifies the primary key and finds a matching row to update, inserting a new one if none is found. But as pointed out in the comments below, you need to make sure you supply values for all columns because even if the row already exists it will replace it with the values you supply (don't assume the new row will have the same primary key value as the old row either).

Documentation: https://dev.mysql.com/doc/refman/8.0/en/replace.html

Sign up to request clarification or add additional context in comments.

4 Comments

Actually, to be specific, MySQL's REPLACE always does an insert, but it will delete the row if it already exists, first. dev.mysql.com/doc/refman/4.1/en/replace.html
It is important to understand that it is an insert+delete and never and update. The consequence of this, is that you will always want to make sure when you do a replace, you should always include data for all the fields.
@Zoredache Technically, it's a 'delete, then insert', as a(n) 'insert + delete' is effectively the same as a delete but that's splitting hairs.
@Agihammerthief there is a very real difference namely that the newly inserted row will NOT have the same primary key as the row that has been deleted. With ON DUPLICATE it will be the same primary key (unless you specifically change it).
33

SQLite supports replacing a row if it already exists:

INSERT OR REPLACE INTO [...blah...] 

You can shorten this to

REPLACE INTO [...blah...] 

This shortcut was added to be compatible with the MySQL REPLACE INTO expression.

1 Comment

It requires that you defined a PRAMARY KEY in your values.
30

I would do something like the following:

INSERT INTO cache VALUES (key, generation) ON DUPLICATE KEY UPDATE (key = key, generation = generation + 1); 

Setting the generation value to 0 in code or in the sql but the using the ON DUP... to increment the value. I think that's the syntax anyway.

1 Comment

This answer should honestly be the selected answer. It's a non-destructive edit if you aren't submitting all fields to an entry.
11

the ON DUPLICATE KEY UPDATE clause is the best solution because: REPLACE does a DELETE followed by an INSERT so for an ever so slight period the record is removed creating the ever so slight possibility that a query could come back having skipped that if the page was viewed during the REPLACE query.

I prefer INSERT ... ON DUPLICATE UPDATE ... for that reason.

jmoz's solution is the best: though I prefer the SET syntax to the parentheses

INSERT INTO cache SET key = 'key', generation = 'generation' ON DUPLICATE KEY UPDATE key = 'key', generation = (generation + 1) ; 

1 Comment

REPLACE is atomic, so there's no period where the row doesn't exist.
8

I don't know that you are going to find a platform-neutral solution.

This is commonly called an "UPSERT".

See some related discussions:

1 Comment

Is that solution really applicable? and portable?
5

In PostgreSQL there is no merge command, and actually writing it is not trivial - there are actually strange edge cases that make the task "interesting".

The best (as in: working in the most possible conditions) approach, is to use function - such as one shown in manual (merge_db).

If you don't want to use function, you can usually get away with:

updated = db.execute(UPDATE ... RETURNING 1) if (!updated) db.execute(INSERT...) 

Just remember that it is not fault proof and it will fail eventually.

Comments

4

Standard SQL provides the MERGE statement for this task. Not all DBMS support the MERGE statement.

Comments

0

If you don't have a common way to atomically update or insert (e.g., via a transaction) then you can fallback to another locking scheme. A 0-byte file, system mutex, named pipe, etc...

Comments

0

Could you use an insert trigger? If it fails, do an update.

1 Comment

Trigger (at least in PostgreSQL) is running when the command worked. i.e. you can't have trigger that runs when base command failed.
0

If you're OK with using a library that writes the SQL for you, then you can use Upsert (currently Ruby and Python only):

Pet.upsert({:name => 'Jerry'}, :breed => 'beagle') Pet.upsert({:name => 'Jerry'}, :color => 'brown') 

That works across MySQL, Postgres, and SQLite3.

It writes a stored procedure or user-defined function (UDF) in MySQL and Postgres. It uses INSERT OR REPLACE in SQLite3.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.