I'm going to throw 410 Gone into the mix. RFC 7231, section 6.5.4 410 Gone says

The 410 (Gone) status code indicates that access to the target resource is no longer available at the origin server and that this condition is likely to be permanent.

So 204 for the first DELETE and 410 for the subsequent DELETEs. It's nice and consistent too, as a GET, PUT, etc will also return 410.

However, it would require the server to keep track of what has been deleted and the spec does go on to say

If the origin server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) ought to be used instead.

So, if it's valuable to keep track of deleted resources (e.g. if you need to prevent them from being re-created), 410, otherwise, 404 as recommended in this answer, or in other words

I'm going to throw 410 Gone into the mix. RFC 7231, section 6.5.4 410 Gone says

The 410 (Gone) status code indicates that access to the target resource is no longer available at the origin server and that this condition is likely to be permanent.

So 204 for the first DELETE and 410 for the subsequent DELETEs. It's nice and consistent too, as a GET, PUT, etc will also return 410.

However, it would require the server to keep track of what has been deleted and the spec does go on to say

If the origin server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) ought to be used instead.

So, if it's valuable to keep track of deleted resources (e.g. if you need to prevent them from being re-created), 410, otherwise, 404 as recommended in this answer, or in other words  "it depends".