79

What is the best way to remove items from a slice while ranging over it?

For example:

type MultiDataPoint []*DataPoint func (m MultiDataPoint) Json() ([]byte, error) { for i, d := range m { err := d.clean() if ( err != nil ) { //Remove the DP from m } } return json.Marshal(m) } 
4
  • @YasirG.: That's for deleting from a map. The OP wants to delete from a slice. Commented Dec 12, 2013 at 14:40
  • ah true sorry for the wrong answer been awake for 25 hours now :S Commented Dec 12, 2013 at 14:41
  • Go desperately needs a descending range construct. Not having it causes a good deal of extra work. Commented Mar 30, 2014 at 11:53
  • How would that change anything? @Vector Commented Nov 26, 2016 at 2:03

12 Answers 12

107

As you have mentioned elsewhere, you can allocate new memory block and copy only valid elements to it. However, if you want to avoid the allocation, you can rewrite your slice in-place:

i := 0 // output index for _, x := range s { if isValid(x) { // copy and increment index s[i] = x i++ } } // Prevent memory leak by erasing truncated values // (not needed if values don't contain pointers, directly or indirectly) for j := i; j < len(s); j++ { s[j] = nil } s = s[:i] 

Full example: http://play.golang.org/p/FNDFswPeDJ

Note this will leave old values after index i in the underlying array, so this will leak memory until the slice itself is garbage collected, if values are or contain pointers. You can solve this by setting all values to nil or the zero value from i until the end of the slice before truncating it.

Sign up to request clarification or add additional context in comments.

11 Comments

No need for semicolons in Go
@Ectomorph: fixed; back then I was a C++ programmer – fortunately these days are gone and took semicolons with them.
Should be accepted answer. Much better than copying array on each removal.
@EugeneBeresovsky I don't think my wording was very accurate here and my only intention was to warn people we're overwriting the original array, so there could be side-effects. Will edit.
Specifically if you have a slice of pointers, you can potentially leak those items at the end of the array, because the backing memory containing the original pointer is still alive to the GC. If you truncate a slice of pointers, always set the truncated pointers to nil. utcc.utoronto.ca/~cks/space/blog/programming/GoSlicesMemoryLeak
|
29

I know its answered long time ago but i use something like this in other languages, but i don't know if it is the golang way.

Just iterate from back to front so you don't have to worry about indexes that are deleted. I am using the same example as Adam.

m = []int{3, 7, 2, 9, 4, 5} for i := len(m)-1; i >= 0; i-- { if m[i] < 5 { m = append(m[:i], m[i+1:]...) } } 

2 Comments

simplest answer, should be the top one
Not efficient though, if removing more than 1 element. It copies array for every removed element. Removing 100 from 10'000 elements copies 500'000 (on average) elements. Where @tomasz solution copies 10'000. 50x faster.
17

There might be better ways, but here's an example that deletes the even values from a slice:

m := []int{1,2,3,4,5,6} deleted := 0 for i := range m { j := i - deleted if (m[j] & 1) == 0 { m = m[:j+copy(m[j:], m[j+1:])] deleted++ } } 

Note that I don't get the element using the i, d := range m syntax, since d would end up getting set to the wrong elements once you start deleting from the slice.

5 Comments

Although I just realized it is a lot straight forward (maybe less efficient?) to just create a second MultiDataPoint object, append valid stuff to it, and Marshal that. Don't know how I didn't see that in the first place :-/
I guess the efficiency depends on the expected ratio of deletions to the total number of elements. The code will most likely be more straight-forward though.
for i, _ := should just be for i :=
@newacct: Oops. Edited.
you should use the form for i := 0; i < len(m); i++ { for the loop, the range might give you an index out of range error.
15

Here is a more idiomatic Go way to remove elements from slices.

temp := s[:0] for _, x := range s { if isValid(x) { temp = append(temp, x) } } s = temp 

Playground link: https://play.golang.org/p/OH5Ymsat7s9

Note: The example and playground links are based upon @tomasz's answer https://stackoverflow.com/a/20551116/12003457

6 Comments

Nice, simple solution.
For initialisation of temp, why do this temp := s[:0] instead of temp := []int{}. What's the difference?
@Daryll s[:0] zeros the length but keeps the same capacity of s, whereas []int{} zeros both. It also avoids re-declaring the type.
Using s[:0] would reuse the same underlying array, while using []int{} would create a new array.
If a different goroutine updates the original s during this loop, this code will wipe the new data and take whatever is in temp. You need to put this whole block in a mutex.Lock() so it is safe for use
|
14

Now that Go has generics, you may use slices.DeleteFunc like this:

m = slices.DeleteFunc(m, func(dp *DataPoint) bool { return dp.clean() != nil }) 

"DeleteFunc" is a fancy name for "Filter" that emphasizes "return true for elements you want to remove".

See full code in the Playground.

Comments

7

One other option is to use a normal for loop using the length of the slice and subtract 1 from the index each time a value is removed. See the following example:

m := []int{3, 7, 2, 9, 4, 5} for i := 0; i < len(m); i++ { if m[i] < 5 { m = append(m[:i], m[i+1:]...) i-- // -1 as the slice just got shorter } } 

I don't know if len() uses enough resources to make any difference but you could also run it just once and subtract from the length value too:

m := []int{3, 7, 2, 9, 4, 5} for i, s := 0, len(m); i < s; i++ { if m[i] < 5 { m = append(m[:i], m[i+1:]...) s-- i-- } } 

Comments

4

Something like:

m = append(m[:i], m[i+1:]...) 

4 Comments

I am trying to use this method on a slice of strings - I get this error: "cannot use m[i + 1:] (type []string) as type string in append" It seems to be expecting a string not a slice of strings as the second parameter. Any ideas?
Nvmd - I just realized that "..." was actually apart of the code.
If you use this while iterating m, you will get runtime error: slice bounds out of range. That's what this question is about. You can use m = append(m[:i], m[i+1:]...) to delete an item from a slice but not while you are iterating through it with for i, element := range m.
0

You don't even need to count backwards but you do need to check that you're at the end of the array where the suggested append() will fail. Here's an example of removing duplicate positive integers from a sorted list:

// Remove repeating numbers numbers := []int{1, 2, 3, 3, 4, 5, 5} log.Println(numbers) for i, numbersCount, prevNum := 0, len(numbers), -1; i < numbersCount; numbersCount = len(numbers) { if numbers[i] == prevNum { if i == numbersCount-1 { numbers = numbers[:i] } else { numbers = append(numbers[:i], numbers[i+1:]...) } continue } prevNum = numbers[i] i++ } log.Println(numbers) 

Playground: https://play.golang.org/p/v93MgtCQsaN

Comments

0

I just implement a method which removes all nil elements in slice.

And I used it to solve a leetcode problems, it works perfectly.

/** * Definition for singly-linked list. * type ListNode struct { * Val int * Next *ListNode * } */ func removeNil(lists *[]*ListNode) { for i := 0; i < len(*lists); i++ { if (*lists)[i] == nil { *lists = append((*lists)[:i], (*lists)[i+1:]...) i-- } } } 

Comments

0

You can avoid memory leaks, as suggested in @tomasz's answer, controlling the capacity of the underlying array with a full slice expression. Look at the following function that remove duplicates from a slice of integers:

package main import "fmt" func removeDuplicates(a []int) []int { for i, j := 0, 1; i < len(a) && j < len(a); i, j = i+1, j+1 { if a[i] == a[j] { copy(a[j:], a[j+1:]) // resize the capacity of the underlying array using the "full slice expression" // a[low : high : max] a = a[: len(a)-1 : len(a)-1] i-- j-- } } return a } func main() { a := []int{2, 3, 3, 3, 6, 9, 9} fmt.Println(a) a = removeDuplicates(a) fmt.Println(a) } // [2 3 3 3 6 9 9] // [2 3 6 9] 

Comments

0

For reasons @tomasz has explained, there are issues with removing in place. That's why it is practice in golang not to do that, but to reconstruct the slice. So several answers go beyond the answer of @tomasz.

If elements should be unique, it's practice to use the keys of a map for this. I like to contribute an example of deletion by use of a map.

What's nice, the boolean values are available for a second purpose. In this example I calculate Set a minus Set b. As Golang doesn't have a real set, I make sure the output is unique. I use the boolean values as well for the algorithm.

The map gets close to O(n). I don't know the implementation. append() should be O(n). So the runtime is similar fast as deletion in place. Real deletion in place would cause a shifting of the upper end to clean up. If not done in batch, the runtime should be worse.

In this special case, I also use the map as a register, to avoid a nested loop over Set a and Set b to keep the runtime close to O(n).

type Set []int func differenceOfSets(a, b Set) (difference Set) { m := map[int]bool{} for _, element := range a { m[element] = true } for _, element := range b { if _, registered := m[element]; registered { m[element] = false } } for element, present := range m { if present { difference = append(difference, element) } } return difference } 

Comments

-2

Try Sort and Binary search.

Example:

package main import ( "fmt" "sort" ) func main() { // Our slice. s := []int{3, 7, 2, 9, 4, 5} // 1. Iterate over it. for i, v := range s { func(i, v int) {}(i, v) } // 2. Sort it. (by whatever condition of yours) sort.Slice(s, func(i, j int) bool { return s[i] < s[j] }) // 3. Cut it only once. i := sort.Search(len(s), func(i int) bool { return s[i] >= 5 }) s = s[i:] // That's it! fmt.Println(s) // [5 7 9] } 

https://play.golang.org/p/LnF6o0yMJGT

2 Comments

This requires more code, is slower (sorting is generally not faster than O(n log n), while iteratively removing from the slice would be O(n)) and doesn't properly demonstrate how you'd use it to remove elements from a slice when you have a filter function and not a comparison function.
I believe this is more memory efficient and could be even faster as it doesn’t copy any element from a slice to another slice. Changing the whole structure of where data is stored for every element to be removed is dangerous and may cost a lot more than just iterating over a slice.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.