If you come from a language such as Python or Java, you might expect to find a standardized way to remove items from a list object in Go. However, as of Go 1.12, there is not a built-in way to perform this action. You can do it with relative ease once you wrap your head around it, though.
Personally, it took me a bit to wrap my head around it, which is why I'm writing this — in hopes of aiding someone else down the line.
I was working on a problem that required me to take a list of items and filter them by removing any items that matched a particular regex, then return the filtered list. Several hours of debugging and brainstorming later, I finally figured out how to do it right.
In my particular case, I did not care about the order of the elements in the slice so I could rearrange them any which way.
First, we start with the function declaration:
func filterExcluded(tl *[]task, excluded []string) []task {
filtered := *tl
lastIndex := (len(filtered) - 1)
The function accepts a pointer to a slice of task-typed objects, along with a slice of strings that will be interpreted as regex matchers, it then returns a slice of task-typed objects.
The filtered variable is a copy of all the items in the tl argument, which we will whittle down later. The lastIndex variable will be updated repeatedly later on and we'll use it to return a subset of data from the filtered slice.
Next, we loop over the regex matchers and make them *regexp.Regexp objects. Nothing fancy.
We'll also created a matched boolean that we'll use to log an error if it matches nothing, so that end users can make sure the regex is actually right and isn't just wasting compute time.
for _, regexString := range excluded {
matched := false
regex, err := regexp.Compile(regexString)
if err != nil {
logrus.Error(err)
continue
}
Now we're gonna go 0 to 100 real quick, but don't panic — I've commented everything in-line so you can see what's happening:
// Do not increment index on each loop.
// Handle incrementing inside of the loop.
for index := 0; index <= lastIndex; {
taskName := &filtered[index].Name
if regex.Match([]byte(*taskName)) {
logrus.WithFields(logrus.Fields{
"task_name": *taskName,
"regex": regexString,
}).Debug("Excluding task")
// Hooray! The provided regex matched something and wasn't a waste of time.
matched = true
// Replace the value at the current index with the value at
// the last index of the slice.
filtered[index] = filtered[lastIndex]
// Set the value of the last index of the slice
// to the nil value of `task`. The value that was previously
// there is now at filtered[index], so we did not lose it.
// We will just NOT increment `index` so that the new value gets checked, too.
filtered[lastIndex] = task{}
// Set the `filtered` slice to be everything up to
// the last index, which we just set to a nil value.
filtered = filtered[:lastIndex]
// The last index will now be one less than before.
// This is the same as if we just did
// lastIndex = len(filtered)
// everytime, except this should be slightly more performant.
lastIndex--
} else {
// If no match was found, increment the index
// so that we check the next value in the `filtered` slice
index++
logrus.WithFields(logrus.Fields{
"task_name": *taskName,
"regex": regexString,
}).Debug("Did not match regex")
}
}
// Log an error if the regex didn't match any tasks.
// This should warn users if they're providing a useless regex.
if !matched {
logrus.Error("No task found to exclude matching: ", regexString)
}
}
Now you just need to return the filtered list and you're golden.
return filtered
Ta-Da! We just filtered a slice while incurring only a slight amount of pain.
TL;DR — Loop over slice; only increment index on non-matches; on match, set slice[i] = last item in slice; set last item in slice to nil (helps w/ GC); set slice = slice[:lastItem]; profit.
Further Reading
- SliceTricks (Golang Github wiki)
Bonus
Putting it all together, it looks like this:
func filterExcluded(tl *[]task, excluded []string) []task {
filtered := *tl
lastIndex := (len(filtered) - 1)
for _, regexString := range excluded {
matched := false
regex, err := regexp.Compile(regexString)
if err != nil {
logrus.Error(err)
continue
}
// Do not increment index on each loop.
// Handle incrementing inside of the loop.
for index := 0; index <= lastIndex; {
taskName := &filtered[index].Name
if regex.Match([]byte(*taskName)) {
logrus.WithFields(logrus.Fields{
"task_name": *taskName,
"regex": regexString,
}).Debug("Excluding task")
// Hooray! The provided regex matched something and wasn't a waste of time.
matched = true
// Replace the value at the current index with the value at
// the last index of the slice.
filtered[index] = filtered[lastIndex]
// Set the value of the last index of the slice
// to the nil value of `task`. The value that was previously
// there is now at filtered[index], so we did not lose it.
// We will just NOT increment `index` so that the
// new value will get checked, too.
filtered[lastIndex] = task{}
// Set the `filtered` slice to be everything up to
// the last index, which we just set to a nil value.
filtered = filtered[:lastIndex]
// The last index will now be one less than before.
// This is the same as if we just did
// lastIndex = len(filtered)
// everytime, except this should be slightly more performant.
lastIndex--
} else {
// If no match was found, increment the index
// so that we check the next value in the `filtered` slice
index++
logrus.WithFields(logrus.Fields{
"task_name": *taskName,
"regex": regexString,
}).Debug("Did not match regex")
}
}
// Log an error if the regex didn't match any tasks.
// This should warn users if they're providing a useless regex.
if !matched {
logrus.Error("No task found to exclude matching: ", regexString)
}
}
return filtered
}