Bug/Decisions are not deduplicated when sent to a bouncer #1158
Description
When a bouncer starts, it requests all the active decisions, but we do not deduplicate anything, which means that if an IP has multiple decisions, the bouncer will try to add the same decision multiple times.
In some cases, this may lead to a missing remediation for the IP, based on the order in which the bouncer processes the decisions, for example:
- 1.1.1.1 has 2 decisions: 1 with 60s left and another one with 1h left, this may lead to only the decision with 60s left being added, which virtually ignores the ban.
We should deduplicate all decisions when a bouncer requests them, keeping only the one with the highest remaining duration.
Logs for reference:
time="12-01-2022 10:32:51" level=debug msg="deleted '49.88.112.115'" time="12-01-2022 10:32:51" level=debug msg="ipset del ban for [49.88.112.115]" time="12-01-2022 10:32:51" level=debug msg="deleted '49.88.112.115'" time="12-01-2022 10:32:51" level=debug msg="ipset del ban for [49.88.112.115]" time="12-01-2022 10:32:51" level=debug msg="deleted '49.88.112.115'" time="12-01-2022 10:32:51" level=debug msg="ipset del ban for [49.88.112.115]" time="12-01-2022 10:32:51" level=debug msg="deleted '49.88.112.115'" time="12-01-2022 10:32:51" level=debug msg="ipset del ban for [49.88.112.115]" time="12-01-2022 10:32:51" level=debug msg="deleted '49.88.112.115'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 222 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 222" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '3m42.989639349s'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 9329 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 9329" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '2h35m29.098288008s'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 13483 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 13483" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '3h44m43.033365753s'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 1754 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 1754" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '29m14.900240076s'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 3622 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 3622" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '1h0m22.02257703s'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 3622 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 3622" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '1h0m22.021953922s'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 8710 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 8710" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '2h25m10.210238383s'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 2394 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 2394" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '39m54.666846316s'" time="12-01-2022 10:32:52" level=debug msg="ipset add ban [49.88.112.115] (for 498 seconds)" time="12-01-2022 10:32:52" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 498" time="12-01-2022 10:32:52" level=debug msg="Adding '49.88.112.115' for '8m18.440998108s'" time="12-01-2022 10:32:53" level=debug msg="ipset add ban [49.88.112.115] (for 12075 seconds)" time="12-01-2022 10:32:53" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 12075" time="12-01-2022 10:32:53" level=debug msg="Adding '49.88.112.115' for '3h21m15.363749544s'" time="12-01-2022 10:32:53" level=debug msg="ipset add ban [49.88.112.115] (for 13578 seconds)" time="12-01-2022 10:32:53" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 13578" time="12-01-2022 10:32:53" level=debug msg="Adding '49.88.112.115' for '3h46m18.592292546s'" time="12-01-2022 10:32:53" level=debug msg="ipset add ban [49.88.112.115] (for 39 seconds)" time="12-01-2022 10:32:53" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec-blacklists 49.88.112.115 timeout 39" time="12-01-2022 10:32:53" level=debug msg="Adding '49.88.112.115' for '39.887366474s'" time="12-01-2022 10:32:54" level=debug msg="ipset add ban [49.88.112.115] (for 12701 seconds)" time="12-01-2022 10:32:54" level=debug msg="ipset add : /sbin/ipset -exist add crowdsec