2

I am trying to figure out how it was compromised. They installed IptabLes and IptabLex in /boot.
They also added /etc/init.d/IptabLes and /etc/init.d/IptabLex which simply call the respective /boot files. It seems this attack uses a lot of bandwidth (probably a DDoS); I noticed it immediately.

The server is running CentOS 6.5 with all the latest updates.
It runs logstash, redis, ElasticSearch, and Cherokee webserver serving Kibana.
I am thinking it must either be ElasticSearch or Cherokee web-server.

ElasticSearch port (9200) was open to the world, because Kibana requires it to view the nice graphs. Redis ports (6379) were restricted to only 5 known hosts via iptables.
Cherokee webserver runs on port (8080) not default of 80 and was open to the world.

SSH does not seem to be compromised. The server uses keys and no password authentication
is allowed. We run SSH on port 2020, which is listed as (xinupageserver) in iptables.

Here are the iptables rules. Notice redis is restricted to web hosts,
but http via Cherokee (webcache) and ElasticSearch (wap-wsp) are open.

➜ ~ iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere REJECT all -- anywhere loopback/8 reject-with icmp-port-unreachable ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:xinupageserver ACCEPT tcp -- web1.mydomain.com anywhere tcp dpt:6379 ACCEPT tcp -- web2.mydomain.com anywhere tcp dpt:6379 ACCEPT tcp -- web3.mydomain.com anywhere tcp dpt:6379 ACCEPT tcp -- web4.mydomain.com anywhere tcp dpt:6379 ACCEPT tcp -- web5.mydomain.com anywhere tcp dpt:6379 ACCEPT tcp -- anywhere anywhere tcp dpt:wap-wsp ACCEPT tcp -- anywhere anywhere tcp dpt:webcache ACCEPT icmp -- anywhere anywhere icmp echo-request REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere

Thanks so much for the help.

Improve this question
7
  • What's in your log files? Commented May 28, 2014 at 1:33
  • Nothing much, no traces in /var/log/secure. Commented May 28, 2014 at 3:11
  • Which version of elasticsearch are you running? Commented May 28, 2014 at 10:29
  • And which version of java are you running your elasticsearch on? Commented May 28, 2014 at 10:35
  • "ElasticSearch port (9200) was open to the world" -- you should never, ever leave Elasticsearch open to the outside world, much in the same spirit as you don't leave Postgres open. Install Nginx/Apache/etc as a proxy with HTTP auth for Kibana. Commented May 29, 2014 at 7:55

4 Answers 4

Reset to default
2

You should never leave Elasticsearch open to the world, in the same way as you wouldn't leave any database server open to the world. Always put a proxy in front of it and use (at least) HTTP auth. Preferably, your proxy should limit what can be done from outside, eg only allowing GET requests, otherwise any user could delete all your data or shutdown your servers.

From version 1.2.0 onwards, dynamic scripting (passing a script as part of eg a search request) has been disabled by default. On older versions it is enabled by default, meaning that any outside user could do anything on your server that the Elasticsearch user can do.

If you're running an older version, you want to add this to your config/elasticsearch.yaml:

script.disable_dynamic: true

See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_enabling_dynamic_scripting for more, and http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/url-access-control.html#url-access-control

Improve this answer
1

@Justin I have the same issue. I have monitoring server in DigitalOcean that uses ElasticSearch too.

I think the problem not in ssh because my server uses keys and no password authentication and disabled login for root users.

Some DO users got the same problem with ElasticSearch: https://www.digitalocean.com/community/questions/my-droplet-is-locked-by-support-staff-because-because-of-an-outgoing-flood-or-ddos-what-do-i-do

EIDT: Here is solution how to clean it http://www.ebel-computing.de/JSPWiki/Wiki.jsp?page=VServer%20Trojan but it does not fix the problem in a future

Improve this answer
0
0

I have had the same issue (IpTablex trojan) on a Debian 7 box with only ssh open via DSL connection. It only ran a postgresql database server which was only listening to internal connections. Highly probable cause was a weak root password, the ssh root login was open. The only other cause could be a compromised windows machine used for logging in to the machine with ssh, but we haven't seen any other machine infected where the windows computers have been used for.

The only good solution is to figure out the cause of infection, backup your important data and reinstall the entire system and fixing the cause of infection. Scripts removing only the /boot/IpTablex and related files is not a solution as it doesn't guarantee there is no other rootkit left somewhere else on the system, and it doesn't fix cause of infection.

Improve this answer
0

Well, immediately, I see a huge flaw in your security posture:

➜ ~ iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere

Your first rule on INPUT is accept any inbound connection on any port... so your reject all at the end (and in fact none of your other INPUT rules) matter....

So any of your running/listening services on the server could have been targeted, just FYI.

[EDIT] See proof below

Script started on Sun 27 Jul 2014 11:32:04 AM EDT ]0;nick@fakeit:/tmp[?1034h[nick@fakeit tmp]$ sudo iptables -L [sudo] password for nick: Chain INPUT (policy DROP) target prot opt source destination ACCEPT all -- anywhere anywhere DROP all -- anywhere anywhere Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ ssh [email protected] The authenticity of host '172.16.16.110 (172.16.16.110)' can't be established. RSA key fingerprint is fc:86:c7:f6:1a:31:5b:9b:4d:a7:85:ca:7c:f2:29:ec. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.16.16.110' (RSA) to the list of known hosts. [email protected]'s password: Last login: Sun Jul 27 11:30:55 2014 from localhost ]0;iptables_demo@fakeit:~[?1034h[iptables_demo@fakeit ~]$ echo "I got in with a default INPUT policy of DROP, an explict DROP ALL at the end, but an ACCEPT ALL rule at the beginning" I got in with a default INPUT policy of DROP, an explict DROP ALL at the end, but an ACCEPT ALL rule at the beginning ]0;iptables_demo@fakeit:~[iptables_demo@fakeit ~]$ exit logout Connection to 172.16.16.110 closed. ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ sudo iptables -L Chain INPUT (policy DROP) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere tcp dpt:http DROP all -- anywhere anywhere Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ ssh [email protected] [email protected]'s password: Last login: Sun Jul 27 11:32:52 2014 from 172.16.16.110 ]0;iptables_demo@fakeit:~[?1034h[iptables_demo@fakeit ~]$ echo "I got in with an ACCEPT ALL as the first rule, an ACCEPT HTTP as the second rule, and a DROP ALL as the end rule with a default policy of DROP" I got in with an ACCEPT ALL as the first rule, an ACCEPT HTTP as the second rule, and a DROP ALL as the end rule with a default policy of DROP ]0;iptables_demo@fakeit:~[iptables_demo@fakeit ~]$ exit logout Connection to 172.16.16.110 closed. ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ sudo iptables -L Chain INPUT (policy DROP) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:http DROP all -- anywhere anywhere Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ ssh [email protected] ssh: connect to host 172.16.16.110 port 22: Connection timed out ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ echo "I was unable to connect when the only ACCEPT rule was for HTTP and a default DROP policy and an explicit DROP rule" I was unable to connect when the only ACCEPT rule was for HTTP and a default DROP policy and an explicit DROP rule ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ sudo iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:http DROP all -- anywhere anywhere Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ ssh [email protected] ssh: connect to host 172.16.16.110 port 22: Connection timed out ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ echo "I was unable to connect with the only ACCEPT rule being HTTP and a default ACCEPT policy but an explicit DROP rule at the end of the INPUT filter" I was unable to connect with the only ACCEPT rule being HTTP and a default ACCEPT policy but an explicit DROP rule at the end of the INPUT filter ]0;nick@fakeit:/tmp[nick@fakeit tmp]$ exit Script done on Sun 27 Jul 2014 11:38:54 AM EDT
Improve this answer
6
  • No, that is not correct. Iptables rules don't stop processing on the first match. The reject at the end is a catchall, of any service not defined. Commented Jul 26, 2014 at 2:54
  • Sorry, Justin, but you don't really know what you're talking about... Commented Jul 27, 2014 at 15:32
  • IPTables works by performing the ACTION bound to the TARGET (first entry on each line of the output produced by issuing iptables -L) If the action is "do some other processing" like "LOG", then after performing the action it will continue to the following rules for additional match. that's how you can do LOG then DROP or LOG then REJECT. If the rule is ACCEPT, iptables is done - no further processing is performed. So, by having the first rule as ACCEPT ALL ANYWHERE you effectively have no INPUT filter. Commented Jul 27, 2014 at 15:56
  • Just looked at iptables-save and the rule is actually -A INPUT -i lo -j ACCEPT not sure why it is showing as ACCEPT all -- anywhere anywhere. Any ideas? Commented Jul 28, 2014 at 19:16
  • I suspect that's to allow for automatic loopback testing. That way if you did something like: iptables -P INPUT DROP (to set a default drop policy) and did iptables -D INPUT 1 (repeat a bunch of times) you'd still be able to ping 127.0.0.1, etc. no guarantee that's why, though. edit: P.S. I didn't mean to sound condescending with my earlier comment :) Commented Jul 31, 2014 at 1:00

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.