Atomic Protector Troubleshooting Guide
Can’t connect to Web Console on port 30000
If you are unable to connect to the Atomic Protector GUI this can be caused by any number of factors, the service could be down, a firewall could be blocking the connection, an issue exists with the client, etc. To assist with the process of troubleshooting these we have put together this check list. Please run through each step.
Step 1: Ensure that you have your system configured for Atomic Protector
Please make sure your system meets and is configured correctly for all the pre-requistes for Atomic Protector. These are documented on the Atomic Protector Prerequisites page.
Step 2: Check to make sure the Atomic Protector GUI is installed
Run the following command as root:
rpm -qa | grep asl-webIf the Atomic Protector GUI is installed, you will see output similar to the following:
[root@host ~]# rpm -qa | grep asl-web asl-web-3.0.25-3.el5.art
To install the Atomic Protector GUI, run this command as root:
wget -q -O - https://www.atomicorp.com/installers/asl |shThis will re-run the installer. It is safe to run the installer over an existing installation. Please check the installer output for any error messages. Generally the web console is not installed either because it was not selected for installation, or because a core dependency for the web console is missing from your system. The most common problem is attempting to install Atomic Protector on systems with third party un-supported versions of mysql. Please be sure to check the Supported Versions of Mysql pre-requisites page if the installer is reporting errors with mysql packages.
Step 3: Check to make sure all Atomic Protector updates are installed
First, check to make sure a third party product has not excluded any Atomic Protector packages from installation or upgrade. You will want to check your /etc/yum.conf file to make sure it is not globally excluding things incorrectly. For example, some third party products that do not use software and package management will try to exclude the systems PHP packages from install/upgrade by incorrectly using a regular expression such as “php*” or “php”. Disable all excludes.
Run the following commands as root:
aum -u yum -y upgrade --disableexcludes=all asl asl-web ossec-hids asl-php* asl -s -f
Step 4: Check that the GUI service is running
Run the following command as root:
ps auwww | grep tortixd
If the service is running, you should any output similar to this (details will vary for your system):
tortix 8732 0.0 0.0 220008 148 ? S May19 0:00 /var/asl/usr/sbin/tortixd root 11227 0.0 0.0 61252 836 pts/10 S+ 15:24 0:00 grep tortixd root 16369 0.0 0.0 219872 200 ? Ss Apr15 2:21 /var/asl/usr/sbin/tortixd tortix 22527 0.0 0.0 220992 188 ? S May09 0:01 /var/asl/usr/sbin/tortixd tortix 22747 0.0 0.0 221232 188 ? S May09 0:01 /var/asl/usr/sbin/tortixd tortix 22819 0.0 0.0 221492 192 ? S May09 0:01 /var/asl/usr/sbin/tortixd tortix 24961 0.0 0.0 220988 192 ? S May09 0:00 /var/asl/usr/sbin/tortixdIf you do not see the “/var/asl/usr/sbin/tortixd” process running as the tortix user, then the Atomic Protector web console server is not running.
If the service is not running, you can start it with this command:
/etc/init.d/tortixd startIf you are missing that file, and you installed Atomic Protector you may have told the installer not to install the GUI or someone may have removed this part of AP from your system. Please re-run the AP installer and reinstall Atomic Protector. The Atomic Protector installer is the best tool for installing AP completely.
If the GUI starts correctly you should see the following:
Starting tortixd: [ OK ]If you get this error:
Address already in use: make_sock: could not bind to address 0.0.0.0:30000Note
Continue to step 5 below
If you get any other message or error, please check to make sure your system is up to date by using yum to update your system. If you do not keep your system up to date you may need to install many updates that are not related to Atomic Protector. We always recommend you keep up with your vendors updates as many of them fix critical security vulnerabilities and to check with your OS vendors and any third party vendors to ensure these updates will work with your OS and third party products. To upgrade your entire system you would run these commands as root:
yum upgrade asl -s -f
Step 5: Make sure you dont have something else listening on port 30000
Run the following command as root:
netstat -anp | grep tortixd | grep 30000You should see output like the following:
tcp 0 0 :::30000 :::* LISTEN 3547/tortixdIf you do not see the tortixd service listening on port 30000, and see something else listening on that port you will need to disable this other application. Please contact the vendor for that application for assistance with disabling their software.
If you do see tortixd listening on port 30000, and you got this error in step 4:
Address already in use: make_sock: could not bind to address [::]:30000Or this error:
Address already in use: make_sock: could not bind to address 0.0.0.0:30000Restart the tortixd service by running these two commands as root:
service tortixd stop killall -9 tortixdOnce the service has been stopped, start the service back again by running the following command:
service tortixd start
Step 6: Check your firewall rules
If you are using any third party firewall management tools, uninstall them and if you can not uninstall them disable them. You will also want to make sure you do not have any pre-existing firewall rules on your system. This is the most common cause of firewall related issues: pre-existing firewall rules.
Check your tortixd ACLs:
If you have configured Atomic Protector to limit access to specific IPs, check this file to ensure your current IP(s) are on the list of custom user defined allowed IPs:
/etc/asl/firewall/tortixd-access-listIf your IP address is not included in this list you will need to add it to the file. Then run this command as root:
service asl-firewall restart
Check to see if you have any firewall rules:
Run the following command as root:
iptables -L -nIf you see output like this:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destinationThe above means you have no firewall rules. The problem is not caused by the firewall on your server, continue to Step 7.
Check to see if there are third-party firewall rules that are causing a conflict
By default, Atomic Protector will generate firewall rules that look like this:
Chain INPUT (policy ACCEPT) target prot opt source destination ASL-GEO-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ASL-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ASL-SMALLPACKETS ah -- 0.0.0.0/0 0.0.0.0/0 length 0:35 ASL-SMALLPACKETS esp -- 0.0.0.0/0 0.0.0.0/0 length 0:49 ASL-SMALLPACKETS 47 -- 0.0.0.0/0 0.0.0.0/0 length 0:39 ASL-SMALLPACKETS 30 -- 0.0.0.0/0 0.0.0.0/0 length 0:31 ASL-SMALLPACKETS icmp -- 0.0.0.0/0 0.0.0.0/0 length 0:27 ASL-SMALLPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 length 0:39 ASL-SMALLPACKETS udp -- 0.0.0.0/0 0.0.0.0/0 length 0:27 ASL-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=128 ASL-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=64 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x37 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x2B ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1A ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0A ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0D ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1C ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x03 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x01 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x29/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x22/0x22 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x11/0x01 ASL-FRAGMENTS all -f 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID ASL-TORTIXD-ACL tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30000 state NEWAll rules in the INPUT chain should start with the word “ASL”, as in the example above, with the exception of the INVALID rule:
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
Check to make sure you don’t have your servers IP addresses blocked in your firewall
Note
If you are using a third party firewall tool, you will need to check with the developers of that application to see if its blocking your servers IP. These instructions will only tell you if Atomic Protector is currently blocking your servers IPs.
Run this command script as the root user:
for ip in `ifconfig -a | grep inet | awk -F: '{print $2}' | awk '{print $1}'`; do iptables -L -n |grep $ip | grep ASL-ACTIVE-RESPONSE; doneIf you do not see any output then Atomic Protector is not blocking your servers IPs.
If you get an output similar to this:
ASL-ACTIVE-RESPONSE all -- 1.2.3.4 0.0.0.0/0When 1.2.3.4 is one of your servers IP addresses, then you have not whitelisted your servers IP address in Atomic Protector.
Check to see if there is any reason Atomic Protector generated firewall rules that are causing a conflict
The quickest way to detetermine if your Atomic Protector firewall rules are causing an issue is to flush all your firewall rules:
/etc/init.d/asl-firewall stopIf your firewall rules are flushed, you should see this:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destinationIf you see anything other than this, you have some third party firewall management product that has modified your firewall rules, or policy defaults. Please contact the developers of this software for assistance removing it, and restoring your firewall policy defaults.
If you are now able to connect, you had a firewall rule problem. If an Atomic Protector generated firewall rule caused this issue it will log this. Check your system logs for any Atomic Protector generated firewall rules, and your IP address. Run this command as root, replacing 1.2.3.4 with your IP address.
grep ASL /var/log/messages | grep 1.2.3.4If you do not see any log events then an Atomic Protector generated rule has not caused this issue.
If you still can not connect, the problem is not caused by the firewall rules on your server, continue to step 7.
Reset your firewall rules
If you have been making modifications to your firewall with the Atomic Protector firewall manager, you can reset your firewall rules to the defaults by running these commands as the root user (not via sudo):
cp /etc/asl/firewall/running.fw /root rm /etc/asl/firewall/running.fw asl -s -fIf you can now connect, then you had configured a firewall rule or rules that were blocking you from accessing the web console.
If you still can not connect, the servers firewall is not the cause of your issue, continue to step 7.
Note
You can restore your custom firewall rules, only if you used the commands above to clear your firewall rules by running these commands as the root user:
cp /root/running.fw /etc/asl/firewall/running.fw asl -s -f
Step 7: Run up a snifer to make sure the problem is not upstream
Note
Make sure you set “eth0” below to the interface that has the IP address assigned to it that you are going to test. If you tell the sniffer to watch a different interface, you will not see the traffic.
Sniffer Option 1: Wireshark
If you do not have it already installed, this command may help you install it. You will want to run this command as root:
yum -y install wiresharkOn some patforms, the command to run wireshark is “tethereal” on others it is “tshark”. Please try both commands. If these do not work for you, please contact your OS vendor for assistance installing a sniffer on your system.
tethereal -i eth0 port 30000If you do not have tethereal installed, please try these commands:
yum install tetherealOR
yum install wiresharkIf you can not install tetheral, see the “tcpdump” example that follows the tethereal example:
Capturing on eth0 0.000000 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=690440808 TSER=0 WS=7 0.000055 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=491123062 TSER=690440808 WS=7 0.047190 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=690440854 TSER=491123062 0.048117 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [PSH, ACK] Seq=1 Ack=1 Win=5888 Len=114 TSV=690440855 TSER=491123062 0.048149 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1 Ack=115 Win=5888 Len=0 TSV=491123074 TSER=690440855 0.403626 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1 Ack=115 Win=5888 Len=1448 TSV=491123163 TSER=690440855 0.403646 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [PSH, ACK] Seq=1449 Ack=115 Win=5888 Len=159 TSV=491123163 TSER=690440855 0.452383 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [ACK] Seq=115 Ack=1449 Win=8832 Len=0 TSV=690441260 TSER=491123163 0.454911 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [ACK] Seq=115 Ack=1608 Win=11648 Len=0 TSV=690441262 TSER=491123163 0.455637 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [PSH, ACK] Seq=115 Ack=1608 Win=11648 Len=198 TSV=690441263 TSER=491123163 0.455652 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1608 Ack=313 Win=6912 Len=0 TSV=491123176 TSER=690441263 0.457887 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [PSH, ACK] Seq=1608 Ack=313 Win=6912 Len=59 TSV=491123176 TSER=690441263 0.508890 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [PSH, ACK] Seq=313 Ack=1667 Win=11648 Len=565 TSV=690441315 TSER=491123176 0.547198 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1667 Ack=878 Win=8064 Len=0 TSV=491123199 TSER=690441315 1.895139 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=1667 Ack=878 Win=8064 Len=1448 TSV=491123535 TSER=690441315 1.895160 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [PSH, ACK] Seq=3115 Ack=878 Win=8064 Len=1042 TSV=491123535 TSER=690441315 1.947302 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [ACK] Seq=878 Ack=4157 Win=17536 Len=0 TSV=690442755 TSER=491123535 1.947693 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [PSH, ACK] Seq=878 Ack=4157 Win=17536 Len=37 TSV=690442755 TSER=491123535 1.947715 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [ACK] Seq=4157 Ack=915 Win=8064 Len=0 TSV=491123549 TSER=690442755 1.948027 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [FIN, ACK] Seq=915 Ack=4157 Win=17536 Len=0 TSV=690442755 TSER=491123535 1.949539 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [PSH, ACK] Seq=4157 Ack=916 Win=8064 Len=37 TSV=491123549 TSER=690442755 1.949610 9.8.7.6 -> 1.2.3.4 TCP 30000 > 46910 [FIN, ACK] Seq=4194 Ack=916 Win=8064 Len=0 TSV=491123549 TSER=690442755 1.997338 1.2.3.4 -> 9.8.7.6 TCP 46910 > 30000 [RST] Seq=916 Win=0 Len=0
Sniffer Option 2: tcpdump
If you do not have wireshark available for your platform, most OS vendors will include tcpdump by default. If tcpdump is not installed on your system, please run this command as root to install it:
yum -y install tcpdump
If this does not work for your OS, please contact your OS vendor for assistance installing a sniffer. tcpdump example:
If you have no problems upstream, you will see something like the following:
10:34:26.227472 IP 1.2.3.4.54405 > 9.8.7.6.30000: S 3615766533:3615766533(0) win 5840 <mss 1460,sackOK,timestamp 771590282 0,nop,wscale 7> 10:34:26.227514 IP 9.8.7.6.30000 > 1.2.3.4.54405: S 1843855213:1843855213(0) ack 3615766534 win 5792 <mss 1460,sackOK,timestamp 511410428 771590282,nop,wscale 7> 10:34:26.274981 IP 1.2.3.4.54405 > 9.8.7.6.30000: . ack 1 win 46 <nop,nop,timestamp 771590329 511410428> 10:34:26.275774 IP 1.2.3.4.54405 > 9.8.7.6.30000: P 1:147(146) ack 1 win 46 <nop,nop,timestamp 771590330 511410428> 10:34:26.275806 IP 9.8.7.6.30000 > 1.2.3.4.54405: . ack 147 win 54 <nop,nop,timestamp 511410440 771590330> 10:34:26.276150 IP 9.8.7.6.30000 > 1.2.3.4.54405: P 1:139(138) ack 147 win 54 <nop,nop,timestamp 511410440 771590330> 10:34:26.327459 IP 1.2.3.4.54405 > 9.8.7.6.30000: . ack 139 win 54 <nop,nop,timestamp 771590382 511410440> 10:34:26.329933 IP 1.2.3.4.54405 > 9.8.7.6.30000: P 147:723(576) ack 139 win 54 <nop,nop,timestamp 771590383 511410440> 10:34:26.369824 IP 9.8.7.6.30000 > 1.2.3.4.54405: . ack 723 win 63 <nop,nop,timestamp 511410464 771590383> 10:34:26.518491 IP 9.8.7.6.30000 > 1.2.3.4.54405: . 139:1587(1448) ack 723 win 63 <nop,nop,timestamp 511410501 771590383> 10:34:26.518514 IP 9.8.7.6.30000 > 1.2.3.4.54405: P 1587:2629(1042) ack 723 win 63 <nop,nop,timestamp 511410501 771590383> 10:34:26.518619 IP 9.8.7.6.30000 > 1.2.3.4.54405: P 2629:2666(37) ack 723 win 63 <nop,nop,timestamp 511410501 771590383> 10:34:26.518660 IP 9.8.7.6.30000 > 1.2.3.4.54405: F 2666:2666(0) ack 723 win 63 <nop,nop,timestamp 511410501 771590383> 10:34:26.570119 IP 1.2.3.4.54405 > 9.8.7.6.30000: . ack 2629 win 100 <nop,nop,timestamp 771590625 511410501> 10:34:26.570586 IP 1.2.3.4.54405 > 9.8.7.6.30000: P 723:760(37) ack 2667 win 100 <nop,nop,timestamp 771590625 511410501> 10:34:26.570654 IP 9.8.7.6.30000 > 1.2.3.4.54405: . ack 760 win 63 <nop,nop,timestamp 511410514 771590625> 10:34:26.570760 IP 1.2.3.4.54405 > 9.8.7.6.30000: R 760:760(0) ack 2667 win 100 <nop,nop,timestamp 771590625 511410501> 10:34:26.617628 IP 1.2.3.4.54406 > 9.8.7.6.30000: S 3610239263:3610239263(0) win 5840 <mss 1460,sackOK,timestamp 771590673 0,nop,wscale 7> 10:34:26.617660 IP 9.8.7.6.30000 > 1.2.3.4.54406: S 1858764235:1858764235(0) ack 3610239264 win 5792 <mss 1460,sackOK,timestamp 511410525 771590673,nop,wscale 7> 10:34:26.660089 IP 1.2.3.4.54407 > 9.8.7.6.30000: S 3617695320:3617695320(0) win 5840 <mss 1460,sackOK,timestamp 771590715 0,nop,wscale 7> 10:34:26.660111 IP 9.8.7.6.30000 > 1.2.3.4.54407: S 1850303361:1850303361(0) ack 3617695321 win 5792 <mss 1460,sackOK,timestamp 511410536 771590715,nop,wscale 7> 10:34:26.664975 IP 1.2.3.4.54406 > 9.8.7.6.30000: . ack 1 win 46 <nop,nop,timestamp 771590719 511410525> (and a lot more)If you dont see a packet exchange, the problem is not with your server - its upstream at some other firewall, or even on your desktop or with your home or office firewall.
If you do see all of this, and still can’t connect - you ARE connecting - check your browser to make sure its not breaking on the connection. For example, if you connect with “http” instead of “https”, or you are connecting to the wrong port, port 3000 instead of 30000.
Step 8: Check to make sure a third party product has not changed Atomic Protector’s permissions
Check that the GUI’s php’s session path:
/var/asl/sessionis owned by ‘root:apache’ with permissions 770.
The permissions and ownership should be:
drwxrwx--- 2 root apache
Step 9: Check to make sure tortix is in the apache group
Run this command as root:
grep ^apache: /etc/group | grep tortix
If tortix is not in the group, you will get a blank response. For example:
[root@host]# grep ^apache: /etc/group | grep tortix
If tortix is in the group, you will see a response similar to this one:
[root@host]# grep ^apache: /etc/group | grep tortixIf none of this resolves your issue, please contact support.
Not getting any emails from Atomic Protector
If you not getting any emails from Atomic Protector this can be cause by any number of factors, your mail server is down, your have spam filtering software thats blocking Atomic Protector, you have firewall rules that are blocking the connection, etc. To assist with the process of troubleshooting these we have put together this check list. Please run through each step.
Note
Run all of the commands in the steps below as root. Do NOT use sudo.
Step 1: Ensure that you have your system configured for Atomic Protector
Please make sure your system meets and is configured correctly for all pre-requisites for Atomic Protector.
Step 2: Check to make sure the Atomic Protector is installed
Run the following command:
asl -v
If Atomic Protector is installed, you will see output similar to the following:
ASL Version 3.2.15-33.el5.art: CentOS 7 (SUPPORTED)If Atomic Protector is not installed, you will see output similar to the following:
bash: asl: command not found...
Step 3: Check to make sure all Atomic Protector updates are installed
First, check to make sure a third party product has not excluded any Atomic Protector packages from installation or upgrade. You will want to check your /etc/yum.conf file to make sure it is not globally excluding things incorrectly. For example, some third party products that do not use software and package management will try to exclude the systems PHP packages from install/upgrade by incorrectly using a regular expression such as “php*” or “php”. Disable all excludes.
Run the following commands:
aum -uf asl -s -f
Step 4: Check to make sure OSSEC is working correctly
Run the following command:
ps auxwww | grep ossecYou should see output similar to the following:
root 3192 0.0 0.0 50352 112 ? S Jun25 0:01 /var/ossec/bin/ossec-execd ossecm 19622 0.0 0.0 91360 668 ? S Aug04 0:10 /var/ossec/bin/ossec-dbd ossecm 19627 0.0 0.0 24520 352 ? S Aug04 0:04 /var/ossec/bin/ossec-maild ossec 19637 1.5 0.6 47176 6424 ? S Aug04 215:45 /var/ossec/bin/ossec-analysisd root 19641 0.0 0.0 38600 364 ? S Aug04 0:44 /var/ossec/bin/ossec-logcollector root 19652 0.1 1.7 99032 17852 ? S Aug04 26:48 /var/ossec/bin/ossec-syscheckd ossec 19654 0.0 0.0 61520 364 ? S Aug04 0:00 /var/ossec/bin/ossec-monitord
If everything is working correctly, you should see all of these processes running:
ossec-execd
ossec-dbd
ossec-maild
ossec-analysisd
ossec-logcollector
ossec-syscheckd
ossec-monitord
If any of these processes are not running, please try running the following command:
service ossec-hids restart
Step 5: Ensure that you have OSSEC configured to send emails
Ensure that you have these settings set to you email address, and that this email address is valid and that you server can resolve the domain.
Check to ensure the domain resolves to a working mail server
Check you server to ensure it can resolve the domain name, and that what is resolving to is the correct server. You can test to make sure the server can resolve the domain by running this command:
host -t mx yourdomain.comReplace “yourdomain.com” with the domain name you used in those settings.
If your server can resolve the domain name you will see output similar to the following:
example.com mail is handled by 10 mail.example.comIf your server can not resolve the domain name you will see output similar to the following:
example.com has no MX record
Check to ensure your server can resolve the mail servers IP address
Then check to make sure your server can resolve the mail servers IP address. You will need to ensure that this resolves to your server, by running the following command:
nslookup mail.example.comAnd then you need to ensure that the IP address returned is the correct IP address to send mail to. If this is not correct, you will need to fix your DNS server(s) to return the correct IP address.
Step 6: Ensure that you have OSSEC configured to send emails
Set the OSSEC_NOTIFY setting to “yes”.
Set the OSSEC_MAX_MSG setting to 0. This will allow Atomic Protector to send emails as alerts occur.
Step 7: Ensure that OSSEC is enabled
Set OSSEC_ENABLED to “yes”.
Step 8: Ensure that OSSEC is set to server mode
OSSEC will only send emails if OSSEC_MODE is set to “server”
Step 9: Ensure the SMTP server is correct
Set OSSEC_SMTP_SERVER to 127.0.0.1
Check to make sure you have a mail server running on that system. If you are using 127.0.0.1 as your mail server, you can run the following command as root to see if you have a mail server running on your local system:
netstat -an | grep LISTEN | grep :25If you do not have a mail server running, please contact your mail server vendor.
If you do have a mail server running locally, and you can not connect to it, please proceed to Step 10 to check your firewall rules.
Step 10: Check your firewall rules
Check to see if you have any firewall rules by running the following command as root:
iptables -L -nIf you see output similar to the following
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destinationYou have no firewall rules and the problem is not caused by the firewall on your server. Proceed to Step 11
Check to see if there are third-party firewall rules that are causing a conflict
By default Atomic Protector will generate firewall rules that look like the following:
Chain INPUT (policy ACCEPT) target prot opt source destination ASL-GEO-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ASL-BLACKLIST all -- 0.0.0.0/0 0.0.0.0/0 ASL-SMALLPACKETS ah -- 0.0.0.0/0 0.0.0.0/0 length 0:35 ASL-SMALLPACKETS esp -- 0.0.0.0/0 0.0.0.0/0 length 0:49 ASL-SMALLPACKETS 47 -- 0.0.0.0/0 0.0.0.0/0 length 0:39 ASL-SMALLPACKETS 30 -- 0.0.0.0/0 0.0.0.0/0 length 0:31 ASL-SMALLPACKETS icmp -- 0.0.0.0/0 0.0.0.0/0 length 0:27 ASL-SMALLPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 length 0:39 ASL-SMALLPACKETS udp -- 0.0.0.0/0 0.0.0.0/0 length 0:27 Atomic Protector-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=128 Atomic Protector-BADPACKETS tcp -- 0.0.0.0/0 0.0.0.0/0 tcp option=64 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x37 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x2B ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1A ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0A ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x0D ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x1C ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x03 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x01 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x29/0x29 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x22/0x22 ASL-PORTSCAN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x11/0x01 ASL-FRAGMENTS all -f 0.0.0.0/0 0.0.0.0/0 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID ASL-TORTIXD-ACL tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30000 state NEWAll rules in the INPUT chain should start with the word “ASL”, as shown in the example above with the exception of the INVALID rule:
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALIDIf you see any rules that do not contain the word “ASL”, then you have some third party firewall management product that has modified your firewall rules, or policy defaults.
Check to make sure you don’t have your server’s IP address blocked in your firewall
Note
If you are using a third party firewall tool, you will need to check with the developers of that application to see if its blocking your servers IP. These instructions will only tell you if Atomic Protector is currently blocking your servers IPs.
Run the following command script as the root user:
for ip in `ifconfig -a | grep inet | awk -F: '{print $2}' | awk '{print $1}'`; do iptables -L -n |grep $ip | grep ASL-ACTIVE-RESPONSE; doneIf you do not see any output then Atomic Protector is not blocking your server’s IP address.
If you get an output similar to the following:
ASL-ACTIVE-RESPONSE all -- 1.2.3.4 0.0.0.0/0When 1.2.3.4 is one of your servers IP addresses, then you have not whitelisted your server’s IP address in Atomic Protector.
Check to see if there are any Atomic Protector generated firewall rules that are causing a conflict
The quickest way to determine if your Atomic Protector firewall rules are causing an issue is to flush all your firewall rules:
/etc/init.d/asl-firewall stopIf your firewall rules are flushed, you should see output similar to the following:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destinationIf you see anything other than the output above, you have some third party firewall management product that has modified your firewall rules, or policy default. Please contact the developers of the software for assistance.
If you are now able to connect, you had a firewall rule problem. If an Atomic Protector generated firewall rule caused this issue it will log this. Check your system logs for any Atomic Protector generated firewall rules, and your IP address. Run this command as root, replacing 1.2.3.4 with your IP address.
grep ASL /var/log/messages | grep 1.2.3.4If you do not see any log events then an Atomic Protector generated rule has not caused this issue.
Reset your firewall rules
If you have been making modifications to your firewall with the Atomic Protector firewall manager, you can reset your firewall rules to the defaults by running the following commands as the root user:
cp /etc/asl/firewall/running.fw /root rm /etc/asl/firewall.fw /root asl -s -fIf you can now connect, then you had configured a firewall rule or rules that were blocking you from accessing the web console.
If you still cannot connect, the servers firewall is not the cause of this issue, continue to Step 7.
Send a test Email
If you have confirmed your firewall rules are not blocking you from sending email to your mail server, send a test email using the method below:
Step 1: Connect to the mail server by running the following command (replacing 127.0.0.1 with your mail server):
telnet 127.0.0.1:25Step 2: The server will respond with something similar to the following
Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. 220 server ESMTP PostfixStep 3: Send the “helo” response with your server’s name, for example
helo www.example.comStep 4: The server will respond with something similar to the following
250 yourmailserversnamesStep 5: Send the mail from line using the setting you configured with OSSEC_FROM
mail from: someuser@atomicorp.comStep 6: If your mail server is setup correctly it will respond with the line below. If the server does not respond appropriately you will need to contact your mail server vendor.
250.2.1.0 OKStep 7: Send the rcpt to line using the setting you configured in the OSSEC_EMAIL setting
rcpt tp: someuser@atomicorp.comStep 8: If you mail server is setup correctly it will respond with the same line in Step 6.
Step 9: Now send a quick test message, first type “data” and hit enter.
Step 10: If your mail server is setup correctly it will respond with a similar response as below:
354 End data with <CR><LF>.<CR><LF>Step 11: Type in a quick message and hit enter twice.
Step 12: If your mail server is setup correctly it will respond with a similar response as below:
250 2.0.0 Ok: queued as D5D6A10D8112
Step 11: Run a sniffer to make sure the problem is not upstream
Note
Make sure you set “eth0” below to the interface that has the IP address assigned to it that you are going to test. If you tell the sniffer to watch a different interface, you will not see the traffic.
Sniffer Option 1: wireshark
If you do not already have it installed, this command may help you install it. Run the following command as root:
yum -y install wireshark
On some patforms, the command to run wireshark is “tethereal” on others it is “tshark”. Please try both commands. If these do not work for you, please contact your OS vendor for assistance installing a sniffer on your system.
Run the following command as root:
tethereal -i eth0 port 25 and hostname 1.2.3.4You should see output similar to the following, if there are no problems with your upstream:
0.000000 127.0.0.1 -> 127.0.0.1 TCP 50849 > smtp [SYN] Seq=0 Win=32792 Len=0 MSS=16396 TSV=596718574 TSER=0 WS=8 0.000014 127.0.0.1 -> 127.0.0.1 TCP smtp > 50849 [SYN, ACK] Seq=0 Ack=1 Win=32768 Len=0 MSS=16396 TSV=596718574 TSER=596718574 WS=8 0.000025 127.0.0.1 -> 127.0.0.1 TCP 50849 > smtp [ACK] Seq=1 Ack=1 Win=33024 Len=0 TSV=596718574 TSER=596718574 0.176700 127.0.0.1 -> 127.0.0.1 SMTP S: 220 yourserversname ESMTP Postfix 0.176713 127.0.0.1 -> 127.0.0.1 TCP 50849 > smtp [ACK] Seq=1 Ack=40 Win=33024 Len=0 TSV=596718751 TSER=596718751Sniffer Option 2: tcpdump
If you do not have wireshark available on your platform, most OS vendors will include tcpdump by default.
If your OS does not have tcpdump installed by default, run the following command as root to install it:
yum install -y tcpdump
Run the following command as root:
tcpdump -i eth0 port 25 and hostname 1.2.3.4
If there are no problems with your upstream, then you should see output similar to the following:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes 13:06:11.808773 IP 1.2.3.4.40930 > 9.8.7.6.smtp: S 3226807163:3226807163(0) win 32792 <mss 16396,sackOK,timestamp 597368964 0,nop,wscale 8> 13:06:11.808790 IP 9.8.7.6..smtp > 1.2.3.4.40930: S 3784740604:3784740604(0) ack 3226807164 win 32768 <mss 16396,sackOK,timestamp 597368964 597368964,nop,wscale 8> 13:06:11.808801 IP 1.2.3.4.40930 > 9.8.7.6..smtp: . ack 1 win 129 <nop,nop,timestamp 597368964 597368964> 13:06:11.950623 IP 9.8.7.6..smtp > 1.2.3.4.40930: P 1:40(39) ack 1 win 128 <nop,nop,timestamp 597369106 597368964> 13:06:11.950636 IP 1.2.3.4.40930 > 9.8.7.6..smtp: . ack 40 win 129 <nop,nop,timestamp 597369106 597369106> (and a lot more)
Step 12: Packets not exchanging
If you dont see a packet exchange, the problem is with the mail server itself. Either your mail server does not accept connections, is not running a mail server, you have an upstream firewall that is blocking you (if you are using a remote mailserver) or you have a network problem that is preventing you from connecting to your mail server. Please contact your mail server vendor for assistance with connections issues with your mailserver.
If everything is working, and you have confirmed that your mail server is listening, and you can successfully send a test message, then check your mail servers logs. This means that Atomic Protector is able to send email, and your mail server is just not sending it on your account. Please contact your mail server vendor for assistance with configuring your mail server to send email to your email account.
If you require additional assistance with this issue, please send us the output of all these steps as we will need this information to assist you. Please do not contact us about issues concerning the correct configuration of your mail server. Please contact your mail server vendor for assistance with their products.
Atomic Protector Web Console Not Running
Step 1: Check to make sure the Web Console is installed
Run the following command as root:
rpm -qa | grep tortixdIf you do not see the package installed, then you are missing components of Atomic Protector. You should run “yum upgrade” and then run the following command:
wget -q -O - https://www.atomicorp.com/installers/asl |sh
Step 2: Start the service
Run the following command as root:
/etc/init.d/tortixd start
Step 3: Follow the steps above in “Can’t connect to Web Console on port 30000” to make sure your firewall and upstream (if any) are configured correctly
You can also re-install the Web Console by running the following command:
yum reinstall asl-web
Empty Web Console
An empty Web Console means that a third party application or user has misconfigured your Atomic Protector directories to prevent tortixd from opening the files it needs to render the GUI.
Check the permissions on the following directory:
/var/asl/sessionThe permissions on the directory above should be:
drwxrwx--- 2 root apache 4096 Jul 29 14:55 sessionIn general, if a third party application or user changes these permissions Atomic Protector may be able to fix this by running the following command:
asl -s -f
No Events in Atomic Protector Web Console
Step 1: Test Atomic Protector Web
First check to make sure that Atomic Protector Web is working. In the Recent Events tab of the Security Events window, set the level filter to 1 so you can see all events. By default Atomic Protector Web is set to a higher level to filter out lower importance events and to only display attacks and highly supicious events by default. If the AP Web is showing lower level events, you may either not have any attacks or suspicious events to report, or you may have disabled or not installed all of AP protection components.
Step 2: Check to make sure Atomic Protector is up to date
If you are running an older version of Atomic Protector, please see the article on how to upgrade your Atomic Protector subscription.
If you are running the most current version of Atomic Protector, run the following commands as root:
/var/asl/bin/aum -u /var/asl/bin/asl -s -f
Step 3: Verify that OSSEC is working correctly
Run the following command as root:
ps auxwww | grep ossecYou should see output similar to the following:
root 3192 0.0 0.0 50352 112 ? S Jun25 0:01 /var/ossec/bin/ossec-execd ossecm 19622 0.0 0.0 91360 668 ? S Aug04 0:10 /var/ossec/bin/ossec-dbd ossecm 19627 0.0 0.0 24520 352 ? S Aug04 0:04 /var/ossec/bin/ossec-maild ossec 19637 1.5 0.6 47176 6424 ? S Aug04 215:45 /var/ossec/bin/ossec-analysisd root 19641 0.0 0.0 38600 364 ? S Aug04 0:44 /var/ossec/bin/ossec-logcollector root 19652 0.1 1.7 99032 17852 ? S Aug04 26:48 /var/ossec/bin/ossec-syscheckd ossec 19654 0.0 0.0 61520 364 ? S Aug04 0:00 /var/ossec/bin/ossec-monitordIf everything is working correctly, you should see all of these processes running.
If you do NOT see all of the processes above, run the following command as root:
service ossec-hids restart
Step 4: Verify that MySQL is working correctly
Check for errors in the MySQL /var/log/mysqld.log file.
Make sure MySQL is listening on the IP and port you configured.
A database that is not listening on a TCP port or the same port that you have configured Atomic Protector to use. Check to make sure your MySQL server is listening on port 3306 on the IP address you have configured AP to use as its database server.
Additionally, a database server that is listening on a different IP address from the one configured for AP can cause start up errors. Check to make sure your MySQL server is listening on port 3306 on the IP address you have configured AP to use as its database server.
Check the OSSEC_DATABASE_SERVER setting and ensure that you can connect to port 3306 on that configured IP or hostname.
Check to make sure the tortix MySQL user permissions are correct
If you are receiving an error during database setup similar to this:
Loading Atomic Protector Web database schema: ERROR 1227 (42000) at line 27: Access denied; you need the SUPER privilege for this operationthen you need to check the MySQL permissions. Specifically if the tortix mysql user is either unable to create or is unable to execute triggers, events will not show up properly.
To resolve this, follow the steps below:
Step 1: Log into MySQL as the tortix user by running the following command
mysql -u `cat /etc/asl/config | grep OSSEC_DATABASE_USERNAME | awk -F\" '{print $2}'` -p`cat /etc/asl/config | grep OSSEC_DATABASE_PASSWORD | awk -F\" '{print $2}'`Step 2: Once you have logged in, select the Atomic Protector database by running the following command
use tortix;Step 3: Verify that triggers have been created by running the following command
SHOW TRIGGERS;Step 4: If you do not see any triggers, then the tortix user does not have the correct permissions. Run the following commands
Note
Log into MySQL as the root/admin user.
GRANT ALL ON tortix.* TO 'tortix'@'localhost' IDENTIFED BY '[tortix user's password]'; GRANT SUPER ON *.* TO 'tortix'@'localhost' IDENTIFED BY '[tortix user's password]';
Step 5: Run Atomic Protector Web Validation
After any of the corrective actions above, or if none of the above resolved the problem, run the following command:
/var/asl/bin/asl --web_validateData will be available in Atomic Protector Web on the first polling after correction, assuming any new events have come in, it does not require a refresh or relogging. Event data that existed prior to the correction will not be repaired.
Atomic Protector Firewall
FTP Not Working?
Background:
FTP is inarguably the most complex protocol in use on the Internet today, if not of all time, it is also the oldest. Unlike other protocols, like SMTP and HTTP, FTP does not work over one port or even a predicable set of ports. FTP uses lots and lots of ports, and it dynamically selects the ports it uses both of the client and server side randomly. You might think FTP just works over port 21, but thats only the tip of the iceberg. FTPs actual “data” connections happen on random dynamically allocated ports that are unique for each client and connection! Which makes it rather difficult for any firewall to support it. To support FTP all firewalls require what are sometimes referred to as “helper” modules. These modules actually look for what appears to be FTP traffic on all those random, dynamic ports. And they have to do this passively, and in real time because the FTP server has no way to tell the firewall what ports it needs open. The firewall has to figure this out as it goes: dynamically opening and closing the ports that FTP needs to use when you are using any firewall. Without these modules FTP simply doesnt work. And, because FTP is unlike any other protocol in use on your system, thats why its important that not only your firewall rules be setup correctly for FTP, but that both your firewall and your operating system support it, understand it and manage the connections in real time correctly. The article below provides a brief overview of how FTP works
Proposed Solutions:
If you are using the Atomic Protector Kernel make sure you are not using a third party firewall.
Make sure you have port 21 open by setting it in the Atomic Protector setting FW_INBOUND_TCP_SERVICES
If you are using passive mode, you will need to allow in the range of ports you configured your FTP server to use for passive mode.
If you still cannot access FTP, check to ensure you are not using any third party firewalls, including upstream firewalls which could also be blocking on this port.
If you are not using the Atomic Protector Kernel make sure you are not using a third party firewall.
Check to make sure your third party kernel includes the standard Linux FTP tracking modules. If you do not have these modules, no Linux firewall will work.
You can check to see if you have the modules loaded by running the following command:
ismod | grep ftpYou should see the following two modules loaded:
nf_nat_ftp nf_conntrack_ftpIf they are NOT loaded, you can load them by running the following commands:
modprobe nf_nat_ftp modprobe nf_conntrack_ftp
If you are using encrypted FTP
As explained above, FTP is a very complex protocol that works over multiple dynamic ports. When Linux is configured to use its kernel based firewall, it uses special “helper” modules to detect the dynamic FTP data connections. These connections do not occur over port 21 or 20, they are dynamic “high ports” (e.g. ports 1024 and higher). The client and server negotiate these (either the client sends the ports it wishes to use, or the server sends the port it wishes to use). When this occurs over an encrypted FTP session Linux can not see this negotiation happen (because its encrypted), and therefore will not dynamically open and close these high port data connections between the server and the client. Therefore, you can not use encrypted FTP with any firewall without permanently opening a range of ports on your firewall(s) for your FTP server(s) to use, and configuring your FTP server(s) to only use that range of ports (this is called “passive mode”), instead of dynamic picking ports (also called “active mode”). Please contact your FTP vendor for assistance with configuring your FTP server. Once you have correctly configured your FTP server, then you will need to permanently open these ports by adding them to your Atomic Protector firewall configuration.
Keep in mind that FTP was never designed to be encrypted, and in some cases when faced with multiple firewalls (including desktop firewalls) it may simply not be possible to pass the FTP data connections through successfully. For this reason, other protocols were designed to replace to FTP, such as SSH and HTTPS.
If you are using ProFTP, please follow the steps below:
Step 1: Edit /etc/prodftpd.conf and add the line below
PassivePorts 60000 60500Step 2: Add the ports above to the Atomic Protector setting FW_INBOUND_TCP_SERVICES
Step 3: Restart ProFTP by running the following command
restart proftp
Resetting Firewall to Defaults
To reset the firewall to only those rules generated by Atomic Protector, follow the steps below:
iptables-save > /etc/asl/firewall/backup.fw
service asl-firewall stop
rm -f /etc/asl/firewall/asl_stop.fw
service asl-firewall start
Additional Information
SELinux policies have been known to interfere with some RPM updates. This is because SELinux policies are not always adjusted for modern platforms and third party packages, such as control panels. This can manifest itself in mysterious failures in %pre and %post macros (confirmed on RHEL4).
Atomic Protector includes an advanced RBAC system that is more powerful and easier to use than SELinux and we recommend you use that instead of SELinux. However, if you wish to use SELinux AP will work fine with SELinux, however you may need to adjust your SELinux policies for your systems specific needs.
If you encounter any issues with rpm installations on your system, and you are not qualified to adjust your SELinux policies that came with your operating system, we recommend you disable SELinux and use the built in RBAC in AP.
To disable SELinux set:
selinux=0
in the kernel boot parameters for your system.
setenable 0, setenforce 0 and disabling SELinux with sysctl are not effective. To disable selinux you must boot with selinux=0 set for your system.