Atomic Protector Error Messages

Installation Error Messages

ERROR 1045 (28000): Access denied for user ‘admin’@’127.0.0.1’ (using password: YES)

  • This error could mean several things are wrong with your system, we have listed them below:

    1. You have supplied the wrong username for your MySQL superuser.

    2. You have supplied the wrong password for your MySQL superuser.

    3. You have skip-name-resolve enabled in MySQL

    4. You have skip-networking enabled in MySQL

    5. MySQL is not listening on the IP addres you configured.

Note

Version 5 and below only.

AP Command Line Errors

Checking directory : not found [FAILED]

  • This means you have a blank entry in your admin users setting. Check to make sure you don’t have a blank entry. For example:

    foo,,bar

foo,bar,

cmd_system ERROR: ‘/bin/echo 16384 > /proc/sys/fs/inotify/max_user_watches >/dev/null 2>&1 (1)’

  • This means that AP can not increase the number of inotify “watches” on the system. The Linux inotify system provides a mechanism for monitoring filesystem events. Inotify can be used to monitor individual files, or to monitor directories. When a directory is monitored, inotify will return events for the directory itself, and for files inside the directory. AP uses Inotify to monitor the file system from changes to software, configuration files and other sensitive parts of the operating system in real time.

    This error generally occurs on a VPS system where Inotify limits are set on the host, and can not be changed on the guest. Keep in mind that a watch is require for each file in the directory being watched. By default those directories are:

    /etc
/var/ossec/active-response
/var/ossec/etc
/var/ossec/agentless
/bin
/lib
/lib64
/opt
/sbin
/usr/bin
/usr/lib
/usr/lib64
/usr/local/bin
/usr/local/lib
/usr/local/sbin

  • Proposed Solutions:

    1. Increase Inotify Watches on hardware node

    On Virtuzzo/Openvz/[VPS] systems it is possible to change the max_user_watches limit on the hardware node only. This will raise the limit for all containers.

    To increase the limit on the hardware node, run the following command:

    [root@vz ~]# sysctl -w fs.inotify.max_user_watches=1000000

    To save it across reboots, redirect the ouput as shown in the command below:

    [root@vz ~]# sysctl -w fs.inotify.max_user_watches=1000000 >> /etc/sysctl.conf

    1. Remove directories from realtime watches (not recommended)

Error: could not find general_firewall_ftp_mod

  • This means that AP could not find any FTP modules loaded into the kernel. If this system is a VPS system, then this is expected behavior. VPS systems do not allow the user (including root) to see what modules are loaded into the kernel. There is no solution to this issue except to check to see if the FTP modules are loaded in the VPS host. Until such time as Virtuzzo/OpenVZ allow VPS systems to see the modules loaded into the kernel, there is no solution to this issue.

    We highly recommend if your are using FTP that you check the VPS host to ensure the FTP kernel modules are loaded to ensure that FTP and firewall rules will work properly.

aum Errors

2 9901 APCommon::cmd_system ERROR: ‘/usr/bin/clamscan -d /var/awp/rules/clamav/* /etc/asl/config

  • This means someone or some third party product has removed clamav from your system. You will need to manually reinstall it with this command as root:

    yum reinstall clamd clamav clamav-db

Note

Do not install clamav from third party products or sources, as documented in the installation instructions for AP

tortixd Errors

Tortixd “You don’t have permission to access / on this server.”

  • This typically means someone or something has removed the AP web console from your system. AP can not remove its own web interface. The recommended solution is to re-install AP, as this is not a normal condition and if this component is missing it is likely other components in AP are missing as well.

Invalid command ‘SSLEngine’, perhaps misspelled or defined by a module not included in the server configuration

  • This means that either mod_ssl is not installed on the system, or someone or some third party product has removed the mod_ssl package from the system. Run this command as root to install mod_ssl:

    yum install mod_ssl

ERROR 3 (HY000) at line 3: Error writing file ‘./some/file’ (Errcode: 28 - No space left on device)

  • This means your file system has run out of space.

Error creating rule: Failed to resolve operator: detectSQLi

  • This means you have an out of date AP installation and you need to upgrade.

Table ‘tortix.awp_user’ doesn’t exist

  • This means that you did not setup your AP database. To manually setup the AP database run the following command as root:

    /var/awp/bin/ossec_database_setup.sh

Note

It is safe to run this script multiple times.

Generic Errors

mod_sed requires httpd

  • If you are using cpanel, you can not install mod_sed as an rpm. Please run the AP installer, cpanel compiles everything for Apache from source, and the use of the AP installer to install mod_sed is the only supported method.

An error occurred attempting to open file /var/awp/data/updates_pending.log

  • This file should be owned by the root user and root group, and it should also have user and group read and write permissions, as in the example below:

    -rwxrwx--- 1 root root 81 Jan 1 00:00 /var/awp/data/updates_pending.log

Note

If you are using third party SELinux rules, check to make sure your SELinux rules are not preventing AP from writing to this file.

yum is reporting rpm is already installed

  • Remove the duplicate entry by running the following command (the command below is an example):

    rpm -e gradm-2.9.1-22.el5.art --nodeps

  • Regenerate the RPM database by running the following command:

    rm -f /var/lib/rpm/__db.00* && rpm --rebuilddb

  • Reinstall the package by runing the following command

    yum reinstall gradm

Deleting old audit records

  • Run the following command and change the number “7” to the number of days of audit records you wish to keep:

    /usr/bin/find /var/awp/data/audit -maxdepth 1 -type d -ctime +7 -exec /bin/rm -rf {} \;

Tomcat is stopped by PAX

  • Java performs certain actions that violate stack protection security models. To allow JAVA to run in this manner, you simply need to this command as root:

    paxctl -mps /path/to/java/bin/java

Up2date Issues

up2date_client.up2dateErrors.CommunicationError: Error communicating with server.

  • This means that Redhats up2date client can not communicate with the Redhat update servers. Please contact Redhat for assistance with this error.

Error communicating with server. The message was:

  • An error has occurred while processing your request. If this problem persists please enter a bug report at bugzilla.redhat.com. If you choose to submit the bug report, please be sure to include details of what you were trying to do when this error occurred and details on how to reproduce this problem.

  • Proposed Solution: This is not an AP error. This means that your system is configured to use Redhat Update Network and you do not have valid credentials to use their server. Contact Redhat support for assistance.

Yum Update Errors

Error performing checksum

  • This usually means that yums cache is out of date, try running this command as root to clear the cache before you run your yum installation or upgrade/update commands:

    yum clean all

There are unfinished transactions remaining.

  • This is not an AP error, that error is generated by your Operating Systems package management system. This error means that your operating systems package management system was previously used to install, remove and/or upgrade software, and the last transaction was not completed. This has left the package management system of your operating system in an incomplete state, and is preventing software from being properly installed, upgraded and/or deleted.

    This tool is not part of AP, is not used by AP and is not supported by Atomicorp. Please contact your OS vendor for assistance with issues with your OSes package management system. The information that follows is provided as a courtesy only.

    If you want to install this tool, please follow the instructions generated by yum. Specifically, this message is stating that the tool yum-complete-transaction is part of the “yum-utils” package:

    The program yum-complete-transaction is found in the yum-utils package. To install that package you need to run this command as root:

    yum -y install yum-utils

    You need to install that package to use that tool. This tool is part of the overall system that allows you to (among other things) pause & resume upgrades.

    If you dont know where a component comes from you can use “yum provides /path/to/file” (wildcards accepted) to search.

    Once this tool is installed, you can tell your OSes package management system to complete the last incomplete transaction with this command:

    yum-complete-transaction

Error: Missing Dependency: httpd-mmn = 20051115 is needed by package

  • This means that the system does not have apache installed. If you have installed apache via a non-package managed means (such as from source code). Contact your apache vendor for assistance with this error.

error: unpacking of archive failed on file /var/awp/etc/httpd/logs: cpio: rename

  • /var/awp/etc/httpd/logs is a directory. This should be a symbolic link to /var/log/tortixd

  • Steps to fix this error:

    Step 1) Remove the log directory from /var/awp/etc/httpd/

    rm -rf /var/awp/etc/httpd/logs*

    Step 2) Update the tortixd package with:

    aum -u

  • Package issue with paxctld 1.2.1. Run the following command as root:

    rpm -e paxctld-systemd-1.2.1-1.el7.art.x86_64 --justdb

HTTP Error 401: Authorization Required

  • This means that your system is not configured with a valid AP subscription account. Please check your username and password in your AP configuration and check to make sure your subscription is up to date.

Update Errors

HTTP request sent, awaiting response… 401 Authorization Required

  • This can mean serveral things:

    1. your license manager username and/or password is incorrect and you need to configure AP to use the correct credentials, or

    2. you changed your license manager username and/or password is incorrect and you need to configure AP to use the current credentials, or

    3. you do not have a license for AP, or

    4. the license has expired, or

    5. you have exceeded your license (AP is installed on more systems than your license allows).

    Note

    The first three reasons are the most common, see the solutions below.

  • Proposed Solutions:

    • If you are installing AP for the first time, then the error means you have entered your credentials incorrectly, or you are attempting to install AP on more systems than you have licenses for.

      Step 1) Check the License Manager to ensure your license(s) is/are up to date.

      Step 2) Run the AP installer. When asked to enter your subscription username and password, please enter the same credentials you used in step 1.

      Note

      If you do not know those credentials, you can reset them here

      Step 3) If you are still getting this error, run these three commands as root:

      rm -rf /etc/asl
rm /etc/yum.d/awp.repo
rm /etc/yum.d/tortix-common.repo

      Then run the AP installer again.

    • If you have successfully installed AP on the system

    Step 1) Check the License Manager to ensure your license(s) is/are up to date.

    Step 2) Follow the process below:

    • Log into the AP Web Console

    • Click on the ‘Settings’ tab

    • Click on ‘AP Configuration’

    • Check to make sure the License username and password are set to your License Manager credentials.

    • Click the ‘Save Changes’ button

    • Run the update process

Error: Another instance of AP appears to be running

You may see the error below:

Another instance of AP is running

This means that either another instance of AP is running (only one instance can run at a time), you can check this by running the following command as root:

ps auxwww | egrep "/awp |/aum " | grep -v grep

If AP is running you should see output similar to the following:

root      7578  7.0  0.4  81476 17796 pts/7    S+   14:55   0:00 awp -s

If you do NOT see an AP instance running, first check to make sure you have not run out of drivespace in /var. If AP cannot write to the /var partition, this error can occur. Once the space issue has been corrected, please run the following command as root:

rm -f /var/awp/tmp/awp.lock

This system is not licensed for this feature

If you get the following error:

This system is not licensed for this feature.
Please click here to get a license for the AP T-WAF

Please run this command as root to install the T-WAF:

yum install awp-waf-module

Checking service for authorized_keys: not found [FAILED]

  • This means that when you installed AP you did not configured your system to use keys for SSH logins, but is instead configured to only use passwords. Passwords are less secure than keys as keys are a simple for of a two factor authentication system, something you have and something you know.

  • You can define admin users for password only based authentication. Please see the following AP settings:

    • ADMIN_USERS

    • SSH_PASSWORD_AUTH

Valid Admin users detected: no [HIGH]

  • This means that when you installed AP you did not configured your system to use non-privileged accounts for logins. This means that you allow root logins to your system which is very dangerous and should be disabled. All users, including admins, should always log into a unique account for each person so that you will know logged into your system, and then those non-privileged users can su to root, or can use sudo to run privileged commands.

Checking username directory : not found [FAILED]

This means that you have defined an ADMIN_USER that does not have a /home directory on the system. For an ADMIN_USER to be valid the following must be true:

  1. The user must exist

  2. The user must have a home directory

  3. The users home directory must have a .ssh subdirectory

  4. The permissions on the .ssh subdirectory must be valid

  5. The user must have a valid SSH key installed on the system

WARNING: SSH will not be reconfigured at this time.

This means either:

  1. You did not configure SSH when you installed AP, so AP will not reconfigure SSH per your instructions.

  2. You told AP to make changes to your SSH configuration, and AP has not done this because they would prevent you from logging into the system.

Note

Most commonly this occurs if an ADMIN_USER is not defined, or if an ADMIN_USER is defined that the user does not have a valid SSH key installed on the system.

FAILED: REmote root logins are still permitted: [HIGH]

  • This means that when you installed AP you did not configured your system to use non-privileged accounts for logins. This means that you allow root logins to your system which is very dangerous and should be disabled. All users, including admins, should always log into a unique account for each person so that you will know logged into your system, and then those non-privileged users can su to root, or can use sudo to run privileged commands.

FAILED: Password authentication is enabled: [HIGH]

  • This means that you did not configured SSH when you installed AP to disallow password based logins. AP will ask you to configure SSH to use key based authentication as it is more secure. If you tell AP not to configure this AP will report this as a high risk vulnerability.

/etc/cron.hourly/awp: Error: AP has not been configured

  • First check to make sure that AP was configured after installation. To configure AP run this command:

    awp -c

    If you did configure AP, check to see if the line “CONFIGURED=yes” is at the bottom of /etc/asl/config. If you do not see this line, add it to your AP configuration file.

Critical Risk: POSIX ACL support was not detected. The T-WAF module cannot be supported in this environment. Please consult your vendor documentation for further assistance on enabling POSIX ACLs.

  • AP will test the /var filesystem to see if its supports ACLs. This error means your operating system does not support ACLs on the /var filesystem.

  • Discussion:

    POSIX ACLs allow Linux systems to use fine grained control on file permissions. POSIX ACLs are required in AP if you want to use the T-WAF, and are are documented as a pre-requisit

    The T-WAF is only used optionally to protect remote web services you configure, and non-Apache web services on the local system, such as other webservers (NGINX, LiteSpeed, etc.). Apache is protected with a special module, and does not require the T-WAF.

    If your system does not support ACLs you can not use the T-WAF. The T-WAF allows you to configure local and remote web services to protect with the AP WAF. This is separate from the embedded WAF AP will install in Apache. The embedded WAF does not require POSIX ACL support.

    Specifically, this error can be caused if:

    • You are not using the AP Kernel and are using a kernel that does not support ACLs

    • /Var filesystem is mounted on a filesystem that does not support ACLs

    • /Var filesystem has ACL support disabled

    • Operating Systems has disabled or crippled ACL support in Linux

  • Solutions:

    • Check your kernel, make sure you are running the AP Kernel, the AP Kernel fully supports ACLs for the filesystems documented in the prerequisites.

    • Check your filesystem TYPE

      Make sure your /var filesystem supports POSIX ACLs. A full list of filesystems that the AP Kernel supports with ACLs are available on the prerequisites page.

      You can check how your filesystems are mounted by running the “mount” command as root. For example:

      # mount

/dev/md0 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/md1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

      The fifth field will tell you what filesystem you are using. In the example above, the / filesystem is using “ext3”.

    • Check to see that the filesystem is mounted with ACL support. Make sure your /var filesystem is mounted with ACL support.

      Check your /etc/fstab file. A normally mounted partition will look something like this:

      /dev/md0                /                       ext3    defaults        1 1

      The fourth field lists the mount options. If your system is using “defaults”, then on many systems this means that ACL support should be enabled. However, your file system may be using a distribution that has changed your defaults, and thereby disabled ACL support.

      If you believe this is your case, attempt adding acl to the as in the example below:

      /dev/md0       /                       ext3    defaults,acl    1 1

      If your system is not using the term “defaults”, then your system does not have ACLs enabled. For example:

      /dev/md0                /                       ext3    noatime,nodiratime        1 1

      In the previous example, the defaults are not used, so ACLs are not enabled. You would need to add acl support to the filesystem options, as in the example below:

      /dev/md0                /                       ext3    acl,noatime,nodiratime        1 1

      If changing your /etc/fstab does not work, then either your kernel does not support POSIX ACLs (the AP kernel support ACLs), you are not using a filesystem that supports ACLs for your /var directory, or your OS has been crippled to not support ACLs. If you are sure your file system is mounted with ACL support enabled, your file system supports ACLs, and that you are using a kernel that supports POSIX ACLs, check with your OS vendor to determine what may be missing from your OS to support POSIX ACLs.

      If you want to try to remount a partition with acl support, run the command below as root. Be sure to change “/” to the mount point you are changing. For example, if you want to remount your root partition, you would use “/”. If you had a dedicated /var partition (not directory) you would change that to “/var”.

    • Check your virtualization solution

      If you are using a virtualization solution, and you are using the AP kernel and your file system both supports and is mounted with ACL support please contact your virtualization vendor to determine what in their virtualization solution may be preventing you from using ACLs in your file system. This is a basic Linux filesystem capability, and is supported in all Linux distributions, however some virtualization vendors have been known to disable this.

  • Testing manually

    Once you believe you have your system fixed, and you have AP installed, you can run this command as the root user to test to see if your /var filesystem supports ACLs:

    touch /var/awp/tmp/testfile;/usr/bin/setfacl -m g:root:rw /var/awp/tmp/testfile; echo $?

    If you system supports ACLs you see the number “0” returned from the following command:

    touch /var/awp/tmp/testfile;/usr/bin/setfacl -m g:root:rw /var/awp/tmp/testfile; echo $?
0

    If you get any other number than 0, that means the /var filesystem does not support ACLs on your system. For example:

    touch /var/awp/tmp/testfile;/usr/bin/setfacl -m g:root:rw /var/awp/tmp/testfile; echo $?
1

Not Found: The requested URL /awp/index.php was not found on this server.

  • This means the AP GUI is missing or was not installed on your system. You can check to see if its installed with this command run as root:

    rpm -q awp-web-gui

  • If the rpm is missing re-install it with:

    yum install awp-web-gui

ModSecurity Errors

Syntax error on line 31 of /usr/local/apache/modsecurity.d/00_awp_whitelist.conf

  • This means that something/someone has removed mod_security from system. AP will not do this. To fix this please follow the process below:

    Step 1: Run the following command as root

    aum -uf

    Step 2: Test the Apache configuration by running the following command

    service httpd configtest

    Step 3: Restart Apache by running the following command

    service httpd restart

    Step 4: If this did not reinstall mod_security on your system, then you will need to run the AP installer again. Do NOT uninstall AP, simply run the following command

    wget -q -O - https://updates.atomicorp.com/installers/awp |sh

Request Body Parsing Failed. Multipart parsing error: Multipart: writing to /somedirectory failed:

  • To fix this issue, free up some disk space.

ModSecurity: Error reading request body: Connection reset by peer

  • This error message means that the client killed the connection. This error is not caused or generated by modsecurity or the server. This error message is simply reporting that the connection was reset by the client before it completed.

  • By default apache does not log these errors (even though they occur), however modsecurity will log these errors as they may be indications of malicious activity. This error is normally benign, but you should investigate any large occurrences of this error.

ModSecurity: Request body (Content-Length) is larger than the configured limit

  • Change this setting in the AP Web Console to the maximum value: MODSEC_REQUESTBODYLIMIT

Note

This is set in bits. Therefore, if you want to set the limit by bytes you will need to compute for bits.

  • The default is 1 gigabit, or 134217728

  • 32 bit systems note: There is a hard limit in apache itself of 2Gb on 32 bit platforms. Apache supports file sizes greater than 2 Gb on 64 bit platforms where Apache is built with large file support. Versions of Apache that are not compiled with 64 bit offsets (large file support), or are operated on 32 bit platforms will have a 2GB maximum limit. This 2GB hard limit is not enforced by modsecurity nor is it something AP establishes, this is an internal limit in apache. Keep in mind this is also request body limit, not a response body limit.

httpd: ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated

  • This means that you are using out of date rules. If you are using Atomicorp rules, then this means you are not using the latest real time rules. The latest real time rules do not use transformations in the SecDefaultAction.

    If you are using third party rules, this means that the rules contain a transform function in the SecDefaultAction setting. This is a deprecated setting.

ModSecurity: Found another rule with the same id

  • This means that you have a modsecurity rule with the same id. All of our rules have unique id’s, and AP configures modsecurity to only load our rules once.

  • This error can only occur for one of two reasons:

    1. Something has configured Apache to load the ModSecurity rules twice

    This can only happen if you have used a third party product to install modsecurity (example: cpanels easyapache), or a third party modsecurity management tool that enables, disables, configures and loads rules. Do not use any third party tools to setup, install or configure modsecurity. Disable these tools, and re-run the AP installer to ensure that modsecurity is setup correctly. You should only use AP to install modsecurity, and to manage your modsecurity rules, configuration, upgrades and installation.

    If you have used any third party tools to install, configure or manage modsecurity uninstall the tools and contact the developers of that software to make sure the changes they have made have been completely removed. AP is not supported with third party modsecurity tools or installations, and these third party tools are not necessary when using AP. You may need to reinstall AP in some cases. If the following does not reconfigure modsecurity to a working configuration, you will need to reinstall AP:

    Follow this process below:

    Step 1) Remove all third party ModSecurity tools

    Step 2) If you used third party tools to install ModSecurity, including installing ModSecurity from your control panel uninstall ModSecurity and disable it’s installtion from your third party tool.

    Step 3) If you AP installtion is new, re-run the AP installer

    Step 3b) If this is an exisiting AP installation, please run the following commands

    aum -uf
awp -s -f

    1. You have other rules that are using the same id’s already assigned to other rules.

    If you are using third party or custom rules, check to make sure they have unique id’s. Modsecurity requires a unique id for each rule. The range 300000-399999 is used by our rules, do not use this range for any custom rules, and if you have third party rules with these id’s be sure to remove these rules.

    If you have used our delayed rules in the past, and setup modsecurity yourself or had a third party setup modsecurity for you, make sure that installation is completely removed from your system. The AP installer will attempt to do this, but as not all installations are the same or follow the same conventions it may be possible your modsecurity installation deviates from the standards, and is not removable by AP.

ClamAV Error Messages

ERROR: Can’t initialize the internal logger =

  • This means another clamd process is running that has locked the clamd.log file. Or permissions on the file or its directory do not allow clamd to read and/or write to the file. Check to see if you have an additional third party clamd running on your system and remove it. Check the permissions on the log files and directory. IF you have not changed the defaults of clamd, then the permissions should look like this:

    drwxr-xr-x  2 clamav clamav    4096 Jun 26 04:23 .
drwxr-xr-x 23 root   root      4096 Jul  1 04:37 ..
-rw-r--r--  1 root   root         0 Jun 26 04:23 clamd.log
-rw-r-----  1 root   root    139593 Jul  1 17:06 freshclam.log

Heuristics.Structured.CreditCardNumber

  • This is caused when the user enables the CLAMAV_StructuredDataDetection option. This means that clamd has detected data in a file that appears to be a credit card.

LibClamAV Warning: RWX mapping denied

  • This means that you are eithe running a very out of date versions of clamav or someone or something has replaced clamav on your system with either an old, or very crippled version of clamav. You will need to ensure that your system is up to date, and that any third party versions of clamav have been removed.

LibClamAV Warning: Bytecode: disabling JIT because PaX is preventing ‘mprotect’ access.

  • This means that you are eithe running a very out of date versions of clamav or someone or something has replaced clamav on your system with either an old, or very crippled version of clamav. You will need to ensure that your system is up to date, and that any third party versions of clamav have been removed. Do not follow the advice of this error message, this will open your system up to a root level compromise. This error occurs only when older versions of clamav are installed, or someone has replaced clamav on your system with crippled version. The AP version of clamav does not have this bug, and will not require you to open this root level hole in your system.

/usr/bin/modsec-clamscan.pl is not installed on the server.

  • This means that someone has removed parts of AP. You will need to reinstall AP and remove whatever software is removing parts of AP.

Exec: Execution failed while reading output: /usr/bin/modsec-clamscan.pl (End of file found)

  • This means that someone has either removed clamav, or disabled the TCP service for clamd.

Note

If you are using a third party clamav installation you will need to remove it. AP is only supported with clamav provided with AP

Error: CLAMAV_ENABLE_DAZUKO set to enabled, but Kernel Malware detection module was not detected.

  • This can happen for one of two reasons:

    1. The system is not running the AP Kernel

    2. The system is running hte AP Kernel, but does not have the dazuko package installed.

      • Check to see if the dazuko package is installed by running the following command:

        rpm -qa | grep dazuko

      • If dazuko is installed, you should see output similar to the following:

        dazuko-kmod-common-2.3.9-6.36 kmod-dazuko-2.3.9-6.36 kmod-dazuko-2.6.32.59-24.art.x86_64-2.3.9-5.24

      • If dazuko is NOT installed, you can install it by running the following command:

        yum -y install kmod-dazuko dazuko-kmod-common

clamd[12345]: lstat() failed on:

  • This means that a third party product has changed the systems antimalware configuration. Specifically, its changed the user that clamd runs as. Make sure that anytime you install or upgrade any software that you run awp in scan and fix mode. AP will look for misconfigurations, vulnerability and holes that third party software may introduce in the system. Just run this command as root, and in this case if you have enabled CLAMAV in the AP configuration AP will automatically fix this issue:

    awp -s -f

    If you have installed a third party version of clamav, please remove it and reinstall the AP version of clamav. It has been specifically modified for the advanced needs for AP, and a generic version of clamav will not work properly. You can run this command, as root, to reinstall clamav:

    yum reinstall clamav clamd clamav-db

clamd[12345]: stream(127.0.0.1@1234): Some.Virus.Found

  • Explanation:

    • An application asked clamd to scan a file via the TCP “stream” method, and clamd reported that the file appeared malicious. The “stream” service is a TCP service. It allows an application to send an object to the clamd server via TCP and to ask the object to be analyzed. All that clamd will see if the IP address of the source (in this case localhost). If an application chooses not to log that its done this, theres not much else you can do to look into this. clamd only sees the request.

      Only one AP component uses the streaming method, that is the web upload scanning component. However, if it detects a malicious file it will also log a WAF event 351000 which will tell you the domain and application involved. If you do not see this event, in the AP GUI, it means some other NON-AP component on your system asked for an object to be scanned, and that object has been found to contain malware. You will need to find out what application(s) on your system do this

  • Recommendations:

    • Only use AP components to scan for malware. We recommend that the realtime Anti virus malware protection system in AP be activated. If it is not, then AP will not be able to stop all malware that may be on the system (upload scanners can only scan what they can see, and there may be other ways to upload content to your system, for example, an attacker can just SSH to the system and manually type in the malware and that would bypass any upload scanner, so use the realtime malware protection system!).

      If you are seeing clamd messages such as these, that may mean that the realtime malware protection system is not active on your system. The realtime malware protection system requires that you run the AP kernel, if you are not using the AP kernel then will not be able to use the realtime malware protection system. Please see the Anti_virus page for documentation on enabling the realtime antimalware system.

      With this said, a stream event such as the one in the example above does not mean necessarily mean that the file was on the system. An application can be sent a file, in memory (for example, if you were uploading a file to a web server it may store the file in memory first), and the application can then ask clamd to examine it and based on the response it gets from clamd can then decide what to do with this. clamd can not force the application to do anything with the file, as clamd is neither handling the file nor is is it managing it when the stream method is used. clamd just tells the application using the stream method if the file is, or is not, a malicious piece of software. At this point its up to the application to do something.

      AP, for example will both do something, and it will log that it did it, when the file was, and what was done with it. Its entirely possible if you see these events that the application that requested the file to be scanned rejected the file (which is what would be expected of a rational application, otherwise whats the point), but the application just may not bother to log that it did this.

      The real time malware protection in AP however can and will prevent all malicious software being saved on the system, from being executed, modified, etc. If thats not active, then AP has “one hand tied behind its back” when it comes to malware. Again, we recommend you enable the realtime malware protection system if you are using the AP kernel.

      As mentioned before, with a STREAM event that does not mean that the file was on the system. More often than not it means it was not on the system as most applications that use the stream method use this for uploads, streaming the file to clamd, and if clamd says its virus they will block the upload .

      If you are using a third party application that uses the stream method, and its not logging anything else (such as that it blocked the file somehow), then we recommend you scan your system. If you do not find any malware then you have an application thats scanning things, but isnt logging what it did if it found a virus.

Cannot re-connect Clamd

  • This error can occur if the clamd daemon is not running, if it has been removed by a third party product, if it has been incorrectly configured by a third party, or if local firewall rules prevent connections to clamds.

    Various applications on the system communication with clamd (freshclam, FTP, apache, etc.). If they can not communicate with clamd, for example if clamd is not running, not listening on a socket or TCP port, or if a firewall is blocking it, the application will generate a connection error. This errors means that one or more of these conditions exists on your system. AP can not create this condition, so if this has happened on your system you may need to perform some troubleshooting to determine the cause so you can implement the correct fix.

    Step 1) Check to make sure you have AP configured with clamd enabled. The correct setting in the AP Web Console CLAMAV_ENABLED=yes

    Step 2) If this setting is set to yes, then check to make sure clamd is configured to listen on an IP address that the application can access. By default this is set to 127.0.0.1. We do not recommend you change this, but if you have setup your interfaces in a non-standard manner you may need to do this.

    Step 3) Also check to make sure you do not have any third party installations of clamd or clamav components. These may conflict with the AP installed clamav components. You only need one installation of the clamav components, and AP will install and manage all of these components for you. If you have installed third party clamav components you may also have a corrupt installation. Please remove any third party clamav components. If you have used a package managed copy of clamv, you may be able to remove the third party installation by finding the packages name and using “yum remove packagename” to remove the package. Please contact the vendor for the third party clamav installation for assistance with removing their packages.

    If you have a source installation of ClamAV, then you will need to contact the third party vendor for assistance with removing their software.

    Once you have removed any third party ClamAV instllations reinstall the AP ClamAV component by running the following command:

    yum reinstall clamv clamd clamav-db

    Step 4) Check to make sure the clamd service is running

    ps auxwww | grep clamd

    Step 5) If you still get a connection refused error, then check your firewall rules to ensure that freshclam can communicate with clamd on the system. You will also want to check to make sure that clamd is both listening via TCP and via a socket. If it is not, then you have a non-AP configured copy of clamd running. Run this command to try to fix this non-AP configuration:

    awp -s -f

This version of the ClamAV engine is outdated.

  • First, don’t panic, this is not an error and nothing is wrong with your system. This just means that a newer version of the clamav engine is available, and this message means that the version of clamav engine you have installed is not the latest. The simple solution is to upgrade to the latest clamav release, vy running this command as the root user:

    yum upgrade clamav

    Sometimes we may not release a clamav update immediately to do some additional testing, if your system will not upgrade to the latest clamav, and you have an active license, please be patient and we will release a new engine when we believe it is stable. If you wish you skip the stable releases, just install testing yum repository.

/usr/bin/modsec-clamscan.pl (The timeout specified has expired)

  • This means that the clamd daemon is either not running, or a response from the daemon has timed out. The later can occur if the system is heavily overloaded to such an extent that the clamd daemon is unable to process the request quickly enough.

    Step 1) Run the following command as root

    ps auxwww | grep clamd

    Step 2) If clamd is running, you should see something similar to this

    root 29859 0.0 6.8 471680 419364 ? Ssl 12:28 0:00 clamd

    Step 3) If clamd is not running, start clamd with this command as root

    /etc/init.d/clamd start

  • Check the load, CPU usage and memory available on the system at the time the event occurred. If the system was overloaded, add additional resources to compensate for system load.

LibClamAV Warning: Detected duplicate databases /var/clamav/main.cvd and /var/clamav/main.cld, please manually remove one of them

  • This can occur if an update is interrupted and clamav does not have a chance to clean up after itself.

    Remove either the /var/clamav/main.cvd or /var/clamav/main.cld file.

    You can also remove both, if you do that run the command “freshclam” as root after you do that.

FATAL: Error inserting dazuko

  1. If you are using the AP Kernel

    This is a harmless error. It simply means that AP is attempting to insert the module, which is already inserted. You can confirm if the module is already inserted by running the following command as root:

    lsmod | grep dazuko

    If you see output similar to this:

    dazuko                 34523  3
redirfs                57062  1 dazuko,[permanent]

    You can ignore this error. If you do not see output similar to this, and instead get no output from this command it means the module is not loaded. You will need to reboot your system (and ensure you are rebooting in the AP kernel) to force the module to load

  2. If you are not using the AP Kernel

    Non-AP kernels do not support the realtime anti-malware system.

ERROR: Clamuko: Dazuko -> Can’t include path

  • This generally occurs if you are not running the AP Kernel, and you do not have the dazuko kernel module installed.

  • Solution:

    Step 1: First ensure that you are running the AP kernel.

    Step 2: Ensure that the dazuko module is loaded by running the following command as root

    lsmod | grep dazuko

    Step 3: If dazuko is installed, you will see output similar to this

    dazuko                 34523  3
redirfs                57062  1 dazuko,[permanent]

    Step 4: You must be running the AP Kernel to use the dazuko module.

    Step 5: Please follow the instructions on the ClamAV configuration page to use the dazuko module. Once the module is installed, you will likely need to reboot yor system.

    • This means that clamd is not running as root.

    • The directory does not exist

    • SELinux is enabled.

ProFTP Errors

CRITICAL:3:a programming/runtime error on proftp upgrade

  • The error looks like the following:

    psa-proftpd-1.3.3e-1.el5.art.x86_64.rpm       | 2.0 MB     00:04
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : psa-proftpd                                   1/1
CRITICAL:3:a programming/runtime error
CRITICAL:3:a programming/runtime error
error: %trigger(psa-triggers-10.10.1-cos5.build1010110207.15.noarch)                         scriptlet failed, exit status 3
error: %trigger(psa-triggers-10.11.0-cos5.build1011110331.11.noarch)                         scriptlet failed, exit status 3

Installed:
  psa-proftpd.x86_64 0:1.3.3e-1.el5.art

    This is not caused by AP. psa-triggers is a piece of software from Parallels company and is not part of psa-proftp and has no adverse impact on the installation of Atomicorps psa-proftp or its upgrade. Please report this bug to Parallels if you have issues with this error and their software.

  • Detailed Explanation:

    • In Linux systems that support RPMs software can be installed to “monitor” changes in other software and then require and force the system to take additional, unrelated and external actions. When that software is installed they can require “triggers” to be executed when that software is changed, upgraded, installed, etc. These triggers are not part of the software, and are not even included in it. Its part of the larger software management system of the operating system, and is something a third party piece of software can require for any piece of software. These triggers are outside the control of the software itself or its authors.

      In this case, when proftp is upgraded, another package (Plesk) has configured the system to require that if the package “psa-proftpd” changes that the “psa-triggers” package must also be run. This is neither requested by proftp, its package or its upgrade. Its completely outside of the package itself. Because of this, in this case proftp is upgraded successfully, however the third party software (Plesk in this case) has a bug that causes an error with the external “psa-triggers” package (and not with proftp or the upgrade). As a result, this does not effect the psa-proftp software, as per the message:

      Installed:
  psa-proftpd.x86_64 0:1.3.3e-1.el5.art

      The software was installed/upgraded successfully.

    • The following error message:

      CRITICAL:3:a programming/runtime error
CRITICAL:3:a programming/runtime error
error: %trigger(psa-triggers-10.10.1-cos5.build1010110207.15.noarch)                         scriptlet failed, exit status 3
error: %trigger(psa-triggers-10.11.0-cos5.build1011110331.11.noarch)                         scriptlet failed, exit status 3

    Is reported from the external and unrelated psa-triggers package from Parallels. This has no effect on the Atomicorp psa-proftp package install, nor does it prevent or effect its installation, upgrade or change. This message is external to the upgrade of proftp and it is not related to any software from Atomicorp.com.

    This is a bug in Plesk, and is not something in AP or proftp, nor is it something the proftp package from Atomicorp requests or executes.

    Report this to the vendor as a bug in their software (in the example above, this is a bug in PSA 10).

Fatal: unknown configuration directive ‘ClamAV’ on line 1 of ‘/etc/proftp-awp.conf’

  • Explanation:

    • This means that you are not using the Atomicorp version of Pro FTP. You are using some other parties version of proftp that does not include malware upload support.

      Some products (such as PSA miroupdates), will install a different version of proftp over the Atomicorp version. These vendors will also not update your rpm repository, so checking your installed packages will also not show this change and will appear to report that the atomicorp version of proftp is install, it is not

      This error is conclusive proof the system is not running the Atomicorp version of psa-proftp.

  • Solution:

    • Reinstall the Atomicorp version of ProFTP

      yum reinstall psa-proftpd psa-proftpd-xinetd

      If you attempt to resinstall our version of proftp, and you get an error from yum similar to this:

      yum reinstall psa-proftpd psa-proftpd-xinetd
Loaded plugins: fastestmirror
Setting up Reinstall Process
Loading mirror speeds from cached hostfile
  * addons: centos.kiewel-online.ch
  * atomic: www7.atomicorp.com
  * base: centos.kiewel-online.ch
  * extras: centos.kiewel-online.ch
  * rpmforge: ftp-stud.fht-esslingen.de
  * updates: centos.kiewel-online.ch
Installed package psa-proftpd-1.3.3c-3.el5.art.x86_64 not available.
Installed package psa-proftpd-xinetd-1.3.3c-3.el5.art.x86_64 not available.
Nothing to do

      This means that you have either configured yum to either exclude you from installing proftp, you have yum priorities setup, you have disabled the awp repository or you have third party repositories also configured on the system that is throwing off yum and not allowing it to install the software it needs.

  • Steps to troubleshoot your yum configuration:

    Step 1: Disable all third party repositories

    Step 2: Remove yum priorities if its installed:

    yum remove yum-priorities

    Step 3: Check your yum configuration to make sure you do not have anything excluded. Exclude entries are sometimes added to /etc/yum.conf or in your /etc/yum.repos.d/ directory, and will look similar to this example:

    exclude=psa-proftp*

    Step 4: Check to make sure the AP respository is enabled. The AP respository is stored in the /etc/yum.repos.d directory in a file titled “awp.repo”

    Step 5: Attempt to reinstall again

    yum reinstall psa-proftpd psa-proftpd-xinetd

    Step 6: If the step above fails, it may be because your ProFTP is out of date, try running this command

    yum upgrade psa-proftpd psa-proftpd-xinetd

Mod_Evasive Errors

cpanel and mod_evasive

  • Currently mod_evasive does not work with cpanel. You may be able to install it manually, but AP will not be able to manage it and this is not a supported configuration. We are working with cpanel to support mod_evasive.

Apache Errors

Error creating rule: Could not add entry 1.2.3.45.6.7.8 in line XX of file /etc/asl/whitelist to IP list

  • This means that there is a bogus entry in your /etc/asl/whitelist file.

[warn] module security2_module is already loaded, skipping

  • This error means that apache has tried to load mod_security twice, and is refusing to do so. This is worth investigating because AP will never try to do this, and this error means that someone or something else has configured apache to do this twice. This could mean that there are two (or more) modsecurity configurations on the system, which will in fact be loaded by apache. Although the module is not loaded, the configuration options will be which could cause all sorts of problems with the system

    Again, AP will never configure the system to do this, so if you see this error it means someone or something else has added another modsecurity configuration to the system which could cause severe performance problems, could disable key settings, or enable other settings that may conflict with the configuration of modsecurity.

    If the module is just configured to load twice, and there are no configuration options then this error may be harmless.

Address already in use: make_sock: could not bind to address [::]:80

  • This is a generic error generated by Apache. This message occurs if Apache (or some other process) is bound to port 80. This error is NOT caused by AP.

In some cases, such as when apache is restarted, the apache init script may not actually shut apache down before starting it. In some other cases, such as if the system is running a processor manager, or a process watchdog, they may restart apache during the restart cycle if Apache takes a long time to stop (the apache “graceful” option may take a long time to shut down all child processes, and this can occur). The process manager may incorrectly (or depending on how you look at the situation, correctly) detect that Apache is “down” and restart it automatically, before the Apache init script can start Apache. This will result in this error which is technically incorrectly.

This can also occur if some other process is listening on port 80. In all cases, when this error message is generated by apache, it means that apache tried to start and tried to bind to port 80 only to find something else (possibly apache) already listening on port 80.

Check to see if apache is already running, by running this command as root:

ps auxwww | grep httpd

If Apache is running, you will see output similar to the following:

root     30070  0.0 11.2 572444 227872 ?       Ss   16:01   0:01 /usr/sbin/httpd
apache   30338  0.0 10.6 445936 215880 ?       S    16:01   0:00 /usr/sbin/httpd
apache   30339  0.2 13.1 652036 266176 ?       S    16:01   0:09 /usr/sbin/httpd
apache   30340  0.1 12.7 633744 256844 ?       S    16:01   0:09 /usr/sbin/httpd
apache   30341  0.2 12.7 641456 257528 ?       S    16:01   0:09 /usr/sbin/httpd
apache   30342  0.2 12.7 641852 257364 ?       S    16:01   0:10 /usr/sbin/httpd
apache   30343  0.2 12.7 635864 256688 ?       S    16:01   0:09 /usr/sbin/httpd
apache   30344  0.2 12.9 644080 261692 ?       S    16:01   0:10 /usr/sbin/httpd
apache   30345  0.2 12.9 644012 261632 ?       S    16:01   0:10 /usr/sbin/httpd
apache   30346  0.2 12.4 634564 251056 ?       S    16:01   0:12 /usr/sbin/httpd
apache   32012  0.1 12.8 644100 259272 ?       S    16:31   0:04 /usr/sbin/httpd
apache   32013  0.1 12.7 643804 256628 ?       S    16:31   0:04 /usr/sbin/httpd
apache   32014  0.1 12.6 641456 255712 ?       S    16:31   0:04 /usr/sbin/httpd
apache   32015  0.1 12.9 645856 262424 ?       S    16:31   0:05 /usr/sbin/httpd
apache   32016  0.1 11.5 631772 233776 ?       S    16:31   0:04 /usr/sbin/httpd
apache   32017  0.1 12.7 641424 258196 ?       S    16:31   0:04 /usr/sbin/httpd

If you see apache running, you can ignore this error. However, you may want to restart apache again just to make sure whatever changes were applied to apache, to cause it need to be restarted, were applied.

If you do not see apache running, try starting it again. If you continue to get this error, then something else is running on port. You can find out what is listening on port 80 with this command:

netstat -anp | grep :80 | grep LISTEN

The output should look like the following:

tcp 0 0 :::80  :::* LISTEN 30070/httpd

Kernel Errors

kernel: xt_lscan: Warning: Pure RST received

  • If you are using the AP kernel:

    This means that the low level portscan detector in AP has detected a RST packet from a source that does not appear to be associated with an active connection. This can happen for at least two know reasons:

    1. The connection is already torn down and the client is continuing to send RST packets to abort the connection, even though it no longer exists. This can occur with bugging IP stacks on clients.

    2. A system is attempting to port scan the system using RST packets to attempt to get the system to report what ports it has open or closed.

    3. This feature in AP is not enabled by default, and can be disabled in AP via this setting: FW_LOWLEVEL_PORTSCAN

    AP does not shun these types of connections. This messages is just informational.

  • If you are not using the AP kernel:

    This means that the low level portscan detector in AP has detected a RST packet from a source that does not appear to be associated with an active connection. This can happen for at least three known reasons:

    This could be a bug in the implementation of of the low level portscan detection kernel module. Please contact your kernel vendor before reporting this issue to us to rule out a kernel level bug.

    1. The connection is already torn down and the client is continuing to send RST packets to abort the connection, even though it no longer exists. This can occur with bugging IP stacks on clients.

    2. A system is attempting to port scan the system using RST packets to attempt to get the system to report what ports it has open or closed.

    3. This feature in AP is not enabled by default, and can be disabled in AP via this setting: FW_LOWLEVEL_PORTSCAN

    AP does not shun these types of connections. This messages is just informational.

FATAL: Module ip_set not found.

  • This means you are either using a third party kernel, or an older kernel that does not include the ip_set kernel module. Please ensure you are using the secure AP kernel to use this feature, or contact your kernel vendor for assistance with this module.

Note

If you do not have this module your system will use the legacy iptables firewall system for large firewall sets, such as blacklists, geoblocking lists, RBLs and other large firewall lists.

/proc/sys/net/ipv6/conf/all/accept_redirects: No such file or directory

  • This means you are not using the AP and your third party kernel is either very old, does not support IPv6, or does not understand how to disable redirects. Please contact your kernel vendor for assistance as this error is only caused by the use of third party kernels.

    In general this harmless, it simply means that AP can not disable redirects on your system.

What do the PAX alerts for software in /usr/libexec/paxtest mean?

  • These messsages are benign, and you can ignore them:

    If you are running the AP Kernel:

    PAX: execution attempt in: <anonymous mapping>, 53181000-53184000 53181000
PAX: terminating task: /usr/libexec/paxtest/anonmap(anonmap):1234, uid/euid: 0/0, PC: 53181000, SP: 23498723984
PAX: bytes at PC: c3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
PAX: bytes at SP-4: 12345465682347509817324059871340598734

    If you are not running an AP Kernel:

    kernel: anonmap[12345]: segfault at 00002aaaaaaab000 rip 00002aaaaaaab000 rsp 00007fff457153f8 error 15

  • Discussion:

    If you see these log messages for any of these programs:

    /usr/libexec/paxtest/anonmap
/usr/libexec/paxtest/execbss
/usr/libexec/paxtest/execdata
/usr/libexec/paxtest/execheap
/usr/libexec/paxtest/execstack
/usr/libexec/paxtest/mprotanon
/usr/libexec/paxtest/mprotbss
/usr/libexec/paxtest/mprotdata
/usr/libexec/paxtest/mprotheap
/usr/libexec/paxtest/mprotshbss
/usr/libexec/paxtest/mprotshdata
/usr/libexec/paxtest/mprotstack
/usr/libexec/paxtest/shlibbss
/usr/libexec/paxtest/shlibdata

    These are caused as part of AP’s built in kernel vulnerability scanner. These messages in syslog are normal and harmless and you can ignore them. They mean the vulnerability scanner is working correctly. These messages do not cause any harm to the system, and are perfectly safe.

    If you are running an AP kernel you are immune to the vulnerabilities the scanner will test for and the “PAX:” messages indicate that AP is working normally and safely.

    If you are not running an AP kernel you will not see the PAX: messages, which means you are vulnerable to some of these tests. The AP GUI will report the specific vulnerabilities, you can also get a report from the command line by running this command as root:

    awp -s

Can I suppress these events so they are not logged?

  • No. These are generated by the kernel itself, and suppressing these events would mean that you wouldnt be notified of an actual attack on your system either. These messages mean that the kernel is working correctly, and this is an important part of the health management system in AP.

kernel: AP_INVALID_INPUT

  • This message simply means that a packet came in that the kernels firewall has detected as a packet not associated with a valid stateful connection. Most often this happens when a packet comes in “late” and the session is either already torn down, or the session has “moved past” this point and does not need this packet and the packet is out of sequence. These messages can generally be ignored, as they are simply the byproduct of the imperfect nature of IP.

    Generally you will see them for cases where a session is shutting down:

    Nov 28 19:11:44 host kernel: AP_INVALID_INPUT IN=eth0 OUT= MAC=<snip> SRC=1.2.3.4 DST=5.6.7.8 LEN=40 TOS=0x00 PREC=0x00 TTL=51 ID=51535 DF PROTO=TCP SPT=59190 DPT=30000 WINDOW=0 RES=0x00 RST URGP=0

    (this is a “reset” packet, telling the server the client is killing the session, but the session is already gone so the kernel doesnt need this packet)

    You may see them for other cases, where a session got our of sync.

    AP can log these packets just in case something more serious is occurring, where an attacker is trying to trick the stateful firewall. You do not need to take any action if you see these messages, these just mean that the stateful firewall is ignoring these packets and if this were an attack, it is being blocked. You can also ignore this for legitimate connections, as AP will not shun on these types of events. Non-malicious traffic is far too common, and we do not recommend you ever shun on these types of events.

denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0

  • The kernel is just reporting that an application tried to core dump, and that you have your system configured to prevent core dumps. When an application core dumps that means it died and its trying to record its state for debugging.

    AP does not configure or enforce this, its just reporting an event. Non-AP kernels do not have the ability to report core dumps. This information can help you to debug a potential bug in an application, or it may indicate that someone is trying to compromise your system by crashing an application.

    Please contact your OS vendor for instructions about how to allow applications to core dump if you want to analyze the core dump to see why the application died. We have also put together a courtesy wiki Apache article which explains how to configure your system to allow core dumps and how to analyze them for your system. Please contact your application vendor with specific questions about why a particular application may have died and generated a core dump

denied resource overstep by requesting 12345 for RLIMIT_*

  • This means that your operating system has a limit configured for something, and your system has exceeded this limit.

    AP does not cause this, the AP kernel also does not cause this. AP neither sets nor enforce these limits, it just reports when the operating system has been configured with a limit, when the limit has been exceeded, and if the operating system has prevented an application from exceeding a limit.

    Normal Linux kernels do not report when limits are exceeded. This special reporting capability in AP makes sure you know when your operating system is enforcing one of these limits. With other kernels, you will have no notice that your OS has preventing something working because you have exceeded a limit on your operating system. However, regardless of what kernel you are using your operating system will still enforce whatever limits you have configured it to enforce, regular kernels just wont tell you that you have exceeded some limit configured in your OS.

    Please contact your OS vendor for instructions about how to configure your systems limits. You may also need to contact your application vendor with specific questions about why a particular limit is inappropriate for your application, and what they recommend you set the limit to.

grsec: banning user with uid xx until system restart for suspicious kernel crash

  • This means a process did something very bad to the kernel and could be an attack on your system.

    This message is generated via a special exploit protection feature in the AP kernel. When a kernel protection alert is triggered due to highly suspicious activity in the kernel (from KERNEXEC/UDEREF/USERCOPY, not just any event) or an OOPs occurs due to bad memory accesses, instead of just terminating the offending process (and potentially allowing a subsequent exploit from the same user), the kernel will take one of two actions:

    • If the user was root, this will lock out the system - if this is happening with root, very bad things are happening on the system.

    • If the user was non-root, AP will log the attempt, terminate all processes owned by the user, then prevent them from creating any new processes until the system is restarted.

    This deters repeated kernel exploitation/bruteforcing attempts and is useful for later forensics

kernel: grsec: chdir to

  • If you are using the AP Kernel, the message above means you have enabled the AUDIT_CHDIR setting in AP.

Note

This option is enabled by default and we do NOT recommend you disabling it.

  • If you are using a third party kernel, please contact the vendor for assistance.

grsec: From xx.xxx.xxx.xxx: use of CAP_SYS_ADMIN in chroot denied

  • This means that an application within a chroot tried to use the SYS_ADMIN linux capability. This means the application is trying to do things that would allow the application to escape the chroot. Denying applications in a chroot this capability prevents them from escaping the chroot and prevents them from compromising the system. Allowing them to have this capability means that the chroot essentially does not exist, as the application can just request root level privileges it wants, and can simply move outside the chroot.

    Allowing applications within a chroot to have this capability is extremely dangerous and should not be granted.

    However, if you need to open this hole in your system please disable the CHROOT_CAPS setting in AP.

grsec: denied untrusted exec

  • The application is not being allowed to run because it is insecurely configured and AP has been configured to prevent insecurely configured applications from running.

    Specifically, AP includes a feature called Trusted Path Execution or TPE for short. When Trusted Path Execution (TPE) is enabled (which is the default), it will only allow system applications owned by root to run on the system and only if they are not modifable by any other user except root. This creates a “trusted” set of software, and protects the system from malicious software, malicious uploads, and local attacks such as via web servers, including those that may change applications on the system or replace them with malicious versions. We do not recommend you disable TPE.

    If you get this message, that means that either the application is owned by an untrusted user or its directories are writeable by untrusted users (who could then replace the software in that directory, even though they may not own it). This protection helps to prevent untrusted users from uploading malware, rootkits, shells and other malicious software to the system. This also prevents untrusted users from changing applications, replacing them with a trojan, mobidying systems paths or otherwise tampering with these. By default the only trusted user on the system is root. If you have an application (or an application in a directory) owned by an untrusted user TPE will not allow it to be executed. You can tune AP or the application to allow it to run by using one of the four documented methods below. They are in order or most secure to least secure:

  • Option 1: Change the ownership of the file and its directory, and parent directories to root.root and make sure the file and directory it is in are not writable by other users (including the root group). Here is an example you will need to adapt to your system:

    Note

    If you do not know what is meant by changing a files ownership to root.root, please contact support and we will put a quote together to perform these services for you. root.root means the user and group must be root. These are example commands to achieve this:

    chown root.root /path/to/directory/of/application
chown root.root /path/to/application
chown root.root /path/to/
chown root.root /path/
chroot root.root /
chmod -R og-w /path/to/application

    Always make sure the parent directories are also owned by root:root. This is the most common issue, some broken applications are known to change key operating systems directories ownership from root:root to something else. We’ve even seen cases where someone changed the ownership of / to something other than root. Thats extremely dangerous and something else thankfully will protect the system from.

    For example, here is an application that will not run with TPE in the fully secure mode because the directory is not set to root.root nor is the application:

    drwxrws--- 2 psaadm swkey-data 4096 May 8 02:32 restart
-r-s--x--- 1 root swkey-data 9240 May 4 05:58 plesk-key-handler

    In this example, the directory is owned by psaadm and swkey-data. These are not set to the root user, so the kernel in the fully secure TPE mode will will not trust this application to run because those users could modify, replace or otherwise tamper with this application and are not trusted by the kernel. To get these applications to run (and you will need to discuss this with your application vendor if the application can run in such a secure manner) the directory and application would need to be owned by the root user and the root group, and must be writable only by the root user (not the root group).

    If you can not set an application to run correct as root.root, then you need to configure AP into a less secure mode. This also means that you will need to accept the risk of running your application and system in a less secure and protected mode. To do this, simply configure AP to be less secure using options 2-4. This situation is rare, and only a few vendors seems inclined to run their applications in this less secure manner.

    In some cases, you can work around this by moving the application to a trusted directory and making sure the directory it is in is owned by root.root. For example, for a setuid script owned by “testuser”:

    [testuser2@ac2 ~]$ pwd
/home/testuser2
[testuser2@ac2 ~]$ id
uid=510(testuser2) gid=510(testuser2) groups=510(testuser2)
[testuser2@ac2 ~]$ cat ~testuser/sensitive_file
cat: /home/testuser/sensitive_file: Permission denied
[testuser2@ac2 ~]$ /usr/local/trusted_apps/special_cat ~testuser/sensitive_file
This data can only be read by setuid
[testuser2@ac2 ~]$ ls -al /usr/local/trusted_apps/special_cat
-rwsr-xr-x 1 testuser testuser 23132 Dec 17 19:45 /usr/local/trusted_apps/special_cat

    As you can see from this example, there is a setuid program called special_cat (its a copy of /bin/cat in this example) that is setuid to testuser. testuser2 tried to open the file ~testuser/sensitive_file with “cat”, but can not because testuser2 must use the setuid program “/usr/local/trusted_apps/special_cat” to read those files.

    The key to making this work with the secure AP kernel is to ensure that the directory in which the application is place must be owned by root.root, and the application itself can still be owned by the untrusted user “testuser”:

    [testuser2@ac2 ~]$ ls -al /usr/local/trusted_apps/
total 36
drwxr-xr-x  2 root     root      4096 Dec 17 19:53 .
drwxr-xr-x 12 root     root      4096 Dec 17 19:53 ..
-rwsr-xr-x  1 testuser testuser 23132 Dec 17 19:45 special_cat

    This helps prevent malicious users from uploading programs to the system that you do not want because nothing can be added to that directory except by root. This also helps to prevent users from running programs you don’t trust in ways that can compromise the security of your system, such as with spam bots or tools to compromise your system such as malware. The key is to make sure no one can modify the file except the user that owns it, and that the file is in a directory that only root can modify or place new files in. This helps to prevent path poisoning attacks and uploading of malicious software to your system.

    Again, if you do understand how to do any of these tasks, please contact a qualified systems administrator to do this for you.

  • Option 2: Change APs behavior so that this restriction only applies to untrusted users. You can do that by turning off this feature, called Trusted PAth Exectuion (TPE) so that it only applies to users in the “untrusted” group

    Run the following command:

    echo 0 > /proc/sys/kernel/grsecurity/tpe_restrict_all

    Keep in mind that this is considerably less secure than option 1. This means all the users on your system, except users in the “untrusted” group, will be trusted unless you specifically tell AP not to trust them. You can see a listing of untrusted users by looking at this option in AP: TPE_UNTRUSTED_USERS

    Or you can run the following command as root:

    grep untrusted /etc/group

    The standard list of untrusted users currently is:

    lp,sync,shutdown,halt,mail,news,uucp,operator,games,gopher,ftp,nobody,vcsa,nscd,sshd,rpc,rpcuser,nfsnobody,mailnull,smmsp,pcap,apache,xfs,ntp,mysql,webalizer,named,alias,qmaild,qmaill,qmailp,qmailq,qmailr,qmails,popuser,psaadm,psaftp,mailman

    These are system users, that is users that programs run as. These users normally run programs run as root, and not as programs owned by themselves.

    Note

    You can only change this proc setting (/proc/sys/kernel/grsecurity/tpe_restrict_all) on boot if you have configured AP to lock the kernel (the default setting is to lock the kernel). Once the boot process reaches stage S99 the kernel is locked and you can not change the security settings. So you will need to set this on boot via a custom init script.

    If you want to set an AP kernel setting, such as /proc/sys/kernel/grsecurity/tpe_restrict_all, you will need to create a custom init script such as the following:

    /etc/init.d/awp-custom

    A simple script that would look like the following:

    #!/bin/bash
echo 0 > /proc/sys/kernel/grsecurity/tpe_restrict_all

    Then you will need to link it depending on the runlevel your system is set as. If you do not know what this means, please consult an experienced Linux Administrator. Most systems are set to run at run-level 3.

    ln -s /etc/init.d/awp-custom /etc/rc3.d/S98awp-custom

    init is not part of AP, its part of your operating system, and if you have additional questions about how to configure your operating system or how to write init scripts please contact your operating system vendor. You can read our primer on customizing your kernel in the Installing custom kernel modules with AP wiki article. This includes a primer on init.

    And last but not least, you need to set the script to be executable in Linux. To do that, as root, you need to run this command:

    chmod u+x /etc/init.d/awp-custom

  • Option 3: Remove the user from the untrusted group in /etc/group (we do not recommend you do this).

    If your application is running as an untrusted user, you can change the configuration of AP to trust this user. The default users are system processes such as apache that should NEVER be trusted by the system. This makes it easy for an attacker to upload a trojan to the system. However, if you must run an application in insecure manner as apache then you would remove that user from this group. AP will not be able to protect the system from uploaded trojan by this user.

  • Option 4: Turn off all TPE protections (not recommended and very insecure)

    This option completely disables the TPE system and makes it possible for any user to upload anything and run it on the system. Although AP does have real time malware detection and protection this system is not 100% foolproof and disabling TPE is extremely insecure and will make it possible for an attacker to upload malicious code to your system that even AP may not be able to detect and then run it.

    To disable TPE either change this setting in the AP Web Console: ENABLE_TPE

    And rebot the machine, or change this proc setting to “0”:

    echo 0 > /proc/sys/kernel/grsecurity/tpe

    This also must be done on boot as with option 3.

    If you are unsure of how to do any of these custom things on your system, please contact us.

GRsecurity ACL database: not found [INFO]

  • This simply means you do not have any GRSEC rules set. This is just an information alert and does not mean anything is wrong with your system.

execution attempt in: <anonymous mapping>

  • An Example from the logs:

    Jan 1 12:00:54 hostname kernel: grsec: From 1.2.3.4: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /opt/cpanel/ea-php70/root/usr/bin/lsphp[lsphp:11604] uid/euid:1001/1001 gid/egid:1000/1000, parent /usr/local/lsws/bin/lshttpd.5.0.18[litespeed:11405] uid/euid:99/99 gid/egid:99/99
Jan 1 12:00:54 hostname kernel: PAX: execution attempt in: <anonymous mapping>, 353513ab000-35351426000 353513ab000
Jan 1 12:00:54 hostname kernel: PAX: terminating task: /opt/cpanel/ea-php70/root/usr/bin/lsphp(lsphp):11605, uid/euid: 1001/1001, PC: 00000353513ab010, SP: 000003c83294db38
Jan 1 12:00:54 hostname kernel: PAX: bytes at PC: 41 54 41 55 41 56 41 57 53 48 8b df 48 83 ec 50 48 8b 43 10

    This means this program is attempting to either perform a dangerous operation that could cause your system to be compromised, or someone is trying to break into your system and the AP kernel is preventing this program from being used to compromise your system. This may also occur with malicious applications, applications that are misconfigured such as PHP, or applications that do things in a dangerous way. You can read more about this kernel protection capability in this article Installing

  • Proposed Solutions:

    1. Most Secure: Report this to your vendor that they need to fix the application so it does not need to open this hole in your system. Modern software shouldnt need to do this.

    1. Secure: In some cases it may be possible to tell the application to not try to open this hole in your system. As root, run this command:

    execstack -c /path/to/application_or_library

    If this does not work, your application may require that this hole be opened in the system. Therefore see Option 3 below.

    1. Very Insecure:

      Note

      Always confirm with your application vendor if this is actually normal and necessary for your application before you make this change! Some vendors simply open these holes in your system because they do not know better. PHP for example does not need to open this hole in your system. Turning off this protection for your application will make it vulnerable to attacks which can result in the compromise of the application, and potentially access to your system. Modern applications should not need to open this hole in your system.

    To allow an application to open this hole in your system, run this command as root:

    paxctl -m /path/to/application_or_library

    Then run the following command as root:

    paxctl -m /path/to/application_or_library

denied RWX mmap of

  • This means this program is attempting to either perform a dangerous operation, that could cause your system to be compromised, or someone is trying to break into your system and the AP kernel is preventing this program from doing this.

    This may also occur with malicious applications, or applications that do things in a dangerous way

  • Proposed Solutions:

    1. Most Secure: Report this to your vendor that they need to fix the application so it does not need to open this hole in your system. Modern software shouldnt need to do this.

    1. Secure: In some cases it may be possible to tell the application to not try to open this hole in your system. As root, run this command:

      execstack -c /path/to/application_or_library

    If this does not work, your application may require that this hole be opened in the system. Therefore see Option 3 below.

    If you continue to get the following error message:

    denied RWX mmap of

    This means option 2 did not work, please see option 3 below.

    1. Very Insecure:

      • Always confirm with your application vendor if this is actually normal for your application before you make this change! In many cases its been found that backdoored malicious versions of software will attempt to perform these dangerous operations, and disabling this will allow them to operate. This may be warning you that your software has been replaced by a malicious backdoored copy!

      Turning off this protection for your application will make it vulnerable to stack, heap and other types of overflow attacks which can result in the compromise of the application, and potentially access to your system. If this application is running as root, then this can open your system to a full root compromise. Modern applications should not need to open this hole in your system.

      To allow an application to pen this hole in your system, run this command as root:

      paxctl -m /path/to/application_or_library

      Then run the following command:

      paxctl -m /path/to/application_or_library

does not have a PT_PAX_FLAGS program header, try conversion

  • Run this command to set a PT_PAX_FLAGS header on your application:

    /sbin/paxctl -c /path/to/your/application

    Then run whatever command you were trying to run with paxctl.

mprotect(): 13 (Permission denied)

  • Example:

    01:01:01 host kernel: grsec: From 1.2.3.4: denied RWX mprotect of /lib64/ld-2.5.so by /usr/local/cpanel/3rdparty/php/53/bin/php-cgi[php-cgi:25915] uid/euid:32003/32003 gid/egid:32003/32003, parent /usr/local/cpanel/cpsrvd-ssl[cpsrvd-ssl:25913] uid/euid:32003/32003 gid/egid:32003/32003

  • Discussion:

    This means an application is trying to do something dangerous on your system. Specifically, the AP kernel protects your system by restricting the mprotect() system call which makes it difficult for an attacker to bypass stack protection systems. This makes it impossible for an attacker to change protection of a specific memory region, for example to mark it as executable if it wasn’t originally executable, or to create a new writeable and executable memory mapping using the mmap() call. Without this feature, all the “Stack Protection and “non-executable memory regions” security features used today are more or less useless, as the attacker just change the permissions on your Stack protection to allow them to compromise the system. Unlike other systems, AP protects you from this vulnerability.

    This protection in the AP kernel is critical to making stack protection meaningful. Therefore, if encounter this message, the application has been stopped from doing something very dangerous to your system. It may not be trying to compromise it, but it is making it much easier for an attacker to compromise your system in the future if it were allowed to do this. Therefore, you should carefully consider if you want to allow an application to do this. If you allow an application to do this you are opening your system to stack and heap based attacks through that application.

    It is important then to ensure that your your application absolutely needs this capability, and that if it does and you want to allow it that you can trust the application, and that you are certain that the application is not going to be used by an attacker to compromise your system.

    Applications that work with untrusted data, such as scanners and servers shouldn’t be allowed to do this unless you know that they have no other vulnerabilities associated with this issue.

  • Proposed Solutions:

    1. Most Secure: Fix the application so it does not need to open this hole in your system.

    1. Secure: Remove the flag from the library that is trying to open this hole in your system. As root, run this command:

      execstack -c /path/to/application_or_library

    1. Very Insecure: Always confirm with your application vendor if this is actually normal for your application before you make this change! In many cases its been found that backdoored malicious versions of software will attempt to perform these dangerous operations, and disabling this will allow them to operate. This may be warning you that your software has been replaced by a malicious copy.

    To allow an application to open this hole in your system, run this command as root:

    paxctl -m /path/to/application_or_library

  • Note on Vulnerabilities:

    In some cases, such as with PHP, you may need to use strace to find the specific library that is opening this hole in your system as your application may not be trying to open this hole in your system. You can do this by running this command as root:

    strace -fF /usr/local/cpanel/3rdparty/php/53/bin/php-cgi

    Will produce a very verbose output as strace tells your what the application is doing. Once the application is prevented from opening this hole in your system it will stop running. Therefore, you want to look at the final output of the strace, which in the case of the example above will look similar to this:

    open("/usr/local/cpanel/3rdparty/php/53/zendopt/ZendGuardLoader.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\33\3\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1082328, ...}) = 0
mmap(NULL, 2145168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6ad6faf57000
mprotect(0x6ad6fb042000, 1048576, PROT_NONE) = 0
mmap(0x6ad6fb142000, 118784, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xeb000) = 0x6ad6fb142000
mmap(0x6ad6fb15f000, 15248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x6ad6fb15f000
mprotect(0x6ad7081c1000, 3460, PROT_READ|PROT_WRITE) = -1 EACCES (Permission denied)

    Whats important to note is that the library “/usr/local/cpanel/3rdparty/php/53/zendopt/ZendGuardLoader.so” is trying to open a hole in tje system and AP is stopping it:

    To prevent the application from opening this hole in your system, run this command as root:

    execstack -c /usr/local/cpanel/3rdparty/php/53/zendopt/ZendGuardLoader.so

    To allow it to open this hole in your system you will need to tell AP to let this application make your system vulnerable to a compromise via this hole for this application or library. We do not recommend you open this hole. Its rare that an application needs to do this, so we recommend you close it, and then test your application to see if it works properly. If it does work properly you do not need this hole opened in your system.

    Before you allow an application to open this hole, always confirm with your application vendor if this is actually normal for your application before you make this change! In many cases its been found that backdoored malicious versions of software will attempt to perform these dangerous operations, and disabling this will allow them to operate. This may be warning you that your software has been replaced by a malicious copy.

    paxctl -m /usr/local/cpanel/3rdparty/php/53/zendopt/ZendGuardLoader.so

grsec: denied RWX mprotect

  • If you see log messages that involve the AP paxtest application suite, for example:

    grsec: denied RWX mprotect of /usr/libexec/paxtest/shlibtest2.so

    This is harmless.

  • If you see log message like this:

    Jan 1 12:00:00 server kernel: grsec: denied RWX mprotect of /lib64/ld-2.12.so by /usr/local/cpanel/3rdparty/php/53/bin/php-cgi[php-cgi:3726] uid/euid:0/0 gid/egid:0/0, parent /usr/local/cpanel/bin/admin/Cpanel/logaholic[logaholic:3715] uid/euid:0/0 gid/egid:0/0

    This means an application is trying to do something dangerous on your system, and AP is preventing it from doing this. This could be an attempt to compromise your system or simply a very vulnerable or poorly constructed application thats operating in a dangerously insecure manner, which would allow users to do bad things to your system including compromising it. You should report this as a vulnerability to the application developer.

    If you want to allow this application to do this dangerous thing on your system, you can configure AP to allow this. Continue reading if you wish to open this hole AP is alerting you to:

    Explanation:

    Specifically, AP protects you system by restricting the mprotect() system call which makes it difficult for an attacker to bypass kernel stack protection systems used on Linux systems. Normal Linux kernels do not protect against this, so its trivially easy to beat the stack protection systems they use, and attackers know this.

    This protection makes it impossible for an attacker to change protection of a specific memory region, for example to mark it as executable if it wasn’t originally executable, or to create a new writeable and executable memory mapping using the mmap() call. Without this feature, all the “Stack Protection and “non-executable memory regions” security features used today are more or less useless, as the attacker can just change the permissions on your Stack protection to allow them to compromise the system. Unlike other systems, AP protects you from this critical vulnerability.

    This protection in the AP kernel is critical to making stack protection meaningful. Therefore, if you encounter this message, the application has been stopped from doing something very dangerous to your system. It may not be trying to compromise it, but it is making it much easier for an attacker to compromise your system in the future if it were allowed to do this. Therefore, you should carefully consider if you want to allow an application to do this. If you allow an application to do this you are opening your system to stack and heap based attacks through that application and no amount of kernel protection will protect you. It is also not necessary for applications to do this with modern programming techniques, and is rarely seen and never with properly developed applications.

    It is important then to ensure that your your application absolutely needs this capability before you open this hole in your system. If it does and you want to allow iit do this, this means that you are completely trusting the application, and you should only do this if you know the application can never be used to compromise your system.

    Applications that work with untrusted data, such as log processing tools, data scanners (virus and antispam) and application daemons should never be allowed to do this.

kernel: grsec: From x.x.x.x: denied ptrace of

  • Parallels tries to prevent users from debugging certain Parallels products. It does this by trying to ptrace itself to detect a debugger. The AP kernel contains protections to prevent ptrace exploits, and controls which users can use this function call. The kernel will block this function, and the Parallels products does not correctly catch this condition, and instead erroneously reports this as the user running a debugger. This is a bug in Parallels products.

    If you are using a version of Plesk that has this bug, and there is no fix available from Parallels for this bug, please follow this process to disable this protection on your system.

    Note

    This workaround is for those users that want to run Parallels products with this bug. This will require you to disable this protection in your kernel. This will be correctly reported as a vulnerability in the system.

    To disable ptrace exploitation protection:

    1. When running AP 3.0 and up simply log into AP, click on the Configuration tab, and select the “AP Configuration” menu option.

    2. Then scroll down to Kernel

    3. Change the “HARDEN_PTRACE” entry from “yes” to “no”. If you kernel is set to lock itself on boot (this is the default), you will need to reboot your system for this setting to take effect.

    If you are not running AP 3.0, please add this to your/etc/sysctl.conf file:

    kernel.grsecurity.harden_ptrace = 0

    And reboot the system.

grsec: process /usr/bin/foo attached to via ptrace by

  • This is an informational message and can be ignored. AP will log ptrace activity to help you detect any potentially malicious ptrace actions. To disable this logging run this command as root:

    echo 0 > /proc/sys/kernel/grsecurity/audit_ptrace

    You will need to set this to occur on boot, or add this to your/etc/sysctl.conf file:

    kernel.grsecurity.audit_ptrace = 0

    And reboot.

error: unknown error 22 setting key

  • When running sysctl -p this error occurs:

    error: unknown error 22 setting key 'some.sysctl.setting'

  • Explanation:

    By default the AP kernel doesn’t allow allow certain security parameters in the kernel to be modified. This prevents attackers from disabling kernel protections, such as stack protections and allowing the kernel to be modified on the fly. If you wish to modify these security settings, there are two options:

    1. Set these security kernel settings and reboot the system.

    2. Configure AP to allow your kernel to be modified at any time.

    This is NOT recommended. Allowing kernel security settings to be changed on a running system makes it possible for an attacker to disable AP kernel level protections, to install rootkits and to otherwise compromise the security of the system. This is similar to allowing a user to disable antivirus on the system, if a user can disable antivirus so can a virus. Kernel level rootkits are the most difficult type of malware to detect and prevent. Once the kernel is compromised, the entire system is compromised and it may not be possible for you even detect that this as occurred. We highly recommend you set any kernel security settings you need, reboot the system and let AP lock the kernel.

    If you wish to use to make your kernel insecure and to allow these security settings to change on the fly, log into the AP GUI, click on AP Configuration and change the setting ALLOW_kmod_loading to yes. You will then need to reboot the system for this change to take effect. AP will report this, correctly, as a critical vulnerability in the system.

modprobe: FATAL: Could not load /path/to/module No such file or directory

  • If you are not using the AP Kernel: Report this error to your kernel vendor. This error is not caused by AP. AP is just alerting you to a problem with your third party kernel. It means your third party kernel is does not have a component your system needs.

  • If you are using the AP Kernel then this means somone or something has removed parts of the AP Kernel. AP will not remove the kernel or any of it’s components.

    To fix this, you will want to make sure to reinstall the AP Kernel.

modprobe: FATAL: Error inserting somemodule (/path/to/module.ko): Operation not permitted

  • If you are using the AP Kernel:

    Discussion:

    By default the AP kernel will prevent the loading of new kernel modules into the kernel, during runtime, once the system has booted up. It will allow kernel modules to be loaded during boot, but once your system has finished the boot up process, when AP will allow you to load kernel modules, AP will then lock the kernel preventing any new kernel modules from being loaded into the kernel. This is to protect the kernel from kernel level rootkits, as it prevents hackers from modifying your kernel and hiding rootkits and malware on your system. Runtime kernel module loading is very dangerous, isnt something most users need, and its extremely unusual to require on a server. We do not recommend you disable this protection.

    Solutions:

    1. Set the modules to load at boot and reboot your system.

    2. Configure AP to allow your kernel to be modified at any time.

    The following is NOT recommended:

    Allowing kernel modules to dynamic load and unload on a running system makes it possible for an attacker to install a kernel module rootkit. Kernel level rootkits are the most difficult type of malware to detect and prevent. Once the kernel is compromised, the entire system is compromised and it may not be possible for you even detect that this has occurred or to remove the rootkit. We highly recommend you load the modules you need on boot and let AP lock the kernel. Remember, AP prevents all LKM rootkits if kernel module loading is prevented at runtime.

    If you wish to configure your kernel to allow dynamic module loading on the fly, log into the AP GUI, click on AP Configuration and change the setting ALLOW_kmod_loading to yes. You will then need to reboot the system for this change to take effect. AP will report this, correctly, as a critical vulnerability in the system.

  • If you are not using the AP Kernel:

    If you are using a third party kernel (that is not the AP kernel), then this information is provided as a courtesy. Please contact your kernel vendor for support with this issue.

    Solutions:

    1. This error generally occurs if the system was not rebooted after AP was installed or if the third party kernel prevents kernel module loading. Make sure the AP setting ALLOW_kmod_loading is set to “yes”, then reboot they system.

    2. Some third party kernels include an option to prevent kernel modules from being loaded. To disable this run this command as root:

      echo 0 > /proc/sys/kernel/modules_disabled

    3. Add the following line to /etc/sysctl.conf and then reboot the system

      kernel.modules_disabled=0

Note

If none of the solutions above worked, please contact your OS vendor.

Warning: cannot open /proc/net/dev

  • AP contains additional safeguards to prevent non-privileged users from reading the full proc table. If you wish to allow a user to see the full proc table, simply add the user to the “procread” group in your /etc/group file and this safeguard will be disabled for that user.

    If you do not know how to add users to a group in Linux, or how to edit your /etc/group file, please contact your OS vendor for assistance.

cannot enable executable stack as shared object requires

  • Examples:

    error while loading shared libraries: libcrypto.so.0.9.8: cannot enable executable stack as shared object requires: Permission denied

grsec: From 1.2.3.4: denied marking stack executable as requested by PT_GNU_STACK marking in /usr/NX/lib/libnxcim.so by /usr/NX/bin/nxserver.bin[nxserver.bin:4626] uid/euid:495/495 gid/egid:10038/10038, parent /usr/sbin/sshd[sshd:4622] uid/euid:495/495 gid/egid:10038/10038

  • The errors above mean that you have an application installed with a serious vulnerability.

    Some application developers may configured their applications insecure to use what is referred to as an “executable stack”. An executable stack allows an attacker to inject raw code into your system, bypassing your operating systems entire security model. This is a well known and widely used method of compromising systems completely.

    Configuring an application in this manner dangerously opens your system to full compromise. Very few, if any applications actually require this insecure state to operate, and often times configuring applications in this manner is done by the application developer in error. You can reconfigure these applications to not do this by following the process below.

    The AP kernel protects you from this dangerous mistake by not allowing these applications to configure your system into this extremely insecure condition.

    There are three options for dealing with this critical vulnerability, from most secure to least secure:

    1. Most Secure: AP will not allow these applications to punch holes in your system, and running the command below will remove that label from the application so it will run properly and securely.

      execstack -c /path/to/application

    Sometimes vendors will do this to the libraries the application uses, and not the binary calling it. If you see a log message with a “.so” in it, that indicates a library has been insecurely configured by your vendor. Here is an example:

    'error while loading shared libraries: libcrypto.so.0.9.8: cannot enable executable stack as shared object requires: Permission denied'

    In which case you may have to remove this vulnerablity from the library itself. You can do this by using the “execstack” comannd and the “-c” switch, which stands for “clear”. This command will remove the setting, or clear it, from the application that is causing this critical vulnerability. For example:

    execstack -c  /opt/splunk/lib/libcrypto.so.0.9.8

    If you aren’t sure which binary or library has the stack execute bit set, you can run this command to query the file:

    execstack -q /opt/splunk/lib/libcrypto.so.0.9.8

    If you see the following output:

    - /lib/libcrypto.so.0.9.8e

    The “-” means that file is safe, and does not have the stack execute bit set, in which case the binary calling the library is the likely culprit and you would want to query that file and remove the flag from it.

    If you see the following output:

    X /opt/splunk/lib/libcrypto.so.0.9.8

    The X means stack execution is allowed, which also means X marks the spot for attacking your stack - which AP will not allow. To close the vulnerability in the above example, remove the stack execute flag from the library by running this command as root:

    execstack -c /opt/splunk/lib/libcrypto.so.0.9.8

    We also recommend you contact your application vendor and ask them to remove these flags from their applications. They are not necessary with modern Linux kernels, and are bad and unsafe practices which will expose your system to compromise if you are not running AP. If you are running AP, you will need to reconfigure these applications per the instructions here.

    1. Insecure Option: Some applications require this vulnerable condition during installation. If you are unfortunate enough to have one of those applications, and the application vendor is not willing to fix their application, reboot the system into a non-secure kernel (a non AP kernel) and install your application. Then reboot the system into the AP kernel and follow the instructions in Option 1 to fix the vulnerable applications.

    2. Insecure Option (Not recommended): It is possible to disable all stack protection in the AP kernel so that it will not be enforced by default and only on executables marked explicitly. No executables are marked in this manner by AP by default, as by default AP automatically protects all executables. If you wish to use this option you will need to mark the executables you wish to protect. Soft mode can be activated by using the “pax_softmode=1” kernel command line option on boot. Furthermore you can control various kernel protection features at runtime via the entries in /proc/sys/kernel/pax.

iptables: Unloading modules: iptable_mangle iptable_nat [FAILED]

  • When using the secure kernel in the default and secure mode, modifications to the kernel are not allowed. This includes loading and unloading modules. You can ignore this error.

FATAL: Module xt_lscan not found.

  • This means you are not using the secure AP kernel and your third party kernel does not include the lowlevel portscan detection kernel module. Please contact your kernel vendor for assistance, this is not caused by AP. Please use the AP Kernel to resolve this issue, or contact your OS vendor and ask them to include this Linux kernel module.

FATAL: Module xt_psd not found.

  • This means you are not using the secure AP kernel and your third party kernel does not include the lowlevel portscan detection kernel module. Please contact your kernel vendor for assistance, this is not caused by AP. Please use the AP Kernel to resolve this issue, or contact your OS vendor and ask them to include this Linux kernel module.

kernel: AP Active Response

  • This is a normal message that AP will log if an IP addressed has been shunned, or firewalled off by AP. This will occur if you have configured AP’s Active Response to “yes”, and an event is triggered that exceeds the minimum event level threshold you have configured. Once that occurs AP will log if any additional network traffic comes from that host, and it will limit this to one log message per minute for that IP address to prevent flooding of the systems logs.

grsec: denied kernel module auto-load of net-pf-10

  • Thats not an error, that means something on your system is trying to enable IPv6, you have IPv6 disabled on boot and AP is configured to prevent modifications to the kernel after boot. By default the AP kernel doesn’t allow loading kernel modules at runtime to protect the system from kernel level rootkits. Once your system finished the boot up process AP will lock the kernel preventing hackers from modifying your kernel and hiding malware on your system. There are four options to resolve this error.

    Option 1: Ignore the message - it is harmless and no harm will come to your system from this message

    Option 2: If you do not want to enable IPv6, find the application and disable whatever in the application is trying to enable IPv6 dynamically

    Option 3: Enable IPV6

    Option 4: Configure AP to allow your kernel to be modified at any time.

  • Options 2 and 3 you will need to discuss with your OS vendor, those are not part of AP and are not supported by us.

  • Option 4, this option is NOT recommended. Allowing kernel modules to dynamic load and unload on a running system makes it possible for an attacker to install a kernel module rootkit. Kernel level rootkits are the most difficult type of malware to detect, remove and prevent. Once the kernel is compromised, the entire system is compromised and it may not be possible for you to even detect that this as occurred. We highly recommend you load the modules you need on boot and let AP lock the kernel. Remember, AP prevents all LKM rootkits if module loading is prevented at runtime.

    If you wish to use to make your kernel insecure and to allow dynamic module loading on the fly, log into the AP GUI, click on AP Configuration and change the setting ALLOW_kmod_loading to yes. You will then need to reboot the system for this change to take effect. AP will report this as critical vulnerability in the system.

kernel: grsec: denied resource overstep by requesting

  • The kernel is reporting that you have configured your system to deny a resource change in an application. The AP kernel does not cause this, it just reports it. A non-AP kernel, such as the default kernel used in most versions of Linux, does not have the ability to report when this occurs (the application just silently fails without any log message).

denied modification of module state

  • This means that you have configured AP to protect your kernel from any changes. This is the default setting for AP, and prevents attackers from inserting malicious code into your kernel. Dynamic modifications to running kernels are dangerous, and many operating system prevent this now (such as 64 bit Windows, and Windows Server.)

denied use of iopl() by

  • AP protects your system from compromise by disabling certain functions in the kernel that could be used to bypass the normal security protocols in a Linux kernel. If you get this message, it means either someone is trying to compromise your system, or if you know this is normal behavior for a trusted application that it needs privileged I/O access. You can disable this protection by changing the kernel level option “disable_priv_io” from 1 to 0. You can do this by logging into the AP GUI, click on the Configuration tab, then select AP configuration. Scroll down to DISABLE_PRIVILEGED_IO and set this to “no”. Then press the update button.

    If AP is configured to prevent changes to the kernel during run time, that is the setting ALLOW_kmod_loading is set to no (this is the default, and recommended setting), then you will need to reboot the system for this change to be applied to the system. We do not recommend you change ALLOW_kmod_loading to yes, that will open your system up to kernel level rootkits.

  • You can resolve this error by appending the following to /etc/sysctl.conf

    kernel.grsecurity.disable_priv_io = 0

    Then reboot the system

grsec: denied open of /dev/port

  • AP protects your system from compromise by disabling certain functions in the kernel that could be used to bypass the normal security protocols in a Linux kernel. If you get this message, it means either someone is trying to compromise your system, or if you know this is normal behavior for a trusted application that it needs privileged I/O access. You can disable this protection by changing the kernel level option “disable_priv_io” from 1 to 0. You can do this by appending this to /etc/sysctl.conf, and rebooting your system:

    kernel.grsecurity.disable_priv_io = 0

MySQL Errors

AP: error while loading shared libraries: libmysqlclient.so.16: cannot open shared object file: No such file or directory

  • This means that a third product, that does not use modern software management techniques, such as package or dependency management, has removed key mysql libraries from your system. AP can not and will not do this, and your operating system will also help to prevent this from occurring. If this has happened to your system, that means you have some software package on your system that has a serious bug or simply does not use modern software management techniques common to all Linux software. We recommend you report this serious bug to the software vendor that caused this to occur to your system. AP can not, and will not do this to your system.

    To fix your operating system you will need to reinstall these libraries. The following command, when run as the root user, may resolve this issue, however this error means that a third product has broken your system and its possible this third party software may have broken other parts of your OS:

    RHEL/Centos 5:

    yum --disableexcludes=all reinstall mysqlclient15

    RHEL/Centos 6:

    yum --disableexcludes=all reinstall mysqlclient16

Failed to connect to database ‘(2002) Connection timed out

  • See the article “Can’t connect to MySQL server on ‘127.0.0.1’” below.

FATAL init Could not establish mysql connection

  • See the article “Can’t connect to MySQL server on ‘127.0.0.1’” below.

Can’t connect to MySQL server on ‘127.0.0.1’

  • This means that OSSEC can not connect to mysql on your system. OSSEC, a component in AP, uses TCP to connect to your mysql server. A number of different things may be the root cause of this error. This checklist provides solutions to the most common causes.

    Step 1: Check to make sure mysql is running, and that its listening on IP address 127.0.0.1.

    Check to see if MySQL is running, by running the following command:

    ps auxwww | grep mysql

    You should see output similar to the following:

    root     17813  0.0  0.0  65988   852 ?        S    Mar19   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock
--pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
mysql    17930  0.5  6.5 551972 264788 ?       Sl   Mar19 116:55 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql
--log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock

    If you do not see a result similar to this, mysql is not running on your system. Start mysql. If you require assistance with mysql, please contact your Operating System or database vendor. If you have neither, please post in the community support section of our forums.

    Also check to make sure mysql is listening on TCP port 3306 and IP address 127.0.0.1:

    netstat -anp | grep 3306

    You should see output similar to the following:

    tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      5543/mysqld

    Or this:

    tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN     5543/mysqld

    If you do not see either 0.0.0.0 or 127.0.0.1 in the fourth column then you may have mysql configured to listen on a specific IP address. You have two choices:

    Option 1: Configure MySQL to listen on all addressed

    • Remove any line in /etc/my.cnf that contains “bind-address” and restart MySQL by running the following command:

      /etc/init.d/mysqld restart

    Option 2: Configure AP to connect to MySQL on a differnt IP address

    • Log into AP, click on AP Configuration, scroll down to “OSSEC_DATABASE_SERVER” and change “127.0.0.1” or “localhost” to the IP address of your mysql server.

    Step 2: Make sure you dont have “skip-networking” in you /etc/my.cnf file.

    Check for the line “skip-networking” in your /etc/my.cnf file by running the following command:

    grep skip-networking /etc/my.cnf

    If you have this line, remove from /etc/my.cnf and save the file.

    Restart MySQL and OSSEC, by running the following commands:

    /etc/init.d/mysqld restart
/etc/init.d/ossec-hids restart

    Now check to make sure MySQL is listening by running the following command:

    netstat -anp | grep 3306

    A correctly configured MySQL will look similar to the following output:

    tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      5543/mysqld

    Or this:

    tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN     5543/mysqld

    If you do not see “tcp” in the first column, and LISTEN in the fifth then MySQL is not configured to listen for TCP connections. You may see output like the following:

    unix  2      [ ACC ]     STREAM     LISTENING     38593710 /var/lib/mysql/mysql.sock

    Step 3: check to make sure mysql is listening on port 3306 and IP address 127.0.0.1:

    A final test to see if mysql is listening on loopback is to telnet, from the system that mysql is running on (not from a remote system), to the loopback interface:

    telnet localhost 3306

    You should see something like this:

    Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
4
5.0.77>

    If you can not connect to mysql, check to make sure that you do not have any lines in /etc/my.cnf that contain “bind-address”. If you do:

    Option 1: Configure mysql to listen on all addressed

    • Remove any line in /etc/my.cnf that contains “bind-address”, and restart mysql by running the following command:

      /etc/init.d/mysqld restart

    Option 2: Configure AP to connect to mysql on a different IP address

    • Log into AP, click on AP Configuration, scroll down to “OSSEC_DATABASE_SERVER” and change “127.0.0.1” or “localhost” to the IP address of your mysql server.

    Step 4: If mysql is listening on port 3306 and on 127.0.0.1, and you still can not connect to it on loopback (127.0.0.1)

    • This is most likely caused by local firewall rules. Disable your firewall to see if you can connect to mysql, run the following command:

      /etc/init.d/iptables stop

    If you can connect to mysql, then your firewall rules need to be adjusted to allow connections to mysql on localhost. Please see the AP Firewall documentation.

MariaDB reports a table is “is marked as crashed and should be repaired”

  • This error is not caused by AP. That means MariaDB was not shut down cleanly or that it crashed. You may have other crashed databases. See the article below for MySQL for instructions to repair crashed databases.

MYSQL reports a table is “is marked as crashed and should be repaired”

  • This error is not caused by AP. That means mysql was not shut down cleanly or that it crashed. You may have other crashed databases.

    Option 1:

    • If this involves one of APs tables, we have added in logic to the database rotation system to try to fix crashed tables. This method is not fool proof.

      As root, run the following command:

      /var/awp/bin/awp_db_rotate

      That will trigger the message from MySQL that the table is crashed:

      AP DB Rotate: Locking failed - Table './tortix/data' is marked as crashed and should be repaired

      That will then trigger a self healing event in AP to fix the table, if possible. Keep in mind that if MySQL was shut down incorrectly or crashed, MySQL tables can be severly corrupted and this Self Healing method may not work. If it does work, you will see an event “60912” in your /var/ossec/logs/self-healing.log log file. You can search for this event with this command:

      grep "tortix.data repair" /var/ossec/logs/self-healing.log

      If the Self Healing event worked, your output will look similar to this:

      Table Op Msg_type Msg_text tortix.data repair note

      If you do not see any events, or if the table is still marked as crashed, then the Self Healing method was not successful because the MySQL table is severely corrupted. See Option 2 below for other methods to repair a corrupt MySQL database.

    Option 2:

    • Sometimes a mysql database may be too corrupt to repair, in which case you will need to re-install the database. To that, run this command as root:

      /var/awp/bin/database-setup

      And answer “y” when the application asks the question “Would you like to re-install the AP database?”

      Then restart the HIDS to ensure it is able to connect to the new database:

      service ossec-hids restart

    Option 3:

OSSEC Errors

OSSEC Directories:

  • /var/ossec/queue/diff/local: This directory is used to store “diffs” of files that have changed, and that the user has configured AP to monitor. “diffs” are special files that record the differences between changes to a file. AP will report these differences when a monitored file has changed. This allows the user to both be notified that a change as occurred, which may not be authorized, and to also see the specific changes that have occurred to the file compared with a previous version.

Authentication key file ‘/etc/client.keys’ not found.

  • This messages is normal and harmless, and simply means OSSEC is setup without any remote clients, which is the default. This feature is not currently supported in AP and will be enabled in a future version.

ERROR: No remote connection configured. Exiting.

  • This messages is normal and harmless, and simply means OSSEC is setup without any remote clients, which is the default. This feature is not currently supported in AP and will be enabled in a future version.

ERROR: Cannot open GeoIP database: ‘/etc/GeoIP.dat’.

  • You can ignore this message. This feature is not currently used and this message is expected when events occur. The message is harmless and you can ignore it.

MySQL server has gone away

  • Please see the FAQ below titled “OSSEC-dbd Reports: Lost connection to MySQL server during query”.

OSSEC-dbd Reports: Lost connection to MySQL server during query

  • This indicates that the timeout parameter for mysql is either too short, or the database server is under considerable load. To increase the timeout period:

    Step 1) Add or modify the “wait_timeout” variable in your /etc/my.cnf file like the following:

    [mysqld]
wait_timeout=28800
interactive_timeout = 28800

    Step 2) Restart MySQL by running the following command:

    /etc/init.d/mysqld restart

ERROR: Queue ‘/queue/alerts/ar’ not accessible: ‘Connection refused’.

  • If this error message is generated when OSSEC is run on a non-Windows platform, you can safely ignore this message. This simply means the Windows AR queue is not available, which is expected with run on a non-Windows system.

ERROR: Unable to connect to active response queue.

  • If this error message is generated when OSSEC is run on a non-Windows platform, you can safely ignore this message. This simply means the Windows AR queue is not available, which is expected with run on a non-Windows system.

Active response command not present: ‘/var/ossec/active-response/bin/restart-ossec.cmd’.

  • If this error message is generated when OSSEC is run on a non-Windows platform, you can safely ignore this message. This simply means the Windows AR command, restart-ossec.cmd, is not installed on the system. Which is expected on non-Windows platforms.

ERROR: Error executing query ‘SELECT id FROM location WHERE name = ‘host->/var/log/mysqld.log’ AND server_id = ‘1’ LIMIT 1’. Error: ‘MySQL server has gone away’.

  • This generally means that mysql is either misconfigured or under heavy load. The most likely scenario is that the maximum number of connections that mysql will allow has been exceeded. Please increase the maximum on your system. You can do that by increasing “max-connections” in your mysql configuration, which is normally stored in /etc/my.cnf.

    Other misconfigurations of mysql can also cause it “go away”, which means that mysql has killed the connection. Check your mysql error logs and contact your OS or database vendor for additional support. A misconfigured database will have other effects on your system, and if AP is unable to communicate successfully with your database your system has serious problems with the database.

    Also, check your mysql logs to make sure you don’t have any errors, a corrupt database can also cause this to occur.

ERROR: Error connecting to database ‘localhost’(ossec): ERROR: Unknown MySQL server host ‘localhost’ (3).

  • This means that your system does not know what the IP address for “localhost” is. This is a default special server name that all systems use to refer to themselves, and the IP address for localhost is 127.0.0.1. Check your systems /etc/hosts file and ensure you have an entry for localhost. Below is an example of a localhost entry:

    127.0.0.1      localhost.localdomain   localhost

    Also check the permissions on your /etc/hosts file to ensure that it is world readable. A correct configured systems /etc/hosts file will have the following permissions:

    -rw-r--r--. 2 root root 866 Mar 31 18:26 /etc/hosts

    If this does not work we recommend you use “127.0.0.1” as this would indicate that something is wrong in your OS that is preventing OSSEC from resolving localhost, and is beyond the scope of what we support.

Command executed: /sbin/service ossec-hids restart

  • This message means that the host intrusion detection system will not start. This generally means one of the following:

    • The HIDS has been shut down manually, and is being automatically restarted by AP

    • The HIDS has been automatically restarted by AP to install updates to the HIDS

    • The HIDS or a part of the HIDS is not running correctly and has been restarted by AP to fix this condition.

    • The HIDS has run into a condition that is preventing it from starting.

    If this condition is continuing to occur, that is you get many alerts within a minute or two of each other, that means AP is not able to restart the HIDS. If you get an email with this alert, and series of these alerts within a minute or two of each other then AP is not able to start the HIDS.

    If you just get a single alert, like this:

    Command executed: /sbin/service ossec-hids restart Exit value: 0 Signal
number: 0 Dumped core?: 0

Shutting down ossec-hids: [ OK ]
Starting ossec-hids: [ OK ]

    And no additional alerts, then everything is fine. AP just needed to restart OSSEC to apply some software updates to OSSEC.

    The most common cause of this condition is when the ossec-dbd process has a problem communicating with the systems database server, or the tables it uses are corrupt or crashed. This means that either the database server (mysql) is not listening on the configured IP address and port, the database is overloaded and is rejecting connections (or is over tuned and is dropping connections), or a table has been corrupted and can no longer be updated or accessed.

  • Proposed Solutions:

    1. If AP is not up-to-date, then run through the AP upgrade proccedure and restart OSSEC by running the following command:

      service ossec-hids restart

    2. Check your MySQL logs first to determine if you have any corrupt databases that require repair by running the following command:

      grep -i error /var/log/mysqld.log

      If there are no errors, check the log below for any error messages:

      /var/ossec/logs/ossec.log

      Then check this FAQ for guidance on those specific errors (just use your browsers search function to look for those messages on this page). In most cases OSSEC will not start because there is a problem communicating with the systems database or the HIDS rules are not up to date.

    1. HIDS is not up-to-date or disabled, run the following commands as root:

      aum -u
awp -s -f

      Make sure the following AP setting is set to “yes”: OSSEC_ENABLED

    1. Firewall rules do not allow connection to the database:

      Temporarily clear your firewall rules by running the following commands:

      service awp-firewall stop
service iptables stop

      If OSSEC can start now, you have configured a firewall rule that is blocking database connections. You will need to remove this rule.

      If OSSEC still can not start, your firewall rules are likely not the cause.

    1. /etc/hosts file is configured incorrectly

      An incorrectly configured /etc/hosts file can cause issues with startups. For example, using a hostname for the database that is not resolving or is not in the /etc/hosts file. For example a missing or incomplete localhost entry in /etc/hosts (localhost should resolve to 127.0.0.1)

    1. Database improperly configured: A non-optimally tuned database that is rejecting connections can cause start up issues. Equally, over tuning mysql can cause it to drop connections. If you believe this is happening to you, please contact a qualified MySQL consultant for assistance configuring your database.

    1. Overloaded MySQL: An overloaded database that is dropping, closing or rejecting connections sporadically. If you believe this to be the case, you will need to either allocate more resources for your database, or remove load.

    1. MySQL not listening on configured IP: A database server that is listening on a different IP address from the one configured for AP can cause start up errors. Check to make sure your MySQL server is listening on port 3306 on the IP address you have configured AP to use as its database server.

ossec-syscheckd: ERROR: Unable to add directory to real time monitoring

  • This error means that the system either:

    • Has run out of file descriptors - This is known to occur on VPS systems, where a limit has been set. Increase the limits on your VPS. Please contact your VPS vendor to determine how to do this with your VPS solution.

    • Does not support inotify - Non AP kernels may not support the modern inotify capability in Linux. This allows files to monitored in real time. Please ensure that your system is running the AP kernel.

    • You have reached the limit for watches on your system - The kernel sets a limit, which can be changed, on the number of real time files that can be monitored. This setting is:

      /proc/sys/fs/inotify/max_user_watches

      In most cases, the default is 8192. Which means the system can watch 8192 files in real time for changes. In practice, that is a lot of files, and should be adequate for the systems key software and configuration files. If you are watching additional directories, in real time, such /home, /var/www and so on, where the system may have tens of thousands of files you will want to increase this number. You can do that either temporarily by changing that value with a command such as this (run as root of course):

      echo 16384 > /proc/sys/fs/inotify/max_user_watches

      Or, you can set this permanently by modifying your systems /etc/sysctl.conf file by adding an entry such as the one below, which increases the limit to 16384:

      fs.inotify.max_user_watches = 16384

      If you do not have a /proc directory or an /etc/sysctl.conf, then this means you are using a virtualization technology that does not allow you to tune the kernel. Please contact your virtualization vendor for assistance with this.

    • The final option is to reduce the number of directories, and therefore files you have AP configured to monitor. Please log into the AP GUI, click on the “AP” tab, then click on “File Integrity”, then select the “Options” button. This will pull up the configuration screen to set which directories or files to monitor. Select the “Directories” button, and configure according to your needs.

ossec-analysisd: INFO: Reading decoder file etc/decoders.d/01-awp-decoder.xml

  • This means that your OSSEC rules are out of date. Please run these commands as root to download the latest rules:

    aum -u
awp -s -f

ERROR: Duplicated decoder with prematch: ‘smtpauth-failed’.

  • This means that your OSSEC rules are out of date. Please run these commands as root to download the latest rules:

    aum -u
awp -s -f

ERROR: Error loading decoder options.

  • This means that your OSSEC rules are out of date. Please run these commands as root to download the latest rules:

    aum -u
awp -s -f

ERROR: Configuration error at ‘etc/decoders.d/01-awp-decoder.xml’. Exiting.

  • This means that your OSSEC rules are out of date. Please run these commands as root to download the latest rules:

    aum -u
awp -s -f

PSMON Errors

Stopping psmon: [FAILED]

  • This means that psmon was either not running it an attempt to restart it was issued, or it was not possible to stop psmon. To see if psmon is running, run this command as root:

    psauxwww | grep "/usr/bin/psmon"

    If psmon is running, you should see output similar to the following:

    root     21851  0.0  0.8 135880  8524 ?        S    17:15   0:00 /usr/bin/perl -w /usr/bin/psmon --daemon --cron
root     21961  0.0  0.0 104744   824 pts/0    S+   17:21   0:00 grep  /usr/bin/psmon

    If you are sure psmon is running, try stopping it manualy and restarting it by running the following commands:

    service psmon stop
service psmon restart

Checking for psmon installation: not installed [FAILED]

  • psmon is not supported with cpanel installations with a locally source compiled PERL. If your system is running cpanel, and you have a locally source compiled PERL you will not be able to use the process monitor. cPanel systems that are using package managed PERL are supported.

Note

This error will only occur if you did not have a package managed PERL installation when AP was installed, or PERL is somehow broken and unable to support PERL packages from rpm. If you have since changed to a package managed PERL, re-run the AP installer to install PSMON.

Command executed: /sbin/service something restart

  • This means that psmon has restarted a process that you have configured your system to start on boot. AP does not configure what services your system will start on boot. AP will check the processes that you have configured to start on boot, and will monitor them to make sure they continue to run. If a service dies, AP will automatically restart it.

  • If you do not want to monitor this service, then you need to disable that process on boot. Run the following command:

    chkconfig --del servicename

    And then reset the AP security policy by running the following command:

    awp -s -f

Apache Errors

httpd: apr_sockaddr_info_get() failed for

  • This error occurs if you do not have your systems IP address and hostname in your /etc/hosts file. Apache requires this to start. Simply add your hostname and IP address to /etc/hosts.

    For Example: If your hostname is www.example.com and your IP is 1.2.3.4 you would add this to /etc/hosts

    1.2.3.4 www.example.com

Note

This eror is not caused by AP

Starting httpd: /usr/sbin/httpd: error while loading shared libraries: libapr-1.so.0: cannot open shared object file: No such file or directory

  • This means someone manually removed the libapr-1.so.0 library from your system. You will need to investigate how this occurred, as AP will not allow itself to be installed on a system that does not report this is installed. If you get this error after installing AP this means your OS believes this library is installed but someone manually removed it without telling your OS.

    You may be able to reinstall this library with these two methods:

    Method 1: Run the following command as root

    yum reinstall apr

    Method 2: Run the following command as root

    yum install apr

CPanel Errors

Error from ssl wrapper: Unable to produce a valid Apache configuration file

[2012-04-05 19:18:33 +1000] warn [ssladmin] Unable to produce a valid Apache configuration file. at bin/ssladmin line 201
[2012-04-05 19:18:33 +1000] warn [cpanel] Cpanel::AdminBin::adminrun(ssl) set error in context ssl
[2012-04-05 19:18:33 +1000] warn [ssl::install] Encountered error in ssl::install: Error from ssl wrapper: Unable to produce a valid Apache
configuration file.

  • The errors above are caused by an insufficient amount of memory being allocated to the CPanel process. The allocated amount of memory may be changed within WHM as follows:

    1. In the menu on the left, click on “Tweak Settings”.

    2. In the main frame, click on the “System” tab.

    3. Increase the value for “Max cPanel process memory”.

    An increase of 64MB should generally be enough, but this may vary depending on your system, the amount of domains you have on the system, etc. If an increase of 64MB is inadequate, keep increasing the limit by 32MB increments.

Segfaults

  • Below is an example of a segfault:

    Jun 8 09:14:16 system kernel: php[5963]: segfault at a801cecc ip a800809b sp bff43d70 error 7 in ld-2.5.so[a8001000+1b000]
23:00:43 system 12 30104 [notice] child pid 26148 exit signal Segmentation fault (11)
exit signal bus error

  • AP gives you more visibility into your systems errors, so in the case of segfaults AP is not actually causing them its just reporting them (which the system didnt actually do very well before AP was installed).

    Segfaults can be indications of a minor or serious problem with a web application, for example, if the segfault was with apache or they can mean that an application died because of a buggy module, or a bug in apache itself. In other cases, a segfault may be caused by a systems configuration. And in still others, it could be an actual hardware problem.

    Its not possible to explain here what may be causing your application to segfault. segfaults are a very generic error, however to help you we have provided this courtesy section to explain these non-AP errors and to provide some cases where we have a good idea of the cause and a solution. In those cases where the issue is generic we provide a guide for tracking the source of a segfault.

  • If you are using the AP Kernel follow the process below:

    Determine what application is causing the segfault. If you are using our secure kernel, it will tell you the exact application that did this, for example if PHP segfaulted you will an error like this:

    Jun 8 09:14:16 system kernel: php[5963]: segfault at a801cecc ip a800809b sp bff43d70 error 7 in ld-2.5.so[a8001000+1b000]

    The kernel is indicating that “php” process 5963 segfaulted, and the library and position in memory where this occurred.

  • If you are NOT using the AP Kernel Your problem is a little more complex, as your system will not log what caused the segfault. Unfortunately, you may have to do some detective work to find out what application is causing the segfault. Please review the final subsection, “apache segfaults” which links to an article that provides generic guidance for tracking down the source of segfaults through core dump analysis.

PHP Segfaults

Note

If you are NOT using the AP Kernel, then AP has nothing to do with the segfault in PHP.

  • The following is an example of a PHP segfault:

    Mar 6 19:25:39 host kernel: grsec: From 1.2.3.4: denied marking stack executable as requested by PT_GNU_STACK marking in /usr/local/cpanel/3rdparty/php/54/zendopt/ZendGuardLoader.so by /usr/local/cpanel/3rdparty/php/54/bin/php-cgi[php-cgi:13019] uid/euid:614/614 gid/egid:615/615, parent /usr/local/cpanel/cpsrvd-ssl[cpsrvd-ssl:13018] uid/euid:614/614 gid/egid:615/615

Jun 8 09:14:16 system kernel: php[5963]: segfault at a801cecc ip a800809b sp bff43d70 error 7 in ld-2.5.so[a8001000+1b000]

Jun 8 09:14:11 system kernel: grsec: From 207.46.13.89: Segmentation fault occurred at a622eecc in /usr/bin/php[php:5952] uid/euid:590/590 gid/egid:590/590, parent /usr/local/apache/bin/httpd[httpd:5684] uid/euid:99/99 gid/egid:99/99 $ /usr/bin/php -v Segmentation fault

  • Some third party PHP modules, such as ioncube and Zend Optimizer are sometimes built in a very insecure manner, that introduces a vulnerability into the system that can lead to full compromise of the server. When they are loaded, they will attempt to open a hole in the system that could lead to a full or partial server compromise. The AP kernel can detect when these applications attempt to open this hole, and will prevent them from doing so. This will cause php to segfault, because these insecurely configured libraries are correctly prevented from opening a hole in your system.

    The solution (described below) is to just set the libraries to not attempt to open this hole, and to report this as a bug to your PHP vendor. These libraries do not need to be configured in this insecure manner to work correctly, it is both unnecessary and introduces a very large hole in your system that attackers can use to partially or fully compromise the entire server.

    AP will try to detect these libraries when they are installed and fix them on installation. With package managed systems this happens automatically.

    However, if you have a customized php install or have installed php from source (as happens with CPanel), then your operating system does not manage this software dynamically and AP can not automatically fix these broken and insecure libraries. You will need to find the insecure third party library that PHP is loading. In most cases this will be the ioncube and zend optimizer libraries, and in a package managed system (Such as Centos and Redhat) they are normally stored in locations such as (this is not a complete list):

    /usr/lib64/php/ioncube/
/usr/lib/php/ioncube/

    On non-package managed systems, such as cpanel systems, where the the package managed versions of PHP have been replaced with locally compiled source builds that location may and often is different. For example, cpanel usually locally compiles PHP from source and places it in locations similar to the list below (these may change, please contact cpanel or the party that build PHP from source for more information):

    /usr/local/IonCube/
/usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/
/usr/local/Zend/
/usr/local/Zend/lib/Guard-5.5.0/php-5.3.x/
(This is not a complete list, you will need to check what external libraries your source built PHP has installed).

    One trick to find these libraries is to use your systems “locate” command. For example, you can use this command to find the ioncube libraries:

    locate -i ioncube | egrep ".so$" | grep lib

    And this one to find the zen libraries:

    locate -i zend | egrep ".so$" | grep lib

  • Proposed Solutions:

    1. Determine if PHP is vulnerable

    There are two conditions in which AP can be associated with a segfault of PHP, and both of these involve conditions when PHP is doing something very dangerous and unnecessary, and AP is preventing PHP from compromising your system. In short, if you are using the secure AP kernel, and you are using an improperly configured version of PHP that is dangerously misconfigured and has a serious vulnerability in it, AP will stop PHP from opening this hole in your system and PHP may generate a segfault. AP will also log the actual vulnerabilities, which is described in more detail below.

    If both of these are true (you are using the AP kernel and you have a dangerously misconfigured PHP) then the AP kernel will detect this dangerous vulnerability in PHP when PHP attempts to open this hole, and the AP kernel will prevent PHP from compromising your system. Run this command to see if PHP is trying to compromise your system with this command, run as root:

    egrep -i "denied rwx|mmap|mprotect" /var/log/messages

    If you see any error messages that say either of the following:

    mprotect(): 13 (Permission denied)

denied RWX mmap of

    Then your system has this dangerous vulnerability, and AP is protecting your from it. We highly recommend you fix this vulnerability.

    If you do not find either the mprotect or mmap error messages occurring on your system check and you have not disabled the option at the link below, then AP is not associated with the cause of your segfaults, and AP is not causing them - RWXMAP_LOGGING

    If you do see these events logged, then your system has this dangerous PHP vulnerability. Move on to step 2 below.

    1. Fix the vulnerability in PHP

    Option 1: Use a package managed PHP

    • This is the easiest and best option. If your control panel company or system administrators are using source built versions of PHP, insist that they do not put execstack on their libraries. Its not needed and its very insecure. Doing so will open your system up to compromise, as as said before there is no need to dangerously misconfigure these libraries in this manner. They work just fine without it.

    Option 2: Run AP in scan and fix mode

    • AP will look for php.ini files in standard locations and will analyze them to attempt to find any libraries that are included via your php.ini. If the insecure libraries are included via php.ini (which is usually the case, but not always especially with third party and source built PHP installs) AP will then remove the unnecessary and highly insecure executable bit flags on those libraries. Just run this command as root to do this:

      awp -s -f

    Option 3: Clear the “executable stack” switch on these libraries manually

    • If AP can not find your php.ini file, or if your libraries are loaded via some non-standard method you will need to manually remove these insecure flags from the libraries. Simply run the following command as root:

      execstack -c /path/to/library

      for example, if you wanted to fix the library “/usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so” you would run this command as root:

      execstack -c /usr/local/Zend/lib/Optimizer-3.3.9/php-5.2.x/ZendOptimizer.so

      Or, if you want to fix the library “/usr/local/IonCube/ioncube_loader_lin_5.2.so”, you would run this command as root:

      execstack -c /usr/local/IonCube/ioncube_loader_lin_5.2.so

      To find these libraries, assuming its ioncube and Zend that are causing this and not some other library, you can use these commands that are typically included with your operating system to typically find them:

      locate -i ioncube | egrep ".so$" | egrep "lib|zendopt"
locate -i zend | egrep ".so$" | egrep "lib|zendopt"
locate -i zend | grep -i optimizer | egrep ".so$"

      If you are using a third party PHP install, as with cpanel, these libraries may be stored in non-standard locations such as:

      /usr/local/cpanel/3rdparty/php/53/zendopt/

      Therefore you may need to use a broader search pattern, and you will need to contact CPanel to see where they may have installed these libraries.

    Option 4: Quick Method

    • This may work, if the vulnerable libraries are ioncube and zend. If you have other vulnerable libraries, you will need to find those libraries and follow the entire process in Option 3. You should also report this as a vulnerability to your PHP vendor.

      execstack -c `locate -i ioncube | egrep ".so$" | grep lib`
execstack -c `locate -i zend | egrep ".so$" | grep lib`
execstack -c `locate -i zend | grep -i optimizer | egrep ".so$"`

    Option 5: Use an Insecure Kernel

    • This is the least secure option, is not necessary, and is recommended. This will leave your system open to root level compromise due to the vulnerability in PHP.

      If you want to do this, you would simply use your the stock insecure kernel that came with your OS.

      A secure kernel, such as the AP kernel, will not allow libraries to do very unsafe things. This protects you from compromise by proactively protecting you from vulnerabilities in applications that do very dangerous things. An insecure kernel will not protect you from these things, and also will allow applications to do dangerous things on your system, and will make it possible an attacker to compromise your system

Tomcat Segfaults

  • Tomcat uses Java, and Java performs certain actions that violate stack protection security models. To allow Tomcat to run in this manner, you simply need to run chpax to configure tomcat and Java to be allowed to do anything they want on your system, and to turn off all security protections for Java. This vulnerability exists on all platforms, with or without AP, if you want to run Java.

    paxctl -mps /path/to/java/bin/java

  • Do the same for Tomcat, for example:

    paxctl -mps /path/to/tomcat

  • You may also need to configure Tomcat suppor tools to run without protection as well, so run the following command:

    paxctl -emrspx /usr/bin/gcj-dbtool

Apache Segfaults

  • These are usually caused by a buggy web application, APR, or an apache module such as an accelerator, or other module. They are not caused by AP. We have put together a courtesy page to assist you.

  • Below is an example of an Apacha segfault:

    Sep 23 01:43:27 srv1 kernel: grsec: From 1.2.3.4: Segmentation fault occurred at 00000063000079bf in /usr/local/apache/bin/httpd[httpd:31167] uid/euid:99/99 gid/egid:99/99, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0

    This generally occurs with systems that do not use a package managed version of apache, or apache compiled from source.

    Some third party apache modules, such as ioncube and Zend Optimizer set their libraries to allow the stack to be executable (a very huge security vulnerability the kernel will prevent, and a setting that is not necessary for this libraries to work). These libraries do not need to be configured in this very insecure state to work. When apache tries to load these libraries the kernel will prevent these libraries from calling a special function, mprotect, in an insecure manner. This will cause apache to segfault.

    AP will try to detect these libraries when it is installed and fix them on installation. With package managed systems this happens automatically.

    However, if you have an apache install compiled from source (as happens with cpanel), then AP can not manage this software dynamically and fix these broken libraries. You will need to find the insecure third party library that apache is loading. In most cases this will be the ioncube and zend optimizer libraries, and in a package managed system (Such as Centos and Redhat) they are normally stored in locations such as (this is not a complete list):

    /usr/lib64/php/ioncube/
/usr/lib/php/ioncube/

  • Proposed Solutions:

    Option 1: Use a package managed by Apache

    • This is the easiest and best option. If your control panel company or system administrators are using source built versions of apache, insist that they do not put execstack on their libraries. Its not needed and its very insecure.

    Option 2: Clear the “executable stack” switch on these libraries.

    • Simply run the following command as root:

      execstack -c /path/to/library