Atomicorp WAF Rules Troubleshooting

Introduction

The follow article outlines various errors you may encounter when using Atomic WAF Rules. If you need further assistance please contact Atomicorp.

Invalid command ‘SecRemoteRulesFailAction’, perhaps misspelled or defined by a module not included in the server configuration

This means that your system is using an out of date version of modsecurity that does not support 2.9.1 and higher rule syntax.

Solutions:

Upgrade to Atomic Protector. Atomic Protector will automatically ensure you have the latest version of modsecurity installed on your system.

Use aum, which will automatically ensure you have the latest version of modsecurity installed on your system.

use our rpms which are built using 2.9.1, and support remote rules.

If you do not wish to upgrade to Atomic Protector, use aum, or use our rpms then you will need to manually upgrade modsecurity to at least version 2.9.1, and enable remote rules support when you compile modsecurity.

Invalid command ‘SecTmpSaveUploadedFiles’

This means that your system is using an out of date version of modsecurity that does not support 2.9.0 and higher rule syntax.

Solutions:

Upgrade to Atomic Protector. Atomic Protector will automatically ensure you have the latest version of modsecurity installed on your system.

Use aum, which will automatically ensure you have the latest version of modsecurity installed on your system.

If you do not wish to upgrade to Atomic Protector, or use aum and can not upgrade to 2.9.0, you will need to remove the ruleset(s) that are generating this error.

Error creating rule: Unknown variable: FILES_TMP_CONTENT

This means that your system is using an out of date version of modsecurity that does not support 2.9.0 and higher rule syntax.

Solutions:

Upgrade to Atomic Protector. Atomic Protector will automatically ensure you have the latest version of modsecurity installed on your system.

If you do not wish to upgrade to Atomic Protector, and can not upgrade to 2.9.0, you will need to remove the ruleset(s) that are generating this error.

Error creating rule: Failed to resolve operator: fuzzyHash

This means that your system is an out of date version of modsecurity that does not support 2.9.0 and higher rule syntax.

Solutions:

Upgrade to Atomic Protector. Atomic Protector will automatically ensure you have the latest version of modsecurity installed on your system.

If you do not wish to upgrade to Atomic Protector, and can not upgrade to 2.9.0, you will need to remove the ruleset(s) that are generating this error.

Error creating rule: Could not open phrase file “/etc/asl/custom-domain-blocks”: No such file or directory

This means that you have not setup this ruleset correctly.

SecReadStateLimit is depricated, use SecConnReadStateLimit instead.

This means your system is using the very buggy 2.8.0 modsecurity release. Do not use 2.8.0. Please see the article below for more information on 2.8.0.

If you must use 2.8.0, this error is benign and you can ignore it.

Error creating rule: Could not add entry

See below if you are using mod_security 2.8.0.

ModSecurity: IPmatch: bad IPv4 specification

This error is not caused by the rule. It is caused by a buggy version of modsecurity, specifically 2.8.0. 2.8.0 has a known critical bug in its IP matching code. It can not properly use IP addresses with rules, as documented in the github bug report here
Solutions:
CPanel:

Step 1) Remove 2.8.0 from your system.

Step 2) Install AUM on your system.

Plesk:
Step 1) Disable buggy modsecurity from the plesk interface under server settings.

Step 2) Remove the buggy modsecurity packages by running the following command:
yum remove plesk-modsecurity*
Step 3) Change the rules path to the correct systemwide path in /etc/asl/config
MODSEC_RULES_PATH="/etc/httpd/modsecurity.d"
Step 4) Make a directory for the rules
mkdir /etc/httpd/modsecurity.d
Step 5) Run the Atomicorp Update Manager by running the following command:
aum -u

httpd: ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated

This means that you are using out of date rules. If you are using Atomicorp rules, then this means you are not using the latest real time rules. The latest real time rules do not use transformations in the SecDefaultAction.

If you are using third party rules, this means that the rules contain a transform function in the SecDefaultAction setting. This is a deprecated setting.

ModSecurity: Failed to access DBM file “/var/asl/data/msa/

This means that you do not have the correct permissions setup for modsecurity to work correctly. Please make sure you have followed all the setup and installation instructions in the Atomic ModSecurity Rules wiki article.

Failed to create subdirectories

This means that the permissions on those subdirectories do not match the user that apache is running as. For example, if apache is running as the user “apache” those directories must be owned by apache.

Important note: If you are using an apache module that changes the user that apache runs as, on the fly, then those directories must be writable by all users. The use of modules that require you to set these directories as world writable and world readable is a serious security vulnerability, and should not be done. This will allow all users on the system to both change these logs, as well as to see both the logs and the payloads for all other users on the system.

Some modules, for example mod_ruid2, write logs as the user for that instance, but will set permissions on subdirectories such that other users can not write or read from them.

The most secure option is for these directories to be writable and readable only by a trusted user, and not by general users on the system.

Error creating rule: Unknown variable: MATCHED_VARS

If you are getting this error, it means that you are using an old version of modsecurity that does not support the modern rule language and you are not running Atomic Protector. We recommend you install Atomic Protector.

Atomic Protector will not allow incompatible versions of the modsecurity rules to be installed. It will also automatically upgrade mod_security provided you have UPDATE_TYPE set to “all”, which is the default. If you have updates disabled in Atomic Protector, then you will need to manually upgrade modsecurity:
yum upgrade mod_security
If you are not using Atomic Protector, then you will need to upgrade modsecurity using whatever method you used to install it. You will need to be running at least version 2.6.3.

Exec: Execution failed while reading output: /usr/bin/modsec-clamscan.pl (End of file found)

Without Atomic Protector

This error occurs if you install the 05_asl_scanner.conf file and/or the 99_asl_scanner.conf ruleset. These rulesets are not supported without Atomic Protector, as malware scanning is not included in the rules only subscription. Atomic Protector comes with malware upload scanning for HTTP, SSH, FTP and other protocols, including real time malware protection and much more. If you want malware upload protection, upgrade to Atomic Protector.

Solutions:

Option 1. Update to Atomic Protector

Option 2: disable this setting in /etc/asl/config: MODSEC_99_SCANNER

Option 3. Remove the file 05_asl_scanner.conf and/or 99_asl_scanner.conf and restart apache.

With Atomic Protector

This means someone or something has disabled the malware protection system on your system. Atomic Protector can not and will not do this.

/usr/bin/modsec-clamscan.pl is not installed on the server.

Malware scanning is not included in the rules only subscription. Atomic Protector comes with malware upload scanning for HTTP, SSH, FTP and other protocols, including real time malware protection and much more. If you want malware upload protection, upgrade to Atomic Protector.

We also don’t include that file or use the methods demonstrated in it because it doesn’t scale very well. Atomic Protector has a binary streaming system that scales.

No action id present within the rule

This means that you using out of date rules. If you are using Atomicorp rules, then this means you are not using the latest real time rules. The latest real time rules have an id action for every rule.

httpd: ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated

This means that you are using out of date rules. If you are using Atomicorp rules, then this means you are not using the latest real time rules. The latest real time rules do not use transformations in the SecDefaultAction.

If you are using third party rules, this means that the rules contain a transform function in the SecDefaultAction setting. This is a deprecated setting

ModSecurity: Found another rule with the same id

This means that you have a modsecurity rule with the same id. All of our rules have unique id’s. This error can only occur for one of two reasons:

You have loaded our rules twice. Please follow this installation guide and ensure that you have modsecurity configured as described in this document. These installation procedures will only load the rules once.

You have rules that are using id’s already assigned to other rules

If you are using third party or custom rules, check to make sure they have unique id’s. Modsecurity requires a unique id for each rule. The range 300000-399999 is used by our rules, do not use this range for any custom rules, and if you have third party rules with these id’s be sure to remove these rules.

Rule execution error - PCRE limits exceeded (-8): (null).

This is an internal limit to prevent a special type of DOS attack on the WAF itself. This is not caused by any of the rules. This is caused by the content the rules are inspecting. In certain cases, the content may be so complex that the WAF is stopping itself from doing too much work which could lead to a DOS attack on the system itself. If your system is generating these kinds of errors, it means you need to set the limits higher on your system, while it is beyond the scope of this article, another solution is to reduce the complexity of the content you are inspecting.

It is also possible this is occurring due to an actual DOS attack on your system. If you are certain this is not a DOS attack, simply increase these limits accordingly for your system. we recommend a minimum of 250000 for a modern system.
SecPcreMatchLimit 250000
SecPcreMatchLimitRecursion 250000
You may have to increase these limits for your system if you continue to get PCRE limit errors.

High Memory Usage

If apache is growing and growing in memory usage, and does not stablize please see the checklist below. If apache is simply using more memory, but has stablized (for example, Apache is using 100MB of memory, as opposed to 40MB of memory without modsecurity) this is normal. modsecurity caches the rules, and certain activities to speed up processing. Memory is infinitely faster than your hard drives, and loading the rules from the hard drive would be terrifyingly slow, too slow to be usable.

Ensure that ModSecurity is setup correctly

Install Atomic Protector. This will ensure ModSecurity is setup correctly.

Manually configure and setup ModSecurity yourself.

Ensure that ModSecurity is built correctly

Install Atomic Protector.

Install a ModSecurity build from our RPM repositories

Check for Leaking Web Application

Web applications can and do leak memory. Check to make sure your memory leak isnt from a web application. Or a web application isnt amplifying a memory leak in something else.

Ensure that Apache is built correctly

The major OS vendors, Redhat, Suse, Ubuntu and others do an excellent job of building Apache correctly. However, all software has bugs, and you should make sure that you are both running the latest updates from your vendor for a currently supported version of operating system from that vendor, and that you have ruled out an actual bug in apache. modsecurity is a complex piece of software, that relies of an even more complex piece of software, Apache, to work correctly. To improve performance, modsecurity will use apaches built in memory caching capabilities, and if there is a bug in these Apache supplied capabilities memory leaks can occur. We always recommend you discuss Apache memory leaks with your OS vendor first.

Please be especially diligent with source builds and Apache builds form non OS vendors. These builds are generally not as well tested, and so far are the only ones we have seen with memory management problems. If you use an Apache build from a non-OS vendor, you should discuss any memory management issues with them first if you have rules out the previous causes, as they are most likely a byproduct of a build error, or a bug in Apache.

Site loads slowly when the rules are loaded

Reason 1: DNS based rules are activated, but there is no local DNS server on the system.

These rules use DNS lookups to check for various characteristics of the connecting system, for example is it on a blacklist, is the IP address known to be used by a trusted source (Google for example). This means that for every connection to your system, a DNS lookup is done against the IP to see if the source IP is listed in some zone or blacklist. If you do not have a local DNS server, this can take a very long time depending on how busy the DNS server is that you are communicating with, and because the request is not cached, your system is asking the DNS server the same question several times in a row. This is because of the way HTTP works. A typical user request may need to open 100 or more connections to your site, and each time a DNS request is made to the RBL.

The simple fast solution is to make sure you are running a local DNS server on your system and that it caches requests. That way your RBL lookups are only to your local system, and if it already has the answer it will not have to ask the RBLs DNS server across the Internet. The difference in performance is night and day with a local DNS server, so we highly recommend you use on if you are going to use any RBL with any product.

Reason 2: The System is low on resources.

As with any piece of software, the WAF will consume some resources on the system. If the system is short on resources the WAF has to compete with other parts of the system for those resources. Check to make sure you do not have any process that is using up RAM or cycles on the system.

Reason 3: Custom rules may be inefficient. If you are using custom rules, try disabling them. If a rule is written in an inefficient manner it can kill performance on the system.

Reason 4: Rules are loaded twice. If you have setup modsecurity yourself, check to make sure you arent loading the rules more than once. One several cpanel installations we’ve seen cases where users inadvertently setup modsecurity to load the same rules twice, and in one case a user set them up to load three times. Apache is a literal animal, if you load the same rule twice it will process that same rule twice, doubling the work on the system! And because our rules use advanced branching logic, if you load them multiple times that can have a very adverse effect on performance as the branching tree logic won’t work correctly and you lose our on all the performance benefits a modern ruleset provides.

Reason 5: Suboptimal custom configuration.

If you setup modsecurity yourself, your configuration could be suboptimal. You may have serial logging enabled, debugging enabled, pour inspection options selected, or any number of things could be wrong. We recommend you use our Atomic Protector product to setup modsecurity, its automatic and it will always configure the optimal settings for your system.

If you prefer to do it yourself, then please make sure you have read the modsecurity documentation and understand the options for modsecurity before you enable them. modsecurity is a powerful piece of software that can easily be misconfigured. Care should be taken when setting up this software yourself.

For do it yourselfers, we have put together a guide to help you on the Atomic ModSecurity Rules page.