Atomic Endpoint Defender Kernel Configuration

Overview

AED includes a special secure kernel, that will proactively protect your system from many different classes of attack.

Do I have the AED Kernel Installed?

Check your rpm repository to see if the kernel is installed by running the following command:

rpm -qa kernel-asl

If you have the AED kernel installed, you will see a series of kernel names with ‘asl’ in them similar to this:

kernel-asl-4.14.26-3791.el6.art.x86_64
kernel-asl-4.14.28-3927.el6.art.x86_64
kernel-asl-4.14.13-3308.el6.art.x86_64
kernel-asl-4.14.27-3910.el6.art.x86_64
kernel-asl-4.14.30-3955.el6.art.x86_64

If you do not see any output, then you need to install the AED kernel.

How do I know if the AED Kernel is running?

You can tell if you are running an AED kernel with the command:

uname -a

If you are running the AED kernel, you should see a kernel name with ‘art’ in the title. For Example:

Linux www.atomicorp.com 4.14.13-3308.el6.art.x86_64 #1 SMP Fri Jan 19 09:51:23 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

Note

If you do not see a kernel name with “art” in the title then you are not running the AED kernel.

Installing the AED Kernel

Note

If you installed AED using the installer, then the kernel is already installed for you.

If you do not have the AED kernel, and you have not run the AED installer, run the AED installer.

If you have run the AED installer and the kernel is not installed, then follow this process:

OpenVZ/Virtuzzo

If you are using a VPS technology, such as virtuzzo or openvz, and you are installing AED inside a VPS you will not be able to install a kernel. This is not limited to the AED kernel, you can not install a kernel inside a VPS. VPS’ do not have their own kernel, they share the hosts single kernel, and you can not replace or modify that kernel from inside a VPS.

If you are concerned with kernel level vulnerabilities, install AED on the host.

Note

VPS technologies are not to be confused with technologies like VMWare, KVM, qemu, ESXi, Xen, and other virtualization technologies. Those technologies do allow you to install your own kernel, and you can install the AED kernel or any other kernel inside those virtual machines.

Xen

Please see the Xen Notes to ensure you have the Xen kernel enabled on your system.

Step 1: Set the kernel channel that is appropriate for your system via the AED web console, see the KERNEL_CHANNEL setting.

Note

Check to make sure you do not have the kernel excluded from your yum configuration.

Step 2 (64 bit system): If your system is 64 bit, run this command as root to install the kernel
el5/el6:
yum --disableexcludes=all --enablerepo=tortix-kernel upgrade kernel
el7:
yum --disableexcludes=all --enablerepo=tortix-kernel upgrade kernel-asl
Xen:
yum --disableexcludes=all --enablerepo=tortix-kernel-xen upgrade kernel
Step 2 (32 bit system): If your system is 32 bit, run this command as root to install the kernel
el5/el6:
yum --disableexcludes=all --enablerepo=tortix-kernel upgrade kernel-PAE
Xen:
yum --disableexcludes=all --enablerepo=tortix-kernel-xen install kernel-PAE
Note

If this does not work, please contact your system administrator or hosting company. This means that your system has been configured to not allow you to install kernels, or something is seriously wrong with the software management in your OS.

Upgrading the AED Kernel

Note

VPS systems do not have their own kernel.

To upgrade the AED Kernel, please follow the process below:

Step 1:
Check and see if you have the latest kernel by running the following command as root
aum -ck
If you have the latest kernel installed and running you will see the updater report OK for the kernel, for example:
Kernel update is available: 3.2.54-60                    [OK]
If you do NOT have the latest kernel installed and running you will see the updater report INFO for the kernel, for example:
Kernel update is available: 3.2.54-60                    [INFO]

Step 2:
Non Xen Systems:
If you have a 64 bit system, run the following command as root:
yum --enablerepo=tortix-kernel upgrade kernel-asl
If you have a 32 bit system, run the following command as root:
yum --enablerepo=tortix-kernel upgrade kernel-PAE-asl
Note

As in examples above, use yum upgrade instead of yum update to upgrade a kernel.

Step 3 (CRITICAL STEP):
The kernel is the “core” of your operating system. If something goes wrong with it, your system will not boot. So before you reboot into a new kernel you want to make sure you have two things:
Access your server if it doesn’t not reboot:

You will want to have either physical access to your server, or remote access to the console via a KVM technology. SSH, telnet and web control panel access is not adequate. If your system hangs on boot only console access can be used to access it.

All reputable hosting companies provide free console access for their customers. If your hosting company does not provide free console access, we encourage you to find a company that does. You will find lots of good hosting companies on our forums .
A working kernel to boot into:
Check to see what kernel your system is currently running. You can run this command as root to do this:
uname -r
You should see output similar to this:
2.6.18-308.11.1.el5
Note

This part of that line “2.6.18-308.11.1.el5” is the version of the kernel you are running.
Check to make sure the kernel is available to boot into. Linux kernels are stored in the /boot directory. Run the following command to see if the kernel is stored in the /boot directory:
ls -al /boot/* uname -r`*
If the kernel is installed, you will see output similar to the following:
-rw-r--r-- 1 root root   67546 Jul 10  2012 /boot/config-2.6.18-308.11.1.el5
-rw------- 1 root root 2717255 Jul 15  2012 /boot/initrd-2.6.18-308.11.1.el5.img
-rw-r--r-- 1 root root  116695 Jul 10  2012 /boot/symvers-2.6.18-308.11.1.el5.gz
-rw-r--r-- 1 root root 1276792 Jul 10  2012 /boot/System.map-2.6.18-308.11.1.el5
-rw-r--r-- 1 root root 2117180 Jul 10  2012 /boot/vmlinuz-2.6.18-308.11.1.el5
If you see output like the following:
ls: /boot/*2.6.18-308.11.1.el5*: No such file or directory
then the kernel you are using is not installed on your system. Do NOT reboot, please contact Atomicorp Support.

Note

Do not contact support if you are using a VPS system.
Check to make sure this kernel is listed in your grub configuration by running the following command:
grep `uname -r` /etc/grub.conf
If your kernel is available in grub.conf you should see output similar to the following:
title CentOS (2.6.18-308.11.1.el5)
 kernel /vmlinuz-2.6.18-308.11.1.el5 ro root=/dev/md0 selinux=0 panic=5
 initrd /initrd-2.6.18-308.11.1.el5.img
Before you reboot, also see the “Setting which Kernel to Boot” section on this page for instructions on configuring grub and confirming your current kernel is configured in grub.

Note

If you are using a third party boot manager, and not grub, please ontact your boot manager vendor for support with configuring boot order and confirming if your kernel is available during reboots.

Step 4:

Access your remote console and ensure you have access. Keep this connection open during step 4 below. If you run into any errors on boot, our support team will need you to provide the errors you see in your console. A serial console is the best tool to use, as this will allow you to cut and paste any errors you see.

If you do not have a serial console, you will need to document the full output of the boot. Sending screenshots of the final boot screen will not be helpful to diagnose your issue. Therefore, we highly recommend you use a serial console so you can cut and paste the output of the boot process

Step 5:
Reboot the system into the new kernel by running the following command as root:
reboot
Once the system reboots correctly, check to see if you have the AED kernel installed by running the following command:
uname -a
Note

If you do not see “asl” in the kernel name, then your system did not boot into the AED kernel.

What do I do if the Kernel is not Installed or won’t Upgrade?

If the kernel is not installed after running the AED installer or the kernel will not upgrade, that can mean three things:
1. If your system is using a container technology like Virtuozzo and Openvz products, you can not install a kernel. Container based systems do not have their own kernel and use the single kernel shared by the host system. Please contact your hosting provider or sysadmin to install the AED kernel on the host system.

Your hosting company wont let you install the kernel and has modified your system to prevent this.

3. If this is not a container based system, the AED installer will not install the kernel if your system is configured to not allow this. No distribution does this by default. If you system is configured this way, it usually means yum is configured on your system to not allow kernels to be installed. Please check your yum configuration to make sure it is not configured to exclude kernels. Some places to check are your /etc/yum.conf file for a line similar to this:
exclude="kernel"
If you see any line like the one above, remove them. Additionally, check in your /etc/yum.repos.d directory to make sure someone did not set this in a repository.
AED honors your systems configuration, so if you configure you system to prevent kernels from being installed via yum, AED will not install its kernel. If you do have any exclude lines configured for yum, please contact the parties that configured/setup your system and ask them to fix your system to allow kernels to be installed.

You can also manually installing the kernel, as described above, but keep in mind that if you have to do this something else is wrong with your system. The AED installer does this automatically, so if the kernel is not installed something on your system is preventing its installation, and not AED.
Please run the command below as root to see if yum is listing the AED kernel as described in the section above:
yum list | grep kernel
The output from the command above should look similar to the following (“art” should appear in the rpm name, if you see “art” then you have the AED kernel):
kernel.x86_64 1:2.6.25.4-4.art installed
kernel.x86_64 1:2.6.26.6-1.art installed
kernel.x86_64 1:2.6.27.7-9.art installed
kernel.x86_64 1:2.6.29.6-1.art installed
kernel.x86_64 1:2.6.32.8-1.art installed
kernel-devel.x86_64 1:2.6.25.4-4.art installed
kernel-devel.x86_64 1:2.6.26.6-1.art installed
kernel-devel.x86_64 1:2.6.27.7-9.art installed
kernel-devel.x86_64 1:2.6.29.6-1.art installed
kernel-devel.x86_64 1:2.6.32.8-1.art installed
kernel-headers.x86_64 1:2.6.32.8-1.art installed

Non Xen Systems

Run the following command as root if you have a 64 bit system:
yum --enablerepo=tortix-kernel upgrade kernel-asl
Run the following command as root if you have a 32 bit system:
yum --enablerepo=tortix-kernel upgrade kernel-PAE-asl

Xen Systems

Run the following command as root if you have a 64 bit system:
yum --enablerepo=tortix-kernel-xen upgrade kernel-asl
Run the following command as root if you have a 32 bit system:
yum --enablerepo=tortix-kernel-xen upgrade kernel-PAE-asl

Rolling back the Kernel

If you wish to use an older kernel, you can use the following command to rollback yum updates provided you have your system setup to allow downloading RPMs .
yum downgrade kernel
If you are using the PAE kernel, run the following command:
yum downgrade kernel-PAE

If you have kernel upgrades disabled in AED

If you select the option of not auto-updating the kernel, this will change the asl.repo file to exclude the kernel. This also means that you can no longer manually install the AED kernel. You must remove this exclusion to use yum to install older kernels. To prevent AED from upgrading to a newer kernel follow this process:

Step 1: Enable Kernel updates by following the process below

Log into the AED Web Console

Select the ‘Configuration’ tab

Select ‘AED Configuration’

Change “UPDATE_TYPE” setting to “all”

Step 2: Install the older kernel

Step 3: Disable kernel updates by following the process below

Log into the AED Web Console

Select the ‘Configuration’ tab

Select ‘AED Configuration’

Change “UPDATE_TYPE” setting to “exclude-kernel” or “rules-only”

Setting which Kernel to Boot

Note

AED will not replace your existing kernel, it will install and set the secure AED kernel to boot, but will leave your default kernel intact should you wish to use a non-secure kernel instead.

If you have AED Installed:

If you have just installed AED, all you need to do is reboot the system. When the system starts a special init called “asl-firstboot” will check the system to make sure everything worked correctly. If it did, then AED will configure the AED secure kernel to be the default kernel.

If AED detects any issues with the secure kernel, it will reboot the system into the default non-AED kernel installed on the system.

If your system is not setup this way, or this is missing from your boot loader please read this entire article. This can only occur if either your system is configured to not allow the installation of kernels, or something is seriously wrong with your systems software management system and is preventing this process from working correctly.

If you want to switch to a different kernel:

RHEL6/Centos 6:
Linux uses a boot loader to select which kernel to boot into (this is not part of AED, its part of your operating system). In most cases your system will use a boot loader called “grub”. If your system is using lilo, the older (really really old) boot loader we recommend you use grub. If you are using a third party boot loader, please contact your boot loader vendor for assistance.
If your system is using grub, you will need to modify this file as root:
/etc/grub.conf
A typical grub.conf file will look similar to this:
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/md1
#          initrd /initrd-version.img
#boot=/dev/md0
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.27.7-9.art.i686)
   root (hd0,0)
   kernel /vmlinuz-2.6.27.7-9.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
   initrd /initrd-2.6.27.7-9.art.i686.img
title CentOS (2.6.26.6-1.art.i686)
   root (hd0,0)
   kernel /vmlinuz-2.6.26.6-1.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
   initrd /initrd-2.6.26.6-1.art.i686.img
title CentOS (2.6.25.4-4.art.i686)
   root (hd0,0)
   kernel /vmlinuz-2.6.25.4-4.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
   initrd /initrd-2.6.25.4-4.art.i686.img
The system will boot into the kernel based on the configuration of this single line in /etc/grub/conf:
deafult=0
Where “0” is the number of the kernel to boot into. In grub the first position, or selected kernel, is “0”, the second is “1”, and so on.

To change which kernel to boot into simply change the following line to:
default=0

To:

default=# , where # represents 1, 2, 3, etc.
RHEL7/Centos 7:

Please see this article for more information about changing the kernel on the platform above.

Preventing the Upgrade or Installation for the Secure Kernel

If you want do not want yum to install the secure AED kernel, and would prefer to stick with your distributions stock kernel, put ‘exclude=kernel*’ in the [asl-2.0] section in /etc/yum.repos.d/asl.repo. AED will actively test and report vulnerabilities in all kernels, so if your stock kernel is reported to have vulnerabilities these are not false positives. We do not recommend you use any kernel other than the AED kernel.

Kernel Options

Please see the Kernel Configuration section for more information.

Testing the AED Kernel

Grub Users

Step 1: Once the Atomic Kernel is installed, determine which position it has been installed in by running the following command
cat /etc/grub.conf
The command above should provide output like the following:
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/md1
#          initrd /initrd-version.img
#boot=/dev/md0
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.27.7-9.art.i686)
   root (hd0,0)
   kernel /vmlinuz-2.6.27.7-9.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
   initrd /initrd-2.6.27.7-9.art.i686.img
title CentOS (2.6.26.6-1.art.i686)
   root (hd0,0)
   kernel /vmlinuz-2.6.26.6-1.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
   initrd /initrd-2.6.26.6-1.art.i686.img
title CentOS (2.6.25.4-4.art.i686)
   root (hd0,0)
   kernel /vmlinuz-2.6.25.4-4.art.i686 ro root=/dev/md1 rhgb quiet selinux=0 panic=5 ramdisk_size=128000
   initrd /initrd-2.6.25.4-4.art.i686.img
Note

default=1, this indicates the kernel the system will boot by default, starting at position 0. Position 0 is “title CentOS (2.6.17-1.art)”, and position 1 is “title CentOS (2.6.9-34.0.2.ELsmp)” in this example, indicating the system is configured to boot into the default CentOS kernel.
Step 2: Type grub and the following will be displayed
GNU GRUB  version 0.97  (640K lower / 3072K upper memory)
[ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename.]
grub>
Step 3: At the grub prompt set the default kernel to 0, and to only boot once with the following
grub> savedefault --default=0 --once
Step 4: Type ‘quit’

Step 5: Reboot the system. If for some reason the kernel does not work with the Atomic Kernel, or is otherwise non-response, powercycling the system will restore the system to the default kernel.

Lilo Users

Step 1: The art kernel should be listed in /boot - for example:
/boot/vmlinuz-2.6.19-7.art
Step 2: Create a symbolic link to this file by running the following command
ln -s  /boot/vmlinuz-2.6.19-7.art   /boot/vmlinuz-art
Step 3: Edit /etc/lilo.conf to add a section for the art kernel - for example:
image=/boot/vmlinuz-art
label=lxart
append="console=tty0 console=ttyS0,57600 panic=30"
Step 4: Type lilo to make the permanent change. Then to test the changes you can boot into the new kernel by running the following commands
lilo -v -v
lilo -R lxart
shutdown -r now
Step 5: When it’s rebooted, run uname -r and you should see the new “art” kernel. Now you can make it permanent by editing /etc/lilo.conf and modifying this line
default=lxart
Step 6: Reboot the sytem by running the following command
reboot

Manually Installing the AED Kernel

RHEL/CentOS EL7

Step 1: Edit /etc/asl/config and set the following setting
KERNEL_CHANNEL="tortix-kernel"

If you are using Xen:

   KERNEL_CHANNEL-tortix-kernel-xen
Step 2: Update the AED configuration file by running the following command
aum -u
Step 3: Install the AED kernel by running the following command
yum install kernel-asl
Step 4: (System with Grub2 ONLY): Set the kernel to boot only once for testing by running the following command
grub2-reboot 1
Step 5: Reboot the system by running the following command
reboot
Step 6: If this is sucessful, set the AED kernel to default boot by running the following command
/usr/sbin/grub2-set-default 0
Step 7: Regenerate the grub configuration file by running the following command
grub2-mkconfig -o /boot/grub2/grub.cfg

Kernel Tuning

The AED Kernel includes four I/O schedulers to suit various system needs and configurations, we describe the schedulers below:

Completely Fair Queuing (CFQ) - This is the default algorithm. As the name implies, CFQ maintains a scalable per-process I/O queue and attempts to distribute the available I/O bandwidth equally among all I/O requests. CFQ is well suited for mid-to-large multi-processor systems and for systems which require balanced I/O performance over multiple LUNs and I/O controllers.

Deadline - The Deadline elevator uses a deadline algorithm to minimize I/O latency for a given I/O request. The scheduler provides near real-time behavior and uses a round robin policy to attempt to be fair among multiple I/O requests and to avoid process starvation. Using five I/O queues, this scheduler will aggressively re-order requests to improve I/O performance.

NOOP - This scheduler is a simple FIFO queue and uses the minimal amount of CPU/instructions per I/O to accomplish the basic merging and sorting functionality to complete the I/O. It assumes performance of the I/O has been or will be optimized at the block device (memory-disk) or with an intelligent HBA or externally attached controller.

Anticipatory - The Anticipatory elevator introduces a controlled delay before dispatching the I/O to attempt to aggregate and/or re-order requests improving locality and reducing disk seek operations. This algorithm is intended to optimize systems with small or slow disk subsystems. One artifact of using the AS scheduler can be higher I/O latency.

Changing the Scheduler

If the default scheduler does not meet your needs, you can change the scheduler by logging in as root and changing the schdeuler for the device in this manner:
echo {SCHEDULER-NAME} > /sys/block/{DEVICE-NAME}/queue/scheduler
You can also view the scheduler you’re using by running the following command:
cat /sys/block/{DEVICE-NAME}/queue/scheduler

Technical Abstract of the AED Kernel

Overview

The kernel installation/upgrade process is for the most part completely automated. Upgrade events are capable of reading the existing configuration information, and adding them to the boot loader.

Major Components:

Kernel, usually /boot/vmlinuz-XXXX

Ram Disk, usually /boot/initrd-XXXX

Modules, usually /lib/modules/XXXX

Boot loader, usually Grub (/etc/grub.conf which is a symlink to /boot/grub/menu.lst)

Module configuration information, /etc/modprobe.conf, or /etc/modprobe.d/*

Minor Components:

New-kernel-pkg, /sbin/new-kernel-pkg

Kenel configuration file (used by new-kernel-pkg), /etc/sysconfig/kernel

mkinitrd config directory (depends on distro), /etc/sysconfig/mkinitrd/

Boot Process:

Grub is configured to load a specific kernel

Kernel boots, this loads the initrd ramdisk

initrd contains the modules needed to read core hardware, like the disks. This is generally where mayhem happens.

System mounts disks and goes multi-user, remaining modules like the ethernet controller, lm_sensors, firewall modules, are loaded dynamically

Initial Installation Process:

Basically the Red Hat OS installer does some really impressive work, its called anaconda and you can read more about it here . In brief, the system boots off of an installer kernel, and anaconda interrogates the system to populate the modprobe configuration file mentioned above. Its the only thing I know of that can actually figure out what the hardware is on a box without human intervention.

Modprobe is what loads the kernel drivers for everything on the box, like the disks or the network card, and the modprobe.conf/modprobe.d part is only configured through anaconda. If you dont go through anaconda, then you dont get these configs for your hardware, and if you dont have these configs a normal upgrade wont work.

Let me reiterate this, if you’re in an environment where someone thought they were doing you a favor by not putting their own kernel on a system from a disk image (1&1, and ovh, I mean YOU), well they arent. You’re about to take the first step down the long road of becoming a kernel/hardware expert. There is no automated way, short of re-imaging the box (and thereby running anaconda again) to resolve this.

Upgrade Process:

Assuming you’ve got a healthy, normal system, its largely automated. At most AED only interacts with the system at the grub level, using tricks like –once to boot kernels in test mode. That aside, heres what happens when you upgrade the kernel with yum.

Yum installs an additional kernel on the system. It may be configured to remove older kernels, but in nearly every environment it is just adding kernels to the system.
Inside the kernel rpm there is a post processing macro called %post, this is a shell script that actually does all the magic. Heres what it looks like from Fedora 10’s 2.6.27.x kernels:
%{expand:%%post %{?-v*}}\
%{-s:\
if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ] &&\
[ -f /etc/sysconfig/kernel ]; then\
/bin/sed -i -e 's/^DEFAULTKERNEL=%{-s*}$/DEFAULTKERNEL=%{-r*}/' /etc/sysconfig/kernel || exit $?\
fi}\
/sbin/new-kernel-pkg --package kernel%{?-v:-%{-v*}} --mkinitrd --kernel-args="selinux=0 acpi=off" --depmod --install %{?1} %{KVERREL}%{?-v*} || exit $?\
%{nil}
Breaking this down, the first step is to update /etc/sysconfig/kernel. The sed line above is passing in information on which base kernel is being used. This is not the version, rather its saying that the default kernel for this system is the kernel-PAE or kernel- package. /etc/sysconfig/kernel itself is one of the config files used by the next step.

new-kernel-pkg is run. This is what creates the initrd ramdisk, and add the kernel to grub. Getting back to the whole modprobe.conf issue, new-kernel-pkg reads /etc/modprobe.conf or /etc/modprobe.d for the information it needs to create the /boot/initrd ramdisk. If this information is wrong, or missing, the kernel will not boot. What you’ll get is a kernel panic.

Once the initrd is created, new-kernel-pkg adds the entries to the bootloader (/etc/grub.conf or /boot/grub/menu.lst), again using /etc/sysconfig/kernel as its config file. You’ll see a setting in there UPDATEDEFAULT=yes. If this is set to yes, then whenever new-kernel-pkg is run, the system will automatically configure that kernel as the default.

Sometimes during this process you will see warnings that certain modules dont exist. This gets back to modprobe.conf again, as the kernel evolves sometimes the module names will change, or even go away. The entries still exist in modpobe.conf, and so new-kernel-pkg will still try to add them. Its generally a bad sign, and something you should worry about, especially if its referencing something important like a network or disk controller. Because of this its a good idea to always boot your kernels in testing mode with the –once flag.

Kernel Panics

Please review Kernel Panics for more information.

Additional Kernel Features

Please see the AED Kernel Features article for more information.

Kernel Modules

For a complete list of AED Kernel Modules, please see this page .

Source Code

You can install the AED Kernel source code by running the following command as root:

yum install kernel-asl-devel