A day in the life…

I just noticed some strange Puppet behaviour. Take a look at this:

class monitoring {
  define nrpe_command (
    $ensure               = present,
    $command,
    $target               = "/etc/nagios/nrpe.d/$title.cfg"
  ) {
    file { $target :
      ensure            => file,
      owner             => root,
      group             => root,
      mode              => 0755,
      content           => "command[$title]=$command\n",
    }
  }
}
class puppet {
  monitoring::nrpe_command {"restart_puppet":
    command => "/etc/init.d/puppet restart"
  }
}

This is the resulting I’ve got:

testhost:/etc/nagios/nrpe.d# ls
puppet.cfg
testhost:/etc/nagios/nrpe.d# cat puppet.cfg
command[restart_puppet]=/usr/local/bin/restart_puppet_safely.sh
testhost:/etc/nagios/nrpe.d#

Somehow, puppet eats everything until the last underscore in the $title variable when using it in the file title, but not when using the variable as content for that file.

Any ideas?

When setting up Pacemaker clusters I tend to reuse my Pacemaker Puppet module in which I distribute a cluster-specific authkey file.

My workflow has always been:

1. Log on to a host that has corosync installed, but not part of a cluster
2. Run corosync-keygen
3. Transfer the generated /etc/corosync/authkey to my Puppet GIT repo on my local machine

You don’t want to run corosync-keygen on an existing cluster as it will overwrite /etc/corosync/authkey.

As it seemed quite a hassle to compile corosync on my Mac just to be able to run corosync-keygen and so delved a bit deeper, finding this post on the openais mailing list: http://lists.linuxfoundation.org/pipermail/openais/2010-February/013845.html

My workflow has thus simplified to:

dd if=/dev/random of=files/authkey bs=1 count=128

Sweet!

This news item struck me today: Bar-peeing is a new fenomenon in Holland (in Dutch)

Quote:

“In a busy, dirty pub with beer all over, I don’t mind. What you pee is mostly beer anyway”, says an anonymous Rotterdammer.

Must be Heineken, methinks…

Shopping on the Dell online shop is absolute horror. Tons of choices without any explanation, like this one:

How am I supposed to know what this is?

What about this one:

Most stupid option, when selecting a LED backlit display, I get this error:

That’s right! I should have chosen the “No camera with microphone for LED backlit display”. Reminds me of the joke about a man that asks for a ham and cheese sandwich without butter to with the clerk responds they only have ham and cheese sandwiches without margarine…

See here…

So, you got a server linked to a SAN box. You define some LUNs on that SAN box. You link the LUNs to the server and you want to start using the LUNs on the server.

Let’s see:

xen02:/dev# ls -l sd*
brw-rw---- 1 root floppy  8,   0 2008-04-15 16:51 sda
brw-rw---- 1 root floppy  8,   1 2008-04-15 16:51 sda1
brw-rw---- 1 root disk    8,  16 2008-04-15 16:51 sdb
brw-rw---- 1 root disk    8,  32 2008-04-15 16:51 sdc
brw-rw---- 1 root disk    8,  48 2008-04-15 16:51 sdd
brw-rw---- 1 root disk    8,  64 2008-04-15 16:51 sde
brw-rw---- 1 root disk    8,  80 2008-04-15 16:51 sdf
brw-rw---- 1 root disk    8,  96 2008-05-09 13:35 sdg
brw-rw---- 1 root disk    8, 112 2008-05-09 13:35 sdh
brw-rw---- 1 root disk    8, 128 2008-05-09 13:35 sdi
brw-rw---- 1 root disk    8, 144 2008-05-09 13:35 sdj
brw-rw---- 1 root disk    8, 160 2008-05-09 13:35 sdk
brw-rw---- 1 root disk    8, 176 2008-05-09 13:35 sdl
brw-rw---- 1 root disk    8, 192 2008-05-09 13:35 sdm
brw-rw---- 1 root disk    8, 208 2008-05-09 13:35 sdn
brw-rw---- 1 root disk    8, 224 2008-05-09 13:52 sdo
brw-rw---- 1 root disk    8, 240 2008-05-09 13:52 sdp
brw-rw---- 1 root disk   65,   0 2008-05-09 13:54 sdq

Phew, how to identify which drive corresponds to which LUN, especially if you have some equal-sized LUNs.
Fortunately, we have this:

xen02:/dev/disk/by-id# ls -l
total 0
lrwxrwxrwx 1 root root  9 2008-05-14 16:11 scsi-1 > ../../sdl
lrwxrwxrwx 1 root root  9 2008-04-15 16:51 scsi-3600a0b8000320d200000038e47e1430a > ../../sdb
lrwxrwxrwx 1 root root  9 2008-04-15 16:51 scsi-3600a0b8000320d200000039047e3f21f > ../../sdd
lrwxrwxrwx 1 root root  9 2008-04-15 16:51 scsi-3600a0b8000320d200000039447f4f916 > ../../sdf
lrwxrwxrwx 1 root root  9 2008-05-09 13:35 scsi-3600a0b8000320d200000041f48073120 > ../../sdh
lrwxrwxrwx 1 root root  9 2008-05-09 13:35 scsi-3600a0b8000320d200000042348076e8c > ../../sdj
lrwxrwxrwx 1 root root  9 2008-05-09 13:35 scsi-3600a0b8000320d200000043748206a2e > ../../sdn
lrwxrwxrwx 1 root root  9 2008-05-09 13:52 scsi-3600a0b8000320d200000043a48244540 > ../../sdp
lrwxrwxrwx 1 root root  9 2008-04-15 16:51 scsi-3600a0b8000322cba0000067c47e3f289 > ../../sdc
lrwxrwxrwx 1 root root  9 2008-04-15 16:51 scsi-3600a0b8000322cba0000067e47e3f2cc > ../../sde
lrwxrwxrwx 1 root root  9 2008-05-09 13:35 scsi-3600a0b8000322cba000007ca48076f1b > ../../sdi
lrwxrwxrwx 1 root root  9 2008-05-09 13:35 scsi-3600a0b8000322cba000007da480dc9cd > ../../sdk
lrwxrwxrwx 1 root root  9 2008-05-09 13:35 scsi-3600a0b8000322cba000007de480f5683 > ../../sdm
lrwxrwxrwx 1 root root  9 2008-05-09 13:54 scsi-3600a0b8000322cba000007e0482445e4 > ../../sdq
lrwxrwxrwx 1 root root  9 2008-05-09 13:52 scsi-3600a0b8000322cba000007e248244632 > ../../sdo
lrwxrwxrwx 1 root root  9 2008-05-14 16:11 scsi-3600a0b8000322cba00000890482afe68 > ../../sdg     <---
lrwxrwxrwx 1 root root  9 2008-04-15 16:51 usb-M-Sys_uDiskOnChip_0F801A713040492E > ../../sda
lrwxrwxrwx 1 root root 10 2008-04-15 16:51 usb-M-Sys_uDiskOnChip_0F801A713040492E-part1 > ../../sda1

Exactly, by using the /dev/disk/by-id virtual directory, you can see which disk id corresponds to which LUN, as you can see in the next (partial) screenshot:

That’s nice! Now we know that our LUN called dpmgmt-root corresponds to 60:0a:0b:80:00:32:2c:ba:00:00:08:90:48:2a:fe:68 which, according to our second listing, corresponds to /dev/sdg.

Using such long device paths isn’t really convenient though, so let’s take this a little further.

Format the device using your preferred filesystem and label your filesystem:

xen02:/dev/disk/by-id# mkfs.ext3 -L dpmgt-root scsi-3600a0b8000322cba00000890482afe68
mke2fs 1.40-WIP (14-Nov-2006)
scsi-3600a0b8000322cba00000890482afe68 is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=dpmgt-root
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
...

Reboot your system and check this out:

xen02:/dev/disk/by-label# ls -l
total 0

lrwxrwxrwx 1 root root  9 2008-05-14 17:28 dpmgmt-root > ../../sdg
lrwxrwxrwx 1 root root 10 2008-04-15 16:51 root > ../../sda1
lrwxrwxrwx 1 root root  9 2008-05-09 13:35 semail-root > ../../sdn
lrwxrwxrwx 1 root root  9 2008-04-15 16:51 tempvm-root > ../../sdf

So now, we can address our device as /dev/disk/by-label/dpmgmt-root on every machine that has access to that LUN, now matter what the actual real device path is on that machine.

What’s even better, is that you even don’t even have to reboot to update /dev/disk/by-label/
Just trigger udev to reload the system information on /dev/sdg:

xen06:/dev/disk/by-label# echo add > /sys/block/sdg/uevent
xen06:/dev/disk/by-label# ls
dpmgmt-root root semail-root tempvm-root

Et voilà!

The last couple of days, it seems there’s some kind of ssh botnet trying to spread out. Since I installed DenyHosts some weeks ago, I usually got 5-10 notifications of blocked IP addresses. Last weekend however, I got more than 200 notifications. 

Although I feel rather safe having installed DenyHosts (which I urge you to install on every SSH accessible server), as a lot of hosts out there aren’t protected, I fear a new botnet is in the making.

Clearly, someone thinks I need to get more junk in my mailbox. Let me tell you though, 1400 spam mails a day is enough already…

That’s why I wrote a very rudimentary script that hooks into DenyHosts and queries a whois server for an abuse email and sends it an email when found.

Update:
I didn’t notice the difference between the PLUGIN_DENY setting and the mail behaviour of DenyHosts.
By default, DenyHost will send you an email everytime it adds a new host to /etc/hosts.deny, whereas the PLUGIN_DENY script will be invoked every time it adds or readds a host to /etc/hosts.deny. That’s why I now first grep a file with hosts whose hostmaster I already notified of the abuse

The script is ridiculously easy:

#!/bin/bash

# Get parameter
IP=$1

# Check whether we've already seen this host.
if `grep $IP /var/lib/denyhosts/notified_abuse > /dev/null` ;
then
        echo host already seen
        exit
else
        echo new host, added to logfile
        echo "`date` $IP" >> /var/lib/denyhosts/notified_abuse
fi

# Try to lookup the abuse mailbox
abuse=`whois $IP|grep ^abuse-mailbox:| tail -n 1 |sed -e "s/abuse-mailbox: //"`

# if found an abuse mailbox, send a mail.
if [ "x$abuse" != "x" ];
then
cat << EOF | mail -a "From: Pieter Barrezeele <xxxxxx@xxxxxxxxxx.be>" -s "SSH brute force attack from $IP" $abuse

Dear Sir/Madam,

Today we experienced an SSH brute force attack originating
from $IP, a host under your responsibility. This probably means
the host in question is compromised.
Please take action to stop this host from attacking us again.

Thanks in advance,
Pieter Barrezeele

PS: this is an automated mail, any errors in this mail are caused by parsing errors.
EOF
fi

 
I’d advise you to send the mails to yourself for a few days until you see only new hosts are added. Alternatively, you could copy the contents of /etc/hosts.deny into /var/lib/denyhosts/notified_abuse as well.

Last weekend the Iris festivities took place in Brussels. One of the headliner acts were performances by Daan and Ozark Henry. Of course I couldn’t miss out on that last one. We arrived just in time to enjoy Housewife, one of Daan’s latest and greatest songs after which it was Piet Goddaer’s turn to entertain the masses arrived at the Paleizenplein just in front of the royal palace.

As usual, this concert was a nice mix of his recent work as well some of his older hits, including “Sweet Instigator”, “Word up”, “Rescue” and even “Inhaling”.

To conclude, I’d like to say: “Merci, Piet!”

And it was good. Well, it’s actually not over yet. A lot of enthousiasts chose to sleep over at the Martelarenplein in front of Leuven’s railway station, to show solidarity with streetchildren in third world countries. Hopefully, this event will reach its goal by gathering enough attention to its cause.
A lot of workshops ranging from street cooking to street dancing, short sets by street artists for this occasion: Milow, Klaas from Yevgueni and Anton Walgrave, an election of the “nicest” street shelter made from rubbish.
I’m really impressed by the courage these people show by sleeping outside on this cold, rainy night. Hats off!

A small impression: