IBM ServRAID tools and Mandriva Linux 2010.0

We recently purchased a couple of Lenovo RD120 servers (exellent quality!) coming with an IBM ServRAID-8k SAS/SATA controller.

This RAID controller is well known from the IBM x35xx series and features RAID levels 0,1,5,6,10 and 1e (whateever level '1e' may be :-), 256MB of RAM and a backup battery to finish write operations once power goes down. So in general, a truly excellent controller.

The only thing that proves to be somewhat tricky is to get the fitting tools up and running that allow you for example to hotswap drives (that is, take one drive offline, take another one online and synchronize it).

So, the first thing to do is to download the latest "IBM ServRAID Application CD" from here [1]. What you get from here is a +400Mb ISO, containing the tools for various operating systems, among them is linux and linux-x86_64.

Once you have downloaded it, copy linux_x86_64/cmdline/arcconf and linux_x86_64/manager/RaidMan-9.00.x86_64.rpm to the server.

The arcconf tool first needs to be made executable (chmod +x arcconf), then you can invoke it without parameters and it will list you all possible options. For example, this gives you more information about the raid controller:

% arcconf GETCONFIG 1 AD
Controllers found: 1
---------------------------------------------------------------
Controller information
---------------------------------------------------------------
   Controller                   : Okay
   Channel                      : SAS/SATA
   Controller                   : IBM ServeRAID 8k
[..]


In general, if you prefer command line tools like myself, arcconf allows you to do anything with the raid controller, that you like (decommission defect drives/commissison new ones, ...).

But of course, sometimes GUI applications are not so bad, after all :-)

Installing the manager application proves to be somewhat more tricky, because Mandriva is not officially supported by IBM. However, it is not too difficult:

% urpmi RaidMan-9.00.x86_64.rpm libxp6 libxt6 libxtst6 libstdc++5 libx11-common

The extra packages are required for java to operate correctly and to allow startup of the manager GUI.

Now, finally, you can invoke

% /usr/RaidMan/RaidMan.sh

You will get a couple of errors about "/usr/lib/libstdc++.so.5" not being preloadable, but as far as I found out, you can safely ignore them.

The only thing yet missing is to allow remote access to the raid agent. So far I was not able to connect to the raid agents on the servers for reasons unknown.

[1] http://www-304.ibm.com/jct01004c/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-61707&brandind=5000020

accessing the CLI of Cisco "Small Business" Switches

Preface: Cisco's "Small Business" network equipment is based on hardware formerly manufactured by Linksys. When Cisco bought Linksys, they incorporated the Linksys hardware as "Cisco Small Business" equipment.

Now, at first glance that doesn't sound too bad. Cisco, being the major player when it comes to network equipment, is well known for its very well designed and quite well supported devices.

However, when the story comes to the formerly Linksys products, Cisco looses everything it has to loose, both in terms of support and also in terms of design.

One example for that failure is documented here [1] for example.

Another story is the completely unuseable web interface of the former managed Linksys switches, like the SRW224G4 [2] for example.

We are unfortunate enough to purchase a couple of those devices, in the hope that the web based management facilities would be useful to some degree.

The truth however is, that the web UI is completely unuseable if you use any other browser than IE6. Neither IE7, IE8, FF2, FF3 or Opera can be used to access the Web UI ...

There are many people complaining about this situation (for example here [3], but obviously - once more - Cisco's support is completely ignorant.

The only thing you can do is to use the command line interface of those switches. In order to do so, you must access the switch like this (taken from here):

- Telnet to the switch & login
- Once you get to the menu, hit ctrl+z, you should now see a ">" (type ? to all available commands from this point on)
- Type lcli, hit enter, and put the user name in again (admin usually)
- now you should see "console#"

How you can configure your script in a similar way known from Cisco's IOS.

A good starting point of how the CLI works can be found here [5].

And what remains for the bottom line is that: just don't buy Cisco's "Small Business" products until they are ready to give real support for them. After all: even those products are business products!

[1] http://www.dslreports.com/forum/r22733060-RVS-4000-Support-email-to-Cisco
[2] http://www.cisco.com/en/US/products/ps9987/index.html
[3] http://davehall.com.au/blog/dave/2007/12/04/linksys-srw224g4-webgui-broken
[4] http://davehall.com.au/blog/dave/2007/12/04/linksys-srw224g4-webgui-broken#comment-393
[5] http://lcli.wikidot.com/

SVN authentication with client SSL X.509 certificates and apache 2.2

We recently moved our subversion repository to a new, now dedicated server. In that course I found it reasonable to streamline our old configuration how authentication and authorization against the repository is done.

Previously, our users were primarily authorized to access the repository using their X.509 certificates. After authorization took place, they were authenticated and thus being asked for their usernames and passwords.

The goal now was to get rid of the second step, allowing authentication and authorization based purely on the certificates.

To be true, the solution was not so easy to find, because apache's mod_ssl module is not really designed for authentication purposes. It has a pretty useless "FakeBasicAuth" option requiring to manually store each user in a htpasswd style file, containing the hardcoded 'password' string as each users password. Pretty ugly, IMHO.

However, I finally managed to get to a resonable result.

Before doing anything else, ensure that you have mod_ssl and mod_dav_svn up and running. I won't go deeper into those basics.

After the modules are ready, put the following in your apache's config for the relevant (virtual)host:

<Location /theLocationOfYourRepository>
  SSLVerifyClient require
  SSLRequireSSL

  SSLOptions +StdEnvVars
  SSLUserName SSL_CLIENT_S_DN_Email

  DAV svn
  SVNPath /the/absolute/filesystem/path/of/your/repository
  SVNListParentPath on
  AuthzSVNAccessFile /etc/subversion/apache-acl-file
</Location>

In my case, the emailAdress attribute of the certificate's subject DN is used to make up the username ("SSLUserName SSL_CLIENT_S_DN_Email"), useable in the AuthzSVNAccessFile, for example:

[/project1]
fred.flintstone@example.com = rw
wilma.flintstone@example.com = r

[/project2]
wilma.flintstone@example.com = rw

# locking out everybody else
[/]
* =

Instead of the emailAddress attribute, you can choose from a number of alternatives, see here [1] and here [2].

For now, I am quite satisfied how it works.

The only thing to be done in the future will be to map those email addresses against LDAP entries and have the usernames retrieved from the DIT based on the matches. And eventually I want the AuthzSVNAccessFile also being served from our LDAP server, of course. But that will probably be a hard fight.

[1] http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#envvars
[2] http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslrequire

capturing videos from your linux desktop

I recently had to produce some screencasts from my linux desktop and wasn't able to find a "quick" solution for this "problem". Of course, google is your friend, so here are two reasonable possibilities I found:

• recordMyDesktop [1]
  The most powerful utility I found, coming as a commandline tool and with qt and gtk frontends as well. Furthermore you can also record audio. Its only limitation (if you call this a limitation) is that it can only produce ogg theora for video and ogg vorbis for audio, so no direct mpeg support (but converting theora to whatever is trivial).
  


• XvidCap [2]
  An "older" utility (last update is from 2006). Besides capturing as a video including audio, it can also produce a series of jpegs instead. It produces mpeg videos.
  

So for now I am quite happy with recordDesktop :-)

[1] http://recordmydesktop.sourceforge.net/
[2] http://xvidcap.sourceforge.net/

rescanning the connected SATA/SCSI devices

Once in a while I stumble upon the problem where I have to attach a SATA or SCSI device to a running Linux box and for some reasons (ie. the device or the interface is not hot plugging capable) it doesn't get recognoized by the system.

Now the solution is very simple, just tell the kernel to rescan the SATA/SCSI bus:

first find out how many SATA/SCSI busses you have:

% ls -l /sys/class/scsi_host/
drwxr-xr-x 2 root root 0 2009-08-24 15:05 host0
drwxr-xr-x 2 root root 0 2009-08-24 15:05 host1
drwxr-xr-x 2 root root 0 2009-08-24 16:26 host2
drwxr-xr-x 2 root root 0 2009-08-24 16:26 host3

So that means 4 host interfaces to query for. Querying or rescanning is very easy:

Find out which hard disks are currently recognoized by the kernel:

% cat /sys/class/scsi_disk/*/device/model
WDC WD1600YS-01S

In other words, a Western Digital WD1600YS hard drive is connected and recognoized by the computer.

Now, if you want the SATA/SCSI bus to rescan all host interfaces for new devices, do this:

% echo "- - -" > /sys/class/scsi_host/*/scan


To theck if any previously unrecognized hard disks have been detected, refetch the list of hard disks:

% cat /sys/class/scsi_disk/*/device/model
WDC WD1600YS-01S
Seagate FreeAgen


So as you see, a new "Seagate FreeAgen" hard disk has been detected.

The procedure is essentially the same for all other types of SATA/SCSI devices such as scanners, tape drives, ...

Linksys/Cisco RVS4000: support madness

Trying to find further information about some problems we are facing with our Linksys/Cisco RVS4000 router, I stumbled upon this blog entry:

http://www.dslreports.com/forum/r22733060-RVS-4000-Support-email-to-Cisco

The poor guy tells his nightmare with Cisco/Linksys support regarding some problems with his RVS4000 router. Among other things, we are seeing exactly the same problems here as well, but obviously contacting Cisco's support seems to be a waste of time.

I can hardly remember a company with such a support reputation to loose fail so miserably ...

Debian+KVM: proxy_arp for individual ip adresses

In one of my older posts [1] I described how you can hide the MAC addresses of KVM virtualized guests living in a dedicated, externally reachable subnet.

If however you don't have a dedicated subnet from your hosting provider but only got only a small number of maybe segmented IP addresses instead of an entire subnet, the solution is much simpler.

So, let's say this is what you want:

• a host
  living under 192.168.10.10 with a netmask of 255.255.255.0


• a KVM guest
  designated to live under 192.168.10.20


• another KVM guest
  designated to live under 192.168.10.30

Due to port security, each of the guests must not reveal their (virtual) MAC addresses to any external communication partner.

For a Debian lenny host, the following prerequisites are required

• install iproute
• install uml-utilities

Using uml-utilities (uml = user mode linux), you can setup virtual network interfaces useable by the KVM guests.

In order to get things going, add the following lines to /etc/network/interfaces

[...]
auto tap0
iface tap0 inet manual
  tunctl_user root
  uml_proxy_arp 192.168.10.20
  uml_proxy_ether eth0
  up ip link set tap0 up
  post-up sysctl -w net.ipv4.ip_forward=1
  post-up sysctl -w net.ipv4.conf.tap0.proxy_arp=1
  pre-down sysctl -w net.ipv4.ip_forward=0
  down ip link set tap0 down

auto tap1
iface tap1 inet manual
  tunctl_user root
  uml_proxy_arp 192.168.10.30
  uml_proxy_ether eth0
  up ip link set tap1 up
  post-up sysctl -w net.ipv4.conf.tap1.proxy_arp=1
  down ip link set tap1 down


The lines above create two to called "tap" devices tap0 and tap1, using the utilites provided by user mode linux (somewhat like an alternative to KVM, vmware, ...). Due to severe Debian magic, the lines also enable proxy_arp for the new devices and tell the kernel to use the MAC address of eth0 instead:

% ifup tap0
[...]
% arp -an
[...]
? (192.168.10.20) at PERM PUB on eth0
% cat /proc/sys/net/ipv4/conf/tap0/proxy_arp
1

And still better, even a specific device route is created:

% route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.10.20   0.0.0.0         255.255.255.255 UH    0      0        0 tap0
192.168.10.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
0.0.0.0         192.168.10.254  0.0.0.0         UG    0      0        0 eth0

If you ifup tap1 as well, you will see further entries like the ones above.

[sidenote]: User mode linux is quite a funny thing to play with, a good starting point for further reading is this link [2].

The only thing remaining for you is to instruct your KVM guests to use their new, dedicated devices. You can do so manually by ensuring that the guests are started with the following options set:
% kvm \
[...] \
-net nic \
-net tap,ifname=tap0,script=no \
[...]

and for the other guest
% kvm \
[...] \
-net nic \
-net tap,ifname=tap1,script=no\
[...]

If you use the libvirt daemon, you can alternatively update the guest's xml definition like this:

[...]
<interface type='ethernet'>
  <target dev='tap0'/>
</interface>
[...]

and for the other guest of course:

[...]
<interface type='ethernet'>
  <target dev='tap1'/>
</interface>
[...]

If you now configure your guests to use once 192.168.10.20 and once 192.168.10.30, you should be able to ping other hosts without revealing the guest's virtual mac addresses. That can be tested using arp again on the targeted hosts, eg.

In the first guest with an ip address of 192.168.10.20, start a ping against 192.168.10.254:
% ping 192.168.10.254
PING 192.168.10.254 (192.168.10.254) 56(84) bytes of data.
64 bytes from 192.168.10.254: icmp_seq=1 ttl=64 time=1.76 ms
[...]

Then log into 192.168.10.254 and use arp to verify the mac address used:
% arp -an
[...]
? (192.168.10.20) at ac:00:00:00:01 [ether] on eth0
[...]

The mac address found (AC:00:00:00:01) should be the same as the KVM hosts physical network card.

Beware once more, that any "proxy_arp" style solution one major drawback: you cannot use DHCP to configure your virtual guests, because a DHCP server has no way to determine their (virtual) mac address.

Happy hacking again :-)

[1] http://riaschissl.blogspot.com/2009/06/port-security-proxying-kvm-mac.html
[2] http://user-mode-linux.sourceforge.net/old/UserModeLinux-HOWTO.html