About a year ago I built several 1.2TB fileservers for a number of my consulting clients which utilized RAID5 arrays for redundancy with LVM running on top for expandability. One of my cleints which does some media work has exhausted the storage space and called a few weeks ago about expanding the storage space on the server.

The four hard drives in the server now were already utilizing all the onboard SATA-II ports. I certainly could have replaced the drives with larger ones (which I did do for another client) but that would have entailed some careful shuffling of data and wouldn’t provide for much future expandability. For another client who uses space much more slowly I could have added a two port SATA expansion card and added two drives in RAID1 but here I expect to need to continue adding space and so I proposed an external storage tower with a multiport SATA link. I was looking for a PCI Express controller which would support eight drives on a single card and would be supported in Debian Linux. I ended up selecting a Highpoint RocketRAID 2322 which seemed to fit the bill.

As it turns out packaged driver support for Linux is only available for Fedora, Red Hat and SuSE. Luckily I found great instructions at this University of Northern Iowa site for building the drivers from source provided by Highpoint. Although there is some grumbling in the open source community about these drivers being non-free licensed (hence no package from Debian) just about everything else is great. The kernel module built without any problems and without a huge number of dependencies and I was able to get the drives up and running without too much work.

Unfortunately, I did not get the module into the initramfs as I had intended and so on reboot it all came crashing down. This entailed a trip to the customer and several hours to fix because the entire system including the root filesystem is LVM on RAID. Luckily, I was able to boot off an Ubuntu CD and build the RocketRAID kernel module again then start the RAID and then the LVM which finally allowed me to mount the filesystem. After doing this a few times I was finally able to get the initramfs straightened out and things working again. Needless to say it was a long night, but a successful one nonetheless.

A few months ago I learned of an interesting website called myNetWatchman. This is a very interesting and free website which aggregates firewall logs from various sources around the Net (you are free to contribute your logs as well) and analyzes them for trends and potential infections by IP address. System administrators are then able to enter IP addresses of servers they manage into the site and see whether those servers have been exhibiting any malicious behavior towards the monitored firewalls. This is just one additional useful tool for sysadmins to monitor the behavior of their servers.

When I used to maintain a large MP3 collection my absolute favorite tool to organize and “tag” the MP3s with artist, album and title information was Magnus Brading’s MP3/Tag Studio. Thde reason this worked so well for me is because as I ripped CDs I would put the music into folders by album and artist using track number and title for the filename. MP3/Tag Studio allowed me to capture all of that information from the path and put it into the tags. But some time ago I switched from MP3s to the lossless FLAC format. FLAC also allows tagging but tools are fewer and far between. Luckily I’ve discovered an open source tagging tool that looks like it will fit the bill. EasyTAG appears to have many of the same features as MP3/Tag Studio but works with many types of files including MP3, MP2, MP4/AAC, FLAC, Ogg Vorbis, MusePack, Monkey’s Audio and WavPack which is a great advantage in my implementation. It’s also a cross platform application available for Linux, Mac OS X and Windows which is useful for those of us who regularly work with several different OSs.

Not long ago I built a new storage and mail server to replace several aging servers. I knew that to provide space for all my files and a little room to grow I would need a little more than a terrabyte of storage space and that I wanted this to be in a redundant RAID array. I mentioned in a previous post how I created a software RAID + LVM setup for this. The one catch is that my motherboard had fewer SATA ports than I initially thought so I had to leave one drive off the srray while I got things up and running.

A few days ago the PCI Express SATA controller came in so I needed to add the fifth drive to the existing array, ideally without breaking anything. The first few site I checked stated this was not yet possible but after doing a bit more checking I found out that if you have a current kernel, mdadm and LVM2 tools it actually is. I based this on information from the Gentoo-wiki but I did make a few changes for my specific scenario.

Really the only additional information I needed was to copy the partition table from one of the other drives (sdd) to the new drive (sde). “sfdisk -d /dev/sdd | sfdisk /dev/sde”

I also use “lsof /home” to find which processes had open files on the volume before I unmounted it. I also set the stride flag on resize2fs to 16 based on my block and chunk sizes. Apparently getting this correct has a great bearing on speed and efficiency. For the record the stride size should be chunk size (from /proc/mdstat) divided by block size (from the file system). In my case my chunk size was 64k and block size 4k.

Overnight I got loads of “452 Insufficient system storage” errors from one of the mailservers I manage which led me on a little hunt for the problem and a delving into the mystical world of Debian log rotation.

To make a long story shorter it turns out that BIND was filling my /var/log/syslog and /var/log/daemon logs with “sysquery: no addrs found for root NS” entries to the tune of 100s a minute bloating my logs to the point they filled my /var partition completely grinding Postfix to a halt because the chroot jail it runs in is on the /var partition.

Obviously the first step was to fix the BIND error that was filling the logs. This is actually a pretty easy fix and simply involved re-creating the db.cache file which was out of date (probably thanks to the use of generally reliable upstream DNS servers). All it took to fix this was updating the file:

cd /etc/bind
dig @m.root-servers.net. ns . > db.cache.tmp
mv db.cache.tmp db.cache
/etc/init.d/bind reload

The next step was to free up some space on the /var partition. I could probably have just deleted the offending files but because they wouldn’t take up much space when compressed and I hate to screw up the log rotation process I decided to rotate them a few times myself. I could probably have done this manually but didn’t know how the log rotation process worked in Debian and was afraid of throwing it for a loop if I did this so I wanted to do it using the rotation tool itself.

As it turns out Debian ships with a highly configurable log rotation tool (it’s really great you can seletivly rotate based on all sorts of conditions) called logrotate which I believe can originally be attributed to coders at Red Hat. Unfortunatly the standard Debian setup does not actually use it for rotating the syslog file which is exactly what I needed to rotate.

After some digging it turns out Debian uses a kludged together script located at /etc/cron.daily/sysklogd for rotating the syslog. This isn’t the first time that someone has pointed out the sillyness of having such a script as this posting from 1999 I found clearly indicates it’s been a longstanding issue. Why the developers didn’t switch to using the logrotate program for the syslog when they started including it is beyond me. It seems that it would be easy enough to make the changeover unless I’m missing something in which case I would appreciate hearing the concern with doing this.

Of course others have pointed out that the standard syslog daemon leaves a lot to be desired so you could even read about how to change that out to something more configurable at the Debian-Administration site.

About the only time I have a problem with one of my MySQL database servers is if the power goes out at just the wrong time during a database write. Luckily for me MySQL comes with a handy utility called myisamchk which can be used to repair corrupted database tables. Instructions for how to use this utility can be found on this page in the MySQL documentation.

Here’s what I had to do to get WINS name resolution working on my Debian based linux:


apt-get install winbind


Note that the winbind program is part of the samba package and I alrady had installed samba and configured it to point to my local WINS server.

I then had to edit /etc/nsswitch.conf adding the winbind and wins entries:


passwd: compat winbind
group: compat winbind
shadow: compat

netgroup: nis


I can now ping and connect to Windows machines by name.