Category Archives: Server Notes

2TB and growing

About a year ago I built several 1.2TB fileservers for a number of my consulting clients which utilized RAID5 arrays for redundancy with LVM running on top for expandability. One of my cleints which does some media work has exhausted the storage space and called a few weeks ago about expanding the storage space on the server.

The four hard drives in the server now were already utilizing all the onboard SATA-II ports. I certainly could have replaced the drives with larger ones (which I did do for another client) but that would have entailed some careful shuffling of data and wouldn’t provide for much future expandability. For another client who uses space much more slowly I could have added a two port SATA expansion card and added two drives in RAID1 but here I expect to need to continue adding space and so I proposed an external storage tower with a multiport SATA link. I was looking for a PCI Express controller which would support eight drives on a single card and would be supported in Debian Linux. I ended up selecting a Highpoint RocketRAID 2322 which seemed to fit the bill.

As it turns out packaged driver support for Linux is only available for Fedora, Red Hat and SuSE. Luckily I found great instructions at this University of Northern Iowa site for building the drivers from source provided by Highpoint. Although there is some grumbling in the open source community about these drivers being non-free licensed (hence no package from Debian) just about everything else is great. The kernel module built without any problems and without a huge number of dependencies and I was able to get the drives up and running without too much work.

Unfortunately, I did not get the module into the initramfs as I had intended and so on reboot it all came crashing down. This entailed a trip to the customer and several hours to fix because the entire system including the root filesystem is LVM on RAID. Luckily, I was able to boot off an Ubuntu CD and build the RocketRAID kernel module again then start the RAID and then the LVM which finally allowed me to mount the filesystem. After doing this a few times I was finally able to get the initramfs straightened out and things working again. Needless to say it was a long night, but a successful one nonetheless.

Keeping an eye on system security

A few months ago I learned of an interesting website called myNetWatchman. This is a very interesting and free website which aggregates firewall logs from various sources around the Net (you are free to contribute your logs as well) and analyzes them for trends and potential infections by IP address. System administrators are then able to enter IP addresses of servers they manage into the site and see whether those servers have been exhibiting any malicious behavior towards the monitored firewalls. This is just one additional useful tool for sysadmins to monitor the behavior of their servers.

Music Tagging

When I used to maintain a large MP3 collection my absolute favorite tool to organize and “tag” the MP3s with artist, album and title information was Magnus Brading’s MP3/Tag Studio. Thde reason this worked so well for me is because as I ripped CDs I would put the music into folders by album and artist using track number and title for the filename. MP3/Tag Studio allowed me to capture all of that information from the path and put it into the tags. But some time ago I switched from MP3s to the lossless FLAC format. FLAC also allows tagging but tools are fewer and far between. Luckily I’ve discovered an open source tagging tool that looks like it will fit the bill. EasyTAG appears to have many of the same features as MP3/Tag Studio but works with many types of files including MP3, MP2, MP4/AAC, FLAC, Ogg Vorbis, MusePack, Monkey’s Audio and WavPack which is a great advantage in my implementation. It’s also a cross platform application available for Linux, Mac OS X and Windows which is useful for those of us who regularly work with several different OSs.

Expanding the size of a Debian Linux software RAID 5 array

Not long ago I built a new storage and mail server to replace several aging servers. I knew that to provide space for all my files and a little room to grow I would need a little more than a terrabyte of storage space and that I wanted this to be in a redundant RAID array. I mentioned in a previous post how I created a software RAID + LVM setup for this. The one catch is that my motherboard had fewer SATA ports than I initially thought so I had to leave one drive off the srray while I got things up and running.

A few days ago the PCI Express SATA controller came in so I needed to add the fifth drive to the existing array, ideally without breaking anything. The first few site I checked stated this was not yet possible but after doing a bit more checking I found out that if you have a current kernel, mdadm and LVM2 tools it actually is. I based this on information from the Gentoo-wiki but I did make a few changes for my specific scenario.

Really the only additional information I needed was to copy the partition table from one of the other drives (sdd) to the new drive (sde). “sfdisk -d /dev/sdd | sfdisk /dev/sde

I also use “lsof /home” to find which processes had open files on the volume before I unmounted it. I also set the stride flag on resize2fs to 16 based on my block and chunk sizes. Apparently getting this correct has a great bearing on speed and efficiency. For the record the stride size should be chunk size (from /proc/mdstat) divided by block size (from the file system). In my case my chunk size was 64k and block size 4k.

Chaining LVM with software RAID for a scalable server

I recently procured and configured a new storage server and knew I wanted a scalable, redundant way of storing the large amount of data this server will hold. Despite what Steve Gibson has said about the so-called hardware RAID on modern motherboards it’s dangerous! It’s not really a hardware solution and requires that the same propriatary chipset be used to recover the data which is a dangerous thing to rely on when it’s your data you need back.

Sidebar: One of these days I’m going to get around to creating a blog or netcast devoted to correcting the misinformation on TWiT and SecurityNow (I understand that it’s difficult for them to stay on top of all this, but speculating on what you don’t understand in front of hundreds of thousands of people who worship the ground you walk on is a poor choice and seems to happen more often that I think is acceptable). They need a technical editor and fact-checker and since they apprently aren’t doing it in house someone else should.

Anyway, with modern processing power there’s little reason to use hardware RAID in a much other than a large corporate environment (or if you like spending a good sized chunk of money). I’ve had good luck with the Linux mdadm software RAID and have actually been impressed with how well it works. What I have not done is to utilize LVM (logical volume manager) to make it a more scalable solution where drive size can be increased or drives added down the road.

In preparation for this configuration I read a great howto on the topic which you can find on the JerryWeb Wiki. Things went quite smoothly and I’ve been very happy with the configuration so far which includes four SATA II drives (with working hot-swap capability in a drive cage) in a RAID 5 array. As soon as I get another AHCI SATA controller in I’m going to be adding another drive to the array so we’ll see how easy the LVM makes that!

Adding Greylisting and SPF support to Postfix

For quite some time now I’ve been running my own Linux (Debian w/ Postfix) mailservers. For the past several years I’ve had good luck basing my installations on the fantastic instructions available at workaround.org but this summer even these systems were failing to filter a lot of my spam. Naturally, I went looking for other anti-spam technologies I could add.

For a few years now I’ve published SPF records for my domains and I’m a strong believer that wider implementation of SPF would greatly reduce the amount of spam using forged addresses. Anyway even though I had been publishing SPF records I had not gotten around to implementing SPF checks on my own server so this was one thing I was looking for.

Another was that I had heard a little bit about something called greylisting. Greylisting is a process by which mail from unknown senders is initially bounced with a “service temporairly unavailable” message but when the remote server tries a second time after waiting some period the message is allowed through. This works on the premise that real mail servers which comply with the mail RFCs will keep retrying to send a message until it goes through or a hold timer expires (usually several days) but spamming programs (often trojans on unsuspecting users’ systems) will only try once. Obviously this won’t stop all spam, especially that from legitimate companies using legitimate spam servers and it could easily be bypassed by the trojan writers by trying to send a message several times. I beleive the latter has not happened because of the additional processing overhead this would create and so far it’s simply not efficient for spammers to track all this.

Eventually I settled on implementing both SPF checking and greylisting based on this guide. The guide actually contains a full howto on setting up a Debian/Postfix mailserver similar to the workaround.org guide I mentioned before. I have only briefly glanced at this other information as I already had a working mailserver but I can say that the method they propose is quite similar, but not entirely the same as the workaround.org method.

Since implementing these changes my own mailserver has been rejecting much more spam (without any increas in false positives) than before. When I temporairly turned this off to migrate to a new mailserver I immediately saw a marked increase in spam getting through. While I have not tested it extensively it is my belief that more is being stopped by the greylisting than the SPF, mostly because many domains do not yet publish SPF records (though several large ISPs which are commonly spoofed now do).

Why does Debian use antiquated syslog rotation?

Overnight I got loads of “452 Insufficient system storage” errors from one of the mailservers I manage which led me on a little hunt for the problem and a delving into the mystical world of Debian log rotation.

To make a long story shorter it turns out that BIND was filling my /var/log/syslog and /var/log/daemon logs with “sysquery: no addrs found for root NS” entries to the tune of 100s a minute bloating my logs to the point they filled my /var partition completely grinding Postfix to a halt because the chroot jail it runs in is on the /var partition.

Obviously the first step was to fix the BIND error that was filling the logs. This is actually a pretty easy fix and simply involved re-creating the db.cache file which was out of date (probably thanks to the use of generally reliable upstream DNS servers). All it took to fix this was updating the file:

cd /etc/bind
dig @m.root-servers.net. ns . > db.cache.tmp
mv db.cache.tmp db.cache
/etc/init.d/bind reload

The next step was to free up some space on the /var partition. I could probably have just deleted the offending files but because they wouldn’t take up much space when compressed and I hate to screw up the log rotation process I decided to rotate them a few times myself. I could probably have done this manually but didn’t know how the log rotation process worked in Debian and was afraid of throwing it for a loop if I did this so I wanted to do it using the rotation tool itself.

As it turns out Debian ships with a highly configurable log rotation tool (it’s really great you can seletivly rotate based on all sorts of conditions) called logrotate which I believe can originally be attributed to coders at Red Hat. Unfortunatly the standard Debian setup does not actually use it for rotating the syslog file which is exactly what I needed to rotate.

After some digging it turns out Debian uses a kludged together script located at /etc/cron.daily/sysklogd for rotating the syslog. This isn’t the first time that someone has pointed out the sillyness of having such a script as this posting from 1999 I found clearly indicates it’s been a longstanding issue. Why the developers didn’t switch to using the logrotate program for the syslog when they started including it is beyond me. It seems that it would be easy enough to make the changeover unless I’m missing something in which case I would appreciate hearing the concern with doing this.

Of course others have pointed out that the standard syslog daemon leaves a lot to be desired so you could even read about how to change that out to something more configurable at the Debian-Administration site.

Recovering from an ext3 hard drive crash

If you run a Linux system and have had the misfortune of having your hard drive crash without a recent backup or RAID configuration you’re in for a world of hurt. Thankfully, there are some free utilities that might make things go a bit easier for you.

My favorite utility is a gem called dd_rescue. This program will work quite hard (and efficiently) to recover as much of your data as possible onto another hard drive or a disk image. It does this by reading smaller and smaller block sizes, isolating as much of the good data as possible before copying it off. Of course this program can’t bring back data that is in the bad area itself, but it does salvage everything it can. I’ve used it a couple of times and one time it worked pretty well, recovering all the important data and the other time I got almost nothing usable. If you need a quick and dirty attempt to get things going again as quickly as possible this is your ticket.

Repairing MySQL tables with myisamchk

About the only time I have a problem with one of my MySQL database servers is if the power goes out at just the wrong time during a database write. Luckily for me MySQL comes with a handy utility called myisamchk which can be used to repair corrupted database tables. Instructions for how to use this utility can be found on this page in the MySQL documentation.

WINS Name Resolution for Linux

Here’s what I had to do to get WINS name resolution working on my Debian based linux:


apt-get install winbind

Note that the winbind program is part of the samba package and I alrady had installed samba and configured it to point to my local WINS server.

I then had to edit /etc/nsswitch.conf adding the winbind and wins entries:


passwd: compat winbind
group: compat winbind
shadow: compat

hosts: files wins dns mdns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files

netgroup: nis

I can now ping and connect to Windows machines by name.