Category Archives: Tips & Tools

Creating Bootable USB Drives

Posted by on February 21, 2014 1 comment

One of my extremely popular posts over the years has been my 2007 instructions for manually creating a bootable DOS USB flash drive. That article presented two possible ways to create a bootable DOS USB drive which appeared as a fixed disk (e.g. C:) with the whole drive available instead of as an emulated floppy disk (e.g. A:) with only 1.44MB available. Both methods were pretty complicated, requiring you to extract source files from the FreeDOS kernel and have a good working knowledge of the command line before attempting. While many people were successful and thanked me profusely for the instructions many others had difficulty following the technical detail.

Luckily, seven years later, things have improved a bit. These days it’s perhaps a bit less likely that you want to boot a USB drive to DOS. The widespread use of NTFS, advent of more user friendly Linux tools on bootable CDs, migration to Windows PE based recovery utilities, etc. has somewhat limited the DOS utility market. Today’s problem seems mostly to be working with computers which no longer have CD/DVD drives in them but having a CD ISO file you want to boot.

The best multi-purpose utility I’ve found for creating bootable USB drives is, hands down, Rufus. Which allows you to create bootable DOS USB drives in Windows in much the same way that my old instructions did but also allows you to turn bootable ISO images of CDs and DVDs into bootable USB drives which is more than likely what you want to do anyway. The source code for Rufus is available from GitHub.

Using Google Apps on iGoogle

Posted by on October 29, 2009 1 comment

I love Google Apps, the sevice which hosts email, calendaring, online docs and chat for your own domain name free of charge at Google. Sure there are some disadvantages and I still don’t like working in the Google Docs word processor too much but the email and calendaring are great. I also love the iGoogle service where you can customize a homepage with RSS feeds and include a Gmail and Google Docs module.

The problem is that internal communication at Google between these projects is pretty much non-existant, at least it appears that way. iGoogle integrates wonderfully with the consumer Gmail and Google Docs service but not at all with the hosted Google Apps service which is becoming even more popular as companies, particularly universities and colleges, around the world start to adopt it. One of my customers was particularly annoyed that documents created with their hosted app couldn’t show up on their iGoogle page and asked me to find a solution. After much frustrations the best solution I could come up with is a series of hacked together iGoogle apps which replicate as much of the functionality as possible from the official Gmail, Calendar and Google Docs iGoogle apps.

1) Google Docs for hosted domains iGoogle gadget

2) Google Mail for hosted domains iGoogle gadget

3) Google Talk for hosted domains iGoogle gadget

4) Google Calendar for hosted domains iGoogle gadget

5) Google Tasks for hosted domains iGoogle gadget

Of course, this is really just a workaround and doesn’t do as nice a job of integration as the regular apps do. In related news the Google account authorization is a mess because of this as well. For example, you can log into the iGoogle site (and other Google services) using a non-gmail address which is linked to a regular gmail account but this is confusing if you also have a hosted apps mail account on that domain. A mess all around. Google…if you’re listening you really need to make some improvements to how your hosted apps service integrates with the rest of your services and sites!

Making Relational Databases Hierarchical

Posted by on October 29, 2009 Comments Off on Making Relational Databases Hierarchical

I’m not one who really enjoys writing code. In fact, I tend to avoid it whenever possible as it’s just not that interesting to me. That said from time to time I have projects to complete where no existing software quite meets my needs and writing my own makes the most sense. This summer I was in the midst of collecting information and publishing a large extended family directory.

Clearly, the best way to do this is to store the data in a database. Because relational databases are most prevalent, I already have quite a bit of experience with MySQL and I knew I wanted to eventually make the data available via a PHP web application I decided to go with a MySQL backend. The trick here is that families aren’t as relational as you might originally think. Families are really hierarchical data (parents, children, grandchildren, etc.) and relational databases have a hard time storing and recalling things based on these types of relationships. For example, a typical family query might be one for a list of all descendants of someone. As it turns out families aren’t the only hierarchical data people might try to store in a relational database. Any business database which shows managers and reporting personnel is also a hierarchical data set albeit usually with fewer levels and much less complication (no need to track divorces, remarriages, spouses, etc.)

I was able to find and read up on several methods for storing the relational information but the one I ended up using is that proposed by Rob Volk of SQLTeam where the expensive (computationally) hierarchical relationships are stored in two additional fields for each record (depth and lineage) with parseable separators between generations. The database is initially loaded through a series of several (expensive) recursive queries but then can be maintained through triggers when information is added or updated.

The Use of S3 and EC2 for Remote Backup

Posted by on December 27, 2008 3 comments

Even before the introduction of the Amazon S3 storage service I was intrigued bye the possibilities of secure backup over the Internet. Over the years I’ve evaluated a number of possibilities such as the use of rsync and Unison either to my own remote servers or to a service. I’m really not too interested in the commercial vendors as most of their software works on Windows or maybe Mac and my files reside on a Linux fileserver. It only makes sense that my backup solution should run on the Linux server as well.

None of these solutions seemed to quite fit the bill for me because of expense, concerns about data security or speed. Since the introduction of S3 I have started playing around with some of the scripts and software which have been developed to take advantage of these powerful services. I was still disappointed though mostly because of some data encryption concerns (on the storage system, not in transit) and the potential charges associated with backing up data to the S3 service. Ideally I would want something rsync like which would only transfer the changed parts of the files instead of recopying the entire file or directory. Unfortunately there is no built in support for anything like this in the low-level S3 system. So after playing with many scripts that suggested they would be able to do something along these lines and remaining unimpressed I decided to put things on hold for a while longer.

Eventually Amazon released the EC2 cloud computing platform but that still didn’t seem particularly useful for my purposes because of the lack of persistent storage between sessions. Once the elastic block storage became available things got more interesting. Now that I could retain data between sessions I had visions of a backup script which would launch an EC2 instance, mount an EBS volume and run rsync or Unison to backup directories on my local server to the remote site. I started playing around with EC2 and soon discovered that although it is very powerful it is a monster to control unless you are writing your own application from the ground up. For a simple job like this that should be easily accomplished by a script it can be a nightmare with several shell variables to set and paths to keep straight. Never mind the several encryption keys and the changing SSH host identifier to deal with. Eventually with some help from two fantastic blog entries (Ereblog and Free Wisdom Online) I was able to get something working…mostly.

It’s quite a fragile thing and you have to make sure that things are executed in the correct paths and with the correct environment variables set. In addition the returned data from the control commands is just awked from the output so it could easily break if the control package were updated, etc. The final nails in the coffin for me were my increased backup storage requirements for photos, audio and video which are huge and can change the economics of doing remote backup quickly. Even for a slimmed down set of documents I found the process to be too slow and fragile for my needs. In the end I have gone back to hauling hard drives with data backups off site and using the rsync program locally to sync these periodically with my live storage.

*Edited 2/2/09 to fix the several times I mistakenly called EC2 EC3 although I knew better. Thanks to the commenter for pointing this out!

Configure Windows XP Network Settings from the Command Line

Posted by on January 21, 2008 Comments Off on Configure Windows XP Network Settings from the Command Line

Once upon a time batch files were king in the PC world. Hardly a magazine issue, BBS discussion or user group meeting would go by without one of these handy scripts to add some functionality or usability to systems. Since the reign of Windows batch file programming has been on the decline. Of course shell scripting remains popular and extremely popular in Linux where most settings on the system are still controlled by text configuration files and command line utilities but in Microsoft land the script has been largely supplanted.

Even though the NT based operating systems (NT, 2000, XP, etc.) have actually made significantly more configuration available from the command line and there was a push for a new ‘Windows Scripting’ language these things have become largely forgotten and there are now an extremely limited number of users comparable to the Windows operating population which are comfortable writing scripts to automate things in Windows. Even among large corporate IT departments where there is perhaps the most to be gained by writing these sort of mini-utilities scripting is a dying art. One of the reasons for this is that most magazines and technical publications no longer regularly mention scripting or command line configuration utilities so there is a limited opportunity to learn about these tools.

Nevertheless these tools exist and when you find them they can be extremely useful. Take the “netsh” program for example. This handy little tool allows you to set and manipulate many of the Windows network settings from the command line and when combined with scripting it is possible to create scripts which will completely reconfigure your network interfaces (say from DHCP to a static address to another static address) for various network configurations all with the simple execution of a script.

You can learn more about this powerful tool from a few different Microsoft sites but while these provide some syntax and information perhaps the best place to get started is at one of the third party sites which covers it. Or now that you know about the utility you could just start experimenting with things.

For example running the “netsh -c interface dump” command will dump all kinds of interesting information about how your interfaces are currently configured to the prompt. It’s possible to capture this information to a text file and then ‘replay’ the data to reconfigure things as they currently are using something like “netsh -f netsettings.txt”

Keep the art alive!

Booting DOS from a USB flash drive

Posted by on August 21, 2007 34 comments

**UPDATE 2014/02/21: Times have changed and you might want to check out the Rufus utility I mention here as an alternative to these technical instructions.**

USB flash drives also called USB keys, pen drives and an assortment of other names can make quite handy reusable boot disks. Today I found myself in need of a DOS boot disk to upgrade the firmware on some hard drives and CD/DVD drives but didn’t want to burn a bunch of bootable CDs or try to find a USB floppy disk drive. I remembered that most newer BIOSs including the one on this system support booting from USB drives so I thought I would investigate that as an option. Unfortunately I found that, being an afterthought, boot support is not an easy thing to do and presents several challenges.

First and foremost is that while there is some information on how to accomplish this on the Internet, there is comparatively little and what information does exist is not as clear or definitive as would be useful. Secondly, there are a number of different methods and tools for accomplishing this and not all methods work with all BIOS implementations of bootable USB. For example, USB flash drives may be booted as fixed disks, floppy drives or USB-Zip drives each of which requires different methods of preparation and the requisite support in the BIOS. Both of the methods I will describe hear treat the flash drive as a fixed disk which seems to be the best method if your BIOS supports it and appears to be becoming the standard for new BIOSs. While these methods are based on readings of other guides and howtos I was unable to find something as simple as I describe here so this method was developed on my own though research based trial and error (and lots of reboots) over the better course of a day.

In its simplest description booting from a flash drive as a fixed disk works almost exactly like booting from a hard disk does. The BIOS invokes the boot sector and master boot record (MBR) on the flash drive which loads the operating system kernel. It would initially seem that it should be no problem to make this work, after all DOS based operating systems worked this way for years. The trick is getting the boot sector and MBR on the flash drive. In ye old DOS days when you wanted to install DOS from a floppy disk onto a fixed disk it was common to invoke the FDISK and SYS commands to create a MBR, boot sector and copy the required system files. The problem is that in most cases the USB flash drive is being prepared from within a recent copy of Windows such as Windows XP which no longer has these commands available for this use. On the other hand if you booted DOS from a floppy disk or bootable CD and had access to FDISK and SYS you would not (normally, without drivers) have access to the USB flash drive to install the files.

After reading the information that was available on the Internet I determined it would be reasonably easy to create a bootable DOS USB flash drive in Linux and possible, yet convoluted and confusing to do so from Windows. I wanted to avoid requiring the use of Linux because the average user of such a drive may not have access readily available to a Linux system. Most of the solutions for creating the drive in Windows either used a creation utility from HP (questionable availability and suitability) or a slew of command line utilities and requiring a floppy drive (or emulator) which seemed like an unnecessary and complicated hack to me. The solutions I present may require you to download a few software packages from the Internet but each only requires one command line utility and should be fairly straightforward. As an added bonus all of the software is free and open source.

Note that these methods were specifically designed for installing FreeDOS, an open source DOS. Similar methods may work for installing MS-DOS, DRDOS or other DOSs; however you will need to obtain the boot sector (probably either from source or via extraction from a floppy disk or disk image) and system files specific to your version of DOS. I recommend using FreeDOS whenever possible as it is generally compatible and provides many additional features not found in vintage DOSs.

The first method described is the FreeDOS direct booting method. The advantages of this method are that the drive boots directly into FreeDOS and requires no files on the flash drive root other than the FreeDOS system files (kernel.sys and command.com). The disadvantage is that you must download an additional software package and FreeDOS is the only OS you may boot from the flash drive.

The second method is the SYSLINUX chained booting method. Advantages to this method include more configuration and customization options and the ability to boot floppy disk images and/or other OSs from the same flash drive using a boot loader menu and chained boot loading. None of these enhancements are covered here, this document will only help you get FreeDOS up and running, for information on booting other OSs from the same drive see the SYSLINUX documentation. The disadvantages of this method include three additional files in the flash drive root (can be moved into other directories, see SYSLINUX documentation) and a slightly more complicated (though transparently so) boot process.
Read more »