code in the ruins

I just think it's really interesting that Oracle is buying BEA and Sun is buying MySQL, and that both acquisitions were announced on the same day.  That is all.

This weekend I finally did something I've been meaning to do for a while - I installed the dd-wrt firmware on my Linksys WRT54G wireless router. dd-wrt is based on the original firmware from Linksys, but it adds a ton of new features.

Since installing it three days ago, I've done various things that I couldn't have done before:

• Static DHCP leases for my machines, so they always get the same IP address. This also makes the router act as a DNS server for free.
• SSH access to the router
• WPA2 wireless security
• Increased the transmission power of the router's wireless signal
• Set up NTP to keep the router's time current
• Set up IPv6 on my home network using 6to4. The router will automatically assign IPv6 addresses to all clients.
• Mounted a Samba share on the router

I love doing this kind of thing. dd-wrt is a fantastic piece of work, and has increased the value of my router a great deal.

The most satisfying bit was the IPv6 setup.  I find it kind of nifty that simply having a single IPv4 address entitles me to several quintillion contiguous IPv6 addresses. You know, just in case.

In the evening on Monday, December 31, my venerable home Linux server, obelix, went down for the last time.  His boot drive developed a bad block and could no longer be mounted.  Efforts are still underway to recover the data from the drive, but on Tuesday, I made the decision to decommission obelix for good.

obelix was a Dell Dimension L667R with a Pentium III and 384 megabytes of RAM.  I've had him for about seven years.  I bought him (refurbished!) on a whim, somewhere in the distant past.  He was extremely versatile; he acted as web, file, and database server, as well as providing services to our network as a Samba domain controller and an LDAP, DHCP, and DNS server.  In his early years, he was also my primary workstation, running countless window managers, desktop applications, and games over the years.  Due to his having a FireWire card installed, I used him to capture DV from my video camera to edit on another machine.  He spent most of his lifetime as a Gentoo Linux box, no doubt logging thousands of hours of software compilation.

Until recently, he served my websites over my home connection, although since I recently began hosting with Dreamhost, he no longer performed this function.  This was a major factor in my decision not to stand him back up after the disk failure.  Although he had still been providing network, file, and database services, I knew that I would no longer need a dedicated server at home once the websites had been moved to external hosting.  So I moved his large data disk to another computer and dismantled the Samba domain, which was more or less unnecessary to begin with.  His primary function as a database server was to host my Amarok music collection database, which was easily rebuilt after he went down.

I will miss having obelix around, though; I named him after my favorite Asterix character, I've had him longer than I've known my own children, and I knew him backwards and forwards.  Tinkering with Gentoo, Apache, MySQL, OpenLDAP, and Samba on obelix was how I began to truly solidify my knowledge of Linux system administration; using obelix absolutely helped get me where I am today.

It's somehow fitting that he should go out on the last day of 2007; it's been quite a year.  There's been a lot of anxiety, some deaths, a birth, some sickness, some health, some successes, some failures, a new job, many highs, many lows.  It hasn't been the easiest year to live through in a lot of ways.  In no way is the loss of obelix anywhere near a significant event in the face of what's happened this year, but it does sort of reinforce the feelings I have about 2008 - cautiously hoping for a clean slate.  The new year always brings new things and does away with some of the old.  Whether that's for good or ill, we'll just have to wait and see.

Today I registered a new domain - mogrify.org - and began the process of migrating my websites to my new host, Dreamhost. The now-venerable mogrify.homelinux.org, as you may have noticed, redirects to the shiny new code.mogrify.org. I'll be moving other things across in the coming days and fixing problems whenever I notice them.

I've been wanting to host offsite for some time, since I'm currently running three separate sites on a single, beige Pentium III Dell box in my study. This is not exactly the most robust of setups, and there have been issues with power and network outages.

I expect to love Dreamhost for the same reason I love hosting at home - because I genuinely enjoy administering Linux systems, and Dreamhost gives you a lot of control - shell access, .htaccess files, log files, email accounts, etc., as well as a whole ton of other options.

So, the dream is becoming a reality.

• Desktop
• Network shares
• NTLM
• Exchange server (mail, contacts, calendar)
• Remote desktop
• Printing
• Limitations

At home, I use Linux. But at work, I have to use Windows. They don't support Linux desktops, and that's not all: our network has a lot of Microsoft services running that don't play nice with Linux. Still, I've managed to set up a Linux box at work; and slowly, I've managed to get enough things working that I only rarely need to go back to Windows to get something done. Details of what I've worked out are below.

The main goal here is to see what's possible without any assistance from the IT department whatsoever, since that's how most people would be doing this. Fair warning, though: if you violate your company's IT policy or break something you can't fix, you're going to be in a sad, lonely place.

First, some background on what I'm looking to do. I'm a web developer. I have access to several servers where I work - some Windows, some Linux. I need to be able to administer them, and I need to be able to modify files on them. I need to be able to access our Exchange server - mail, contacts, calendar. I need to be able to access our Intranet, which uses NTLM (a.k.a. Integrated Windows Authentication). And there are certain pieces of proprietary software that I have to use. As I said, I haven't gotten everything to work, but I've put together a pretty good setup that solves most of these problems.

Linux on the desktop

The first step is to install Linux onto a workstation. The first option is to just nuke a workstation's hard drive. Non-destructive partitioning is also possible, and you might be able to put a new drive into the machine and dual-boot. Or bring in your own Linux box to use at work. Many of these things will get you into trouble with your IT department, and they may not even be possible if they have password-protected the machine's BIOS or have some means of preventing unauthorized PCs from connecting to the network.

I took option one - nuking a workstation's hard drive. My IT folks can restore a standard drive image, so this seemed like a minimally invasive way to go. So I kissed it goodbye and installed Ubuntu (although I eventually ended up using Kubuntu instead, because of Kontact. But more on that later).

Once I had a running Linux box, I started hooking up the network stuff.

I had to abandon the idea of logging into the Linux box with my Windows domain credentials. It requires that the machine be added to the domain, which requires a domain admininstrator. I created a local account on the machine to use.

Network shares

Thanks to the excellence of Samba, geeks everywhere can access Windows shares flawlessly. Or, almost. Initially, I used Gnome to connect to the various Windows shares I need, and everything was fine. But I began to notice an annoying problem - when opening a file on a Windows share in GEdit, it would tell me that it couldn't access the file. Only after trying again would it open the file. Apparently this is a bug in gnome-vfs. But it was sufficiently troubling that I went looking for a better way to access files on network shares.

I settled on autofs. Autofs will automatically mount a list of, well, anything, somewhere on the filesystem. It has a nifty 'ghost' feature that makes the shares visible, but does not mount them until they're accessed. I listed out my network shares, and now they're all available under /mnt/auto whenever I want them.

The details:

sudo apt-get install autofs smbfs

In /etc/auto.master:

/mnt/auto /etc/auto.misc --timeout=3600 --ghost

In /etc/auto.misc:

share -fstype=cifs,rw,credentials=/path/to/smb.auth,uid=user,gid=grp,file_mode=0644,dir_mode=0755 ://server/share
hidden -fstype=cifs,rw,credentials=/path/to/smb.auth,uid=user,gid=grp,file_mode=0644,dir_mode=0755 ://server/hidden\\$

In /path/to/smb.auth (make sure this file is chmod 600):

username=domainuser
password=domainpassword

Autofs will automatically create the directories /mnt/auto/share and /mnt/auto/hidden. When you (or your applications) access these directories, they'll be mounted and the file ownership and permissions will be set as specified.

The only drawback: it doesn't appear to be possible to mount directories within a share directly. You have to mount the root of the share.

NTLM

Our Intranet uses NTLM for authentication to provide personalized features. Good thing Firefox supports it. One authentication box and you're set until the browser closes.

Exchange Server

We use Exchange Server 2000 and Outlook 2000 where I work. The obvious choice of client for working with Exchange servers in Linux is Evolution, which has had Exchange interoperability for some time now. Unfortunately, I've never gotten the Exchange connector to work. It's possible that our server has WebDAV disabled, or that the authentication box that pops up when I visit my Outlook webmail is getting in the way. At any rate, I had to drop Evolution and look for something else.

I'm able to access the Exchange server via IMAP and retrieve my mail that way. Not perfect, but it works. The calendar (especially meeting request handling) and the user directory are more problematic.

First, the calendar. I first tried Thunderbird with the Lightning extension. It worked reasonably well, but I noticed that it appeared to be getting the time zone of my meetings wrong. When I accepted meeting invitations, the appointments would be scheduled several hours away from when they were actually supposed to occur.

Next, I tried Kontact, which turned out to work very well. For mail, I use IMAP to connect to the server. Kontact can also work directly with Exchange calendars - when I added a new calendar, Exchange was one of the options (though it says it is experimental). Happily, I saw that it added all of my current appointments, complete with notes, other participants, reminders, etc. Creating and editing events worked fine. Meeting requests received by mail have response links, and the meetings are scheduled on the calendar as you'd expect. The only problems I've encountered so far have been performance-related; there are long pauses while communicating with the server, and Kontact has very occasionally become unresponsive.

For the user directory, I was able to get Kontact to connect to our backup domain controller via LDAP. It works well, except there is a hard limit of 1000 records, and Kontact doesn't seem to be able to retrieve more than that. As a result, there are some employees who do not appear in Kontact's local version of the directory. We don't have much more than 1000 employees, so it's not a huge problem.

I should say that I'm a little privileged there. As a normal user, I wouldn't be able to access Active Directory and retrieve records via LDAP in our shop. I happen to have a user account with more abilities that is used by the web-based employee directory that I wrote. However, I believe that most domain controllers will allow a domain user to list records. Could be wrong though.

With mail, appointments, and contacts all managed by Kontact, the integration is quite good - as I've said, meeting requests are handled well, and contacts are conveniently suggested as you compose a new message. It's a very usable setup.

Remote desktop

Obviously, administering a Linux server from Linux via SSH is no problem. Windows servers are easy too, thanks to krdc and rdesktop. Both of these clients make it easy to log in to Windows boxes and do what you need to do.

Printing

We use HP network printers here, which can be accessed directly via TCP/IP on port 9100. They work flawlessly.

Limitations

The only reason I can't drop Windows entirely at this point is because of proprietary software that I still need to use.

As a web developer, I have to use Internet Explorer. It makes me cringe, but I have no choice - when 70% of our visitors still use it, it means I have to test everything in IE before it goes out. I've tried IEs4Linux (IE on Wine), but it was almost unusable - I couldn't type addresses in at all, though I could navigate with the mouse. Not good enough for rigorous testing.

Also, we use Dreamweaver to maintain site templates. Dreamweaver doesn't run on Wine because of its license enforcement component, so I still have to go back to Windows to modify templates. Soon we will be migrating to an open source content management system with a web interface, so I'll be able to drop Dreamweaver for good.

And finally, I need to use either Enterprise Manager or SQL Server Express to work with SQL Server databases. I haven't found anything that runs on Linux and still does everything I need, so it's back to Windows for those tasks. We'll be moving to Oracle gradually, which isn't my first choice but has a better Linux presence since its tools are written in Java.

And that's how I carved out a little Linux niche in my Windows shop.

Update: smbfs is required for mounting Windows shares using autofs.

How do you get into a Windows box when:

1. all user accounts are locked out,
2. the Administrator password is unknown,
3. the box has no working CD drive, and
4. you're too cheap to buy any extra hardware to boot from?

The answer, of course, is to spend all weekend learning how network booting works. I had the pleasure (yes, pleasure, for I am weird like that) of this experience. I Googled much, and tried many things that didn't pan out. I read many guides to PXE booting, TFTP, and such things.

I found the Offline NT Password and Registry Editor, which provides a handy boot disk. After getting it to boot with PXELINUX, I found that it would refuse to mount a "dirty" NTFS partition as writable. Even though it forced Windows to run a disk check, it still didn't clean up the drive, so there was no way for it to get in and change the Administrator password.

I remembered reading about NTFS-3g, the recently released NTFS filesystem driver for Linux, and how it was far more stable than older attempts to deal with NTFS volumes. It's included in Knoppix now, so I firmly resolved to figure out how to boot Knoppix over the network.

By this time I had a fully working network boot setup (which involves a combination of DHCP server, a TFTP server, and the PXELINUX bootloader). Knoppix also requires an NFS server thrown into the mix - although the NT password boot disk is entirely contained in RAM, Knoppix is normally not, so there needs to be a network location where it can find its goodies. Luckily, I found an excellent guide over at BabyTux, which is why I won't write a full one here. After a little tweaking of the Knoppix boot options, I was watching Knoppix's familiar colorful boot sequence.

On to the matter at hand - resetting the Administrator password to allow access to the machine. With Knoppix's support for NTFS-3g, I was able to mount the NTFS partition as writable (although it complained that it was dirty). I downloaded the source code for the NT password utility, but it wouldn't build - for one, Knoppix doesn't ship with OpenSSL headers, and there are also some deprecated techniques in the code that were causing GCC 4 to give up. I don't know much about C, but I did find a diff that someone had posted for this problem. I built the utility on another box, dropped it in Knoppix's NFS share, and ran it. It worked.

So, the utility successfully changed the Administrator password and unmounted the volume. The fact that the partition was mounted dirty didn't seem to bother Windows - it booted, chkdsk'd, and rebooted. A quick F8 during boot and I was able to log in as Administrator. And the rest is history.

I recently put together a new website, HenricoCrime.org. It's a crime data/Google Maps mashup in the style of ChicagoCrime.org and RichmondCrime.org.

The crime data comes from the Henrico County, VA police department website… the heavy lifting is handled by a set of PHP classes that search the site and scrape the HTML for event data. I use Yahoo's geocoding service to find the latitude and longitude of each event. I first tried Google's but I found that it guessed wrongly too often… Henrico County has several different localities, and I have no way of knowing which one I'm looking for. Google will return hits on similar street names in the wrong places, where Yahoo is more strict. I then store the whole mess in a MySQL database on my server.

All that happens on the backend - the website itself just queries the database and assembles a Google Map with info markers for each event. I also generate some basic statistics from the database for each day.

The other interesting bit is The Cloud. I was trying to think of interesting ways to display trends over a long period of time… The Cloud loads hundreds of crime events at once and marks each one with a tiny, nearly transparent dot on the map. As events stack up in the same place, the marks become darker. So if you pull in a few thousand events, you can see where much of the police activity is happening. Generally, it seems like many of the events cluster along the main roads in suburban Richmond. The best places to be, crime-wise, appear to be Glen Allen (the area in the North right around 295), and eastern Henrico (which is mostly rural).

I'd like to keep coming up with different ways of looking at the data; for instance, weekly, monthly, and annual statistics, breakdowns by crime type, RSS feeds, etc. Most events have more data than I'm actually displaying here, so there should be some other possibilities.

Oh, and I also recently completed a site for my aunt, who runs Connections Speech-Language Therapy in Boerne, TX. So, shameless plug there.

Been trying out Opera 9 - great browser. Probably not as versatile as Firefox when you factor in the extensions, but it's fast, sexy, and highly polished. Where it particularly shines over Firefox is in the memory footprint - I've been using both browsers all day, and Firefox is right up around 173MB, with Opera sitting at a cool 47MB - and that's with the mail window open. Unscientific, yes. But you can feel it - with lots of tabs open in Firefox, the computer's not as responsive. So I'd like to use Opera more.

But it's lacking a killer feature for me - SOCKS support. When I get to work every morning, I open an SSH session to my home server. It sets up a dynamic tunnel that Firefox can use as a SOCKS proxy. And that ensures that I'll never be asked about "excessive Internet usage" - all they'd see would be a bunch of nonsense being sent over a nonstandard port. Not that I'm doing anything wrong; I just feel better not being monitored.

But anyway, this won't work with Opera, because it doesn't support SOCKS proxies. And that's kind of sad - or was, until I figured out a workaround. I installed tinyproxy (I like tiny), but it could just as well be Squid or Privoxy or some other HTTP proxy. It's set up on my home server. To prevent outside connections, I changed the "Listen" and "Allow" directives to "127.0.0.1". And I set up another tunnel in SSH - not dynamic, just a local tunnel from my work desktop to the home server on tinyproxy's port. And then I told Opera to use localhost as the HTTP proxy server.

Still, it'd be cool if Opera came out with SOCKS support - it'd probably be useful to some corporate IT departments in addition to the usual gang of tinfoil-hat types. A big problem with a setup like this is that DNS requests get leaked - they aren't passed through the proxy, so if those are monitored, it's still possible to guess what content you're viewing. With native SOCKS support, it would be possible to pass ALL network traffic through the proxy - HTTPS, DNS, email, etc.

The other option is something like FreeCap, which hooks into the Windows network stack to redirect connections through the SOCKS server. I haven't tried this one yet, but I've tried the non-free SocksCap and found that it made things pretty unstable. Maybe I'll give it a try…

Update, July 6: FreeCap works really well, much better than SocksCap. Score one for open source. So there's no need to do all the crazy server-admin stuff with tinyproxy. Unless you want to

One of my duties at work is maintaining and extending our Intranet site, which is written in classic ASP and VBScript. I consider this an unfortunate situation, because I don't like ASP or VBScript. Or IIS, or Windows for that matter. But there it is.

Recently I fed a troll on Slashdot in a thread about ASP and PHP. Another comment on the same thread mentions that you can use other languages than VBScript and JScript in ASP pages if you want - he mentions Ruby, Python, and Perl, and PHP and Lua are available as well. That reminded me that this was something I always wanted to look into. So I installed Python, Ruby, PHP, and Perl. I haven't gotten PHP working in IIS yet, but the others work quite well.
I wrote some simple pages to test each language, and then I started experimenting more with Ruby. It takes some getting used to to program in this environment; you have to remember to use Response.Write instead of puts, for instance. The script's working directory is c:\windows\system32 instead of where the page actually lives. But it's incredibly nice to have Ruby's large class library and syntactic sugar. You have the choice of doing things through OLE (i.e. Server.CreateObject("Microsoft.XMLHTTP")) or through native Ruby (i.e. Net::HTTP.get(…)).

It gets complicated when you're trying to convert properties from the ASP built-in objects (Request, Response, Server, etc.) to Ruby objects. Everything provided by ASP is a WIN32OLE object in Ruby. So, unless I'm missing something, I'm going to have to deal with converting whatever I need from ASP to Ruby before I can use it.

An example: I said before that the initial working directory of the Ruby script (Dir.getwd) is c:\windows\system32, I guess because that's where the DLL lives. In ASP, you can get the directory the script is in from Request.ServerVariables("APPL_PHYSICAL_PATH"). But you can't do Dir.chdir(Request.ServerVariables("APPL_PHYSICAL_PATH")) because that property is a WIN32OLE object, not a string. And its to_s method just returns its instance.

In fact, none of the methods you can call on that object return its value, and although you can iterate with Request.ServerVariables.each, all you get are the names of the items in the collection, not their values.

It took some experimenting, but eventually I figured out a way to get to the value. I was thinking of it like a hash, but ServerVariables is sort of a collection of collections. Each collection in the collection has one item in it: a Ruby string containing the value of the variable. But a hash would be easier to work with, so I wrote a mixin to return a hash representation of ServerVariables:

<%@language="RubyScript"%>
<%
module W32Helpers
def to_hash
if not is_a? WIN32OLE then return nil end
hash = Hash.new
each do |key|
value = nil
Item(key).each do |member|
value = member
break
end
hash[key] = value
end
hash
end
end

sv = Request.ServerVariables
sv.extend(W32Helpers)
Dir.chdir(sv.to_hash['APPL_PHYSICAL_PATH'])
Response.Write Dir.getwd # sweet.
%>

This should work on any WIN32OLE object that is Enumerable. I imagine there's a way to check if that's the case, but I havent found it yet. And it's only one-dimensional. An interesting aside: chdir appears to be application-wide and persistent across sessions.

I'm sure there'll be other ways to improve the Ruby/ASP environment. Maybe I'll work out a way to load some modules at the beginning of a script and set everything up beforehand. Should be good.

A couple of months ago, I got mt-daapd, the open-source iTunes music server, running at home. I don't use iTunes, but sharing my music collection with Samba or NFS seemed like a lot of overhead. It was very easy to do, and thanks to the Zeroconf/Bonjour/Rendevous support, even easier to use. It was pretty satisfying to fire up Banshee and see my server just pop up in the list without doing anything.

My music collection is a mix of Ogg Vorbis and MP3, which is no problem for mt-daapd, or Banshee for that matter. So I was all set on my Linux notebook. Then I started to look around for a way to connect to the server from Windows. iTunes was out - it doesn't support Ogg, and even if it did, it's too big and proprietary for just this one project. But it turns out that, beyond iTunes, there's not much available on the Windows platform that does DAAP. Even VLC, which normally plays anything you throw at it, doesn't have DAAP support yet.

I did find Get It Together, though, which is an open-source Java DAAP client. It worked great - it found and connected to mt-daapd just like Banshee did. MP3s worked flawlessly, but the Ogg files were grayed out, and wouldn't play.

Kinda makes sense - it's supposed to be an iTunes client, so why would it ship with Ogg support? I decided to get the source and see if I could make it play my precious Oggs.

Step one was to keep the Ogg entries from being disabled in the playlist.  In JavaPlayer.java, I added "ogg" to the SUPPORTED_FORMATS constant, recompiled that class, and replaced the ones in git.jar with the new files. No problems - GIT would now try to play the Ogg files. Of course, it failed. But it was a start.

So next, I had to figure out how to build Ogg support into the program. And here's where I got lucky. GIT uses the JavaZoom SPI to provide MP3 support. This is nifty because it works at runtime - if it's in the program's classpath, then the program can play MP3s just like any built-in format like WAV or AIFF, without caring whether it's an MP3 or not. JavaZoom also maintains a Vorbis SPI, and there are additional ones available for FLAC, Speex, Monkey's Audio, and probably more. (OnJava.com has links to these in an article.)

So I downloaded the Vorbis SPI from JavaZoom, and moved it and its supporting library JARs into Get It Together's classpath. I had to modify the launch script for Get It Together to include these four JARs. And it worked… GIT would now play both the Oggs and the MP3s in my mt-daapd library. Sweet.

This isn't enough to get GIT to include Oggs in its local media library, but since I only need to use it as a DAAP client, I'm happy. And if I ever end up with files in other formats, I'll be able to extend GIT in the same way. Nice to see it all come together like that.