I just started the wheels moving for a FedEx Day at work. Here’s what we wrote up for it:

We would like to propose taking a page from Atlassian's playbook and instituting a "FedEx Day". The idea is that for one day, all of the developers and QA would be free to work on a product or feature of their own choosing. The only rules would be that they have to work with someone who they don't normally work with and that the next morning they have a finished product to demo to the whole team, something that they can “ship”, ergo “FedEx Day”. This is a lightweight version of Google’s 70/20/10 policy, where the majority of an employee’s time is spent doing their traditional job, 20% on company related innovation, and 10% on anything.

People don’t stay at Texterity because of the money. We pay our developers competitively, but not lucratively (nor should we). People stay here because they like the stuff we’re working on and the people they’re working with. We haven’t lost a lot of developers, but when we do, the very competitive job market makes it a challenge to fill those positions. Jason A, Vijay, Arun, Dasheng; each of those losses has been painful in its own way. The best way that we can encourage tech workers to stay is to make them feel needed and give them purpose. The most exciting projects I’ve ever worked on were the ones where I had pretty much complete autonomy and was passionate about the end result. It’ll give all of the team members a chance to try something new and challenging and maybe even give us some outside-of-the-box features and products.

A FedEx Day would have some other benefits, too. For a day, the developers would be decision makers where they typically are not. They’d be able to see what it looks like from the product owner’s point of view. What’s a valuable feature? What’s worth spending time and testing on? How do you sell its value to management when you demo it? And there’s a good chance we’d end up with some features or products that we wouldn’t have come up with otherwise – Atlassian says they end up shipping about half of the FedEx Day projects.

We have plans for making sure that emergencies get handled and we have ideas for scheduling so that day-to-day operations are impacted as little as possible, but those discussions can come later.

If you’d like to hear more about motivation and purpose in preparation for the meeting, I suggest watching this fantastic ten minute presentation by Dan Pink:

www.youtube.com/watch

Here’s a post on Atlassian’s blog about their first FedEx Day: http://blogs.atlassian.com/rebelutionary/archives/000495.html

Here’s an article from Inc. magazine on the topic: http://www.inc.com/news/articles/2010/01/side-projects.html

My laptop at work is getting a bit long in the tooth and is due to be replaced in the next couple of months. I currently run Ubuntu on a Dell laptop. The laptop is elevated on a dock next to another screen. I have two choices:

Choice 1: MacBook Pro. Arguments in favor:

• Apple makes the best laptop hardware anywhere. Period.
• I'd be able to dual-boot to OSX, which I have nothing against, it's just not my preferred environment. I could hypothetically do things like play Steam with my OSX partition.
• Multitouch mousepad is so nice.

Choice 2: Dell E5520. Arguments in favor:

• The Linux support for the MacBook Pro is imperfect, especially wifi.
• I like the way I do stuff in Linux, and if I switched to OSX (for wifi, for instance), there will be a learning curve.
• There are docks (whereas MacBooks have some third-party stands but no real dock solution).
• 15.6 inch widescreen, instead of 13" (which I'd have to get to keep the MacBook Pro under budget).

What are your thoughts?

Some days, nothing goes right.

mysql> drop table node_details; ERROR 1051 (42S02): Unknown table ‘node_details’ mysql> create table node_details ( id integer ); ERROR 1050 (42S01): Table ‘node_details’ already exists mysql> drop database local_nodes; ERROR 1051 (42S02): Unknown table ‘node_details’

This exercise in un-collapsed waveforms brought to you by this not-a-bug bug.

I got a promotion yesterday. Effective more or less immediately, I am the manager of one of two teams of developers. I’ll have two great developers reporting directly to me – one of whom is on vacation and is probably going to learn of this change while reading this very blog entry when he comes back next week. Hi, Bryan! – and my very first task is to hire a third. The first thing I learned about being a manager is that there are a hell of a lot more things to think about when you’re the crux of a hiring decision than when you’re just another interviewer voice.

I’ve told a couple people about this already, and there have been two repeated questions: First, “Are you still going to be able to do technical work?” Absolutely. In fact, it was one of my requirements for accepting. This is where I’ve always wanted to go with my career – hence my dual BS in Computer Science and Management – but I never (ever) (EVER) want to stop coding or playing with new technology. With such a small (and, I believe, low-maintenance) team, I expect to spend at least three-quarters of my time doing exactly what I did last week.

Second, “Did you pursue this, or did your boss just offer it out of the blue?” I think it was a fortuitous combination. The company is growing (we had 4 developers when I started here, we now have 8, plus 2 interns, and hope to add 3 more before the end of the calendar year), and the manager of the development team was having a harder and harder job keeping everything planned. At the same time, I’m one of the most senior developers here, and I believe I’ve had an unofficial leadership position here for a while. I mentioned in my annual review last month that I was interested in more responsibility, and I think that just solidified thoughts my manager had already been having.

This transition will be easier than it could be, otherwise. In fact, these are probably the ideal conditions. There’s no seniority clash. There’s not really anything Broken about the culture in the development group or at the company as a whole. I sincerely enjoy the people I’m working with, and I think they’re all smart and hard working without being jerks or workaholics. I don’t feel the need to change a lot, organizationally, since it’s all working very well. I’ll be able to ease into this new role with some hiring and planning (and soon, budgeting for fiscal ‘09).

I’m scared, but excited. It’s like the clickety-clack part of the roller coaster ride. You know what I mean.

Update 6/5: The new group has a name, although it’s possibly temporary: “Publishing Technology”. We wanted to steer clear of words like “internal” (since we’ll do publisher-facing apps), “tools” (which implies smaller utilities than what we work on), and “workflow” (that’s just a small fraction of our purview). The other half of development is very simply the “Products” group.

One of M’s co-workers is giving an interesting sounding talk this week. If it wasn’t in the middle of a work day, I might try to wrangle my way into it:

"Have you ever watched a cardiac arrest on TV and thought 'That is not how we do it'? Do you remember seeing Dr. Carter shocking asystole and wondering why the cast of ER did not follow ACLS guidelines? Has a patient's family member ever said to you 'Meredith Grey was coded for four hours and was fine. What do you mean there might be neurological damage?' All these questions and more will be addressed this coming Thursday during my Senior Talk entitled CPR: As Seen on TV."

Update: This reminds me a bit of the so-called CSI Effect, where juries are becoming more and more demanding when it comes to physical evidence in trials.

At work, each new user is assigned a totally random alphanumeric 12-character ID. They’re random instead of sequential because this is what goes into the user’s cookie (and in some cases into URLs) and we didn’t want the IDs to be discoverable. Sometimes we need to do what we call a subscriber load and generate thousands (or sometimes many thousands) of IDs at once. The subload process tends to be very slow, and one of my co-workers was tasked with making it faster. While profiling the code, he discovered that a big time sink was the ID generation procedure. After more research, we discovered that it was written in 2004 and had never been modified after the original checkin. It was hundreds of lines long, used all kinds of global variables (Perl hasn’t had static variables until 5.10), and involved big math with a magic prime number close to 70¹². Worse, it was implemented as a hash function. And it was always passed a salt. And that salt was always random.

We replaced it with this code:

my @chars = ('A' .. 'Z', 'a' .. 'z', '0' .. '9');
sub randid {
    $rv = '';
    for my $i (1 .. 12) {
        $rv .= $chars[rand(@chars)];
    }
    return $rv;
}

It used to generate about 100 IDs per second. Now it can do 175,000 per second.

After four different positions in three different groups and two cities in the past seven years, this its my last day working for Priceline.com. I’ve had an excellent time working here. The people are friendly and excited, and we occupy an interesting place between tiny startup and mega-corporation, which allows us to do some interesting and cutting-edge stuff.

I’m sad to be leaving, but I’m excited about my new gig at Texterity, as well as the prospects of owning a condo in Southie and just generally moving to the Boston area. May is exciting!

Recently, Sun released version 3.1 of the Sun Ray Server Software. It now has Linux support for USB devices connected to the thin clients. Instead of being kernel-level, this support is user-level — in the form of a modified libusb. Unfortunately, roughly 97% of applications expect kernel devices, and thus lack support for libusb. Luckily, pilot-link belongs to the minority. The following are my (roughly chronological) notes on getting a Palm (in the form of an old Handspring Visor) to sync with my Sun Ray.

• You'll need to have libusb 0.1.8 or newer installed. For reference, RHEL3 doesn't have a new enough version; I needed to find a non-standard RPM.
• Get the latest version of pilot-link from http://www.pilot-link.org/. I used 0.12.0-pre4. You'll almost certainly need to compile it, since even if your distro has a package for it, it probably won't be compiled with libusb support.
• Be sure to add the --enable-libusb flag when you run configure for pilot-link. (I assume the reader knows how to compile and install stuff under Linux)
• When you run commands that you want to use the Sun Ray libusb support, you need to run them with the LD_PRELOAD environment variable to include "/opt/SUNWut/lib/libusbut.so.1".
• I'm not sure what most of the executables that pilot-link installs do, but I know the important one is pilot-xfer. This command line works for me: bin/pilot-xfer --port usb: -s $HOME/.pilot-link This command will fail unless you have the Palm already trying to sync. You'll also need to run it as root, although the pilot-link guys say they're working on a fix for that.

I'm still on the hunt for something that will allow me to access data on a USB thumb drive.

I just got an email from an HR person at Google. It was really personalized, not just a form letter, saying that I might fit in well with their Server Engineering team. And from the description of the job, I think I would. I had to tell him that I didn’t know where I’d be a year from now, but I might contact him in March.

Oh, even better, he told me how he came across me: my goddamn contributions to fucking Wikipedia.

Yesterday, I passed my Red Hat Certified Engineer exam with flying colors. I’m willing to call it a legitimate certification. The entire test was hands-on troubleshooting (“Here’s a machine that won’t boot. Fix it”) and installation and configuration (“Here’s bare metal, and a four page description of how we want the machine setup. Go”). It’s very indicative of the kind of work I do on a daily basis, so it’s far more useful than a multiple-choice test would have been.

Today, I gave an intro to wiki class at work. Ever since I moved the Unix group’s home page from a wad of hand-edited HTML files to wiki, it’s been a far bigger hit than I expected. There are now nine teams using wikis for their documentation or interested in migrating, and I expect there are probably a few more that will be converting soon. It’s great to be able to feel that I’m making a bottom-up difference at work, and that open source (not to mention Wikipedia) is touching a few more people who might not otherwise ever feel it.

Yesterday, I was surfing through the “24” tag on del.icio.us when I found a site that had the ring that CTU uses in several formats. After I remembered that I have the exact same phone at work, I decided to see if I could get this ring on my desk phone.

The first step was to figure out if this was even possible. There are a number of rings already on the phones at work, and they’re apparently fairly programmable, so I wasn’t surprised when a couple quick searches led me to a document explaining how to make custom rings for the Cisco IP phone model 7960. The file format requirements at the bottom were a little challenging, though. I wasn’t sure what to do about “uLaw compression” or “Raw PCM”, But I figured it probably wasn’t going to be extremely hard.

Half an hour later, I had two free applications: wavetools, “a library consisting of 8 programs for manipulating mono WAV files” that let me resample the file to 8000 Hz and cut it to 16080 samples (the maximum length), and AFsp, which then converted it to a raw PCM. When all was said and done, I had my ringtone in the right format. I only hoped that it still sounded right.

The next step was less technological and more political, and I wasn’t kidding myself into thinking it would be easy. The Cisco IP phones use a centralized server to get their configuration information, firmware, and (apparently) ringtones. I sent an email to Paul the phone guy, asking him if it would be a big deal. As I expected, he referred me to a middle-manager, but luckily I knew him pretty well. I asked him if it would be possible, and at first he seemed receptive, but he started to realize that it might be setting a bad precedent. I could only imagine how many people would want to have their own ringtones on there, and chances are they wouldn’t have the skills to convert it to the exactly right format.

I was concerned that this was the end of my 24 ringtone story. I emailed the PCM and the link to the author of the original page, telling him that it was untested but that I was confident it was right. Then I went home.

The next morning, in the shower (as if you doubted it), I realized: the phones just use TFTP to get all of the stuff from the server. TFTP servers are, well, trivial to set up. In fact, I know of at least one of our servers at work that’s currently running a TFTP daemon. The hard part was telling the phone to use a TFTP server other than the one that it was told to use:

1. Go to Settings -> Network Configuration
2. Hit "* * #". Nothing will happen, but you've just made the settings editable.
3. Find the "Alternate TFTP" setting. On my phone it's #32. Set it to "Yes" by hitting the appropriate soft button.
4. Go back up to the "TFTP Server 1" setting (#8 on my phone), and hit the "Edit" soft button
5. Write down the current TFTP server. We'll need it later, and you should probably store it in a safe place, just in case something dies sometime down the road.
6. Type in your new TFTP server IP.
7. Save the changes.

Now, it was just a matter of figuring out what files I needed to get from the old TFTP server. I started my new server with verbosity turned way up, and rebooted the phone (well, I unplugged it and plugged it back in). When the phone requested files, I saw them in the log. I then used a TFTP client program to download them from the old server. I modified the RingList.xml and added my raw PCM to the directory, and rebooted my phone again.

When it came up, the ring was on the phone and looked and sounded just like any other ring.

What’s the attraction behind printing things out? The new guy at work has printed at least a dozen emails this week, and also printed several of the documentation pages out and posted them in his cube. When I’m showing him something, I’m sure to say “This has a Wiki page. You can just copy and paste these commands I’m running, just like I’m doing right now.” But he has to write down every command I run on a big ol' pad of paper.

I’ve been trying to stay virtually paperless. I take notes on my PDA instead of dead trees. I do have a notebook handy in case I need to do something that requires it, like diagramming an application or environment, but I rarely have to use it. I kinda feel like the wasteful habits of my co-worker are invalidating my good intentions. Looking around me in the last week, I’ve seen a lot of desks covered in stacks of papers, and I’m finally realizing how common this is. Why do so many people need paper copies of everything, when they just end up being unused clutter?

Good luck to Brian and Nomad on their first days of work at new jobs. Stop reading this blog and get back to work!

The aforementioned major deploy went out this morning. It went well. More than 5000 man-days culminated in a very smooth site rollout this morning. I feel like a weight has been lifted from my shoulders.

Remember a few months ago, when I said that we had gone with SuSE instead of Redhat? Apparently, before we could sign a contract with Novell, Redhat turned around and told us “We want your business. We will match any offer Novell gives you.” So although nothing (still) has been totally resolved, it looks like we will be going with Redhat for our Linux needs. It’s been a long, wild ride, but it looks like we’ll soon finally have enterprise support for the roughly sixty servers under my domain.

Just to allay any fears, I have not forgotten this blog. There’s a huge new site launch next week for which I’m in the middle of configuring and tweaking 50 Linux servers. I’ve heard some people say that this is our single biggest launch since the dot-com bubble burst. I can believe it.

I was called for Jury duty (petit jury, thank goodness) on May 3. As a heavy-duty Law and Order fan, I look forward to watching the real thing from the inside. Believe it or not, I actually hope I’ll get picked. I just hope it doesn’t interfere with E3.

Yesterday, after three months of indecision, we finally decided on a Linux distribution. We’re going, not with the (essentially) industry standard Redhat, but with SuSE. The codebase is obviously virtually identical, and the featuresets are pretty comparable, and Novell (who bought SuSE January 2004) gave us an amazingly good deal. It’ll be an interesting exercise trying to get all of our Dev and QA stuff moved over in the next few weeks, and there are apparently a couple of vendors we’ll have to pressure to hurry up their SuSE support, but it’s nice to have this decision finally made.

Update 3-23: It didn't go as easily as expected.

As of roughly 15:30 this afternoon, I am no longer a programmer. I am a Unix System Administrator. Go, me. As my first Unix Admin-related post, I’d like to announce that the Sun v20z is a clean, precise, fast-as-all-hell piece of genius.

I'd say more, but it's Friday evening, and I'm feeling terse.

I just rewrote a piece of code and tested it against a ridiculously huge dataset (63,000 points), just to see how much it has improved. The new code ran under 14 seconds. The old one – well – all I know is that it’s taken more than nine minutes and is still running.

Success is a nice feeling.

Update: It ended up taking 14 minutes and 53 seconds. More than a 10x 60x speedup.

It’s official, I’m going to be working for Priceline.com’s UNIX group as the in-house Linux guru. Go, me.

I do three kinds of things at work:

1. Work. Sub-things: programming, monitoring management
2. Not work. Sub-things: surfing, programming
3. Semi-work. Usually proof-of-concept programming. This is the most interesting thing I do. WEBoggle started as a "what's this DOM thing I keep hearing about" semi-work. Here's another one.

In the break room here at Priceline, the basket that the ‘hot cup’ lids are in is apparently where the tea bags used to be, and it’s still labelled “regular” and “decaf”. Every time I go in there to get a lid, I think of making the joke “Damn, they’re out of decaf lids again!” But no one I know is ever in there at the same time, so I never have the opportunity to make the joke.