Monday, March 23, 2009

My First Experience with Open Source

I first got into open source through a project that I developed while at IBM on the Visual Age for Java technical writing team. Specifically, they needed a build system. Why do tech writers need a build system you ask? Because they had a hell of a lot of documentation. Specifically, they had over 10,000 HTML files to manage. In English. Then toss in the eleven translations of the product and you are talking about 120,000 files that have to get processed and packaged correctly for use in the VAJ help system. I was tasked to create this system because of my background in software development.

After a couple of false starts at building my own web-based mod-perl system on AIX from scratch, I stumbled across Ant. If you've ever done any Java development, you've almost certainly used Ant at least once. Its a Java based build system which uses XML files to describe how a build is executed. The original author, James Duncan Davidson, had written it while working on Java projects at Sun. He was tired of the cross-platform build headaches he was facing, so he set out to build a system using Java as the cross-platform source. And yes, he really did write the core of it on a plane. It was, as you probably know, a huge success.

Aside: While writing this I went to look up JDD's latest info, and it turns out he's now a full time professional photographer. He's the guy who took the picture of Bill Gates releasing a jar of mosquitoes on-stage at TED09. Cool.

I picked up Ant and started to use it for our project. It wasn't the fabulous web based system that I had originally envisioned, but it had one major thing going for it that my other attempts didn't: it worked. At the end of the day, that's what mattered the most. That is, it mostly worked.

I tripped across the usual mismatches between the documentation and what had actually been done. Not to mention bugs. Lets face it folks, every bit of software out there has bugs. We are used to it, and we either work around it or fix it. And that's one of the things I really like about open source: the code is open and you can fix it yourself. So I did.

But that was where I stopped. I fixed the problems in my local copy of Ant. Then the next version came out, and many of the bugs were fixed. Yay! Except they weren't fixed exactly the same way I did, so I had to spend a bunch of time manually merging my changes (I wasn't using the CVS tree at the time, just the release tarball). Boo.

By this point I had been monitoring the Ant mailing lists pretty avidly. It was great to see this collegial atmosphere, where people could ask questions and get well thought out answers by the people who actually wrote the product. Most of the people on the list were welcoming of newcomers, and were willing to steer people in the right direction. That is, most people were, and a couple were complete jackasses. At the time, these people really got my back up, but they were developers on the project, and I was just a lurker. What did I know? A fair bit, as it turns out, but I didn't necessarily think so at the time.

The list also pointed out that not everyone on the list was strictly equal. That is, there were committers, and everyone else. In Apache parlance, committers are the people who have write access to the source code repository of the project. There's more to the role than that, but its enough to get us going. In an Apache project, each committer has a veto vote. That is, if they don't like a source code change, they can vote to have it undone. The voting is also used to determine what new features are going to be added, their design, and details of the actual implementation.

I had noticed that people were sending in patches to the list to fix bugs or add enhancements. This was often described as "scratching their own itch," since it fixed something about the project that the submitter had a personal interest in. After having gone through a frustrating time of merging my changes with theirs, I decided to try posting my own patches in the hopes that they would be accepted, cutting down on my overall work. 'Cause at the end of the day, I'm a lazy, lazy man.

I dug into CVS, learned how to checkout a copy the repository, and started hacking. Once I fixed a problem, I figured out how to create a universal diff. From that, I sent in my first patch. As I recall, I checked and rechecked the list a couple of dozen times waiting breathlessly for someone to comment. At some point a committer said "Thanks, we'll look at it later." Crushed, I returned to my build, convinced I had failed.

After a while, however, my patch did get picked up. Yay! No longer crushed. So I submitted another, and it too was accepted. I started to participate more in the email discussions, offering my humble opinion. And it was humble, as I was learning the system and realized that many of the developers on the list had a much better grasp of how things had come to be the way they were than I did. I learned to search the archives of the list before asking a question, to make sure I wasn't covering old ground.

I'm not trying to say I was the perfect citizen on the mailing list. I'm more than sure I made a few gaffes and came off like an idiot at least once. Who doesn't? That being said, I like to think I learned from my mistakes reasonably quickly. Others, however, did not. They were the help vampires, and they started to take up more and more of everyone's time with useless emails. If I recall correctly, this is when someone posted a link to the indispensable essay on how to ask smart questions in an attempt to stop the tide of inanity. If you haven't read it before, do it now. Really, go read it, its worth the time.

At this point JDD had to leave the project for a while. He had other commitments that kept him from being able to dedicate enough time to the project, so he announced that he would be stepping for some time and would return when he could give Ant the attention it deserved. I didn't really know who JDD was at the time, so I didn't actually pay much attention to his departure.

Also around this time, I was invited to become a committer on Ant. I was thrilled! I was being asked to take a very active role in the project, with direct access to the project source code. I was still developing my VAJ help system build, so I was actively working with Ant on a daily basis.

Things were going along pretty well for a few months, and then JDD came back on the scene. He had a vision for Ant 2.0. He laid out that vision for us (sorry, I don't remember the details), and we said "no." JDD was metaphorically stunned. Who were we to reject his view of the project he had created? Technically, we committers were the stewards of the project. According to the Apache rules, we each had an equal say in the direction of Ant. Just because James was the one to start it didn't mean he had any more influence. He was not "first among equals." I think that point came as a shock to more people than just JDD. At this point, JDD left Ant and started a new Java build system named Amber.

After a few months, I moved onto another project at IBM, working on the Eclipse help system (this was before Eclipse was open-sourced). I was no longer working on Ant builds, and my participation in the project waned. I left IBM shortly after that to work at a startup, and my Ant involvement dropped even more. I made an effort to get back into it, but working at a startup tends to not leave you a lot of free time. In the end I quietly withdrew from participating on the project, and became an emeritus committer.

So if you want to participate in an open source project, the morals of the story are:
  • you need to read the email discussions regularly,
  • be respectful of peoples time by crafting short, readable emails,
  • ask smart questions,
  • don't be a help vampire,
  • scratch your own itch,
  • people working on an open-source project as an offshoot of their day jobs aren't necessarily the best long term participants,
  • be patient, as people are often working on the project on their spare time, so don't expect instant answers to your questions, and the really big one
  • abide by the rules of the community
I can't stress that last point enough. Each open source project has its own community, with its own set of established rules. You need to ensure you understand those rules and are willing to abide by them if you want to be taken seriously. If you can't abide by them, then its probably not the project for you.

Participating in an open-source project can be very rewarding and enriching experience. You can gain a lot of experience in the reality of distributed development even if that isn't part of your day-to-day job. It can also offer a way to get those extra skills that your current job just can't offer you. Its a committment, but a very worthwhile one.

1 comment:

  1. I occasionally use little programs to make things easier, such as Zotero - but had never given much thought to the human element behind it. So this is an interesting 'behind the music' of open source :)
    EN

    ReplyDelete