My most trusty digital workhorses

While I wouldn't necessarily call myself a gadget geek, I have amassed a not unimpressive amount of gear over the years and seen most of it fade into obsolescence.

Take the Psion 5 organizer. It was gorgeous. You could actually type on it with eight fingers and it fits in your jacket pocket. Just watching it slide open is techno-rapture. If I hadn't dropped it eight years ago and cracked the touch-screen (everything else still works) and if it had had USB, I might still use it to this day. There isn't a single device on the market today that gets the same mileage from standard AA batteries (because of its monochrome LDC display), the same portability, the instantaneous boot, the same comfortable keyboard. Nothing.

As for PCs, my carbon footprint is a very hefty size:

1992: 386 laptop with a whopping 20 Mb disk
1994: 486-DX266 PC
1997: Pentium I 120Mhz laptop
1999: Pentium II PC
2000: second-hand IBM Thinkpad
2003: IMac 17"
2006: Macbook
2008: My Linux PC for daily use.

Everything else, except for the Macbook, I have either sold, scrapped, given away or painted yellow and used as a stage prop. No machine has managed to keep up with the state of the art in software land, or with Steve Jobs' master plan. How many of today's websites will run on Internet Explorer 3? Who would want to use a 256-colour 640x480 monitor? The first portable music players sported a laughable 32 Mb of built-in memory. Your batteries would outlive your playlist many times over. I never owned one of them, though.

There are however two ancient digital workhorses that have stood me in good stead for over two decades. I have never bothered to replace them. I don't need to, because they do everything I want them to. More importantly, I don't have to replace them, because of incompatible firmware upgrades, new connector jacks, file formats or standard wars.

Free as in free from frustration

How many small open source projects have a dedicated testing team, comparable in size to the available development horsepower? How many have a dedicated usability professional, or even integrate usability standards throughout the development process?

If you have limited time to spend on something you will not get paid for, you will pick the fun stuff. In the workplace fun, cool and kudos are not the sine qua non before programmers will do any work. Your paycheque is your motivation, as Alfred Hitchcock once said to an actor.

I attended a seminar once about Spring MVC, listening to two over-enthusiastic developers telling me how their platform was so much better than the 1356 Java web frameworks vying for my attention. First question from the audience: "When are you guys going to bring the documentation up to date?". I rest my case.

Although no documentation is often better than bad documentation I could possibly live without documentation, provided the software is so intuitive it doesn't need a manual. I can hear your ironic chuckles already.

I was dabbling with music notation software on Linux, installing the latest version of MuseScore.
It's an impressive piece of work, but it kept crashing. It's been around for a while, and yet it still manages to crash randomly. I was running the stable release, and yet it kept crashing on me. The crashes were inexplicable and completely non-reproduceable. One single keystroke was enough to let everything vanish.

I bet it doesn't crash that often on the developers' machines. I reckon there's a lot of satisfied users who don't uninstall the app out of sheer frustration. Why don't I file a bug report and help the developers fix this? Well, I consider some bugs too trivial to report and others - like this one - too glaring. I'm using the latest version of Ubuntu, installing it using the standard Synaptic package manager. As far as I'm concerned the problem is not my problem. Sure, there's likely some pesky bit of configuration to do with sound drivers, but statistically I have only forty more years left on this earth, and my time is too precious to bother.

Mac OS comes with Garageband. It has only a subset of all the nifty features that MuseScore offers. It's made for the amateur home studio, i.e. for me. It's so intuitive it doesn't need a manual. It's certainly not free, neither in freedom or in beer, but it came installed with the OS, so at least it feels free.

Best of all, it hasn't crashed on me yet.

Eat your own dogfood part 3: JDBC4Me

I've done it. After almost ten years of fruitless hacking at ambitious architectures that didn't really do anything, I have finally uploaded a first version of a working piece of open source software: JDBC4Me. I won't expound on how it will radically alter our conception of data or how it wil effect a dramatic paradigm shift in database access. It won't.

It's nothing earth-shattering. For the non-programmers: it helps you retrieve, insert and update data from a program from and into a database in a more user-friendly way. It does nothing new, but it does it in a way I found lacking in other tools. If you're not a Java programmer who's reasonably well-versed in JDBC you probably have no use for it whatsoever. That's perfectly fine.

One of the biggest pitfalls in building software is trying to be all things to all people. The choice of what to leave it out is often harder than what to put in.
JDBC4Me does one thing, and I think does it reasonably well for a sub 1.0 release. It has documentation and lots of unit tests. While writing these tests I discovered and fixed scores of obvious and some pretty nasty bugs. I deliberately tested against only three database products and their most common Java drivers: Oracle10, MySQL5 and Derby10. Use anything else, and the tool will default to settings that may or may not work. I'm quite happy to include Microsoft SQL server and PostGreSQL, but only when I have time to test them properly. If there's one thing I discovered in working with databases in Java, it's that vendor-neutrality in generic tools is overrated. Hibernate and other ORM tools are keen to lure you into this trap and pretend that changing vendors is as simple as updating a configuration file, but the soldiers in the trenches know better.

Here's another thing I learned. The more you zoom in on a programming challenge, the hairier it gets. You start off thinking that all driver vendors conform neatly to the JDBC standard, especially a multi-billion dollar empire like Oracle. Yeah, right. I actually discovered that Oracle had the worst standard compliance of all, and I was unlucky enough to base the first version of my code on Derby, which had the best compliance (not surprising, as Derby is written in Java). I made assumptions that the driver could provide certain metadata, which it didn't in Oracle, so my well-intentioned vendor-neutral code quickly had to go and I found myself writing an Oracle10Proxy, a Derby10Proxy and a Mysql5Proxy. It hurts my eye, it flies in the face of everything JDBC stands for, but it's the only way to make sure that each database performs as it is supposed to perform.

Having lots of features to please lots of users counts for nothing if the features you need are poorly implemented. A housing site that covers 20% of all properties in a country is less useful than one that has 80% in a single town, if that happens to be the town I want to live in.

Next time I'll share an experience of using other people's open source software. I'll try to be friendly and forgiving, BUT IT KEPT CRASHING ON ME.
"Yeah, but it has many more features than [...]"
- BUT IT KEEPS CRASHING!!
"You can write your own macros..."
- BUT IT KEEPS CRASHING!!
"It's free"
- BUT IT KEEPS CRASHING!!

Eat your own dog food - part two

The problem with Yourstreet, I reckoned, was that in essence it was a business venture backed by a website, more than just a web application. It was a typical chicken and egg/necessary critical mass quandary: nobody is going to join if not enough people are already using it. It would have taken a serious investment is sales and marketing. We didn't really want to run a software company: we just wanted to code. But...

• Rule one: there's a lot more to programming than just writing code.
• Rule two: there's a lot more to software development than just programming.
• Rule three: there's a lot more to running a software company than development.

Any programmer should be rudely awoken by rule one when she gets involved in any non-trivial application. Violations of this rule are responsible for more than half of all curious perversions on Daily What the Fuck. Good programming is about being considerate for your co-workers, not showing off your skills just for the hell of it, and basically treating your work as if the next guy to maintain it is a homicidal maniac who knows your address. Writing code that works is one thing. Writing code that other coders can read and improve is a necessary feature of good programming.

Software development involves business cases, requirements, project briefs, planning meetings, progress reports, release schedules, deployment diagrams, service level agreements. Well, you get the point. Especially in smaller projects you can't avoid becoming involved in one or more of these chores that have very little to do with coding, or even programming.

All of the above still has to do with building and maintaining the product. Elsewhere in the building they're hiring new staff, replacing lightbulbs (and planning the resources for it), answering the phone. O yes, and finding people who might want to give you money in exchange for yourstreet.org. Rule three is about creating Joel's development abstraction layer.

I don't want to run a software company, I really don't. Not even one with a casual dress code and gorgeous female developers who like classical music. That's why our Scotland partnership broke up -- that, and my marriage, but this being a software blog I won't elaborate. I moved back to the Netherlands and started work for a company in Rotterdam.

Meanwhile I had all this Perl code and too much time on my hands in the evenings. So I decided to reafactor yourstreet into something that didn't require a sales force to reach critical mass and was something that I could see myself using. I started building a web-based CMS/file synchronization/version-management apparatus. Started it, and abandoned it.

Don't even think for one moment that a team of one (or two) developers can out-perform a team of 500. If you want to write successful software on your own, you have to keep it small, smaller, smallest. More about that next time.

Eat your own dog food - part one

I could be really boastful and pretend that I have 25 years' worth of programming experience, starting with the 100-line Commodore VIC-20 game Operation Crocodile when I was fifteen (no copies remain). At least in those days I did the right thing: I built little ramshackle programs entirely for my own use and enjoyment, and for as long as my adolescent attention span could muster.

In May 2000, while I was living in Edinburgh, Scotland, I quit a very boring job doing technical support in order to start a web development company with a fellow Dutchman. I got my feet wet with good old Perl/CGI code interspersed with lots (no really, I mean LOTS) of hardcoded HTML in a riot of highly artistic anti-pattern programming madness. But hey, it worked!
Around the turn of the century they paid you good money to hone your skills in a work environment that consisted almost exclusively of amateurs, and I mean that in the best sense of the word. Nobody could boast years of experience in web development, simply because the whole discipline was in its infancy.

I knew I was destined for better things, and during the time I spent in between contracts me and my partner in crime were hatching the killer app. It was called yourstreet.org (don't bother looking it up). The idea wasn't bad, provided you had the non-technical manpower to pull it off.
Based on Comiston Road in Edinburgh's well-heeled Morningside area we set up a website for local news, including free as well as paid advertising for the local shops. We acted locally, and too quickly though globally. That's where we went wrong.

Geeks are typically not interested in the non-technical aspects of making a web enterprise successful. Our codebase mushroomed at a frightening rate. I was not just interested in Morningside road. I thought areas, cities, municipalities, the whole of Lothian, Scotland, the UK, Europe, the galaxy. I wasn't interested in actual users. Worse, I wasn't even interested in using it myself.

Take a brief moment to think about that.

A lot of the mistakes I made was textbook stuff

1 - Re-inventing the wheel.

If you're writing a non-trivial application that needs to contain, say, a calendar to schedule events with multiple attendees, you have what is called a wheel. Wheels have been around a long time. They're available in many sizes, flavours and prices. Here's for the obligatory car analogy: even the most revolutionary hydrogen-powered car will probably have standard wheels.

2 - No unit testing (or any of those pesky best practices)

Unit tests? Why should I need them? I don't make bugs! I was astonishingly naïve in those days, but at least I wasn't the only one. I once worked in a team once where we had Human Semaphore File Locking (HSFL). “Nobody touch index.html. I'm working on it!”. At least we all worked within shouting distance.

1 – Programming with a blunt saw

I was a very inexperienced programmer. Although I was learning at a steady rate I had no peer supervision. I should have spent more time sharpening my saw by concentrating on the quality of my work instead of the quantity.

But there's one lesson that's not in any of the textbooks. It has to do with motivation.

Be prepared to eat your own dog food.

If you're investing time and energy in open source software, it must be to scratch some personal itch. Thinking up impressive architectures that don't actually do anything is a dead-end street.
If you really think you can out-PhotoShop PhotoShop, then do it, and use it to edit all your pictures. I don't think you stand a chance in hell, to be frank.
Being user-centric means being egocentric. Put yourself first. Don't fall into the trap of thinking what the software can do. Think what you want to do with the software.
To be continued...

Long live the average programmer



They say that great programmers are an order of magnitude more productive than average programmers. Wikipedia would want me to specify who "they" are (Frederick Brooks and Joel Spolsky for starters), but this is not a scientific assertion. Any programmer who has worked with really smart colleagues will, perhaps grudgingly, admit that it is by and large true.
It doesn't only go for programming, though. True excellence in any endeavour is rare (Mozart, Michelangelo, Monty Python). Excellent achievements really stand out. That's why we call them excellent. If excellent programmers weren't rare they would be average. So much for tautology.

Experience doesn't count for much. It doesn't take you a lifetime to become excellent at something, and that's a soothing though. You find out quickly enough whether you really have great talent. I started playing the piano almost thirty years ago and I quickly made progress during my early teens, but after that it kind of stagnated. Child prodigies, by comparison, don't stagnate after a few years. They start playing when they're four and by the time they're sixteen they baffle the crowds at the major concert halls. Great athletes don't win the Tour de France once or twice: they win it at least five times (Merckx, Hinault, Indurain, Armstrong).

There's no shame in being out of these people's league. Most of us are. You can still be a competent pianist and avoid the fiendishly difficult sonatas of Franz Liszt. You can be a competent programmer and not be able to reverse-engineer the Linux kernel.

Should a software company fire people like myself and hire a genius, who is ten times more productive but takes up the same amount of office space? Of course not. To begin with, geniuses are very rare and even if you paid them triple wages (which would make very good business sense) a genius is primarily motivated by the satisfaction that practising his art gives him, only secondarily by the amount of money he can make with it. If you have a genius in your team, you better make sure to give them some real challenges, or they get bored before you can say "private office with Aero chair".

Most programming work I have done over the years takes smart people, not geniuses. In fact, some of it was pretty mundane and boring. Most software jobs don't require inventing faster sorting algorithms or more reliable floating-point arithmetic. At least mine didn't. I would have been completely clueless anyhow. More likely they require building a user-friendly web form that handles phone numbers intuitively rather than insulting customers with "010-123456" does not match regex validPhoneNumber Javascript popups.

If it takes me a day to fix a simple task, the star could probably do it in half that time, but I guess the simpler the task, the smaller the difference. Partly because the star would get bored or consider it an insult of their intelligence. Molest me not with this pocket calculator stuff, they will reply haughtily when asked to hack a few AJAX web forms. Give them something really difficult to do, because that's where they shine. How much faster than myself could they reverse-engineer the Linux kernel? It does not compute, because I simply wouldn't be up to it. Not in this lifetime, with this set of brains at least.

The Page Paradigm

Following Jeff Atwood's tip I read Dan Ariely's excellent book Predictably Irrational, which gave me all sorts of revealing insights into my own irrational psyche.

Anchoring is the psychological phenomenon explaining how we human beings are reluctant to update our ingrained opinions. This relates to our sense of value (cheap or expensive) as well as our conception of the validity of doing things a certain way. We like to stick to what we know. If you grew up in the fifties, 200,000 Euro for a house will always feel like an obscene amount, and if the Commodore VIC20 was your first computer any file over a megabyte just feels huge. I remember a C programmer lambasting me in very derisive terms for programming with mod perl (an embedded Perl interpreter in the Apache web server) because it added a mere 4Mb to Apache's memory footprint. Note that this was seven years ago, in 2002.

If you built dynamic web sites around the turn of the century, you'll have HTTP, HTML, JavaScript with either Perl, JSP, PHP or ASP paradigm anchored in your brain and you'll be used to doing things in a very time-consuming and cumbersome way. I don't care what anyone may say to the contrary: building fast, usable, secure and reliable web applications was and is a very tough job.

You could fill entire bookshelves lamenting the inadequacies in terms of security, usability and browser compatibility nastiness, but what's the point? Clearly the benefits outweigh the drawbacks. Embedded Java applets had their chance and we didn't want them. Web applications are here to stay. The underlying technology has evolved impressively, yet for a revolution we should stop thinking of web applications in terms of pages. It's time to haul that anchor and move on.

Originally the web was conceived as an information universe of static content, randomly accessible through hyperlinks. The back button has always been an indispensable tool. It allows you to follow any link on a page and quickly retrieve your steps. It's the equivalent of the undo function and it works well for static content.

A typical web application however does not take kindly to it. The function of a web application is performing a mandatory sequence of actions in which the user and the remote server typically exchange data through HTML forms. You search and pick a book of your choice, you provide shipping and payment details and then you submit your order. Presenting these stages as navigable pages with a back button is asking for trouble. Your browser knows this. It will warn you that you are “attempting to submit a form twice”, or something to that effect, leaving the less technically inclined users clueless. Bad programmers will even book you three seats on the same flight as a penalty for hitting the refresh button a couple of times. Where's the logic in that? The back and forward buttons look like undo and redo functions, but they're not. You can even bookmark a submitted form like this: www.initrode.com/order.cgi?orderId=12345;creditCard=1234234534564567;type=VISA;valid=10_2015.

Don't get me started.

Ajax to the rescue. Browser technology has now progressed to the point where you can build an entire client-server application to be operated from a single HTML page. Embedded or linked JavaScript libraries generate page elements on the fly and communicate asynchronously with a remote server without having to refresh the page. No more form submissions. And no more quirky back buttons. The Google Web Toolkit (GWT) framework even has undo functionality built in. Users can hit the back button and the application will behave appropriately, which also means notifying the user when an action is not undoable.

Usability guru Jakob Nielsen still did not think much of AJAX technology in 2005, when it was still a budding technology http://www.usabilityviews.com/ajaxsucks.html but he's slowly coming round and for a thousand bucks he'll tell you why.

Existing page-based technologies like PHP and JSP remain anchored in the pages metaphor. If you're a Java developer with plenty of experience in Swing GUI programming you should give GWT a spin. It's the most radical departure from the old school and I absolutely love it.