Lessons From The Science of Nothing At All - Part 1

[came across this good, long overview of Software Development and Agile thinking by Richard P Gabriel - will be published in two parts given the length of the content]:

Where I come from we make things from nothing - from dreams and fantasies. The laws of physics don't apply. Our products weigh zero. We've explored just about every product development approach there is - extreme or otherwise: waterfall, iterative, rapid prototyping, community development, and mobs.

Imagine this: Dionysus has deleted all the software on the planet - what's going to happen? You won't be able to surf the Web, send email, make and run spreadsheets, use word processors, and download music. No more anonymously available dirty pictures, no more mapping services, no more reading newspapers from around the world, no more computer and video games, no more pocket organizers, no more modern warfare, governments will come to a halt. Well, that's not so bad.

Oh, and movies will have cheesier special effects, Playboy centerfolds will seem to have more blemishes, buying airline tickets won't work well, booking hotels will take longer, no one will be going into space, you'll mostly be watching local TV, you won't be getting the New York Times, Wall Street Journal, and The Economist in the mail, some airplanes will fall out of the sky.

Wait, what's that? - This is getting serious. No more telephones, some buildings will stop being livable, most cars will stop running, food won't be delivered efficiently, health care - don't want to think about that. We'd be thrown suddenly back into the 1950s or 1940s. Just about everything we do every day will change or disappear.

Software is a real part of our lives, a real part of the world. If it wasn't real, then why does deleting it break so many things? Well of course it's real: Big companies make lots of money creating it: Microsoft, Oracle, IBM; even smaller ones: Netscape, Borland, Adobe. A few million people earn their livings writing software, an equal number attend to those who create programs: documentation writers, testers, managers, customer support folks.

What's writing software like?

Suppose I showed you two rooms where in one a group of programmers was developing a program to monitor a heart patient and keep him alive until the doctors arrive if something goes wrong and in the other a group of street people was using Microsoft Word to write letters to their parole officers. You would not be able to tell the rooms apart.

Here's the problem and the intriguing possibilities of studying software and how to make it as exemplars for all sorts of making - or if you prefer, product development.

What a lot of you make uses stuff in the physical world. You hammer it together, stick one thing to another, screw this into that. The stuff wears out, you put oil on it to make it run smoother. A guy in overalls can fix it. You deliver it to people in boxes. When you take your product to the post office and they put it on a scale, it weighs something.

That is, you have reality on your side. You make things in 3-dimensional space. Maybe it seems to you that these are limitations that you'd like to unload. Don't. Hang onto those things. In my business, we don't have simplifying constraints like these. On the other hand, we have some things better. I've never had a few thousand gross of bad IF statements delivered.

Let's look at some statistics. You can hardly pin anyone down on numbers like these, but a typical number you'll see for an average programmer is 50 lines of tested code produced per day. At delivery, there can be about 30 defects or bugs per 1000 lines of code.

Bugs are probably not quite what you think. Imagine you've designed a car and built it, but someone designed the wrong size tires for it. If the car were software, maybe this would cause the car to not start, or you wouldn't be able to steer it, or maybe the car would explode. Or maybe everything would go fine until the driver tuned to NPR and then the gas tank would spring a leak.

Windows NT has nearly 50 million lines of code in the full build. That would take one person about 2700 years to complete. Oh yeah, and it would have about 1.5 million bugs. Adding people to a project can speed things up, but not linearly. If you add a programmer to a team larger than 50 people, it's not unlikely the productivity of every programmer will drop, and at some point, increasing the number of people on a team will reduce its total productivity.

Bugs are not a good thing. Some of the recent Ariane rocket failures were the result of bugs - some of them downright silly.

The Ariane 501 was the European Space Agency rocket that self-destructed after its engine nozzles all swiveled into an extreme position, causing the rocket to veer abruptly, which caused it to self-destruct. The bug had to do with some alignment software on the navigational computer that normally runs right after they put the rocket on the launchpad, but in this case it kept running well into the launch, during which time it computes nothing meaningful. This software eventually encountered a bug since no one was aligning anything. The navigational computer sends navigational data to a command computer, which directs the rockets. The navigational computer put a textual error message on the data bus to the command computer and then shut itself down. The command computer took the textual error message as extreme navigational data and swerved the nozzles. Then it noticed the navigational computer was off and switched to the other one, which had also shut down for the very same reason. The stumped command computer turned to its backup, which was also stumped, and after noticing the extreme aerodynamic stress on the rocket, one of the two decided to self-destruct.

You might think that the software running on the navigational computer wasn't mature or well-tested. In fact, it was used by the Ariane 4 rockets for many years without any problems at all, and not a single line of code was changed when it was put on Ariane 5 rockets. Software for space rockets is written much more carefully than commercial software. Here's how the software for the space shuttle is written.

SEI Level 5 is the highest rating for a software methodology assigned by the CMU Software Engineering Institute. Lockheed Martin Corporation employs an SEI Level 5 software group that puts together the shuttle software. The software is not very big - only 420,000 lines of code. It was developed over a 20-year period by a group of 260 people. These numbers imply that each line of code has had 25 person-hours of attention, and that each person has been responsible for a little over 1600 lines of code - 25 pages. $700,000,000 was spent to develop this code, or about $1700 per line.

Further, the software had been completely specified in pseudo-code agreed to beforehand. The task of a programmer in this regime is to translate pseudocode to actual code. But, all this work does buy NASA only 1 error in each of the last three releases of the code. Because we cannot afford to have deaths in the space program, the cost and effort are worth it. The cost amounts to making each subroutine a career-long research project. Twenty years, 8 hours a day, studying and eliminating errors from 1600 lines of code.

This sort of slow, deliberate process of creating software is good at producing bug-free code. Variants of this approach have been used since the beginning of programming. One popular variant created in the 1970s is called the Waterfall Model which breaks up the process into 5 stages: The first stage is understanding the concept, doing analysis, and defining the requirements; the second stage is design; the third stage is implementation and coding; the fourth stage is integration of the parts and testing; and the fifth stage is operation and maintenance.

The Waterfall Model was a response to earlier methodologies that quite essentially started with someone shouting, "hey, let's make a text editor," and proceeded with people running toward the keyboards. Deep thinkers realistically believed that by using the Waterfall Model, the quality of software could be improved and the process of creating software could become much more predictable.

Shallow thinkers quickly noticed that this reality thing was going to be a problem. Recall, software creation doesn't get the benefit of reality and 3-dimensional space. What does this mean? When you're designing a real product, you have centuries of experience with the world and real artifacts to base your designs on. Designing a new hand tool? - 60 centuries of experience, has to fit human hands, has to accomodate human motion. Does it cut? Done that. Does it fasten? Done that. Does it make holes? Done that. Designing a new telephone? Two or three generations of experience there. New door? Has to open and close, has to be big enough to let what's to move through, move through. New stove - people have been cooking for hundreds of centuries.

But, consider the first people to design and build a text editor. Before that, there was never a text editor. Changes to a manuscript were always made by retyping or retypesetting. How would people want to make textual changes? How would people want to navigate? Searching? - no one ever heard of that before. Systematic changes? Huh? By the way, there were no display terminals, so how do you even look at the manuscript?

So what do you think would happen if you tried to design and implement a text editor for the first time? You'd maybe talk to some folks and decide what it was going to do. Then you'd design and build the software - keeping in mind that the slightest, most picayune mistake and maybe nothing works at all. Designing and implementing it so it works maybe would take a couple of people 6 months. Once it sort of worked, you'd let people use it, and once they had seen what can be done, then they would start saying, "oh, now I get it, well, how about if you add this and that to it and change this so it works like that."

Then the developers would go back and make another few months of changes, and the process would repeat. Why? No one has ever seen a text editor before or maybe only the previous versions, and so they can't imagine how they would really use it and what the experience would be like until they had one. Then they would start to use their imaginations to guess what it could be like, but they are only guesses. The people who are burdened with coming up with the requirements for the first text editor cannot do it because first, they have never even heard about a text editor before, and second, they have no idea what is possible or impossible for software to do. Usually they require only modest and conservative things because they think what they can imagine is too hard or impossible, and once they see that something they thought would be impossible can be done, they start expanding their imaginations. So they iterate and try and try things.

For products in the real world, you have already had the decades, centuries, and millennia to go through this iterative process, and we software guys have had 50 years total.

Let's go back to that Waterfall Model. Here's its first step: Understand the concept, do analysis, and define the requirements. If you can do that, then the model will produce a program that satisfies the requirements. The Waterfall Model depends on having the right requirements, and initial requirements are usually just a guess. A number of software processes in use assume that the real requirements can be fully and accurately specified beforehand. Why do they think that when it is clear that when you're building something no one has ever seen or had experience with before, you are at best guessing every step of the way?

In response to the Waterfall Model, other methods were proposed: the Iterative Method and Rapid Prototyping, for example.

Rapid prototyping uses a special prototyping language to make quickly a prototype of a system. With this prototype, potential users can try out the software, and the real requirements can be figured out. Then a more traditional method like the Waterfall Model can be used to implement the requirements using a real programming language.

In the old days - 15 years ago or so - prototyping languages were distinguished from real programming languages by not requiring as much description of the program as is needed for real programming languages. The reason real programming languages require a lot of description is so that a compiler can produce very efficient programs using highly compressed data representations - a compiler is a program that translates computer programs that people write into code that computers can execute. A way to look at the difference between prototyping and real programming languages is that the prototyping language spares the programmer from typing in a lot of extra stuff, which means the computer needs to do more work while the real programming language makes the programmer do the extra stuff so a computer doesn't have to do as much. The result is that with a prototyping language, you can write code a lot faster and with fewer bugs than you can with a real programming language, and with a real programming language it takes a lot longer to program, but the code runs fast. In the old days when computers were slow, it was worthwhile to make people work hard in service of computers. In the old days, Java would probably be called a prototyping language.

The Iterative Method employs use cases to define what the software will do. A use case is a scenario of usage and what should happen - that is, a person does this, that, and the other thing, and this is what the software does in response. The total specification of what an interactive program should do can be made by describing all the possible use cases.

The iterative method proceeds by gathering use cases, then implementing enough of the system to implement a small batch of the use cases, trying the system out, then adding new use cases or modifying existing ones, and then doing more implementation, etc. The idea is to intersperse understanding the nature of the software being written with its implementation.

With both rapid prototyping and the iterative method, the goal is to speed up understanding what software should do and how people can best interact with it - because with software we are building things that have no physicality and which do not adhere well to physical metaphors. In many cases we are creating things which have never been created before, and we have no idea how people will use them, what they will find hard, what they will find confusing, and the process of writing software is so arduous, time-consuming, and money-intensive, that we have not yet been able to try enough experiments to know the answers to those questions.

Here is one way to look at it: If you place almost anyone from a Western culture in front of any desk that they have never seen before, he or she will be able to look through the drawers, find pens and pencils, and otherwise do their work there. The primary computer metaphor is the desktop, developed by Xerox PARC in the 1970s and "borrowed" by Apple and then Microsoft. But it isn't a real desktop - it's a metaphor, and not such a good one at that. Stuff appears on a vertical screen and not a horizontal desk. The sorts of things that appear there are not books, pencils, and papers, but documents and applications - or nicknames for applications. You don't reach for these things with your hands, but maneuver a small arrow using a round brick on your desk or by twirling a ball mounted on a holder; you don't grasp them with your fingers but operate them by pushing buttons on the brick or ball-holder.

Here, the designers have tried to use a physical metaphor to make understanding the software easier, but operating within a metaphor is not the same as operating in the world. Moving a mouse and clicking buttons to get things done is not natural to people nor are there the centuries of experience with machines similar to computers. Today we take for granted the relatively new idea of a steering wheel on a car. But such steering wheels have been around for about a hundred years, and the use of a "steering wheel" for controlling the direction of ships has been around since the 18th century, where it was found to produce better control with less effort because it used a block and tackle for mechanical advantage. And the steering wheel is a physical device - not a picture of one - and it's operated by placing one's hands on it - not by pointing an arrow at it and pushing a little button twice.

The two most successful software applications have been the spreadsheet and the text editor. The spreadsheet, developed by Dan Bricklin and Bob Franskton in 1979, mimicked the tabular notation of bookkeeping and married it with the functionality of popular business calculators of the time. The result - called "VisiCalc" - was something both familiar and new. Given the familiar, the new was easier to learn than if it were all new. The familiar parts were common to people who did business financials - the layout on the screen was not a metaphor, but a duplicate of what accountants would use on paper. What was new was the command-style programming of the spreadsheet.

Nevertheless, the spreadsheet was something never seen before. A chart indicating the 64 greatest events in accounting and business history contains VisiCalc.

The text editor is nothing more than the combination of a piece of paper in a typewriter with other commands and functionality that before could only be performed by retyping or retypesetting. Again, the software combined the familiar with the new: typing, moving around on the page using simple commands like , , and along with new functionality like search and replace, text justification, font control, and line wrapping. Nevertheless, habits die hard. Some fiction writers are used to revising their stories and novels by literally retyping their manuscripts and regaining the feeling and rhythms of typing the words. Such writers continue to do so: Printing out the manuscript and retyping it rather than fooling with small changes, looking for larger changes with deeper significance.

The first recognizable text editors were developed about 30 years ago. Since then we have gone through 3 generations and we're on the 4th - Microsoft Word being somewhere between the 3rd and 4th generations, not having become competent at typesetting. Yet people do not fully utilize the functionality available, or they get confused by the user interface. There are icons and tool interfaces all over the screen, and it's hard to tell what is controlled by what. Except for the typewriter part of the interface, nothing about what confronts a writer is familiar and is not part of the physical world. The physical controls of a computer control the virtual world of the software, and that virtual world need not obey any laws of physics or common sense. People are not used to making up paragraph templates or styles, they cannot make cross-references or indexes. Nevertheless, they can operate complex physical machines like lathes and trucks, cook using sophisticated stoves and ovens, ride bicycles with 24 gears, fly airplanes, and sail boats

These, though, are the simple cases. Most business software has no physical equivalents, and even metaphors are hard to come by. The people who pay for the software sometimes aren't the ones who will use it, and the people who use it will not be well-trained - they will simply have to "pick it up," so to speak. Because there is no requirement for the innards of software to make physical or common sense, and because the connections between the cartoonish controls on a computer screen don't have to have a physical or logical connection to what happens inside, people - whether trained or not - can develop incorrect mental models of what's going on inside the software and how their actions control things.

Designers of physical devices have an expert collaborator: the physical world. Because the physical world can, in theory, make it possible for someone to figure out the innards of the device and how they work, the designer can take advantage of his or her helper to make things sensible to a person using it. The designer of software has no one to help simplify things. The complexity of the model embodied in the software - which sometimes doesn't correspond to any real-world situation or any "model" of the problem being solved - is inscrutable, and leads directly to difficulty using the software. Keep in mind that the software not only has to model the objects and operations involved in the problem but also has to model objects and operations for presenting itself to the user, and for interacting with the user. The physical world of inanimate objects does not care about interacting with us, and leaves that messy job to the laws of physics. For software developers, it's left up to the laws of the imagination.

As software developers we face two grave difficulties: One is that we are building things that have never been seen on the planet before, and the other is that the tools we use to build it with are so brittle that it takes an enormous effort to get it done, and the smallest slip-up can cause none of it to work.

We learned these lessons in the 1970s, but things take time to percolate to the industry. In the mid-1980s, Harlan Mills at IBM introduced the Cleanroom Process. This process considers the program you are trying to write as an expression of a mathematical function. Here's how it works: First, you specify all the components of what you want to write and how they interact. Each component is assigned to a person or team which performs stepwise refinement using structures called "box structures." Through reviews and verification, all the components are shown to be correct. Then by testing statistically determined uses, an operational view of the software including its performance and correctness can be drawn. This testing is done by a separate group, and no code is executed by the development team prior to the independent testing.

If at some point a problem is discovered with the design or specification, the development group goes back to the design phase. In this sense it is an incremental process. With this process as well as with the Waterfall Model, you had better know what you want to build, because if it turns out you've built the wrong thing, going back and fixing it is going to take a long time given the programming languages and tools we use.