PragPub 2012-07: Issue #37

Table of Contents

Features

- Tales from the Ops Side
by Michael Nygard
Michael gives a blow-by-blow account of dealing with “the only load test that matters.”

- Technical Debt Counselors
by Christoph Pater
When technical debt is out of control, who you gonna call?

- Threads
by Michael Swaine
In writing about the history of the personal computer, the author finds some threads that run through the whole yarn.

- Scala for the Intrigued
by Venkat Subramaniam
In the latest installment of his series on the Scala language, Venkat shows the power of chaining traits.

Departments

- Up Front
by Michael Swaine
As one series winds down, two others begin.

- Choice Bits
The early reviews of Microsoft Surface are in and they’re... too early.

- Calendar
Author sightings, upcoming conferences, and other events of note.

- Shady Illuminations
by John Shade
John discovers that email is broken and offers no solutions.

- But Wait, There’s More...
Coming attractions and where to go from here.

Except where otherwise indicated, entire contents copyright © 2012 The Pragmatic Programmers.

Feel free to distribute this magazine (in whole, and for free) to anyone you want. However, you may not sell this magazine or its content, nor extract and use more than a paragraph of content in some other publication without our permission.

Published monthly in PDF, mobi, and epub formats by The Pragmatic Programmers, LLC, Dallas, TX, and Raleigh, NC. E-Mail support@pragprog.com, phone +1-800-699-7764. The editor is Michael Swaine (mailto:michael@pragprog.com). Visit us at http://pragprog.com

ISSN: 1948-3562

Up Front

Tales from the Ops Side
by Michael Swaine
Michael Nygard wrote an excellent book for us a few years back called Release It! We were looking over that book recently and rediscovering just how good it was. There are some great case studies in the book—truly dramatic stories from the trenches. So we thought, why not share some of those stories with the readers of PragPub?

So that’s what we’re doing. The first in this series recounts the chaos and quick thinking that occur when a new e-commerce site meets the dreaded day after Thanksgiving and the associated shopping madness.

Technical debt is a useful metaphor invented by Ward Cunningham for necessary work on a project that has been deferred. Like any good metaphor, it should be extensible, and in an essay in this issue, Christoph Pater asks the question, if there is technical debt, shouldn’t there be technical debt counselors?

Venkat Subramaniam has been conducting a class in Scala in the magazine for several months now, and the series is nearing its end. The current lesson involves Scala traits, how you can chain them, and why you would want to.

PragPub’s editor has written an article for this issue, too. It’s on personal computer history and a couple of people whose names just keep popping up.

John Shade is on deck, of course, this time with his eccentric take on email.

Choice Bits

Tweeted Tics and Trips

The early reviews of Microsoft Surface are in and they’re... too early.

What’s Hot

Top-Ten lists are passé—ours goes to 11. These are the top titles that folks are interested in currently, along with their rank from last month. This is based solely on direct sales from our online store.

1

1

Seven Databases in Seven Weeks

2^

NEW

Lean from the Trenches

3

3

Agile Web Development with Rails

4v

2

Practical Vim

5

5

Programming Ruby 1.9

6v

4

Working with Unix Processes

7^

8

The RSpec Book

8^

9

The Cucumber Book

9^

11

Seven Languages in Seven Weeks

10v

6

The Rails View

11^

NEW

Deploying Rails

Microsoft Surface: The Early Snark

I can’t put my finger on it, but there’s something superficial about the early reviews of Microsoft’s recent tablet announcement. Personally, I’m still waiting to see Microsoft Depth, but while I wait I thought I’d share some of the early smudges on Microsoft Surface.

Why Microsoft Surface has a keyboard? How else would you do CTRL+ALT+DEL? — @honzasterba

Sorry but Microsoft Surface looks like it’s afraid of being touched—kickstand, keyboard, stylus all keep fingers away from touchscreen. — @riptari

Danny Sullivan pointed out that Microsoft wasn’t too keen on letting journalists touch the Surface, either, which kind of undermined all those “hands-on” reviews. Also he contributed this prime snark:

I don’t care that much about a kickstand, much less one that has perfect-pitch sound. — Danny Sullivan

A lot of the early snark was about the announcement itself.

The “Surface” video, which is of a computer rendering, is hosted on youtube. That says everything. — @perrymetzger

THIS STORY CONTAINS NONE OF THOSE THINGS RT @rodtrent: Microsoft Surface specs, price, release date and more. | CNET UK — @JBrodkin

No idea on battery life, adapter size, price, or other miscellany. Yesterday’s Microsoft Surface announcement was severely incomplete. — @ChrisPirillo

That burger was sloppier than a Microsoft Surface announcement. — @SteveStreza

But this is Microsoft, so we can always look forward to Announcement 2.0:

Next week, Steve Ballmer will demonstrate the Microsoft Surface by using it from the inside of an underwater volcano. On a snowboard. — @cjoh

Steve’s quite the showman, but I’m not sure Google didn’t raise the bar with its recent Google Glass event. (“If the skydivers live, this’ll be an awesome demo.”)

Some took the occasion of Microsoft’s announcement to make fun of others, like Dell:

People don’t want a Microsoft tablet, says former CEO of dying company whose products no one wants, either. (Dell) — @mims

Acer:

The first sign that Microsoft’s vaporware tablet might be successful: Acer saying it won’t: http://www.huffingtonpost.com/2012/06/21/acer-microsoft-surface-tablet_n_1615939.html — @pragpub

Apple:

Microsoft’s hands were forced—an engineer left a 60-inch surface table in a bar. — @jkendrick

British politics:

Mr Cameron on the phone. He’s put a #microsoft sticker on the back of Mr Clegg’s Etch-a-Sketch and told him it’s a new Surface Tablet. — @Queen_UK

and everybody else:

To take a break from the snark, this does look better than what any of the Windows hardware partners have been producing lately. — @jeff_lamarche

Danny could be wrong about the kickstand. It seems to be a meme. Although I’m not absolutely certain that these tweets were about Microsoft:

I need a kickstand because I’m on lean. — @ForkWhippinRich

$1.50 a cup & I’m leanin’ like kickstand. — @RealMikeMontana

Yo momma gotta peg leg with a kickstand. — @ImSoEfficient

Life Lessons Learned

Learning to accept that you may very well already have had the best idea of your life is tough. — @dhh

You haven’t really learned anything until your behavior has changed. — @ellnestam

I’m astonished at the value to be gained from predicting the outcome of every programming experiment out loud beforehand. — @KentBeck

Every webcam should come with a small comb. And a toothpick. #justsaying — @gvwilson

APIs are the new business contracts where you don’t have to do a deal! Great until the terms are changed on you and you are screwed. — @dalmaer

With big push to online education http://bit.ly/MELQh6, how will students learn vital business skill of staying awake during staff meetings? — @marick

Who Are Those Guys?

They’re not all guys. Also, we have to confess that we cleaned up their punctuation and stuff a little. OK, who they are: Dion Almaer, Kent Beck, Jon Brodkin, Ola Ellnestam, David Heinemeier Hanssen, Clay Johnson, James Kendrick, Jeff LaMarche, Sean Love, Brian Marick, Perry E. Metzger, Christopher Mims, Mike Montana, Natasha, Chris Pirillo, Purple Promethazine, Honza Sterba, Steve Streza, Danny Sullivan, Michael Swaine, Greg Wilson, and Elizabeth Windsor. You can follow us at www.twitter.com/pragpub.

Tales from the Ops Side

Black Friday
by Michael Nygard
Michael gives a blow-by-blow account of dealing with “the only load test that matters.”

Several years ago, I worked for a company that offered 24x7 operations for websites that we didn’t create. As crazy as it sounds, we would not only support applications and systems that our clients built but also offer uptime and performance guarantees. During that time, I got what you could call a “crash course” in operations and in what it meant to build resilient systems. These systems taught me how well our typical applications were prepared to survive the harsh rigors of production. The answer was, “Not well at all.”

In this series, I’ll relate some incidents from that time. Names have been changes to protect the parties involved, but the essential details and interactions are all accurate. You may find these vignettes entertaining or enlightening, or you may shake your head at how unprepared we all were. As you read, please bear Kerth’s Prime Directive in mind. “Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.” Besides, it’s no fun to tell stories about times when everything went right!

In the middle 1500s, a Calabrian doctor named Aloysius Lilius invented a new calendar to fix a bug in the widely used Julian calendar. The Julian calendar had an accumulating drift. After a few hundred years, the official calendar date for the solstice would occur weeks before the actual event. Lilius’s calendar used an elaborate system of corrections and countercorrections to keep the official calendar dates for the equinoxes and solstices close to the astronomical events. Over a 400-year cycle, the calendar dates vary by as much as 2.25 days, but they vary predictably and periodically; overall, the error is cyclic, not cumulative. This calendar, decreed by Pope Gregory XIII, was eventually adopted by all European nations, although not without struggles, and even by Egypt, China, Korea, and Japan (with modifications for the latter three). Some nations adopted the Gregorian calendar as early as 1582, while others adopted it in only the 1920s.

It’s no wonder that the Catholic church created the calendar. The Gregorian calendar, like most calendars, was created to mark holy days (that is, holidays). It has since been used to mark useful recurring events in certain other domains that depend on the annual solar cycle, such as agriculture.

No business in the world actually lives by the Gregorian calendar, though. The business community uses the dates as a convenient marker for its own internal business cycle.

Each industry has its own internal almanac. For an insurance company, the year is structured around “open enrollment.” All plans take their bearings from the open enrollment period. Florists’ thinking is dominated by Mother’s Day and Valentine’s Day. These landmarks happen to be marked with specific dates on the Gregorian calendar, but in the minds of florists, and their entire extended supply chain, those seasons have their own significance, with no bearing on the official calendar date.

For retailers, the year begins and ends with the euphemistically called “holiday season.” Here we see a correspondence between various religious calendars and the retail calendar. Christmas, Hanukkah, and Kwanzaa all occur relatively close together. Since “Christmahannukwanzaakah” turns out to be difficult to say in meetings with a straight face, they call it “holiday season” instead. Don’t be fooled, though. Retailers’ interest in the holiday season is strictly ecumenical—some might even call it cynical. Up to 50% of a retailer’s entire annual revenue occurs between November 1 and December 31.

In the United States, Thanksgiving—the fourth Thursday in November—is the de facto start of the retail holiday season. (Some retailers have lobbied Congress asking the government to move Thanksgiving two weeks earlier.) By long tradition, this is when consumers start getting serious about gift shopping, because there are usually a little fewer than 30 days left at that point. Apparently, motivation by deadline crosses religious boundaries. Shopper panic sets in, resulting in a collective phenomenon known as Black Friday. Retailers encourage and reinforce this by changing their assortment, increasing stocks in stores, and advertising wondrous things. Traffic in physical stores can quadruple overnight. Traffic at online stores can increase by 1,000%.

This is the real load test, the only one that matters.

Baby’s First Christmas

My client had launched a new online store in the summer. The weeks and months following launch proved, time and time again, why launching a new site is like having a baby. You must expect certain things, such as being awakened in the middle of the night and routinely uncovering horrifying discoveries (as in, “Dear God! What have you been feeding this child? Orange Play-Doh?” or “What? Why would they parse content during page rendering?”) Still, for all the problems we experienced following the launch, we approached the holiday season with cautious optimism.

Our optimism was rooted in several factors. First, we had nearly doubled the number of servers in production. Second, we had hard data showing that the site was stable at current loads. A few burst events (mispriced items, mainly) had given us some traffic spikes to measure. The spikes were large enough to see where page latency started to climb, so we had a good feel for what level of load would cause the site to bog down. The third reason for our optimism sprang from the confidence that we could handle whatever the site decided to throw at us. Between the inherent capabilities of the application server and the tools we had built around it, we had more visibility and control over the internals of the online store than any other system on which I’d ever worked. This would ultimately prove to be the difference between a difficult but successful Thanksgiving weekend and an unmitigated disaster.

A few of us who had pulled weekend duty through Labor Day had been granted weekend passes. I had a four-day furlough to take my family to my parents’ house three states away for Thanksgiving dinner. We had also scheduled a twenty-four-hour onsite presence through the weekend. As I said, we were executing cautious optimism. Bear in mind, we were the local engineering team; the main Site Operations Center (SOC)—a facility staffed with highly skilled engineers twenty-four hours a day—was in another city. Ordinarily, they were the ones monitoring and managing sites during the nights and weekends. Local engineering was there to provide backup for the SOC, an escalation path when they encounter problems that have no known solution. Our local team was far too small to be on-site twenty-four hours a day all the time, but we worked out a way to do it for the limited span of the Thanksgiving weekend. Of course, as a former Boy Scout (“Be prepared”), I crammed my laptop into the packed family van, just in case.

Taking the Pulse

When we arrived on Wednesday night, I immediately set up my laptop in my parents’ home office. I can work anywhere I have broadband and a cell phone. Using their 3Mb cable broadband, I logged in to our jumphost and start up my sampling scripts. (Jumphost: a single machine, very tightly secured, that is allowed to connect via SSH to the production servers.)

Way back during the run-up to launch, I was part of load testing this new site. Most load tests deliver results after the test is done. Since the data come from the load generators rather than inside the systems under test, it is a “black-box” test. To get more information out of the load test, I had started off using the application server’s HTML administration GUI to check such vitals as latency, free heap memory, active request-handling threads, and active sessions.

If you don’t know in advance what you are looking for, then a GUI is a great way to explore the system. If you know exactly what you want, the GUI gets tedious. If you need to look at thirty or forty servers at a time, the GUI gets downright impractical.

(This particular application server, ATG Dynamo, still has the best admin GUI. It’s not as pretty as WebLogic or WebSphere, but it exposes every single component of the application server. Like the original Volkswagen Beetle engine, you can see every part, how well it’s working, and how it’s wired to other components. ATG was doing dependency injection long before Martin Fowler coined the term and way, way before Spring had sprung.)

To get more out of our load tests, I wrote a collection of Perl modules that would screen-scrape the admin GUI for me, parsing the HTML for values. These modules would let me get and set property values and invoke methods on the components of the application server—built-in as well as custom. Because the entire admin GUI was HTML based, the application server never knew the difference between a Perl module and a web browser. Armed with these Perl modules, I was able to create a set of scripts that would sample all the application servers for their vital stats, print out detail and summary results, sleep a while, and loop.

Shell scripts calling Perl modules that screen-scrape the admin GUI... it sounds like a Rube Goldberg machine, but it is totally normal for Operations to cobble something together when needed.

They were simple indicators, but in the time since site launch, all of us had learned the normal rhythm and pulse of the site by watching these stats. We knew, with a single glance, what was normal for noon on Tuesday in July. If session counts went up or down from the usual envelope, if the count of orders placed just looked wrong, we would know. It’s really surprising how quickly you can learn to smell problems. Monitoring technology provides a great safety net, pinpointing problems when they occur, but nothing beats the pattern-matching power of the human brain.

Thanksgiving Day

As soon as I woke up Thanksgiving morning, before I even had a cup of coffee, I hopped into my parents’ office to check the stats windows that I left running all night. I had to look twice to be sure of what I saw. The session count in the early morning already rivaled peak time of the busiest day in a normal week. The order counts were so high that I called our DBA to verify orders were not being double-submitted. They weren’t.

By noon, customers had placed as many orders as in a typical week. Page latency, our summary indicator of response time and overall site performance, was clearly stressed but still nominal. Better still, it was holding steady over time, even as the number of sessions and orders mounted. I was one happy camper over turkey dinner. By evening, we had taken as many orders in one day as in the entire month to date. By midnight, we had taken as many orders as in the entire month of October—and the site held up. It passed the first killer load test.

Black Friday

The next morning, on Black Friday, I ambled into the office after breakfast to glance at the stats. Orders were trending even higher than the day before. Session counts were up, but page latency was still down around 250 milliseconds, right where we knew it should be. I decided to head out around town with my mom to pick up the ingredients for chicken curry. (It would be Thanksgiving leftovers for dinner on Friday, but I wanted to make the curry on Saturday, and our favorite Thai market was closed on Saturday.)

Of course, I wouldn’t be telling this story if things didn’t go horribly wrong. And, things wouldn’t go horribly wrong until I was well away from my access point. Sure enough, I got the call when I was halfway across town.

“Good morning, Michael. This is Daniel from the Site Operations Center,” said Daniel.

“I’m not going to like this, am I Daniel?” I asked.

“SiteScope is currently showing red on all DRPs. We’ve been doing rolling restarts of DRPs, but they are failing immediately. David has a conference call going and has asked for you to join the bridge.”

In the terse code we’ve evolved in our hundreds of calls, Daniel was telling me that the site was down, and down hard. SiteScope—our external monitoring tool—shown in Figure 16.1, accesses the site the same way that real customer’s access it. When SiteScope goes red, we know that customers aren’t able to shop and we’re losing revenue. In an ATG site (a J2EE application server commonly used by online retailers), page requests are handled by instances that do nothing but serve pages. The web server calls the application server via the Dynamo Request Protocol, so it’s common to refer to the request-handling instances as DRPs. A red DRP indicates that one of those request-handling instances stopped responding to page requests. “All DRPs red” meant the site was down, losing orders at the rate of about a million dollars an hour. “Rolling restart” meant they were shutting down and restarting the application servers as fast as possible. It takes about ten minutes to bring up all the application servers on a single host. You can do up to four or five hosts at a time, but more than that and the database response time starts to suffer, which makes the start-up process take longer. All together, it meant they were trying to tread water but were still sinking.

“OK. I’ll dial in now, but I’m thirty minutes from hands on keyboard,” I told him.

Daniel said, “I have the conference bridge and passcode for you.”

“Never mind. I’ve got it memorized,” I said.

I dialed in and got a babel of voices. Clearly, a speakerphone in a conference room was dialed in to the bridge as well. There’s nothing like trying to sort out fifteen different voices in an echoing conference room, especially when other people keep popping in and out of the call from their desks, announcing such helpful information as, “There’s a problem with the site.” Yes, we know. Thank you, and hang up, please.

Vital Signs

The incident had started about twenty minutes before Daniel called me. The operations center had escalated to the on-site team. David, the operations manager, had made the choice to bring me in as well.

Figure 16.1: SiteScope Hits Front-End Store, Simulates Users

Too much was on the line for our client to worry about interrupting a vacation day. Besides, I had told them not to hesitate to call me if I was needed.

We knew a few things at this point, twenty minutes into the incident:

- Session counts were very high, higher than the day before.

- Network bandwidth usage was high but not hitting a limit.

- Application server page latency (response time) was high.

- Web, application, and database CPU usage were low—really low.

- Search servers, our usual culprit, were responding well. System stats looked healthy.

- Request-handling threads were almost all busy. Many of them had been working on their requests for more than five seconds.

In fact, the page latency wasn’t just high. Because requests were timing out, it was effectively infinite. The statistics showed us only the average of requests that completed. Requests that didn’t complete never got averaged in.

Other than the long response time, which we already knew about since SiteScope was failing to complete its synthetic transactions, none of our usual suspects looked guilty.

To get more information, I started taking thread dumps of the application servers that were misbehaving. While I was doing that, I asked Ashok, one of our rock-star engineers who was on-site in the conference room, to check the back-end order management system. He saw similar patterns on the back end as on the front end: low CPU usage and most threads busy for a long time.

It was now almost an hour since I got the call, or ninety minutes since the site went down. This means not only lost orders for my client but also that we were coming close to missing our SLA for resolving a high severity incident. I hate missing an SLA. I take it personally, as do all of my colleagues.

Diagnostic Tests

The thread dumps on the front-end applications servers revealed a similar pattern across all the DRPs. A few threads were busy making a call to the back end, and most of the others were waiting for an available connection to call the back end. The waiting threads were all blocked on a resource pool, one that had no timeout. If the back end stopped responding, then the threads making the calls would never return, and the ones that were blocked would never get their chance to make their calls. In short, every single request-handling thread, all 3,000 of them, were tied up doing nothing, perfectly explaining our observation of low CPU usage: all 100 DRPs were idle, waiting forever for an answer that would never come.

Attention swung to the order management system. Thread dumps on that system revealed that some of its 450 threads were occupied making calls to an external integration point, as shown in Figure 16.2. As you probably have guessed, all other threads were blocked waiting to make calls to that external integration point. That system handles scheduling for home delivery. We immediately paged the operations team for that system. (It’s managed by a different group that does not have 24/7 support staff. They pass a pager around on rotation.)

Figure 16.2: Front-End Store Hits Back-End Order Management

I think it was about this time that my wife brought me a plate of leftover turkey and stuffing for dinner. In between status reports, I muted the phone to take quick bites. By that point, I had used up the battery on my cell phone and was close to draining the cordless phone. (I couldn’t use a regular phone because none of them took my headset plug.) I crossed my fingers that my cell phone would get enough of a charge before the cordless phone ran out.

Call in a Specialist

It felt like half of forever (but was probably only half an hour) when the support engineer dialed in to the bridge. He explained that of the four servers that normally handle scheduling, two were down for maintenance over the holiday weekend, and one of the others was malfunctioning for reasons unknown. To this day, I have no idea why they would schedule maintenance for that weekend of all weekends! Figure 16.3 shows the relative sizes of the three systems involved.

Figure 16.3: Order Management Hits “Enterprise” Scheduling System

The sole remaining server could handle up to twenty-five concurrent requests before it started to slow down and hang. We estimated that the order management system was probably sending it ninety requests right at that moment. Sure enough, when the on-call engineer checked the lone scheduling server, it was stuck at 100% CPU. He had gotten paged a few times about the high CPU condition but had not responded, since that group routinely gets paged for transient spikes in CPU usage that turn out to be false alarms. All the false positives had quite effectively trained them to ignore high CPU conditions.

Our business sponsor, on the conference call, gravely informed us that marketing had prepared a new insert that hit newspapers Friday morning. The ad offered free home delivery for all online orders placed before Monday. The entire line, with fifteen people in a conference room on speakerphone and a dozen more dialed in from their desks, went silent for the first time in four hours.

So, to recap, we have the front-end system, the online store, with 3,000 threads on 100 servers and a radically changed traffic pattern. It’s swamping the order management system, which has 450 threads that are shared between handling requests from the front end and processing orders. The order management system is swamping the scheduling system, which can barely handle twenty-five requests at a time.

And it’s going to continue until Monday. It’s the nightmare scenario. The site is down, and there’s no playbook for this situation. We’re in the middle of an incident, and we have to improvise a solution.

Compare Treatment Options

Brainstorming ensued. Numerous proposals were thrown up and shot down, generally because the application code’s behavior under that circumstance is unknown. It quickly became clear that the only answer was to stop making so many requests to check schedule availability. With the weekend’s marketing campaign centered around free home delivery, we knew requests from the users were not about to slow down. We had to find a way to throttle the calls. The order management system had no way to do that.

We saw a glimmer of hope when we looked at the code for the store. It used a subclass of the standard resource pool to manage connections to order management. In fact, it had a separate connection pool just for scheduling requests. I’m not sure why the code was designed with a separate connection pool for that, probably an example of Conway’s law, but it saved the day—and the retail weekend. Because it had a component just for those connections, we could use that component as our throttle.

If the developers had added an enabled property, it would have been a simple thing to set that to false. Maybe we could do the next best thing, though. A resource pool with a zero maximum is effectively disabled anyway. I asked the developers what would happen if the pool started returning null instead of a connection. They replied that the code would handle that and present the user with a polite message stating that delivery scheduling was not available for the time being. Good enough.

Does the Condition Respond to Treatment?

One of my Perl scripts could set the value of any property on any component. As an experiment, I used the script to set max for that resource pool (on just one DRP) to zero, and I set checkoutBlockTime to zero. Nothing happened. No change in behavior at all. Then I remembered that max has an effect only when the pool is starting up.

I used another script, one that could invoke methods on the component, to call its stopService( ) and startService( ) methods. Voila! That DRP started handling requests again! There was much rejoicing.

Of course, because only one DRP was responding, the load manager started sending every single page request to that one DRP. It was crushed like the last open beer stand at a World Cup match. But at least we had a strategy.

I ran my scripts, this time with the flag that said “all DRPs.” They set max and checkoutBlockTime to zero and then recycled the service.

The ability to restart components, instead of entire servers, is a key concept of recovery-oriented computing. (See “Recovery-Oriented Computing” below.) Although we did not have the level of automation that ROC proposes, we were able to recover service without rebooting the world. If we had needed to change the configuration files and restart all the servers, it would have taken more than six hours under that level of load. Dynamically reconfiguring and restarting just the connection pool took less than five minutes (once we knew what to do).

Almost immediately after my scripts finished, we saw user traffic getting through. Page latency started to drop. About ninety seconds later, the DRPs went green in SiteScope. The site was back up and running.

Winding Down

I wrote a new script that would do all the actions needed to reset that connection pool’s maximum. It set the max property, stopped the service, and then restarted the service. With one command, an engineer in the operations center or in the “command post” (that is, the conference room) at the client’s site could reset the maximum connections to whatever it needed to be. I would later learn that script was used constantly through the weekend. Because setting the max to zero completely disabled home delivery, the business sponsor wanted it increased when load was light and decreased to one (not zero) when load got heavy.

We closed out the call. I hung up and then went to tuck my kids into bed. It took a while. They were full of news about going to the park, playing in the sprinkler, and seeing baby rabbits in the backyard. I wanted to hear all about it.

The Moral of the Story: Recovery-Oriented Computing

The Recovery-Oriented Computing (ROC) project is a joint Berkley and Stanford research project. The project’s founding principles are as follows:

- Failures are inevitable, in both hardware and software.

- Modeling and analysis can never be sufficiently complete.

- A priori prediction of all failure modes is not possible.

- Human action is a major source of system failures.

Their research runs contrary to most work in system reliability. Whereas most work focuses on eliminating the sources of failure, ROC accepts that failures will inevitably happen. Their investigations aim to improve survivability in the face of failures.

Many of the concepts of ROC can be implemented in today’s languages and platforms. Follow their focus on damage containment, automatic fault detection, and component-level restartability, and you’ll derive great benefits.

Michael is a veteran software developer and architect. His background is either “well-rounded” or “checkered” depending on how charitable you'd like to be. He has worked in Operations (including for a “Top Ten” internet retailer), sales engineering, and as a technology manager and executive. In 2007, Michael wrote Release It! to bring awareness of operational concerns to the software development community. This early influence in the DevOps movement showed developers how to write systems that survive the real world after QA. Send the authors your feedback or discuss the article in the magazine forum.

Technical Debt Counselors

A Modest Proposal
by Christoph Pater
When technical debt is out of control, who you gonna call?

In 1992 Ward Cunningham drew the comparison between technical complexity and debt, writing: "Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite...."

I remembered this as I watched a documentary series about a debt counselor on German television. "Hey, if there is something like technical debt in our software projects," I thought, "we should have technical debt counselors too!"

The more I thought about it, the more appropriate this stretching of the metaphor seemed. So let me stretch this metaphor a little more. If there were a technical debt counselor, what would such a person actually do?

Developing a metaphor requires mapping elements of one domain to corresponding elements in another. So first, we will map a feature that we need in our project to some cool thing that we can buy for money. The price we will pay for the cool thing is then a proxy for the time we need to develop and ship the new feature.

Credit Cards Ready?

OK, here we go. We are going into debt.

In the documentary series I mentioned above, people who are now loaded with debt got there by falling for (or to be less judgmental, taking advantage of) offers like: "buy now, pay later" to get all the cool things of their dreams.

So what about our cool, new software project? How do we get into technical debt? Well, first we need developers. (The inexperienced ones are cheaper so we hire more of them for the team.) Then we start the project, with good requirements, a great design, and the resolution to do it right.

Which lasts for a while, but pretty soon we remember that time is money, so we take a few shortcuts and do some hacks to get the features out the door. These are our first technical debts.

There is no problem with debt in general. But debt becomes a problem as soon as we can't keep up the payments. As soon as we can't both maintain our daily life and keep up the repayments. Or on our software project, if we need too long to deliver a feature or if we even can't deliver the feature any longer.

That's bad.

Help! Now What Do We Do?

It gets worse if we don't face the situation, or if we sugarcoat it.

Once you're in over your head, emergency measures are needed. If we went into financial debt, we could go into insolvency to get rid of it. In the documentary, the debt counselor comes to the rescue just before the people go into the insolvency. He tries to avoid the insolvency for them. He sorts out their finances, makes a plan for how to pay the bills and repayments, and as the most important point, regardless of his ability to avoid their insolvency, he trains them in how to organize their lives and how to live without going into debt again.

In our software projects we know such a thing as insolvency too. We call it rewrite.

But why do we not have technical debt counselors? We need such a person, who tells us how to save our project without a rewrite. And most important, as with the financial debt counselor, a technical debt counselor would show us how to write the code and how to care about and avoid technical debt.

He should help us to create a working process, coach the team how to take care for their project and train them on techniques to work with legacy code and how to use tests. He inspires us to do quality work.

In short: He helps us to do a great project!

Where Are the Technical Debt Counselors?

If you've ever heard a coworker say, "A rewrite every three years is normal" or saw a project that couldn't deliver needed features, you know it's true: We need technical debt counselors.

Too bad we don't have any.

That is, I don't know anybody whose business card reads "technical debt counselor." Not yet. But there is a movement called software craftsmanship. That's where I would look for the technical debt counselors of tomorrow.

Christoph Pater works as software engineer and scrum master at adesso mobile solutions. He writes the German blog "Software Leid(enschaft)" and organizes the local software craftsmanship community in Dortmund (Germany). Send the authors your feedback or discuss the article in the magazine forum.

Threads

Lee Felsenstein and Adam Osborne
by Michael Swaine
In writing about the history of the personal computer, the author finds some threads that run through the whole yarn.

Everything is deeply intertwingled.–Ted Nelson

Paul Freiberger and I are currently writing the third edition of Fire in the Valley, our history of the personal computer—to be published by The Pragmatic Bookshelf, naturally—and I’m struck by how often the same names and ideas recur, weaving in and out of the story.

I thought I’d try pulling out some of these threads and examining them in the pages of PragPub. I extricated two threads here. If this seems to work, I may tug on another thread next month.

Lee Felsenstein

The last time we were revising this book Paul was working at Paul Allen’s thinktank, Interval Research. One of his colleagues there was Lee Felsenstein. Lee gave us great interviews and insights for that revision, but that was no surprise, because he’d been hugely helpful to us when we were writing the first edition, including posing for a picture with all his inventions.

One reason Lee was so helpful is that the thread of his contributions to personal computing is long and is woven through some of the richest fabric of personal computer history.

The personal computer revolution caught fire with the Popular Electronics cover story on the MITS Altair in January 1975. But lots of revolutionaries were already loading their metaphorical muskets, and the revolution already had its Tom Paine and his Common Sense. (Ted Nelson and Computer Lib, respectively.)

Increasingly, the revolutionaries were getting together in clubs to exchange knowledge about computer technology, and the most famous of these clubs was the Homebrew Computer Club in Silicon Valley. The club had no official membership or dues, but it soon had an official master of ceremonies: Lee Felsenstein.

Lee proved to be quite adept at the job. “He kept order, he kept things moving, he made it fun to go to the meetings,” Homebrew member Chris Espinosa said. “There were 750 people in that room at one time, and he worked it like a rock concert.... [T]o see him work a crowd like a Baptist preacher.... He was great.”

Lee as master of ceremonies made perfect sense. Since moving to California for college he had spent his time at the intersection of social action and technology, looking for ways to connect people. He’d worked for The Berkeley Barb and another underground publication, The Tribe, where he combined counterculture goals and technical know-how. In 1972, three years before the Altair, he went to work as chief engineer for Resource One, an attempt to unify the switchboards of the Bay Area using a $120,000 XDS 940 from SRI they’d somehow got their hands on. A Resource One offshoot that Lee helped start, Community Memory, was trying to install public terminals in storefronts, where anyone who came in the door had access to a free computer network. Felsenstein was a connector and he was connected in both the tech and the social action communities: he’d been to Xerox PARC to see the innovations there before Steve Jobs got his famous peek.

The tech projects that Lee was pursuing fit right in with his activist agenda. His Pennywhistle Modem and his Tom Swift Terminal, needed to improve the user experience over the terminals used for Community Memory (putting it less delicately, the terminals broke down regularly), were both intended to empower people to connect. Appropriately, when Lee opened public discussion of the design of a better terminal, he did so on Community Memory.

That invitation to community design led to his producing what was for its time the coolest personal computer: the Processor Technology Sol. Bob Marsh manipulated Lee into designing a computer by telling him he’d pay him to design that terminal he kept talking about.

And designing the Sol set Lee up for his big gig: designing the Osborne 1.

Adam Osborne

The Osborne 1 debuted at the West Coast Computer Faire in April, 1981. IBM hadn’t thrown its PC into the market yet. Graphical user interfaces were years in the future. This was still the age of command-line computing and CP/M, and the Osborne 1 was a hit. For the first time, a computer came with all the software you needed. And it was portable—in the sense that you could carry it around with one hand and you could fit it under an airplane seat. All for $1795.

At that price, and considering that the bundled software would cost more than that if bought separately, you could overlook a lot of interesting design decisions. Squeezed by technological and budgetary limits, the screen was tiny. A pixel was bigger in those days, too, so all the screen would hold was 52 characters on a line. The off-the-shelf software bundled with the machine assumed a wider screen. Lee overcame that problem by making the screen a window on a larger virtual screen and giving users system-level ability to scroll the screen horizontally and vertically. Because it was to be portable, Lee made it tough, cushioning the picture tube. There wasn’t much precedent for any of this, so he made it up as he went along.

The company and the idea for the computer were the creation of Adam Osborne. Adam stood out in an industry of scruffy hippies with his precise diction, British accent, natty wardrobe, erect posture, and authoritative and dogmatic pronouncements. Adam would dismiss the leaders of this nacent industry with comments like, “Early personal computer companies were managed by amateurs who deluded themselves into believing that their transient success had something to do with good management or foresight.” Not that he was wrong, but he was also one of those early personal computer company managers.

It turned out that Adam’s most notorious pronouncement was “adequacy is sufficient.” It was a directive he gave to Osborne Computer Company (OCC) employees, and was ironic because months before OCC formed he was writing a column called “From the Fountainhead” in which he berated manufacturers for shoddy work. But for his computer, for his company, good enough was good enough.

And it looked for a while like he was right. OCC was the big overnight success story for the young personal computer industry, climbing more vertically than Apple or any other personal computer startup had. In a few months it was doing a million dollars in sales per month, which for the time was impressive. Within two and a half years it flamed out as spectacularly as it had blazed into prominence, but it was a bright light while it lasted.

Adam Osborne didn't come from out of nowhere. He was also a founder of the personal computer book industry. He got that started by selling his own book out of a box at the back of the Homebrew meetings. From this he was able to start a book company that was later purchased by McGraw-Hill. Osborne’s first big seller other than his own book was a book on Gary Kildall’s CP/M operating system by my friend Thom Hogan. That book helped Thom get hired as editor of InfoWorld, which is particularly meaningful to me since he then hired me. Thom later left InfoWorld to become director of software for OCC.

Osborne’s own books sold well, and his cred for writing them was substantial. When Intel was developing the microprocessors that hobbyists would use to launch the personal computer revolution, Osborne was the engineer they asked to write up the documentation for the chips. In truth, the Fountainhead in the title of his later magazine column referred to Intel and Silicon Valley, not to Adam himself, but if you knew him or read his writing, it was an easy mistake to make.

Closing the Loop

Adam also hired my friends Roger and Roy, who had worked with Thom and me at Data Domain, one of the first personal computer stores. That would be just a personal detail unworthy of mentioning, except that it lets me link back to the quote at the top of this article.

Because the founder and owner of the Data Domain, Ray Borrill, also started a company called Itty Bitty Computers. And his partner in that venture was none other than Ted Nelson.

About whom I think I may write next issue.

Michael Swaine is the editor of PragPub. Send him your feedback or discuss the article in the magazine forum. The photo of Lee Felsenstein is by Levi Thomas.

Scala for the Intrigued

Chaining Traits
by Venkat Subramaniam
In the latest installment of his series on the Scala language, Venkat shows the power of chaining traits.

We discussed traits in the previous article of this series. In this article we take that discussion further to show how traits can chain. Unlike the method collision worries in multiple inheritance, methods of traits can cooperate to elegantly implement the decorator pattern.

Let’s look at our first assignment....

We’ve been tasked by a large company to check records of candidates for employment. We’re told that there may be a number of criteria to check before a person can be hired. Let’s design a mechanism for checking in such a way that new criteria can be easily added and the combination of criteria can be easily changed.

Let’s first start with the Person class.

case class Person(first: String, last: String) {

override def toString = first + " " + last

}

Since there are different criteria, let’s create an abstract base class for them to conform to.

abstract class Checker {

def check(applicant : Person) : Boolean

}

The check method in the implementation classes will tell us whether a given applicant qualifies by returning a boolean result.

We will use this Checker in an evaluateApplicant function.

def evaluateApplicant(applicant : Person, checker: Checker) = {

println("Received application for " + applicant)

val result = if(checker.check(applicant)) "approved" else "disapproved"

println("Application " + result)

}

The evaluateApplicant method receives an applicant and an instance of Checker. It prints details of the applicant and then utilizes the services of the given Checker to decide if the applicant qualifies.

The company tells us we need to verify the education records of any candidate. The EducationChecker class will take care of that for us. It extends the Checker and overrides the check method.

class EducationChecker extends Checker {

override def check(applicant : Person) = {

println("checking education...")

//...

applicant.last.length > 1

}

}

The check method of EducationChecker will evaluate the education records of the given applicant by communicating with appropriate services that can provide necessary details. Here we will simply fake a solution that returns true if the person’s last name has more than one letter. Let’s use the above code on an applicant with a single-letter last name.

val john = Person("John", "X")

evaluateApplicant(john, new EducationChecker)

//Received application for John X

//checking education...

//Application disapproved

The education checker was invoked and it determined that the candidate is not qualified.

In addition to the education being checked for all candidates, the company wants the credit scores of some candidates (maybe those who will deal heavily with finances) to be checked.

We don’t want to add a credit check to the EducationChecker as it does not belong there. Furthermore, not all candidates need that check, so we need a way to mix the education check with credit check when we desire. In order to facilitate that, we will create a trait that will allow such mixing.

trait CreditChecker extends Checker {

abstract override def check(applicant : Person) = {

println("checking credit...")

//...

super.check(applicant) && (applicant.last.length > 1)

}

}

The CreditChecker trait extends the Checker abstract base class. This trait can be mixed into any class that extends from the Checker. When mixed in, this trait will form a chain to connect to the class it is mixed into. If multiple traits are mixed in, they form a chain in the order in which they’re mixed in. For example if two traits are mixed into a class, then the chain will contain an instance of the class, followed by an instance of the first trait mixed in, and then an instance of the second trait.

We need to override the check method of Checker in the CreditChecker trait. At the same time, we need the destination class, the class into which this trait will be mixed in, to provide the check method. We indicate this with a rather odd combination of keywords abstract override.

In the check method of CreditChecker, we evaluate the credit and combine that result with the result of the previous checker in the chain using the call to super.check(...).

Let’s use the CreditChecker in combination with the EducationChecker.

evaluateApplicant(john, new EducationChecker with CreditChecker)

//Received application for John X

//checking credit...

//checking education...

//Application disapproved

To use only the EducationChecker we can send an instance of that class to the evaluateApplicant function. If we want to add a check for credit, we can simply mix the CreditChecker trait in with the EducationChecker at the time of the call to evaluateApplicant.

The mixing of traits went fairly well. Now, we hear from the company that for some candidates we need to check education and criminal records. To accommodate this requirement let’s first create a CrimeChecker trait.

trait CrimeChecker extends Checker {

abstract override def check(applicant : Person) = {

println("checking criminal records...")

//...

super.check(applicant) && (applicant.last.length > 1)

}

}

Let’s use this trait along with the EducationChecker.

evaluateApplicant(john, new EducationChecker with CrimeChecker)

//Received application for John X

//checking criminal records...

//checking education...

//Application disapproved

Our design allows us to choose an EducationChecker and, in addition, optionally add another checker. Our design is quite extensible to accommodate more than one additional check. For instance, if the company wants us to apply all the above three checks, we can do that without writing any additional classes. Let’s try that on another candidate.

evaluateApplicant(Person("Mark", "Who"), new EducationChecker

with CrimeChecker with CreditChecker)

//Received application for Mark Who

//checking credit...

//checking criminal records...

//checking education...

//Application approved

We see how the traits collaborate with each other rather than collide. The order in which the methods are evaluated is from right to left, the reverse order of how the traits are mixed in.

When we design with traits, we can avoid class proliferation and mix in the appropriate traits to create a flavor or combination of features. The ability to mixin traits on instances provides the added advantage that we don't have to determine these combinations ahead of time.

Even though Scala allows mixing traits on instances, it does not compromise compile-time checking. Traits are entirely handled at compile time and this does not involve any runtime surprises.

Congratulations! In this article you learned how traits can form a chain and collaborate to implement a combination of behaviors. You can now mix this into your set of available design tools.

Dr. Venkat Subramaniam is an award-winning author, founder of Agile Developer, Inc., and an adjunct faculty at the University of Houston. He has trained and mentored thousands of software developers in the US, Canada, Europe, and Asia, and is a regularly invited speaker at several international conferences. Venkat helps his clients effectively apply and succeed with agile practices on their software projects. Venkat is the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy: Dynamic Productivity for the Java Developer and Programming Scala: Tackle Multi-Core Complexity on the Java Virtual Machine. His latest book is Programming Concurrency on the JVM: Mastering Synchronization, STM, and Actors. This series started in the September 2011 issue and has been running continuously since then. If you’d like to read the whole series, here are the links to the articles published so far:

1. 9/11: The Elegance of Scala

2. 10/11: Sensible Typing and Optional Items

3. 11/11: Cute Classes and Pure OO

4. 12/11: Functional Style of Programming

5. 1/12: Working with Collections

6. 2/12: Creating Higher Order Functions

7. 3/12: Pattern Matching

8. 4/12: XML as First Class Citizen

9. 5/12: Recursions and Tail Call Optimization

10. 6/12: Using Traits

11. 7/12: Chaining Traits

Send the author your feedback or discuss the article in the magazine forum.

Calendar

Author sightings, partner events, and other notable happenings.

Thanks to David Copeland for creating a tool to make it easy for authors to generate the code I need to post these! Here’s what’s coming up in the next three months:

Author Appearances

Who’s where, and what for.

- July 12, Columbia Java user Group, Columbia, MD
“Programming Groovy”Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- July 13–14, Greater Maryland Software Symposium, Columbia, MD
Talks on JavaScript, HTML5, Scala, and ConcurrencyVenkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- Jul 18, OSCON 2012, Portland OR
“You Ain’t SPDY”Chris Strom, Author of Dart for Hipsters and The SPDY Book

- July 19, OSCON 2012, Portland, OR
“Threads and JRuby, the Simple Alternative to Evented”David Copeland, Author of Build Awesome Command-Line Applications in Ruby

- July 20, OSCON, Portland, OR
“Your Own Metric System”Ian Dees, co-author of Using JRuby and author of Scripted GUI Testing With Ruby

- July 20–22, Lone Star Software Symposium, Austin, TX
Talks on JavaScript, HTML5, Scala, Concurrency, Groovy, Spock, and DSLVenkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- July 29–31, GR8Conf, Minneapolis, MN
Keynote, workshop, and multiple talks related to GroovyVenkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- Aug 3–5, Central Iowa Software Symposium, Des Moines, IA
Talks on JavaScript, HTML5, Scala, and ConcurrencyVenkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- August 9–11, CocoaConf, Columbus OH
“Core Audio All-Day Tutorial,” “Mobile Movies with HTTP Live Streaming,” and "Core Audio Cranks It Up”Chris Adamson, co-author of iOS SDK Development

- August 9–11, CocoaConf, Columbus OH
“Intro to iOS Programming”Bill Dudney, co-author of iOS SDK Development

- August 9–11, CocoaConf, Columbus OH
“iOS Performance Tuning with Instruments”Bill Dudney, co-author of iOS SDK Development

- Aug 12–17, Agile 2012, Dallas, TX
“Helping with Testing and Quality Assurance Stage Production”Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- Aug 24–26, Research Triangle Software Symposium, Raleigh, NC
Talks on Groovy, Grails, Concurrency, and HTML5Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- Aug 30, Colorado Springs User Group, Colorado Springs, CO
TBDVenkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- Sept 5–7, Swipe Conference, Sydney, Australia
“Intro to iOS Programming”Bill Dudney, co-author of iOS SDK Development

- Sept 17, HTML5 Denver, Denver, CO
“CoffeeScript and Testing JavaScript”Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

- Sept 30–Oct 4, JavaOne, San Francisco, CA
Multiple topics on Java, Groovy, and ScalaVenkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

O’Reilly Events

Upcoming events from our friends at O’Reilly.

- July 16–20, O’Reilly Open Source Convention Portland, OR
OSCON: “If you work with open source, you’ll want to be here. With a dozen plus tracks and hundreds of sessions, you’ll find practical tutorials, inspirational keynotes, and a wealth of information on open source languages, platforms, and development.”

USENIX Events

What’s coming from our USENIX friends.

- July 23–24, Asia-pacific Systems Workshop Seoul, South Korea
APSYS 2012: “Building on the success of its 2010 and 2011 predecessors in New Delhi, India and Shanghai, China, APSys 2012 will be a lively forum for systems researchers and practitioners across the world to meet, interact, and collaborate with their peers from the Asia/Pacific region.”

- Aug 8–10, 21st USENIX Security SymposiumBellevue, WA
USENIX Security ’12: “The USENIX Security Symposium brings together researchers, practitioners, system administrators, system programmers, and others interested in the latest advances in the security of computer systems and networks. The Symposium will span three days, with a technical program including: Keynote Address by Dickie George, Johns Hopkins Applied Physics Laboratory; over 40 Refereed Papers covering the latest research including topics such as browser security, CAPTCHAs, and passwords; privacy enhancing technologies; botnets and Web security; and more.”

- Aug 6–7, 2012 Electronic Voting Technology Workshop/Workshop on Trustworthy ElectionsBellevue, WA
EVT/WOTE ’12: “EVT/WOTE ’12 brings together researchers from a variety of disciplines, ranging from computer science and human-computer interaction experts through political scientists, legal experts, election administrators, and voting equipment vendors.”

- Aug 6–7, 3rd USENIX Workshop on Health Security and PrivacyBellevue, WA
HealthSec ’12: “The focus of HealthSec ’12 will be on the development of new techniques and policies to ensure the privacy and security of next-generation healthcare systems and devices.”

- Aug 6–7, 6th USENIX Workshop on Offensive TechnologiesBellevue, WA
WOOT ’12: “WOOT ’12 aims to bring together researchers and practitioners in systems security to present research advancing the understanding of attacks on operating systems, networks, and applications.”

- Aug 6, 2nd USENIX Workshop on Free and Open Communications on the InternetBellevue, WA
FOCI ’12: “FOCI ’12 seeks to bring together researchers and practitioners from technology, law, and policy who are working on means to study, detect, or circumvent practices that inhibit free and open communications on the Internet.”

- Aug 6, 5th Workshop on Cyber Security Experimentation and TestBellevue, WA
CSET ’12: “The science of cyber security is challenging for a number of reasons. Meeting these challenges requires transformational advances, including understanding of the relationship between scientific method and cyber security evaluation, advancing capabilities of underlying experimental infrastructure, and improving data usability.”

- Aug 7, 7th USENIX Workshop on Hot Topics in SecurityBellevue, WA
HotSec ’12: “HotSec places its singular emphasis on new ideas and problems. HotSec takes a broad view of security and privacy and encompasses research on topics including, but not limited to, large-scale threats, network security, hardware security, software security, physical security, programming languages, applied cryptography, privacy, human-computer interaction, emerging computing environment, sociology, and economics.”

- Aug 7, Seventh Workshop on Security MetricsBellevue, WA
MetriCon 7.0: “How to define, generate, and communicate security metrics you can use today.”

Other Happenings

- July 9
Browser pioneer Marc Andreessen is 41.

- July 16
VisiCalc co-creator Dan Bricklin is 61.

- Aug 9
Alfred Aho, co-creator of AWK language, is 71.

- Aug 11
Steve Wozniak is 62.

- Aug 12
The IBM PC was released 31 years ago today.

- Aug 20
Game developer John Carmack is 42.

- Aug 21
Linus Torvalds announces Linux project, 1991.

- Aug 29
Boy genius Stephen Wolfram is 53.

- Sept 1
This month the GNU Project turns 29.

- Sept 2
First message sent over Internet, 1969.

- Sept 2
sendmail creator Eric Allman is 57.

- Sept 9
C creator Dennis Ritchie is 71.

- Sept 10
Space tourist Charles Simonyi is 64.

- Sept 27
Perl creator Larry Wall is 58.

Shady Illuminations

Going Postal
by John Shade
John discovers that email is broken and offers no solutions.

Email is like oxygen—it is essential to sustain life on this planet, and yet too much of it makes one dizzy. —Alexander Sidorkin

“Going postal” is poised for a comeback. Soon it won’t be just disgruntled employees of an outsourced government service who are freaking out. Email is trying its best to turn us all into psycho postal workers. Even the strongest of you are bending under the strain. Eventually you’re going to break.

Because we’re drowning in unwanted emails, threatened by scams and malicious code hidden in emails, and unable to manage the flood of incoming messages. Email is broken. You can confirm this in the usual way: by counting the number of hits you get for “fix email.”

Somebody Ought to Fix This

It’s time to fix email.

I’m not talking about fixing your default email account after Facebook silently switches it to you at them dot com. I’m talking here about fixing email itself. The technology. The ubiquitous aspect of life in century 21.

It’s not that nobody has tried to fix email. To reinvent it. AOL tried. Google tried several times. It’s just that the attempts to fix email are all themselves broken.

David Pogue says he wants to fix email, but he really means he wants people to be better at it. Presumably the people sending him emails. Well, he’s a popular guy. I’m sure he gets a lot of crappy email from people who are total numbskulls. And since he has a platform to talk to the world, why not use it to chastise his fan base and nag them to send him less sucky emails?

But while that might be a good plan for a popular industry columnist, the strategy does not scale to the rest of us. Anyway, it probably won’t work even for Pogue because he’s trying to fix people. Very hard. It’s much easier to fix technology. Or the protocols and ceremony thereof.

(By the way, back when Apple was using the phrase “the rest of us,” did you flash on an image of Ronald Reagan with no legs every time you heard it? No? I guess it was just me, then.)

Paul Graham’s a smart guy. He would like to fix email. He says email is a todo list. He’s crazy. And I say that as a person who sends himself emails late at night reminding his morning self about things to do. But email isn’t a to-do list. If other people can email items into your to-do list, you don’t have an email problem, your life just sucks.

This person apparently thinks email isn’t complicated enough. These people think it’s a platform to build on. So if you have the app you can read the message? So broken. This person wants corporate email to work like the post office. Except that she wants more control over who can email her. But email isn't a newsgroup, it’s an open communication channel; it’s mail. The discovery aspect of it is crucial.

One thing that doesn’t fix email is wholesale blocking. Blocking email from people you know you don’t want to hear from ever again, sure, that’s smart. But blocking email from people you have no idea whether you want to hear from is like—old school reference here, people under 30 may have trouble following this—ripping the phone out of the wall because you don’t like to hear it ringing. You do that only because the system is broken and you’re desperate. You’re in fact breaking email yourself. You’re ripping out the discovery element of the tool. You’re hiding from the world. And while it sounds good to me, people tell me that’s a bad thing.

I Don’t Do Solutions

So how do we fix email? Beats me. I don’t do solutions. I specialize in problems. They’re easier to find.

I mean, I guess I could propose replacing the “letter in an envelope” metaphor with the “postcard” metaphor. If we have no privacy anyway, why not eliminate the whole time-consuming “opening the email” step? Turn the subject line into the message and require anything longer to be packaged up in a separate Dropboxed file. The point being to make it less easy to write long emails. Then people might think about them longer, improve the content, decide not to send dumb and unnecessary emails.

How about if it were impossible to archive emails? Once you open an email you have to do something with it, because it poofs away when you close it. And disappears after a few days even if unopened?

Probably all bad ideas. I told you I don’t do solutions. But something else I don’t do is change my behavior unless forced. If the trouble with email is a people problem, caused by the behavior of both the senders and receivers of email, then fixing email depends on changing their behavior.

Technology should discourage bad email behavior. Just don’t ask me how.

John Shade was born under a cloud in Montreux, Switzerland, in 1962. Subsequent internment in a series of obscure institutions of ostensibly higher learning did nothing to brighten his outlook. If even after reading this you just have to, you can send the author your feedback or discuss the article in the magazine forum.

But Wait, There’s More...

Coming Attractions

This may be the last page, but that doesn’t mean it’s the end of the conversation. Keep up with what’s going on in the Pragmatic universe by subscribing to our newsletter and drop by the Pub again next month for more Pragmatism. We’ll keep a seat warm and a mug chilled and a candle burning in the window.

Coming Soon in PragPub

In the works: Sam Aron is writing an article on using data structures to represent musical compositions and synthesis with working examples in the Clojure music toolkit Overton.

Right. But to really be in the know, you need to subscribe to our weekly newsletter. It’ll keep you in the loop, it’s a fun read, and it’s free. All you need to do is create an account on pragprog.com (email address and password is all it takes) and select the checkbox to receive newsletters.

While you’re waiting for the next issue of the newsletter or of PragPub to come out, you can follow us on twitter at @pragpub, @PragmaticAndy, or @pragdave. Or on Facebook at facebook.com/PragmaticBookshelf. Or email us at michael@pragprog.com.