PragPub 2012-11: Issue #41

Table of Contents

Features

Thinking Functionally with Haskell

by Paul Callaghan

In which Paul explores some powerful ideas about plumbing.

The Cloud Saves Money

by Jesse Anderson

You know how to manage the move to The Cloud, but what if you’re asked to cost-justify it? Could you?

The JavaOne Snooze

by Brian Tarbox

Brian returns to the big Java conference and finds it changed.

Threads

by Michael Swaine

Gary Kildall was a programmer’s programmer.

Departments

Up Front

by Michael Swaine

Haskell and monads and functional programming, how to cost-justify the move to the cloud, the return of our Quiz department, a report on the JavaOne conference post-Oracle, John Shade on pesky users and their annoying sense of entitlement, our events calendar, Choice Bits, and a recollection of Gary Kildall.

Choice Bits

What books are hot right now, what Twitter is for, and a new featurette, The Talk of the Tech.

Calendar

Author sightings, upcoming conferences, and other events of note.

The Quiz

An occasional diversion at least peripherally related to programming.

Shady Illuminations

by John Shade

John thinks users are a grinding noise in the gears of progress.

But Wait, There’s More...

Coming attractions and where to go from here.

Except where otherwise indicated, entire contents copyright © 2012 The Pragmatic Programmers.

Feel free to distribute this magazine (in whole, and for free) to anyone you want. However, you may not sell this magazine or its content, nor extract and use more than a paragraph of content in some other publication without our permission.

Published monthly in PDF, mobi, and epub formats by The Pragmatic Programmers, LLC, Dallas, TX, and Raleigh, NC. E-Mail support@pragprog.com, phone +1-800-699-7764. The editor is Michael Swaine (mailto:michael@pragprog.com). Visit us at http://pragprog.com

ISSN: 1948-3562

Up Front

Now with More Antioxidants

by Michael Swaine

Paul Callaghan is back this month with another adventure in functional programming. Paul’s language of tutorial choice is Haskell, but the topics he covers are not language-specific. This month he tells you more than you thought you wanted to know about monads.

Brian Tarbox is another author whose work has appeared here before. This month Brian has something different for us. He attended this year’s JavaOne conference, and he thought you might be interested in knowing if and how it has changed since it came under the Oracle umbrella.

Jesse Anderson’s article is a bit of a departure, too. It’s the first of two articles on moving to the cloud. But Jesse isn’t telling you how to execute the move to the cloud. He’s offering help with the problem you’re likely to run as you pitch the move. He shows you how to cost-justify the move for decision-makers you may have to convince.

If you don’t know how important Gary Kildall is in the history of the personal computer, you’ll want to read the article your humble editor offers up this month. And our irascible columnist John Shade is irascible as usual, taking on users with a sense of entitlement.

The part of the table of contents headed “Departments” lists the recurring elements of the magazine, as opposed to the feature articles. This month there is some activity in Departments beyond the routine. We’re experimentally adding a minor element to “Choice Bits” called “Talk of the Tech”—brief news items of a generally technical nature. If you like it, we’ll keep it. Also, our Quiz department is back this month, with a simple cryptarithmic puzzle, the answer to which, if you need it, will appear next month.

Meanwhile, take care of yourself. It’s a scary world out there. Hurricanes hook up with winter storms and trash the neighborhood. The online store decides to crash on the day after Thanksgiving. You find out that you’re a node on the critical path when you come down with the flu and the project screeches to a halt.

So keep yourself healthy. Get plenty of antioxidants. Eat pomegranates. November in the USA is National Pomegranate Month. Be well. And enjoy the issue!

Choice Bits

Hot Books, Deep Tweets, and The Talk of the Tech

Now with extra talk.

What’s Hot

Top-Ten lists are passé—ours goes to 11. These are the top titles that folks are interested in currently, along with their rank from last month. This is based solely on direct sales from our online store.

1^ | 3 | Practical Vim
2^ | NEW | Core Data
3^ | NEW | Build Awesome Command-Line Applications in Ruby
4^ | NEW | tmux
5^ | 6 | Agile Web Development with Rails
6^ | NEW | Practical Programming
7v | 1 | The Pragmatic Programmer
8^ | NEW | The Developer's Code
9^ | 11 | The RSpec Book
10v | 7 | Programming Ruby 1.9
11^ | NEW | Technical Blogging

The Talk of the Tech

Curiosity is in his title. John Grotzinger, NASA’s Curiosity project manager, has a pretty cool job. He gets to direct the research efforts of NASA's Curiosity rover. He and his team have already demonstrated evidence of liquid water on ancient Mars, and they are now in hot pursuit of measurable methane on the planet. Even if Grotzinger’s team find methane—and convincingly confirm that finding—it still won’t be unequivocal evidence of microbial life. But it will be tantalizing enough that their next step will be to focus on tracking down the source.

Not-so-fine Corinthian leather. Reacting to Scott Forstall’s abrupt exit from Apple, John Pavlus at Technology Review gives an impassioned argument for not throwing the skeuomorphic baby out with the Scott Forstall bathwater. Thankfully, Pavlus doesn’t go so far as to defend the Corinthian leather look of some iApps. As streakers at football games invariably learn, the natural look can be taken too far.

The best thing on Facebook? If you’re on Facebook, skip this item, because it’s old news to you. But if you’re not on Facebook, congratulations. You have more time than the rest of us. Use it wisely. On the other hand, I have to inform you that you are missing out on George Takei’s wonderful stream of found digital objects. Right, the actor who played Sulu on the original “Star Trek.” He’s found a new career, or maybe it’s a hobby.

Getting the picture. Looking at a few examples very often clarifies a confusing problem. Here’s a math problem presented in words: If n is an even integer, there is a function f(i,n) that produces, for values of i from 2 through n-1, a sequence of integers the last half of which counts down from n/2 - 1 to 1. What is f? Chances are the answer is not immediately leaping out at you. But if you write down the values of f(i,12) for i from 2 to 11, paired with the corresponding values of i, you’ll may see it immediately. If not, the answer is below.

C++ waking up? C++ is not a sexy language these days, but it is an important one. Peter Bright at Ars Technica is reporting that the future of C++ is getting—well, not sexier, but let’s say brighter. Microsoft, Intel, Google, IBM, and several other companies are committing to doing something about the doldrums the language has been in. They aren’t going to make it any sexier to program in C++, but they are talking about faster updates to language standards and more conformance to those standards.

Easy come, easy go. Sometimes we could benefit from communication media that forced us to slow down a little. News comes out that George Lucas is selling LucasFilm to Disney for four billion dollars. There follows a flurry of sniping and grumbling. (“George Lucas sold my childhood!”) Then comes the announcement that Lucas is giving the entire four billion to charity. Ulp.

HTTP headers, in bed. Thanks to @snipeyhead, here’s a full list of HTTP headers as fortune cookies. In bed. Some particularly apt ones: 405, 411, 412, 417, 429, 451, and the reassuring 200.

Just for you. If you’re a Pragmatic Bookshelf author and haven’t checked your Author Dashboard lately, you’ll find that we’ve made some changes. Nothing drastic, just some tweaks that should make it easier to get to the information you’re looking for. Also some nifty Buy-Now widgets for your website. Hope you like it.

Answer to math problem: f is n mod i.

Debris from the Twitterstorm

Please tell us what you’ve been up to lately.

Thinking about Literate programming. Trying to write my code as more of a story, but it feels more like a “choose your adventure.” — @dalmaer

Writing rebase jokes for parody lyrics to “call me maybe,” designing an API for db migrations, drinking coffee. — @selenamarie

Kind of a November tradition—I’m blogging encouraging tips for writers every day in parallel with NaNoWriMo at http://dimsumthinking.com. — @dimsumthinking

Torn between the desire to suck out all the marrow of this trip, and the desire to be chill about it cuz i can & will come back. — @amyhoy

For those wondering if I was interviewing for a job today: if I have to wear a suit to interview, that’s not a job I want. — @chadfowler

I said that whichever team increases code coverage the most gets to pick my next hair color. — @pamelafox

Do share your protips and truefacts.

Pro tip: always add a phone number in your e-mail signature, so that the conversation could move to the next level any time. — @afilina

You'd be surprised how many “highly performing teams” are really just a number of self-esteeming individuals who self-promote. — @tottinge

There are countless ways to express an idea. This makes it quite interesting and challenging at the same time. — @venkat_s

True fact: @jimweirich walks up to little kids and pulls Y-combinators out of the ears instead of nickels. #rubyconf — @therealadam

OH: “If Steve Jobs was still in charge Mobile Safari on iOS6 wouldn’t fire duplicate onreadystatechange events!” — @dalmaer

“After you've been tased, the world is new.”-@BecomeUseful — @thisisstar

And just ask, because somebody must know.

Why do all English hotels seem right out of Fawlty Towers? Seems there must be a training program for the proprietors. — @jwgrenning

How can we see light from the early universe arriving just now? How did we outrun the light in the first place? @StarTalkRadio — @jeffcohen

Isn’t there a word for when you’re on a ship and you have to dump heavy things in order to go faster? — @amyhoy

Talked to a software developer who had never heard of Dropbox. How is that possible...? — @PragmaticAndy

Who Are Those Guys?

First, they’re not all guys. Second, we have to confess that we cleaned up their punctuation and stuff a little. OK, who they are: Dion Almaer, Jeff Cohen, Selena Deckelmann, Anna Filina, Chad Fowler, Pamela Fox, James Grenning, Amy Hoy, Andy Hunt, Adam Keys, Tim Ottinger, Star St. Germain, Daniel Steinberg, and Venkat Subramaniam.

You can follow us at www.twitter.com/pragpub.

Thinking Functionally with Haskell

Is There Anything Left to Say about Monads?

by Paul Callaghan

In which Paul explores some powerful ideas about plumbing.

Apparently, there’s a view that Haskell is 99% monads (maybe more), and that monads are some arcane mystical concept that only a few can master. Wrong and wrong.

My aim this month is to talk a bit about why patterns like monads are useful, particularly explaining what such patterns means for programming and how to use them to keep your code under control—but not to over-use them.

I Remember the Time before Monads

Haskell and similar languages did exist before these ideas were applied to programming, so there really was a time before monads. It was not a barren wasteland, where all we could do was write programs to manipulate trees (or burn electricity), and had no interaction with the outside world at all.

We really could do real-world stuff, including file operations, console IO, and IPC, though it was a bit clumsy in places. Around that time, I was doing my PhD in the context of a large Natural Language Processing system, around 60K lines of Haskell—so one of the largest functional programs of its time. The program could process and analyze a Wall Street Journal article in a few seconds, build a complex semantic representation of the story, output various summaries of it, and allow interactive Q&A on the contents. It didn’t use a single monad.

It was, however, a time of exploration, when researchers explored various ideas to find a good way of both having our cake and eating it. Monads are one of the solutions they found, and essentially gave us a small but flexible API for working with computations (like IO operations or state modifications, or various combinations thereof) as opposed to simple data values, and did so elegantly within the standard langage. It got even better when syntactic sugar was added. This point of operating within the language is important: avoiding ad hoc extensions does help to keep the language simple.

This simple idea provided an excellent structuring pattern to tame a lot of clumsy code, and even more useful, gave us a solid framework for exploring more powerful ideas.

So monads are highly useful for some aspects of programming work, but they are certainly not an essential or core part. You will probably find that most large, well-written Haskell programs contain about 50-80% code that does not involve monads at all—the bulk is just pure data manipulation. Of the remainder, the monad use is mostly straightforward and follows certain common idioms. Real scary stuff is pretty rare.

Monads Are One Kind of Plumbing

Remember what we’re trying to do as (functional) programmers: use the language in full to let us program in a clear and direct way.

What kinds of things get in the way? Put another way, have you written code and wished that you could abstract out certain “noise” elements to leave a clearer “signal” for the key operations of your code? Here are a few of the noisy elements:

verbose syntax

(frequent) type annotations

error handling

passing common arguments

threading mutable state

handling multiple results

Haskell & Co. already score well on the first two, as we explored in past months. In a functional context, dealing with the other items boils down to needing more flexible ways to join operations together, or to compose them. For example, we can write and understand (ruby-style) code like this:

| if (res_1 = do_step_1).nil?
| error_1
| elsif (res_2 = do_step_2(res_1)).nil?
| error_2
| else
| use_vals(res_1, res_2)
| end

But wouldn’t it be preferable to write something like the following, which makes it quite clear that the operation is a particular sequence of steps that uses the intermediate values in a certain way? And let all of the error checking be handled behind the scenes?

| res_1 = do_step_1
| res_2 = do_step_2(res_1)
| use_vals(res_1, res_2)

Of course, we can hide some of the error checking with conventional exceptions, but exceptions can be a bit of overkill sometimes, i.e. a sledgehammer. Plus, they only deal with error handling and none of the other phenomena we might want to hide away, like state handling. Is there a more general mechanism we can use?

Basically, yes, and monads are one of several ideas that can be used. Let’s start by revisiting the idea of a pipeline of transformations first.

The Simplest Form of Pipeline

A few of my examples have already used pipeline-style code, like

take 3 $ map reverse $ words "one two three four"

using $ to chain several transformations together. Recall that you can read such pipelines in either direction, e.g. take 3 of the reversed words in the string... The $ operator is also a close relative of . (function composition), though it’s easier initially to explain things in terms of $. It is defined as follows, including the precedence and associativity declaration and the type signature for reference.

| infixr 0 $
| ($) :: (a -> b) -> a -> b
| f $ x = f x

As mentioned earlier, the definition is boring—applying its function argument to its other argument, and the main trick of $ comes from its declaration as a right-associative operator, which means we can write f $ g $ h x instead of f (g (h x)). The precedence level of 0 means it has the lowest priority level, so reverse $ "a" ++ "b" means reverse ("a" ++ "b") and not (reverse $ "a") ++ "b".

Sometimes it is handy to write the pipeline the other way round, so let’s define € as an alternative to $—exactly the same as above but the parameters are in a different order.

| infixl 0 €
| (€) :: a -> (a -> b) -> b
| x € f = f x

The initial example can now be written in a more OO-chaining style as

"one two three four" € words € map reverse € take 3

This operator has been declared as left-associative and lowest priority, hence the above is parsed as a left-nested

((("one two three four" € words) € map reverse) € take 3)

which is the grouping that makes sense.

Passing Values along the Pipeline

Can the conditionals example be written in this style too? Basically yes, though the € is too simple and we’ll need something else—let’s call it £ for now. This £ needs some way to grab the intermediate result values and pass them along for use later. Conveniently, lambdas (anonymous functions) are an excellent fit for this! So how about this:

| do_step_1 £ \res_1
| -> do_step_2 res_1 £ \res_2
| -> use_vals (res_1,res_2)

To understand how the lambdas help here, we need to understand a key detail of lambda syntax: the important rule is that in \var -> ..., the expression after the arrow extends as far to the right as possible, and the parameter name is available throughout the body of the function, even inside nested functions (unless the name gets shadowed). Technically, it’s not an operator like + or $, but you can think of it as acting like a right-associative operator with a low precedence of (-1). So the above parses as a right-nested tree:

| do_step_1 £ (\res_1
| -> (do_step_2 res_1 £ (\res_2
| -> (use_vals (res_1,res_2))))

Or in tree form, the parse result looks like this:

| £ ---- do_step_1
| |
| +-- \res_1 -> £ ---- do_step_2 res_1
| |
| +-- \res_2 -> use_vals (res_1,res_2)

The top £ node has a big function as its second argument, and the function grabs the input (from the first argument) and uses it in various places in the body of the function: as input to do_step_2 and also again when computing the result value. The inner function gets the result from do_step_2 res_1 under the name of res_2 and then returns the combined result.

So, we’ve seen that we can express code in an imperative-ish style using a suitable operator, and that the anonymous function support in Haskell helps to tie inputs and outputs together in the pipeline. Next, we need to consider what we’re hiding and how we’re hiding it, which means, what is the definition of £ going to be?

A Pipeline for “Conditional” Code?

The glue we need depends on what the component operations actually do, which depends also on their types.

The original Ruby-style code suggested operations that return a value or nil. Thankfully, Haskell has no notion of nil—we only deal with values and it’s impossible to not have a value. (Franklin Chen has a great talk on “various aspects of nil” - definitely worth a look. Do you know what the “billion-dollar mistake” is?) The Haskell approach is to use an “option type” to distinguish between a value being useful/present, or not. The standard version of this is Maybe. Some people (including me) see it as a kind of box: it is either empty (Nothing) or full (Just something), and the polymorphism allows it to contain values of any type. The full type also reflects what kind of thing is in the box, e.g. Just "foo" has type Maybe String—i.e., a String in the box.

| -- already defined in the Prelude
| data Maybe a = Nothing | Just a

This Maybe type is ideal for representing the return results from operations which may or may not return a useful value. For example, looking up a key in a table may or may not find a value. Haskell’s prelude contains a simple function called lookup for small tables, e.g. lookup 'a' [('a', 10), ('b', 20)] returns Just 10 but lookup 'c' [('a', 10), ('b', 20)] returns Nothing.

| lookup :: Eq a => a -> [(a, b)] -> Maybe b
| lookup k [] = Nothing
| lookup k ((x,y):xys) | k == x = Just y
| | otherwise = lookup k xys

Maybe has other uses too, like representing when some component of a data structure is optional, e.g. someone’s age can be Nothing or Just 42. When using such values, the extra detail ensures that you handle the 'nil' cases properly: you can’t treat a Maybe Int value as an Int—it has to be unpacked explicitly. There are no null pointer exceptions. Ever.

First Attempt

On the other hand, the extra wrapping does sometimes get in the way, but that’s what we’re looking to do here: find some definition of £ which hides the annoying details away. Let’s think types first. We have (€) :: a -> (a -> b) -> b but now want a version that weaves Maybe in there as well.

Intuitively, we want Maybe around both arguments and around the result. What happens if we try (£) :: Maybe a -> Maybe (a -> b) -> Maybe b? And when both arguments are Just, then go ahead and apply the function (line 1). Otherwise, for any other combination of inputs, return Nothing (line 2).

| (£) :: Maybe a -> Maybe (a -> b) -> Maybe b
| Just x £ Just f = Just (f x) -- line 1
| _ £ _ = Nothing -- line 2

We can test with a few expressions, e.g. Just "foo" £ Just reverse gives Just "oof", whereas Nothing £ Just reverse and Just "foo" £ Nothing give Nothing.

It’s encoding the idea that the calculation should only proceed when both sides are OK, but is it what we want? Unfortunately, almost but not quite, because it doesn’t fit the pattern we were aiming for from above (do_step_1 £ \res_1 -> do_step_2 res_1 ...) because the lambda is floating outside the second step and not wrapped inside it—which means it won’t be so easy to pass values down this pipeline.

So what have we invented? Briefly, it’s something between a functor (which represents mapping) and a monad, and is useful enough to have a name and a standard library: it’s an “Applicative Functor.”

Recall that functors signify the general idea of mapping over a container data type (e.g. mapping on lists, mapping on trees). For Maybe it means the following:

| fmap_maybe :: (a -> b) -> Maybe a -> Maybe b
| fmap_maybe f Nothing = Nothing
| fmap_maybe f (Just x) = Just (f x)

So, fmap_maybe (\x -> x + 1) Nothing gives Nothing, and fmap_maybe (\x -> x + 1) (Just 10) gives Just 11. It’s a useful operation to do something to the value if one is there, else to leave the box empty.

Applicative functors support slightly more functionality by bridging or chaining between two values. Notice how the definition of £ differs from fmap_maybe—and in particular, notice that fmap_maybe f v is equal to v £ Just f, i.e. you can define mapping via the slightly stronger concept, but can’t go the other way. (This isn’t just theory—it’s an illustration of how one concept is stronger than the other and so can be used for more things.) A similar relationship exists between applicative functors and monads.

Functors, Overloading, and Constructor Classes

Some technical details of Haskell now, which make concepts like functors and monads more convenient in use. Let’s talk overloading. Haskell’s type class system allows overloading of names based on the types in play, and this is used to avoid an explosion of names like fmap_maybe, fmap_either, ... for things that are inherently the same concept and have the same pattern in their types.

Mapping is a key notion for all container-like types, where a function of type a -> b can be applied to some structure of type f a to yield f b, retaining the shape of the input data but applying the function to all values it contains. In concrete terms, fmap show (Just True) gives Just "True", or fmap show [1,2,3] gives ["1", "2", "3"]. The following code, already included in the Prelude, first introduces the functor type class as an interface with one member, then declares how mapping works on lists and Maybe values.

| class Functor f where
| fmap :: (a -> b) -> f a -> f b
| instance Functor [] where
| fmap = map
| instance Functor Maybe where
| fmap f Nothing = Nothing
| fmap f (Just x) = Just (f x)

Notice the switch from overloading on actual types (like Int) to overloading on type constructors like Maybe, which are not types themselves but can be used with actual types to construct new types, e.g. Maybe Int or even Maybe (Maybe Int). The general principles for these “constructor classes” are very similar to those for type classes, and allow fmap to be used as freely as other overloaded functions like show, and so on.

Behind the scenes, the overloading mechanism uses the types to select which of the various definitions of fmap to use. For example, fmap (fmap show) [Just 1, Nothing, Just 2] works on a list of Maybe Int values, and selects the list version for the outer map and the Maybe version for the inner map, thus gives [Just "1", Nothing, Just "2"].

Applicative functors are also captured as a type class. Our £ operator becomes the overloaded operator <*> (note the change of argument order), and each suitable container type can have its own custom implementation of it. The class interface also contains pure :: a -> f a, which represents the conversion of a simple value into the “wrapped” type, e.g. pure for Maybe means wrapping the value with Just, and pure is the way we get simple values into the pipeline.

| -- from the Control.Applicative standard library
| -- use "import Control.Applicative" to include it
| class Functor f => Applicative f where
| pure :: a -> f a
| (<*>) :: f (a -> b) -> f a -> f b
| instance Applicative Maybe where
| pure x = Just x
| Just f <*> mx = fmap f mx
| Nothing <*> _ = Nothing

A simple example now: a computation that requires two table lookups before it can combine the result. We want to add the values if both are OK, else return Nothing (because one part of the operation failed). Using pure to put the add function in the chain, and assuming letters = [('a', 10), ('b', 20)], we can write:

pure (\x y -> x+y) <*> lookup 'a' letters <*> lookup 'b' letters

and get Just 30 as a result. Try changing one of the keys to something else and see what happens. To summarize, we wanted to hide the details of error handling, and the above is fairly successful at it—though the passing of values down the pipeline is still missing.

One technical point: the type checker will accept a definition for an instance as long as the types match, but it’s recommended to ensure that your definition has some other properties or “laws.” The “library docs” explain the laws expected. These ensure that <*> mirrors the associativity property of function composition and that pure works with <*> as a kind of identity element. Recall that associativity means f . (g . h) is the same as (f . g) . h, i.e. that it doesn’t matter which way you do the computation. Do these laws matter? Well, the monad police will not come knocking, not yet anyway, but checking the laws hold will give you confidence that the code will behave as you expect.

(Compare: would you freak out if (1 + 2) + 3 turned out to be different from 1 + (2 + 3)? Or if 1 == 2 differed from 2 == 1?)

A Monad for Maybe

We tried (£) :: Maybe a -> Maybe (a -> b) -> Maybe b, so let’s see what happens when the second Maybe gets pushed inside the function, to give this type:

(££) :: Maybe a -> (a -> Maybe b) -> Maybe b

Guided by the type, we can start filling in the code. Line 1 is the case where the first argument is Nothing, and if we don’t have a value to use then we have Nothing to return. Line 2 is the other case, where we unpack an x and then feed it to the second argument—and this second argument produces the Maybe b we want.

| Nothing ££ _ = Nothing -- line 1
| Just x ££ k = k x -- line 2

Again, compare this definition against that for the applicative functor and functor. The change is small but what it adds is significant: the second argument can now use the result of the first Maybe to decide what to do. This is a new dependency of the second argument on the first that we’ve not seen so far, and it enables the passing of values down the pipeline.

Consider how it works in the context of our first example. I’ve changed the layout to emphasize the imperative aspects, but it’s still the same code. The first line can be read as “perform step 1 and get the result under name res_1.” If step 1 returns Nothing, then the computation stops and returns Nothing overall. Otherwise, the result is fed into the second argument function which makes the result available as res_1 and the same idea applies again: all stop if step 2 fails, else continue with the result as res_2. Finally, the last step can use res_1 and res_2 to compute the final result.

| do_step_1 ££ \res_1 ->
| do_step_2 res_1 ££ \res_2 ->
| use_vals (res_1,res_2)

Note that use_vals will return a Maybe value, but we don’t care here whether it returns Nothing or not—it’s not relevant to the code above. The caller of this code can decode it if it needs to.

So we’ve reached the goal and found an operator for glueing together two Maybe values so that the error handling is hidden, and we can pass values from one stage to successive stages. This plumbing, effectively, is a monad. Wasn’t so bad, was it?

You probably realize that this kind of plumbing can apply to other types too. Haskell overloads the plumbing via another constructor class. Here are the key details:

| -- from the Prelude (via library Control.Monad)
| class Monad m where
| return :: a -> m a
| (>>=) :: m a -> (a -> m b) -> m b
| instance Monad Maybe where
| return = Just
| Nothing >>= _ = Nothing
| Just x >>= k = k x

Like for pure, the overloaded return function just allows a value to be turned into a stage in the pipeline, particularly in a way that allows the following stage to retrieve it. There are also a few laws to check, basically corresponding to a notion of associativity and two checks that return behaves like an identity element. So, a monad is anything for which you can invent a suitable definition of >>= and return! However, some monads are more useful than others...

A More Convenient Notation

Haskell has some syntactic sugar for writing down pipelines based on monads, called “do notation.” Instead of combinations of explicit lambdas and >>= operators, we can write this:

| do res_1 <- do_step_1
| res_2 <- do_step_2 res_1
| use_vals (res_1,res_2)

do is a keyword, and must be followed by a block of statements. Each statement is either a call to some (monadic) operation (aka “action”) or has form pattern <- action, which represents running the action and capturing its output by matching against the pattern. The simplest pattern is a variable name, as used above to capture the first and second results.

This “do” notation is nothing magical—it is just a shorthand that is translated into some combination of >>= and lambdas early in compilation. But, I hope you agree, it does make the code easier to read and follow. It’s no accident that it looks like imperative code either—we are (for most monads) describing some sequential process anyway, e.g. do step one first, if all OK then do step two, etc.

Being able to use this notation also helps understand how to use monads. Firstly, monads are the things that make this notation work, and secondly, we are using monads in order to program in this style—so once you know what kind of operations you want to do (and so what kinds of type will be involved), then you can start writing more abstract code in do notation and rely on monads to handle the plumbing.

Programming with the Interfaces

One of the many advantages of programming to interfaces is that you can write code that works with any instance of the interface. One important operation for monads is sequence :: Monad m => [m a] -> m [a], which is used to convert a list of monadic values to a list of values, by chaining them together (with >>=) and collecting a list of all of the results generated.

| sequence [] = return []
| sequence (m:ms) = do x <- m
| xs <- sequence ms
| return (x:xs)

You could write it as a fold (try it?) but the above version is also a good example of the programming style with monads. The first line says, if no actions to perform then return [] (i.e. so that the caller can retrieve the [] value). Otherwise, perform the first action m to get a value x, then call sequence recursively to process the remaining actions to give the values xs, and finally we return the combined list of values.

Here’s an example:

sequence $ map (\k -&gt; lookup k letters) ['a','a','b','a']

will try to look up all of the characters in the list, and return a list of results Just [10,10,20,10] because all of the lookups succeeded. However,

sequence $ map (\k -&gt; lookup k letters) ['a','c','b','a']

will return Nothing because the monad for Maybe bombs out at the first fail. Compare the result to map (\k -> lookup k letters) ['a','c','b','a']: the key difference is that sequence is computing the overall result from the lookups in sequence, and the monad definition dictates a fail when any of the stages fails.

The libraries contain other imperative-style control structures too, like while loops, and it’s easy enough to add our own if nothing fits. Note also how the monadic code fits alongside other bits of Haskell, i.e. we could use map with a monadic action over a list of values, to get a list of monadic values, then use sequence to compute the overall result. In other words, the monadic values are just like any other piece of data in Haskell, and no extension to the language is needed.

A Quick Look at State

Haskell has no notion of mutable state (i.e. variables that you can assign new values to), but some algorithms make essential use of updateable state. You probably realize that many algorithms that look like they need state can actually be rephrased as simpler data manipulations, such as how breadth-first search was done last month, but there are some algorithms and cases where this doesn’t work well, such as more complex search on graphs where you need an explicit representation of the graph explored so far in order not to duplicate work.

What to do?

The simplest approach mirrors how thread-safe libraries work: we carry a thread’s state value around explicitly and safe operations will work on that state value rather than some global state—hence the state must be one of the parameters. In FP terms, we can use the following function type for state-manipulating entities:

type StateChg s a = s -> (s,a)

That is, a state-changing operation can be represented as a function that takes a state and returns a (possibly new) state plus some other “visible” non-state value. Here are some simple examples of state operations in this style.

| add_num :: Int -> StateChg Int ()
| add_num val = \total -> (total + val, ())

| fetch_state :: StateChg s s
| fetch_state = \s -> (s,s)

| zero_state :: StateChg Int ()
| zero_state = \total -> (0, ())

add_num adds a value to whatever state value is passed in, so it could be used to add a number to a running total. Alongside the new state value, the empty tuple or dummy value () is returned as the “visible” value—it’s a common Haskell idiom to indicate lack of an interesting value, kind of like void in C, and reinforces the idea that the operation is being done for its effect on state (i.e. side-effect) rather than to compute a value.

fetch_state provides a way to consult the current state, hence it returns the incoming state value as the “visible” value, as well as passing on the unchanged state. Notice that it’s totally polymorphic—it doesn’t depend on the type of the state.

zero_state provides a way to reset the (Int) state back to zero, and does so by ignoring the incoming state and passing on 0 instead. The visible value is () in keeping with the “just for side-effects” nature of this operation.

Now, it’s going to be boring unless we can chain a few state manipulations together in a pipeline. We’d like to write code like the following, on the left-hand side. The more familiar conventional mutable-variable version appears alongside as comments.

| do zero_state -- x = 0
| add_num 3 -- x += 3
| t <- fetch_state -- t = x
| add_num (t * 2) -- x = x + t * 2

Notice how the monadic do-notation is giving us a DSL for state-manipulating computations, and it’s not too awkward to use.

We can also define more flexible operators, e.g. change_state :: (s -> s) -> StateChg s () to apply an arbitrary change to the state value, hence replace add_num 3 with change_state (+3). Multiple “variables” are possible too, e.g. replacing the single state value with an “environment” (mapping names to values), and adjusting the state operations to fetch and change entries in the environment according to names.

In fact, we can introduce whatever abstractions we want, anything that will help to make the code clearer.

Let’s consider how the monadic >>= operator works here, i.e. the glue or plumbing that is used to combine two smaller stages together in the pipeline to create a larger stage. Each stage takes a state and produces a new state plus a value, so part of the work is to thread the state from the first stage to the second. We also have the second stage depending on the output of the first (which allows passing values down the pipeline). The following is a sketch of the code (a few technical details, like newtype and class instances have been left out).

(>>=) :: StateChg s a -> (a -> StateChg s b) -> StateChg s b

| step1 >>= step2_fn = \s_in -> let (s_mid, res1) = step1 s_in
| step2 = step2_fn res1
| in step2 s_mid

The “monad” part in the type signature is StateChg s, i.e. a StateChg with the same (polymorphic) state type throughout. We don’t want the state type to change during the pipeline! Otherwise, the signature has the usual m a -> (a -> m b) -> m b pattern. The result of >>= must be another state transformation, so we use \s_in -> ... to capture the expected incoming state and then compute the result from this state. The let x = y in z notation allows us to give local names to intermediate results, particularly to show the plumbing in full detail. We get the intermediate state s_mid and first value res1 by passing the incoming state to the first step. We get step2 from the second argument step2_fn by passing it res1 (remember: the second argument isn’t a state change itself, but a function that determines the actual state change when applied to res1). Finally, the overall result is whatever comes out of step2 when it is run on the intermediate state s_mid.

I was careful to avoid presenting this in imperative terms, i.e. “we get X from Y and ...” rather than “do Y to get X then ...,” because Haskell doesn’t necessarily evaluate them in that sequence. I should have made this clearer earlier: Haskell does not prescribe a particular order, like Java does. (Incidentally, do you know what the evaluation order is for function arguments in C? If you don’t, you might be surprised.)

Instead, Haskell’s evaluation mechanism is guided by dependency: at each stage it only does enough work to decide which clause of a function to use, e.g. to select the empty list case vs the non-empty list case, and no more—so it need not touch arguments that are irrelevant to the immediate decision. Apart from that, the compiler is free to choose which order to do work. For example, in (a + b) + (c + d) which operand is computed first? The language doesn’t fix this, and the lack of side effects also means that it doesn’t actually matter (and the compiler doesn’t have to attempt to infer this). Indeed, the compiler can even do both in parallel!

With monads, however, we are allowing sequential behavior and side effects in a very controlled way, and the pattern of dependencies inside >>= gives the expected sequential progression. This gives us precise control over ordering of state changes so that they do happen in the right order, e.g. updating a variable before reading its value.

You might be concerned that this extra structure makes the code slow. This is partly true, since there is more work to do, but do you remember the silver rule from the first article? “Trust your compiler!” Haskell compilers (GHC especially) are able to inline and transform out some of the overheads of monadic code. So, don’t worry about it until it’s shown to be a bottleneck.

IO Is a Kind of State Manipulation

Space is getting short, so I need to be brief now. Plus, enough has been written elsewhere on the topic! I particularly recommend “Tackling the Awkward Squad” by Simon Peyton Jones as a starting point.

The key points are:

IO is a key case where we need precise control of the order of events.

The monadic concepts and framework provide exactly the kind of DSL we want for IO.

The IO type is usually defined as a compiler primitive, but internally it is basically a state manipulation, like type IO a = RealWorld -> (RealWorld, a).

IO actions are effectively changes in the outside world, and the monad structure ensures the changes to the RealWorld occur in the right order. Plus, the compiler can optimize this state transformation away to leave a sequence of calls to underlying OS libraries.

Haskell libraries provide various IO operations like putStr :: String -> IO () and getLine :: IO String that can be chained together using do notation etc. putStr prints a string, and getLine reads a line of input, e.g. do { i <- getLine; putStr ("Line is:" ++ show i) } to read a line and then say something about it.

The libraries also contain full support for file and network IO, exceptions, pointer-like variables....

It’s also a good basis for concurrency and threads.

One subtle point that catches some people out regarding how monadic values work: What is this line of code going to do?

| [ putStr "hello", putStr "world",
| do {system "echo bang"; return ()}

Answer: nothing! It is just a list of data, where each element is some IO action, i.e. with type [IO ()].

Such actions can be passed around like normal values, because that’s exactly what they are—just pieces of data. The actual effects won’t actually occur until such data is threaded into the sequence of actions being run by the top level of a program. Having such actions as data also leads to nice ways of using callbacks, listeners, etc. For example, GHCI’s REPL takes some input, parses it, type-checks it, then attempts to convert the value to a string and shows it on screen with putStr. However, when it detects a value of type IO a, it runs the action instead of printing something. Can we convert the list of actions to a single action? Remember sequence? The following causes the list of actions to be run, and to be run in the right order too.

| sequence [ putStr "hello", putStr "world",
| do {system "echo bang"; return ()}

More Monads, and Guidelines

We’ve seen some simple monads, but potentially any type constructor can be a monad if you can find definitions of return and (>>=) that meet the requirements. One interesting group comes from combinations of simpler monads, which results in monads that combine the behaviors of their components. Their plumbing is a mixture of the simpler bits of plumbing. The blogs are full of interesting (and some scary) combinations: see what you can find. Many of these have serious practical uses too; for example, several parser libraries use a combination of state, multiple values (“non-determinism”) to present a simpler algorithm.

However, don’t go mad.

DON’T MONADIFY ALL THE THINGS!!!

A good guideline: only use monads where they are essential, such as the key parts of state-based algorithms or the outer IO-bound levels of a program.

You could write all of your programs in a monadic style, but this would get tedious quite quickly, obscure key details about data transformations, and force execution of your code in a particular order (and limit what the compiler can do with it). Haskellers often refer to monadic code as a “sin bin”—tricky code is put there so we can keep an eye on it, not to reward it.

With a bit of thought about the data involved, you can often re-cast some imperative-looking operation to a simpler functional transform, maybe with a bit of monad on the outside. For example, if you’re wanting to process some numbers in a file but wanting to catch invalid entries, you could write monadic code that reaches down through handling of lines and cells on lines to each individual number, but this could be overkill. It could be simpler to read the file as a string, split it into tokens, then map some operation that parses the numbers to return a Maybe Int result, etc., finally having some outer operation that uses sequence on the Maybe values to determine if the whole sequence is valid. (For detailed error handling, you could switch to the similar Either type.)

Also consider whether the intermediate option of applicative functors gives you enough functionality.

Dr Paul Callaghan has temporarily run out of motorbike metaphors. He hopes that there’s enough of the wider ideas in here to help you use monads appropriately in your code. Bits of his bio can be seen on earlier articles. Paul also flies big traction kites and can often be seen being dragged around inelegantly on the beaches of North-east England, much to the amusement of his kids. He blogs at free-variable.org and tweets as @paulcc_two.

Send the author your feedback or discuss the article in the magazine forum.

The Cloud Saves Money

But How Does You Know That’s True?

by Jesse Anderson

You know how to manage the move to The Cloud, but what if you’re asked to cost-justify it? Could you?

Ask virtually anyone about The Cloud and they will tell you what marketing has driven into their heads. The Cloud saves money. Well, it’s true, or it can be. But how does it save you money?

The Old Guard

When you start pitching a move to The Cloud, you may find that there are some members of the Old Guard who haven’t heard that marketing message. Maybe they haven’t even heard about The Cloud, or at least they haven’t heard the pitch that it saves them money. Worse yet, they might want you to prove that you can actually save money by moving your infrastructure.

Grumble, grumble.

For the most part the marketing engine has done a great job of drumming savings into people’s psyches, but hasn’t put as much effort into showing how this savings from moving to The Cloud is brought about.

So it may fall upon you to explain it.

No worries. It’s not a hard case to make. You just need to compare current costs with projected costs due to moving to The Cloud, and one way to do this is by calculating the cost per unit. This allows you to make a relatively objective comparison between your current infrastructure costs and the estimated Cloud costs.

Performance

Your first step is to figure out your current performance for a common unit of work that your software is doing. Sometimes your software may be doing several tasks at once. If this is the case, choose the most commonly run task or the most performance-intensive task. In my Million Monkeys research, I had an easy choice of unit of work: Everything in the Monkeys code is about character groups.

Once you have chosen your unit of work, you need to get your performance numbers. That is, how many units can be calculated/produced/transformed per hour. You’ll need to get those performance numbers for both your current setup and for the system using your planned Cloud provider.

However you collect these numbers, the process should be easily repeatable. It will be similar to, but not necessarily, a unit test. Make sure that you run any test for a few hours to get a good average.

If you have the time, try to get performance numbers for as many instances as possible. You may not use these instances in the short term, but you can get an idea of how your software will perform as you scale up.

Calculating Costs

The next step is calculating total cost per hour. This can be a big job in itself. Although some Cloud providers offer cost calculators, you still need a decent amount of domain knowledge to come up with a cost estimate. Keep in mind, this a total cost of everything. That includes things like compute time, storage, routing/load balancing, and bandwidth. Anything that you have in your current IT infrastructure (unless you are reengineering things) needs to be in this cost estimate.

This work is guaranteed to make you feel like you are being nickeled and dimed to death. Alas, that is the nature of the Cloud beast.

Certain costs may not lend themselves to an hourly cost estimate. This is where you may have to fudge things. Try to give it your best guess or take an average.

Calculating your current costs may be difficult as well. These need to include time spent by staff dealing with hardware issues, for example. Once again, this may be a case where fudging and best guesses are in order.

Cost Per Unit

You’ve done the hard parts and now it’s time to see the fruits of your labor. You’re ready to calculate your cost per unit.

To be sure that we’re clear on the concept, let’s step outside your domain and look at cost per unit in terms of a different domain, say buying food. If you have a giant box of crackers that costs $3 and a small box that costs $1, which one is a better buy? The answer is that neither is demonstrably better, because you don’t have enough information to make an informed decision. You only know the cost, but you need to know the unit. For crackers, this is usually in weight, like ounces or kilograms.

If you have a box of crackers that weighs 6 pounds and costs $3 and a box of crackers that weighs 1 pound and costs $1, which one is a better buy? OK, now you have the information you need. You take the unit (pounds) divided by the cost. So the big box costs $0.50 (6/3=0.5) per pound and the small box costs $1 per pound (1/1=1). The big box is the better buy, provided you actually consume the entire box.

Your performance and costs for assessing the move to The Cloud are calculated the same way. Take your performance per hour and divide that by the cost per hour. Do this same calculation for every instance that you have performance numbers for. I highly recommend graphing this data, as it helps everyone visualize the comparison. Take a look at my research for how I visualized the data.

Show Your Work

This approach is straightforward, but the point is that it is a very objective way to compare things. Keep in mind that there are lots of other variables in choosing a Cloud provider, or even deciding whether or not to move to The Cloud. Just looking at straight costs may lead you to a bad decision.

As you get into Cloud usage, keep in mind that the fastest or cheapest instance may not have the best cost-per-unit ratio. In my research, I found that the Hi-CPU Medium instance performed the best relative to the cost. If you only get performance numbers for one instance type, you may be missing the instance type that saves you the most money.

Want to impress your bosses? Have these numbers and charts ready from the beginning, or at least early on in the process. You may not want to show every detail, but a chart showing cost-per-unit comparisons might get you some kudos.

The costs per unit calculations take some work. You have to compile performance numbers and costs, but it’s really worth the effort once you can back up “The Cloud saves money” with actual numbers.

Jesse Anderson is a Creative Engineer in Reno with many years of experience in creating products and helping companies improve their software engineering. He works at Cloudera on the Educational Services team as a Curriculum Developer and Instructor. He does both professional and personal projects. Personal projects like the Million Monkeys project went viral and gained international notoriety. His interviews appeared in such prestigious places as the Wall Street Journal and Fox News. To help the local community, he volunteers his time as the President of the Northern Nevada Software Developers Group and he sits on the Technology Advisory Committee at Morrison University. His blog and website is www.jesse-anderson.com.

Send the author your feedback or discuss the article in the magazine forum.

The JavaOne Snooze

In the Age of Oracle

by Brian Tarbox

Brian returns to the big Java conference and finds it changed.

Having attended JavaOne a number of times in the pre-Oracle era (winning RockStar and Duke's Choice awards along the way) I was curious to see what the conference had become. My paper on “Advanced Beginner Scala” had been accepted, so the trip would be at relatively lower cost, or so I thought.

In the Age of Oracle

One impact of combining JavaOne with Oracle Open World (OOW) is that instead of 15,000 attendees there were now closer to 60,000 (though only about 2,000 of them were for JavaOne). This meant that hotels were obscenely expensive. My Holiday Inn in the Tenderloin (a sketchy part of town) was $300 a night!

Knowing the conference had shrunk from its glory days was one thing; experiencing it was another. The opening Keynote talk was held in a room with a capacity of about 1,000 and it wasn’t close to full. A far cry from the days of filling Moscone to overflowing.

Another thing that was just plain weird was that they didn’t provide coffee in the morning until after the first set of talks each day. That may seem picky but this is a Java conference, right?

There seemed to be two distinct but unofficial tracks at the conference: traditional Java and non-Java JVM languages. Of the 400 or so talks, fully 40 of them included Scala in the title, and during one of the Keynote panel discussions several panelists talked about using alternate languages. It was interesting to see “non-Java” morphing into “Scala, Groovy and others.” Haskell, Clojure, and JRuby were mentioned, but only as “other choices.”

In the traditional Java track the main emphasis was on the upcoming (as in, sometime next year) release of Java 8. This had a feel of desperation to it that reminded me of the Samsung advertisements mocking Apple fans waiting in line for the next phone. “This time we’re going to get everything we didn’t get last time,” was the message. “And if not this time, then next time for sure!” So Java will finally get closures but not a module system. The module system, code named JigSaw, has been deferred until Java 9, sometime in 2015.

Along with the hype for Java 8 was the assertion that “almost everyone has upgraded to Java 7 so going to 8 will happen quickly.” This of course is pure spin, as quickly became evident if it wasn't already, when several presenters polled the audience for the Java version they were on. The results were consistent: most people were on 6, some on 5, some on 7 and then some still on 4. So, your average Java programmers working in a moderately conservative company (which might be a redundant statement) likely won’t have access to Java 8 for two to three more years at best.

Tired

Many of the Java talks just seemed tired, like the presenters couldn’t dredge up the enthusiasm. I attended one talk on techniques for writing bug-free code that spent the first 30 minutes of the hour convincing us that finding bugs early in the waterfall cycle was less expensive than finding them in the field. As I tweeted upon leaving: a great insight from about 1985. In many of the talks the presenters, upon advancing to each new slide would say (and I’m not making this up) “So, ah, what does this slide say?” They’d then read the slide, figure it out and then explain it to us. Really?

A typical view of the screen in the hotel conference rooms

A clue that the organizers were either bored or expected the attendees to be bored was the absence of survey sheets for the talks. In past years you couldn’t leave a session without being handed a survey card you were supposed to fill in to rate the talk. In my experience, past conference organizers and speakers took these very seriously because high ratings could lead to “Rock Star” status... and that’s the only way I’ll ever get to be called a rock star! This year the surveys were missing... unless you happened to stumble upon the “survey” section of the Schedule Builder. When I asked several conference organizers about this, they replied “Oh yeah... we should probably tell people about that.”

A strange confirmation of this “tired” assessment came from a conversation with someone from Oracle’s Java Magazine. The cover story was on James Gosling’s latest project: semi-autonomous solar powered surfboards. These highly instrumented drones wander the oceans collecting data which is then sent to central servers. Very cool technology. Java-based of course. The title on the magazine cover was “Java At Sea.” When I asked the person if they were aware that the title could be interpreted in different ways, they just smiled.

The Good Stuff

But it wasn’t all bad. On the positive side, there were a number of very good talks on the care and tuning of the Garbage Collector, and there were deep dives into the byte codes generated for the new language features such as closures.

Venkat Subramaniam was there and his talks were, as always, interesting and entertaining. His talk on Java Generics was outstanding... especially as I thought I already had a quite thorough understanding of the topic. (Turns out I still had a lot to learn.)

As for my contribution, I gave a talk on my experience rewriting my open source music project log4jfugue from Java into Scala. The project shrank from 2,500 loc to 250 loc. I emphasized the fact that I’m at best an advanced beginner in Scala, and that you can do quite a lot using just the simplest subset of this powerful language’s features. Did my talk come in on the boring or interesting side of the ledger? Well, the talk seemed to be well received... at least no one walked out during it! I’d tell you what I got for survey results, but....

In summary, JavaOne seems over, and Java is well on its way to becoming the language you code in when you don’t have a choice.

Late-Breaking News

Brian sent the following update as we were going to press:

I actually did get my survey results back. I had 98 people attend the talk and four people found the survey URL buried on the Oracle web site. I got a 4.5 out of 5 but with only four responses I can't be too excited.

Brian Tarbox is a Distinguished Member of Technical Staff at Motorola in the Systems Engineering group, designing next generation video products. He writes a blog on the intersection of software design, cognition, music, and creativity at briantarbox.blogspot.com. His primary contact point is about.me/BrianTarbox .

Send the author your feedback or discuss the article in the magazine forum.

Threads

Gary Kildall

by Michael Swaine

Gary Kildall was a programmer’s programmer.

Paul Freiberger and I are currently writing the third edition of Fire in the Valley, our history of the personal computer—to be published by The Pragmatic Bookshelf, naturally. As we go back over the material, we keep finding threads that run through the whole narrative. I am sharing some of these as brief history articles here in PragPub.

Gary Kildall’s story weaves in and out of the early history of the personal computer, and the color of the thread that runs through it all is that of a programmer enjoying the challenge of solving tough problems.

Sailors and Spies

As the curtain goes up, we find Gary Kildall already on stage, a respected computer science professor at the Naval Postgraduate School in Monterey, California. He’s a born teacher, happiest when he has a piece of chalk or a pencil in his hand. And he’s really into these new semiconductors he gets before anybody else from his contacts at Intel. There’s a cutting-edge microcomputer lab in the back of his classroom. His students are in-service military personnel pursuing graduate degrees in operations and technical areas, or future spies. The CIA is the funding agency for many of the scholarships on campus. It’s 1972, but there aren’t a lot of anti-war protests going on at NPS. It’s a pretty serious, straight-arrow place, and Gary’s students are motivated.

Apparently teaching computer science to sailors and spies is not enough of a challenge, because Gary is also consulting with Intel, where some of the geekier engineers are rather loosely interpreting their contract to design a controller chip for a calculator. They’ve decided to view it as permission to develop a computer on a chip. Gary writes key software for the chips, the 4004 and 8008. In a clear case of mission creep, he writes a full-blown HLL for the infant processors, an implementation of PL/I that he called PL/M.

(Later, when we talked to Gary’s friend and colleague Alan Cooper about these days, Alan told us that Gary wrote a PL/I implementation precisely because it was hard. The challenge was where the fun was.)

Entrepreneur

As time goes on, Intel’s engineers see enormous potential in the new microprocessors they are developing. They are certain that successful businesses will be built on the back of their innovations. Wristwatches, Bob Noyce thinks. Or microprocessor-controlled microwave ovens, or blenders, or carburetors or stereo sets. Small computers, meh. Not much of a market there, Intel’s boss thinks. But some of these kinds of plays would pay off. There are discussions within Intel about getting a piece of that action, but ultimately the company sticks to what it knows: semiconductors. Mostly memory.

Intel’s marketing manager, Mike Markkula, starts looking for opportunities. So does Intel documentation specialist Adam Osborne. Gary gets the fever, too. He and UC Berkeley computer science professor John Torode build a computer with a disk operating system and sell a few, but this is really just a hobby operation. Gary’s idea of building a business isn’t quite what you’d expect from a scientifically-minded professor teaching sailors and spies. He decides to build an Astrology Machine.

With San Francisco hardware designer constructing the actual hardware and Gary doing the software, they release their machines on the Bay Area. Gary’s sense of integrity demands that he compute the star positions accurately, even though he considers astrology a joke. The machines look like arcade games of the period, and a number of them get placed in supermarkets around the area. Unfortunately, the printer that delivers the horoscope jams a lot and the business is a failure.

Digital Research

Out of his work at Intel, though, Gary has crafted a just-right-size operating system for the new microcomputers being built from the chips, and the Astrology Machine gives him a chance to field-test it. He calls it CP/M. Designed to work with Intel’s existing and future microprocessors, CP/M isolates the machine-specific code into a component he called the BIOS. This makes it easily portable across Intel’s chips, but also across different vendors’ microcomputers. Gary asks the Intel executives if they have any objections to his marketing CP/M on his own. They shrug and say go ahead.

It would be an exaggeration to call this collection of hobbyist entrepreneurs and the motley products they are offering assembled or in kit form through mail order at this frenzied time an industry. But CP/M is just what it takes to turn into one. Breakthroughs in technology industries aren’t always technological breakthroughs. This time, though, that’s what’s was needed: the solution to a tough technical problem. That’s what CP/M delivers. Everybody wants it.

This time when Gary starts a business it is a little more conventional, and it is an immediate success. You can tell he’s going to have fun doing it, though: the company is officially named Intergalactic Digital Research. Its core product is CP/M.

The breakout microcomputer in 1974 is the MITS Altair. Its announcement on the cover of Popular Electronics in January inspires a generation of hobbyist entrepreneurs. One of them drops out of Harvard to start a company he calls Micro-Soft. Other microcomputer companies come on line, notably IMSAI. It is IMSAI that puts Digital Research in the black with a $25,000 contract. For the next seven years, Digital Research owns the personal computer operating system market.

Then IBM decides to produce a personal computer.

When Gary Went Flying

The story of when IBM came calling and Gary went flying has been told many times, and with varying details. It is the stuff of personal computer industry myth and legend.

But the key facts of the story are not in doubt. They are that Bill Gates gets his guys in the room with IBM and seals the deal—even though he doesn’t have a product to sell them—while Gary does not pursue IBM and fails to get the deal and as a result loses the personal computer operating system market that he created.

That isn’t the end of the story, but it marks the point beyond which Gary is done with the business. He takes time off, writes a version of the language LOGO as a present for his son. Of course he does such a good job that it’s a marketable product, and Digital Research sells it. But Gary doesn’t need that. He phases out of the business. It just isn’t fun any more.

He goes back to inventing. He does groundbreaking work on CD-ROM software and on interfacing computers and videodiscs. A company, KnowledgeSet, is spun off from that work. It enables the CD-ROM-based Grolier’s Encyclopedia, which shows everyone how to do CD-ROM content right.

Legacy

Alan Cooper once pointed out a good example of how Gary looked at his work. In MS-DOS, to copy a file from drive A to drive B, you would write: copy A: B:. In CP/M, to do the same thing, you would write: pip B: A:.

When users complained that the pip command was confusing, Gary was dismissive. Gary thought that any halfway intelligent person could learn that PIP was the command you used to copy, and that you copied (or “pipped”) from right to left, not left to right. No matter that this was arbitrary, confusing, and counterintuitive, or that customers complained about it.

Bill Gates listened to his customers. “That difference in attitude,” Cooper said, “is worth twenty million dollars.”

So Gary wasn’t Bill Gates. But, as we said in the second edition of Fire in the Valley, he left a remarkable legacy. He created the first microprocessor disk operating system, which eventually sold a quarter million copies. He defined the first programming language and wrote the first compiler specifically for microprocessors. He developed the file system and data structures for the first consumer CD-ROM. He created the first computer interface for videodiscs that allowed automatic nonlinear playback, presaging today's interactive multimedia. He created the first successful open-system architecture by segregating system-specific hardware interfaces in a set of BIOS routines, an innovation that made the whole third-party software industry possible.

Finally, he personified an approach to work that many of us find admirable: The belief that science and technology advance through the open sharing of discoveries, and that it’s more important to create the next invention than to protect the last one.

Calendar

Author sightings, partner events, and other notable happenings.

Here’s an update on what our authors and friends are up to for the next three months.

One of the interesting talks coming up is Michael Nygard’s “Exploiting Loopholes in CAP” at QCon San Francisco. Andy Lester suggested that Michael explain CAP for those of us not in the know, and Michael supplied the following explanation:

“CAP is ‘Consistency, Availability, and Partition-Tolerance.’ We say CAP as shorthand for Brewer’s Conjecture, which says that a distributed system can achieve at most two of those three qualities. CAP was proved in a 2002 paper by Gilbert and Lynch. This theorem spawned the entire crop of eventually-consistent databases. (Mongo, Cassandra, Couch, Riak, etc.) Like any proof, however, the Gilbert & Lynch proof relies on some strict definitions and assumptions. In my talk, I’ll be showing how to ‘dodge’ CAP by working with different definitions of ‘consistent’ and ‘available,’ or by otherwise changing the assumptions that underlie the proof.”

Nov 6–9, Pragmatic Studio, Denver, CO

“iOS Programming”

Daniel H Steinberg

Nov 6–9, Pragmatic Studio, Denver, CO

“iOS Programming”

Bill Dudney, author of Core Animation

Nov 7, Øredev, Malmo, Sweden

“Vim—precision editing at the speed of thought”

Drew Neil, author of Practical Vim

Nov 8, QCon San Francisco, San Francisco, CA

“Exploiting Loopholes in CAP”

Michael Nygard, author of Release It!

Nov 8–9, RubyWorld Conference, Matsue, Japan

Keynote

Dave Thomas

Nov 9-11, NFJS, Chicago, IL

Talks on JVM languages, Concurrency, HTML5, JavaScript, and Testing

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Nov 12, Devoxx, Antwerp, Belgium

“Intro to iOS 6 for Java Developers”

Daniel Steinberg

Nov 13, Bath Scrum User Group, Bath, UK

"Talk: Coaching teams through Change

Rachel Davies, co-author of Agile Coaching

Nov 14, .NET Rocks!, Houston, TX

Keynote

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Nov 16, Clojure/conj, Raleigh, NC

Keynote: “Whence Complexity?”

Michael Nygard, author of Release It!

Nov 16-18, NFJS, Denver, CO

Talks on JVM languages, Concurrency, HTML5, JavaScript, and Testing

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Nov 19–23, Organized by ProgramUtvikling, Oslo, Norway

“Test Driving your .NET Apps” (course)

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Nov 27-30, RichWeb Experience, Fort Lauderdale, FL

Talks and workshops on JavaScript, CoffeeScript, HTML5, Testing

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Nov 28, YOW! Australia, Melbourne, Victoria, Australia

Workshop: “Production Ready Software”

Michael Nygard, author of Release It!

Nov 29–Dec 1, CocoaConf, Raleigh, NC

Keynote, “Less code, more fulfilling,” and “Storyboards—This time it’s personal”

Daniel Steinberg

Nov 29–Dec 1, Cocoa Conf, Raleigh NC

TBD

Bill Dudney, author of Core Animation

Nov 29–Dec 1, CocoaConf, Raleigh, NC

“Core Audio in iOS 6,” “Mobile Movies with HTTP Live Streaming,” and “All-Day Core Audio Workshop”

Chris Adamson, co-author of iOS SDK Development

Dec 4–6, JavaOne Latin America 2012, São Paulo, Brazil

“Regular Expressions in a Nutshell”

Staffan Nöteberg, author of Pomodoro Technique Illustrated

Dec 5, Copyright & Technology, New York, NY

“DRM and E-Publishing Panel”

Jim Dovey, author of iOS/Mac Networking

Dec 7, DevIgnition, Tyson Corners, VA

“Your Brain & Dart”

Chris Strom, Author of Dart for Hipsters and The SPDY Book

Jan 8–11, CodeMash, Sandusky, OH

“Building Productive iOS 6 Apps” and “Objective-C Is Not Java”

Chris Adamson, co-author of iOS SDK Development

O’Reilly Events

Upcoming events from our friends at O’Reilly.

Nov 7, Mini TOC Charleston, Charleston, SC

Mini TOC Charleston is geared to librarians, large & small publishers, self-published authors, creative collectives, information hobbyists, journalists, historians, bloggers, and online experts seeking a better understanding of what ubiquitous content means for the future of commercial publishing.

Dec 4–5, Velocity China Conference, Beijing, China

Most dynamic website companies face the same challenge: pages must load fast, the infrastructure has to be able to expand, sites and services must be reliable, and you have to do it all in a timeframe the team can handle while keeping it within budget. Velocity is the best place for Web professionals to meet to exchange ideas and share best practices and lessons learned.

USENIX Events

What’s coming from our USENIX friends.

Dec 3–7, Middleware 2012, Montreal, Quebec, Canada

ACM/IFIP/USENIX 13th International Conference on Middleware: “a premier forum for the discussion of innovations and recent advances in the design, implementation, experimentation, deployment, and usage of middleware systems.”

Dec 9–14, LISA ’12, San Diego, CA

26th Large Installation System Administration Conference: “The annual LISA conference is the meeting place of choice for system and network administrators and engineers; it is the crossroads of Web operations, DevOps, enterprise computing, educational computing, and research computing.”

Other Happenings

Nov 8

Bill Joy is 58.

Nov 6

Jerry Yang is 44.

Nov 10

Bert Bos, co-creator of Cascading Style Sheets, is 49.

Nov 10

Richard Brodie, creator of Microsoft Word, is 52.

Nov 18

Pamela Kyle Crossley, historian and creator of SIMPLE, is 57.

Nov 19

Dries Buytaert, Drupal founder and lead developer, is 34.

Nov 22

Randall Schwartz is 51.

Nov 22

PHP creator Rasmus Lerdorf is 44.

Nov 22

And it’s Charles Simonyi’s wedding anniversary.

Dec 3

John Backus would have been 87.

Dec 4

Eric Raymond, aka ESR, is 43.

Dec 9

Grace Hopper’s 105th birthday.

Dec 10

Ada Lovelace’s 196th birthday.

Dec 10

Ada, the programming language, is 31 today.

Dec 16

John Bardeen and Walter Brattain created the first transistor on this day in 1947.

Dec 17

Ken Iverson, creator of APL, would have been 91 today.

Dec 17

Les Earnest, inventor of first spell checker and of the finger protocol, is 81.

Jan 10

MUD co-creator Richard Bartle is 53.

Jan 10

Donald Knuth, who continues to inspire authors who are having trouble finishing that book, is 75.

Jan 11

Tony Hoare, author of Quicksort, is 79.

Jan 20

Sims creator Will Wright is 53.

Jan 21

Paul Allen is 60.

Jan 22

On this day in 1984, Superbowl viewers watched a woman with a hammer smash Big Brother.

Jan 24

Alain Colmerauer, Prolog’s creator, is 72.

Jan 25

Pac-Man creator Toru Iwatani is 58.

Jan 31

Guido van Rossum, author of Python and possessor of an excellent last name for a programmer (Google “R.U.R.”), is 57.

And be sure to celebrate December 10th. As you probably know, December 10th is the birthday of the first programmer, Ada Lovelace, colleague of Charles Babbage and daughter of Lord Byron. It’s also, incidentally, the birthday of poet Emily Dickinson, of abolitionist William Lloyd Garrison, and of newscaster Chet Huntley. For Hardy Boys fans, it’s the 71st birthday of both Tim Considine and Tommy Kirk. It’s the anniversary of the end of the Spanish-American War, and shouldn’t we look for any excuse to celebrate the end of a war? It’s a day of auspicious beginnings, too, since it’s the day of the year on which women were first allowed to vote in the United States. December 10th is a particularly romantic day, being the day on which King Edward VIII abdicated the throne of England to marry Mrs. Wallis Simpson. And it’s the day on which the millionth Model T rolled off the Ford production line, which may be sort of romantic for some people. It’s the anniversary of the establishment of the Metric System in France, the day the Nobel Prizes are awarded, and the day on which the UN General Assembly adopted the Universal Declaration of Human Rights. A significant day, surely. And on December 10, 1884, Huckleberry Finn was published, for which we are all richer.

The Quiz

Solving Donald Knuth

I thought it was about time for another head-scratcher. This one’s simple. Just replace each letter with a digit so that the addition works. The puzzle uses all ten digits, which is more than can be said for your humble editor, and the solution is unique.

All will be revealed next month. But I’m sure that by then you’ll have written a program in Erlang that solves all addition problems using the names of famous computer scientists. Or you’ll have devised the Google search expression that finds the answer online. So I’ll also reveal the author of the program that I used to construct this puzzle.

Shady Illuminations

Users

by John Shade

John thinks users are a grinding noise in the gears of progress.

Remember the good old days, when your computer arrived via Railway Express? And you opened the box and it was just a bag of parts? Remember? And after you assembled it and made a few trips to Radio Shack to replace the defective components and got it working, if you wanted to actually do anything with it you had to write the software yourself? In 8008 assembly language? And enter it one bit at a time through toggle switches? Remember those glory days?

OK, maybe you don’t. Take my word for it, computer users back then had grit.

Computer users today are grit. They’re a grinding noise in the gears of progress. They’re a bunch of wimps with a highly developed sense of entitlement. Present company excepted, of course.

I place in evidence as Exhibit 1 the “Facebook, I want my friends back” meme.

Facebook is not letting you see all the posts of all your friends. Not unless they pay for the privilege. Your friends are being held for ransom. It’s a stick-up, it’s robbery, it’s bait and switch.

Well, one of those hysterical claims is right. It is bait and switch. That’s a social media business model being taught in Harvard Business School now. Monetizing Technology Ventures through Bait and Switch, three credits. But just because you’re the carp in the metaphor that doesn’t mean we have to listen to you carp over Facebook charging for what you’ve been getting for free. Facebook went public. This means it has to answer to stockholders. It has to have a business model now.

Oh, and there’s the fact that this is actually a benefit to people with large friends lists. But the part that I like is the sense that Facebook is stealing something from anybody. That sense of entitlement.

Then there’s the “Apple is forcing me to upgrade twice a year” meme.

Right, Apple is ripping you off if it shortens its upgrade cycle. Because you are forced to buy every new model, so they get twice the money. This obviously calls for a revolution. Apple customers have to rebel against the oppressive regime and, and, not buy every new model that comes out. Nice try, Apple, one guy sneers, I’ll see you in two years.

Well, yeah. It’s a bold assault on the castle, but it just might work.

John Shade was born under a cloud in Montreux, Switzerland, in 1962. Subsequent internment in a series of obscure institutions of ostensibly higher learning did nothing to brighten his outlook. He’s not looking for a handout or a hand-up, but he has been told that he could benefit from a handler. Follow John on Twitter, send him your feedback, or discuss the article in the magazine forum.

But Wait, There’s More...

Coming Attractions

This may be the last page, but that doesn’t mean it’s the end of the conversation. Keep up with what’s going on in the Pragmatic universe by subscribing to our newsletter and drop by the Pub again next month for more Pragmatism. We’ll keep a seat warm and a mug chilled and a candle burning in the window.

Coming Soon in PragPub

I can’t tell you what’s in the December issue, because that would spoil the surprise. Plus, I don’t know. What I do know is that an unprecedented number of authors have something in the works. And there are some really cool topics in the list.

Recent Releases, Betas, and Upcoming Books

iOS SDK Development, Outsource It!, Mac Kung Fu, 2nd Edition, Core Data, 2nd Edition, Practical Vim, The Definitive ANTLR 4 Reference, The ThoughtWorks Anthology, Volume 2, and The Pragmatic Programmer: From Journeyman to Master.  

Right. But to really be in the know, you need to subscribe to our weekly newsletter. It’ll keep you in the loop, it’s a fun read, and it’s free. All you need to do is create an account on pragprog.com (email address and password is all it takes) and select the checkbox to receive newsletters.

While you’re waiting for the next issue of the newsletter or of PragPub to come out, you can follow us on twitter at @pragpub, @PragmaticAndy, or @pragdave. Or on Facebook at facebook.com/PragmaticBookshelf. Or email us at michael@pragprog.com.