PragPub 2011-12: Issue #30

Table of Contents

Features

- Extending Java
by Sven Efftinge, Sebastian Zarnekow
Java is the most successful language on the planet—and one of the most derided. Maybe it’s time for a new language with all of Java’s strengths and none of its weaknesses.

- Scala for the Intrigued
by Venkat Subramaniam
In this fourth installment of his series on the Scala programming language, Venkat delves into the functional style of programming in Scala.

- Revisiting Graphics Hardware
by James Bowman
James revisits the world of 8-bit game programming in creating the Gameduino, an 8-bit graphics and sound shield for the Arduino.

- Just-In-Time Logging
by Brian Tarbox
Brian proposes moving logging from the province of the developer’s discretion to the circle of architecture and design.

- When Did That Happen?
by Dan Wohlbruck
Three giants in the history of computing meet for lunch during World War II.

Departments

- Up Front
by Michael Swaine
In this issue we revisit the days of 8-bit game programming and extend Java into the future.

- Choice Bits
Drinking from the Twitter firehose.

- Meet the Team
Meet David Kelly, Pragmatic Bookshelf’s typesetter.

- Calendar
Author sightings, upcoming conferences, and other events of note.

- Author Index
From Chris Adamson to Marcus Zarra, over eighty authors have contributed to our first two and a half years.

- Shady Illuminations
by John Shade
Commerce without the whining.

- But Wait, There’s More...
Coming attractions and where to go from here.

Except where otherwise indicated, entire contents copyright © 2011 The Pragmatic Programmers.

Feel free to distribute this magazine (in whole, and for free) to anyone you want. However, you may not sell this magazine or its content, nor extract and use more than a paragraph of content in some other publication without our permission.

Published monthly in PDF, mobi, and epub formats by The Pragmatic Programmers, LLC, Dallas, TX, and Raleigh, NC. E-Mail support@pragprog.com, phone +1-800-699-7764. The editor is Michael Swaine (mailto:michael@pragprog.com). Visit us at http://pragprog.com

ISSN: 1948-3562

Up Front

A Better Java
by Michael Swaine
In this issue we revisit the days of 8-bit game programming and extend Java into the future.

We’re looking at an article from Jonathan Penn on MacRuby for the January issue. Could MacRuby be the future of iOS development? Not without some commitment from Apple. So Jonathan Penn has decided to do something about that. He’s launched a campaign to get Apple’s attention. We think it’s intriguing. Read about it, think about it, and give Jonathan your feedback. Our selfish interest: your feedback will make his January article even richer.

This is the 30th monthly issue of PragPub. In these two and a half dozen outings, we have been honored to publish the words of more than eighty authors. Partly to recognize our contributors and partly to provide a convenient reference, I put together an index to all the articles published in PragPub, organized by author. I’ve included it at the end of this issue and have also parked it, at least temporarily, here, where I’ll try to keep it up to date.

But you want to know what’s in this issue of PragPub besides an index.

Java is the most successful language on the planet—and one of the most derided. Sven Efftinge and Sebastian Zarnekow decided maybe it was time for a new language with all of Java’s strengths and none of its weaknesses. In this issue they unveil Xtend, a language that extends Java with an emphasis on functional style and a couple of lumps of syntactic sugar.

James Bowman revisits the world of 8-bit game programming in this issue, the days when graphics displays were not directly pixel-addressable, but made you work with 8x8 pixel bitmaps called background tiles. He shows how to exploit the surprising power in this retro approach in creating the Gameduino, an 8-bit graphics and sound shield for the Arduino.

Logging is the poor stepchild of applications, even though for many some apps it is the primary means of diagnosing problems. Brian Tarbox proposes moving logging from the province of the developer’s discretion to the circle of architecture and design to achieve what he calls just-in-time logging.

In the fourth installment of his series on the Scala programming language, Venkat Subramaniam delves into the functional style of programming in Scala.

Dan Wohlbruck tells how three giants in the history of computing met for lunch during World War II and what resulted. Maybe.

Plus Choice Bits, our Events Calendar, a profile of another Pragmatic Bookshelf staffer, and another column by John Shade.

Choice Bits

Someone is Wrong on the Internet

Applying the axiom of choice behind the bar results in a Long Island Iced Tea.

What’s Hot

Top-Ten lists are passé—ours goes to 11. These are the top titles that folks are interested in currently, along with their rank from last month. This is based solely on direct sales from our online store.

1^

NEW

The Developer's Code

2^

NEW

Rails Recipes

3^

NEW

Build Awesome Command-Line Applications in Ruby

4^

NEW

Agile Web Development with Rails

5^

NEW

Mac Kung Fu

6^

NEW

Web Development Recipes

7^

NEW

The Cucumber Book

8^

NEW

New Programmer's Survival Manual

9^

NEW

Programming Ruby 1.9

10^

NEW

CoffeeScript

11^

NEW

Programming Clojure

I tweet this:

There’s one xkcd cartoon that will be relevant as long as there’s an internet. You know the one I mean?

and within seven minutes some total stranger has replied,

@pragpub All those other guesses by people on the internet are *WRONG*, and I will not rest until they are corrected!

I invent a clever phrase and google it. Sure enough, dozens of people have already thought of it.

Somebody out there in the intertubes will get my obscure jokes. Somebody else will have had my insights.

I’m not sure whether to feel comforted or redundant.

Coding Wisdom from Our Agents in the Field

“What I cannot create, I do not understand.”—on Feynman’s blackboard when he died. tinyurl.com/7wy2kjf Good motto for programming. — @gvwilson

When someone’s been in captivity all their life, it can be hard to embrace freedom. So they continue to use static types in Groovy code. — @daveklein

One reason to do hard things while programming is so they won’t be hard next time. Another reason is because they are fun. — @KentBeck

We have so much technical debt on this project that we might need a TARP bailout.... — @scottdavis99

RT @hal_hildebrand: Parallel processors running perpendicular today. — @KentBeck

What’s the name for the moment you open your redesigned web site in Internet Explorer for the first time and want to cry? — @ginatrapani

I’m sorry but “Ice Cream Sandwich” is just a ridiculous code name for software. — @bketelsen

How to Tell CodeWisdom from HackerNewsTips

“Java is to JavaScript what Car is to Carpet.”—Chris Heilmann — @CodeWisdom

Finally. @dhh moves from battling @groupon to fight the real enemy RT @dhh: How did 7-Up lose out to Sprite? It’s endlessly more delicious. — @HackerNewsTips

“XML is like violence—if it doesn’t solve your problems, you are not using enough of it.”—Unknown — @CodeWisdom

Be aware of Java 7 license changes. Larry Ellison can legally commandeer robotics startups using it, in order to build an Iron Man suit. — @HackerNewsTips

“Computer science is no more about computers than astronomy is about telescopes.”—Edsger Dijkstra — @CodeWisdom

We’re excited about @Heroku kicking off their “Language Creator as a Service” offering with their acquisition of Yukihiro Matsumoto. — @HackerNewsTips

“Pasting code from the Internet into production code is like chewing gum found in the street.”—Mike Johnson — @CodeWisdom

California Proposition 215 gives @TechCrunch the right to reveal your child’s sex to your in-laws before you do. — @HackerNewsTips

Two Moves Behind in the Game of Life

So the reason I suck at life. I’m in San Antonio 24 hours prior to the start of the race. The calendar in my head doesn’t reflect reality. — @tswicegood

I may be male, unshaven and in need of caffiene, but yes, I do know more about horses than you. #lifeonthetrain — @adamgoucher

I think I’ve finally struck the right balance between beauty and professionalism on my Twitter profile design. — @chadfowler

The busy are doing it wrong. calnewport.com/blog/2011/11/1… I need to pay attention to this. A lot! — @jaredrichardson

The milk run takes longer, but is a much more relaxed ride. — @adamgoucher

I have solved the Halting Problem: I’ve halted. — @gvwilson

Who Are Those Guys?

This month we galled the kibes of Kent Beck, Scott Davis, Chad Fowler, Adam Goucher, Brian Ketelsen, Dave Klein, Jared Richardson, Travis Swicegood, Gina Trapani, Greg Wilson, Hacker News Tips, and Programming Wisdom. You can follow us at www.twitter.com/pragpub.

Meet the Team

David Kelly

Each month we profile a member of the Pragmatic Bookshelf team. This month it’s David Kelly, our talented typesetter. David joined the team in the usual way: he ran into Andy Hunt at the Central Carolina Songwriter’s Association, where they both occasionally display their writing and performance licks, and the rest is pragmatic history.

David Kelly, Typesetter

David is responsible for the look and feel of the Pragmatic Bookshelf books. He handles the XSL style sheets for the PDF, epub, and mobi versions of the books. His job includes all the fine-tuning adjustments to hyphenation, page breaks, figure sizes, and so forth to make the books look professional and pretty. He’s also the guy who designs many of the covers for our books.

David brings to Pragmatic Bookshelf a background in technical writing, documentation management, and process engineering. After teaching himself DITA, XSL, XSL-FO, and a little Ant, he has worked primarily in document automation and production. And then he ran into Andy.

“Since coming here I’ve learned a lot about the commercial publishing business that I had not been exposed to in the world of corporate technical publishing,” David says. But he’s always been a book person. “I majored in physics only because there was no degree offered in science fiction. I wrote short stories at the age of eight, wrote a novel when I was fifteen, and published two fantasy novels in my mid-twenties. At one point I owned a printing press and used it to produce and publish poetry books for North Carolina poets. That was a labor of love! So I’m very fortunate to be able to do professional work so closely related to one of my long-standing passions.”

Outside of his work with Prags, David is working on a science fiction novel about human evolution and music. He lives with his wife Anne and son Thad in Raleigh, North Carolina, and goes bicycling whenever he can.

Email: david.kelly@pragprog.com

Extending Java

Xtend: a New Language for Java Developers
by Sven Efftinge, Sebastian Zarnekow
With an emphasis on functional style and a couple of lumps of syntactic sugar, Xtend hopes to retain all of Java’s strengths and none of its weaknesses.

Java is the most successful language on the planet. There are close to 10 million developers writing all kinds of software systems in it; the best IDEs on the planet are Java IDEs; and the vast amount of exciting open-source frameworks, libraries, and tools are very good reasons to use it. Also, a lot of the concepts in Java are well thought out. Even more importantly, they are mature and well understood.

But—

Java is also known to be a very inflexible and ceremonial language, forcing the developer to repeat redundant information over and over again. In fact, this is why Java IDEs have to be so powerful: to help us to write all that boilerplate!

But as we all know, code is read much more often than it is written. So this aspect of Java is a real problem. We could try to fix it with IDEs, but that’s just throwing yet another load of tools on the pile. We should fix the problem where it’s caused: we need a better Java.

Java + Xtend == Better Java

Xtend is a statically-typed programming language developed at Eclipse.org. It has a strong focus on leveraging all the good parts of Java, including seamless integration with the huge amount of Java frameworks and libraries out there. Experienced Java developers can get started with Xtend in almost no time, since everything is so familiar. The editor integrates tightly with Eclipse’s Java tooling to ensure a seamless integration in the IDE as well. If you know how the Java tools work, you will feel at home with Xtend’s UI.

The good parts of Java make for a great basis, but Xtend is all about extending that. For instance, you can now simply discard all that boilerplate you are used to generating using your IDE. Reasonable defaults, type inference, and some other powerful features can make code surprisingly readable. Closures and operator overloading are fun things to have as well. With Xtend you can use such beyond-Java features without learning a whole new language.

Xtend is not meant to be a replacement for Java but serves more as an add-on. When working with Xtend you still write the interfaces, enums, and annotations in Java. There’s little value in changing the syntax of these concepts, as they are already very compact, well-known, and tooling-wise nicely supported. The added value with Xtend is the very concise way to implement Java classes. And if you feel like shifting some bits around, you can always go back to Java. The two languages integrate seamlessly in both directions.

Hello World!

So let’s write some Xtend code. The ubiquitous “Hello World” program would look like this:

class HelloWorld {

def static void main(String[] args) {

println(“Hello World!”)

}

}

At a first glance, the most obvious difference from Java is the def keyword, which is used for method declarations. Other than that you might notice that there are no mandatory semicolons and no explicit visibility modifier. In Xtend every class and method is public by default. Fields, on the other hand, are private if you don’t specify the visibility differently. Xtend supports private, protected, and public visibility, sharing the exact same semantics you know from Java. Java’s default visibility (package-private) is not supported.

The println function is automatically imported, similar to members of the package java.lang. The method is defined in a thin runtime library that is part of Xtend. It contains only a dozen classes, adding some convenient functionality on top of the Java SDK.

Getting Rid of the Boilerplate

Let’s make it more interesting by looking at a more realistic example. Let’s say we have a method that should turn a list of strings into their uppercase versions. This is the noisiest version you could write in Xtend:

def public List<String> toUpperCase(List<String> strings) {

val List<String> result = new ArrayList<String>();

for (String string : strings) {

result.add(string.toUpperCase());

}

return result;

}

Besides the def and the val keyword it looks like the Java code most people would write. Let’s now make it a bit more concise by applying type inference and other sugar:

def toUpperCase(List<String> strings) {

val result = new ArrayList<String>()

for (string : strings) {

result += string.toUpperCase

}

return result

}

This is already a bit better. We removed the majority of explicit type information, since the compiler doesn’t need it to understand the code. You can have it if you think it improves readability, but in Xtend you are not forced to write it down.

Many people claim that Java generics are bad because they force you to specify the type arguments on the client side. Type Inference reduces this problem dramatically, as you can see.

You might also have noticed that we’ve replaced the invocation of List.add(T) with the operator +=. In Xtend, operators are just an alternative syntax for method invocations, hence you can easily use any operators on any type if you want to. The += operator for java.util.Collection is defined in the runtime library mentioned earlier. This feature not only allows you to have nice arithmetic operators on numeric types like java.math.BigDecimal, it also comes in handy in many other scenarios. In Xtend you can, for example, create an instance of a HashMap like this :

val map = newHashMap( 1 -> ‘one’, 2 -> ‘two’, 3 -> ‘three’ )

In fact, the -> operator is an extension method defined like this:

def static <A, B> Pair<A, B> operator_mappedTo(A a, B b) {

return new Pair<A, B>(a, b);

}

But let’s get back to the previous example, where we turned strings into their uppercase versions. Although the applied sugar has improved the readability of the code a lot, the code is still very imperative, which is hard to grasp and error-prone.

Enter Functional Programming

Functional programming advocates a stateless, side-effect-free programming style, avoiding all kinds of imperative programming based on state changes. In strict functional programming, statements don’t exist, but everything is expressed in terms of pure functions that evaluate to the same result if invoked with the same arguments. There’s ultimately no global state or anything like that.

Being a Java-like language, Xtend doesn’t go that far but it goes pretty far. For instance, it doesn’t have statements. Everything that looks like a statement is actually an expression. Even a block is just an expression. Now you can use things that look like statements in places where it’s just not allowed in Java. Here’s an example:

val data = try {

fileContentsToString(‘data.txt’)

} catch (IOException e) {

‘dummy data’

}

In Java it wouldn’t be possible to use a try-catch on the right-hand side of an assignment. Since it’s a statement it doesn’t result in a value. In Xtend, however, everything is an expression and the result would be either the result of fileContentsToString or the string “dummy data” in case an IOException was thrown. By the way: Xtend doesn’t force you to catch checked exceptions. They are just rethrown in such a way that the compiler doesn’t complain, using a technique introduced by Lombok.

Another important ingredient of functional programming is higher-order functions. These are functions that get other functions passed in as an argument. We already have functions like this in Java. For instance java.util.Collections#sort(List, Comparator) is effectively a higher-order function since a Comparator usually is really just a function able to compare two instances of the same type. However, in Java there’s no concise syntax to implement an instance of Comparator. The most concise way would be an anonymous class.

Xtend, on the other hand, allows you to use function literals instead of anonymous inner classes. Those function literals are way more readable and can be passed to higher-order functions. They adapt transparently to the type that the invoked function expects. Xtend uses a syntax similar to the one known from Smalltalk and adds a couple of very useful higher-order functions to Java’s collection library by means of extension methods. This allows you to transform the list of strings from the previous example like this:

def toUpperCase(List<String> strings) {

strings.map( string | string.toUpperCase )

}

The functional style is not only much more readable but also less error-prone. Also note that in Xtend, as in most other modern languages, a return is implicit. The last expression in a block is automatically returned if the method’s return type is not void.

Extension Methods

In case you were wondering how java.util.List got this new fancy map method, it’s defined in the class ListExtensions, which is implicitly imported as an extension method. It’s part of Xtend’s runtime library and ultimately delegates to the collection library of Google Guava. If you want to use your own extension methods or want to use some existing methods as extension methods, you can import them using the extension keyword. Let’s have a look at an example:

import static extension java.util.Collections.*

class Test {

def printSorted(List<String> strings) {

// calls Collections.sort(List)

println( strings.sort() )

// calls Collections.sort(List,Comparator)

println( strings.sort [ a, b | a.compareTo(b) ] )

}

}

Both calls to sort will invoke a static member of java.util.Collections. The second example also shows how closures are automatically converted to existing Java types. In this case the closure is translated to an anonymous class of Comparator.

Extension methods are even more interesting in chained expressions, because they effectively allow you to write and read from left to right. That’s much more natural than reading inside out as you need to do when using the typical functional infix syntax. Imagine the following nested invocation of methods:

writeToFile( escape(myText), “Greeting.txt”)

You need to read this inside out if you want to follow how it is executed. Using extension method syntax, you can write it like this:

myText.escape.writeToFile(“Greeting.txt”)

Unlike other languages, Xtend doesn’t force extension methods to be static. Any locally declared method can be used as an extension and it’s even possible to use methods from local fields as extension methods. This is especially useful if you use dependency injection, as we will see later. But how can you define polymorphic extension methods for a whole type hierarchy?

Multi Methods (aka Dynamic Dispatch)

Let’s say you have a type hierarchy of shapes, Shape being the abstract super type of Circle and Rectangle, which are the two initial concrete cases. There are all kinds of computations you might want to do with shapes in the future, and we don’t want to modify and recompile the shape hierarchy every time we add some new functionality or add a new concrete subclass.

This problem is commonly known as the expression problem, which Xtend solves in a very elegant way.

New functionality is defined using extension methods. For instance, computing the area of a shape could be implemented like this:

class ShapeGeometry {

def getArea(Circle c) {

PI * c.radius * c.radius

}

def getArea(Rectangle r) {

r.length * r.width

}

}

Using extension methods we are now able to compute the areas of circles and rectangles:

new Circle(5).area

new Rectangle(4, 5).area

But of course we want to be able to call area on any element of type Shape and of course it should behave in a polymorphic way: The actual method to call will be picked at runtime using the runtime type of the shape. To do that you would add the dispatch keyword to the set of overloaded methods:

class ShapeGeometry {

def dispatch area(Circle c) {

PI * c.radius * c.radius

}

def dispatch area(Rectangle r) {

r.length * r.width

}

}

Now the compiler will infer one synthetic entry method, using the most common method signature. In this case that would be the signature def int getArea(Shape shape) since the common super type of Circle and Rectangle is Shape. The implementation of that method dispatches every call to the right case based on the runtime type information. It always finds the best match.

This is how you would use the extension methods in your code:

@Inject

extension ShapeGeometry

def testArea() {

val Shape shape = new Rectangle(4, 5)

assertEquals( 20, shape.area )

}

As you can see we didn’t use static extensions here, but have the extension method injected. We typically prefer Google Guice, but any dependency injection container should do and you could of course also use old-fashioned factories or the like to initialize the field.

Extension fields don’t need to have a name, since we will not directly refer to them. The expression shape.area is translated to this._shapeGeometry.area(shape).

Adding New Functionality

Let’s now assume you want to add a Triangle to the hierarchy, but you still cannot modify or recompile any code. What would you do?

In Xtend you would define the class Triangle and create a subclass of ShapeGeometry, where you add another dispatch method:

class ShapeGeometryForTriangles extends ShapeGeometry {

def dispatch area(Triangle t) {

(t.base * t.height) / 2

}

}

Xtend will infer a new synthetic dispatch method and override the one from the super class. Of course it will dispatch to every declared dispatch method including the ones from the super classes. Now you only need to tell your dependency injection framework to inject an instance of ShapeGeometryForTriangles if asked for an instance of ShapeGeometry.

There are other interesting language features in Xtend, like the sophisticated whitespace handling in template expressions and the cool switch expression. But one thing stands out.

Exceptional DSL Support

A domain specific language (DSL) is a small executable language solving a particular problem. Imagine you are working on a web framework. A typical task is to map http requests to controller methods or other resources. Instead of configuring that in an untyped, clumsy XML file, you could define it like this:

GET /index            -> StartController.index()

GET /list/Persons     -> PersonController.list()

GET /edit/Person/{id} -> PersonController.edit(id)

Wouldn’t it be cool to have all the great tool support you see in Xtend, including specific validations, which for instance would warn you about certain unreachable mappings? Also the references to classes and methods should be statically typed, including content assist, and even cross-language rename refactoring would be extremely nice. With Xtend and its support for DSLs this is an easy thing to do.

Xtend is built with Xtext, Eclipse’s language and IDE development framework. With that tool chain you can build such a request mapping language in no time. It will enable you to embed Xtend’s expressions, Java types, and annotations everywhere you want. These languages provide much more functionality and IDE support than anything you can do with internal DSLs and the definition of such a language is not rocket science. Have a look at the tutorial on Xtext’s website to understand how to do that.

Try it!

While learning a completely different programming language is something every developer should do from time to time, switching a whole project and all its team members to a new language is a huge effort. Xtend lets you use all your Java knowledge and existing Java code. At the same time the lightness of its concise syntax and the encouraged functional style will make programming much more enjoyable. The tooling is pretty close to what Eclipse’s Java Development Tools can do today and in some aspects it even works a bit better. The compiler, on the other hand, is completely Eclipse-independent and can be executed in any Java process. There is even a Maven plug-in to translate Xtend code to Java source code.

If you want to try it out, you could use the Eclipse Market Place Client from within Eclipse or go to the website and download a ready-to-use Eclipse distribution. Next you could materialize a tutorial project using the wizard. But as you’ve read this far you should be able to start right away. Just use the file wizard in the Java perspective to add an Xtend file to any of your Java projects. Unit tests are often a good starting point. ;-)

Have fun!

Sven Efftinge is the lead designer of Eclipse’s Xtend and Xtext. He lives and works in Kiel, Germany and runs an open-source development lab for itemis, a company specialized on Eclipse-based tooling and domain-specific languages.

Sebastian Zarnekow works for itemis and lives in Kiel, Germany. As the co-architect of Xtend and Xtext, he is specialized in language design, language implementation, and IDE development. Sebastian presents at international conferences and writes articles for magazines. Send the authors your feedback or discuss the article in the magazine forum.

Scala for the Intrigued

Functional Style of Programming
by Venkat Subramaniam
In this fourth installment of his series on the Scala programming language, Venkat delves into the functional style of programming in Scala.

Hello again! In this fourth installment of our series on the Scala language, we discuss the functional style of programming in Scala.

In the last article we saw the pure OO side of Scala. Scala also supports functional style; however, it does not enforce the style, so it’s often called a fully OO, hybrid functional language.

When designing Scala, Martin Odersky took a bold, unconventional step of bringing two different paradigms together, the OO and functional. This is not a trivial undertaking as these two styles are quite dissimilar and this marriage of distinct paradigms poses some real challenges.

Let’s first take a look at what it means to be functional. There are two defining aspects to the functional style of programming—the purity of functions and programming with higher-order functions.

Purity means that functions have no side-effects. The output of a function is predictably the same as long as the input is the same. A pure function is not affected by and does not affect anything outside; it also does not mutate any value.

There are two benefits of function purity. It’s easier to understand and prove the correctness of such a function. Furthermore, pure functions promote referential transparency; in plain English, pure functions can be easily rearranged and reordered for execution on multiple threads, making it easier to program concurrency on multicore processors.

Scala does not enforce purity, though it makes it easy to detect where mutable variables are used—simply search for vars. It is a good practice for us to use vals and immutability as much as possible when programming in Scala.

The other aspect of function style is working with higher-order functions—this is the facility that treats functions as first-class citizens. We can pass functions to functions, create functions within functions, and return functions from functions. This allows for functional composition and so in Scala we can design using both functional composition and object composition, as we desire or as appropriate.

Let’s explore these with some examples. We’ll start with some small examples that manipulate numbers so we can easily learn how to use higher-order functions (we’ll learn how to create functions that accept other functions in a future article). We will conclude with a practical example where we’ll apply the concepts we’ve learned.

Let’s start with a simple iteration: given a list of stock prices, we’d like to print each price on a separate line.

At first thought, the traditional for loop comes to mind, something like this in Java:

for(int i = 0; i <= prices.size(); i++)

or is it < instead of <= in the loop condition?

There is no reason to burden ourselves with that. At the least we should use the for-each construct in Java, like so,

for(double price : prices)

Let’s follow this style in Scala.

val prices = List(211.10, 310.12, 510.45, 645.60, 832.33)

for(price <- prices) {

println(price)

}

Scala’s type inference determines the type of the prices list to be List[Double] and the type of price to be Double. The above style of iteration is often referred to as an external iterator. Scala also supports internal iteration, so we could write the above example using the foreach function of List.

prices.foreach { e => println(e) }

The foreach function is a higher order function: it receives as parameter another function.

In general a function has four parts: name, parameter list, return type, and body. Of these four, the body is the most important. In the previous function that’s passed to foreach, the body is between the => and the closing }. The parameter list has one parameter e and since Scala can infer type, we did not have to say e : Double, though we could. Scala already knows the return type of this function based on what foreach expects and this function is anonymous, so does not have any name. The anonymous function we passed above is called a function value in Scala.

Scala has the smarts to allow us to reduce the code even further. If we simply pass the parameter received in the function value to another function in the body, like in this example, we can let Scala do that job.

prices.foreach { println }

That reduced some noise in the code; the parameter was received in the function value and passed to println.

We can take this up another notch: in an earlier article in this series we discussed how Scala makes quite a few things optional, including dot. So, we can refactor the above code to

prices foreach println

Here we’re sending the println function itself as a parameter to the foreach method instead of wrapping a call to it into a function value—the println function itself is treated here as a function value.

All right, we know how to use the functional style to iterate over elements: simply pass a function value that operates on an element to the internal iterator and it takes care of calling that function value with each element in the list as a parameter.

There are different flavors of internal iterators available on collections in Scala. Let’s look at the benefits a few of these offer.

If we want to pick the first price that’s greater than $500, we can do that without mutating any variables.

prices find { e => e > 500 }

If we want all the prices that are greater than $500, then replace find with the filter function.

prices filter { e => e > 500 }

If we have to compute ten percent of each of the prices given, once again we can achieve this elegantly using the map function.

println(prices map { e => e * 0.1 })

will print

List(21.11, 31.012, 51.045, 64.56, 83.233)

The map function applies the function value given, once for each element in the list, collects the result from the function value (which is 10% of the price in this example) into a list, and finally returns the collected list.

Finally, say we’re asked to total the prices given. We’re quite familiar with how to do that in the imperative style.

var total = 0.0

for(price <- prices) {

total += price

}

println("Total is " + total)

to get the output of

Total is 2509.6

However, a variable was tortured in the making of this example. We can avoid that with functional style again.

println("Total is " + prices.reduce { (e1, e2) => e1 + e2 })

//Total is 2509.6

The reduce function takes a function value that accepts two values. In the first call to the function value, e1 is bound to the first element in the list and e2 is bound to the second element. In each of the subsequent calls to the function value, e1 is bound to the result of the previous call to this function value and e2 to the subsequent elements in the list. The reduce function returns the result from the last call to the function value once it has been applied for each of the elements in the list.

We’ve seen a few functions here: foreach, find, filter, map, and reduce. It’s time to put these to a practical use.

Functional programming emphasizes immutability, but it’s equally about designing with state transformation and function composition.

In OO programming we strive for good object composition. In functional programming we design with function composition. Rather than mutating state, it is transformed as it flows through the sequence of functions.

Let’s prepare an example to see the difference between imperative style and functional style. We’re given a list of ticker symbols and our goal in this example is to find the highest priced stock not exceeding $500.

Let’s start with a sample list of ticker symbols.

val tickers = List("AAPL", "AMD", "CSCO", "GOOG", "HPQ", "INTC",

"MSFT", "ORCL", "QCOM", "XRX")

For convenience (and to avoid cryptic symbols in code) let’s create a case class to represent a stock and its price (case classes are useful to create immutable instances in Scala that provide quite a few benefits including ease in pattern matching).

case class StockPrice(ticker : String, price : Double) {

def print =

println("Top stock is " + ticker + " at price $" + price)

}

Given a ticker symbol we want to get the latest stock price for that symbol. Thankfully, Yahoo makes this easy.

def getPrice(ticker : String) = {

val url = "http://ichart.finance.yahoo.com/table.csv?s=" + ticker

val data = io.Source.fromURL(url).mkString

val price = data.split("\n")(1).split(",")(4).toDouble

StockPrice(ticker, price)

}

We fetch the latest stock price from the Yahoo URL, parse the result, and return an instance of StockPrice with the ticker symbol and the price value.

To help us pick the highest-priced stock valued not over $500, we need two functions, one to compare two stock prices and the other to determine if a given stock price is not over $500.

def pickHighPriced(stockPrice1 : StockPrice, stockPrice2 :

StockPrice) =

if(stockPrice1.price > stockPrice2.price) stockPrice1

else stockPrice2

def isNotOver500(stockPrice : StockPrice) = stockPrice.price < 500

Given two StockPrice instances, the pickHighPriced function returns the higher priced. The isNotOver500 will return a true if the price is less than or equal to $500, false otherwise.

Here’s how we would approach the problem in imperative style:

import scala.collection.mutable.ArrayBuffer

val stockPrices = new ArrayBuffer[StockPrice]

for(ticker <- tickers) {

stockPrices += getPrice(ticker)

}

for(stockPrice <- stockPrices.reverse) {

if(!isNotOver500(stockPrice)) stockPrices -= stockPrice

}

var highestPricedStock = StockPrice("", 0.0)

for(stockPrice <- stockPrices) {

highestPricedStock =

pickHighPriced(highestPricedStock, stockPrice)

}

highestPricedStock print

//Top stock is AAPL at price $377.41

Let’s walk through the code to see what we did.

First we create an instance of ArrayBuffer, which is a mutable collection. We invoke the getPrice() function for each ticker and populate the stockPrices ArrayBuffer with the StockPrice instances.

Second, we iterate over these stock prices and drop from this mutable collection those stocks that are valued over $500. This results in possibly fewer elements than we started with.

Finally we loop through the collection one more time to pick the stock that is valued the highest among them, again mutating the highestPricedStock variable as we navigate the collection using the external iterator.

We can improve on this code further, use multiple collections if we desire, wrap the code into separate functions, and put them into a class if we like. However, that will not affect the fundamental approach we took: imperative style with mutable data structure. The state of the collection of stocks and their prices went through quite a few mutations.

Let’s write this code in functional style. Ready?

tickers map getPrice filter isNotOver500 reduce pickHighPriced print

That’s it, it’s small enough to fit into a tweet. OK, this conciseness takes some getting used to.

tickers map getPrice first transforms the collection of tickers into a collection of StockPrice instances. For each ticker symbol, now we have the name and price in this collection. The filter function then applies the isNotOver500 on that collection and transforms that into a smaller collection of StockPrices with only stocks whose price does not exceed $500. The reduce function takes that further to pick the highest-priced stock, which we finally hand over to the print method of StockPrice.

In addition to being concise, the code does not mutate any state. The state goes through transformations as it flows through the composed functions.

While this code is elegant and concise, what about debugging and performance? These are two questions I’m often asked.

From the debugging point of view, there are no mutable states, so there are fewer options for errors than in code with several mutable parts. We can write unit tests on each of the intermediate steps separately and also on the collective results. We can step through the code individually or collectively. You can also store the intermediate values in immutable vals along the way so we could examine those.

If the collection is fairly small we won’t see any significant performance impact. If the collection is fairly large, we may face some copy overhead. Let’s not reject the goodness of functional style in the name of performance. Prototype and see if performance is a concern for what you have to implement. If the performance is adequate, then you have gained from what the paradigm has to offer. If performance is not reasonable then you can explore options to improve it. One option is to use persistent data structures—these are immutable data structures that perform intelligent selective copying to provide close to constant-time copy performance.

I hope this article whets your appetite for learning further and using the functional style of programming in Scala. In the next article we’ll see how Scala collections make use of this style to provide a concise and fluent interface.

Dr. Venkat Subramaniam is an award-winning author, founder of Agile Developer, Inc., and an adjunct faculty at the University of Houston. He has trained and mentored thousands of software developers in the US, Canada, Europe, and Asia, and is a regularly-invited speaker at several international conferences. Venkat helps his clients effectively apply and succeed with agile practices on their software projects. Venkat is the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy: Dynamic Productivity for the Java Developer and Programming Scala: Tackle Multi-Core Complexity on the Java Virtual Machine. His latest book is Programming Concurrency on the JVM: Mastering Synchronization, STM, and Actors. Send the author your feedback or discuss the article in the magazine forum.

Revisiting Graphics Hardware

Using modern computing horsepower to stretch retro graphics hardware
by James Bowman
James revisits the world of 8-bit game programming in creating the Gameduino, an 8-bit graphics and sound shield for the Arduino.

The 80s: big hair, muscle cars, and graphics systems that were... interesting. To save precious bytes, early systems like Nintendo had graphics displays that were not directly pixel-addressable. Instead, they let you define a few hundred background tiles, typically 8x8 pixel bitmaps. The display was made by reference to these tiles, using a dedicated RAM called the tile map.

Systems like this were very efficient for rendering Mario World, with its repeating block elements, but had trouble showing arbitrary images. Eighties artists spent hours—sometimes days—working around this restriction, carefully crafting a screen image to fit in the limited tile budget.

Since we now live in the glittering future, is it possible to automate the process, and show arbitrary images on this hardware?

Gameduino: Tiles Are Back!

I recently revisited this world when I made the Gameduino, an 8-bit graphics and sound shield for the Arduino. The background graphics are tile-based; the tiles are 8x8 pixels, and the screen is 50 tiles wide by 38 tiles high. As you might expect from an 8-bit graphics system, there are 256 unique tiles.

Instead of a storage location for every pixel on the screen, the system only needs storage for a tile map—the array of bytes that give tile numbers— and the graphics for the tiles themselves. This saves a lot of RAM, and has other advantages.

First, it can make drawing graphical elements very efficient: for example to paint a 32x56-pixel element, the CPU needs to update a 4x7 rectangle to the tile map, so it only writes 28 bytes of video memory. In the days of 8-bit CPUs running at a Megahertz, this would have been a significant economy.

Second, 8-bit game authors used the tile map hardware for animation effects. A small graphics update can cause a dramatic on-screen change by using a tile repeatedly on the screen and then changing the tile’s graphics. Conveyor belts, ocean waves, lightning strikes: this trick is one of the staples of 8-bit game animation.

Displaying Any Image

Since there are 256 tiles available, drawing an image is straightforward if it uses 256 or fewer tiles. If the image is small, say 128x128 pixels, then it requires 16x16=256 tiles. Easy enough.

The challenge is how to deal with a larger image. The maximum case, of an image that covers the whole screen, uses 2000 tiles. With only 256 unique tiles available, each tile should be used an average eight times. Hopefully, some parts of the image contain less detail—for example a dark shadow—and can share the same tile. Some parts of an image contain such important detail—an eye, for example—that they should be allocated a tile that is used only once.

The Math: K-means Clustering

This is a classic scarcity problem: we’d like 2000 unique tiles, but only have 256. The perfect tool for this job is k-means clustering. The k-means algorithm clusters a collection into groups based on likeness: we supply it with 2000 tiles, and after a prodigious amount of computation, it gives back 256 clusters of tiles, where the tiles in each cluster are reasonably alike.

The k-means algorithm operates on points distributed in space; in the illustration above it is 2-dimensional, but k-means can use any N-dimensional space we choose. The input to k-means is this collection of points in N-dimensional space—in our case each point represents a tile—and the algorithm returns a cluster: each original tile is in a group of tiles that are visually similar.

For implementation, I chose to work in Python with OpenCV. OpenCV is a popular computer vision library, and has a really excellent Python interface (it should be mentioned here that I’m the original author of its Python interface.) In addition to an implementation of the k-means algorithm, OpenCV has various other imaging operations that came in handy as I refined the implementation. OpenCV from Python gives very fast graphics operations, with the easy prototyping that comes from Python. Happily, Gameduino’s data preparation tools are Python-based, as well.

The key to getting good results from k-means is to choose a good space for it to operate in.

Choosing a Space

One simple approach would be to have k-means work in an RGB color space. The coordinates of a tile are simply the (R,G,B) average of the tile’s pixels. So tiles that are approximately the same color are clustered together. But this ignores all the actual pixel detail in each tile, and our perception of color changes. For example, we are much more sensitive to green shading differences than blue ones.

Much better to use a couple of well-known image processing tricks to place the tiles in a perceptually metric space. “Perceptually metric” means that visually similar tiles are close together in this space. Put another way, if two tiles look alike, they should be near to each other. This means that k-means can know that they are neighbors, and treat them as candidates for clustering.

In order to do this perfectly, you’d need a complete simulation of the human visual system, so that you could compute just what it means for two tiles to “look alike.” Fortunately an approximation does quite well enough. Representing pixels in a color space like YUV makes similar-looking colors have similar YUV coordinates—which is why it is used extensively in image compressions systems like JPEG and MPEG:

cv.CvtColor(tile, yuv, cv.CV_BGR2YCrCb)

The next trick is to not use the tile’s pixel values directly, but to apply the DCT—Discrete Cosine Transform—to them:

cv.DCT(chan, dct, cv.CV_DXT_FORWARD)

Without going into too much detail, using the DCT expresses the tiles’ features from coarse to fine. For example, two tiles that have a dark blob on the left will have similar DCT values, even if the shapes of the blobs differ.

What do these things give us? For each tile, originally containing 64 pixels of RGB data, we convert the pixels to YUV space and apply the DCT to each channel. This results in a string of 192 numbers—coordinates in a 192-dimensional space where visually similar tiles are close together. After the samples are prepared, the k-means call itself is:

cv.KMeans2(samples, tilecount, labels, tc, flags = 2)

The labels argument is the k-means output: a list of labels, one per input tile, that says which cluster the tile belongs to.

Who’s Afraid of the 192nd Dimension?

The k-means implementation works just fine in any space—including 192 dimensional ones—and we might as well use all the data available so that it can make good choices about tile clustering. The fact that each tile is represented by a point in 192-dimensional space is really just an implementation detail—while we might have a lot of difficulty imagining a 192-dimensional space, the math of distances works just the same in higher dimensions, which is all k-means needs to know about.

This is not to say that very high dimensionality is always a good idea—in many cases it can be a recipe for numerical disaster. For example a tiny amount of noise in every dimension can utterly thwart the search for a signal. But in this case the goal is the expression of proximity, and higher dimensionality is fine.

Average of Clusters

After k-means gives us 256 clusters, the rest of the work is straightforward. For each cluster, the code merges the member tiles—just a simple average of pixels—and uses this average tile in place of all the original members:

s = cv.CreateMat(8, 8, cv.CV_32FC3)

cv.SetZero(s)

for t in tiles:

cv.Add(s, t, s)

cv.Scale(s, s, 1.0 / len(tiles));

This completes the process: for every 8x8 tile in the original image we now have a ‘closest match’ in the carefully-chosen set of 256 tiles.

How does it look? Well, it depends on the image. For images that contain a lot of fine detail, the algorithm does not do well: it tries to reuse tiles even though they are not visually similar, resulting in a scrambled mess. But for images that have a limited amount of detail, like this portrait, it does quite well.

The k-means clustering has fully used its ‘budget’ of 256 tiles. A handful of tiles reused many times make up the low-detail segments of the image—the background and the table. And it has used a few tiles once for the high-contrast details like eyes and the outline of the shirt.

How Much Does It Cost?

The encoding process takes about 1 second and 22 megabytes on a modern workstation-class computer. A rough estimate, given 30 years of Moore’s Law, is that the encode program would have taken about 12 days on a 1981 machine, assuming you could have found one with enough memory, and enough time between hair-care and car-polishing duties.

There is an online version of the conversion tool here The source for the program is here. Prerequisites are OpenCV and the Gameduino Python tools.

Many thanks to Scott Lembcke for his original suggestion and implementation for using k-means for tile allocation. CC source image by Steve Evans.

James Bowman wrote games for 8- and 16-bit consoles, and later made graphics software and hardware at SGI, 3dfx, and NVIDIA. James now consults on graphics, imaging, and hardware. Send the author your feedback or discuss the article in the magazine forum.

Just-In-Time Logging

A Modest Proposal
by Brian Tarbox
Moving logging from the province of the developer’s discretion to the circle of architecture and design results in just-in-time logging.

Logging is the poor stepchild of many applications, even though for many enterprise applications it is the primary means of diagnosing problems. You cannot, after all, attach a debugger to Amazon.com.

Many existing products simultaneously suffer from excessive logging and insufficient information in those logs. This often proves to be an enormous performance penalty, and makes diagnosing systems in the field problematic. We often write so many hundreds of thousands of lines of logs that most support personnel have developed libraries of grep routines to throw away 99% of the text. In addition, so much is written to the logs that they roll over too quickly and the information about the original cause is often lost. Over the years many teams have repeatedly attempted to trim the amount of logging, with limited or temporary success.

Insanity is defined as doing the same thing and expecting different results. It is time to take a new approach to logging. The new approach proposed here adds formality and rigor to the topic of logging, while at the same time improving performance and ease of post-hoc analysis.

Details of the Problem

While we have at times enforced a discipline of design with regard to large matters such as architecture, the field tends to treat logging as something left to the discretion of the individual developer. We have quite appropriately wanted to leave developers with some freedom of action in which to exercise their craft. This approach, while admirable, has nonetheless had its drawbacks.

Few teams exhibit discipline about which messages should be marked debug and which should be marked info. Fewer still exhibit discipline about putting the relevant noun in the log messages, where the noun would be the ID of the thing being described by the message. This has meant that very often in order to have access to sufficient information one needs to log at the worst level; i.e., debug. This in turn has resulted in our systems logging literally millions or tens of millions of lines per hour. This has had an enormous negative effect on our overall performance, to say nothing about the increased difficulty of managing the system.

Previously Attempted Solutions That Have Failed

Over the last ten years of working with various teams I’ve seen any number of bug scrubs or decrees issued. “All log messages will contain nouns,” “all debug messages shall be protected by if(log.isDebugEnabled) tests,” and “the number of log messages shall be strictly limited.” Each of these endeavors resulted in a significant and brief improvement, which was quickly overcome by the entropy of unstructured coding.

Suggesting that we perform another such log scrub flies in the face of reality. Another approach is required.

Proposed Solution

The proposed solution moves logging from the province of developer’s discretion to the circle of architecture and design. It is to be something that is designed and proscribed. The proposed solution is similar to a logging solution that was successfully implemented for stored procedures. The approach recognizes that different levels of information should be logged for successful versus unsuccessful operations. In a perfect world any failure could be fully analyzed from the single log message written at the point of detecting the failure. This is not usually the case, however. If a video stream create fails because the selected streamer has no remaining resources it is interesting to know how the streamer was selected. Successful operations by definition have fully completed, while failed operations by definition do not fully complete.

The problem is that until the operation completes one cannot tell whether it will be successful or not. The stored procedure logging solution was to write log messages into an in-memory-temp table in the database. This operation is close to free in terms of performance. If the stored procedure succeeds the temp table is dropped. If the stored procedure fails it performs a bulk insert from the temp table into a normal database log table.

The log4j approach is to buffer all log messages in a HashMap indexed by the relevant noun. This is a very fast operation and so one can write as many log messages as one wants. When the operation completes, the system does one of two things. If the operation failed the HashMap contents can be stringified and the buffer can be written in a single step incurring only a write. If the operation succeeded we assume that the string buffer contains far more information than we need, so we write a summary of that buffer. So in the success case we write a single line of output to the log. Since we assume that in a normally running system at least 95% of operations will succeed, this approach ought to reduce the logging volume by close to 90%. This will have a dramatic effect on how long log files live. That would be a game changer in terms of the debuggability of systems both in our labs and in the field. This alone might be reason enough to make the change.

Implementation

There are several parts to the suggested implementation. The first part is a new buffered logger class that extends the standard log4j Logger class. This class contains a static ConcurrentHashMap<Long, StringBuffer> to hold the various log messages. It also contains overloaded versions of the standard info, debug, and warn methods. These overloaded versions take two parameters: the first being the noun ID and the second being the message. Instead of writing a log message like this:

log.info(“something happened”);

you will instead write:

log.info(nounID, “something happened”);

The result of this call will be that your new message gets appended to the string buffer of messages associated with that particular noun instead of being directly written to the log. The BufferedLogger class supports a flush method that is called when the client wants to output the accumulated string buffer. The flush method contains a normal log4j logger which writes the stream buffer to the actual server.log file.

There is also a remove method that is called when an operation succeeds. This method takes a nounID and a string message. It removes all entries for the nounID from the hash map and logs the supplied success message to the real log4j log file. The other major part of the implementation is of course the changes to client code to use the new approach. Please note that only high volume operations need to be converted to this approach. To use this style of logging a client class would create a static BufferedLogger just as we normally create a standard logger at the top of each class. One could even create both a standard logger and a BufferedLogger within the same class.

For those log messages associated with a given verb each log.info(message) call would be changed to log.info(msid, message). This means that each log message involved in the processing of a particular verb must include the nounID, but after all that is a goal we have been striving for unsuccessfully for years. This approach simply mandates it by including the nounID as part of the method signature.

The verb operation must be surrounded by the appropriate try, catch, finally blocks so that the code can determine if the operation succeeded and so the buffered logger can call flush or remove. Again that structure should already be in place.

Enforcing the New Approach

There is actually a very simple way to ensure that this new approach is used. You can create a unit test that calls the code for the verb in question. In that test’s overload Logger.createInstance (which is how code obtains the old style log4j logger instance), the overload will return a mock logger instance that calls assertTrue(false) if it is called. So, if any code calls the old-style logging during stream create the unit test will fail. In this way we do not rely on developers to follow the rules, we make rule enforcement automatic.

What about Message Formatting?

Another area that can yield dramatic improvements in system performance is the proper use of formatting directives in log messages. It’s hard to believe but I’ve seen numerous groups use formatting directives whose JavaDoc says “never use this directive in a production system.” The worst example of this is the %M directive which uses Java Introspection on each and every log call to determine the name of the method being logged. Tests show that the presence of this directive can have a massive effect on the overall performance of the system.

The following graphs show the time to execute a key system verb on systems running with and without the %M directive. Note that removing this directive caused the overall system throughput to dramatically increase.

Since the proposed change to logging already involves looking at the log messages that are emitted during the critical paths it is here asserted that the extra cost of changing those messages to indicate where they come from is small, especially compared to the benefit gained by using log4j as its designers intended.

Conclusion

The upside of this approach is

- An enormous reduction in the amount of logging that we perform

- greatly increased lifetime of log files allowing one to deal with customer issues more than 36 hours old

- automatic grouping of log messages for a given operation

- high level of information available for trouble shooting failures (in fact given that we will be logging so little we may well be able to add more debug level information knowing that it will only be written when needed)

- more informative log messages since the new logging method signatures require a noun ID

Brian Tarbox is a Distinguished Member of Technical Staff at Motorola in the Systems Engineering group designing next generation video products. He writes a blog on the intersection of software design, cognition, music, and creativity at briantarbox.blogspot.com. His primary contact point is about.me/BrianTarbox . Send the author your feedback or discuss the article in the magazine forum.

When Did That Happen?

A Holiday Fable
by Dan Wohlbruck
Three giants in the history of computing meet for lunch during World War II.

During the first or second week of December 1942—it’s impossible to be more precise—three men met for lunch at the Hay-Adams House in Lafayette Square in Washington, D.C. All of the men were geniuses and each of the three made separate, landmark contributions to the creation of electronic computers in the twentieth century.

Two of the men arrived together. One of them, Alan Turing, had been sent to the United States by the British government to review the work being done in the US to break the codes used by the Nazis to communicate with their U-boat fleet. Turing had volunteered for service on the day that Britain declared war on Germany, and he had designed a machine that deciphered messages sent by German commanders during the Battle of Britain. While Turing was in the US, he also worked on a project to encrypt the voice communications used by Churchill and Roosevelt. Since the telephone cables connecting the two countries ran under the Atlantic, the Allies feared that Germany would splice into the cables and intercept the conversations between the Allied leaders.

Turing worked on the encryption problem with Claude Shannon from AT&T. While a postgraduate student, Shannon had created a way to apply Boolean logic to the design of digital circuits. Subsequently he had become a recognized expert on electronic communications. And, although at different times, both Turing and Shannon had studied at Princeton’s Institute for Advanced Studies, both tutored there by one John von Neumann.

After hearing about Turing’s visit, von Neumann arranged for a lunch with his two former students.

Neither Turing nor Shannon had previously dined at the Hay-Adams and as they got out of the taxi cab they shared, they paused. Turing turned to look at the White House just across Lafayette Square. Shannon, a true American from the Midwest, had seen the White House but had never enjoyed the luxury of a restaurant like the Lafayette Room. A bit intimidated, Shannon waited for Turing to lead the way. The men were seated at the table von Neumann had reserved and waited for their host to arrive.

The two men could hardly have been more different. Turing was small of stature and well-dressed. Shannon was tall, rangy, and uncomfortable. Other than being former students of von Neumann, the only thing that the two men shared was their common project, and that did them no good because they were not permitted to talk about it in public. Left with no other subject, they traded stories about Johnny.

John von Neumann was well known and well liked in Washington. His intelligence was obvious without being showy. He dispensed his expertise and wisdom without desire for favor. When he testified before Congressional committees, senators and representatives hung on every phrase. He was believed and trusted and he knew everyone.

Suddenly, the room grew quiet, as all eyes turned to the man making his way, table by table, from one end of the restaurant to the other. Johnny von Neumann had arrived. He stopped here to greet Senator Truman and there to say a word to a White House staffer. Without conscious thought, both Turing and Shannon rose and waited by their chairs as a show of respect. Von Neumann finally got to their table and the three men shook hands with warm affection, then sat and ordered lunch.

Over coffee when the lunch dishes were cleared, von Neumann led his former pupils to a discussion of a problem he was facing.

What followed was something rare and possibly unique. These three shared a different type of genius. Turing’s mind knew no limits. He was without prejudice or inhibition and could frame a solution without concern for conventional wisdom. Shannon, for his part, was able to connect two dissimilar fields of study by finding how they interacted. John von Neumann was an assimilator. Where Turing and Shannon could explain what could be done, von Neumann could describe how to bring it about.

“Alan,” he said, “I need a machine that can reason. I want something that I can feed with information, and to which I can then pose a question, and wait for an answer.”

Turing considered the challenge. After a moment of thought, he suggested an approach. Shannon had an inspiration and added his insight. As the two men grew more excited, von Neumann made notes and diagrams on a linen napkin. Forty minutes later, it was done.

With a final flourish, Turing said, “And thus a machine can think.”

“What, I wonder, would such a machine think about?” von Neumann mused.

In the midst of a world war, it wasn’t surprising that Turing replied, “Peace on earth.”

Nodding his agreement, Johnny added, “And goodwill.”

Ever the democrat, Shannon finshed for them all, “To all of mankind.”

Author’s note: The three men did meet, as indicated, during December 1942. Their actions and attributes are reconstructed from reliable sources. The product of their meeting is historical, holiday fiction.

Dan Wohlbruck has over 30 years of experience with computers, with over 25 years of business and project management experience in the life and health insurance industry. He has written articles for a variety of trade magazines and websites. He is currently hard at work on a book on the history of data processing. Like this article? Hate it? Send the author your feedback or discuss the article in the magazine forum.

Calendar

Author sightings, partner events, and other notable happenings.

Two pioneers of computer programming were born in the month of December: Ada Lovelace on December 10, 1815, and John Backus on December 3, 1924. I share a birthday with Lovelace and Backus died four years ago just a few miles from where I live, so I think of them at this time of year. But there are several better reasons to celebrate their birthdays.

Wherever you stand on the “first programmer” claim, Ada is certainly a model for women in technology. As a child she showed great interest and promise in mathematics, and one of her tutors (no less than Augustus De Morgan) praised her originality and suggested that she might become a first-rate mathematician. Struggling with illness most of her brief life, working on mathematical problems while raising children, she nevertheless grasped the workings of Charles Babbage’s Analytical Engine so thoroughly that she was able to document it and code algorithms that would have run on the machine if Babbage had ever been able to build it. Babbage called her “The Enchantress of Numbers.”

John Backus created FORTRAN, the first widely-used high-level programming language. He developed the hugely important Backus-Naur Form (BNF), a formal notation in which one can define any context-free programming language. He was one of the first people to promote a functional style of programming and even created a functional language, FP, whose innovations are still visible in functional programming languages today. His work won him a Turing Award.

I’m raising a birthday glass to Ada Lovelace and John Backus. -Michael Swaine

Author Appearances

Who’s where, and what for.

- Jan 11-13, CodeMash, Sandusky, OH
“Developing With iOS 5 ” and “iOS Networking: Bonjour, iCloud!”Chris Adamson, co-author of iPhone SDK Development

- Jan 11–13, CodeMash, Sandusky, OH
“Mac Development for the iOS Developer” and “Storyboards: New development in iOS5”Daniel H Steinberg, co-author of iPad Programming: A Quick-Start Guide for iPhone Developers and author of Cocoa Programming: A Quick-Start Guide for Developers

- Jan 23–27, OOP 2012, Munich, Germany
Tutorial: “Influence and Authority: Using Your Personal Power to Get Things Done,” Talk: “Six Behaviors to Consider When Hiring for an Agile Team”Johanna Rothman, author of Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects, Manage It!: Your Guide to Modern, Pragmatic Project Management, and Behind Closed Doors: Secrets of Great Management

- Jan 24, By the best from DevSum, Stockholm, Sweden
“Advanced Regex in .NET”Staffan Nöteberg, author of Pomodoro Technique Illustrated

- Feb 13–15, Jfokus, Stockholm, Sweden
“Regex Applied—when regex is a winner”Staffan Nöteberg, author of Pomodoro Technique Illustrated

- Feb 28, Softdevon III, Crakow, Poland
“Pomodoro Technique—can you focus for 25 minutes?”Staffan Nöteberg, author of Pomodoro Technique Illustrated

O’Reilly Events

Upcoming events from our friends at O’Reilly.

- Feb 13–15, TOC, New York, NY
O’Reilly Tools of Change for Publishing Conference 2012: “O’Reilly’s TOC Conference is where the publishing and tech industries converge, as practitioners and executives from both camps share what they’ve learned from their successes and failures, explore ideas, and join together to navigate publishing’s ongoing transformation.”

- Feb 28–Mar 1, Strata Conference, Santa Clara, CA
O’Reilly Strata Conference: “Strata brings together practitioners, researchers, IT leaders and entrepreneurs to discuss big data, Hadoop, analytics, visualization, and data markets.”

USENIX Events

What’s coming from our USENIX friends.

- Dec 12–16, Middleware 2011, Lisbon, Portugal
ACM/IFIP/USENIX 12th International Middleware Conference: “The scope of the conference is the design, implementation, deployment, and evaluation of distributed system platforms and architectures for future computing and communication environments.”

- Feb 5–8, NDSS 2012, San Diego, CA
19th Annual Network & Distributed System Security Symposium: “The Network and Distributed System Security Symposium fosters information exchange among researchers and practitioners of network and distributed system security.”

- Feb 14–17, FAST ’12, San Jose, CA
10th USENIX Conference on File and Storage Technologies: “FAST ’12 brings together storage system researchers and practitioners to explore new directions in the design, implementation, evaluation, and deployment of storage systems.”

Author Index

Thirty Months of Articles, Sorted by Author

From Chris Adamson to Marcus Zarra, over eighty authors have contributed to our first two and a half years.

Chris Adamson

- Writing an iPhone App, August 2009

- The Nostalgia of the New, February 2010

- Nice Apps, If You Can Afford To Write Them, March 2010

- Punk Rock Languages, March 2011

Jorge Aranda

- Against SEMAT, January 2010

Kent Beck

- Responsive Design, September 2009

- iPad Roundtable, February 2010

- Agile @ 10, February 2011

Aaron Bedra

- Create Unix Services with Clojure, July 2011

- Hello, ClojureScript!, August 2011

Jeremy Bingham

- Your Halloween Costume, October 2010

David Bock

- State Secrets, October 2009

Ambrose Bonnaire-Sergeant

- Growing a DSL with Clojure, July 2011

Nicolas Bourbaki

- iPad Roundtable, February 2010

Tony Bove

- iPad Roundtable, February 2010

Daniel Brolund

- The Mikado Method, June 2010

Ed Burnette

- iPad Roundtable, February 2010

- Android's Next Move, October 2009

Trevor Burnham

- A CoffeeScript Intervention, May 2011

- Decouple Your Apps with Event-Driven CoffeeScript, August 2011

Paul Butcher

- Forgive and Remember, December 2009

- Great Expectations, August 2010

Jeff Cohen

- Open Source Community Values, October 2011

Ward Cunningham

- Agile @ 10, February 2011

Frederic Daoud

- Stretching Stripes, November 2009

James Duncan Davidson

- iPad Roundtable, February 2010

- The Daily Shoot, January 2010

Rachel Davies

- The Agile Coach, November 2009

- Building Trust, September 2010

Scott Davis

- Beauty in Code, September 2009

Ian Dees

- iPhone, Meet Cucumber, August 2009

- New Series: Everyday JRuby, December 2010

- Everyday JRuby, January 2011

- Testing Arduino Code, April 2011

Zach Dennis

- Practical Mock Advice, June 2011

Bill Dudney

- Bonjour, iPhone, November 2009

- Inside iOS 5, November 2011

Ola Ellnestam

- The Mikado Method, June 2010

Chad Fowler

- Clone Yourself, September 2009

- Chad Fowler on Ruby, December 2010

Eric T. Freeman

- Embedding a Movie on iPad, April 2010

James Edward Gray II

- TextMate Tricks, October 2009

- iPad Roundtable, February 2010

- The Red Dirt Interview, April 2010

Dana Gray

- The Red Dirt Interview, April 2010

James Grenning

- Agile @ 10, February 2011

Adam Goucher

- Page Objects in Python, August 2010

- Testing for the Cloud, March 2011

- Lightsabers, Time Machines, & Other Automation Heuristics, October 2011

Phil Haack

- Why ASP.NET MVC?, June 2010

Stuart Halloway

- When Things Go Wrong, July 2009

Dan Haywood

- Going Naked, December 2009

Jean-Francois "Jeff" Heon

- Clojure Building Blocks, July 2011

Rich Hickey

- Why Clojure?, July 2009

Brian Hogan

- Better CSS with Sass, February 2010

- HTML5: Accessibility For All, October 2010

Rob Holland

- BDD on iPhone: iCuke, July 2010

Dave Hoover

- Mail on Rails, October 2009

Mike Hostetler

- A Troubleshooter's Tale, April 2010

Jason Huggins

- Automating Screencasts, December 2009

- JavaScript: It's Not Just for Browsers Any More, March 2010

Andy Hunt

- Welcome to PragPub, July 2009

- Pragmatic Life, August 2009

- Painting with Sound, October 2009

- iPad Roundtable, February 2010

- All Your Context, August 2010

- Guru Meditation, September 2010, October 2010, November 2010, December 2010, January 2011, May 2011, June 2011, Guru Meditation, August 2011

- Agile @ 10, February 2011

John Jainschigg

- iPad Roundtable, February 2010

Ron Jeffries

- Agile @ 10, February 2011

Jon Kern

- Agile @ 10, February 2011

David Koelle

- ...And Your Bugs Can Sing, September 2009

Jeff Langr

- Agile Flash Cards, September 2010

- What Agile Is Not, October 2010

- Shu Ha Ri, November 2010

- Cohesive Software Design, December 2010

- Code Coupling, January 2011

- Abstraction, February 2011

- Software Volatility, March 2011

- Test Abstraction, April 2011

- Agile Reflections, May 2011

- Pair Programming in a Flash, June 2011

- Pair Programming Benefits, July 2011

- How Virtuous Is Your Code?, August 2011

- The Only Agile Tools You'll Ever Need, September 2011

- But We Have These Distributed Folks, October 2011

- Test-Driven Development, November 2011

Andy Lester

- The Layoffs Are Coming!, July 2009

- The Working Geek, December 2009, January 2010, February 2010, March 2010, April 2010, May 2010, June 2010, July 2010

David McClintock

- iPad Roundtable, February 2010

Jonathan McCracken

- Agile Microsoft, June 2010

Chris McMahon

- Writing and Performing, November 2010

- Rediscovering QA, January 2011

Stephen Mellor

- Agile @ 10, February 2011

Staffan Noteberg

- The Pompatus of Pomodoro, November 2009

- De Morgan to the Rescue, February 2011

Alan Oppenheimer

- iPad Roundtable, February 2010

Tim Ottinger

- Agile Flash Cards, September 2010

- What Agile Is Not, October 2010

- Shu Ha Ri, November 2010

- Cohesive Software Design, December 2010

- Code Coupling, January 2011

- Abstraction, February 2011

- Software Volatility, March 2011

- Test Abstraction, April 2011

- Agile Reflections, May 2011

- Pair Programming in a Flash, June 2011

- Pair Programming Benefits, July 2011

- How Virtuous Is Your Code?, August 2011

- The Only Agile Tools You'll Ever Need, September 2011

- But We Have These Distributed Folks, October 2011

Paolo Perrotta

- Much Ado About Nothing, January 2010

- Three Bundler Benefits, December 2010

Steve Peter

- Our Daily Bread, December 2009

- Our Daily Beer, March 2010

Susannah Pfalzer

- Let's Talk about Your Book, November 2010

Noel Rappin

- The Virtues of Mockery, May 2010

- Testing for Web Services, March 2011

Jonathan Rasmusson

- The Indispensable Developer, June 2010

- Way of the Agile Warrior, October 2010, November 2010, December 2010, February 2011, March 2011, May 2011, June 2011

- Self-Inflicted Scope Creep, November 2011

Steven Reynolds

- Clojure Collections, July 2011

Jared Richardson

- Trench Warfare, May 2011

Craig Riecke

- JSON -- The Last 10 Feet, March 2010

- Refactoring Your Job, February 2011

Johanna Rothman

- Managing Your Life Projects, August 2009

- Barriers to Agility, February 2010

Maik Schmidt

- Meet the Arduino, July 2010

- Advanced Arduino Hacking, April 2011

- Create your Own Arduino IDE, April 2011

- Make Your Own Video Game System, August 2011

Ben Scofield

- In Defense of Mediocrity, August 2010

Ken Schwaber

- Agile @ 10, February 2011

Liz Sedley

- The Agile Coach, November 2009

John Shade

- Shady Illuminations, July 2009, August 2009, September 2009, October 2009, November 2009, December 2009, January 2010, February 2010, March 2010, April 2010, May 2010, June 2010, July 2010, August 2010, September 2010, October 2010, November 2010, December 2010, January 2011, February 2011, March 2011, April 2011, May 2011, June 2011, July 2011, August 2011, September 2011, October 2011, November 2011, December 2011

Eric Smith

- TDD on iPhone: DIY, July 2010

Alan Smithee

- iPad Roundtable, February 2010

Daniel Steinberg

- Get a Life, August 2009, September 2009, October 2009, November 2009, December 2009, January 2010, February 2010

- Not Quite New in iOS 4, July 2010

- Embedding a Movie on iPad, April 2010

Venkat Subramaniam

- Scala for the Intrigued, September 2011, October 2011, November 2011

Michael Swaine

- Meet the Team: Miles Forrest, September 2011

- Meet the Team: Susannah Pfalzer, October 2011

- Meet the Team: Janet Furlow, November 2011

- Meet the Team: David Kelly, December 2011

- ...and a bunch of regular or semi-regular departments: Up Front, Choice Bits, Swaine's World, Calendar, The Quiz, Interviews, But Wait, There's More....

Brian Tarbox

- ...And Your Bugs Can Sing, September 2009

- Interrupt Mood, November 2009

- Testing as Contract, March 2010

- Medicine Making Music, April 2010

- The Floating Finish Line, May 2010

- Better Tools and Training, June 2010

- The Key to Better Software, September 2010

- Catch the Pig!, May 2011

- Code As Dialog, August 2011

- Scala Traits, September 2011

- Root Cause Analysis from Long Ago, November 2011

Bruce Tate

- The Bruce Tate Interview, May 2010

- Grokking Pattern Matching and List Comprehensions, January 2011

Mike Taylor

- Tangled Up in Tools, April 2010

Dave Thomas

- Welcome to PragPub, July 2009

- Pragmatic Publishing, July 2009

- The Quiz, July 2009

- How Do We...?, July 2009, August 2009, September 2009, October 2009, November 2009, December 2009

- iPad Roundtable, February 2010

- What's New in Ruby 1.9.2, December 2010

- Agile @ 10, February 2011

Arie van Bennekum

- Agile @ 10, February 2011

Gregg Williams

- Getting Clojure, November 2010

Dan Wohlbruck

- When Did That Happen?, February 2010, March 2010, April 2010, May 2010, June 2010, July 2010, August 2010, September 2010, October 2010, November 2010, December 2010, January 2011, March 2011, April 2011, May 2011, June 2011, July 2011 When Did That Happen?, August 2011 September 2011 October 2011 November 2011

Hew Wolff

- Good Code Tells the Truth, October 2010

Marcus Zarra

- Touching the Core, January 2010

- iPad Roundtable, February 2010

Shady Illuminations

Here’s to the Crazy Ones
by John Shade
Commerce without the whining.

According to TechCrunch, the hot trend in commercial startups is the “Labs” concept. Churn Labs, Tasty Labs, Nest Labs, Foursquare Labs, Dogpatch Labs. What’s it all about? You could read the TechCrunch article to find out, or we could just reason it out together.

Labs are for research. Commercial startups have presumably been created to produce products. So clearly, the Labs trend is about selling your research as a product. That’s not such a crazy idea, and I’ll tell you why.

Early adopters.

Early adopters are the best. They jump in fearlessly even after you warn them that your product probably doesn’t actually work. Early adopters are willing to pay a premium for a version of the product that inevitably will be shown to lack key features, run slowly, crash often, and generally look like crap compared to the improved version you’ll release three months later.

This premium is known as the early adopter tax. Early adopters will stand in line for the right to pay the early adopter tax for your unpolished product, and you might wonder why. Marketing geniuses have an answer, or several answers: they’ll tell you that early adopters want the prestige or that they are doing product research or that they really, really need what you’re promising. Maybe. All you need to know is that this is what early adopters do.

An early adopter ate the first oyster. Another early adopter ate the first death cap mushroom. Early adopters will eat your dogfood and tell you how it tastes. And you can use that feedback to improve the product, assuming that you want to do that. But why would you?

Those same marketing geniuses will tell you that you should use the feedback from early adopters to make your product more palatable to normal customers. There’s a whole body of research on this. There’s this chasm, see. The chasm between early adopters and normal customers. It’s really, really hard to cross that chasm, because normal customers are a completely different species.

The theory is that to cross the chasm, you need to get your early adopters to pitch the product to the normals. Not just mention, but actively promote. How you get them to do that is an interesting question. To some people. To me, an interesting question is, why would you want them to?

Think about it: You improve the product, get some bugs out, improve the user experience, make it more palatable to the unwashed, somehow get those early adopters to become your unpaid marketing department, and what are you accomplishing? You’re just trolling for normal customers, with all the headaches they bring.

Normal customers are whiners. Normal customers give you no points for effort or good intentions. Normal customers don’t even love technology. They think of it as a tool for getting things done. Or worse, they think technology is scary. InfoWorld has them figured out. I looked at 74 InfoWorld stories recently and 28 of them were scary stories about data security. Do you really want to have to care that much about data security? Normal users will force you to, because they are whiny wimps.

It would be great if you could just avoid those complainers altogether. Stick with the pioneers. Apple has done pretty well, and by their own admission they are targeting the crazy ones. That’s who you want, the ones who appreciate you for the concept artist that you are.

It may be a pipe dream, but it sure seems like the smart thing would be to build a business in which all your customers are early adopters. Forever. Software-product nirvana: build it, launch it, kill it. Skip the stage where you have to support it. That’s my brilliant idea.

Or did I just describe Google’s business model?

John Shade was born under a cloud in Montreux, Switzerland, in 1962. Subsequent internment in a series of obscure institutions of ostensibly higher learning did nothing to brighten his outlook. He has never sold a product, or even research packaged as a product. Send the author your feedback or discuss the article in the magazine forum.

But Wait, There’s More...

Coming Attractions

This may be the last page, but that doesn’t mean it’s the end of the conversation. Keep up with what’s going on in the Pragmatic universe by subscribing to our newsletter and drop by the Pub again next month for more Pragmatism. We’ll keep a seat warm and a mug chilled and a candle burning in the window.

Coming Soon in PragPub

We’re looking at articles on topics ranging from agile practices to MacRuby from the likes of Jeff Langr, Tim Ottinger, and Jonathan Penn. We’re still working on the interview with David Hussman, which will be worth the wait. And Venkat will dig even deeper into Scala.

Coming Soon on the Bookshelf

New in beta: Seven Databases in Seven Weeks. Data is getting bigger and more complex by the day, and so are your choices in handling it. From traditional RDBMS to newer NoSQL approaches, you’ll get a handle on the hottest open source databases today.

Right. But to really be in the know, you need to subscribe to our weekly newsletter. It’ll keep you in the loop, it’s a fun read, and it’s free. All you need to do is create an account on pragprog.com (email address and password is all it takes) and select the checkbox to receive newsletters.

While you’re waiting for the next issue of the newsletter or of PragPub to come out, you can follow us on twitter at @pragpub, @PragmaticAndy, or @pragdave. Or on Facebook at facebook.com/PragmaticBookshelf. Or email us at michael@pragprog.com.