PragPub 2012-06: Issue #36

Table of Contents

Features

The Beauty of Concurrency in Go

by Alexander Demin

It's good to learn a new language every so often, but you have to get beyond "hello World."

Moneyball and the Science of Building Great Teams

by Peter Varhol

It's not the unsolved problems that trip you up, it's the ones you solved too easily.

Scala for the Intrigued

by Venkat Subramaniam

Scala traits remove duplication of code and provide a cleaner design.

Retrospective Velocity

by Brian Tarbox, and Heather Mardis

Looking in the rear-view mirror to see how fast you're moving.

Departments

Up Front

by Michael Swaine

The Go and Scala languages and insights for software teams.

Choice Bits

Twitter is like a zen garden. Admire the pretty stones, but don’t swallow them.

Meet the Team

Meet John Osborn, Development Editor.

Calendar

Author sightings, upcoming conferences, and other events of note.

Shady Illuminations

by John Shade

If we’re in the post-PC Era, shouldn’t we also be post-Windows?

But Wait, There’s More...

Coming attractions and where to go from here.

Except where otherwise indicated, entire contents copyright © 2012 The Pragmatic Programmers.

Feel free to distribute this magazine (in whole, and for free) to anyone you want. However, you may not sell this magazine or its content, nor extract and use more than a paragraph of content in some other publication without our permission.

Published monthly in PDF, mobi, and epub formats by The Pragmatic Programmers, LLC, Dallas, TX, and Raleigh, NC. E-Mail support@pragprog.com, phone +1-800-699-7764. The editor is Michael Swaine (mailto:michael@pragprog.com). Visit us at http://pragprog.com

ISSN: 1948-3562

Up Front

Go. Team.

by Michael Swaine

Advice is easy; examples are hard.

You should learn a new language every year. That’s good advice. But how, exactly, do you go about learning a new language?

Do you dive right into the deep end of the pool, taking on a commercial assignment with a serious deadline and tackle it in a language you’ve never used?

Do you read the manual and write some simple “Hello World” programs to try to get the feel for the language and then conclude that you’ve “learned” it? Or, more honestly, that you haven’t?

Alexander Demin loves learning new languages, and he was eager to dive into Google’s Go language, but he wanted a real problem that would take him deeply into the character of the language, yet be possible to write in a reasonable amount of time.

His real problem is something he calls TCP/IP Proxy, and he shares it with you in this issue of PragPub. When you’ve worked through it, you’ll have a good sense of how Go handles strings, file and network I/O, and concurrency.

We use the word “team” when speaking of developers working together on a project, and the parallels with sports teams are real. Peter Varhol points out one more way in which software teams are like sports teams in his article on Moneyball and software teams. He argues that the kinds of insights Oakland Athletics General Manager Billy Beane brought to baseball can improve the efficiency of software teams.

We have a team here at The Pragmatic Programmers, and in this issue you’ll meet another member of that team in the latest installment of our series of Staff Profiles. Venkat Subramaniam has a new installment of his series on the Scala programming language, too. And Brian Tarbox and Heather Mardis offer some thoughts on retrospective velocity that your team may find helpful.

Choice Bits

Tweeted Tics and Trips

Twitter is like a zen garden. Admire the pretty stones, but don’t swallow them.

What’s Hot

Top-Ten lists are passé—ours goes to 11. These are the top titles that folks are interested in currently, along with their rank from last month. This is based solely on direct sales from our online store.

1^ | 7 | Seven Databases in Seven Weeks
2v | 1 | Practical Vim
3^ | 4 | Agile Web Development with Rails
4^ | NEW | Working with Unix Processes
5^ | 8 | Programming Ruby 1.9
6v | 2 | The Rails View
7v | 3 | Rails Recipes
8^ | NEW | The RSpec Book
9 | 9 | The Cucumber Book
10v | 6 | tmux
11^ | NEW | Seven Languages in Seven Weeks

All in a Day’s Work

Just automated all the admin for our DNS: 20+ domains in Route53 via a Ruby DSL and cli53. Love it. — @pragdave

We’re sitting at our neighborhood bar, having a beer (thomas) and a diet coke (me) and trying to name our office. — @amyhoy

"Who’s _really_ working?" - @Brilliantcrank, trying to round people up to go get donuts. — @ryanirelan

Six dumb people blocked this morning. My work is done. — @shelly

Deep Thoughts

Only @richhickey could show us two lines of code and declare “It does way too much!” #hbb — @mtnygard

@gdarklighter I’m saying basic coding skills are worth it if you use computers, the way basic plumbing helps when you live in a house. — @ginatrapani

Why don’t HTML guides that say "deprecated, use styles" tell you what the <...style="foo"...> equivalent would be? #reader_considerate — @marick

American Mcdonalds coffee is not nearly as tasty as Canadian. — @adamgoucher

Life on the Road

Hey, looks like I’ll be speaking at @WindyCityRails this year. Looks like it’ll be a lot of fun with a great speaker lineup, and also me. — @noelrap

Hello June: Bergen, Oslo (#ndc), Mumbai (#coderetreat), Sangli, Denver (#uberconf), London, Poland. — @venkat_s

I think I envy people who can eat when they are flying. — @adamgoucher

Is it me or is United’s latest site updates something from 1998, just with rounded corners? — @jeffcohen

I was just patted down by #TSA Officer Stiffarm. (his real name) #theJokesJustWriteThemselves — @scottdavis99

Weird! Walked through downtown Austin and no one handed me sunglasses, bottled water, or an invite for free beer in a tent. #sxswisover — @shelly

It’s a Lifestyle Thing

@eileentv @jeffjarvis ha! Loved the homage to the girl who fell in the mall fountain while texting! — @ginatrapani

Hacking the ActionDispatch gem on a Friday night. Yes, I’m an introverted geek. — @jeffcohen

Living in SF there are so many cool nerdy events it can be really hard to choose. — @stubbornella

Even my cat is suspicious of me. Can’t understand why. All I’ve done for the last 8 hours is snore on the couch. — @mollydotcom

We Boffos (boring old farts) of software process should *manage* our decline rather than deny it. — @marick

All of the @shellys on Twitter who are not me lead such interesting lives. Most of their friends are illiterate, however. #misdirectedtweets — @shelly

Twitterville Is a Helpful Place

Question: Do liquor stores deliver? I assume there are legal reasons they can’t. But that should be a thing. — @bphogan

@bphogan There are definitely liquor stores that deliver here in Boston (despite otherwise crazy liquor laws). — @trevorburnham

@bphogan They do in NY. — @david_a_black

@bphogan they have drive-thru daiquiri stands in New Orleans. I miss those. — @codefinger

@bphogan When I lived in LA, pink dot delivered booze. I think http://yummy.com does in chicago. — @neovintage

@bphogan I’ve seen drive through liquor stores. — @jeffklawiter

@bphogan depending on your state laws, you can buy liquor online... — @ryanbriones

The Right Tool for the Job

Just got a Rigol DG4062 waveform generator. I didn’t have high hopes for it being good, but it turned out to be really nice. — @jeriellsworth

Found Post-it tape rolls in local stationery store today. Not sure where to use them, but their sticky reuseness is impressive. — @staffannoteberg

My new favorite prototyping material is Premo Sculpey thermally setting plastic clay. — @jeriellsworth

Final Thoughts

The only goal of zomblatt is purity of process. And effectiveness. Those two, and clarity. And maybe malleability. — @tottinge

MT @dietsch New MacBooks with OS X Housecat will only remain in your lap when you’re on the toilet, sitting at dinner, or tying your shoes. — @shelly

"Should we integrate with salesforce?" "Maybe, but then we’d attract salespeople as customers, and we don’t really want that." "RIGHT." — @amyhoy

In the past 12 months, 28 people have written feature articles for http://pragprog.com/magazines: 100% male. #canthelpnoticing — @gvwilson

Who Are Those Guys?

They’re not all guys. Yes, we may have a little trouble attracting women to write for the magazine, but the people whose tweets we follow are not all guys. Also, we have to confess that we cleaned up their punctuation and stuff a little. OK, who they are: David A. Black, Shelly Brisbin, Trevor Burnham, Ryan Carmelo Briones, Jeff Cohen, Scott Davis, Jeri Ellsworth, Adam Goucher, Brian P. Hogan, Molly E. Holzschlag, Amy Hoy, Ryan Irelan, Jeff Klawiter, Joe Kutner, Brian Marick, Staffan Nöteberg, Michael Nygard, Tim Ottinger, Noel Rappin, Rimas Silkaitis, Venkat Subramaniam, Nicole Sullivan, Dave Thomas, Gina Trapani, and Greg Wilson. You can follow us at www.twitter.com/pragpub.

Meet the Team

John Osborn

Each month we profile a member of the Pragmatic Bookshelf team. This month it’s Development Editor John Osborn.

John Osborn, Development Editor

I never planned to be an editor; I thought I was going to be a prize-winning reporter, an aspiration of which I was reminded at a recent reunion when a friend pointed to me across a crowded banquet hall and shouted, “Hey, there’s the guy who said we should watch for his by-line in the New York Times.”

So much for that.

Throughout my more than forty years of professional life, I have attended to three loves: writing, teaching, and technology. Out of college, I carried coffee and copy for reporters in the Washington Bureau of the New York Times and covered the Florida State legislature for United Press.

Later, I taught physics and chemistry to high school students in Africa and Maine. I built one of the first Altair computers for my students soon after the kits were announced in Popular Electronics magazine, while devouring news of the microcomputer revolution in magazines edited by the Prag’s own Michael Swaine.

At Digital Equipment Corporation, I wrote the first VAX/VMS self-paced training course and later managed Digital Press, the company’s in-house publisher (later sold to Elsevier). I acquired and published computer books for IDG Books (later bought by Wiley) and for O’Reilly Media (still very much independent, and the distributor of The Pragmatic Programmer Bookshelf).

A list of the titles I have published over the years reads like a history of modern enterprise computing, including Common Lisp, by Guy Steele; VAX/VMS Internals, by Ruth Goldenberg and Larry Kenah; XML Bible, by Elliot Rusty Harold (a Publisher’s Weekly bestselling computer book); X Windows System, by Robert Scheiffler, Programming C#, by Jesse Liberty, and Programming Windows Communications Foundation (WCF), by Juval Lowy.

Now, I’m helping a new generation of aspiring computer book authors compose their thoughts for the Prags. I have to credit Susannah Pfalzer, Pragmatic’s energetic Managing Editor, for introducing me to Andy Hunt and Dave Thomas a year ago. She and I had worked together at IDG Books, and shortly after I decided to go freelance as a developmental editor, she called. It took little persuasion to sign me up.

What I like best about working for the Prags is the deep respect the founders and my colleagues have for good writing. It’s not enough that a book be cutting edge or technically accurate; it also has to tell a good story and engage readers. I have never seen so much time spent on getting a book right. At the Prags, editors are collaborators not gatekeepers.

But there is life outside publishing. When I’m not sitting in the town library with a manuscript and a red pen or wrestling with PML at my homebrew Windows workstation, you’ll find me outdoors with my family, on a bike, in a kayak, on skis, or sweating on a steep White Mountain trail in New Hampshire.

As for that reporting thing? Well, if you’re in Harvard, Massachusetts, pick up a copy of the Harvard Press. You’ll find my by-line there.

Email: john@pragprog.com

Twitter: @johnatlarge

Web: OsbornEditorial.com (under construction)

LinkedIn: John Osborn

The Beauty of Concurrency in Go

Beyond "Hello World"

by Alexander Demin

It's good to learn a new language every so often, but you have to get beyond "hello World."

It's good for you to learn a new programming language from time to time. This is true even if the language doesn't take off or is ancient. Tackling old problems in a new language pushes you to rethink your current views, approaches, and habits.

I love trying new stuff, especially programming languages. But after implementing "Hello, world!" or the Fibonacci sequence in a new language, you usually feel almost nothing, get no taste whatsoever. You could try implementing the Sieve of Eratosthenes to explore, a little, data structures and maybe performance. But I wanted something real, something I could maybe even reuse afterwards. So some time ago I invented for myself a problem that helps me to get the feel of a language in just a few hundred lines of code.

The problem involves several very important elements of a language: strings, file and network I/O, and, of course, concurrency. The problem is called TCP/IP proxy (or you could call it a network debugger). The idea is, you have a TCP/IP listener (single- or multi-threaded) accepting a connection on a given port. When it receives an incoming connection, it has to connect to another host and pass through the data in both directions between the caller and the remote host. Additionally, the proxy can log the traffic in various formats to help in analyzing the data.

I stopped counting the occasions when I needed this kind of tool. Any time network programming is involved, such a tool is essential. I have implemented it many times in my life in different languages: C, C++, Perl, PHP. The two latest implementations were in Python and Erlang. It represents the kind of real problem I was looking for.

We can specify more concrete requirements. The application must serve multiple connections simultaneously. For each connection, it needs to log data in three ways: a dump presenting data sequentially in the both directions in the form of a hexadecimal dump, and two binary logs with incoming and outgoing data streams in separate binary files.

We're going to implement the program in this article, and the language we're going to use is Go. The Go authors claim that they designed the language with concurrency and multi-threading in its blood stream. I intend to take them at their word.

If I developed such an application in boostified C++, I would probably go for the main listener thread, plus threads for each connection. Hence, an individual connection would be fully served (I/O and logging) by a single thread.

Here are the threads I'll use to serve each connection in the Go implementation:

a bi-directional hex dumper thread

two threads logging incoming and outgoing streams in the binary form

two threads passing through data from the local host to the remote and vice versa

In total: 5 threads.

Again, five threads are serving each individual connection. I implemented all these threads not for the sake of multi-threading per se, but because Go encourages multi-threading, while C++ discourages it (even with the new C++x11 standard's steroids). Multi-threading in Go is natural and simple. My implementation of the TCP/IP proxy in Go doesn't use mutexes and conditional variables. Synchronization is elegantly managed by Go's channels.

Okay, here's the source, with explanations. If you are not familiar with Go, the commentary should help. My intention was to focus not just on the functionality of the program, but also on the Go language itself.

Let's Go

In lines 2-11 we declare the packages we are going to use. Notably, if a package is included but not used, Go treats this as an error and enforces removing unused declarations (remember when you gave up last time and didn't bother to clean up the list STL includes in your C++ project?)

| 1 package main
| 2 import (
| 3 "encoding/hex"
| 4 "flag"
| 5 "fmt"
| 6 "net"
| 7 "os"
| 8 "runtime"
| 9 "strings"
| 10 "time"
| 11 )

In lines 12-16 we declare global variables presenting the command line flags. Further down we will see how to parse them.

| 12 var (
| 13 host *string = flag.String("host", "",
| "target host or address")
| 14 port *string = flag.String("port", "0", "target port")
| 15 listen_port *string = flag.String("listen_port", "0",
| "listen port")
| 16 )

In lines 17-20 we see the syntax of variadic function arguments in Go.

| 17 func die(format string, v ...interface{}) {
| 18 os.Stderr.WriteString(fmt.Sprintf(format+"\n", v...))
| 19 os.Exit(1)
| 20 }

In lines 21-28 there are two functions launching the hex dump and the binary loggers. The only difference is in the log name.

| 21 func connection_logger(data chan []byte, conn_n int,
| local_info, remote_info string) {
| 22 log_name := fmt.Sprintf("log-%s-%04d-%s-%s.log",
| format_time(time.Now()), conn_n, local_info, remote_info)
| 23 logger_loop(data, log_name)
| 24 }
| 25 func binary_logger(data chan []byte, conn_n int, peer string) {
| 26 log_name := fmt.Sprintf("log-binary-%s-%04d-%s.log",
| format_time(time.Now()), conn_n, peer)
| 27 logger_loop(data, log_name)
| 28 }

In lines 29-43 the real Go fun begins. The function logger_loop creates a log file and then begins spinning in the infinite loop (lines 35-42). In line 36 the code waits for a message from the channel data. There is an interesting trick in line 34. The operator defer allows us to define a block of code guaranteed be executed at the end of the function scope (similar to finally in Java). If empty data is received, the function exits.

| 29 func logger_loop(data chan []byte, log_name string) {
| 30 f, err := os.Create(log_name)
| 31 if err != nil {
| 32 die("Unable to create file %s, %v\n", log_name, err)
| 33 }
| 34 defer f.Close()
| 35 for {
| 36 b := <-data
| 37 if len(b) == 0 {
| 38 break
| 39 }
| 40 f.Write(b)
| 41 f.Sync()
| 42 }
| 43 }
| 44 func format_time(t time.Time) string {
| 45 return t.Format("2006.01.02-15.04.05")
| 46 }
| 47 func printable_addr(a net.Addr) string {
| 48 return strings.Replace(a.String(), ":", "-", -1)
| 49 }
| 50 type Channel struct {
| 51 from, to net.Conn
| 52 logger, binary_logger chan []byte
| 53 ack chan bool
| 54 }

In lines 55-88 there is a function that reads data from the source socket from, writes to the log, and sends it to the destination socket to. For each connection there are two instances of the pass_through function copying data between the local and remote sockets in opposite directions. When an I/O error occurs, it is treated as a disconnect. Finally, in line 79 this function sends the acknowledgment back to the main thread, signalling its termination.

| 55 func pass_through(c *Channel) {
| 56 from_peer := printable_addr(c.from.LocalAddr())
| 57 to_peer := printable_addr(c.to.LocalAddr())
| 58 b := make([]byte, 10240)
| 59 offset := 0
| 60 packet_n := 0
| 61 for {
| 62 n, err := c.from.Read(b)
| 63 if err != nil {
| 64 c.logger <- []byte(fmt.Sprintf("Disconnected from %s\n",
| from_peer))
| 65 break
| 66 }
| 67 if n > 0 {
| 68 c.logger <- []byte(fmt.Sprintf("Received (#%d, %08X)
| %d bytes from %s\n",
| packet_n, offset, n, from_peer))
| 69 c.logger <- []byte(hex.Dump(b[:n]))
| 70 c.binary_logger <- b[:n]
| 71 c.to.Write(b[:n])
| 72 c.logger <- []byte(fmt.Sprintf("Sent (#%d) to %s\n",
| packet_n, to_peer))
| 73 offset += n
| 74 packet_n += 1
| 75 }
| 76 }
| 77 c.from.Close()
| 78 c.to.Close()
| 79 c.ack <- true
| 80 }

In lines 81-107 there is a function processing the entire connection. It connects to the remote socket (line 82), measures the duration of the connection (lines 88, 101-103), launches the loggers (lines 93-95) and finally launches two data transferring threads (lines 97-98). The pass_through functions run until both peers are active. In lines 99-100 we wait for acknowledgments from the data transferring threads. In lines 104-106 we terminate the loggers.

| 81 func process_connection(local net.Conn, conn_n int, target string) {
| 82 remote, err := net.Dial("tcp", target)
| 83 if err != nil {
| 84 fmt.Printf("Unable to connect to %s, %v\n", target, err)
| 85 }
| 86 local_info := printable_addr(remote.LocalAddr())
| 87 remote_info := printable_addr(remote.RemoteAddr())
| 88 started := time.Now()
| 89 logger := make(chan []byte)
| 90 from_logger := make(chan []byte)
| 91 to_logger := make(chan []byte)
| 92 ack := make(chan bool)
| 93 go connection_logger(logger, conn_n, local_info, remote_info)
| 94 go binary_logger(from_logger, conn_n, local_info)
| 95 go binary_logger(to_logger, conn_n, remote_info)
| 96 logger <- []byte(fmt.Sprintf("Connected to %s at %s\n",
| target, format_time(started)))
| 97 go pass_through(&Channel{remote, local, logger, to_logger, ack})
| 98 go pass_through(&Channel{local, remote, logger, from_logger, ack})
| 99 <-ack // Make sure that the both copiers gracefully finish.
| 100 <-ack //
| 101 finished := time.Now()
| 102 duration := finished.Sub(started)
| 103 logger <- []byte(fmt.Sprintf("Finished at %s, duration %s\n",
| format_time(started), duration.String()))
| 104 logger <- []byte{} // Stop logger
| 105 from_logger <- []byte{} // Stop "from" binary logger
| 106 to_logger <- []byte{} // Stop "to" binary logger
| 107 }

In lines 108-132 is the main function running the TCP/IP listener. In line 109 we ask the Go runtime to use all physically available CPUs.

| 108 func main() {
| 109 runtime.GOMAXPROCS(runtime.NumCPU())
| 110 flag.Parse()
| 111 if flag.NFlag() != 3 {
| 112 fmt.Printf("usage: gotcpspy -host target_host -port target_port
| -listen_post=local_port\n")
| 113 flag.PrintDefaults()
| 114 os.Exit(1)
| 115 }
| 116 target := net.JoinHostPort(*host, *port)
| 117 fmt.Printf("Start listening on port %s and
| forwarding data to %s\n",
| *listen_port, target)
| 118 ln, err := net.Listen("tcp", ":"+*listen_port)
| 119 if err != nil {
| 120 fmt.Printf("Unable to start listener, %v\n", err)
| 121 os.Exit(1)
| 122 }
| 123 conn_n := 1
| 124 for {
| 125 if conn, err := ln.Accept(); err == nil {
| 126 go process_connection(conn, conn_n, target)
| 127 conn_n += 1
| 128 } else {
| 129 fmt.Printf("Accept failed, %v\n", err)
| 130 }
| 131 }
| 132 }

This is it, just 132 lines. Please note: we used only the standard libraries, coming out of the box.

Now we are ready to run:

go run gotcpspy.go -host pop.yandex.ru -port 110 -local_port 8080

It should print:

Start listening on port 8080 and forwarding data to pop.yandex.ru:110

Then you can run in another window:

telnet localhost 8080

and enter, for instance, USER test [ENTER] and PASS none [ENTER]. The three log files will be created (the time stamp, of course, could be different in your case).

Bi-directional hex dump log log-2012.04.20-19.55.17-0001-192.168.1.41 -49544-213.180.204.37-110.log:

| Connected to pop.yandex.ru:110 at 2012.04.20-19.55.17
| Received (#0, 00000000) 38 bytes from 192.168.1.41-49544
| 00000000 2b 4f 4b 20 50 4f 50 20 59 61 21 20 76 31 2e 30
| |+OK POP Ya! v1.0|
| 00000010 2e 30 6e 61 40 32 36 20 48 74 6a 4a 69 74 63 50
| |.0na@26 HtjJitcP|
| 00000020 52 75 51 31 0d 0a
| |RuQ1..|
| Sent (#0) to [--1]-8080
| Received (#0, 00000000) 11 bytes from [--1]-8080
| 00000000 55 53 45 52 20 74 65 73 74 0d 0a
| |USER test..|
| Sent (#0) to 192.168.1.41-49544
| Received (#1, 00000026) 23 bytes from 192.168.1.41-49544
| 00000000 2b 4f 4b 20 70 61 73 73 77 6f 72 64 2c 20 70 6c
| |+OK password, pl|
| 00000010 65 61 73 65 2e 0d 0a
| |ease...|
| Sent (#1) to [--1]-8080
| Received (#1, 0000000B) 11 bytes from [--1]-8080
| 00000000 50 41 53 53 20 6e 6f 6e 65 0d 0a
| |PASS none..|
| Sent (#1) to 192.168.1.41-49544
| Received (#2, 0000003D) 72 bytes from 192.168.1.41-49544
| 00000000 2d 45 52 52 20 5b 41 55 54 48 5d 20 6c 6f 67 69
| |-ERR [AUTH] logi|
| 00000010 6e 20 66 61 69 6c 75 72 65 20 6f 72 20 50 4f 50
| |n failure or POP|
| 00000020 33 20 64 69 73 61 62 6c 65 64 2c 20 74 72 79 20
| |3 disabled, try |
| 00000030 6c 61 74 65 72 2e 20 73 63 3d 48 74 6a 4a 69 74
| |later. sc=HtjJit|
| 00000040 63 50 52 75 51 31 0d 0a
| |cPRuQ1..|
| Sent (#2) to [--1]-8080
| Disconnected from 192.168.1.41-49544
| Disconnected from [--1]-8080
| Finished at 2012.04.20-19.55.17, duration 5.253979s

Binary log of outgoing data log-binary-2012.04.20-19.55.17-0001 -192.168.1.41-49544.log:

| USER test
| PASS none

Binary log of incoming data log-binary-2012.04.20-19.55.17 -0001-213.180.204.37-110.log:

| +OK POP Ya! v1.0.0na@26 HtjJitcPRuQ1
| +OK password, please.
| -ERR [AUTH] login failure or POP3 disabled, try later. sc=HtjJitcPRuQ1

It seems to work, so let's try to measure the performance by downloading a bigger binary file directly and then via our proxy.

Downloading directly (file size is about 72MB):

| time wget http://www.erlang.org/download/otp_src_R15B01.tar.gz
| ...
| Saving to: `otp_src_R15B01.tar.gz'
| ...
| real 1m2.819s

Now let's start the proxy and then download through it:

go run gotcpspy.go -host=www.erlang.org -port=80 -listen_port=8080

Downloading:

| time wget http://localhost:8080/download/otp_src_R15B01.tar.gz
| ...
| Saving to: `otp_src_R15B01.tar.gz.1'
| ...
| real 0m56.209s

Let's compare the results.

diff otp_src_R15B01.tar.gz otp_src_R15B01.tar.gz.1

It matches, which means the program works correctly.

Now the performance. I repeated the experiment a few times on my Mac Air. Surprisingly, downloading via the proxy worked for me even a bit faster than directly. In the example above: 1m2819s (directly) vs 0m.56209s (via proxy). The only explanation I can imagine is that wget is single-threaded, and it multiplexes incoming and outgoing streams in one thread. In turn, the proxy processes the streams in the individual threads, and perhaps this causes a tiny speedup. But the difference is small, almost negligible, and maybe on another computer or network it would disappear completely. The main observation is: downloading via proxy doesn't slow things down, despite the additional overhead of creating quite massive logs.

In summary, I'd like you to look at this program from the angle of simplicity and clarity. I've pointed it out above but I'd like to underline it again: I had started using threads in this application gradually. The nature of the problem gently pushed me to identify concurrent activities in processing a connection, and then the ease and safety of concurrency mechanisms in Go had finished it off, and eventually I used concurrency without thinking about the efficiency vs complexity (and difficulty to debug) trade-off.

Agreed, sometimes a problem simply needs to thrash bits and bytes, and the linear efficiency of the code is the only thing you care about. But more and more you encounter problems where the capability of concurrent, multi-threaded processing becomes the key factor, and for this kind of application, Go will shine.

I hope this serves for you as a representative example showing off the ease and even beauty of concurrency in Go.

Alexander Demin is a software engineer and Ph.D. in Computer Science. Constantly exploring new technologies, he believes that something amazing is always out there. He can be contacted at alexander@demin.ws or through his homepage and blog.

Send the author your feedback or discuss the article in the magazine forum.

Moneyball and the Science of Building Great Teams

Winning the Software Development Game

by Peter Varhol

It's not the unsolved problems that trip you up, it's the ones you solved too easily.

Moneyball is a book and movie about baseball: about fielding, pitching, and especially scoring. But it's also about building great teams, and about recognizing systemic errors in human thought processes. That's where it has relevance for software development. Understanding these errors and how to overcome them can help you build better software teams, and manage them more effectively.

Moneyball came into being because the Oakland Athletics major league baseball team had a big problem. It had a payroll that was a tenth the size of the very best teams in the league, and it had to field a winning team in order to remain profitable in its small market. Its general manager, Billy Beane, started looking deeply into the characteristics that produced winning teams, and came to a surprising conclusion.

Beane discovered that the baseball experts were all wrong. The prototypical baseball player simply didn't have a lot of impact on the ability of a team to win. Instead, Beane embarked on a detailed examination of characteristics that were highly correlated with winning games.

In short, he found a set of systemic errors in the way that baseball experts evaluated and used talent. The experts evaluated individual players by projecting their own opinions on the value of individual abilities to team goals, rather than by looking at characteristics that directly related to wins.

We can apply this to software development and testing teams. To do so, we have to look at problems the way Billy Beane did, and delve more deeply into the realm of human error. The best place to start is with Daniel Kahneman's new book, Thinking, Fast and Slow. Kahneman describes two approaches to human thought and decision-making, which he labels System 1 and System 2.

System 1 is immediate, reflexive thinking that is for the most part unconscious. We do this sort of thinking many times a day. It keeps us functioning in an environment where we have numerous external stimuli, some important, and many not.

But System 1 thinking is sometimes fooled. Consider this question: A baseball and bat together cost $1.10. The bat cost $1.00 more than the ball. What does the ball cost? If you didn't think before responding, you likely said ten cents, which is incorrect. That's System 1 talking.

System 2 is more deliberate thought. It is engaged for more complex problems, those that require mental effort to evaluate and solve. System 2 makes more accurate evaluations and decisions, but it can't respond instantly, as is required for many types of day-to-day decisions. And it takes effort, which means it can tire out team members.

We need both systems. Most of our normal reactions in life are governed by System 1. Our senses pick up a cue, and we respond to it in a familiar way. We do System 1 thinking "without thinking." But then when we are presented with a new problem, or have to deal with an unfamiliar situation, we have to think it through. That is where System 2 kicks in action. The two modes of thinking work together to enable us to function across a wide range of sensory inputs and subsequent decisions.

But this very natural way of thinking can result in systemic errors that cause us to make mistakes on software projects. Those mistakes are related to our very expertise on such projects. By relying on our beliefs based on past practices, we are sometimes right, but often wrong. And that hurts our decision-making throughout the project.

How We Make Mistakes in Thinking

Most errors in decision-making are the result of fallacious System 1 thinking. There are a number of ways of classifying reflexive, or System 1, errors in thinking. These include:

Priming

Heuristics

Anchoring

Regression to the mean

Priming is probably the oddest of these errors. It codifies the fact that any information presented prior to the problem, even if it is completely unrelated to the problem, influences the decision. For example, Kahneman set up an experiment where a "wheel of fortune" stopped at either 30 or 65. It turned out that the number people obtained on that wheel influenced their answers to a follow-on question, which had nothing to do with the wheel of fortune numbers.

Anchoring is related to priming, in that a previous value affects a subsequent decision. But in this case, the previous number is suggested as a possible direction for the ensuing decision. I might ask the question "Was John Kennedy older or younger than 40 when he was killed?" Whatever your answer, you are now anchored to age 40 if subsequently asked how old he was when he died.

All of us use heuristics, or rules of thumb, in our decision-making. Heuristics enable us to categorize situations and make fast, pre-conceived decisions, and are useful and valued. They are formed through as few as one prior experience with a similar situation. That's quick learning, and there lies the danger.

Because heuristics can fool us. Years ago when I was training to be a pilot, my instructor put me "under the hood" in practicing recovery from unusual attitudes. With the hood down, he put the plane into an unusual flying position from which I had to recover as quickly as possible. When he brought the hood up, I could see only the instrument panel. I rapidly developed a heuristic that enabled me to quickly identify and correct an unusual attitude. Quick response without any thought.

I continued to use my heuristic in my next lesson, but my instructor had figured out what I was doing. Now things weren't working out as I expected. My turn indicator and artificial horizon were centered, but I was still losing over 1000 feet a minute! I was stumped. My instructor had "crossed" the controls, leaving me in a slip that my heuristic couldn't account for. Now my heuristic was working against me. I was worse than wrong; I couldn't follow through with any critical thought at all once my heuristic failed.

Regression to the mean is a particularly insidious flaw in our thinking. In a general sense, regression to the mean denotes that extreme values are highly uncommon, so that subsequent values are more likely to trend back to a more likely value. We might praise an exceptionally good performance, then be dismayed when subsequent attempts aren't as good. Likewise, criticizing a poor performance will likely result in a better performance the next time, not because of the criticism, but because the poor performance had more to do with bad luck than it did poor ability or a lack of trying. We fallaciously conclude that praise makes us complacent and criticism makes us try harder.

Regression to the mean is not itself an error or a mode of thinking; it is a statistical phenomenon. It simply notes that long-tail events in a probability distribution are rare. If we happen to observe one, then random fluctuation is likely to make subsequent observations closer to the average. The System 1 error in thinking comes in the form of an inability to grasp the fact of regression to the mean.

Regression to the mean, in practical experience, means that performance is made up of ability and luck. It's possible, though unusual, for a person to have exceptional ability. And exceptional ability usually leads to exceptional performance. In all likelihood, though, any particular exceptionally good performance is the result of competent ability plus very good luck in this specific instance. While we know this intellectually, it is very difficult to realize it in practice.

Back to Software Teams

What does human error have to do with software project teams? It turns out that we make all these kinds of errors in planning and executing our application projects. And this leads to familiar problems.

Having preconceived notions about our project success is a particular problem, similar to what we know as the Pygmalion effect, or self-fulfilling prophecy. Our expectations can ultimately influence the final result, even if we aren't aware of the influence, or even swear that it doesn't exist.

Even worse than self-fulfilling prophecies is the bandwagon, where most people on the project know better, but subscribe to the views of a minority in order to go along with the crowd. It is clear that there is belief and buy-in among some segment of the users, but the belief doesn't extend to the entire team.

When evaluating project data, we recognize that there are data and performance points that are outside what we expect, but rather than attributing them to luck, good or bad, we usually attempt to explain them. This results in our drawing conclusions that are both inaccurate and misleading.

How do we overcome these biases? We have to bring System 2 thinking into play, and that means we have to be cognizant of the biases. Reminding ourselves that exceptional performances, either good or bad, are probably based largely on luck helps us maintain our perspective in the face of extreme results.

Another way to overcome our thinking is to understand how we can make decisions on a project for peak efficiency. Many of our biases come from System 1 thinking, where we react and respond immediately, and where prior events play a significant role in our decisions. But as managers, our decisions impact people beyond ourselves. When our decisions affect others, they often need to be questioned and thought through.

So we should strive to engage System 2 thinking more frequently in projects. Of course there is a downside to this. System 2 thinking is hard work; too much of it causes us to mentally tire and make mistakes. System 1 thinking has value in many situations, because of its efficiency and relative accuracy.

Mental mistakes aren't the sort of things that are at the top of our mind when we assess our software projects, and make decisions concerning direction, resource allocation, or priorities. Many of these decisions are made with System 1 thinking, and are often both efficient and correct.

But such decisions are often wrong, too. This seems to present a conundrum. If we fear the errors of System 1 thinking, and System 2 thinking is laborious and time-consuming, what is the solution?

One way to overcome some of the errors I've mentioned is to pause to think through the implications of an immediate decision. If the decision comes automatically, and influences team direction or use of resources, a minute to examine it isn't a lot to ask. Another method is to assign one or more trusted team members to rethink all of your impulse decisions. Feedback from others can be an important check on unthinking responses to daily decisions.

Thinking errors are critical to the outcome of software development projects. Managers, leads, and team contributors all look at information on the project and make decisions daily. By understanding how we evaluate these decisions, how our own thought processes work, and where we are likely to make thinking errors, we can train ourselves and our team to make better decisions regularly.

I have developed a presentation around this topic that I would be happy to share. Just email me at peter@petervarhol.com, tell me something about what you do and what kind of software projects you're involved in, and I'll send it to you.

Peter Varhol is a well-known writer and speaker on software development and testing topics. He has over 20 years experience in software and software development, and has been a college professor, software product manager, and technology journalist. He is currently tools evangelist at Seapine Software, and a consultant to lifecycle tools vendors and users.

Send the author your feedback or discuss the article in the magazine forum.

Scala for the Intrigued

Using Traits

by Venkat Subramaniam

Scala traits remove duplication of code and provide a cleaner design.

Inheriting from a class or abstraction is common in OO programming, but Java restricts us to live with single implementation inheritance. Those of us who’ve tried multiple inheritance (MI) in languages like C++ know the consequences. MI is like a gorilla: it’s cute at first but you don’t bring it home. You have to deal with method collision and base instance duplication. While Java avoided these issues, simply prohibiting MI does not solve the real problem. In this tenth article of the series we’ll see how traits elegantly avoid the MI issues and provide an elegant modeling technique.

Our application that manages finance has gone through an initial audit and we’ve been asked to encrypt some data to increase security. The customer class we wrote,

| class Customer(val firstName : String, lastName : String) {
| //...
| override def toString =
| "Details for customer " + firstName + " - " + lastName
| }

needs a special encrypt method. As a first attempt, we add that method in.

| class Customer(val firstName : String, lastName : String) {
| //...
| def encrypt = {
| // some complex logic to encrypt the data
| toString.reverse
| }
| }

Here we have added an encrypt method in the Customer class. For this example we merely reverse the output of the call to the toString() method.

We can now directly call the encrypt method on an instance of Customer.

| val customer = new Customer("John", "Doe")
| println(customer.encrypt)
| //eoD - nhoJ remotsuc rof sliateD

Soon, however, we realize that there are other classes in the application, like Stock, that requires this treatment. We know that duplicating the encrypt method in this and other classes is an act of criminal negligence. One possibile approach is to create an abstract base class and move the encrypt method into it. Then the Customer class can extend from that class. Unfortunately we quickly realize this will not work since the Stock class already has a base class, and not all securities may need the encrypt method.

| class Security(val name : String)
| class Stock(override val name : String) extends Security(name) {
| override def toString = "Details for stock " + name
| }

The Java way to solve this problem would be to create an interface. That still does not prevent the duplication, as each of the classes needs the implementation. We will have to resort to delegation to avoid duplication. In Java, we would create a class with a static method. In Scala, that would be a singleton object.

| object Encryptor {
| def encrypt(data : String) = {
| // some complex logic to encrypt the data
| data.reverse
| }
| }

Then in the Customer class we can invoke the encrypt method of the Encryptor.

| class Customer(val firstName : String, lastName : String) {
| //...
| override def toString =
| "Details for customer " + firstName + " - " + lastName
| def encrypt = Encryptor.encrypt(toString)
| }

| val customer = new Customer("John", "Doe")
| println(customer.encrypt)
| //eoD - nhoJ remotsuc rof sliateD

The Stock class would have to do the same.

| class Stock(override val name : String) extends Security(name) {
| override def toString = "Details for stock " + name
| def encrypt = Encryptor.encrypt(toString)
| }

| val stock = new Stock("XYZ")
| println(stock.encrypt)
| //ZYX kcots rof sliateD

The core logic for encrypt is encapsulated, but there’s a thin layer of duplication in the classes that use it. Let’s see how traits solve this problem.

Scala does not have interfaces; instead, it has traits. A Scala trait with only method declarations and no implementation translates to a mere interface at the bytecode level.

Let’s create a trait called Encrypt.

| trait Encrypt {
| def encrypt : String
| }

Compile this trait using scalac and then view the generated bytecode using javap.

| > scalac Encrypt.scala
| > javap Encrypt
| Compiled from "Encrypt.scala"
| public interface Encrypt{
| public abstract java.lang.String encrypt();
| }

Scala traits directly map to Java interfaces. If we place implementations in the trait, then the Scala compiler creates an interface and an abstract base class for us. Let’s implement the encrypt method in our trait.

| trait Encrypt {
| def encrypt = {
| // some complex logic to encrypt the data
| toString.reverse
| }
| }

Compile this modified version of the Encrypt trait and take a look at the bytecode generated.

| > scalac Encrypt.scala
| > javap Encrypt
| Compiled from "Encrypt.scala"
| public interface Encrypt extends scala.ScalaObject{
| public abstract java.lang.String encrypt();
| }

| > javap Encrypt\$class
| Compiled from "Encrypt.scala"
| public abstract class Encrypt$class extends java.lang.Object{
| public static java.lang.String encrypt(Encrypt);
| public static void $init$(Encrypt);
| }

You probably have a hunch where this is going. Rather than requiring us to create the delegation, the Scala compiler takes care of it for us.

Let’s see how we can use this with the Customer class.

| class Customer(val firstName : String, lastName : String) extends Encrypt {
| //...
| override def toString =
| "Details for customer " + firstName + " - " + lastName
| }

| val customer = new Customer("John", "Doe")
| println(customer.encrypt)
| //eoD - nhoJ remotsuc rof sliateD

We extended the Customer class from the Encrypt trait in much the same way we would have extended from an abstract base class. The encrypt is now automatically part of Customer instances.

Since Stock already extends from Security, Scala allows us to bring the methods of the trait into the Stock class using a special with keyword.

| class Stock(override val name : String) extends Security(name) with Encrypt {
| override def toString = "Details for stock " + name
| }

| val stock = new Stock("XYZ")
| println(stock.encrypt)
| //ZYX kcots rof sliateD

Now all instances of Stock also have the encrypt method. If we take a look at the bytecode generated for the Customer and Stock classes using the javap -c command, we will see the encrypt method added to each of these classes during compile time.

We can bring multiple traits into a class at the same time simply by using multiple withs in the class definition. In the next article in this series we’ll see how we can force method collisions between multiple traits and use it to our advantage.

Scala allows us to apply traits to instances also. Let’s look at a CheckingAccount that does not have the traits.

| class CheckingAccount(val number : Int) {
| override def toString = "Details for account " + number
| }

| val anAccount = new CheckingAccount(1)
| println(anAccount.encrypt) //ERROR

Since CheckingAccount does not have any encrypt() method or the Encrypt trait, we can’t call the encrypt method on the instance we created. But even though the class does not have it, we can add traits to specific instances at the time of their creation.

| val secretAccount = new CheckingAccount(2) with Encrypt
| println(secretAccount.encrypt)
| //2 tnuocca rof sliateD

While, in general, instances of CheckingAccount don’t have the encrypt method, our secretAccount is special and can be encrypted.

Traits in Scala are handled entirely at compile time. When you use instance-level traits, the compiler creates an anonymous inner class for that instance.

In this article we saw how traits remove duplication of code and provide a cleaner design. In the next article we’ll see how method collisions are handled and how we can put this to a greater use.

Dr. Venkat Subramaniam is an award-winning author, founder of Agile Developer, Inc., and an adjunct faculty at the University of Houston.

He has trained and mentored thousands of software developers in the US, Canada, Europe, and Asia, and is a regularly invited speaker at several international conferences. Venkat helps his clients effectively apply and succeed with agile practices on their software projects.

Venkat is the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy: Dynamic Productivity for the Java Developer and Programming Scala: Tackle Multi-Core Complexity on the Java Virtual Machine. His latest book is Programming Concurrency on the JVM: Mastering Synchronization, STM, and Actors.

This series started in the September 2011 issue and has been running continuously since then. If you’d like to read the whole series, here are the links to the articles published so far:

9/11: The Elegance of Scala

10/11: Sensible Typing and Optional Items

11/11: Cute Classes and Pure OO

12/11: Functional Style of Programming

1/12: Working with Collections

2/12: Creating Higher Order Functions

3/12: Pattern Matching

4/12: XML as First Class Citizen

5/12: Recursions and Tail Call Optimization

6/12: Using Traits

Send the author your feedback or discuss the article in the magazine forum.

Retrospective Velocity

When Will This Project Be Done?

by Brian Tarbox, and Heather Mardis

Looking in the rear-view mirror to see how fast you're moving.

So how long is this going to take? This is one of the most common questions to ask or be asked in a traditional software development environment, and one of the most difficult to answer. The agile methodologies, and Scrum in particular, address this problem through the notion of velocity. Whatever your team accomplished in the last time period (sprint) is a good predictor of what they'll be able to accomplish in the next one. Scrum's use of burn down charts, standard definitions of "done" and retrospectives allows for the discovery of a team's average velocity. One can argue about details like hours versus story points, but almost any implementation of Scrum allows for at least tracking how much work was done per unit time.

If you've read Black Swans or The Age of the Unthinkable you may be troubled by our assertion that the past can be used as a predictor of the future. Both books strongly make the case that this is not in fact true. One of the books (I forget just now which one) cites the statistic that something like 80% of the total volatility of the US Stock Market can be accounted for by five individual trading days. They also make the case that grand Five Year Plans are almost always doomed to fail due to unforeseen developments. While all of that is true, and I am a fan of both books, this actually strengthens the case for Scrum's use of historical velocity. Scrum measures velocity sprint by sprint, and each measurement is a data point that can be used to correct assumptions that may be fading in relevance. Scrum assumes the existence of change and embraces it rather than trying to defend against it. And yet, even Scrum acknowledges that most of the time today will be very similar to yesterday and so we can use that data to make some assumptions about tomorrow.

Sadly, many projects are still using "traditional" methods (which seems to be one of the new names for waterfall). One of the drawbacks of such an approach is that the time scales are typically relatively large compared to Scrum, which means there are fewer time periods to measure and thus fewer opportunities for correction. There is simply less data available is you have two six-month buckets versus twelve one-month buckets. This is one reason that "traditional" projects do not typically end up producing a team velocity. This in turns make it substantially challenging to estimate how long the next project will take.

I recently attended the No Fluff, Just Stuff conference and one of the speakers said something striking about velocity (there were so many good talks that I cannot recall who made this particular statement).

He said "I've been developing software for 30 years. I wish that I had written down all of my estimates of how long something would take and then how long it had taken. If I had done that, imagine how accurate my estimates would be by now." I think it's safe to say that almost none of us have done what this speaker regrets not doing. Does that mean we just give up? Of course not. But...

One of the key take-away messages from one of the experts in the field of estimation is not to estimate at all. To quote Steve McConnell: " If you can't count the answer directly, you should count something else and then compute the answer by using some sort of calibration data." As it happens, for our problem of assessing velocity, we do have lots of data that might be helpful. The data in question live in your requirement tracking system, bug database, and source code control system. While these systems are often, shall we say, imprecise, they do contain useful data.

So, if we can't count our velocity directly we can count the number of bugs in our last project, how long they stayed open, how many requirements were present, and so on. Our calibration data is that all of those requirements and bugs took place in the time interval of the project. It's not especially fine-grained but in a waterfall model we are not looking for fine-grained data.

Some may argue that bug reports and requirements are not precisely defined and standardized. What passes as a single requirement for one team might be 3-5 separate requirements for another team. Some teams have bugs like "it doesn't work" while others might enter a dozen or more particular bugs to cover the same underlying defect. Here is the strange thing, though: it doesn't matter.

Just as story points are not standardized, all that matters is that within your team you tend to be consistent. Whatever your team's definition of a story point is if you're doing Scrum, it is likely to represent about the same quantum of work next month that it represented last month. In the waterfall world, the level of granularity you bring to your bug reports is likely to be fairly constant. The point is that we're not looking to equate points or bug counts against anything but other points and other bug counts in the same work environment. So, if your last several projects had 6 requirements, generated 60 bugs, and took 6 months, you have a velocity of 1 requirement and/or 10 bugs per month. If your next project arrives with 15 requirements and a deadline of three months from now we can safely conclude that you are in trouble!

Keep in mind the distinction between accuracy and precision. In the proceeding case we can say with high confidence that you are probably hosed.

All of this will just work better in Scrum than in Waterfall. So if Scrum is so much better why not just use it instead of waterfall? The problem of course is the little matter of changing your company's culture. But while it may take a huge effort to get your organization to move to Scrum, you, as an individual, can go compute your personal velocity right now. Any bug tracking system out there will allow you to gather data about the number of bugs assigned to you and/or fixed by you per unit time. If it were a database query you'd do something like select count(*) from bugs where assignee=="me" group by month(assignedDate). Now, you may object saying "but I do lots of other things besides fix bugs." You would likely be correct, and it still doesn't matter. If a third of your time historically gets absorbed by mind-numbing meetings, what makes you think next month will be any different?

As an individual you might pick from several data sources to establish a velocity. The selection may depend on your job function (sustaining engineer, development engineer) and/or on the systems you are using for bug tracking, source code management, and the like.

You might track the number of bugs fixed by you, the number of source code check-ins you do, the number of code inspections you participated in, or some combination of these and others. One interesting strategy is to gather the data from as many relevant sources as possible and then graph the results for the longest time period for which you have data. If we assume that your actual theoretical velocity has been relatively constant, you can select a data source or sources that results in a flat graph. Of course if you have switched languages or processes or problem domain, your velocity likely has not been constant, so your mileage may vary.

Another point to keep in mind if using the traditional approach is that quite often the project goes through distinct lifecycle phases. I think of these as the EPM, MS Word, Visio, Eclipse, and Jira phases of the project. Many of us are familiar with the pain of spending months in Word and Visio before we get "approval" to go back to the happier world of Eclipse (and then Jira). In these cases it's important to know how to combine your various data streams to get an overall velocity.

Figure 1 shows a somewhat fictionalized analysis of my work in a preceding year. The counts for requirements, defects, and check-ins are raw counts, while the Functional Specification word count is scaled by a factor of 250. Although the graph does bounce around a bit, it also shows an average of about four and a half work units per month. This is even more clear in Figure 2, which shows each month's velocity with standard deviation markers. From this chart it is easy to see that the velocity generally hovers between 4 and 6 work units per month.

Figure 1 Sample Retrospective Velocity

Figure 2 Average Velocity Range

Of course, a side benefit of such changes in velocity is that you can in fact document the change in velocity. In my own organization we have be increasing the mandate for code inspections of all code. While we selected a very lightweight inspection tool (Code Collaborator, highly recommended) the transition to an inspection regime was not without cost. Naturally management would like to see data showing some positive effects caused by this change. If one is measuring velocity, either at the individual or team level, you are well positioned to observe changes in the metrics. We expect to see a decline in field reported bugs for code that has been through the inspection process.

This approach can also be used as support for a stealth project. My coworkers are naturally wary of my new-found fascination with Scala. "There he goes again" has been muttered more than once. This is in fact a natural and reasonable reaction, until I start producing evidence of the benefit of Scala (or any other technology I might introduce to the team). If I present brand new data of a type the team is unfamiliar with, they can be forgiven some skepticism. If however I present the "standard" velocity measurements that we have become accustomed to collecting and analyzing, and if that data shows a delta for Scala-based stealth projects, then my arguments are on much firmer ground.

All discussions of Black Swans aside, whatever last month looked like is a good starting point for guessing what next month might look like. By all means measure each month and look for trends. Perhaps you and/or your organization is getting better and your velocity is increasing. Now we have more confirmation of that fact. Or perhaps in response to missed deadlines you're attending still more mind-numbing meetings to discuss such gems as why things take too long, and so your velocity is decreasing. Cold comfort perhaps, but now you have data to bring to those discussions.

Retrospective Bugs and Code Inspections

While we're talking about looking behind us let's explore another often overlooked opportunity for learning from the past. Even after instituting a policy of near-total code inspection we still of course have field-reported defects. Hopefully fewer, but they still exist. This reminds me of the standard interview question "So, candidate, what is your biggest weakness?" This is a boring question but it can have a powerful followup: "And what are you doing to address this weakness?" In the context of bugs and inspections, the followup question is "So if this code was inspected why didn't the inspection find the defect?" This is not meant to be a rhetorical question!

As the inspection process progresses it become more likely that the code containing any bug will be code that passed through an inspection. Rather than viewing this as a failure of the inspection process, we can view it as an opportunity to enhance that process. After a bug has been resolved, which by definition means we have identified what was wrong, we can and should take the additional step of identifying why the inspection did not catch the defect.

Perhaps the inspectors lacked the subject matter expertise to see a semantic flaw in the code. Perhaps the inspection checklist (be it formal or informal) did not emphasize looking for synchronization issues. Perhaps the inspectors had too much faith in an "expert" coder and so performed just a cursory inspection. Whatever the reason, it should be identifiable and then actionable.

While you might see this as yet another process to slow down development, it can actually provide enormous benefit. When you test or inspect code, you are improving that one section of code. When you perform meta-analysis of your inspection process, you are improving all code that will pass through the inspection process.

We've seen that you don't have to be doing Scrum to get the benefits of knowing your Velocity and tracking how it is changing. And we've seen that looking backwards at your inspections can lower your defect rate. And you can do both without requiring changes or approvals from management. Perhaps it's time to give these ideas a try.

Brian Tarbox is a Distinguished Member of Technical Staff at Motorola in the Systems Engineering group designing next generation video products. He writes a blog on the intersection of software design, cognition, music, and creativity at briantarbox.blogspot.com. His primary contact point is about.me/BrianTarbox .

Heather Mardis is a Senior Engineer at Motorola Mobility in the On Demand Video group, providing release and build infrastructure, tools, and processes. She has a long background in configuration and release management, most recently succeeding in bringing continuous integration and build-automation to the ODV team at Motorola to support their three-continent development needs. Her credits include conference presentations at Atlassian Summit including the "you've got issues" award in 2010. She strives to be the go-to person in ODV, keeping tabs on all the many projects in progress at once and staying one step ahead of what the team might need next.

Send the authors your feedback or discuss the article in the magazine forum.

Calendar

Author sightings, partner events, and other notable happenings.

Thanks to David Copeland for creating a tool to make it easy for authors to generate the code I need to post these! Here’s what’s coming up in the next three months:

Author Appearances

Who’s where, and what for.

June 6–8, Norwegian Developer Conference, Oslo, Norway

Multiple talks related to Functional programming, F#

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

June 10–15, Better Software/Agile Development Practices West, Las Vegas, NV

keynote: “Influence and Authority: Using Your Personal Power to Get Things Done”

Johanna Rothman, author of Behind Closed Doors: Secrets of Great Management Manage It! Your Guide to Modern, Pragmatic Project Management Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects

June 12, Connected World Conference, St. Charles, IL

Home Automation Panel

Mike Riley, author of Programming Your Home

June 13, YAPC::NA (Yet Another Perl Conference), Madison, WI

“29 ways to get started in open source today”

Andy Lester, author of Land The Tech Job You Love

June 19–22, UberConf , Denver, CO

Multiple talks related to Scala, HTML5, Concurrency, JavaScript, Test automation

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

June 23, Goruco 2012, New York, NY

“Maintaining Balance While Reducing Duplication: Part II”

David Chelimsky, co-author of The RSpec Book

June 26–29, Open Source Bridge, Portland, OR

“Logic Lessons That Last Generations”

Ian Dees, co-author of Using JRuby and author of Scripted GUI Testing With Ruby

June 27, Bootstrapd—a community gathering for software craftspeople with an independent spirit, Edinburgh, UK

Bootstrapd—conference organizer

Matt Wynne, author of The Cucumber Book

June 28–30, CocoaConf, Herndon, VA

“Mobile Movies with HTTP Live Streaming” and “Core Audio Cranks It Up”

Chris Adamson, co-author of iOS SDK Development

June 29, Scottish Ruby Conference, Edinburgh, UK

“TDD your command line apps for fun and profit”

David Copeland, Author of Build Awesome Command-Line Applications in Ruby

June 29–30, Scottish Ruby Conference, Edinburgh, UK

“Hexagonal Rails”

Matt Wynne, author of The Cucumber Book

June 29–30, Utah Software Symposium, Salt Lake City, UT

Multiple talks on JavaScript, HTML 5, Scala, Concurrency, and automated testing

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

July 19, OSCON 2012, Portland, OR

“Threads and JRuby, the Simple Alternative to Evented”

David Copeland, Author of Build Awesome Command-Line Applications in Ruby

July 20, OSCON, Portland, OR

“Your Own Metric System”

Ian Dees, co-author of Using JRuby and author of Scripted GUI Testing With Ruby

Aug 9–11, CocoaConf, Columbus OH

“Core Audio All-Day Tutorial,” “Mobile Movies with HTTP Live Streaming,” and “Core Audio Cranks It Up”

Chris Adamson, co-author of iOS SDK Development

O’Reilly Events

Upcoming events from our friends at O’Reilly.

June 25–27, Velocity Conference Santa Clara, CA

Velocity Conference: “Velocity is the best place to learn from peers, exchange ideas with experts, and share what has worked (and equally importantly, what has not worked) in real world applications. The focus is on practical, large-scale web development, presented by experts.”

July 16–20, O’Reilly Open Source Convention Portland, OR

OSCON: “If you work with open source, you’ll want to be here. With a dozen plus tracks and hundreds of sessions, you’ll find practical tutorials, inspirational keynotes, and a wealth of information on open source languages, platforms, and development.”

USENIX Events

What’s coming from our USENIX friends.

June 4–6, 5th Annual International Systems and Storage Conference Haifa, Israel

SYSTOR 2012: “SYSTOR 2012, the 5th annual international systems and storage conference, promotes experimental and practical computer systems research.”

June 7–8, 4th USENIX Workshop on Hot Topics in Parallelism Berkeley, CA

HotPar ’12: “HotPar will bring together researchers and practitioners doing innovative work in the area of parallel computing. HotPar recognizes the broad impact of multicore computing and seeks relevant contributions in all fields, including application design, languages and compilers, systems, and architecture.”

June 12–13, 4th USENIX Workshop on Hot Topics in Cloud Computing Boston, MA

HotCloud ’12: “HotCloud brings together researchers and practitioners from academia and industry working on cloud computing technologies.”

June 12–15, 2012 USENIX Federated Conferences Week Boston, MA

FCW ’12: “Back for 2012, USENIX is combining established conferences and workshops into a week of research, trends, and community interaction.”

June 12, 2012 USENIX Women in Advanced Computing Summit Boston, MA

WiAC ’12: “This is an all day event for those interested in advancing women in computing representing both academia and industry.”

June 13–15, 2012 USENIX Annual Technical Conference Boston, MA

USENIX ATC: “USENIX ATC has always been the place to present groundbreaking research and cutting-edge practices in a wide variety of technologies and environments. USENIX ATC ’12 will be no exception.”

June 13, 4th USENIX Workshop on Hot Topics in Storage and File Systems Boston, MA

HotStorage ’12: “Over the past few years, storage has been changing like never before. Architectures are evolving rapidly, with many fundamental shifts. The HotStorage workshop aims to provide a forum for the cutting edge in storage research where researchers can exchange ideas and engage in discussions with their colleagues.”

June 13, 3rd USENIX Conference on Web Application Development Boston, MA

WebApps ’12: “WebApps ’12 is designed to bring together experts in all aspects of developing and deploying Web applications. The program includes refereed papers on topics such as distributed systems and browser extensions, Java and client extensions, and more, in addition to keynotes by Douglas Crockford (Yahoo!), and Yehuda Katz (Tilde, Inc.)”

June 14–15, 4th USENIX Workshop on the Theory and Practice of Provenance Boston, MA

TaPP ’12: “Provenance provides important documentation that is an essential part of the quality of data, and it is essential to the trust we put in, for example, the data we find on the Web and the data that is derived from scientific experiments. TaPP ’12 covers topics related to theoretical or practical aspects of provenance, including but not limited to: provenance in databases, workflows, programming languages, security, software engineering, or systems; provenance on the Web; real-world applications of or requirements for provenance.”

June 15, 6th USENIX/ACM Workshop on Networked Systems for Developing Regions Boston, MA

NSDR ’12: “NSDR ’12 will provide a venue for researchers to propose and discuss ideas concerning the design, implementation, and evaluation of new computing and communications technologies to support the sustainable development of developing regions. NSDR ’12 specifically focuses on the technical networking and systems research challenges that arise in the design, implementation, and deployment of new computing solutions appropriate for developing regions.”

July 23–24, Asia-pacific Systems Workshop Seoul, South Korea

APSYS 2012: “Building on the success of its 2010 and 2011 predecessors in New Delhi, India and Shanghai, China, APSys 2012 will be a lively forum for systems researchers and practitioners across the world to meet, interact, and collaborate with their peers from the Asia/Pacific region.”

Aug 8–10, 21st USENIX Security SymposiumBellevue, WA

USENIX Security ’12: “The USENIX Security Symposium brings together researchers, practitioners, system administrators, system programmers, and others interested in the latest advances in the security of computer systems and networks. The Symposium will span three days, with a technical program including: Keynote Address by Dickie George, Johns Hopkins Applied Physics Laboratory; over 40 Refereed Papers covering the latest research including topics such as browser security, CAPTCHAs, and passwords; privacy enhancing technologies; botnets and Web security; and more.”

Aug 6–7, 2012 Electronic Voting Technology Workshop/Workshop on Trustworthy ElectionsBellevue, WA

EVT/WOTE ’12: “EVT/WOTE ’12 brings together researchers from a variety of disciplines, ranging from computer science and human-computer interaction experts through political scientists, legal experts, election administrators, and voting equipment vendors.”

Aug 6–7, 3rd USENIX Workshop on Health Security and PrivacyBellevue, WA

HealthSec ’12: “The focus of HealthSec ’12 will be on the development of new techniques and policies to ensure the privacy and security of next-generation healthcare systems and devices.”

Aug 6–7, 6th USENIX Workshop on Offensive TechnologiesBellevue, WA

WOOT ’12: “WOOT ’12 aims to bring together researchers and practitioners in systems security to present research advancing the understanding of attacks on operating systems, networks, and applications.”

Aug 6, 2nd USENIX Workshop on Free and Open Communications on the InternetBellevue, WA

FOCI ’12: “FOCI ’12 seeks to bring together researchers and practitioners from technology, law, and policy who are working on means to study, detect, or circumvent practices that inhibit free and open communications on the Internet.”

Aug 6, 5th Workshop on Cyber Security Experimentation and TestBellevue, WA

CSET ’12: “The science of cyber security is challenging for a number of reasons. Meeting these challenges requires transformational advances, including understanding of the relationship between scientific method and cyber security evaluation, advancing capabilities of underlying experimental infrastructure, and improving data usability.”

Aug 7, 7th USENIX Workshop on Hot Topics in SecurityBellevue, WA

HotSec ’12: “HotSec places its singular emphasis on new ideas and problems. HotSec takes a broad view of security and privacy and encompasses research on topics including, but not limited to, large-scale threats, network security, hardware security, software security, physical security, programming languages, applied cryptography, privacy, human-computer interaction, emerging computing environment, sociology, and economics.”

Aug 7, Seventh Workshop on Security MetricsBellevue, WA

MetriCon 7.0: “How to define, generate, and communicate security metrics you can use today.”

Other Happenings

June

In this month in 1957, computer science legend Donald Knuth published his first article, “Potrzebie System of Weights and Measures,” in Mad Magazine.

June 1

Tim Paterson, whose operating system made Bill Gates a jillionaire, is 55.

June 8

Tim Berners-Lee is 57.

June 23

Vint Cerf is 69.

June 23

Mathematica is 24.

June 28

Firefox co-creator Dave Hyatt is 30.

July 2

Gordon Letwin, one of first dozen Microsoft employees and lead Microsoft architect on OS/2, is 60.

July 5

Linguist Victor Yngve, creator of COMIT, the first string processing language, is 92.

July 6

Adi Shamir, the S of the RSA encryption algorithm, is 60.

July 7

Smalltalk co-creator Adele Goldberg is 67.

July 9

Marc Andreessen is 41.

July 10

Legendary game designer Scott Adams is 60.

July 16

Dan Bricklin is 61.

July 22

Alan Cox, early Linux kernel contributor, is 44.

July 24

Al Lowe, creator of Leisure Suit Larry series of computer games, is 66.

July 26

Hakon Wium Lie, creator of Cascading Style Sheets, is 47.

July 29

James Duncan Davidson is 42.

Aug 9

Alfred Aho, co-creator of AWK language, is 71.

Aug 11

Steve Wozniak is 62.

Aug 20

Game developer John Carmack is 42.

Aug 29

Boy genius Stephen Wolfram is 53.

Shady Illuminations

The Post-Windows Era

by John Shade

John wonders, if we’re in the post-PC Era, shouldn’t we also be post-Windows?

Microsoft, now that you’ve killed off Zune, don’t you think it’s time to put Windows out of our misery?

But wait. That’s not right. Before I start offering you advice, I should identify the problem you have that I’m trying to address. That’s just fair and reasonable.

A Serious Lack of Lack

A couple of problems you have, as I see it, are too much money and too much talent.

You’re not alone in these problems. People complain about Facebook making drastic changes in its user interface. When I hear this I think, it's like the weather. If you don't like it, wait a few weeks and it'll change again. It has to. Facebook has a ton of money (at close of market today three quarters of a ton) and lots of clever programmers, and they have to use these resources. And from what I’ve heard about the Facebook code base, tinkering with it is an excellent way to use up a lot of programmer time and investor dollars.

Nobody has a deeper bench of talent than you, but the difference is, what Facebook’s talented programmers inevitably produce is a more annoying version of Facebook. What results from the marriage of marketing and coding at Microsoft is inevitably two things stitched together. Ars Technica said Zune had a “strange schizophrenia of spirit,” but there was nothing strange about it. Everything you do comes out like the front page of a news site where every department thinks it should be above the fold. Your products try to please many masters, unlike Apple’s, which in recent years had to satisfy just one master: Steve Jobs’ mental model of the customer.

Startups don’t have your problem focusing because they don’t have unlimited money and talent. So you fear startups. You should.

So that’s one problem you’ve got. Too much money and talent that you don’t know what to do with.

You Don’t Kill Your Babies

Another problem you have is everything you’ve ever done. Because, unlike Apple, you don’t kill your babies. You did it with Zune, I admit, and that was smart. It also feeds your Xbox business, also smart. But this is not generally what you do. Fundamentally you are in the business of supporting an installed base. You’re not Facebook, you’re MicroFocus.

I know, I cited Apple again. But Apple is obviously doing a lot of things right, and even I can identify some of the more obvious things it’s doing right. You can, too. You just can’t do them.

Much as you might like to, you can’t just sit back and enjoy a nice profit from your healthy Xbox business and ignore the rest. Because “the rest,” especially the Windows brand, is a heavy burden of legacy. It is to you what 8080 instructions are to Intel. You might like to get rid of it, you don’t think it’s helping you move forward, but it’s inextricable. It defines you. It’s your junk DNA.

At least that’s how I think you see it. I suggest you’d be better offf thinking of it as a boat anchor or albatross. Either way, it’s something you can cut loose.

A Few Modest Proposals

So, having identified your problems—too much money and talent and everything you’ve ever done—I’d like to offer some advice.

First, kudos on deciding to go into Iraq. I predict that will help with the too much money and talent problem. Open an office in Afghanistan, too.

Second, the name Windows.

I get it that you are unable to abandon the Windows user base. So you’re going to continue to create schizophrenic products like Windows 8, with its Metro/Rural multiuser experience. (It’s a little bit Country, it’s a little bit Charles Aznavour.) But the name: is that really working to your advantage?

Think about it: If we had started out with smart phones as computing devices rather than with TV sets attached to boxes of electronic parts, would we have thought of overlapping or tiled windows? Or windows at all? Is that a metaphor that would have served us well? And now that we’re abandoning the 1980s vision of what a computing device should look like, shouldn’t we abandon the no longer useful user interface metaphors as well? Or at least quit using the no longer meaningful words?

The Windows code is a different matter entirely, I know. But even there I have a suggestion for you.

Open-source Windows. Could open source developers resist probing that codemaze? And once in, could they extract themselves? Wouldn’t their coding practices be Borged? Think of it as a giant tarbaby. And you could release it as a huge tarbomb.

These are just a few suggestions. I hope you find them useful.

John Shade was born under a cloud in Montreux, Switzerland, in 1962. Subsequent internment in a series of obscure institutions of ostensibly higher learning did nothing to brighten his outlook. Send the author your feedback or discuss the article in the magazine forum.

But Wait, There’s More...

Coming Attractions

This may be the last page, but that doesn’t mean it’s the end of the conversation. Keep up with what’s going on in the Pragmatic universe by subscribing to our newsletter and drop by the Pub again next month for more Pragmatism. We’ll keep a seat warm and a mug chilled and a candle burning in the window.

Coming Soon in PragPub

In the works: Sam Aron is writing an article on using data structures to represent musical compositions and synthesis with working examples in the Clojure music toolkit Overton. Julius Seporaitis has some ideas to share about a new way of creating web applications. And Christoph Pater suggests that, if there is technical debt, maybe we should have technical debt counsellors.

Recent Releases from the Bookshelf

Working with Unix Processes is now available, Seven Databases in Seven Weeks: A Guide to Modern Databases and the NoSQL Movement is now in print, and Edit Text at the Speed of Thought is now in beta.  

Right. But to really be in the know, you need to subscribe to our weekly newsletter. It’ll keep you in the loop, it’s a fun read, and it’s free. All you need to do is create an account on pragprog.com (email address and password is all it takes) and select the checkbox to receive newsletters.

While you’re waiting for the next issue of the newsletter or of PragPub to come out, you can follow us on twitter at @pragpub, @PragmaticAndy, or @pragdave. Or on Facebook at facebook.com/PragmaticBookshelf. Or email us at michael@pragprog.com.