PragPub 2009-10: Issue #4

Table of Contents

Features

Android’s Next Move

interviewed by Michael Swaine

An interview with Hello, Android author Ed Burnette.

Mail on Rails

by Dave Hoover

Delivering the mail is a pain. So why configure mail servers and domain keys when you can get existing Web APIs to do the dirty work for you?

TextMate Tricks

by James Edward Gray II

With a little help from Ruby, James spiffs up his favorite text editor.

State Secrets

by David Bock

State machines, workflow, and international treaties.

Painting with Sound

by Andy Hunt

Basic tips for home recording to help you tell your story more effectively and sound better.

Departments

Up Front

by Michael Swaine

Help us shape PragPub.

Choice Bits

Tweets, posts, and chatter on programming, publishing, people, and PragPub.

How Do We...?

by Dave Thomas

People often ask how we do what we do. This series explains…

Get a Life

by Daniel Steinberg

Write that book you’ve always wanted to. We’ll help.

Language Workbenches

by Michael Swaine

DSLs rule! Now if only they weren’t so unruly.

Shady Illuminations

by John Shade

How Google hijacked Internet Explorer.

The Quiz

by Michael Swaine

A monthly diversion at least peripherally related to programming.

Calendar

by Jackie Carter

Author sightings and other notable events.

Except where otherwise indicated, entire contents copyright © 2009 The Pragmatic Programmers.

Feel free to distribute this magazine (in whole, and for free) to anyone you want. However, you may not sell this magazine or its content, nor extract and use more than a paragraph of content in some other publication without our permission.

Published monthly in PDF, mobi, and epub formats by The Pragmatic Programmers, LLC, Dallas, TX, and Raleigh, NC. E-Mail support@pragprog.com, phone +1-800-699-7764. The editor is Michael Swaine (mailto:michael@pragprog.com). Visit us at http://pragprog.com

ISSN: 1948-3562

Up Front

It’s About You

by Michael Swaine

The magazine as coffeehouse.

In my formative years, I used to hang out in a coffeehouse in Elkhart, Indiana, named Partly Dave. It was run as a sort of co-op, with patrons serving on the board and serving up the coffee, and its slogan was “Partly Dave is partly you.”

Corny, I know. But it was a good idea then, and it’s a good idea now. I think of PragPub as partly magazine, partly coffeehouse, and yes, partly you.

Unlike the Wizard of Oz, we do want you to pay attention to what’s going on behind the screen. We want you to jump right in and contribute to the conversation. This magazine has no Letters page. (Letters pages are so 20th century.) Our reader discussions take place live in our Forums.

You can do more than comment on articles: you can also write articles for PragPub. There is no entrance exam. All you have to do is come up with an article that our other readers would like to read.

The same goes for books. In this issue’s “Get a Life,” Daniel Steinberg describes an exciting Pragmatic Programmers project that could help you to finally write that book you’ve been contemplating. If it turns out to be a book that our other readers would like to read, you could be on your way to becoming a Pragmatic Bookshelf author.

Daniel’s piece is one of the more reader-involving elements of this issue. Our fourth opening of this Pragmatic Pub also features articles on Google Android, efficient e-mailings with Ruby, producing PDFs from TextMate, workflow modeling with state machines, and home recording tips for amateur musicians. Our columnists delve into Google Chrome Frame, the rise of DLL programming, and how we produce book-specific markup. And as always, there’s a quiz to exercise your little grey cells.

But I was arguing that PragPub is partly you, and there’s another important way in which you can be involved with the Pub. You can help us decide what to do next. You can help us create the next iteration of PragPub.

As it says on the cover, this is “The First Iteration” of PragPub. We’re actively gathering ideas about what the second iteration should be, and we really want your input. Everything is on the table: format, delivery platforms, content, you name it. These four issues have primed the pump. We have smart people with interesting ideas queueing up to produce articles for PragPub. Now we want to think creatively about how best to share their insights.

We invite your brilliant suggestions.

Choice Bits

Overheard on the Intertubes

Wading into the chatterstream in hip boots, we wield the Axiom of Choice with gleeful abandon.

What’s Hot

Top-Ten lists are passé—ours goes to 11. These are the top titles that folks are interested in currently, along with their rank from last month. This is based solely on direct sales from our online store.

1^ | NEW | Beginning Mac Programming
2^ | 3 | The RSpec Book
3v | 1 | Agile Web Development with Rails, Third Edition
4v | 2 | iPhone SDK Development
5 | 5 | Programming Ruby 1.9
6^ | NEW | Cocoa Programming
7^ | 10 | Language Implementation Patterns
8 | 8 | Core Data
9v | 4 | Metaprogramming Ruby
10v | 7 | Programming Clojure
11^ | NEW | Manage Your Project Portfolio

And Here’s What Folks Are Saying…

For starters, let’s get it out of the way and admit that the Kindle’s user interface is just horrible. — Lyle Johnson

In Tribes, Seth Godin makes a rather startling assertion, that a Tribe must “exclude outsiders.” This doesn’t seem like a way to grow a tribe, but he says that you should go for quality rather than size. I think this applies to weblogs as well.... — Bruce Eckel

It hasn’t been all roses, they have had experience of accidentally deleting walls and also removing avatar clothing. — Rachael Davies

Most managers can’t hear no if they literally cannot imagine how to work themselves out from under the pile of work. — Johanna Rothman

Pragmatic programmers should not be re-termed duct tape programmers. — Dhananje Nene, challenging Joel Spolsky

“Clone Yourself” .. a most thought provoking article! A big ‘thank you’. — Roy Daly in PragPub Forum

I often find that a nice design can come from just being really anal about getting rid of duplicated code. — Martin Fowler

I am rarely happier than when spending entire day programming my computer to perform automatically a task that it would otherwise take me a good ten seconds to do by hand. — Douglas Adams

Communications person: Do you think we need to tell the field? Change manager: Well, it all depends on what level of confusion you want to communicate. — OverheardInTheOffice.com

A dialog box is another room. Have a good reason to go there. — Alan Cooper

There are only two industries that refer to their customers as “users.” — Edward Tufte

The Microsoft Paradox: Every generalization about Microsoft is simultaneously true and false. — Stephen Cawood

Enjoying PragPub magazine from the pragmatic programmers. Printed it onto dead trees. Is that bad? — Glenn Halstead via Twitter

We can all imagine an old-style editor getting a scoop and saying “this will sell a lot of papers!” Cross out that final S and you’re describing their business model. The reason they make less money now is that people don’t need as much paper. — Paul Graham

AUTHOR TOUR: A hazing ritual intended to make authors compliant to their publishers. — Thomas Christensen, Tom’s Glossary of Publishing Terms

One day, you’ll turn off the feature that emails you every time someone buys your software. That’s a huge milestone. — Joel Spolsky

How Do We...?

How Gerbils Make Sausage

by Dave Thomas

People often ask how we do what we do. This series explains…

This month's question: How do you create book-specific markup?

As we discussed last month, one of our goals is to use proper semantic markup for our books. This means that we have to accommodate books that have specific markup requirements.

Last month we looked at the technique we use to allow what is effectively subclassing of our book DTDs—the specifications of the markup permitted for each book. Using this, we can have a single master DTD for all the regular markup and per-book DTDs for the book-specific stuff. This allows us to validate each book is syntactically correct during the build. But a book’s no good unless you can read it, so this month we’ll look at the next step—how we support per-book specialized formatting. This is what lets us convert the book-specific markup into the correct representations for paper books, PDFs, and the various eBook formats.

We convert our books from XML into the required output format using XSLT. XSLT is a processor that takes an XML document and transforms it into another form. We use it to generate XML, HTML, and TeX versions of our books. XSLT works by reading a set of pattern-matching transforms, and then applying appropriate transforms to a source XML document.

Now, the Rails book talks a lot about key/value pairs—elements of Ruby hashes. After I wrote these inline a couple of times, I realized that I really should be using semantic markup for them. So I added a new PML element just for the book. You write “Saying <hashentry key=":id">product</hashentry> is idiomatic” which renders as “Saying :id => product is idiomatic.”

So how do we make this particular transform local to just the Rails book? We actually use something vaguely reminiscent of Ruby’s mixins.

Each book’s source tree contains a set of XSLT transforms, one for each of our target formats. When we first set up a title, the transform (for example) for the mobi format is trivial:

| <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
| version="1.0">
| <xsl:import href="../../../../../Shared/xml/ppb2mobi.xsl"/>
| </xsl:stylesheet>

All it does is load up the global set of mobi transforms. In effect, the default transforms are mixed into the book-local XSLT file.

Then, when we want to add new markup, we can simply add the transform to the book-local file, somewhere after the mixin. (Apologies for the low-level HTML—the Kindle doesn’t support stylesheets.)

| <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
| version="1.0">
| <xsl:import href="../../../../../Shared/xml/ppb2mobi.xsl"/>
| <xsl:template match="hashentry">
| <code><xsl:value-of select="@key"/></code>
| <xsl:text> </xsl:text>
| <code><xsl:text>=></xsl:text></code>
| <xsl:text> </xsl:text>
| <code><xsl:apply-templates/></code>
| </xsl:template>
| </xsl:stylesheet>

We use TeX to create our printed books and DRM-free PDFs. In this case, the XSLT transforms the XML to TeX source, making use of macros in our TeX stylesheet.

| <xsl:template match="hashentry">
| <xsl:text>\hashentry{</xsl:text>
| <xsl:call-template name="scape">
| <xsl:with-param name="string">
| <xsl:value-of select="@key"/>
| </xsl:with-param>
| </xsl:call-template>
| <xsl:text>}{</xsl:text>
| <xsl:apply-templates/>
| <xsl:text>}</xsl:text>
| </xsl:template>

Because the TeX macro \hashentry is not one of our standard ones, we have to define it locally for the book. For TeX, we use a slightly different approach to get book-local functionality.

TeX supports the idea of a load path—a list of directories that TeX will search when loading files. All of our regular TeX macro packages are stored in a shared directory that appears last in this search path. And many of these macro packages are simply empty placeholders, loaded up during formatting at strategic places but having no effect on the build.

But… we include a book-local directory close to the front of TeX’s search path. If there’s a file in that directory with the same name as a global one, TeX will load it in preference. And that lets us add book-local TeX formatting. We just put formatting macros into a file called local.tex in that directory, and those macros are available during that book’s build. For the \hashentry macro, the file contains:

| \newcommand{\hashentry}[2]{%
| \codething{#1\,=>\,#2}%
| }

(The \, sequence is a narrow space, which formats the entry nicely.)

Dave Thomas is one of the Pragmatic Programmers.

Get a Life

It’s time to write that book

by Daniel Steinberg

Join us in the month of November for an experiment in authorship.

When I tell people that I mainly write and edit books for a living, they look a little past me as if gazing at what might have been and say, “I’ve always wanted to write a book.”

I always answer, “You should try,” and I mean it. I don’t mean that they need to publish a book or that they even need to finish writing their book. But if you’ve always wanted to do something, then you should try to do it and find out if you can or not.

If you want to be a writer, you’ve got to have something to say. Think for a minute. What type of book would you like to write? What would it be about? If you don’t come up with any book ideas—that’s OK. You’ve saved yourself a lot of time by understanding that ahead of time. A surprising number of authors don’t realize they have nothing to say until they’ve said it.

One of the hardest things about writing your book is that there are no real deadlines. You can resolve to start writing tomorrow, but really, what’s the difference if you don’t start for another day—or another month, or more? And so you never start. There are always more urgent things that you need to take care of. Years go by and you think “I’ve always wanted to write a book.”

What are you waiting for?

So no more putting it off. Between now and the end of the year let’s find out whether or not you can write that book. We’ll give you a couple of weeks to think about whether or not you’re really ready for this challenge, and then on November 1st a group of us will spend the month working on our books. You can write a Prag-style book or you can write that novel. It’s up to you.

If you’ve always wanted to write a novel, then you should sign up with NaNoWriMo. Each November they celebrate and support National Novel Writing Month. You start writing on November 1 and try to write a 50,000 word novel by the end of the month. 50,000 words is about 175 pages. You focus on quantity, not quality—there will be plenty of time to rework your novel, but the month of November is spent getting the first draft out. At that point you’ve written a novel.

Of the more than 100,000 participants in 2007, more than 15,000 of them finished. Around 15% of those who tried, succeeded. That’s amazing. If you decide to participate, tell us about it on the magazine’s forums. Let us know how it went.

That’s if a novel is the kind of book you’ve been wanting to write.

Introducing PragProWriMo

But what if a novel isn’t for you? What if you’re more of a non-fiction author-to-be? We’ve been thinking about that a lot and we’ve come up with a plan. We’re piggy-backing on NaNoWriMo for authors who are working on one of the two kinds of books we publish:

Our Pragmatic Bookshelf series—which helps developers master some language, technique, or technology that makes them more effective at work; or

Our Pragmatic Life series—which helps developers and others improve their life outside of work.

If you are thinking about either of these types of books, we’d love for you to participate in PragProWriMo. Really, there’s not much to it. We’ll provide you with some information about writing a book and encourage you to write 80 pages during the month of November. If you write every day, that’s a little less than three pages a day. If you write five days a week then it’s about four pages a day.

This gives you a concrete time period. In about two weeks from now we’ll post information on the Prag site about this project and you’ll be able to sign up. At the end of November you’ll know whether or not you can write a book. You may find by the end of the month that you are most of the way there.

If you’re successful (and we think you will be), then we’d love for you to submit a proposal for your book to us. On the other hand, it’s your work. You can do with it what you want. You can publish it for free, you can do print on demand, you can hide it from the world and keep it to yourself, or you can take it to another publisher.

Maybe some great new Prag books will result from this project. Maybe not. Our only real goal in this is to help you write the book you’ve always wanted to write.

I think you should try.

Daniel is the editor for the new series of Mac Developer titles for the Pragmatic Programmers. He writes feature articles for Apple’s ADC web site and is a regular contributor to Mac Devcenter. He has presented at Apple’s Worldwide Developer Conference, MacWorld, MacHack, and other Mac developer conferences. Daniel has produced podcasts for Apple featuring the work of developers and scientists working on the platform. He has coauthored books on Apple’s Bonjour technology as well as on Java Programming and using Extreme Programming in Software Engineering classes.

Language Workbenches

Is This the Era of the DSL?

by Michael Swaine

Under names like Language-Oriented Programming, Language Workbenches, and Intentional Programming, a new paradigm of software development is gaining in importance.

Even if you do most of your coding in Ruby, I’ll bet you don’t call yourself a Ruby programmer. Ditto for Java, C# or C++ or Objective-C, assembler or Erlang or Clojure. You’re way too savvy to characterize yourself in terms of one programming language. Your current language of choice doesn’t define you personally or professionally.

But it does make some sense to identify eras of programming in terms of a dominant language. There was a time when you couldn’t pass yourself off as a professional software developer unless you were proficient in C; while back in the dawn of time when I enrolled in my first programming class, it was taken for granted that programming meant Fortran. I’m not suggesting there aren’t scores of other languages in use in any era, and plenty of people who program exclusively in these non-dominant languages. But it still means something to characterize the period, say, from 1973 (Unix kernel rewritten in C; now there’s a killer “app”) to the rise of object-oriented languages in the mid-80s as the C Era.

So if every era can be characterized by a dominant programming language, what era would you say that we are in today? It’s a tough question, isn’t it, because there seems to be more freedom to choose your language to fit the task today than at any other time in the history of programming. In fact it’s been said that we’re in a “post-modern era of language diversity.” Sounds good to me.

But I’d like to push it a step further and suggest that the blade of the plow of progress today is not even in the field of general-purpose programming languages at all. I’d argue that this is the era of the DSL.

Domain-speciﬁc languages (DSLs) are, as the name suggests, specific to a particular problem domain. A DSL’s syntax reflects the natural language of experts in the domain. As Venkat Subramaniam says in Programming Groovy, “You don’t use them for general-purpose programming like you use Java, Groovy, or C++, because DSLs have a very limited scope and capability.” A DSL can be internal, meaning that it is mapped onto the syntax of a general-purpose language, or external, in which case it stands alone. Creating an external DSL takes more work, but provides more control. And as Terence Parr demonstrates in Language Implementation Patterns and The Definitive ANTLR Reference: Building Domain-Specific Languages, it’s not that hard to create external DSLs.

But you knew all that. You would have to know about DSLs, because they’re ubiquitous. Ant and make are examples of external DSLs, gant and rake are internal DSLs. As Paolo Perrotta points out in Metaprogramming Ruby, the Unix shell is really a DSL for gluing command-line utilities together. Rails is a DSL.

But why claim that this is the Era of the DSL? After all, domain-specific languages are nothing new. The whole tradition of Little Languages in Unix is just another take on the idea of DSLs. NASA was using DSLs as far back as the early days of the Apollo program in the 60s.

What’s new today is a shift in the center of gravity of software development toward DSL-based programming. It started to be noticed four or five years ago and there have been indications just this year that it is ready to come into its own.

Five years ago, Sergey Dmitriev of JetBrains articulated a model of DSL-centric programming called Language-Oriented Programming (LOP). Rather than:

Analyze the problem and model a solution

Choose a general-purpose language

Code the solution in the GPL

the LOP model calls for you to:

Analyze the problem and model a solution

Choose a domain-specific language or write one

Code the solution in a syntax closer to the domain using the DSL

JetBrains has been following up on this model and developing tools to make it practical, while other companies have been exploring similar models, including Charles Simonyi’s Intentional Software. An umbrella term for these approaches, used by Martin Fowler, is Language Workbenches.

Four years ago, Fowler spelled out the shortcomings of DSLs and identified what was needed to overcome the drawbacks to DSL programming. He described an approach to programming that puts DSLs in the center of the process and provides the kinds of tools you expect to have in a real development environment. He pointed out examples where people were exploring this approach, including the JetBrains and Intensive folks. And he captured all these approaches under the umbrella term Language Workbenches, a term that suggests that the tools needed for DSL-centric programming are all hanging on the wall above the workbench.

Intentional Software, is working on a system that completely implements all of Fowler’s requirements for a Language Workbench. Simonyi says that it’s crucial to separate the problem from the program, and his company plans to empower a domain expert to encode the problem using a DSL.

Now things are heating up. Fowler is giving lectures and writing a book on Language Workbenches. In May at the DSL Developer’s Conference, Intentional Software took the wraps off its secretive DSL-centric Intentional Domain Workbench. And in July JetBrains announced the 1.0 release of its Meta Programming System, or MPS, for DSL-centric programming. Attendees at the Intentional demo were effusive, comparing it to the legendary 1968 “Mother of all Demos” at which Doug Engelbart previewed pretty much everything. Whether you call it Language-Oriented Programming or Language Workbenches or something else, DSL-centric programming seems to be experiencing some heat right now.

Four years ago Martin Fowler wrote, “[W]e probably have little idea what DSLs will look like once we’ve had experience with language workbenches.... If language workbenches live up to their hopes, in ten years time we’ll look back and laugh at what we now think DSLs should look like.” OK, we’re almost halfway to ten years. Are we starting to crack a smile?

Michael Swaine is the editor of PragPub.

Android’s Next Move

The significance of the new release of Google’s Android

interviewed by Michael Swaine

Small changes could make a big difference for Android developers.

Google has baked the Donut and it’s time for a taste test.

The recently-released 1.6 version of Google’s Android mobile operating system, based on the Donut track of the project, brings changes that should brighten the eyes of Android developers. Ed Burnette has been scrambling to keep up with the changes—and to keep Hello, Android up to date. We asked him to take some time out to answer our questions about Android: where it stands today and where it’s heading.

ms: Hello, Ed. Susannah says you assimilated all the changes in Android 1.6 in record time. Now I’m hoping you have some free time to chat with me about where Android’s going.

eb: Sure, I’d be glad to.

ms: Let’s get right to it then. How significant for developers is this latest dot release?

eb: It’s a minor release but it could have a major impact on developers for two reasons. First, it includes a totally redesigned Android Market that has the potential of making paid applications much more visible and attractive to the typical user. Second, it expands the potential hardware base of Android by supporting screen sizes larger and smaller than the 320x480 standard that all Android phones (and, I might add, iPhones) have sported up to this point.

ms: Let’s start with Android Market. What’s changed, and why is it so important to developers?

eb: Paid applications will get top billing in the new Market, instead of being relegated to an obscure menu option.

ms: Up to now, paid apps have been sort of ghettoized.

eb: Right. Also, developers can post screenshots showing their apps in action before the user downloads a thing. These two little features, which helped contribute to the success of the iPhone app store, could make targeting the Android platform much more profitable for developers.

ms: I know you’ve given some thought to how to market Android apps effectively. What advice would you give developers?

eb: One of the most important thing that developers can do is to listen and respond to their users. Developers of the most popular apps often put out frequent updates to address problems and ideas reported by their users in comments and e-mail. I’ve seen many cases where a 1-star rating has been turned into a 5-star rating simply by giving the users a little attention. It’s hard, sometimes, because complaints can be harsh. As I wrote in my book, “A thick skin and a sense of humor are invaluable tools of the trade.”

ms: Developer tools didn’t get a lot of respect early on. Has that situation changed?

eb: Actually, I’m not sure if I agree with that. From the first version, Google provided a pretty nice Eclipse plug-in that took care of building and debugging Android applications. In 1.5 and 1.6, they added a way to manage multiple emulator configurations called Android Virtual Devices, or AVDs. The only thing lacking has been a nice GUI layout editor. The community has stepped up tools like DroidDraw, but hopefully in future versions of the SDK, this kind of capability will be integrated into the plug-in. I should point out that a GUI editor is a luxury but certainly not a requirement for Android development.

ms: The situation seems to be getting complicated regarding versions and platforms. Motorola seems to be developing a phone for December release that is based on Android 1.5, version 1.6 is here, and Android 2.0 appears to be in the works, although Google isn’t sharing much information. How difficult is it going to be for developers to track which phone uses which version? What are the implications for software development?

eb: Right now it’s not very complicated. Almost all phones are running 1.5 at a resolution of 320x480. The only challenge has been all the differing input methods. All the phones so far have touch screens, but some have trackballs, some have Directional pads (D-pads), and at least one (the Motorola Cliq) has neither unless you slide out a keyboard.

ms: That will change, though, right?

eb: Yes, by the middle of 2010, there is going to be a mixture of 1.5 and 1.6 or later, and a mixture of screen sizes. The 1.5 vs. 1.6 situation is pretty easy to handle: just stick with the 1.5 APIs so you’ll run on either platform. There’s nothing especially compelling in the 1.6 APIs, so this won’t be a problem for 99% of apps. For the other 1%, they can test the platform version at runtime and only call the 1.6 APIs if they’re running on 1.6.

ms: Then there’s the layout manager/screen size issue. There’s a phone coming out with a smaller screen size, on which apps created using earlier versions won’t work. How big a problem is this, and what’s the solution?

eb: Screen sizes are trickier. The first step is to get yourself out of compatibility mode. When Android 1.6 loads an app that was built for 1.5 it runs it in compatibility mode, which means it will assume the app was coded for a 320x480 resolution and it will stretch or shrink the user interface as necessary to make it fit the real resolution. This doesn’t always work well, especially on smaller screens, to the point that Google decided not to even show 1.5 apps in the Android Market if you have a 1.6 phone with a small screen such as the HTC Tattoo. Luckily, a one-line change in the Android manifest file will take care of that.

ms: So a one-line fix and you’re done?

eb: Not exactly; you still have to actually support all the screen sizes. The key to this is testing. Before releasing your app, try it using different skins in the emulator. A skin defines the width and height of the emulated device along with its pixel density (dots per inch). The 1.6 SDK comes with several skins that correspond to real devices like the Tattoo, the Sholes, and the Archos that will be out in the next few months.

ms: How about Skytone’s Android-powered Netbook? Does that even make sense? Does Android have the oomph to play in that field?

eb: I don’t think an Android Netbook makes as much sense as an Internet tablet. An Internet tablet will sport a touch screen somewhere in the 5 to 7 inch diagonal range, probably without a physical keyboard, with a resolution between 800x480 and 1024x600. If you’re a Star Trek fan, think of the PADDs they’re always tossing around.

ms: OK, there’s a successful technology. PADDs were in general use from the 22nd to the 24th century and throughout the galaxy. Google should be so lucky.

eb: Ha! As long as they don’t get Shatner to do their ads I’m OK with that. But seriously, with the right screen technology, good price point, high speed wireless connectivity and long enough battery life, I think these general purpose devices could define a popular niche. They could obsolete e-readers like the Kindle, and possibly even become standard and required for students, business folks, and starship captains alike. A common operating system and application development platform like Android could finally help make that vision a reality.

ms: What about multi-touch technology? Is Apple going to be able to lock it down legally?

eb: The idea that Apple has a legal lock on multi-touch technology is bogus. Multi-touch existed long before Apple ever brought it to the iPhone, although you have to give them credit for making it available and desirable to the masses. Eventually, I predict all touch-screen devices including Android phones will support multi-touch.

ms: Google seems to be better at making changes than at communicating about them. How can a developer keep up? Read your writings, of course, but doesn’t Google need to provide more information about when new versions will be released?

eb: It’s really hard for a developer to keep up, especially with changes in products and versions that haven’t been announced yet. Google needs to realize that developers need to make plans for the future, which means we need some predictability in basic things like release schedules. This is where Google could learn a lot from the Eclipse Foundation, which for many years has kept to a yearly rhythm of milestones and new versions. This lets all members of the ecosystem time their efforts and releases around the platform.

I decided I’d better let Ed get back to researching and writing about Android, which is no small task. Android 1.6 includes many changes that we didn’t touch on here. Search, in particular, has been significantly enhanced. There’s support for VPNs, and user interfaces to several tools have been improved. There’s the Pico multilingual text-to-speech engine. There’s the GestureBuilder tool and the Gestures framework for adding gestures to your app. There’s a new Accessibility framework that lets you make your app more, uh, accessible. All good. But the big news is Android Market, which now should start to give Apple’s iPhone App Store some competition.

If you haven’t already done so, you can download the Android 1.6 SDK here. More and more devices supporting Android 1.6 are hitting the streets.

Thanks to Susannah Pfalzer and Jürgen Fey, who contributed to this interview.

Ed Burnette is a software industry veteran with more than 25 years of experience as a programmer, author, and speaker. His development credits include everything from commercial video games to high-performance grid computing systems. Ed is a founding member and senior researcher at the SAS Advanced Computing Lab.

Ed has authored numerous technical articles and books, including Hello, Android from The Pragmatic Programmers. He writes the Dev Connection blog for ZDNet, and is the creator of Planet Android.

In his copious spare time, you can often find him playing video games with his two sons at their home in central North Carolina.

Mail on Rails

Three Scenarios for Cloudsourcing Your E-mail Services

by Dave Hoover

Dave Hoover was instrumental in Michael Koziarski's revamping of our email system at Pragmatic Programmers.

The design and delivery of HTML email is a pain. It’s also a solved problem, so why not use the existing Web APIs to do the heavy lifting? In this article I work through three quick scenarios where a Ruby developer can outsource email functionality by integrating with an email marketing service.

There are many existing email marketing services with APIs, such as Constant Contact, iContact, and MailChimp. In this article, I’m going to show examples of integrating with Mad Mimi, a more recent player in the email marketing space. The principles here should work with any service; I’m using Mad Mimi because it makes it more concrete to use a specific tool and, well, because I know Mad Mimi very well. I’m the lead developer.

The first two scenarios focus on a typical Rails application, and then the final scenario shows general-purpose integration.

Scenario 1: Out of the HTML Business

If you’re ready to throw your failed attempts at email design out the window, or you’re starting a Rails app from scratch and don’t have the budget to put any research and development into making your HTML emails look perfect, you can set up the emails using the WYSIWYG-style tools that many email marketing services provide. Ideally, someone other than you (like, someone non-technical) can take on this task. These services have already gone through the effort of creating an HTML template that is optimized for all known email clients, and in most situations, there is no compelling reason for you to reinvent that wheel.

The simplest way to accomplish this integration with the Mad Mimi service is via the mad_mimi_mailer Ruby gem. This gem extends ActionMailer::Base to give you a remarkably familiar API.

| class UserNotifier < MadMimiMailer
| def mimi_welcome(user)
| subject "Welcome to WidgetHub"
| recipients user.email
| bcc ADMIN_PEEPS
| from "system@example.com"
| body :username => user.name,
| :email => user.email,
| :password => user.password
| end
| end

Let’s walk through this example. First, instead of inheriting from ActionMailer::Base, we’ve inherited from MadMimiMailer. You would have needed to install and configure the mad_mimi_mailer gem in your environment in order for the MadMimiMailer class to be available. You would have also needed to configure your Mad Mimi API credentials so that the MadMimiMailer can authenticate to the web service. This is unsurprisingly simple:

| MadMimiMailer.api_settings = {
| :username => 'dave@example.com',
| :api_key => '75a6c6f1SomeFakeApiKeycbcc300'
| }

The rest of the method looks exactly like you’d expect an ActionMailer::Base class to look. Then, what actually happens when you call UserNotifier.deliver_mimi_welcome(current_user)? First, the mimi_ prefix tells MadMimiMailer to step in and handle the delivery. Without that mimi_ prefix, the delivery is handled like a normal ActionMailer::Base. MadMimiMailer grabs the rest of the method name welcome, passes that name to Mad Mimi along with the rest of the data you’ve passed in, and sends your welcome promotion to the recipient you’ve specified.

The biggest difference between a typical ActionMailer::Base and a MadMimiMailer is the body hash. Typically, the body method takes a hash of any sort of Ruby objects, often ActiveRecord::Base instances. But when you’re calling out to Mad Mimi, you need to pass in a simple hash of strings. This will, naturally, evolve over time, and as Mad Mimi’s Mailer API grows in popularity, more powerful templating features will be supported, but for now, the keys of the body hash will expect to match their counterparts inside {curly} braces in your Mad Mimi “promotion.” So, for the example above, you would expect to find {username}, {email}, and {password} somewhere in the body of the content. The values need to be plain old strings.

That’s how you do it with Mad Mimi, and you could do something similar with any other email marketing service that supports this functionality. The beautiful thing about this approach is that once it’s set up, non-technical people can change the design and content of your emails without involving developers. They just need to be careful of the {placeholders} and not rename the promotion.

Scenario 2: Sticking to Your Guns

If you’re comfortable designing excellent HTML emails, or have someone who can do it for you, there are still some good reasons to outsource the delivery of those emails to a service.

First and foremost, while your site is under development, or even in the first few months while it is still small and sending relatively few emails, all of your emails will magically appear in inboxes instantaneously. But as you grow, and you’re sending hundreds, thousands, and tens of thousands of emails a day, suddenly your deliverability will begin to suffer unless you start learning the black magic of email deliverability. Unless email is the core of your application, email deliverability is a distraction and should be handed off to a service that has already solved these tough problems for you.

Secondly, one of the most important features of an email marketing service is statistics. Standard stats include reads, clicks, opt-outs, forwards, and bounces. You get all of this information automatically from the email marketing service.

Here’s the Mad Mimi approach. Given that you want to stick with your standard ERB-based emails, simply let MadMimiMailer know:

| class UserNotifier < MadMimiMailer
| def mimi_welcome(user)
| use_erb true
| subject "Welcome to WidgetHub"
| recipients user.email
| bcc ADMIN_PEEPS
| from "system@example.com"
| body :user => user
| end
| end

In this case, we call use_erb to tell MadMimiMailer to use the typical Rails ERB template, mimi_welcome.text.html.erb, to send the email. This means we don’t need to have any emails set up within the service beforehand (like we did in the last example). Also, since we’re using ERB, there’s no longer any reason to flatten out the user object into strings, so we simply pass in the object for use in the template.

One less obvious requirement of using the ERB approach is that you need to include the following placeholder somewhere in your template:

[[peek_image]]

Typically, people wrap that image in an invisible p tag:

| <p style="visibility:hidden;" title="Web beacon">
| [[peek_image]]
| </p>

This placeholder is used by the service to jam a little img tag into the email, which triggers a request back to the service, letting it know that the recipient opened their email, which is obviously critical for gathering statistics about your users’ behavior.

Scenario 3: Just the Facts, Ma’am

If the MadMimiMailer feels too magical, or you need lower-level control over the call to the API, it’s relatively straightforward to drop down to Ruby’s net/http library to make the call. If I wanted to mimic the core behavior of my first example, the underlying code would look like this:

| body = {
| :username => user.name,
| :email => user.email,
| :password => user.password
| }
| params = {
| 'username' => 'dave@example.com',
| 'api_key' => '75a6c6f1SomeFakeApiKeycbcc300',
| 'promotion_name' => 'welcome',
| 'recipients' => user.email,
| 'subject' => 'Welcome to WidgetHub',
| 'bcc' => ADMIN_PEEPS,
| 'from' => 'system@example.com',
| 'body' => body.to_yaml
| }
| url = URI.parse('https://madmimi.com/mailer')
| request = Net::HTTP::Post.new(url.path)
| request.set_form_data(params)
| http = Net::HTTP.new(url.host, url.port)
| http.use_ssl = true
| response = http.start do |http|
| http.request(request)
| end

There’s nothing very fancy going on here. The only out-of-the-ordinary detail is that I’m serializing the body hash into YAML before I post the request. If I wanted to use my own HTML rather than a pre-existing email that’s been set up in the service, I would simply swap out the body parameter with the raw_html parameter, whose value is, not surprisingly, a bunch of raw HTML (this is what use_erb uses in MadMimiMailer) instead of a hash. And just like in ERB-based approach above, you’ll want to ensure that [[peek_image]] is included in the HTML string.

This Mailer API from Mad Mimi is just the tip of the iceberg of what email marketing service APIs can take off your plate. Other API features include campaign statistics, list importing, list sync’ing, and list management. We’re living in a time when an increasing number of peripheral services can be outsourced from our projects, allowing smaller teams to focus more intensely on the heart of the problems they are trying to solve.

In this article, I’ve focused on email design and deliverability, which is just one of many services that are experiencing extremely rapid uptake in our industry, such as source control (GitHub), continuous integration (RunCodeRun), functional web tests (Sauce Labs) and of course, our CPUs (Amazon EC2) and hard disks (Amazon S3). I hope this article is the beginning of the end to your email struggles!

Some useful links: Mad Mimi Mailer Ruby gem, Mad Mimi API, Reviews of top 10 email marketing services.

Dave authored the book Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman for O’Reilly, instigated the Software Craftsmanship North America conference, and is the Chief Craftsman at Obtiva. Dave began teaching himself to program in 2000, back when he was a family therapist. Dave lives near Chicago with his wife and three children. In his spare time, Dave competes in endurance sports.

TextMate Tricks

An Automation to Produce PDFs

by James Edward Gray II

Make your editor work for you—and work hard.

The people I work with are often surprised by how fast I crank out the tedious tasks. At the risk of ruining my image, I must confess that I’m really just a compulsive user of TextMate’s automation features. What continually surprises me is how easy it is to teach TextMate new tricks, customized to my personal tastes. I spend a little effort up front when embarking on some new project and TextMate pays me back over and over again as I do the real work.

For example, let’s say I decided it was time to branch into writing fiction. Most people would start by looking into all of the software available for authoring fiction, but I start by thinking about how I’m going to get TextMate ready for this new venture. I’m already invested in TextMate and it is now super painful for me to write anything without it. When typing an email into Apple’s Mail.app, I regularly highlight some text, type a quotation mark, and freak out when half of my message is replaced. Composing a larger work in anything dumber than TextMate would just be far too stressful for me.

I bet most of you are thinking that I’ve already lost the plot. I mean, let’s face it, aside from saving a simple text file, our output options kind of suck with TextMate. How are we going to compose something we wouldn’t be embarrassed to show to a publisher? We could resort to a hardcore geek tool like LaTeX, but then we might end up focusing more on the format than the content. Well, I say we’ve just found the first automation we are going to need. Let’s teach TextMate how to produce decent PDF files from our simple content.

I know, I know. Publishers aren’t clamoring for manuscripts in PDF format. I’m talking about producing something professional-looking for the elevator pitch. And, frankly, something that makes me feel like this prose is really publishable.

A Novel Approach

Let’s assume I have my novel project started, and that it looks, at this point, something like this screen shot:

What I need is a command that can read through the chapters and transform them into a pretty PDF. Let’s make that happen.

We’ll surely need more than one command, though, so let’s build a bundle, TextMate lingo for a group of related automations, for these fiction commands. In the Bundles menu I’m going to choose Bundle Editor and then Show Bundle Editor. Then I’m going to click and hold the plus button in the lower left corner of that window and select New Bundle. After that, I can just type in the name of the bundle I am creating, which is Fictional Work in this case.

Now that I have a place to hold commands, I’m ready to write one. Of course, I have no desire to build a PDF manually. It’s time for a quick cheat. I’m going to do my scripting with my favorite scripting weapon, Ruby, which has a powerful PDF generating library called Prawn. We need to make sure we have a recent version of that before we ask it to do the heavy lifting for us, so I’m going to bounce over to the Terminal and run the install command:

sudo gem install prawn

Now let’s head back to TextMate and the Bundle Editor so we can build a command. First I’m going to click on the Fiction bundle to make sure it’s highlighted, then it’s back to the plus button in the lower left, and this time I’ll select New Command. I’ll name this one Build Fictional PDF. I also need to initialize a few settings for this command. First, I’ll flip Input to None since the command will just read everything from the chapters directory. Output needs to be Show as HTML, because that’s the easiest way to stream some progress information to the user as we work.

We’re now ready to write the actual code. Remember how I said I can’t really type outside of TextMate’s main editing window though? Well, that includes the Bundle Editor. I’m just going to click my cursor into the Command(s) box and then go into the Edit menu after the Edit in TextMate command. If you don’t have that one installed, just ask TextMate to add it for you by opening the Bundles menu and selecting TextMate and then Install "Edit in TextMate".

We’re finally ready to write some Ruby:

| #!/usr/bin/env ruby -KU
| require "pathname"
| require "fileutils"
| require "yaml"
| require "erb"
| require "#{ENV['TM_SUPPORT_PATH']}/lib/textmate"
| require "#{ENV['TM_SUPPORT_PATH']}/lib/web_preview"
| require "rubygems"
| require "prawn"
| require "prawn/format"
| # Figure out paths, title, and user's name
| PROJ_DIR = Pathname.new(ENV["TM_PROJECT_DIRECTORY"])
| CHAPTERS_DIR = PROJ_DIR + "chapters"
| PDFS_DIR = PROJ_DIR + "pdfs"
| FILE = PDFS_DIR +
| "#{Time.now.strftime('%Y-%m-%d-%H-%M-%S')}.pdf"
| TITLE = PROJ_DIR.basename.to_s
| USER = %x{dscl . read /Users/$USER realname}.
| sub(/\A\S*realname:\s*/, "")
| # Configure options
| options = (yaml = PROJ_DIR + "config" + "pdf.yaml").exist? ?
| YAML.load(yaml.read) : { }
| margins = { :top_margin => options.fetch("top_margin", 72),
| :left_margin => options.fetch("left_margin", 72),
| :right_margin => options.fetch("right_margin", 72),
| :bottom_margin => options.fetch("bottom_margin", 72) }
| font_size = {:size => options.fetch("font_size", 10)}
| font_size[:spacing] = options.fetch("spacing", font_size[:size] * 1.5)
| header_font_size = { :align => :center,
| :size => font_size[:size] * 2,
| :spacing => font_size[:spacing] * 2 }
| # Build the PDF
| html_header("Fictional PDF Builder")
| puts "<ol>"
| Prawn::Document.generate(FILE, :skip_page_creation => true) do
| start_new_page(margins.merge(:top_margin => 360))
| font(options.fetch("font", "Courier"))
| text("<b>#{TITLE}</b><br/>by #{USER}", header_font_size)
| CHAPTERS_DIR.each_entry do |chapter|
| if (chapter = CHAPTERS_DIR + chapter).file?
| puts "<li>Processing #{chapter.basename}</li>"
| start_new_page(margins)
| text( "<b>#{chapter.basename('.txt').sub(/\A\d+\s+/, '')}</b>",
| header_font_size )
| text( chapter.read.gsub("\n", "<br/>"),
| font_size.merge(:align => :justify) )
| end
| end
| end
| puts "</ol>"
| FileUtils.ln_sf(FILE, PDFS_DIR + "#{TITLE}.pdf")
| # Report success and finish up
| html_safe_path = ERB::Util.h(FILE.realpath).gsub("'", "&#39;")
| puts %Q{<p>Done: <a href="javascript: } +
| %Q{TextMate.system('open \\'#{html_safe_path}\\'', null);">} +
| %Q{open PDF</a>.</p>}
| html_footer
| TextMate.rescan_project

I’m sure that looks like a lot of code, but you may be surprised by how boring it really is when you give it a good look. I start by loading several libraries to save me work: four standard libraries that ship with Ruby, two sets of TextMate helpers, and then Prawn.

After that, I set a slew of constants and variables. I read the project path out of an environment variable that TextMate sets up and use that to build the paths I will need, as well as to figure out the project’s title. Then I ask the operating system for the name of the user. Finally, I read some options out of a YAML file, if it exists, and prepare a bunch of settings from that.

The next section is the real meat of the code. Using Prawn’s nice DSL, we construct a good-looking PDF. I’m using the Prawn Format extension in here, which allows for some basic HTML-like tags for bold and line-breaks. Note that I’m also writing status messages to $stdout as I work. I’m using HTML there too, and since I called the helpers html_header() and html_footer(), my output will be styled like a normal TextMate command.

The last chunk of code adds two important finishing touches. First, it builds a link to the generated PDF. I could have shown that file in the TextMate output window with a normal file path, but I decided it would be nicer to open the document in the system’s preferred PDF reader. To do that, I make use of the JavaScript function TextMate makes available to me for executing some shell code: TextMate.system(). The other important task is to use one last helper to get TextMate to rescan the project. This will make sure it notices the files I’ve added to the project.

Now, I made it possible to configure things like margins and fonts for the PDF with a YAML file. It doesn’t exist yet and the code will run fine without it, but let’s add one more shortcut for that. This time, we don’t need any code and we can just build a snippet. A snippet is a chunk of boilerplate code we can customize to our needs.

Click on that plus in the lower left of the Bundle Editor one more time and choose New Snippet this time. Give it a name like Configure PDF Builder and replace the examples with this code:

| ---
| # Set margins below. These are in PDF points (72 per inch).
| top_margin: ${1:72}
| left_margin: ${2:72}
| right_margin: ${3:72}
| bottom_margin: ${4:72}
| # Set font, size, and line spacing below.
| font: ${5:Courier}
| font_size: ${6:10}
| spacing: ${7:15}

This is just a trivial YAML document with some placeholders in it. The placeholders are the elements like ${1:72} where 1 is the placeholder’s order and 72 is the default value. When we trigger this snippet, TextMate will allow us to tab between those values in numerical order, replacing them with whatever we type. Try it out. Close up the Bundle Editor, create a file called pdf.yaml in config directory, and then go into the Bundles menu and select Fiction followed by Configure PDF Builder. Tap the tab key a few times and watch how your cursor jumps around. You can also use shift-tab to go back to previous values.

When things are configured the way you want them, it’s time to check out the rest of our hard work. Take another trip into the Bundles menu and choose Fiction then Build Fictional PDF. You should see the command output window pop up and run through your chapters. Then you can click the link to view the output. You should see a title page and sections for each chapter.

Further Automations

This is just one tool that would help us write fiction—among other things—with TextMate. There are a lot more we could build, more targeted to fiction. Personally, I would like to have a way to insert notes in the text that I can view as needed, but that wouldn’t really be a part of the story and wouldn’t print. I would also like a way to autocomplete character and place names no matter which file I’m in (overcoming a limitation of TextMate’s built-in autocompletion). It would also be neat to find a way to storyboard the chapters and allow for easy rearranging. I’m sure you could dream up more automations that would be handy as well.

That’s what makes TextMate or any other editor we can customize great, though. We can add on the exact tools that help our specific situation the most. Give it a shot. You might just be surprised how big the payoff can be.

I’ve placed the bundle created in this article on GitHub in case anyone wants to extend it some more. I should also mention for fiction authors that there is a Screenwriting bundle for TextMate.

James Edward Gray II is a Ruby and Rails programmer by day. However, he often spends hobby hours helping to maintain TextMate's Ruby integration. This gives him an ongoing excuse to play with his two favorite toys pretty much all the time. He is the author of TextMate: Power Editing for the Mac.

State Secrets

Or, How I Stopped Worrying and Learned to Love Workflow Modeling

by David Bock

State machines model workflow and get the Russians and the Estonians talking.

“We will not tell the delegate from Russia how we manage our export control process in Estonia,” the Estonian delegate said at the 1997 Conference on Export Controls in Oxford, UK, and thus began my journey into the large and complex world of modeling workflow. From 1996 through 2007 I worked on an import/export control system developed by the US State Department that was given to former Soviet Union countries to help them control their borders.

There is a lot of stuff sold around the world that can be dangerous in the wrong hands. Riot control gear manufactured in Lithuania being sold to a police force in Latvia is probably a legitimate sale. That same gear going to a tribal leader in Afghanistan might not be. Brewery equipment being shipped from Germany can be used to make beer in one person’s hands, or weaponized anthrax in another’s. The same equipment that can make carbon fiber golf clubs can be repurposed to make nose cones for warheads. In the international trade community, these are called “dual-use goods,” and under many circumstances their import and export is controlled; companies have to apply for licenses to move the material across international borders into “areas of concern.”

When the Soviet Union dissolved, the newly independent nations found themselves with all kinds of “interesting” pieces of military hardware and manufacturing facilities, weak economies, and smart people looking for employment. It was in their interests to be able to position themselves on the world market, and it was in the US interest to make sure they followed international treaties on the sale of this stuff.

Tracker

In 1996, my company was hired to build a system to help track the licensing, approval/denial, and transport of these goods. While we could get cooperation from the involved governments most of the time, there were plenty of roadblocks to sharing information about their internal processes. Writing software when you know what needs to be done is hard enough, but when your end users won’t tell you what it needs to do, it becomes almost impossible.

There were many interesting technical challenges on that project—from the translation into several Eastern European languages through the output of legally binding paperwork in each country—but the most interesting challenge was trying to model workflow that varied from country to country, when half the time we couldn’t know what the official workflow even was.

This forced us not to solve their particular problems, but to build a generalized solution that could be tailored to their workflow. The system we built is called Tracker; it was refined over an 11-year period and is in use today, protecting the borders of several countries. (More information is available at www.trackernet.org). Key concepts developed under this program have been used in a modernization effort for the US import/export control and in the United Nations Effort on Humanitarian Demining.

Modeling the Impossible

We wrote that tool in Java. When we started, Java was relatively new on the technical landscape. There were no vendors selling workflow solutions, so the work we did was groundbreaking on many levels. In particular, we chose to model the complexity of their workflows as a State Machine. Years later, our customer told us that he “never actually expected the software to succeed, [he] just wanted it as an excuse to get the delegates from the countries cooperating with each other.” A primary reason we succeeded was because of the open-ended nature of state-based workflow modeling.

Today, most vendor workflow solutions are process-based, not state-based. Marketing literature would have you believe that process-based is somehow superior. In truth, both methodologies have their purposes, and they can be used together to model very complex workflow situations. The work we did on Tracker took state-based workflow modeling further than I have seen it taken elsewhere, and into a proprietary Java-based workflow tool called Journeyman. Since then, I have taken these concepts into other environments to solve other workflow-related problems, and this has grown into a comprehensive methodology for workflow modeling called StonePath. These projects have led to the creation of the StonePath project, a state-based workflow modeling tool for Ruby (open sourced under the MIT license and available on github). While the StonePath reference implementation is in Ruby, the modeling concepts can be useful in just about any modern language.

Basics of StonePath Modeling

Almost everything I needed to know to start modeling workflow, I learned from Schoolhouse Rock. I am probably betraying both my age and my nationality with that statement, but Schoolhouse Rock was a series of educational cartoon/music videos that aired Saturday mornings in the US from the mid 70s to the early 80s. One of the cartoons taught children How a Bill Becomes a Law in the US Government. In that cartoon, a singing and dancing paper scroll (named Bill, of course), describes how he becomes a law. “First, I’m just a good idea that someone tells their congressman,” he says... and he goes on to talk about how he is in committee, then voted on, how he might get vetoed, put up for a revote, and so on. We are anthropomorphizing the bill, and by doing so we find out about the different “states” he can be in and how he moves from state to state. “Anthropomorphizing” an object—imagining your object as an agent with its own desires and actions as it goes through a workflow—is a great way to start planning your workflow. We will use that technique in an exercise below.

Your Boss’s Boss

In one exercise with a group of business analysts defining a system for managing legal actions, I said, “Imagine that this is a high profile legal action that the New York Times is doing an exposé on. Your Boss’s Boss is calling you every hour to find out what’s going on, and is expecting an answer in five words or less. What are you going to say to her?”

“It’s in Data Entry.”

“It’s under investigation now.”

“It’s on Todd’s desk, awaiting legal review.”

“We need to verify the investigators’ findings.”

These also point to some of the things we need to think about when defining a state-based workflow, but there is more here than just states... there seem to be actions to be performed, assignments to particular people, and so on.

These two techniques—“Anthropomorphizing the Work Item” and “Your Boss’s Boss”—are two important requirement elicitation techniques for modeling a state-based workflow. We will see these techniques used as we learn to model state-based workflows.

The major components of StonePath workflow are:

The WorkItem

The WorkOwner

The Task

The WorkBench

We will discuss each in turn.

A Real-World Example

Let’s start with a real-world example; something you probably wouldn’t really model in software, but something that we all should have some experience with—a trip to the grocery store.

Tammy needs to do some grocery shopping. She knows she needs to get something for dinner tonight, has a prescription she needs to get filled, and probably needs to get a few other things while she’s there. How does this relate to workflow?

First, Tammy is going to do some planning for her trip. She might be the kind of person who goes to the store with a discrete shopping list and a bunch of coupons, or she might be the kind of person that just thinks “I feel like making spaghetti tonight” and wanders around the store picking stuff up. We know she is going to get a prescription filled, and pick up a few other things while she’s there.

Second, she actually needs to go to the store and buy her stuff.

Third, she finishes shopping, comes home, puts stuff away, and makes dinner.

Believe it or not, that is a great first-pass at a workflow we can model. It starts out looking like this:

That is very oversimplified, but once we have a notional “3-5 bubble” diagram, we can start to hang ideas off of it as we elicit business requirements.

What happens in that “shop” bubble? Well, we know that Tammy has a prescription to fill. Most people would go to the pharmacy, drop it off, and continue to shop while they wait for it... There is no reason to stand by the pharmacy for 15 minutes.

Most shoppers know that you don’t put ice cream in the shopping cart until you are almost done shopping—Tammy certainly won’t want to put it in while we have 15 minutes to wait for her prescription; it might melt. Taking this a little further, she’d probably plan to get all of her dry goods on her shopping list first, followed by perishable refrigerated things (milk, eggs, veggies), followed by frozen stuff.

What does our “shop” bubble look like now?

Wait! What about picking up the prescription? If you noticed that, I’m proud of you—you will have the kind of eye necessary for debugging workflows. Let’s save the prescription talk for a little later... after all, that isn’t so much a “state” we are in, so much as a “task” the pharmacist is performing and one we need to do when he’s done.

Long-Running Tasks

At my grocery store there is a deli counter that often has a long line. They recently installed a touchscreen kiosk that lets people place their order, go do their shopping, and come back later for their deli meats and cheeses. This is a lot like the prescription-filling... I can start a long-running task, go deal with the other states in my flow, and come back to it later. so let’s rename that first bubble to “start long-running tasks.” Again, we’ll talk more about actual tasks later.

So there we have it—a simple state-based workflow that models Tammy’s shopping trip. It doesn’t model every single detail... depending on what we wanted to model, we could easily think of other states to add. But let’s leave that for later iterations of refinement. We still have other details to talk about.

The Lie That Teaches You the Truth

“State machine purists” reading along right now are probably muttering to themselves, “these aren’t states he’s modeling!” And they are right. I have led you down this path because, in dialogues with clients, this is how it typically happens. We talk about states, but in truth, the clients are often more interested in talking about the transitions that put them into states rather than the states themselves. This is an important part of this as a methodology—let the client speak in their language, and slowly bring it around to the model we need. What does this mean to the work we have done?

States vs. Transitions

In the classical computer science concept of a “State Machine,” states are connected by a series of transitions. In order to move from one state to another, there must be a transition connecting them. If there is no transition, there is no way to move from the first state to the second state.

In workflow modeling, we want those transitions to have names. The best names for these transitions are action verbs, and this is why it’s easy to get the client to talk about transitions over states. Some examples in common workflows are “approve,” “deny,” “escalate,” “decide,” “perform,” “open,” “close,” “authorize.” One nice side-effect for the design of software systems is that these transition names often appear as menu choices or button names. They work well for that.

States are actually the uninteresting time period in between those actions where things are sitting in people’s inboxes, something is waiting for approval, and so on. We often have to extract those names or derive them from the workflow. Let’s remodel our state diagram keeping this in mind:

See how those transitions are all actions? Start, get, finish... that is a good test that we are modeling that correctly. While in each state, things are happening to close that state out in preparation for the next transition. For instance, while in the dry_goods_phase, we will be picking things off the shelves in the grocery store and putting them into our cart.

See how the state names can all be used to answer the “Boss’s Boss” question? “We are in the frozen goods phase of our shopping trip now.” That might not sound much like the way you would normally describe a shopping trip, but if we were modeling an approval process for license applications, we could probably identify states that map back to domain terms.

There are several simple suggestions for good state names:

start state names with “positional” terms like “in,” “on,” or “under,” such as “in process,” “in final review,” “on hold,” or “under investigation.” Or,

end state names with terms that imply a temporary condition, like “phase” (like we did above). Or,

use past tense actions, like “done,” “reviewed,” or “countersigned.”

Do not use action verbs—these best describe transitions.

Do not use or imply time or duration in your names, like “3-day waiting period.” Time and trigger events are another concept in this modeling methodology.

States often have pretty generic names like “in-process” and “pending,” or past-tense verbs of the action that brought them to this state, like “decided” or “activated.” When possible, a more descriptive name is better, but when in doubt, the descriptive names are probably the names of transitions.

Exactly What is our WorkItem?

In modeling workflows, we model a “Work Item”—our object with state. Exactly what are we modeling here?

In all object-oriented software construction, we often end up modeling concepts. While we might have a tangible thing in some kinds of modeling (like our Bill above—or if we were modeling a photographer’s workflow for a photograph), sometimes the thing we model is an intangible. In this example, our intangible workitem being modeled is a ShoppingTrip.

The WorkOwner

In the StonePath modeling methodology, every WorkItem has one and exactly one WorkOwner. In this example, it is Tammy. If Tammy had instead given her husband Chris the shopping list and said “You’re doing the shopping today,” then that would be a “reassignment of the workitem” to Chris. At this point, Tammy technically has no say in how the shopping is done. If Chris decides he wants to pick up some Oreos, that is Chris’s call.

So what about the pharmacist and the deli meister? What are they in relation to this workflow? They aren’t responsible for the shopping list, but they are playing some role in this whole shopping experience. In order to understand that, we have to model what they are doing.

The Task

In the execution of any workflow, there are “tasks” that people are going to perform. The pharmacist has a prescription to fill. The Deli meister has a pound of turkey and a half pound of provolone cheese to slice. Tammy is going to pick up some ice cream. These are Tasks in the workflow.

In addition to the states we model, we also need to think about the tasks that fire off while in these states. We can see now in the bubble model, the “start long-running tasks” starts to make some sense... these tasks need to start then, but they don’t necessarily need to finish then. They can finish anytime between now and the time Tammy checks out.

Tasks have a life of their own. A task can be:

“pending.”.. that is, it hasn’t started yet.

“in-process,” as in “it has started, but we are still waiting for it to finish.”

“completed,” as in “you can come pick up your prescription now.”

...and several other states related to it being cancelled, expiring, and so on.

Hmm... this starts to look like another state machine. Each task has its own state (although a very simple and limited one).

The interesting thing about tasks is that they can also be assigned to people (well technically, “agents”—we could assign them to other computer programs for processing). Our prescription task is assigned to a pharmacist, our deli order task is assigned to the deli meister, and so on. A task belongs to both the workitem that created it and the user (actually “workbench”) that is assigned to handle it.

You might begin to see what the other tasks are for the other bubbles... “Get Oreos,” “get Eggs,” “get Ice Cream” are the kinds of tasks we will start in the other bubbles.

The WorkBench

The assignment of a task to a person (or agent) is done through a WorkBench. You might also think of this as an “in box” for assignments. Workbenches can be private (as in “all pharmacy orders for narcotics need to go to the senior pharmacist on duty”), or can be shared amongst a group of people (as in any pharmacist on duty can fill a prescription for antibiotics). In simpler workflows, it is common to model the agent as the workbench. In more complex workflows, it is useful to have this separation so that tasks can be assigned to groups of people.

WorkItem, WorkOwner, Task, WorkBench are the major ingredients of State-Based Workflow.

While there is a lot more in the details of the StonePath Methodology, these four concepts make up the backbone that, once understood, allow developers, business analysts, clients, and users to communicate to define the user stories for the development of the system. The rest of StonePath are details that fall out of some of the implications of the interaction of these four elements.

Note how some nice features fall out of this model... If Chris decides to go to the store with Tammy, she can start tasking him: “Go get the lettuce while I go get the eggs.” This is modeled as a task assignment to Chris’s workbench. I always like when correct behavior I didn’t explicitly plan for comes out of a model; it feel like the model is “correct” on some level.

State Guards

As our workflow stands now, technically, it is possible for us to finish our shopping and get to “done” before our prescription is filled or our deli order has been completed. While we seem to have a good start, there are clearly some problems with this model. What we need is a mechanism of saying “Don’t transition into the ‘done’ state until the long-running tasks are complete.” In fact, we might not even want to start shopping for our frozen goods, just in case those tasks take a long time.

This is the concept of a State Guard. In order to build a complete state-based workflow, we need the ability to say things like “Don’t enter this state if some other condition isn’t true,” or even “Don’t leave this state unless some condition is true.” That is, we want to guard the entry and exit of states with Boolean conditions. For now, let’s just draw a little lock on our diagram that says this for the frozen food state.

This all seems pretty simple at this point, really... We are modeling things that you might already do without thinking about it—“Don’t put the frozen food in my shopping cart if I’m likely to be here a while.” We are just modeling it with some explicit concepts.

Parallel and Sequential Activities

In any workflow, state or process based, we need to think about what activities happen sequentially and what activities happen in parallel. In our state-based modeling, these concepts map quite naturally to states and tasks... Our states are sequential—we move from one to the other in a logical, defined order. Our tasks, on the other hand, can happen independently of each other unless we explicitly tie them together (our deli order and prescription can happen independently, as can our dry and refrigerated goods... but all those happen before we get the frozen goods).

This is one of the strengths of the state-based workflow modeling—for me at least, the activities that can happen in parallel map naturally to my thought processes. Defining parallel, independent activities means that our workflows can scale by adding more resources to the problem. That is, if Tammy and Chris brought their daughter to the store too, she’d be able to go get stuff, making the whole trip go faster.

Multiple Paths

Our simple state machine had a linear path through the states. There is no reason why this has to be the case; plenty of state machines have alternate paths where one state can branch into one path or the other, where states can point back to previous states, and so on—the key to understanding how this works is by realizing that, while there are many paths, the WorkItem is still only in one state at any time. We will see some parallel activities shortly.

Long-Running Tasks—A Closer Look

From Tammy’s perspective, she gave a task to the pharmacist—“Fill this prescription.”

From the pharmacist’s point of view, this task creates a new WorkItem... That is, he is going to be the Owner of a WorkItem that will have its own flow.

There are tasks to this flow (check customer’s insurance, check prescription for drug interactions, fill bottle with pills, bill insurance, notify customer prescription is ready for pickup), and may have a workflow that varies from simple (in-process - done, to something involving complexity surrounding the distribution of controlled substances).

Likewise, the deli meister may have a workflow for their task. The register clerk may have a workflow associated with cash vs. credit, bagging the groceries, and so on. Each task might simply be something that is “done,” or could spawn a whole sub-flow to the parent flow of “Tammy going shopping.” The best part is that the encapsulation, and communication through tasks and workbenches, keeps each workflow isolated. Tammy’s plans for shopping didn’t have to change because the pharmacist has new procedures for billing insurance companies. Again, this sounds like common sense coming through in our design—but how often have you seen poorly coupled software that wouldn’t allow one thing to happen because of some other, seemingly unrelated thing?

Let’s take a closer look at the pharmacist’s workflow for filling a prescription. What does a pharmacist do?

First, he has certain safety checks he’s going to perform. He’s going to make sure the prescription isn’t forged, he’s going to check the customer’s records to see other prescriptions they have filled to make sure there are no dangerous interactions, he will make sure they haven’t filled the prescription recently, and he’ll check with the insurance carrier to make sure their prescription card is valid. These can pretty much happen in any order, and could even be given to pharmacy assistants to perform.

Second, assuming everything is OK, he will fill the prescription. Depending on the medicine this might simply be a prepackaged bottle, a liquid that needs to be reconstituted, or pills that need to be dispensed into a pharmacy bottle. He will also print a label for the bottle with handling instructions for the medicine like “take 2 pills twice a day with food.” Again, there are two steps here that can happen in any order once we know we are filling the prescription. If there is a problem with the prescription, the pharmacist will call the customer back to handle it.

Third, after the prescription has been filled, they will notify the customer. When the customer returns they will sign for the prescription and be given a chance to ask the pharmacist any questions about the medication.

Let’s model this workflow:

We have something new in our workflow—we have taken multiple branches. But something isn’t quite right yet. Do you see it? Again, we have started by modeling the interesting activities (tasks and transitions), not the states. Look at the names—“perform,” “fill,” “notify,” and so on. Those are clearly action verbs. These don’t pass our “Schoolhouse Rock Test.” If our prescription were a singing, dancing piece of paper, he’d be more likely to say “I’m waiting for the pharmacist’s approval,” and “I’m waiting for the client to pick me up.”

With these naming conventions in mind, let’s redraw this workflow:

Note that with this refined version, we have a new arrow... from customer input, the flow can go back to “pending pharmacist approval.” Perhaps the insurance denied the prescription and Tammy said, “Fine—I’ll pay for it myself.” At that point, the flow goes back to the pharmacist where it will either be approved, or the next issue is brought to the attention of the customer. The “pending pharmacist approval” bubble will have many of the tasks we talked about that can happen in any order... check the insurance card, check for dangerous drug interactions, make sure the prescription isn’t forged, etc.

Transition Guards

Just as we had guards on the entry and exit of states, we can also guard explicit transitions. This most often occurs when there are transitions that can only be performed by certain roles with authority... for instance, on checkout, a cashier who is not of legal drinking age might not be able to sell alcohol. In this situation, a manager can authorize the sale (the transition from pending sale to sold).

The Role of Time in Workflow

When we model workflow, we often have time constraints on some of the things we need to do. Often, people try to model this into the states themselves, like “3-day waiting period” as a state name. Incorporating time into the state name is generally a bad idea; you end up with a proliferation of states for rule variants, like a “2-day waiting period,” “5-day waiting period,” and so on. If we “anthropomorphize our workitem,” we quickly see that the workitem is the thing that should be managing its time.

In our pharmacist’s workflow, we might have a rule that “prescriptions waiting for pickup more than 3 days trigger a phone call to the customer.” In order to do this, we don’t create new states for “waiting customer pickup after late notification.”.. we add that rule to the prescription workitem. We add a method named something like “pickup_overdue?” to our object, with logic that says “if in this state for more than 3 days, return true.” We then add a method that creates a task to call the client and assigns it to a pharmacist’s assistant. We can then have a simple piece of logic:

| if (prescription.pickup_overdue?
| && prescription.no_outstanding_notification?)
| prescription.create_overdue_notice_task
| end

So how and when does that piece of logic get checked?

Periodic Processes

In larger workflow “engines,” there are continually running processes that push things like this along. In this workflow methodology, software agents can be workflow participants as well. In the simplest case, that piece of logic will be put into a periodic task that is automated to run every few hours. In more complex cases, a software agent may actually have its own workbench and receive task assignments to act on (again, just like a human, waking up every now and then to do some work).

With a periodic task, it is possible to do more conventional “workflow routing” activities. For instance, you could have a workflow state that isn’t meant for “human eyes”—you have a periodic task that wakes up, gets all the work items in that state, and uses some logic to decide what the next state should be.

If you are interested in using the StonePath gem in a Rails application, I suggest you also look at the Crondonkulous gem (also at http://www.github.com/bokmann) for automating periodic tasks within a Rails application with cron.

Access Control

The final piece of the workflow puzzle is controlled access to information. Let’s take, for example, the pharmacist who is delegating out tasks. Perhaps a pharmacist’s assistant is going to check for drug interactions, and might brief out the patient when the medication is picked up. Regulations might prevent that assistant from knowing the customer’s insurance information, exact medical condition, or home address, but all of this information might be part of the pharmacist’s WorkItem for the prescription. At another point in time, the pharmacist’s assistant might be doing the data entry and need to be able to not just see, but modify all of this information. How can we control this?

Access Control in StonePath is controlled at the intersection of the WorkItem’s state, and the User’s Role. Users in a StonePath Workflow have a “role” they are performing in the workflow—Customer, Pharmacist, Deli Meister, and Pharmacist Assistant are all roles we have discussed so far.

When we define the states a WorkItem can be in, we can control the access any role has to the workitem at that time. We can say, for instance:

| WorkItem: Prescription
| state: "pending pharmacist action"
| role: Pharmacist
| allow: all
| role: Pharmacist Assistant
| deny: customer address
| state: "in fulfillment"
| role: Pharmacist
| allow: all
| role: Pharmacist Assistant
| deny: all
| state: "waiting customer pickup"
| role: Pharmacist
| allow: all
| role: Pharmacist Assistant
| deny: customer address
| deny: insurance information

Of course, the exact details of this language are a little more complex, but this give us the ability to have a fine-grained access control at the intersection of the job responsibilities of people by role and the state of the workitem.

Putting it All Together

The StonePath Reference Implementation is implemented in Ruby, and relies on Scott Baron’s excellent “Acts as State Machine” gem for the state and transition definitions (After the release of Rails 3, the reference implementation will transition to using the new ActiveRecord::StateMachine module, to keep dependencies at a minimum). Let’s actually look at some source code for how this would work.

The ShoppingTrip class

| class ShoppingTrip < ActiveRecord::Base
| include StonePath
| stonepath_workitem
| owned_by :Person
| target_of :ShoppingTasks
| aasm_initial_state :plan_trip
| aasm_state :in_planning
| aasm_state :dropoff_orders_phase
| aasm_state :dry_goods_phase
| aasm_state :refrigerated_goods_phase
| aasm_state :frozen_goods_phase
| aasm_state :done
| aasm_event :start_shopping do
| transitions :to => :drop_off_orders_phase,
| :from => :in_planning
| end
| aasm_event :get_dry_goods do
| transitions :to => :dry_goods_phase,
| :from => :drop_off_orders_phase
| end
| aasm_event :get_refrigerated_goods do
| transitions :to => :refrigerated_goods_phase,
| :from => :dry_goods_phase
| end
| aasm_event :get_frozen_goods do
| transitions :to => :frozen_goods_phase,
| :from => :refrigerated_goods_phase,
| :guard => !long_running_tasks?
| end
| aasm_event :finish_shopping do
| transitions :to => :done, :from => :dry_goods_phase
| end
| def long_running_tasks?
| # return true if there are outstanding long running
| # tasks, like prescription filling
| end
| end

This is real working code for a ShoppingTrip class. On line 1, we include the StonePath library. This brings in everything we need to be able to define the rest of the class, including the state machine library.

On line 4 we declaratively define this class as a stonepath_workitem, which allows it to “hook into” workflow in several ways.

Line 6 states that this workitem will be owned by instances of the Person class.

Line 7 says that this workitem is the subject of a particular kind of task, shopping_tasks.

We will see how those work in a little bit.

Line 9 defines out first state—the plan_trip state. The rest of this class defines the states and transitions that make up the rest of our state-based workflow. Notice how get_frozen_goods defines a guard? This guard calls the method long_runing_tasks?, and if that returns true, we will not be able to follow this transition into the frozen_goods_phase.

The Person class

| class Person < ActiveRecord::Base
| include StonePath
| stonepath_workowner
| workowner_for :ShoppingTrips
| stonepath_workbench
| workbench_for :ShoppingTasks
| end

Of course, a real Person class is going to contain things like a name, a password, perhaps ldap authentication credentials, and so on... but this is all we need to be able to create a Person object representing Tammy, and have her be an owner for one or many ShoppingTrip workitems. Since we can also give tasks to Tammy and Chris (such as “get oreos”) the Person needs to say that they are a workbench_for :ShoppingTasks. That might seem a little, odd, but we’ll talk about that a little more when we talk about the Pharmacist. Note that it would be possible, in more complex workflow scenarios, to be an owner_for more than one kind of workitem and a workbench_for more than one kind of task.

We have referenced a ShoppingTask class twice—once in each model above. Let’s define that class now.

| The ShoppingTask class
| include StonePath
| stonepath_task
| task_for :ShoppingTrip
| assigns_to :Person
| end

Of course, the ShoppingTask will have some data associated with it; perhaps the item to be picked up, or the prescription to be filled.

If you download that sample project, you can now fire up a command-line and begin manipulating these objects:

| > tammy = Person.create
| > trip = ShoppingTrip.create
| > trip.owner = tammy
| > trip.aasm_current_state
| plan_trip
| > perscription = trip.shopping_tasks.create("Prescription")
| > ordeos = trip.shopping_tasks.create("get Oreos")
| > ice_cream = trip.shopping_tasks.create("get Ice Cream")
| > trip.start_shopping!
| > pharmacist = Person.create
| > pharmacist.shopping_tasks << perscription
| > # the pharmacist now knows about the persciption
| > # and can begin processing it.
| > trip.get_dry_goods!
| > # go get the oreos
| > trip.get_refrigerated_goods!

...and so on. We can keep going with this workflow, but we now have the opportunity to discuss two other aspects.

A Person Is a Workbench?

While that commonly does occur in workflows, this is really just a shorthand notation that allows us to assign tasks directly to people. You could just as easily model a workbench as an “inbox” or a “todo list,” and have this make more sense.

| class GroceryCounter
| include StonePath
| stonepath_workbench
| workbench_for :shoppingTasks
| end

with this class, we can now say:

| > deli_counter = GroceryCounter.create
| > pharmacy_counter = GroceryCounter.create

This seems to make a lot more sense. Now several pharmacists can share responsibilities at the pharmacy_counter.

In order to make this work with the code above, we would need to be able to change the ShoppingTask class slightly:

assigns_to :Person, :GroceryCounter

As of the 1.0 reference implementation in Ruby, though, assignments to multiple workbench types is not supported. This will be available in StonePath 1.1.

Event Methods

In our interactive shell above, we can see that every event we defined becomes a method name we can execute. We can find out which ones are available in the current state by asking:

> trip.aasm_events_for_current_state

As mentioned earlier, it is very common to take the currently available states and assign them to button names or menu items.

In some cases though, it can be inconvenient in our code to have to figure out what our current state is and what the next thing we want the code to do is. In the case of our ShoppingTrip workflow, it is pretty much a linear path, so wouldn’t it be convenient if we could just say:

> trip.next!

and have it transition to the next state in the chain? We can do that by defining the next event like so:

| aasm_event :next do
| transitions :to => :long_running_tasks, :from => :plan_trip
| transitions :to => :dry_goods_phase, :from => :plan_trip
| transitions :to => :refrigerated_goods_phase,
| :from => :dry_goods_phase
| transitions :to => :frozen_goods_phase,
| :from => :refrigerated_goods_phase,
| :guard => !long_running_tasks?
| transitions :to => :done, :from => :dry_goods_phase
| end

Defined this way, the next! method will simply “do the right thing” depending on the state of the workitem.

We have covered a lot of concepts and code in this introduction to the StonePath Workflow Methodology. As with any modeling discipline, some things are immediately obvious, but the craft of handling edge cases comes only with experience using the tools. I encourage you to download the sample ShoppingTrip project and play with it yourself.

As a group of concepts, “workflow” can be baked directly into your domain modeling; you do not need an expensive, monolithic, or resource-intensive “workflow engine” to handle the concepts most of us will need in our typical workflow needs.

David Bock is a Principal Consultant at CodeSherpas, where he consults as a software engineer, project manager, and team mentor for commercial and government clients. David is the President of the Northern Virginia Java Users Group, is a founder of the Northern Virginia Ruby User Group, served as the Editor of O'Reilly's OnJava.com website, and is a frequent speaker on technology in venues such as the No Fluff Just Stuff Software Symposiums. In January 2006, Mr. Bock was honored by being awarded the title of Java Champion by a panel of esteemed leaders in the Java Community in a program sponsored by Sun. There are approximately 100 active Java Champions worldwide. David has also served on several JCP panels, including the Specification of the Java 6 Platform and the upcoming Java Module System.

Painting with Sound

Critical Tips for Home Recording

by Andy Hunt

Grab that guitar or horn that you haven’t touched for years and get to it.

Ain’t technology wonderful? It wasn’t that long ago that owning a professional 24-track recording studio was a very, very expensive proposition. Home recording was doable, sort of, but the technology really couldn’t compete with the pros. If you hadn’t noticed, technology has caught up.

These days you can use the very capable GarageBand that comes with your new Mac laptop, or snag Ardour or Audacity (multi-platform) for free, or go upscale with something like Logic Pro or Pro Tools. You can get a full and impressive 24 track studio—or 48, 96 tracks, or more—right on your laptop. You can record, mix, master and post your creation on the web or sell it through iTunes, all by yourself. So technology has solved all our problems and made home recording a reality, unleashing the creativity of millions, right? Well, not exactly.

It also wasn’t that long ago that people typed messages on paper using typewriters. I know, because I found several examples from the 1980’s in my filing cabinet last week. Apparently one former acquaintance of mine couldn’t be bothered with using the shift key, because all of their letters were TYPED IN UPPER CASE. SHOUTING THE WHOLE TIME. It got the message across, but it was pretty crude looking.

Of course all that went away with the advent of what we called Desktop Publishing at the time. Now the average person could produce brochures, pamphlets, letters and such all by themselves. You could mix and match fonts! In different SIZES. With images, stuck randomly on a whim. The possibilities were glorious; it got the message across, but it was pretty crude looking.

What? All that technology, marvelous fonts and images, and it still looked crappy? Yup. Just because the tools became easy to use and popular didn’t mean that the users were trained in graphic design, typography or layout. Similarly, just because now you’ve got the full horsepower of a multitrack recording studio at your fingertips, it doesn’t automatically train you in all the subtle arts required.

Now back in the day, as people shrunk in horror at the wild abuse of multiple fonts, clichéd use of Old English and random images carelessly spilled on the page, articles began to appear to try and convey some of the more important design principles. Without learning everything about graphic design, we were at least advised to show some restraint on font selection, perhaps use a grid of thirds for object placement and layout, and so on. The point of layout is to help tell the story effectively, to help highlight that which deserves highlighting, and without distracting from the message—the story you’re trying to tell.

So in this article I’d like to do something similar. I’ll try and convey some basic tips for home recording to help you tell your story more effectively and sound better. So grab that guitar or horn that you haven’t touched for years and get ready to tell your story.

Let’s start by looking at sound itself. There are two fundamental aspects to sound you want to think about: amplitude and frequency, and changes to both over time.

EQ

In the frequency domain, instruments produce notes at certain frequencies, but also produce overtones—additional frequencies stacked up at multiples of the basic frequency. For instance, suppose I play a concert A on the trumpet. The note ‘A’ at that octave is defined to be 440Hz. If you played a pure sine wave on a synthesizer, and then looked at it using a frequency spectrum analyzer on the computer, you’d see a narrow spike around 440Hz.

But on a trumpet, you’d see the fundamental frequency at 440Hz and a bunch of spikes at other frequencies as well. It’s these harmonics that give an instrument its unique timbre (no, you don’t pronounce it like wood “timber”, it’s tam-burr).

Now some of these extra frequencies are critically important, and some aren’t. Some are downright undesirable, in fact, and we’d like to get rid of them completely. For this, we need to use an equalizer, or EQ.

EQ lets you sculpt the sound. You can use it to help fit sounds together in a mix, or improve different aspects of a sound. For instance, with a vocal track, the frequencies around 1-2kHz affect vocal intelligibility. If you add a slight boost to that area on a vocal track, you’ll make it easier to understand the lyrics.

A big problem many home recordists have is a muddy bottom. That is, the bass part can’t be heard clearly or distinctly, it just kind of sinks in some kind of ill-defined audio “mud.” The problem is that all the non-bass instruments in the mix also contribute some lower bass frequencies, especially the kick drum, other guitars, piano, even voice and horns.

It’s common practice to cut these unwanted lower frequencies from all these other instruments in order to give the bass room to breathe—room to be heard. To do that, you use an EQ plug-in or piece of hardware to “roll off” (remove) the frequencies below, say, 250Hz or so.

Now, as with any powerful tool, EQ can be dangerous. EQ is designed to be a small, subtle tool—not an effect. You really don’t want to use it to make a male voice sound female, or make a trombone sound like a trumpet. There are better tools for such things (hint: you might want to change the fundamental frequency as well as the formants.) Also, conventional wisdom says you generally want to use EQ to cut, not boost, specific frequencies.

This is where relying on presets might get you into trouble. Virtually all the EQ presets in Logic Pro, for instance, boost certain desired frequencies instead of cutting the undesired ones. That’s backwards, and it’s not necessarily a good practice to emulate. If you want to bring out a certain frequency, you might be better off lowering all the other frequencies by the desired amount. (As a side note, you also want to prefer to use Logic’s Linear Phase EQ instead of the standard EQ, or any phase-coherent EQ in other systems. Phase corrected EQ’s typically have a much nicer, more natural sound, at the expense of extra CPU processing required).

You can also use EQ as a listening aid for yourself, and not as part of the final product. For instance, when you’re mixing and want to increase clarity in the midrange frequencies, it’s a great trick to slap an EQ across the whole mix that cuts out everything below 250Hz and everything above 5000Hz. It makes it sound like an old-fashioned AM radio (or really bad laptop speakers). You mix like that (in mono, for extra credit) to get a nice balance between all the instruments, then take off the EQ and listen to the extra highs and lows in all their glory.

If a particular instrument sounds bad for some reason, use EQ to first make it sound worse. Emphasize the worst part of the sound by boosting those frequencies using EQ. Once you’ve identified the bad parts that way, reverse the settings and use that EQ to cut the nastiness right out.

Parts of a Sound

Now let’s move on to amplitude. Consider that the loudness of a sound might go through several phases:

Attack

Decay

Sustain

Release

The attack comes first. The first few milliseconds or tens of milliseconds tends to be the most interesting; a lot of our perception of whether a sound is a voice or a trumpet or a kazoo comes from the attack phase. Very brief spikes of audio energy during the attack are called transients. More on that in just a second.

Most natural sounds have some level of decay as the initial attack tends to wear off, then might sustain at a more-or-less fixed level for a while. When the player takes their fingers off the keys, stops blowing, bowing or banging, the sound typically doesn’t just stop cold; there’s a release period as the sound energy tapers off to nothing.

Many synthesizers use envelope generators to create control waveforms that go through these same phases. You can get some interesting effects by taking a natural sound and changing its volume envelope. You might create a snare drum that sustains, or a cymbal with an abrupt release, and so on.

Compression

It seems that all the woodworking magazines out there regularly run columns on how to hand cut dovetails. It’s a tricky skill to learn, beginners have a hard time with it, and there are many different approaches and schools of thought.

So it is with compression. One of the more powerful, and probably most misunderstood, tools in the recording toolset is the compressor. It’s a tricky tool to learn, beginners have a hard time with it, and there are many different approaches and schools of thought. I doubt I can do better than any of the many tutorials out there, but here’s a few thoughts to get you headed in the right direction.

First, remember those transients that spike during the attack phase of a sound? Well they present a problem, because if they are really loud, they’ll overload the system and we’ll have to turn down this track. But now you can’t hear it as well, and it might get lost in the mix. It gets fun because actual amplitude and perceived loudness are different things; we can take advantage of that. What we need is a magic knob to turn down the volume on the loudest parts, and quickly turn the volume back up when needed.

In essence, that’s what the compressor does. It compresses the signal when it gets louder than a certain threshold. For instance, suppose you’ve got a vocal track. Vocals can be notoriously hard to tame, with loud shouty parts and whisper-quiet parts in the same breath. Here’s a piece of vocal:

You can adjust the threshold so that the compressor squashes down those loud parts; this lets us turn up the overall gain (“makeup gain”). There’s also usually a ratio control to adjust how much it squishes. The result looks like this:

You can control how quickly the compressor activates, and how soon it deactivates, using the attack and release controls. A quick attack will compress initial transients, and a slow attack will let them through. For instance, here’s a snare drum hit:

With a very fast attack and an extreme ratio, you can get a louder, rounder body of the sound at the expense of the crack:

You don’t want to compress the transients out completely, because that’s often the most important character of the sound. In the end, the best way to learn how to use a compressor well is to play with the knobs. But consider it can be too distracting to watch the numbers and meters and graphs. You need to listen with your ears, not watch with your eyes.

Reverb

Reverb, short for reverberation, is natural effect of a sound occurring in an acoustic environment. When a sound is generated, by whatever means, the sound waves don’t make a straight line to your ears. They bounce around, hitting walls and other objects, getting absorbed, reflected, amplified and attenuated. It’s a mess. But it’s a mess we’re used to hearing; that’s the world we live in. If you record a sound completely dry—with no reverb—it doesn’t sound right. So we’ll usually add a little reverb to sweeten it up a bit; to create a space that sounds great and highlights the tracks.

Some writer once said that using reverb had similarities to using crack. At first, everything is shiny, but then very rapidly it all gets blurry and confused. With reverb, less is definitely more; it’s very easy to use more than you need and end up with a mess.

There are two major ways of using reverb in a mix. In the first model, you set up one reverb on its own bus channel. Then you route all the other channels that need reverb through this one bus; you control how “wet” a track is (how much reverb it gets) by adjusting how much you send to the bus.

For instance, here is a simple four-channel mix with a lot of reverb on the piano, just a little bit on the drums, and none on the bass. I’m using the SpaceDesigner reverb on Bus 1 (abbreviated here as SpaceDsn).

The other model is to put a separate reverb as an insert effect on each individual track, as shown here. Notice there’s no bus, just a SpaceDsn inserted on each track.

Each has its advantages; by using one reverb you get a more coherent effect that produces the illusion that everything’s in the same room. It also conserves CPU resources. But you can get more interesting effects by using individual reverbs; it all depends on what you’re after.

In general though, lower frequency instruments get less reverb. Bass probably gets none at all, and instruments with a lot of natural sustain such as cymbals and acoustic guitar don’t need extra reverb either. If the overall mix gets too muddy, you might be better off using a short echo or delay instead of a reverb effect.

Modern convolution and algorithmic reverbs are very powerful and can do a really nice job of simulating a natural space. But don’t forget that recording in a shower stall, a hallway or a large room has its own sound as well.

Tragedy of Plenty

Modern recording technology has given you a horn of plenty. Unfortunately, that can also be a big drawback. Too much of anything can be bad.

Just because you have 48 tracks, you don’t have to use them all. Remember you’re telling a story, and it’s hard to follow a story when there’s more than one person speaking at a time. The flow of the story can pass between instruments, but the focus needs to be on one element at a time. The Beatles, you might recall, accomplished quite a bit with just four and eight tracks.

Then there’s volume. When mixing, you want to listen at very, very low volume levels–your ear is much more sensitive at lower volumes. Louder is not better at this stage.

In this land of plenty of tracks and plenty of effects, it’s easy to lose track of the balance of tension and release. Too much tension is noise; too much release is Kenny G. Neither is enjoyable. Loud and quiet are relative concepts; there is no loud without quiet to compare it to. It’s the contrast that generates interest and power. The worst thing to do is slap in a loop and just let it drone on for three and half minutes.

Well, maybe it’s not the worst sin (notice I haven’t talked about AutoTune at all), but it’s an easily avoidable menace. Some folks like to set up a loop and bang away on top of it, fearful of introducing too much change in case they might hit a wrong note. But consider that perhaps a boring note is also a “wrong” note.

Why Now?

There’s no better time to record your own material. Digital recording software is inexpensive and readily available, synthesizers, convincing virtual instruments and effects are plentiful. Good quality microphones and outboard gear are the cheapest they’ve ever been. You can get the gear, all you need now is a reason.

So let me offer you the following thought: music is vital to people’s lives. A commencement address by Boston Conservatory’s Karl Paulnack reveals the ancient Greek understanding of the fundamental nature of music: that music and astronomy were “two sides of the same coin.” The Greeks defined astronomy as the study of relationships between observable, permanent, external objects.

Music, on the other hand, examines that which is far more subtle: the study of relationships between invisible, internal, hidden objects. It’s all about the moving pieces within our hearts and souls.

We’re waiting to hear what you’ve got. Knock our socks off.

Andy Hunt is a Pragmatic Programmer and amateur musician.

Shady Illuminations

Framing the Issue

by John Shade

John finds inspiration in Google’s hijacking of Internet Explorer.

John Gilmore famously said, “the Net interprets censorship as damage and routes around it.”

A pearl happens when an oyster immunizes itself against a sliver by building a shell around it.

A teen-ager who finds her iPhone just too Apple-friendly-looking puts a Goth skin on it.

A missionary or Java developer civilizes a primitive type by slapping a wrapper on it.

These are all examples of framing. In the social sciences, the concept of framing has been popularized by George Lakoff, who built a career in linguisitics on misinterpreting Noam Chomsky. Anyway, that’s how Chomsky frames Lakoff. Frames are the irrational, emotional filters through which we attempt to inflict a modicum of sense on a senseless universe. Or a pallor of Goth on anything. Anyway, that’s how I frame Lakoff’s framing of frames.

Google got a lot of attention recently by framing Internet Explorer in a particularly clever way.

Back in 1997, Microsoft introduced support for Browser Helper Objects, third-party DLLs that extend Internet Explorer. Google has exploited that technology to hijack the rendering from IE and hand it off to Google Chrome. This has the effect of upgrading IE with the Webkit developer tools, faster JavaScript, and support for HTML 5 features. You’re cruising along in your Model T browser and it starts acting like a Maserati Bora. And in fact it is a Maserati: only the chassis is still intact; the body and engine have been magically replaced.

Microsoft, you can bet, is not happy. But as one blogger points out, Google didn’t hack undocumented APIs or something; they did it the right way, using technology that Microsoft documents and supports.

Pretty sneaky, that.

A couple of points are overlooked in most of the reporting on this: First, this isn’t the first time Google did this to Microsoft: in June, they turned Microsoft Outlook into a platform for accessing Google’s cloud services. And second, it’s not just IE: Google’s ChromeFrameNPAPI plugin should do the same favor for Firefox, Safari, and Opera on Windows. They’re preparing to hijack everybody.

Frames considered harmful

Personally, I think this is pretty cute of Google. I also think it adds flavor to the story that the word “framing” has such deliciously disreputable connotations.

In politics, “framing the issue” means spinning the scandal.

In the underworld, “to be framed” means to be arrested for the wrong crime.

And in technology there’s the little trick where one Website uses HTML frames to incorporate large chunks of another Website, compounding fraud and copyright infringement with design crimes. It’s this connotation of framing that lends Google’s trick a particularly nefarious tone. Website framing was a hot issue when the Web was young, but apparently it’s still a live one, with some of the current culprits being URL shorteners and Facebook and Digg users grabbing other sites and news content and framing it. Good luck to those pushing back against the framers. Once decanted, it’s rare that a genie can be recanted.

My vote for the most nefarious type of frame, though, is the atrocity of HTML frame tags. I’ve read that there are valid uses for frames in HTML, but I’m sure the same could be said for the blink tag. Me, I’m of the Frames Considered Harmful school of thought.

I think the framing meme is a powerful one. You could expand Gilmore’s aphorism, for example, beyond censorship to any attempt to wall off a part of cyberspace. Do good fences make good neighbors or is there just something that doesn't love a wall? The Internet is with Robert Frost on this one. When any technology has become more of a wall than a door, it is asking to be framed.

Given adequate bandwidth, storage, and processing power, any technology can be treated as a sliver to be encapsulated and turned into a pearl. You can virtualize your Mac into a Windows machine, and the Mac OS is really a frame disguising the fact that you’re running a Unix box. Your blog postings can become Tweets and vice-versa, or turn into e-mails, or e-mails can turn into Tweets or blog posts, all with a little framing. So what else could be framed?

My vote is for banks. My bank is located conveniently nearby. I’d like to keep that convenience, but frame the bank with another bank that doesn’t sit on my deposits forever before posting them.

It’s just a thought.

John Shade was born in Montreux, Switzerland, in 1962. Subsequent internment in a series of obscure institutions of ostensibly higher learning did little to enlighten him. He makes a better door than he does a window.

The Quiz

The Mother and Child Reunion

by Michael Swaine

Paul Simon says that he wrote the song “Mother and Child Reunion” after seeing the phrase on a Chinese restaurant menu describing a chicken-and-egg dish. Last month we presented a Chicken-and-Egg quiz, and this month we’re continuing this arbitrary and accidental homage to Paul Simon with this Mother and Child Reunion quiz.

So here’s the thing. We’ve somehow gotten the pairs of items below all mixed up. Your job is to reunite mothers and children. Pair each item on the left with the appropriate item on the right to create a Mother-and-Child reunion. Either the item from the left column will be (in some sense) the mother of the item from the right column, or the item in the right column will be (in some sense) the child of the item in the left column. “In some sense” is the key phrase: there are several different kinds of mother-child relationships represented here, but almost all of them pertain to computers and programming, and there is only one right way to pair up the moms and kids.

Mother

Child

1. Black, Estrada, Coronado, Collins, and Zappa

A. All markets, according to John Sculley

2. Grace Murray Hopper

B. WikiWiki

3. Mary Lou Jepson

C. Chicago

4. Mamma.com

D. Truth

5. Ada Lovelace

E. Ted Nelson

6. Jean E. Sammet

F. DEMO Conference

7. Adele Goldberg

G. DIMM

8. SNOBOL

H. FORMAC

9. Fortune

I. Grendel

10. Personal communicators

J. N. Katherine Hayles

11. A computer

K. Icon language

12. A demo in 1968

L. COBOL

13. Actress Angelina Jolie

M. Steven Paul Jobs

14. Actress Celeste Holme

N. Computer programming

15. Actress Hedy Lamarr

O. One Laptop Per Child

16. The ’60s

P. Wendi Shasta Leonardo

17. Susan Kare

Q. Search engines

18. HyperCard

R. Smalltalk

19. Time

S. Spread-spectrum crypto

20. Mobo

T. Invention

Answer to Last Month’s Quiz

Last month’s Chickens and Eggs quiz presented a series of pairs of events and asked you to say which of each pair happened first. Here are the answers. In each case, we list the earlier item first.

ARPAnet 1969, Ethernet 1973.

The IBM System/360 1965, “Go To Statement Considered Harmful” (Dijkstra) 1968

Etch A Sketch late 1950s, The first instance of the same program being run on different computers from different vendors and producing the same result 1960

Cray delivers the first Cray-1 supercomputer 1976, Gary Kildall creates the BIOS for CP/M 1977

Tron (the original movie) 1982, C++ 1984

Apple “releases” Steve Jobs September 1985, Microsoft releases Windows November 1985

The C Programming Language (K&R) 1978, VisiCalc 1979

Buck Rogers 1929, Flash Gordon 1934

The patenting of the integrated circuit 1959, the first legal article on computer law 1960

Apple’s HyperCard 1992, the Mosaic browser 1993

The birth of William Henry Gates III 1955, the birth of Fortran 1957

PHP mid-1995, Ruby late 1995

The UNIVAC 1 June 1951, The Day the Earth Stood Still September 1951

The Soul of a New Machine (Tracy Kidder) 1981, Thinking Machines Corporation (Danny Hillis) 1982

The Art of Computer Programming Volume I: Fundamental Algorithms (Knuth) 1968, Forth 1970

Extreme Sports’ X Games 1995, Extreme Programming 1996

Tinkertoys 1914, the use of magnetic tape for computer storage 1951

Color television 1950, the Shell Sort 1959

Yahoo 1994, XML 1998

Calendar

by Jackie Carter

Oct 2, Code Works Los Angeles

Topic TBA

Travis Swicegood, author of Pragmatic Version Control Using Git

Oct 2-4, Twin Cities Software Symposium, Minneapolis, MN

“Third time’s a charm: What’s new in Spring 3.0,” “WAR and Pieces: Breaking down monolithic web applications with Spring-DM and OSGi,” “That old Spring magic has me in its SpEL: DI Wizardy with the Spring Expression Language,” and “Beyond JUnit: Powertools for Test-Driven Development”

Craig Walls, author of Modular Java

Oct 3-4, Los Altos Hills, CA

Silicon Valley Code Camp

Oct 4, Code Works Dallas

Topic TBA

Travis Swicegood, author of Pragmatic Version Control Using Git

Oct 4

John Atanasoff’s birthday

Oct 4-6, JAOO, Denmark

Behaviour-driven development—writing software that matters” (1/2 day tutorial) and “Adventures of an Agile Architect”

Dan North, author of The RSpec Book

Oct 4-7, Los Angeles, CA

Adobe MAX

Oct 4-7, Orlando, FL

VSLive

Oct 4-9, Denver, CO

Models ’09, the ACM/IEEE 12th International Conference on Model Driven Engineering Languages and Systems

Oct 5, Supreme Court chambers, Washington, DC

Season Opener: South Carolina v. North Carolina

Oct 5-7, Bloomington, IN

SIGDOC 2009

Oct 7-9, Mumbai, India

Interop Mumbai

Oct 10-16, Atlantic City, NJ

2009 IEEE Information Visualization Conference (InfoVis)

Oct 11-14, San Antonio, TX

2009 IEEE International Conference on Systems, Man and Cybernetics (SMC)

Oct 11-14, Big Sky, MT

22nd ACM Symposium on Operating Systems Principles

Oct 11-15, San Francisco, CA

Oracle OpenWorld

Oct 11-16, Grenoble, France

CODE/ISSS, International Conference on Hardware-Software Codesign and System Synthesis

Oct 12, Skills Matter, London

“Refactoring Your Wetware, Full Day Workshop”

Andy Hunt

Oct 13

Arthur Burks’ birthday

Oct 13-14, Rails Summit Latin America, Säo Paulo, Brazil

Topic TBD

David Chelimsky, author of The RSpec Book

Oct 15-16, Lake Buena Vista, FL

ESEM 2009, International Symposium on Empirical Software Engineering and Measurement

Oct 16-18, Voices That Matter: iPhone Developers Conference

“Peer to Peer Networking for Games and More”

Bill Dudney, author of Core Animation for Mac OS X and the iPhone and iPhone SDK Development

Oct 16-18, Voices That Matter: iPhone Developers Conference

“Custom User Interfaces with Core Animation”

Marcus Zarra, author of Core Data

Oct 16-18, PostgreSQL Conference West 2009, Seattle WA

“Practical Full-Text Search in PostgreSQL” and “Practical Object-Oriented Models in SQL”

Bill Karwin

Oct 17-18, Voices That Matter, Boston, MA

Topic TBD

Marcus S. Zarra, author of Core Data

Oct 19-21, Denver, CO

Advanced Ruby Studio

Dave Thomas and Chad Fowler, author of The Passionate Programmer

Oct 19-22, SpringOne and Groovy/Grails Experience, New Orleans LA

“Grails Quick-Start” and “Not Your Father’s Custom Tags”

Dave Klein, author of Grails Quick Start

Oct 19-22, SpringOne and Groovy/Grails Experience, New Orleans LA

“That old Spring magic has me in its SpEL: DI Wizardy with the Spring Expression Language”

Craig Walls, author of Modular Java

Oct 19-22, San Jose, CA

ZendCon

Oct 19-22, ZendCon, San Jose CA

“Getting Git”

Travis Swicegood, author of Pragmatic Version Control Using Git

Oct 19-22, New Orleans LA

SpringOne and Groovy / Grails Experience

Oct 19-23, Cambridge, MA

Software Test & Performance Conference

Oct 19-24, Beijing, China

ACM Multimedia 2009

Oct 20-22, San Francisco, CA

Web 2.0 Summit

Oct 21, JUG, Richmond VA

“Refactoring Your Wetware”

Andy Hunt

Oct 21-23, Santa Clara, CA

ARM techcon3

Oct 22, Nashua Scrum Club

“Manage Your Project Portfolio: A Lean and Agile Approach”

Johanna Rothman, author of Manage It! Your Guide to Modern, Pragmatic Project Management and Behind Closed Doors: Secrets of Great Management

Oct 22, Compaid webinar

“Pragmatic Lessons Learned in Project Management”

Johanna Rothman, author of Manage It! Your Guide to Modern, Pragmatic Project Management and Behind Closed Doors: Secrets of Great Management

Oct 22-23, New York, New York

HotNets-VIII

Oct 22-24, Fairfaz, VA

SIGITE 2009

Oct 23, University City, MO

Strange Loop Conference

Oct 24, Minneapolis MN

The Twin Cities Code Camp

Oct 24, The Twin Cities Code Camp, Minneapolis MN

“Meet Ruby”

Brian Hogan, author of Web Design for Developers

Oct 24, The Twin Cities Code Camp, Minneapolis MN

“No More Excuses: Test Your Javascript!”

Kevin Gisi

Oct 25-28, PNSQC, Portland OR

Topic TBD

Johanna Rothman, author of Manage It! Your Guide to Modern, Pragmatic Project Management and Behind Closed Doors: Secrets of Great Management

Oct 25-29, Orlando, FL

OOPSLA

Oct 26-28, PNSQC, Portland OR

Topic TBA

Johanna Rothman, author of Manage It! Your Guide to Modern, Pragmatic Project Management and Behind Closed Doors: Secrets of Great Management

Oct 26-29, Biloxi, MS

OCEANS 2009

Oct 27-28, Prague

A Whistlestop Tour to Naked Object 4.0

Dan Haywood, author of Domain-Driven Design Using Naked Objects

Oct 28

Ted Hoff’s birthday

Oct 30-31, DevOpsDays ’09 Conference, Ghent, Belgium

“Non-Functional Requirements: do user stories really help?”

Rachel Davies, author of Agile Coaching

Nov 2, Skillsmatter, London

“In the Brain of Dan Haywood”

Dan Haywood, author of Domain-Driven Design Using Naked Objects

Nov 2-4, Agile Vancouver

Topic TBA

Johanna Rothman, author of Manage It! Your Guide to Modern, Pragmatic Project Management and Behind Closed Doors: Secrets of Great Management

Nov 6-8, Las Vegas, NV

7th Annual UFO Crash Retrieval Conference

Nov 7, Bletchley Park, Milton Keynes, UK

ACCU Security Yesterday, Today, and Tomorrow

Nov 8-12, AYE conference

Topic TBA

Johanna Rothman, author of Manage It! Your Guide to Modern, Pragmatic Project Management and Behind Closed Doors: Secrets of Great Management

Nov 9-13, Orlando FL

Agile Development Practices Conference 2009

Nov 9-13, Agile Development Practices, Orlando FL

“Pragmatic Thinking and Learning”

Andy Hunt

Nov 9-13, Agile Development Practices, Orlando FL

Several topics in the testing sub-conference

Jared Richardson, author of Ship It!

Nov 12, Agile Development Practices Conference 2009

“Focused Continuous Testing”

Rod Coffin

Nov 14-15, Greater Boston Chapter of ACM, in Boston, MA

Craig Walls is sharing the stage for two days with Ken Sipe doing an overview of Spring, covering much of the Spring portfolio.

Craig Walls, author of Modular Java

Nov 15-17, International PHP Conference, Karlsruhe

Topic TBA

Travis Swicegood, author of Pragmatic Version Control Using Git

Nov 16-17, Antwerp

“Introducing Scimpi”

Dan Haywood, author of Domain-Driven Design Using Naked Objects

Nov 16-20, Devoxx09, Antwerp

“Pomodoro Technique”

Staffan Nöteberg

Nov 17-19, Los Angeles

Microsoft PDC09

Nov 20, Redmond, WA

Microsoft Windows’ birthday

Nov 20, Italian Agile Day, Bologna, Italy

“SOAP is dead, long live REST”

Paolo Perrotta, author of Metaprogramming Ruby

Dec 1-3, Long Beach, CA

Pragmatic Studio - Ruby on Rails

Dave Thomas and Chad Fowler, author of The Passionate Programmer

Dec 1-4, Long Beach, CA

Pragmatic Studio - 4 day ‘how to’ for iPhone SDK development

Bill Dudney, author of Core Animation for Mac OS X and the iPhone and iPhone SDK Development, and Pragmatic Bookshelf editor Daniel Steinberg

Dec 2-3, Agile ’09, Copenhagen

“Pomodoro Technique”

Staffan Nöteberg

Dec 5

ARPANET expanded to four nodes 40 years ago today, while the Grateful Dead played live at Fillmore West

Dec 7

Tom Waits’ birthday

Dec 10, London Geek Nights, London

“Naked Objects 4.0”

Dan Haywood, author of Domain-Driven Design Using Naked Objects

Dec 10

Ada Lovelace’s birthday

Dec 16-19, Kochi, India

International Conference on High Performance Computing

Dec 25, Canada, USA

“The Imaginarium of Doctor Parnassus” opens

Writer and Director, Terry Gilliam; starring Heath Ledger, Christopher Plummer, Tom Waits, Johnny Depp, Colin Farrell, Jude Law

Dec 28

Linus Torvalds’ birthday

Jackie Carter is an editor at The Pragmatic Programmers. Jackie pulled together the author events for his calendar.