PragPub 2012-08: Issue #38

Table of Contents

Features

Tales from the Ops Side

by Michael Nygard

Did you know there's a website equivalent to having your doors broken down by hordes of deal-crazed shoppers?

Functional Thinking For The Imperative Mind

by Michael Bevilacqua-Linn

In which we dip our toe into the functional programming waters and answer the question, how does functional thinking differ from the imperative kind?

Thinking Functionally with Haskell

by Paul Callaghan

In which we begin an exploration into the Haskell language and dive deeply into functional programming.

Threads

by Michael Swaine

Thoughts on the Tom Paine of the personal computer revolution.

Scala for the Intrigued

by Venkat Subramaniam

In the final installment of his series on the Scala language, Venkat explores concurrency.

Departments

Up Front

by Michael Swaine

Stampeding customers, Scala, Haskell, functional thinking, and the Tom Paine of the personal computing revolution.

Choice Bits

Which Pragmatic Bookshelf books are hot right now—plus some tweets.

Calendar

Author sightings, upcoming conferences, and other events of note.

Shady Illuminations

by John Shade

With some problems, the only problem is thinking there’s a solution, John thinks.

But Wait, There’s More...

Coming attractions and where to go from here.

Except where otherwise indicated, entire contents copyright © 2012 The Pragmatic Programmers.

Feel free to distribute this magazine (in whole, and for free) to anyone you want. However, you may not sell this magazine or its content, nor extract and use more than a paragraph of content in some other publication without our permission.

Published monthly in PDF, mobi, and epub formats by The Pragmatic Programmers, LLC, Dallas, TX, and Raleigh, NC. E-Mail support@pragprog.com, phone +1-800-699-7764. The editor is Michael Swaine (mailto:michael@pragprog.com). Visit us at http://pragprog.com

ISSN: 1948-3562

Up Front

Something Is a Foot

by Michael Swaine

“Sometimes when you ask questions but don’t get answers, it means nobody knows the answers. At other times, though, it means nobody wants to be seen answering the questions. On this project, it was some of both.”

That’s Michael Nygard, back this issue with another tale of terror from the ops side. This time he shares his experience on a very big website project that went south very badly. He details the rapid response actions he took and does a post mortem on why it all went so badly and how it could have been avoided.

Also in this issue, we’re responding to a reader request for an article on “functional programming for the imperative mind.” We have two articles, in fact, because not everyone will be starting from the same place. One is about getting your feet wet in the functional programming pool, by Michael Bevilacqua-Linn. The other is a deep dive into functional programming emphasizing Haskell, by Paul Callaghan. Paul’s article kicks off a series on Haskell that will open your eyes about Haskell’s pristinely functional paradigm and functional programming in general.

Your editor has another computer history essay in the issue, this time reflecting on the legacy (so far!) of Ted Nelson, who more or less invented the Web before the Web. Only better.

Venkat Subramaniam is here, too, wrapping up his 12-issue series on the Scala language. This installment is all about concurrency.

And as usual we have a quirky column by our own John Shade, and events calendar, and a few other Choice Bits.

Choice Bits

Hot in August

Which Pragmatic Bookshelf books are hot right now—plus some tweets.

What’s Hot

Top-Ten lists are passé—ours goes to 11. These are the top titles that folks are interested in currently, along with their rank from last month. This is based solely on direct sales from our online store.

1^ | 3 | Agile Web Development with Rails
2^ | 11 | Deploying Rails
3v | 1 | Seven Databases in Seven Weeks
4 | 4 | Practical Vim
5^ | NEW | The ThoughtWorks Anthology
6v | 5 | Programming Ruby 1.9
7^ | NEW | Deploying with JRuby
8v | 7 | The RSpec Book
9v | 8 | The Cucumber Book
10^ | NEW | iOS SDK Development
11v | 10 | The Rails View

Skirmishes in the War on Bugs

If you‘re a web designer, you really, really need to get a cheap Dell monitor so you can see how bad your site looks on it and fix it. — @anna_debenham

.@tobins I think bug trackers should automatically close any bug with no description and “sometimes” in the title as “cannot reproduce.” — @invalidname

Ignoring feedback means that the system will eventually experience a massive unpleasant surprise rather than a small unpleasant surprise. — @jwgrenning

On code encapsulation: “You’re trying to protect against Murphy, not Machiavelli.” — @malcolmt

Pearls of Wisdom

All things become clear when you extract the method. — @jrasmusson

Every decent programmer I know learned by bashing together crappy bits of code in a crappy programming language. Start there then get better. — @nzkoz

On average, cross-language benchmarks say a lot about 1 programmer, and a little about n languages. — @stuarthalloway

Arrogance is a strength when you’re right and everybody else is wrong. Not so much when you’re wrong though. — @KentBeck

Seize the day, bro! In 10,000 years, all that will be left of you is your skeleton. And your plastic Live Strong bracelet. — @mktgdouchebag

Status Reports

[Neal Ford] is in some circle of hell right now—the AutoUpdater is in the process of updating itself, so that it can check for updates. — @neal4d

Someone actually sent me a friend request on Facebook today. It felt very 2010. — @gvwilson

My little brother and I couldn’t decide if we wanted to go for a run this AM or a sauna. Luckily, in Omaha, they aren’t mutually exclusive. — @scottdavis99

Random Thoughts

Documenting someone else’s code is a bit like writing fan-fiction. — @stuartsierra

Trying to figure out what I was going do with my extra leap second, then wasted it on this tweet. — @neal4d

OH: “Thin Mints is just a marketing scheme. If they called them Fat Mints no one would buy them.” — @jaredrichardson

Idea for a new programming language: Subjective C. You write code and it behaves differently depending on the compiler’s mood. — @zeh

I really hope Jessica Biel names her first child Batmo. — @ali_harter

If anyone else has ever tweeted about Calvin Coolidge and (an allusion to) phone sex, I’d like to know. #could_be_my_unique_contribution — @marick

Rea Life, or Something Like It

Can you imagine being Vint Cerf and having to respond to moronic “the government deserves no credit in creating the internet” op-eds? — @marick

Now that Higgs Boson has been found, they’re just using LHC to smash together new Taco Bell menu items. — @pourmecoffee

Michigan’s record-breaking temperatures prove hot enough to bake pizza in car. — @invalidname

As long as science has worse marketing than politics and religion, we’re stuck. — @bmf

Who Are Those Guys?

They’re not all guys. Also, we have to confess that we cleaned up their punctuation and stuff a little. OK, who they are: Chris Adamson, Kent Beck, Scott Davis, Anna Debenham, Marketing Douchebag, Zeh Fernando, Neal Ford, James Grenning, Stuart Halloway, Ali Harter, Michael Koziarski, Mike Lee, Brian Marick, pourmecoffee, Jonathan Rasmusson, Jared Richardson, Stuart Sierra, Malcolm Tredinnick, and Greg Wilson. You can follow us at www.twitter.com/pragpub.

Tales from the Ops Side

Trampled by Your Own Customers

by Michael Nygard

Did you know there's a website equivalent to having your doors broken down by hordes of deal-crazed shoppers?

In the world of physical retail stores, there's a promotion called a "doorbuster." It's a special offer for the first few people through the doors, usually available during the holiday shopping season. Sometimes, the figurative doorbuster becomes a literal doorbuster, when the eager crowds break windows and doors in their haste to get to the deals.

Countdown and Launch

After years of work, the day of launch finally arrived. I had joined this huge team (more than three hundred in total) nine months earlier to help build a complete replacement for this retailer’s online store, content management, customer service, and order-processing systems. Destined to be the company’s backbone for the next seven years, it was already more than a year late when I joined the team. For the previous nine months, I had been in crunch mode: taking lunches at my desk and working late into the night. Minnesota winter will test your soul even under the best of times. Dawn rises late, and dusk falls early. None of us had seen the sun for months. It often felt like an inescapable Orwellian nightmare. We had crunched through spring, the only season worth living here for. One night I went to sleep in winter, and the next time I looked around, I realized summer had arrived.

After nine months, I was still one of the new guys. Some of the development teams had crunched for more than a year. They had eaten lunches and dinners brought in by the client every day of the week. Even today, some of them still shiver visibly remembering turkey tacos.

Today, however, was the day of triumph. All the toil and frustration, the forgotten friends, and the divorces were going to fade away after we launched.

The marketing team—many of whom hadn’t been seen since the requirements-gathering meetings two years earlier—gathered in a grand conference room for the launch ceremony, with champagne to follow. The technologists who had turned their vague and ill-specified dreams into reality gathered around a wall full of laptops and monitors that we set up to watch the health of the site.

At 9 a.m., the program manager hit the big red button. (He actually had a big red button, which was wired to an LED in the next room where a techie clicked “Reload” on the browser being projected on the big screen.) The new site appeared like magic on the big screen in the grand conference room. Where we lurked in our lair on the other side of the floor, we heard the marketers give a great cheer. Corks popped. The new site was live and in production.

Of course, the real change had been initiated by the CDN. (CDN: Content Delivery Network, also known as an “edge network,” an accelerator that caches images and static content near the browser. This removes up to 80% of requests from your site’s web servers.)

They had a scheduled update to their metadata set to roll out across their network at 9 a.m. Central time. The change would propagate across the CDN’s network of servers, taking about eight minutes to be effective worldwide. We expected to see traffic ramping up on the new servers starting at about 9:05 a.m. (The browser in the conference room was configured to bypass the CDN and hit the site directly, going straight to what the CDN calls the “origin servers.” Marketing people aren’t the only ones who know how to engage in smoke and mirrors.) In fact, we could immediately see the new traffic coming in to the site.

By 9:05 a.m., we already had 10,000 sessions active on the servers.

At 9:10 a.m., more than 50,000 sessions were active on the site.

By 9:30 a.m., there were 250,000 sessions active on the site. Then the site crashed.

Aiming for QA

To understand why the site crashed so badly, so quickly, we must take a brief look back at the three years leading up to that point.

It’s rare to see such a greenfield project these days, for a number of good reasons. For starters, there’s no such thing as a website project. Every one is really an enterprise integration project with an HTML interface. Most projects have at least some kind of back end with which they must integrate. When the back end is being developed along with the front end, you might think the result would be a cleaner, better, tighter integration. It’s possible that could happen, but it doesn’t come automatically; it depends on Conway’s law. The more common result is that both sides of the integration end up aiming at a moving target.

Early in my time on the project, I realized that the development teams were building everything to pass testing, not to run in production. Across the fifteen applications and more than five hundred integration points, every single configuration file was written for the integration-testing environment. Hostnames, port numbers, database passwords: all were scattered across thousands of configuration files. Worse yet, some of the components in the applications assumed the QA topology, which we knew would not match the production environment. For example, production would have additional firewalls not present in QA. (This is a common “penny-wise, pound-foolish” decision that saves a few thousand dollars on network gear but costs more in downtime and failed deployments.) Furthermore, in QA, some applications had just one instance that would have several clustered instances in production. In many ways, the testing environment also reflected outdated ideas about the system architecture that everyone “just knew” would be different in production. The barrier to change in the test environment was high enough, however, that most of the development team chose to ignore the discrepancies rather than lose one or two weeks of their daily build-deploy-test cycles.

When I started asking about production configurations, I thought it was just a problem of finding the person or people who had already figured these issues out. I questioned, “What source control repository are the production configurations checked into?” and “Who can tell me what properties need to be overridden in production?”

Sometimes when you ask questions but don’t get answers, it means nobody knows the answers. At other times, though, it means nobody wants to be seen answering the questions. On this project, it was some of both.

I decided to compile a list of properties that looked as if they might need to change for production: hostnames, port numbers, URLs, database connection parameters, log file locations, and so on. Then I hounded developers for answers. A property named “host” is ambiguous, especially when the host in QA has five applications on it. It could mean “my own hostname,” it could mean “the host that is allowed to call me,” or it could mean “the host I use to launder money.” Before I could figure out what it should be in production, I had to know which it was.

Once I had a map of which properties needed to change in production, it was time to start defining the production deployment structure. Thousands of files would need changes to run in production. All of them would be overwritten with each new software release. The idea of manually editing thousands of files, in the middle of the night, for each new release was a nonstarter. In addition, some properties were repeated many, many times. Just changing a database password looked as if it would necessitate editing more than a hundred files across twenty servers, and that problem would only get worse as the site grew.

Faced with an intractable problem, I did what any good developer does: I added a level of indirection. The key was to create a structure of overrides that would remain separate from the application code base. The overrides would be structured such that each property that varied from one environment to the next existed in exactly one place. Then each new release could be deployed without overwriting the production configuration. These overrides also had the benefit of keeping production database passwords out of the QA environment (which developers could access) and out of the source control system (which anyone in the company could access), thereby protecting our customers’ privacy.

In setting up the production environment, I had inadvertently volunteered to assist with the load test.

Load Testing

With a new, untried system, the client knew that load testing would be critical to a successful launch. The client had budgeted a full month for load testing, longer than I had ever seen. Before the site could launch, marketing had declared that it must support 25,000 concurrent users.

Load testing is usually a pretty hands-off process. You define a test plan, create some scripts (or let your vendor create the scripts), configure the load generators and test dispatcher, and fire off a test run during the small hours of the night. The next day, after the test is done, you can analyze all the data collected during the test run. You analyze the results, make some code or configuration changes, and schedule another test run. Time elapsed before the next test: about three or four days.

We knew that we would need much more rapid turnaround. So, we got a bunch of people on a conference call: the test manager, an engineer from the load test service, an architect from the development team, a DBA to watch database usage, and me (monitoring and analyzing applications and servers).

Load testing is both art and science. It is impossible to duplicate real production traffic, so you use traffic analysis, experience, and intuition to achieve as close a simulation of reality as possible. Traffic analysis gives you nothing but variables: browsing patterns, number of pages per session, conversion rates, think time distributions, connection speeds, catalog access patterns, and so on. Experience and intuition help you assign importance to different variables. We expected think time, conversion rate, session duration, and catalog access to be the most important drivers. Our first scripts provided a mix of “grazers,” “searchers,” and “buyers.” More than 90% of the scripts would view the home page and one product detail page. These represented bargain hunters who hit the site nearly every day. We optimistically assigned 4% of the virtual users to go all the way through checkout. On this site, as with most ecommerce sites, checkout is one of the most expensive things you can do. It involves external integrations (CCVS, address normalization, inventory checks, and available-to-purchase checks) and requires more pages than almost any other session. A user who checks out often accesses twelve pages during the session, whereas a user who just scans the site and goes away typically hits no more than seven pages. We believed this mix of virtual users would be slightly harsher on the systems than real-world traffic would be.

On the first test run, the test had ramped up to only 1,200 concurrent users when the site got completely locked up. Every single application server had to be restarted. Somehow, we needed to improve capacity by twenty times.

We were on that conference call twelve hours a day for the next three months, with many interesting adventures along the way. During one memorable evening, the engineer from the load-testing vendor saw all the Windows machines in their load farm start to download and install some piece of software. The machines were being hacked while we were on the call using them to generate load! On another occasion, it appeared that we were hitting a bandwidth ceiling. Sure enough, some AT&T engineer had noticed that one particular subnet was using “too much” bandwidth, so he capped the link that was generating 80% of our load. But, aside from the potholes and pitfalls, we also made huge improvements to the site. Every day, we found new bottlenecks and capacity limits. We were able to turn configuration changes around during a single day. Code changes took a little longer, but they still got turned around in two or three days.

We even accomplished a few major architecture changes in less than a week.

This early preview of operating the site in production also gave us an opportunity to create scripts, tools, and reports that would soon prove to be vital.

After three months of this testing effort and more than sixty new application builds, we had achieved a tenfold increase in site capacity. It could handle 12,000 active sessions, which we estimated to represent about 10,000 customers at a time (subject to all kinds of caveats about counting customers). Furthermore, when stressed over the 12,000 sessions, the site didn’t crash anymore, although it did get a little “flaky.” During these three months, marketing had also reassessed their target for launch. They decided they would rather have a slow site than no site. Instead of requiring 25,000 concurrent users, they thought 12,000 sessions would suffice for launch during the slow part of the year. Everyone expected that we would need to make major improvements before the holiday season.

Murder by the Masses

So after all that load testing, what happened on the day of the launch? How could the site crash so badly and so fast? Our first thought was that marketing was just way off on their demand estimates. Perhaps the customers had built up anticipation for the new site. That theory died quickly when we found out that customers had never been told the launch date. Maybe there was some misconfiguration or some mismatch between production and the test environment?

The session counts led us almost straight to the problem. It was the number of sessions that killed the site. Sessions are the Achilles heel of every application server. Each session consumes resources, mainly RAM. With session replication enabled (it was), each session gets serialized and transmitted to a session backup server after each page request. That meant the sessions were consuming RAM, CPU, and network bandwidth. Where could all the sessions have come from?

Eventually, we realized noise was our biggest problem. All of our load testing was done with scripts that mimicked real users with real browsers. They went from one page to another linked page. The scripts all used cookies to track sessions. They were polite to the system. In fact, the real world can be rude, crude, and vile.

Things happen in production—bad things that you can’t always predict. One of the difficulties we faced came from search engines. Search engines drove something like 40% of visits to the site. Unfortunately, on the day of the switch, they drove customers to old-style URLs. The web servers were configured to send all requests for .html to the application servers (because of the application servers’ ability to track and report on sessions). That meant that each customer coming from a search engine was guaranteed to create a session on the app servers, just to serve up a 404 page.

Another huge issue we found was with the search engines spidering the site. Some of the spiders (particularly for the lesser-known search engines) do not keep track of cookies, for legitimate reasons. They do not want to influence marketing data or advertising revenue. The spiders generally expect the site to support session tracking via URL rewriting. Without the cookies, however, they were creating a new session on each page request. That session was then going resident in memory until it expired (thirty minutes). We found one search engine that was creating up to ten sessions per second.

Then there were the scrapers and shopbots. We found nearly a dozen high-volume page scrapers. Some of them were very clever about hiding their origins. One in particular sent page requests from a variety of small subnets to disguise the fact that they were all originating at the same source. In fact, even consecutive requests from the same IP address would use different User-Agent strings to mask the true origin. (User-Agent: an HTTP header sent by the browser to identify itself. Nearly all browsers claim to be some form of Mozilla, even Microsoft’s Internet Explorer.)

ARIN can still identify the source IP addresses as belonging to the same entity, though. These commercial scrapers actually sell a subscription service. A retailer wanting to keep track of a competitor’s prices can subscribe to a report from one of these outfits. It delivers a weekly or daily report of the competitor’s items and prices. That’s one reason why some sites won’t show you a sale price until you put the item in your cart. Of course, none of these scrapers properly handled cookies, so each of them was creating additional sessions.

We also had the amateur shopbots to handle. Several source IPs hit the same product detail page URL from the old site once per minute. It took us a while to identify the product, which ultimately turned out to be a PlayStation 2. Three years after PS2’s famous shortages, scripts were still running to look for the quantity available on the console, which created even more sessions.

The Testing Gap

Despite the massive load-testing effort, the site still crashed when it confronted the real world. Two things were missing in our testing.

First, we tested the application the way it was meant to be used. Test scripts would request one URL, wait for the response, and then request another URL that was present on the response page. None of the load-testing scripts tried hitting the same URL, without using cookies, 100 times per second. If they had, we probably would have called the test “unrealistic” and ignored that the servers crashed. Since the site used only cookies for session tracking, not URL rewriting, all of our load test scripts used cookies.

In short, all the test scripts obeyed the rules. It would be like an application tester who only ever clicked buttons in the right order. Most testers I’ve known are perverse enough that if you tell them the “happy path” through the application, that’s the last thing they’ll do. It should be the same with load testing. “Noise” might just bleed away some amount of your capacity, but it could bring your site down.

Second, the application developers did not build in the kind of safety devices that would cut off bad things. When something was going wrong, the application would keep sending threads into the danger zone. Like a car crash on a foggy freeway, the new request threads would just pile up into the ones that were already broken or hung.

Aftermath

The grim march in the days and weeks following launch produced impressive improvements. In one day, the CDN engineers used their edge server scripting to help shield the site from some of the worst offenders.

The home page was completely dynamically generated, from the JavaScript for the drop-down category menus to the product details and even to the link on the bottom of the page for “terms of use.” One of the application platform’s key selling points was personalization. Marketing was extremely keen on that feature but had not decided how to use it. So, this home page being generated and served up five million times a day was exactly the same every single time it got served. It required more than 1,000 database transactions to build the page. (Even if the data was already cached in memory, a transaction was still created because of the way the platform works.) The JavaScript drop-down menus with nice rollover effects required traversal of eighty-odd categories. Also, traffic analysis showed that a significant percentage of visits per day just hit the main page. Most of them did not present an identification cookie, so personalization wasn’t even possible. Still, if the application server got involved in sending the home page, it would take time and create a session that would occupy memory for the next thirty minutes. So, we quickly built some scripts that would make a static copy of the home page and serve that for any unidentified customers.

Have you ever looked at the legal conditions posted on most commerce sites? They say wonderful things like “By viewing this page you have already agreed to the following conditions....” It turns out that those conditions exist for one reason. When the retailer discovers a screen scraper or shopbot, they can sic the lawyers on the offending party. We kept the legal team busy those first few days. After we identified another set of illicit bots hitting the site to scrape content or prices, the lawyers would send cease-and-desist notices; most of the time, the bots would stop. (Like shooing a dog away from the dinner table, though, they always come back—sometimes even in disguise.)

One of the most heroic efforts in that chaotic time happened the week of launch. The IT operations manager identified six extra servers that matched our configuration. They had been requisitioned by a different department but were not in use yet. The manager reallocated them for the commerce site (and presumably ordered replacements) as extra application servers. One of our sysadmins spent a marathon 36-hour shift provisioning them: operating system install, network configuration, filesystem configuration, SAN access, and monitoring. Once he got to that point, we had someone drive him back to his hotel room where he could crash. I was then able to get the application server and applications installed and configured the same day. We doubled the capacity of the application server layer in two days, from bare metal to serving requests.

This particular application server’s session failover mechanism is based on serialization. The user’s session remains bound to the original server instance, so all new requests go back to the instance that already has the user’s session in memory. After every page request, the user’s session is serialized and sent over the wire to a “session backup server.” The session backup server keeps the sessions in memory. Should the user’s original instance go down—deliberately or otherwise—the next request gets directed to a new instance, chosen by the load manager. The new instance then attempts to load the user’s session from the session backup server. This mechanism works well (and scales surprisingly well), considering that the sessions are all kept in memory rather than in a database or on disk; that is, it scales well so long as the session data is kept small. For instance, it is common to include the user’s ID, her shopping cart ID, and maybe some information about her current search, such as keywords and results page index. It would not be typical to put the entire shopping cart in the session in serialized form or the entire search results (up to 2,000 results). Sadly, that is exactly what we found in the sessions. We had no choice but to turn off session failover.

All these rapid response actions share some common themes. First, nothing is as permanent as a temporary fix. Most of these remained in place for the next year or two. Second, they all cost a tremendous amount of money, mainly in terms of lost revenue. Clearly, customers who get throttled away from the site are less likely to place an order. (At least, they are less likely to place an order at this site.) Disabling session failover meant that any user in the checkout process on an instance would not be able to finish checking out when that instance went down. Instead of getting an order confirmation page, for example, they would get sent back to their shopping cart page. Most customers who got sent back to their cart page, when they had been partway through the checkout process, just went away. Making the home page static made personalization difficult, even though it had been one of the original goals of the whole rearchitecture project. The direct cost of doubling the application server hardware is obvious, but it also brought added operational cost. Finally, there is the opportunity cost of spending the next year in remediation projects instead of rolling out new, revenue-generating features.

The worst part is that no amount of those losses were necessary. It is now more than two years since that site launched. Today, the site handles more than four times the load, on fewer servers, without having gone through a hardware refresh. The software has improved that much. If the site had originally been built the way it is now, the engineers would have been able to join marketing’s party and pop a few champagne corks instead of popping fuses.

An Aside: Conway’s Law

In a Datamation article in 1968, Melvin Conway described a sociological phenomenon: “Organizations which design systems are constrained to produce designs whose structure are copies of the communication structures of these organizations.” It is sometimes stated colloquially as, “If you have four teams working on a compiler, you will get a four-pass compiler.”

Although this sounds like a Dilbert cartoon, it actually stems from a serious, cogent analysis of a particular dynamic that occurs during software design. For an interface to be built within or between systems, Conway argues, two people must—in some fashion—communicate about the specification for that interface. If the communication does not occur, the interface cannot be built.

Note that Conway refers to the “communication structure” of the organization. This is usually not the same as the formal structure of the organization. If two developers embedded in different departments are able to communicate directly, that communication will be mirrored in one or more interfaces within the system.

I’ve found Conway’s law useful in a proscriptive mode—creating the communication structure that I wanted the software to embody—and in a descriptive mode—mapping the structure of the software to help understand the real communication structure of the organization. Conway’s original article is available on the web at the author’s site.

Michael is a veteran software developer and architect. His background is either “well-rounded” or “checkered” depending on how charitable you'd like to be. He has worked in Operations (including for a “Top Ten” internet retailer), sales engineering, and as a technology manager and executive.

In 2007, Michael wrote Release It! to bring awareness of operational concerns to the software development community. This early influence in the DevOps movement showed developers how to write systems that survive the real world after QA.

Send the authors your feedback or discuss the article in the magazine forum.

Functional Thinking For The Imperative Mind

Getting Your Feet Wet

by Michael Bevilacqua-Linn

In which we dip our toe into the functional programming waters and answer the question, how does functional thinking differ from the imperative kind?

It's difficult to pin down exactly what functional programming is.

It's difficult because the phrase has come to imply several different, though related, things. If you talk to a Clojure hacker, you'll probably get an earful of macros. A Haskell programmer might talk about monads, and an Erlang programmer about actors.

These are all different concepts. Macros give programmers extremely powerful metaprogramming, monads allow us to model changing state in a purely functional way, and actors provide a robust way of doing distributed and concurrent programming.

Yet all of these come from functional languages. The breadth of concepts can make it a bit difficult for someone new to figure out what it's all about.

At its core, functional programming is about programming with pure, side-effect-free functions.

This is a pure function: f(x) = x².

So is this:

| public int incrementCounter(int counter) {
| return counter++;
| }

This is not:

| private int counter = 0;
| public void incrementMutableCounter() {
| counter++;
| }

The first example increments a counter by returning a new integer that's one higher than than the passed-in integer. The second example does so by mutating a bit of state that may be shared across many pieces of a program.

A function like incrementCounter that doesn't rely on mutating state is called a pure function. Purity has many benefits. For instance, if you've got a pure function that does some expensive computation, you can optimize your program by only calling the function once and caching the result, a technique known as memoization.

Pure functions also make programs easier to reason about. An object oriented program is a graph of objects, each with its own bundle of mutable state. Modifying one object's state can lead to another's state being modified, possibly many nodes in the graph away. In a program with only pure functions, this sort of action at a distance is impossible.

That's the simplistic description of functional programming. Unfortunately, strict purity and the real world don't play well together. Pure functions can be used to model some domains well, others not so much. Compilers are pure functions. Google's search is not.

Practical functional programming languages emphasize immutability and functional purity, though they must have some means of modeling a changing world that stops short of total functional purity. Haskell is probably the most strict. In Haskell, you can model change using monads, and otherwise maintain strict purity.

Other languages have their own techniques for minimizing and controlling state change that may not be quite as strict as Haskell's. Clojure, for instance, uses a software transactional memory system in combination with a set of reference types and fiendishly clever immutable data structures to maintain a high degree of purity while still letting programmers deal with a changing world.

So to a first approximation, functional programming is about programming with pure functions and immutable state, while imperative programming relies heavily on mutability. Around this immutable core there is a set of language techniques and features that replace imperative techniques that rely on mutability. Looking at these will give us a deeper feeling for what it is to think and program functionally.

Let's take a look at a simple example, filtering a list so it only contains odd numbers.

| public List<Integer> filterOdds(List<Integer> list) {
| List<Integer> filteredList = new ArrayList<Integer>();
| for(Integer current : list) {
| if(1 == current % 2) {
| filteredList.add(current);
| }
| }
| return filteredList;
| }

This is fine imperative code. We iterate through a list, and for each element we check to see whether it's odd by computing its modulus. We could, perhaps, make its intent a bit clearer if we were to pull that check out into a helper function and name it.

| public List<Integer> filterOdds(List<Integer> list) {
| List<Integer> filteredList = new ArrayList<Integer>();
| for (Integer current : list) {
| if (isOdd(current)) {
| filteredList.add(current);
| }
| }
| return filteredList;
| }
| private boolean isOdd(Integer integer) {
| return 1 == integer % 2;
| }

Now, what if we want to create a function that allows us to filter evens instead of odds? The only bit of code that needs to change is calling an isEven instead of isOdd.

| public List<Integer> filterEvens(List<Integer> list) {
| List<Integer> filteredList = new ArrayList<Integer>();
| for (Integer current : list) {
| if (isEven(current)) {
| filteredList.add(current);
| }
| }
| return filteredList;
| }
| private boolean isEven(Integer integer) {
| return 0 == integer % 2;
| }

This works, but we committed one of the cardinal coding sins. Most of filterOutEvens is cut and pasted from filterOutOdds. What we really want is a way to have a filter that can use some arbitrary bit of logic to do its filtering.

Let's take a look at how we might accomplish this in Java. Both isOdd and isEven take a single argument and return a boolean value. Let's define an interface that captures the essence of this computation. We'll call it Predicate, which is a mathy name for a function that returns a boolean value.

| public interface Predicate {
| public boolean evaluate(Integer argument);
| }

Now we can rewrite filterEvens and filterOdds to be more generic.

| public List<Integer> filter(List<Integer> list, Predicate predicate) {
| List<Integer> filteredList = new ArrayList<Integer>();
| for (Integer current : list) {
| if (predicate.evaluate(current)) {
| filteredList.add(current);
| }
| }
| return filteredList;
| }

Then we define our two predicates.

| class isEven implements Predicate {
| public boolean evaluate(Integer argument) {
| return 0 == argument % 2;
| }
| }
| class isOdd implements Predicate {
| public boolean evaluate(Integer argument) {
| return 1 == argument % 2;
| }
| }

Now we can simply instantiate one of the predicates and pass it into the filter method. If we come up with a new way to filter our list, say we only want to keep integers that are perfect squares, we can just define a PerfectSquare predicate rather than having to cut and paste the entire filtering function.

What's we've just done with the filter method and Predicate interface simulates a concept from the functional world, higher order functions. A higher order function is a function that can be passed into, or returned from, another function. Let's take a look at how we'd do similar filtering in Clojure, a modern, functional variant of lisp.

| (filter odd? [0 1 2 3])
| (filter even? [0 1 2 3])

That's it! The first thing you probably noticed it that it's much shorter than that Java version. The second is probably that the parentheses aren't in their usual spot. Clojure, and other lisps, use prefix notation for functions calls. This means that the function that's being called is the first thing inside the parentheses, with its arguments coming afterward.

Syntactic differences aside, notice how the Clojure version uses all built-in functions and language features? There's no need for us to define a Predicate interface. odd? is a function that takes a number and returns true if it's odd, while even? does the same for even numbers. We can pass those functions directly into the filter function using the power of higher order functions.

This turns our original imperative solution, in which we wrote code that was concerned with the nitty gritty details of iterating through a list, into a very declarative one. We're working at a higher level of abstraction that often lets us describe what results we want, rather than the details of getting them.

So when people talk about functional programming, they're generally talking about at least two separate, but very closely related things. First, they're talking about programming with pure functions. Since this is a pipe dream for most real world problems, practical functional programming languages generally settle for making it easier to use immutability than not, and for facilities that control mutation when you absolutely need to do it.

Second, they're talking about the style of programming that has grown up around this functional core. As we've seen, this style relies heavily on higher-order functions and other related techniques. These techniques often produce code that operates at a higher level of abstraction, such as using the filter function we saw above rather than explicit iteration.

These two facets of functional programming have significant benefits. The extreme emphasis on immutability makes programs easier to reason about. The behavior of a function can be understood just by reading the code in the function itself, rather than worrying about some bit of mutable state that it may rely on that's hundreds or thousands of lines away. The user of higher-order functions often leads to declarative code, which is shorter and more direct than the imperative equivalent.

So if you think of these two things when you think of functional programming, you won't be wrong: a preference for programming with pure functions, and a style of programming that involves higher-order functions in declarative code.

If this peek into the functional mindset has inspired you to look into functional programming in more detail, both Scala and Clojure are excellent languages for practicing programmers to learn functional techniques. They both run on the JVM and interoperate well with Java, so they're suitable for any development task you'd care to throw at them. Thanks for reading, and happy hacking!

Michael Bevilacqua-Linn has been programming computers ever since he dragged an Apple IIGS that his parents got for opening a bank account into his fifth grade class to explain loops and variables to a bunch of pre-teenagers. He currently works for Comcast, where he builds distributed systems that power infrastructure for their next generation services. In his spare time he likes rock climbing and good beer, though not at the same time. He blogs here.

Send the authors your feedback or discuss the article in the magazine forum.

Thinking Functionally with Haskell

A Deep Dive into the Functional Pool

by Paul Callaghan

In which we begin an exploration into the Haskell language and dive deeply into functional programming.

Ever wondered how functional programmers think? I aim to give you a glimpse into the programming style and mindset of experienced functional programmers, so you can see why we are so passionate about what we do. We'll also discuss some wider ideas about programming, such as making our languages fit the problem and not the other way round, and how this affects language design.

Few of these ideas get the exposure they deserve in textbooks or tutorials, and in my view they are essential for coming to grips with a functional language and using it productively in real apps.

Syntax and semantics, the meat and veg of most books and university courses, are ok for basic language use, but to really master a language that embodies a paradigm that is new to you, you need to know about the deeper pragmatic ideas. Let's see if we can do something about that.

I used Lisp for a few years before university, then switched to Haskell and have been using it for around 20 years. However, inspired by learning about Rails and Ruby when revamping a tired web technology course, I changed career to do full-time Rails work, and have spent the last four years having fun on a variety of apps, including Spree (#2 committer at one point) and recently a big bespoke lab management system.

Ruby feels like naughty fun for a Haskell programmer. Many of the ideas are very similar, like the very natural use of blocks and lambdas and having lots of scope for bending the rules. I really enjoy programming in Ruby, though some times I do get homesick and pine for a bit more oomph.

Most of this article will refer to Haskell, though many of the ideas do apply to other, similar languages as well. Haskell has a few advantages and a good balance of features. Haskell has its weaknesses too, and I hope to explore these in due course.

The Golden Rule

The most important idea in modern functional programming is this:

It's all about data

That is, functional programming is all about putting data first. We think about what kinds of data we have in a problem domain, and what kinds of transformations we want on them. Then we start building up the data structures and the code to do the transformations.

Functional programming is not about pure functions any more (or we'd still be using lambda calculus). What modern functional languages are about is developing ever-better tools to help in this style of programming, thus providing easier ways to specify powerful data types, manipulate them at a high level, break them apart and recombine them, and to do all this with a degree of safety and reusability and without too much syntactic baggage or implementation detail cluttering up the code.

Remembering to put data first is key if you want to use these tools well. Taking a fully imperative approach in a functional language doesn't end happily. The tools aren't designed to work that way.

A First Exercise

Ever used pipes in Unix shells? They are a very good introduction to functional thinking, and it's worth lingering a moment in Unixland before jumping into Haskell.

For example, suppose you want the three most memory-hungry processes owned by users whose login names begin with "foo"? There's no magic Unix command for this, but you can easily build one by assembling smaller pieces. This is a key idea in Unix, which provides small tools to perform steps on data, and a way to glue the steps together. I'll leave the fine details of this example to you, but basically you use ps to list processes, use grep to select matching rows, then sort on a particular column, and finally use head to grab the first few rows.

Another example: how do you count how many processes run by users Fred or Joe use more than 100M in virtual memory? You might need wc to count rows, and awk (or even perl or ruby) to do the numeric filtering. The details aren't important.

What is important is to notice what data we're using (rows of lines, each with certain fields) and how we're transforming that data through a pipeline of steps, one simple step at a time. Look at the code too--it tends to say pretty clearly what it is doing.

A Second Exercise

Let's see how much Haskell you can understand just by first grasping an informal, hand-wavy solution and then looking at the corresponding code. It's very much a "think in data" solution, and quite different from how you've probably seen this kind of problem approached before. Without modulo division or if statements, I give you "Functional Fizz Buzz".

The task is this: you want Fizz every three steps, and Buzz every five steps. Note that sometimes the cycles coincide. Let's talk cycles then.

| threes = cycle ["", "", "Fizz"]
| fives = cycle ["", "", "", "", "Buzz"]

where cycle is defined like this (the real library def is more efficient, but less clear)

cycle xs = xs ++ cycle xs -- SIMPLE version of lib

So threes just spits out ["","","Fizz","","","Fizz",... ] until we stop it, and similarly for fives. Next, we want to merge two streams into one: this is quite common so there's a library function for it. zipWith pairs up elements and uses some operation to combine each pair:

| zipWith g [a,b,c,...] [d,e,f, ...] ===>
| (computes to) [g a d, g b e, g c f, ...]
| eg zipWith max [1,2,3] [2,2,2] ===> [2,2,3]
| eg zipWith (*) [1,2,3] [2,2,2] ===> [2,4,6]

Think zippers in clothes. Now, that's just what we want for merging our streams. It works for infinite streams too (why shouldn't it?)

fizzbuzz = zipWith (++) threes fives

(++) is string concatenation, and then we just push the list of lines to the screen. And hit ^C when we get bored.

main = putStr (unlines fizzbuzz)

If we want numbers in there between the fizzes and buzzes instead of blanks, we can just zip in another list that contains the numbers from 1 up to infinity, and just add in a number if the string would otherwise be empty.

So: it's a short piece of code which obviously works, built from small pieces and glued together in a simple way (think Unix pipes), and there's no worry about loops, division, variables, memory limits...

Now, this isn't a one-off trick. Programming with Haskell is like this most of the time.

Data Types Are Cheap

We'll now look at the tools that modern functional languages provide. The first one is the sheer flexibility for declaring new data types and the scope for reusing them. This is very important: rather than encoding some data in a hash and hoping that you remember to keep the encoding consistent, Haskell allows you to add a new type that expresses exactly what you want to store. Such declarations are much shorter than the equivalent in imperative or OO languages. And Haskell can automatically generate certain standard functions to use with the type, such as ordering tests or conversion to strings. I'll say it again--it's very quick and easy to add the data type that fits your problem domain well, and the close fit really helps code quality.

Some examples now.

data Bool = False | True

That's right--Bool isn't baked into the language, the language is powerful enough to add such "primitive" notions directly into the core language. The various boolean operators, including shortcut semantics, are just standard Haskell definitions. The "core" of Haskell is a surprisingly small language, and the rest of the standard language is defined in straightforward Haskell.

data RGB = Red | Blue | Green deriving (Eq, Ord, Show, Enum)

This defines three constants (Red, Blue, Green) and automatically generates equality and ordering tests, a show (ie. to_s) function and the ability to use .. notation; e.g., [Red .. Green] is the list [Red, Blue, Green]

data Maybe a = Nothing | Just a

This is a (parametric) polymorphic type, and represents a box which is either empty (Nothing) or contains a single value of some type. This has various uses; e.g., passing in an optional value or returning a value that is absent or a sensible value. The parametric polymorphism means it can be used with any value (and any type) we choose, so it's not limited, say, to just containing strings. Note that parametric polymorphism is not the polymorphism seen in OO (though Haskell has a version of the latter, as explained below). A quick explanation is: parametric polymorphism means using the same code for everything, whereas the main kind of polymorphism in OO is more about allowing different values to be treated the same by virtue of calling object-specific code.

| data Person = Person { name :: String, age :: Maybe Int,
| fav_col :: RGB, address :: [String] }

Here's a simple record that stores a name, an optional age, a favourite color, and zero or more lines of an address (square brackets means lists in Haskell). Notice that a record field can contain an arbitrarily complex value, so not just primitive types. An example value is

joe = Person "Joe" (Just 25) Red ["Durham Cathedral", "Durham"]

Haskell also has syntactic sugar for accessing and updating record-style values which uses the names provided for the fields, in effect giving us accessors and setter functions.

We can also have recursive types like lists or trees, like the (polymorphic) binary tree below, which has values of some type at the leaves, and its internal nodes each have a left tree and a right tree.

data PTree a = PLeaf a | PNode (PTree a) (PTree a)

Lists are defined in a similar way, with a [] case and a cons case. Again, this isn't baked in, apart from a bit of syntactic sugar that Haskell provides to allow a simpler notation. Haskell also allows more complex examples, such as the following, which is effectively parametrizing which type holds the subtrees. This allows us to vary how the child trees are stored; e.g., we could have zero or more (c = lists), or precisely one for each of the RGB colors (with c a as a function from RGB to some polymorphic a). Not an everyday construct, but it does have its uses!

data X c a = XLeaf a | XNode (c (X c a))”

We're able to combine constants, records, recursion, and polymorphism quite freely, and to mix these with types in the standard libraries, like arrays, hashes, lists.... This gives us a lot of flexibility and convenience to model data in our problem domains, and to do it without much code. This modelling can be very accurate too, which helps to eliminate certain classes of errors. That is, if we can use these data type definitions to precisely say what is allowed, then our code need only deal with those cases, and we can easily check coverage of those cases. For example, representing optional values with Maybe forces us to explicitly handle the "nothing" case vs. the "something" case. Compare this to Ruby, where nils are often used for this but it's quite common to forget to check for nils before calling a method.

In fact, Haskell does not have nil values (and does not need them), so that's one class of error we never see.

Don't underestimate how important this flexibility and accuracy is!

A Very Important (and Under-Appreciated) Type

Like most functional languages, Haskell has first-class functions, meaning we can treat functions like almost any other piece of data--build them, pass them around, use them. We should not forget how such functions sit with the data types above. The notation for the type is A -> B, indicating a conversion from some type A to some other type B. Tests on color values will have type RGB -> Bool, or determining the max of two colours with RGB -> RGB -> RGB. Values of these types can appear in record fields, etc., for (a very contrived) example, each person could include a mapping from an int to a color which expresses what colour that person associates with a number.

We can also represent "waiting for an X" as a function value; e.g., if you have a person record but are waiting for their address to come from somewhere, this could be represented as a function of type [String] -> Person which is supplied with the address when it is available, and will return the complete person thereafter. Using the example above, we can do it like this, using Haskell's syntax for anonymous functions:

\address -> Person "Joe" (Just 25) Red address

Pattern Matching

Doing stuff with values in the above types is easy: we just write clauses in our functions to deal with the various patterns we expect to see.

Example, mirror-reversing a tree. There are two main cases, leaf or node, and each clause says what to do with the contents of the tree.

| mirror (PLeaf x) = PLeaf x
| mirror (PNode l r) = PNode (mirror r) (mirror l)

Notice that we're covering all possible cases of tree here. A value which is of tree type is either a leaf or a node, and we provide code to handle both cases in full. We'll never get a run-time error when an unexpected input is received. Some of the compilers track this "totality" for us, and can give warnings when functions don't cover all cases. Also, it doesn't matter what kind of data is on the leaves--this operation is just manipulating the tree structure. So quite naturally, this function is (parametrically) polymorphic and can be used on any PTree value.

Also note, we don't need if statements so much now. Pattern matching does most of the checking work for us. We can still have boolean-conditional tests; e.g., if 2 > 3 then a else b, or there's a shorthand for combining these with the patterns above. Finally, we can arbitrarily nest patterns for more complex conditions; e.g., the following from a compiler for Java, which tests for lifting a side effect out of a binary operator expression.

| rwE (BINOP op l (ESEQ s r))
| | commutes s l
| = chg $ ESEQ s (BINOP op l r)
| w otherwise
| = chg $ ESEQ (MOVE (TEMP t) l)
| (ESEQ s (BINOP op (TEMP t) r))
| where t = new_tmp_var

Recursion, Control, and Higher Order Functions

Pattern matching above gives us decision making, and we can also make arbitrary calls to other functions (including recursive calls). What about loops and so on? Most of the time, we don't really need them.

Think data and transformations again. When we have a collection of things to process in some way, say a list of numbers that we want to turn into a list of colors, or a tree containing strings for which we want the leaf count, most of the transformations we want fall into two main patterns: mapping and folding.

Mapping is about applying the same operation to everything in a collection, but keeping the shape the same. For example, we can add two onto elements in the list [1,2,3] to get [3,4,5]. The order and count of elements stays the same. The operation to perform is supplied as a function. Ruby programmers will know this pattern already, and know how convenient it is for replacing an explicit loop.

The other and more powerful pattern is folding. Ruby programmers will know this for arrays as inject; e.g.,

[1,2,3].inject(1, &:*)

to get the product of a list of numbers, and it works exactly the same. Another way to think of folding is to replace the "nodes" of some data structure with functions (or constants). Writing [1,2,3] in lisp style (cons 1 (cons 2 (cons 3 nil))) then [1,2,3].inject(i,f) will give us (f 1 (f 2 (f 3 i))). In the product example, this is (* 1 (* 2 (* 3 1))). Notice that we're collapsing or folding the data into a different type (list of numbers into a single number), though with appropriate choice of f and i we can produce a list of numbers again, or even produce more complex values--like lists of trees of numbers.

Now, this folding idea, of replacing constructor nodes with functions, applies to any data structure, so we can easily adapt it to other types, like trees or records. Here's folding on the simple trees from above.

| foldPTree node_case leaf_case (PLeaf x)
| = leaf_case x
| foldPTree node_case leaf_case (PNode l r)
| = node_case (foldPTree node_case leaf_case l)
| (foldPTree node_case leaf_case r)

So a leaf count can be done with foldPTree (\x y -> x + y) (\_ -> 1); i.e., count one for each leaf value then add up the results from subtrees. Once you understand the folding pattern, then such definitions suddenly become a lot clearer than the explicit version. Compare with this.

| leaf_count (PLeaf x) = 1
| leaf_count (PNode l r) = leaf_count l + leaf_count r

The code is simple enough, but you still need to check each line carefully to see that it has all the details right and doesn't contain mistakes, like calling leaf_count l instead of leaf_count r. In a sense, mentally you have to rewrite it as a fold! It's similar to explicit loops in Ruby, when you realize that they can be written more directly as a map or an inject. Isn't it nice to shorten and simplify the code, to make it say what you mean more directly? Yes.

So these maps and folds are a natural way to transform your data, and thus are highly useful tools for programming--in terms of both saying simply what you mean and avoiding verbose code. In fact, explicit recursion in FP is a bit of an anti-pattern, particularly when it's not needed--it can be annoying to read extra code when a more direct version works, plus more code means more chance of slipping up. Tony Hoare contrasted code with obviously no deficiencies vs. code with no obvious deficiencies--it helps to aim for the former!

Now, you might not be able to spot the maps and folds in some piece of code right away--it takes practice--but when you have sketched out an explicit version, do go back and think about whether it has bits of mapping, bits of folding, or particular sub-cases like filtering, and see if you can simplify the code. Also think in terms of data and transformations, and the opportunities might become more obvious. Don't feel compelled to use folds, etc., if you're not confident, but do try to reflect on your code afterwards and see if you can refactor. There will be cases when your pattern of recursion does not fit a fold, or will look worse if coded as a fold, but these are pretty rare. Similar advice applies for loops in other languages too: look for the mapping aspects, filtering, folding, and try to use the higher-level operations instead. Again, you should find that explicit loops aren't required that often (and if they are, perhaps wider refactoring would help; i.e., is there a bad design choice upstream that is forcing your hand?)

Three last technical points. First, folding corresponds to the "vanilla" pattern of processing values in a data type, and this pattern is inherently connected to how the datatype is defined. It's not a coincidence. Second, we're passing functions into the maps and folds to control what happens in the various cases, hence maps and folds are examples of Higher Order Functions. We don't need to dwell on this much--it just means functions that do stuff with functions. Third, mapping can be defined in terms of folding (exercise: try to write map in Ruby in terms of inject).

Functional Glue

The other side of higher order function use is how we use the building blocks to create larger blocks. Haskell is sometimes called an excellent "glue" language, because of the ease with which code units can be assembled. You've already seen pipelining--building large transformations from a sequence of smaller transformations, each contributing some piece towards the final result. Here's a Haskell example, written in various styles.

| foo1 input = unlines (map (\line ->
| unwords (map reverse (words line))) (lines input))
| foo2 input = unlines $ map (\line ->
| unwords $ map reverse $ words line) $ lines input
| foo3 = unlines . map (unwords . map reverse . words) . lines

The above reverses each word on each line of the input. Conceptually, we split the input into lines, then each line into words and then reverse each word, then reassemble the words and lines. The first version is the explicit version, the second cuts down some of the parentheses by using a syntactic trick, and the third shows the idiomatic version. Clearly, the first two still have some noise, but the third says very concisely what we are thinking. Reading right to left, we split the lines, do something to each line, and reassemble. And for each line, we split to words, reverse them, then reassemble. Again, no trickery--just using the language to say what we mean.

The dot deserves special mention. It is functional composition in Haskell, and basically means joining the output of one function to the input of another, so forming a bigger function. It is defined like this, as a piece of Haskell, and not baked in either:

f . g = \x -> f (g x)

So, foo . bar is a new function that takes some value x, applies g to it then applies f to the result. The two functions can be anything (providing their input/output types are compatible, of course). For example, show . (\x -> x + 3) shows the result of adding three to a value (assumed numeric). Or, not . (\x -> x > 10) . (\x -> x + 3) tests for a number + 3 being more than 10, and inverts the result. We're not stuck with this definition of composition either--we can define and use "reverse composition" too, where f gets applied first then g--whatever we find most useful.

Here's an example that often annoys me in Ruby. Sometimes I want to do one mapping on a list, then another mapping, so I have to write

stuff.map(&:foo1).map(&:foo2)

In Haskell it looks like this: map foo2 $ map foo1 $ stuff. But often, conceptually it is nicer to have one map doing two operations, so can rewrite it to map (foo2 . foo1) $ stuff. Ruby doesn't support this kind of rewrite without a lot of extra syntax. (I suggest this flexibility is the acid test of full functional support in a language.)

Finally, if we need other kinds of glue, we can just define it ourselves.

Clean Syntax

Haskell's syntax is one of its key features, and a model for other languages to follow. Several aspects are deliberately designed to cut down on syntactic baggage and allow more focus on the ideas in the code. This really emphasizes the declarative style of programming. Some key points:

Function application is kind of lisp style, but without so many parentheses; e.g., map reverse (words "foo bar"), where map takes two arguments and words takes one.

Partial application is natural and unobtrusive; e.g., map foo4 is a function that applies some function foo4 to any list it is supplied later. Compare other languages, where we'd have to write something like ->(list) {map foo4 list} to get the same effect.

Modern precedence and associativity rules for operators, which can be extended to include new user-defined operators--which really helps in the design of embedded domain-specific languages (DSLs)--although one can go too far and end up with APL...

Semicolons. Yes, semicolons. And braces (curly brackets). The Haskell grammar actually requires these elements to delimit and separate groups of expressions, and most implementations of the language have a pre-processing stage that inserts braces and semicolons according to how indentation is being used in the code, typically through a variant of Landin's "off-side rule." (Key point: Haskell's use of layout is more flexible than Ruby's.)

Performance

You may have been worried by the seemingly infinite loop in the fizz-buzz example. However, Haskell compilers generate "lazy" code, which means (as a first approximation) that work is only done when needed, so we only generate as much of the list as we need (here, until ^C is used), plus garbage collection reclaims unused space.

Note that there are several kinds of "lazy" available, including a few options we can set during compilation, and that the Haskell language definition only requires "non-strict" rather than "lazy".

Put another way, the language passes more control over execution order to the compiler and runtime. We don't have to micro-manage the steps of computation, and the compiler is free to transform our code in powerful ways to improve performance. It's quite common for FP compilers to optimize away some of the intermediate data structures in a chain of functions (called "deforestation") and so save on memory and time. Some FP compilers (Ocaml in particular) can even match the top C compilers in performance now because of the complex optimizations they can do when side-effects are less of an issue.

The key point from this is the silver rule: "Trust your compiler." Your job is to write good code, and you can let the compiler deal with some of the lower-level details. It is much better at complex optimizations than you are--and less error-prone(!) If the program runs slowly, there are a good range of tools for profiling and analyzing that help to pinpoint bottlenecks. You can then work on optimizing the crunch points. Haskell's GHC compiler offers several simple techniques for speeding up bottlenecks, from reduction of laziness, through user-supplied rewrite rules, to even making it easy to code key steps in C. Meanwhile, most of your code can remain high-level and clear.

Summing Up

There is a lot more I would bring up, but my editor is tapping his foot. Next time? For now, I hope you have a clearer view of how functional programmers think: that most of it is about deciding which data structures you need, then coding up various transformations. And how we use the language to program at a high level and use it to express the big ideas in a straightforward and clear way. It is kind of addictive, just like Ruby. You have a lot of flexibility and convenience, but you also get strong tools to do even more complex stuff and keep control of it.

Should you all go and learn Haskell? I would be chuffed if you were intrigued enough now to spend a few hours on it, and manage to absorb some of the different ideas. I think the main benefit is to be aware of the data structures that your program is manipulating, and aim to write your code to bring this more to the fore; then you will see more functional ideas appearing quite naturally in your code. It also helps to think about the strengths and weaknesses of your existing tools, to use their strengths well and to avoid getting caught by their weaknesses.

This goes for Haskell too. It has strengths, and it has weaknesses, and researchers are already looking at stronger languages.

I said before that Ruby feels like naughty fun for a Haskell programmer. This is partly due to a lot of common ground between the two. So I really hope that Ruby programmers get something useful out of these articles which feeds back into their Ruby or Rails work. It is possible that you hit obstacles and get frustrated, but do try to step back, think of the bigger picture ("data, dude!") and try a different approach. To this end, I would like to do a kind of code clinic in a future article, and use the PragPub forums to get people past some of the obstacles. It'd be great to hear your views.

In the coming months, I aim to cover (provisionally)

understanding the maybe type

practical tips for working in haskell (including type-directed development)

tests, types, and learning from both

haskell code clinic

writing a ruby refactorer in haskell

dependent types

and yes, yet another explanation of monads

I need a picture and a bio.

Dr Paul Callaghan rides a big, fast motorbike, and suggests that this informs his programming language opinions. He doesn't like to wait around, or do more stuff than he has to, and loves the freedom. He started with Ruby and Rails around 6 years ago and soon decided to take up Rails work full time, and is currently working on a range of bespoke systems for a small software house. Before this, he was a university lecturer and did research on advanced programming techniques using dependent types. He did his PhD in the area of Natural Language Processing as part of a group that competed in several international information extraction competitions, developing significant chunks of the NLP system's 60k lines of Haskell. He blogs here and also flies big traction kites and can often be seen being dragged around inelegantly on the beaches of North-east England, much to the amusement of his kids.

Send the authors your feedback or discuss the article in the magazine forum.

Threads

Ted Nelson

by Michael Swaine

Thoughts on the Tom Paine of the personal computer revolution.

Paul Freiberger and I are currently writing the third edition of Fire in the Valley, our history of the personal computer—to be published by The Pragmatic Bookshelf, naturally. Going through the material, I keep finding topics that feel like article material. Last issue I reflected on the contributions of Lee Felsenstein and Adam Osborne. This issue I’m sharing some thoughts on the legacy (so far!) of Ted Nelson.

Project Xanadu

In 1982, Playgirl magazine named Ted Nelson one of America’s most eligible bachelors.

This is just one of the facts that makes it hard to pin down Ted’s unique place in the history of the personal computer.

What Ted might do with a fact like this, I think, is to put it down on a notecard and save it to link it with other facts later.

That’s how he approached writing as a grad student at Harvard back in 1960, and it led him through frustration to invention. Only with computer technology could he manage the exponential connections between all the ideas in his fertile mind and on those mountains of cards he was accumulating. Only with computer technology could he realize the kind of nonsequential writing that he wanted to do. But the computer technology of that time wasn’t up to the task. So he set out to rethink the way computers worked.

He came up with an idea he called hypertext, by which he meant linked text and other media, but he kept thinking about how linking should work and took it far beyond the one-way, unlabeled, unversioned links of the (then unimagined) World Wide Web. Today developers aspire to the DRY goal in software—don’t repeat yourself. Ted applied the principle to data with an idea he called transclusion—the inclusion of a document or part of a document into another document by reference, and with the insight that document storage should mostly be about storing the difs—an idea useful both for the modular design of documents and for versioning. He looked farther ahead yet and saw that transclusion would stir the copyright waters—think of the arguments about deep linking on the Web—and came up with a plan of micropayments that could be built into the infrastructure of hypertext. Seeing that all these ideas needed to be implemented simultaneously in order to work, he designed Project Xanadu, a universal information repository to function as the future intellectual home of mankind, with streetcorner access points serving up information as a kind of McDonald’s for the mind.

He came up with that in 1960.

Computer Lib

Theodor Holme Nelson is the son of actress Celeste Holme (Academy Award, “Gentlemen’s Agreement”) and director Ralph Nelson (Emmy, “Requiem for a Heavyweight”). He was born in 1937 and has degrees in philosophy, sociology, and media and governance from Swarthmore, Harvard, and Keio University. He grew up in Chicago and Greenwich Village.

He is justly called the Thomas Paine of the personal computer revolution, and this is why:

In 1974, Ted pulled together all (well, probably not all) of his thoughts on computer technology, the computer industry, and the interplay of computers and society in an extraordinary book called Computer Lib/Dream Machines. Computer Lib was inspired by The Whole Earth Catalog and shares some of its eccentric layout, but goes farther in the direction of anarchy and ADD. The book is a passionate call to arms: “You can and must understand computers NOW” is the text above an upraised fist on the cover.

Computer Lib was an extraordinary book, and extraordinarily, it was recognized as such at the time. When Ed Roberts was creating the Altair computer that would kick off the revolution for computer hobbyists in 1975, he had a copy of Computer Lib on his desk. Steve Wozniak was reading it when he created the Apple 1. Computer Lib was the Common Sense of the personal computer revolution. The enemy in that revolution, as Stewart Brad put it, was “Central Processing, in all its commercial, philosophical, political, and socio-economic manifestations.” The revolution happened, Central Processing was overthrown, and, as Ted wrote in 1987, “computers are just as oppressive as before, but smaller and cheaper and more widespread. Now you can be oppressed by computers in your living room.” I’ll leave it to you to decide where the revolution has gone since 1987.

But I wanted to pull together a lot of different facts about Ted, and I’ve mostly focused on Xanadu and Computer Lib. I have all these other topics and facts left over:

Ted has ranted against the tyranny of hierarchical file structure and against the application-centric view of computing. “‘Word processing’ is not a category of human activity,” he said.

He has ridiculed WYSIWYG, saying it was really What You See Is What You Get When You Print It Out, and that we should not aspire to have our electronic documents mimic typewriter output.

He has criticized the whole process of software development, arguing for an approach more like movie directing, and emphasizing auteurs. He compares software development today to the movie industry before 1904, when the cameraman made the movie.

I could go on. Ted does. There is need for him to do so. As Ted might say, the revolution is not over. There are still ramparts to be stormed.

Scala for the Intrigued

Concurrency in Scala

by Venkat Subramaniam

In the final installment of his series on the Scala language, Venkat explores concurrency.

We saw the object-oriented paradigm and the functional style of programming interplay in Scala in the previous articles in this series. In this article we’ll use the functional purity for greater good—programming concurrency.

We all desire to make our applications faster, more responsive. There’s no shortage of resources with multiple cores in our hands. The hard part is writing the code correctly to reap the benefits of the power on hand.

To fully exploit the power of concurrency and perform various complex tasks, concurrency libraries like Akka are quite helpful. (See also Programming Concurrency on the JVM by one of our favorite authors -ed.) However, in this article, we’ll stay within the facilities provided directly in the Scala language and make use of parallel collections.

We want our applications to be responsive and faster, but we don’t want to compromise their correctness. The purity of functional programming is our ally in this area. When we set out to make code concurrent, we must ensure the purity of operations, making sure they don’t have side effects. This means using vals (rather than vars) and immutable objects. For avoiding side effects, Scala rewards us with faster response without compromising the correctness of the results.

Let’s first create a utility function that will allow us to measure the time operations will take. This will help us compare the sequential processing and its concurrent counterpart.

| object Time {
| def code(block: () => Unit) = {
| val start = System.nanoTime
| try {
| block()
| } finally {
| val end = System.nanoTime
| println("Time taken: " + (end - start)/1.0e9)
| }
| }
| }

The code function of the Time singleton accepts a function value and measures the time it takes to apply or execute the function value (code block). For example, to report the time for a simple block of code that takes a short nap, we can write the following:

| Time.code { () => Thread.sleep(1000) }
| //Time taken: 1.00088

We’ve seen different Scala collections in the previous articles. The collections provide a special method named par that returns to us a parallel collection.

Let’s create a list of names and get a parallel collection from it.

| val names = List("Jane", "Jacob", "Brenda", "Brad")
| println(names)
| //List(Jane, Jacob, Brenda, Brad)

| println(names.par)
| //ParVector(Jane, Jacob, Brenda, Brad)

Certain operations on parallel collections run concurrently and we can make use of these to speed up operations that take significant time. Suppose we’re creating a messenger application and want to check the status of multiple people we’re interacting with. The function for this operation might look like the following:

| def checkStatus(name : String) = {
| Thread.sleep(1000) //simulate the delay to communicate
| String.format("%s's status", name)
| }

Sequential calls to this function will incur delay in proportion to the number of elements in the list.

| Time.code { () => println(names.map { checkStatus }) }
| //List(Jane's status, Jacob's status, Brenda's status,
| // Brad's status)
| //Time taken: 4.005623

We can enjoy a speedup if we apply the map function on the parallel collection we created.

| Time.code { () => println(names.par.map { checkStatus }) }
| //ParVector(Jane's status, Jacob's status, Brenda's status,
| // Brad's status)
| //Time taken: 1.021915

Unlike the execution of the map function on the List, the map function on the parallel collection ran the given function value concurrently for each element. The number of concurrent executions depends on the number of threads in a pool that Scala allocates, which in turn depends on the number of cores available. This is simple, but is somewhat limiting. If we need finer control on the thread pool, libraries like Akka provide some good solutions.

Suppose we want to find if all our meeting participants are connected. We can do this sequentially.

| def isConnected(name : String) = {
| Thread.sleep(1000) //simulate the delay to communicate
| name.length > 4 //simulated response
| }

| Time.code { () =>
| println("All connected? :" + names.forall { isConnected })
| }
| //All connected? :false
| //Time taken: 4.004065

Unlike the map function, the forall function needs to collect the result of the evaluation of the function value for all the elements. However, the evaluations can be performed concurrently as in the next code.

| Time.code { () =>
| println("All connected? :" + names.par.forall { isConnected })
| }
| //All connected? :false
| //Time taken: 1.018888

We saw how the parallel collection runs some operations concurrently. Not all operations can be executed concurrently, however. For example, the parallel collection provides the foldLeft, foldRight, and reduce methods so we can conveniently invoke them on these kinds of collections like we do on regular collections. Based on the context, we have to keep in mind that such operation will be sequential and not concurrent.

| def addToConnected(connected : Int, name : String) =
| connected + (if (isConnected(name)) 1 else 0)

| Time.code { () =>
| println("Number of folks connected: " +
| names.foldLeft(0) { addToConnected })
| }
| //Number of folks connected: 2
| //Time taken: 4.005546

| Time.code { () =>
| println("Number of folks connected: " +
| names.par.foldLeft(0) { addToConnected })
| }
| //Number of folks connected: 2
| //Time taken: 4.014092

There is another caveat with parallel collections: the concurrent version may do more work. Suppose we want to find the first connected person. We could use the find function, like so:

| Time.code { () =>
| println("A connected person: " + names.find { isConnected })
| }
| //A connected person: Some(Jacob)
| //Time taken: 2.003715

The sequential version evaluated the isConnected function for the first two persons in the list before it found a matching element. Let’s take a look at the parallel version.

| Time.code { () =>
| println("A connected person: " + names.par.find { isConnected })
| }
| //A connected person: Some(Jacob)
| //Time taken: 1.020151

The concurrent version gave the same result as the previous version and took less time, but there’s a catch. To understand this, let’s introduce a print statement, an impurity, to make visible the actual tasks executed.

| names.find { name => println("seq: " + name); isConnected(name) }
| //seq: Jane
| //seq: Jacob

| names.par.find { name => println("conc: " + name); isConnected(name) }
| //conc: Jane
| //conc: Jacob
| //conc: Brad
| //conc: Brenda

The concurrent version ends up doing more work than the sequential version, as it’s trying to evaluate all the cases concurrently. We have to weight this in and ensure the extra executions are not prohibitively high and impact either the performance or the outcome.

Let’s put the parallel collection to a practical use. In the December article on “Functional Style of Programming” we looked at concise code to fetch and process stock prices from Yahoo. Let’s revisit that code and make it concurrent.

For this example we’ll use the following tickers:

val tickers = List("AAPL", "AMD", "CSCO", "GOOG", "HPQ", "INTC", "MSFT", "ORCL")

The code to fetch the price and construct it into a StockPrice object is repeated from that article here.

| case class StockPrice(ticker : String, price : Double) {
| def print = println("Top stock is " + ticker + " at price $" + price)
| }

| def getPrice(ticker : String) = {
| val url = "http://ichart.finance.yahoo.com/table.csv?s=" + ticker
| val data = io.Source.fromURL(url).mkString
| val price = data.split("\n")(1).split(",")(4).toDouble
| StockPrice(ticker, price)
| }

The helper functions we needed to check if the stock prices are less than $500 and to pick the higher priced stock are show next:

def isNotOver500(stockPrice : StockPrice) = stockPrice.price < 500

| def pickHigherPriced(stockPrice1 : StockPrice, stockPrice2 : StockPrice) =
| if(stockPrice1.price > stockPrice2.price) stockPrice1 else stockPrice2

Finally here’s the sequential code to compose all these to produce the desired result. We’ll measure the time to execute this code.

| Time.code { () =>
| tickers map getPrice filter isNotOver500 reduce pickHigherPriced print
| }
| //Top stock is ORCL at price $30.01
| //Time taken: 17.777705

The code took around 17 seconds to get the prices from Yahoo and determine the highest-priced stock not over $500.

Let’s make a small change to the code to turn this into concurrent execution.

| Time.code { () =>
| tickers.par map getPrice filter isNotOver500 reduce pickHigherPriced print
| }
| //Top stock is ORCL at price $30.01
| //Time taken: 3.805312

We inserted the call to par and invoked the map on the resulting parallel collection. All the requests to Yahoo and the subsequent calls to isNotOver500 are done concurrently. The only sequential part is the reduce operation and the resulting calls to pickHigherPriced. The concurrent version took only around 3 seconds to produce the same result.

The functional programming style combined with powerful libraries make concurrent programming not only easier but also fun.

It has been a pleasure writing this series, and we’ve now arrived at the culminating article. I sincerely hope you enjoyed this exploration of the Scala language. Thank you for reading.

Dr. Venkat Subramaniam is an award-winning author, founder of Agile Developer, Inc., and an adjunct faculty at the University of Houston. He has trained and mentored thousands of software developers in the US, Canada, Europe, and Asia, and is a regularly invited speaker at several international conferences. Venkat helps his clients effectively apply and succeed with agile practices on their software projects. He is also the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy: Dynamic Productivity for the Java Developer and Programming Scala: Tackle Multi-Core Complexity on the Java Virtual Machine. His latest book is Programming Concurrency on the JVM: Mastering Synchronization, STM, and Actors.

This series started in the September 2011 issue and concludes with this August 2012 issue. If you’d like to read the whole series, here are the links to the individual articles: 9/11: The Elegance of Scala, 10/11: Sensible Typing and Optional Items, 11/11: Cute Classes and Pure OO, 12/11: Functional Style of Programming, 1/12: Working with Collections, 2/12: Creating Higher Order Functions, 3/12: Pattern Matching, 4/12: XML as First Class Citizen, 5/12: Recursions and Tail Call Optimization, 6/12: Using Traits, 7/12: Chaining Traits, and 8/12: Concurrency

Send the author your feedback or discuss the article in the magazine forum.

Calendar

Author sightings, partner events, and other notable happenings.

Here’s what’s coming up in the next three months:

Author Appearances

Who’s where, and what for.

Aug 3–5, Central Iowa Software Symposium, Des Moines, IA

Talks on JavaScript, HTML5, Scala, and Concurrency

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

August 9–11, CocoaConf, Columbus OH

“Core Audio All-Day Tutorial,” “Mobile Movies with HTTP Live Streaming,” and "Core Audio Cranks It Up”

Chris Adamson, co-author of iOS SDK Development

August 9–11, CocoaConf, Columbus OH

“Intro to iOS Programming”

Bill Dudney, co-author of iOS SDK Development

August 9–11, CocoaConf, Columbus OH

“iOS Performance Tuning with Instruments”

Bill Dudney, co-author of iOS SDK Development

Aug 10, CocoaConf,Columbus OH

“Storyboards for iOS”

Daniel H Steinberg, co-author of iPad Programming

Aug 11, CocoaConf,Columbus OH

“What's new in Mountain Lion”

Daniel H Steinberg, co-author of iPad Programming

August 12–17, Agile Alliance,Grapevine, TX

“What Will This Project Cost?” and “Building a Team Through Feedback”

Johanna Rothman, author of Behind Closed Doors: Secrets of Great Management, Manage It! Your Guide to Modern, Pragmatic Project Management, and Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects

Aug 12–17, Agile 2012, Dallas, TX

“Helping with Testing and Quality Assurance Stage Production”

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Aug 13–17, Agile 2012 Dallas, TX

“Agile In A Nutshell”

Jonathan Rasmusson, author of The Agile Samurai

Aug 14, ThatConference, Wisconsin Dells, WI

“FUD-Free Accessibility for Web Developers” also, Cake

Brian Hogan, co-author of Web Development Recipes, author of HTML5 and CSS3, and author of tmux: Productive Mouse-Free Development

Aug 14, Agile 2012, Dallas, TX

“Lean from the Trenches”

Henrik Kniberg, author of Lean from the Trenches

Aug 17, JRubyConf EUBerlin, Germany

“Deploy, Scale and Sleep Better with JRuby”

Joe Kutner, author of author

Aug 24–26, Research Triangle Software Symposium, Raleigh, NC

Talks on Groovy, Grails, Concurrency, and HTML5

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Aug 25, Madison Ruby, Madison, WI

“The Revenge of method_missing()”

Paolo "Nusco" Perrotta, author of Metaprogramming Ruby

Aug 30, Colorado Springs User Group, Colorado Springs, CO

TBD

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Aug 31, ALE 2012, Barcelona, Spain

Keynote

Henrik Kniberg, author of Lean from the Trenches

Sept 5, Swipe Conference, Sydney, Australia

“Understanding the iOS Graphics System Workshop”

Bill Dudney, co-author of iOS SDK Development

Sept 5–7, Swipe Conference, Sydney, Australia

“Intro to iOS Programming”

Bill Dudney, co-author of iOS SDK Development

Sept 6, Swipe Conference, Sydney, Australia

“Working with Core Data”

Bill Dudney, co-author of iOS SDK Development

Sept 6–7, Agile Brazil, Sao Paulo, Brazil

“Agile Portfolio Planning: Managing Your Project Portfolio” and “Workshop: Hiring For Your Agile Team” and “Workshop: Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects”

Johanna Rothman, author of Behind Closed Doors: Secrets of Great Management, Manage It! Your Guide to Modern, Pragmatic Project Management, and Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects

Sept 8–9, Barcelona, Spain

“The Three Laws of Bulls*it”

Paolo "Nusco" Perrotta, author of Metaprogramming Ruby

Sept 11, CVRegGlen Allen, VA

“The Rails View”

John Athayde, co-author of The Rails View

Sept 11, CocoaHeads, Toronto, CA

“Live Document Syncing with iCloud”

Jim Dovey

Sept 11, Agile RTP Raleigh, NC

“Uncomfortable with Agile”

Andy Hunt

Sept 14–15, GoGaRuCoSan Francisco, CA

“Deploy, Scale and Sleep Better with JRuby”

Joe Kutner, author of Deploying with JRuby

Sept 17, HTML5 Denver, Denver, CO

“CoffeeScript and Testing JavaScript”

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Sept 19–20, Frozen RailsHelsinki, Finland

“Deploy, Scale and Sleep Better with JRuby”

Joe Kutner, author of Deploying with JRuby

Sept 25, Agile Carolinas Charlotte, NC

“Uncomfortable with Agile”

Andy Hunt

Sept 27–28, SurgeBaltimore, MD

TBA

Michael Bevilacqua-Linn

Sept 30–Oct 4, JavaOne, San Francisco, CA

Multiple topics on Java, Groovy, and Scala

Venkat Subramaniam, the author of .NET Gotchas, the coauthor of 2007 Jolt Productivity Award winning Practices of an Agile Developer, the author of Programming Groovy, Programming Scala and Programming Concurrency on the JVM

Oct 1–2, Crisp, Stockholm, Sweden

“Advanced Agile” (co-teaching with Alistair Cockburn)

Henrik Kniberg, author of Lean from the Trenches

Oct 2, GOTO, Aarhus, Denmark

“Continuous Delivery”

Michael Nygard, author of Release It!

Oct 6, AgileEE, Kiev, Ukraine

Keynote

Henrik Kniberg, author of Lean from the Trenches

Oct 10-11, RiconSan Francisco, CA

TBA

Michael Bevilacqua-Linn

Oct 23, AgileDCWashington, DC

“Test Abstraction: Keeping Your Test-Driven Tests Healthy”

Jeff Langr, co-author of Agile in a Flash

O’Reilly Events

Upcoming events from our friends at O’Reilly.

Oct 1–2, Strata Conference, London, England

Strata Conference: “The O’Reilly Strata Conference in London will bring together the best minds in data to explore the opportunities made possible by the latest state-of-the-art big data tools and analytic approaches.”

Oct 2–4, Velocity Conference, London, England

Velocity Conference: “This October, hundreds of web ops and performance professionals will gather in London to share their ideas and expertise—and to discover the technical skills, tools, tips, tricks, and best practices they need to exceed the continually increasing expectations for a faster, stronger web.”

Oct 9, TOC Frankfurt, Frankfurt, Germany

TOC Frankfurt: “Tools of Change Frankfurt returns for a fourth year on Tuesday, 9 October 2012—the eve of the Frankfurt Book Fair (10–14 October)—gathering the best and brightest in our global publishing and technology community for a full day of intriguing keynotes, sessions, and networking.”

Oct 16–17, Strata Rx Conference, San Francisco, CA

Strata Rx Conference: “Strata Rx is the first conference to bring data science to the intractable and urgent issues confronting healthcare.”

Oct 20, Mini TOC Vancouver, Vancouver, B.C.

Mini TOC Vancouver: “While the focus at Mini TOC is on the intersection of technology and publishing—just like at the ‘big’ TOCs—at Mini TOC, the crowd is small, the atmosphere is intimate and informal, and all the emphasis is on attendees and presenters sharing the conversations.”

Oct 23–25, O’Reilly Strata Conference and Hadoop World, New York, NY

O’Reilly Strata Conference and Hadoop World: “Now in its second year in New York, the O’Reilly Strata Conference explores the changes brought to technology and business by big data, data science, and pervasive computing. This year, Strata has joined forces with Hadoop World to create the largest gathering of the Apache Hadoop community in the world.”

USENIX Events

What’s coming from our USENIX friends.

Aug 8–10, 21st USENIX Security SymposiumBellevue, WA

USENIX Security ’12: “The USENIX Security Symposium brings together researchers, practitioners, system administrators, system programmers, and others interested in the latest advances in the security of computer systems and networks. The Symposium will span three days, with a technical program including: Keynote Address by Dickie George, Johns Hopkins Applied Physics Laboratory; over 40 Refereed Papers covering the latest research including topics such as browser security, CAPTCHAs, and passwords; privacy enhancing technologies; botnets and Web security; and more.”

Aug 6–7, 2012 Electronic Voting Technology Workshop/Workshop on Trustworthy ElectionsBellevue, WA

EVT/WOTE ’12: “EVT/WOTE ’12 brings together researchers from a variety of disciplines, ranging from computer science and human-computer interaction experts through political scientists, legal experts, election administrators, and voting equipment vendors.”

Aug 6–7, 3rd USENIX Workshop on Health Security and PrivacyBellevue, WA

HealthSec ’12: “The focus of HealthSec ’12 will be on the development of new techniques and policies to ensure the privacy and security of next-generation healthcare systems and devices.”

Aug 6–7, 6th USENIX Workshop on Offensive TechnologiesBellevue, WA

WOOT ’12: “WOOT ’12 aims to bring together researchers and practitioners in systems security to present research advancing the understanding of attacks on operating systems, networks, and applications.”

Aug 6, 2nd USENIX Workshop on Free and Open Communications on the InternetBellevue, WA

FOCI ’12: “FOCI ’12 seeks to bring together researchers and practitioners from technology, law, and policy who are working on means to study, detect, or circumvent practices that inhibit free and open communications on the Internet.”

Aug 6, 5th Workshop on Cyber Security Experimentation and TestBellevue, WA

CSET ’12: “The science of cyber security is challenging for a number of reasons. Meeting these challenges requires transformational advances, including understanding of the relationship between scientific method and cyber security evaluation, advancing capabilities of underlying experimental infrastructure, and improving data usability.”

Aug 7, 7th USENIX Workshop on Hot Topics in SecurityBellevue, WA

HotSec ’12: “HotSec places its singular emphasis on new ideas and problems. HotSec takes a broad view of security and privacy and encompasses research on topics including, but not limited to, large-scale threats, network security, hardware security, software security, physical security, programming languages, applied cryptography, privacy, human-computer interaction, emerging computing environment, sociology, and economics.”

Aug 7, Seventh Workshop on Security MetricsBellevue, WA

MetriCon 7.0: “How to define, generate, and communicate security metrics you can use today.”

Oct 8–10, OSDI ’12, Hollywood, CA

10th USENIX Symposium on Operating Systems Design and Implementation: “The 10th OSDI seeks to present innovative, exciting research in computer systems. OSDI brings together professionals from academic and industrial backgrounds in what has become the premier forum for discussing the design, implementation, and implications of systems software.”

Other Happenings

Aug 9

Alfred Aho, co-creator of AWK language, is 72.

Aug 11

Steve Wozniak is 62.

Aug 12

The IBM PC was released 31 years ago today.

Aug 20

Game developer John Carmack is 42.

Aug 21

Linus Torvalds announces Linux project, 1991.

Aug 29

Boy genius Stephen Wolfram is 53.

Sept 1

This month the GNU Project turns 29.

Sept 2

First message sent over Internet, 1969.

Sept 2

sendmail creator Eric Allman is 57.

Sept 10

Space tourist Charles Simonyi is 64.

Sept 27

Perl creator Larry Wall is 58.

Oct 2

The Great Quux is 58.

Oct 4

The Free Software Foundation is 37.

Oct 4

Birthday of John Vincent Atanasoff, one of the inventors of the computer.

Oct 5

Linux is 21!

Oct 6

Adobe co-founder John Warnock is 72.

Oct 12

Birthday of Ole-Johan Dahl, co-creator of the Simula language and thus of object-oriented programming.

Oct 16

Mike Muuss, who invented Ping, was born on this date in 1958.

Oct 25

Peter Naur, the N in BNF, is 84.

Oct 28

Former software executive Bill Gates is 57.

Oct 29

On this date in 1969, ARPAnet first linked Doug Engelbart’s lab at SRI with Leonard Kleinrock’s at UCLA.

Oct 31

The traditional date for the leaking of confidential internal Microsoft documents outlining strategies for undermining open source software.

Shady Illuminations

Problems Solved

by John Shade

With some problems, the only problem is thinking there’s a solution, John thinks.

You won’t find me on Facebook. I don’t need fake “friends” who will ask me to voluntarily self-defriend if I don’t feel worthy of their friendship or when they have reached their friend quota. Like Mal Reynolds, I prefer real friends who will stab me in the back to my face.

In the real world, false friends eventually sort themselves out. You don’t know whom you can trust. You get stabbed in the back. Problem solved. Generally, I’ve found, things have a way of sorting themselves out. It’s what makes me such an optimist.

Evil Corporate Monopolies

Take evil corporate monopolies. In the middle of the 20th century, IBM dominated the computer industry. Corporate purchasers bought IBM to keep their jobs, not because of any technical supriority of its products. IBM employees wore the company uniform, sang the company songs, and toed the company line. IBM’s competitors were referred to as dwarfs. (That was the official spelling of the plural of dwarf before Peter Jackson wrote “Lord of the Rings.”) And then there was the whole supplying the Nazis thing.

Mainly, though, IBM represented the dehumanization of humanity or something like that. People were being folded and spindled and mutilated. I forget just what. It was bad, though. And when, in 1981, IBM revealed its sinister plan to take over the personal computer industry, too, it was clearly the beginning of the end for humanity.

But by 2004, IBM had sold off its personal computer division to the Chinese and was but a shadow of its former dark power. Problem solved.

In the meantime, Microsoft had replaced IBM as the evil corporate de facto monopoly to fear. Microsoft, as any sensible person could see, was going to crush all so-called “competitors” and embrace and extend all the life out of the industry and, with its commercials, all the joy out of life.

Then Microsoft began to fade. It would be fun to blame it all on Steve Ballmer, and personally I’m fine with that. Now Microsoft has posted its first quarterly loss and I’m sure we all agree that it’s all downhill from here. Pretty soon they’ll be selling off their software operation to China and will focus entirely on cloud services.

Meanwhile, Apple has become the evil corporate de facto monopoly to fear. (Google, of course, is protected from temptation by its magic do-no-evil mantra.) Apple, it turns out, was evil all along, but it never had the power to crush its competitors and enslave developers and users. Now it does.

But just this year the cracks began to form. You’ve seen the Genius commercials. You know where they are leading. Pretty soon Apple will hire Jerry Seinfeld or Jerry Lewis as its spokesperson and it’ll be all over.

My point is, there’s really no point in worrying about the unlimited power of some corporation over every aspect of human life because eventually that company is going to be dethroned by some other horror from hell.

These things have a way of sorting themselves out.

Guns

Or take guns. There’s apparently some sort of debate in the US about access to guns. As a coward, I naturally avoid debates that involve weapons. It would be my policy to befriend those who have them and hang out with those who don’t, except that I’m not into befriending or hanging out.

But soon the whole matter will be moot. It will be sorted out, as all important issues are, by technology.

It is now possible to print your own guns at home on your 3D printer. Proof of concept created, 200 rounds fired, debate resolved. Oh, and I said “gun,” but I guess we’d better make that “assault rifle.”

So check that one off. Before long practically everyone will have the means to print their own guns in any quantity they want, in the privacy of their own homes. (Once you’ve printed the first one, you have the means to acquire the money to print as many as you like.) Guns are going to go from a topic of national debate to a track at Maker Faires. These things have a way of sorting themselves out.

Lessons Learned

So if I can leave you with three lessons, they would be: ignorance is bliss, procrastination now!, and leave it for the night shift.

The TV series “Breaking Bad” almost didn't happen. As its creator explains, if he had known about “Weeds,” a TV show about a single mom supporting her family by selling pot, he wouldn’t have had the nerve to pursue his TV series about a dad supporting his family by selling meth. The moral of the story is, what you don’t know can help you. Also, if you can’t support your family on your coding skills, there are other options.

Yes, we could solve the problem of global warming if we had the will to do so, but if we procrastinate, the problem’s not going to go away and in the meantime we may put an end to the puzzle of why Greenland is icy and Iceland is green. Also, Greenland’s melting ice may expose large stores of precious metals. So there’s your silver lining.

Leaving it for the night shift is going to have a new meaning when workers who don’t sleep invade the workforce. Yes, unemployment is still unacceptably high, but remember that you’re only unemployed if you are actually looking for a job. And it looks as though the long-awaited robot workforce may be starting to arrive, making that problem moot.

These things have a way of sorting theselves out.

John Shade was born under a cloud in Montreux, Switzerland, in 1962. Subsequent internment in a series of obscure institutions of ostensibly higher learning did nothing to brighten his outlook. If ignorance is truly bliss, he doesn’t want to know it. Send him your feedback or discuss the article in the magazine forum.

But Wait, There’s More...

Coming Attractions

This may be the last page, but that doesn’t mean it’s the end of the conversation. Keep up with what’s going on in the Pragmatic universe by subscribing to our newsletter and drop by the Pub again next month for more Pragmatism. We’ll keep a seat warm and a mug chilled and a candle burning in the window.

Coming Soon in PragPub

This issue features two articles on functional programming, but that’s just a taste of what’s coming. We expect to have at least one article on thinking functionally in each issue through the end of the year.

Recent Releases from the Bookshelf

Deploying Rails: Automate, Deploy, Scale, Maintain, and Sleep at Night is now in print and shipping. Get a copy to dog-ear on your desk. Also, in addition to synching via Dropbox (see our FAQ), which we’ve had for a while, you can now also send your epub to Readmill, for social highlighting and sharing.  

Right. But to really be in the know, you need to subscribe to our weekly newsletter. It’ll keep you in the loop, it’s a fun read, and it’s free. All you need to do is create an account on pragprog.com (email address and password is all it takes) and select the checkbox to receive newsletters.

While you’re waiting for the next issue of the newsletter or of PragPub to come out, you can follow us on twitter at @pragpub, @PragmaticAndy, or @pragdave. Or on Facebook at facebook.com/PragmaticBookshelf. Or email us at michael@pragprog.com.