PragPub 2013-08: Issue #50

Table of Contents

Features

Programming Elixir

by Dave Thomas

Elixir is a modern, functional programming language designed for high availability and concurrency. It has Ruby-like syntax married to the power and reliability of the Erlang VM. If you wanted to get into functional programming but were put off by the academic feel, now’s the time to jump in.

Identity, Value, and State in Clojure

by Michael Bevilacqua-Linn

Michael looks at Clojure’s novel philosophy on identity, value, and state.

Finding the Joy in Legacy Code

by Jeff Foster

The people who wrote that code knew more than you do about the problem they were solving.

Big Problems

by Jesse Anderson

A cautionary tale.

3D Graphics with SceneKit

by David Rönnqvist

The ramp from 2D to 3D is steep, but a really good framework helps a lot.

Departments

Up Front

by Michael Swaine

It’s our 50th issue and the beginning of a new era.

Choice Bits

We follow Twitter so you don’t have to.

Rothman & Lester

by Johanna Rothman & Andy Lester

Johanna and Andy talk about how to articulate your own value in a résumé.

Puzzle

by Michael Swaine

A short session in mental calisthenics.

Calendar

Want to meet one of the Pragmatic Bookshelf authors face-to-face? Here’s where they’ll be in the coming months.

Bookshelf

What’s new and what’s hot from the Pragmatic Bookshelf.

Shady Illuminations

by John Shade

John embraces the metaphor of the software developer as a downhill skier, and encourages developers to embrace the trees they’ll encounter on their downhill run.

Rear Window

What Ed Roberts did after creating the personal computer.

Except where otherwise indicated, entire contents copyright © 2013 The Pragmatic Programmers.

Feel free to distribute this magazine (in whole, and for free) to anyone you want. However, you may not sell this magazine or its content, nor extract and use more than a paragraph of content in some other publication without our permission.

Published monthly in PDF, mobi, and epub formats by The Pragmatic Programmers, LLC, Dallas, TX, and Raleigh, NC. E-Mail support@pragprog.com, phone +1-800-699-7764. The editor is Michael Swaine (mailto:michael@pragprog.com). Visit us at http://pragprog.com

ISSN: 1948-3562

Up Front

The Second Iteration

by Michael Swaine

This month marks a transition for PragPub. As of this issue, the magazine is no longer free. After four years of giving the magazine away, Dave and Andy tasked me with developing a plan to sustain PragPub into the future.

We all agreed that we didn’t want to go the advertising route. So that meant asking readers to invest a little in the magazine. All 49 past issues remain available for free at the Pragmatic Bookshelf site, and new issues can be purchased at my site. Still available in pdf, epub, and mobi formats, still DRM-free and ad-free.

The other change happening this month is that, for the first time, you can also subscribe to PragPub. Others by author or as indicated.

As for the issue in front of you, we begin our Second Iteration with lots of goodies, including contributions by Dave Thomas, Jesse Anderson, Michael Bevilacqua-Linn, Jeff Foster, David Rönnqvist, John Shade, Andy Lester, and Johanna Rothman.

Dave concludes his popular series on the Elixir language, which may just be the language that gets you into functional programming for real. Michael B continues his exploration of another hot language, Clojure, while Jeff takes it in another direction altogether, sharing his wisdom on how to find the joy in legacy code. David dives into 3D graphics with a powerful framework from Apple, Jesse offers up a nice little article about big problems, Johanna and Andy tell you what really belongs in your résumé, and John pays peculiar reverence to an Agile hero.

Plus there’s all the regular stuff: Choice Bits, the Puzzle, Calendar, Bookshelf, and Rear Window.

In other words, the kind of cool stuff you’ve come to expect from PragPub. Now, with a little income, we should be able to keep bringing you the cool stuff. And in future issues, we may have a few surprises in store as well.

Keep in touch,

Mike

Photo Credits: p. 17 ©iStockphoto.com/zmeel, p. 27 ©iStockphoto.com/Michael_Dodd, p. 29 ©iStockphoto.com/Palto, p. 24 ©iStockphoto.com/StephanHoerold.

Choice Bits

From the Twitterstream

We follow Twitter so you don’t have to.

Programmers tweet the darnedest things.

This Coding Life

I’m not sure if I like this framework or I’m just suffering from Stockholm Syndrome. — @benrady

Drinking game at OSS conferences: Take a drink when speaker mentions GitHub. In other news, scores dead of alcohol poisoning at MonkeySpace. — @haacked

TIL: When you date a security researcher, getting handcuffed != sexy time, it means he decided it’s time for you to learn to pick a handcuff. — @pamelafox

I bet I’m the only person who’s been using Ruby since 2001 who still can’t remember whether it’s =~ or ~=. — @marick

I have an object that wants to be stateful for one operation and stateless for another, closely related, one. The tension is terrible. — @KentBeck

Very productive day so far. I’ve deleted more than 2,000 lines of code. — @mtnygard

Simple Questions

Why don’t our developers get fully engaged in the work we tell them to shut up and do? — @tottinge

The hardest thing about learning new tech is getting the conceptual framework. My question may be simple, but I don’t know how to ask. — @ultrasaurus

Re not peeing on that problematic third rail, is it true that urine is more conductive than water? — @GreatDismal

We all agree, today stops at 3, right? #millionmph — @tswicegood

The Post-PC Era

..@mattbraun @drbarnard One hint it’s already happening: when was the last time you heard anyone talk about the “Post-PC Era?” — @invalidname

Florida just banned all computers and smartphones. By accident. Supposedly. — @mims

“Don’t run with the iPad!” is the new “Don’t run with scissors!” — @hugs

This whole Internet thing is a fad. It’ll never catch on. — @jdalrymple

Instant Aphorisms

“Grandiose” is not a desired quality in design or code. — @PragmaticAndy

Poor code that works will carry the seeds of its own destruction. — @tottinge

If you can’t enjoy the quotidian, you can’t enjoy life FULL STOP — @amyhoy

Create fast should not imply drop quality, but to focus on the most essential features and business value. — @venkat_s

...@EthanZ on why the code words “metadata, not content” aren’t as comforting as they’re meant to sound http://goo.gl/bcyfC Recommended. — @jayrosen_nyu

Any rule of thumb is inherently opposable. — @pragpub

Deep Thoughts

The hipster is strong in downtown #Austin today. Did Portland or Brooklyn start shipping them to us? — @tswicegood

Joseph Campbell’s theory of monomyth, as explained by puppets: http://www.youtube.com/watch?v=yZxs_jGN7Pg — @antiheroine

Pretty sure there is a new iTunes version every day. — @SaraJChipps

@neal4d “Relax,” said the TSA agent. “We are programmed to receive.” — @scottdavis99

All these years as a tech writer, and preparing illustrations and screenshots STILL feels like eating my vegetables. — @Ihnatko

Douglas Engelbart

We should start an Engelbart Award, given to a promising technologist who develops truly new ideas and shares them freely. — @davewiner

Watch Douglas Englebart give the “Mother of all Demos” in 1968. http://is.gd/6tavIU — @GuyKawasaki

For some deeper insight into the contributions to high tech and culture made by Doug Engelbart, check this out: http://visualinsight.net/_engelbart/engelbart_mural.jpg — @MrAlanCooper

I met Doug Engelbart only once, briefly. What to say? Happily I found the right words: “Thank you.” — @judell

Who Are Those Guys?

First, they’re not all guys. Second, we have to confess that we cleaned up their punctuation and stuff a little. OK, who we followed this month:

Chris Adamson, Sarah Allen, Kent Beck, Sara Chipps, Alan Cooper, Jim Dalrymple, Scott Davis, Neal Ford, Pamela Fox, William Gibson, Phil Haack, Amy Hoy, Jason Huggins, Andy Hunt, Andy Ihnatko, Guy Kawasaki, Brian Marick, Christopher Mims, Jen Myers, Michael Nygard, Tim Ottinger, Ben Rady, Jay Rosen, Venkat Subramaniam, Michael Swaine, Travis Swicegood, Jon Udell, and Dave Winer.

Fair’s fair. You can follow us at www.twitter.com/pragpub.

Rothman & Lester

Articulating Value

by Johanna Rothman & Andy Lester

Johanna and Andy talk about how to articulate your own value in a résumé.

Johanna: This month, we’re talking about what to put on a résumé. It’s one of those things that everyone thinks: they know, but many people don’t.

Andy: What do you look for on a résumé?

Johanna: I look for an objective when I look at a résumé. Not because it’s a decider, but because I want to see what people put at the top of their résumé, after their name and address. But too many people put something useless, such as “Challenging position, doing blah blah blah.” That doesn’t help me at all. That just lumps a candidate in with everyone else.

Andy: I’ve often said that an objective is the worst way to start a résumé, because all it says is either A) nothing or B) that the writer is looking for the job for which he’s applying.

The “Nothing” is when the writer says, like in your example, “challenging position that will allow me to utilize my skills and knowledge to further the business goals of a successful organization.” That says nothing at all because that is why people get hired. Everyone employed uses their skills and knowledge to further the business goals of the organization.

The other one that I see is “Make your objective tailored to the position,” so then you see stuff like “Looking for a position as software developer using PHP and PostgreSQL,” when the job you are applying for is a software developer using PHP and PostgreSQL.

Johanna: Right. And, it’s even worse when a technical candidate starts listing “technical” skills such as Word, Excel, or PowerPoint.

Andy: What’s an example of an objective that you see as useful?

Johanna: When a manager is looking to become an architect, and you see something like that in an objective. “Looking for a senior technical position.” Well, that was wimpy, but it’s not a management position.

When a candidate differentiates what he or she is looking for, from the kind of position he or she has now, especially if it’s a different kind of position, such as manager to technical position, or developer to tester, or tester to business analyst. When you want to make a career change, tell me.

Andy: I’ve always thought that that sort of information should be in a cover letter, then.

Johanna: It should be. And, sometimes, it’s difficult to find a way to make a cover letter/email in the online application process. So put it in the objective, too. Cover letters don’t always stay with the résumé either.

Andy: So in the case of changing careers or positions, an objective might make sense. What about the other 95 percent of the time?

Johanna: Maybe an objective doesn’t make sense. I have a résumé that some of my clients want to see, because it’s a summary of my work. I don’t have an objective on it! It depends what you are using your résumé for.

Andy: So in the absence of an objective, how do you start off? I suggest that all résumés start off with contact information (no need for street address), and then 3 to 5 bullets listing their biggest strengths, depending on how much experience they have. Each bullet should be distinct from the others, so don’t make your three bullets Perl, Python, and PHP. At least one of the bullets should include a summary of your work experience such as “7 years’ web application development experience in Ruby” or “4 years’ experience in system administration with heterogeneous Windows and Unix environments.”

Johanna: Well, I like to know whether the person is local, so I know if we’re talking relocation, so I DO want to see a street address.

Andy: Can they get by with a city and state?

Johanna: Yes, that would work.

When you say biggest strengths, do you mean, “Successfully managed projects as small as 4 people/3 months to as large as 76 people/18 months?” Could that be a bullet?

Andy: Sure, that’s about right.

Johanna: That means, gentle readers, that listing skills such as Word, Excel, and PowerPoint are not what we want to see, right Andy?

Andy: No, not at all, and certainly not at the top of the résumé in prime visual real estate.

Johanna: That makes me crazy when I see that on a technical résumé. To me, it means I’m not looking at a technical person at all.

Andy: What people have to remember is that I am probably not going to read your entire résumé unless there is something in the first half of the first page that makes me think reading the rest of it will be worthwhile.

Johanna: Yup.

Andy: That’s why I feel so strongly about the objective. The objective starts out the one-way exchange with “Hi, my name is Bob, and here’s what I want from you,” rather than “Hi, my name is Bob, and I’ve got a decade of app development under my belt with X, Y, and Z.” Which one is going to make me want to read the rest?

Johanna: Oh, definitely the second. That’s why I want to read the experience in reverse chronological order. Just show me your experience, starting with the most recent experience first.

Andy: What do you want to read about in the experience? For me, I want details that the writers seem to gloss over.

Johanna: I want the candidate to tell me about the value that he or she provided to the organization. For each line in the résumé, there has to be some value in that experience. Quantify it. See here and here.

Andy: For example?

Johanna: If I managed a project, did I help that project meet its release date? Did I facilitate something that made the team better for some value? Did I save the company money? Did that project attract more customers? Did the project retain customers? Was the product from that project responsible for some amount of revenue?

Relate the work to the business. Now you have my attention.

You might be saying, “I’m a developer, I don’t know how to do that.” Sure you do. You may have paid down technical debt. You may have implemented a feature in fewer weeks. Maybe you helped a customer solve a problem.

Andy: In many cases, that requires writing in something other than “Résumé Bullet-ese.” It makes me sad when I see people hampered by what they imagine is a prohibition against writing sentences. I’m not saying write paragraphs, but you can write short, direct sentences that tell what it is you have to tell.

It’s OK to have a bullet point that says: “Created a front-end interface in node.js for a website that gathered 150,000 hits/day.”

You don’t have to write in stilted “Implemented node.js interface for website.”

And use numbers! Numbers add detail to your résumé that give a sense of scope and scale to the work you’ve done.

Johanna: Right. That tells me you know your stuff and the value of your work.

Andy: And it also doesn’t make me guess. Say you list “First-level technical support for the company” as a bullet. What’s that company? Your uncle’s real estate office with him and a secretary? That’s what I’m going to have to assume, unless you spell it out “First-level technical support for 200-seat office, answering average of 14 trouble tickets daily.”

The reason people write stuff like “First-level technical support for the company” is that when they write that, they can imagine what it is that they’re talking about. They haven’t considered that the reader does not have that same mental image of the work, and that that is what has to come across in the résumé.

Johanna: When you provide data in your résumé, you provide me, as the hiring manager, a chance to compare my position against your résumé. I see how close your experience is. Not only do I see how close the experience is, you look good to me.

I have a chance to ask you questions, first in the phone screen, and next in the in-person interview about how you’ve worked. You have a much better chance of making it to the interview with real data on your résumé.

At least, if I reject your résumé, it’s because your background is not close to what I need, it’s not because I imagine you are not what I need.

So what else do you look for, aside from numbers, so people articulate their value?

Andy: I want to hear about things they achieved, not day-to-day operations. “Wrote code for company IT department” is nothing. “Created a company-specific object-relational mapping (ORM) module, from design to coding to test suite to documentation. This ORM is now in use by 17 different developers in 3 different offices.”

Johanna: You want to read about completed projects, and their value?

Andy: Yes, projects. Yes, value. Yes, people/departments that she’s worked with. Yes, customer-focused. In the example above, the customers are the 17 other developers using her ORM.

Johanna: If you are an operations-type person, where you might not do real projects, you can talk about the effect your work has on people. You always talk about your value to the organization.

Andy: Right, you talk about your customers. And you talk about numbers that relate to the customers. How many calls did you take? How many print jobs did you run and for which departments in the company? How big was the data center you had to keep humming?

Johanna: If you’re a tester, you don’t just say, “Developed tests.” You say something like (and this is an off-the-cuff example) “Developed exploratory and automated tests for whiz-bang product to assess product readiness. We later received an award from so-and-so for quality. Product exceeded its expected sales by 1000 percent for the first year.”

Andy: One note there about your example, which I know is off the off-the-cuff, leads to another pet peeve of mine: “Résumé-ese” that tries to sound more “businessy” than it is.

Johanna: Oh, I was actually giving a real example of a real product in my past :-)

Andy: But I think that “to assess product readiness” is probably assumed by the reader, and doesn’t need to be said, and to my ears sounds like the writer is trying to put one over on me. You wouldn’t say to a friend of yours that your job is “to assess product readiness,” right? If you wouldn’t say it to a friend in conversation, don’t put it in your written résumé. Keep it simple like “Developed tests that decided if the product was ready to ship.”

Johanna: Good point. This is why you should always have a friend review your résumé. Always.

Andy: Heck, read it out loud! The act of voicing the words “to assess product readiness” should trigger something in your ear. (Not to pick on your example TOO much. :-))

Johanna: Why not? I wrote it! This is the value of reading your own words out loud. If you wince, someone else will, too.

So, articulate your value. And don’t forget to add awards, books, and patents to your résumé. You earned them. Talk about them.

Andy: What’s your #1 résumé annoyance that you wish you could shout from the rooftops and everyone would heed?

Johanna: My #1 annoyance is when people don’t put the dates of their jobs in an easy-to-read manner, say down the left margin, so I can read it. Don’t make me work to see how long you were in a job. I can subtract. I can tell when you went to school, or if you graduated. Don’t make me work for that information. Just tell me.

Andy: For me, it’s the absolute importance of getting every bit of your spelling and mechanics correct. We’re in the computer industry, and precision matters. When I see “PERL” or “Sql” or “JAVASCRIPT” it drives me nuts.

If I see “PERL” on your résumé, it tells me that you’ve actually spent very little time actually using Perl, or else it tells me you don’t care about spelling. In the Perl community, the spelling of “Perl” is a shibboleth. When we see “PERL,” we know it’s someone who’s new.

Johanna: Right. If I see things like that, it says to me that you don’t care enough to portray yourself in the best light.

Andy: Or that you don’t pay attention to detail.

Andy: Next month, we’ll discuss the ins and outs of interview preparation. What do you do in advance of the interview? How do you prepare, so you can nail the interview and get the job?

Johanna Rothman helps leaders solve problems and seize opportunities. She consults, speaks, and writes on managing high-technology product development. She enables managers, teams, and organizations to become more effective by applying her pragmatic approaches to the issues of project management, risk management, and people management. She writes the Pragmatic Manager email newsletter and two blogs on www.jrothman.com.

Andy Lester has developed software for more than twenty years in the business world and on the Web in the open source community. Years of sifting through résumés, interviewing unprepared candidates, and even some unwise career choices of his own spurred him to write his nontraditional book on the new guidelines for tech job hunting. Andy is an active member of the open source community, and lives in the Chicago area. He blogs at petdance.com, and can be reached by email at andy@petdance.com.

Send the authors your feedback or discuss the article in the magazine forum.

Programming Elixir

Part 3: Processes and Messages

by Dave Thomas

Elixir is a modern, functional programming language designed for high availability and concurrency. It has Ruby-like syntax married to the power and reliability of the Erlang VM. If you wanted to get into functional programming but were put off by the academic feel, now’s the time to jump in.

Last month, we rounded out a look at the basics of sequential Elixir code by looking at functions. So let’s expand our world out into a second dimension by introducing some concurrency.

The Actor Model

The actor model of concurrency is actually fairly old. It probably dates back to a couple of hundred thousand years ago, when humans first started communicating and remembering things. Fred would ask Wilma if she’d seen Barney, and Wilma would respond back to Fred, but only after she’d first told Bamm-Bamm to stop teasing Dino.

The essence of the actor model is just that—independent, asynchronous actors that communicate only by sending each other messages. The word only is significant—there is no state shared between actors.

Step forward to the mid 1970s, and Carl Hewitt proposed actors as a model of computation. In his mind, actors weren’t simply a means of achieving concurrency. They were instead a fundamental building block of software systems, much as objects are to OO programmers.

In fact, there’s a lot of similarity between an actor and an object. Both maintain their own independent state, which they encapsulate behind an API. But the two also differ. The obvious difference is that objects interact sequentially (threading notwithstanding) while actors can operate in parallel. But there’s another, deeper, difference implicit in this distinction. Actors do not have to process messages in the order they arrive. In fact, you don’t necessarily know the order in which messages arrived. And it turns out that this is liberating.

Actors and Elixir

Elixir runs on the Erlang virtual machine. And Erlang has support for actors built into the language. In fact, the actor model is one of Erlang’s defining characteristics.

And, perhaps surprisingly for something so rich, it’s really simple.

An actor in Elixir is called a process. You create new processes using one of the variants of the spawn function, send messages to it using the <- operator, and receive messages using a receive expression.

Let’s start with spawm:

| process_id = spawn fn ->
| IO.puts "In process #{inspect self}"
| end
| IO.puts "Spawned process #{inspect process_id}"

The spawn function runs the function you pass it as a separate process, returning the process id (from now on, we’ll use pid).

Notice that we use self inside the process we create. Anywhere in Elixir, self returns the current pid. Called in the spawned process, it will return that process’s pid. Called in the main process, we’ll get the main process’s pid instead.

If you run this code, you’ll likely see:

| $ elixir spawn1.exs
| Spawned process #PID<0.36.0>
| In process #PID<0.36.0>

However, you might also see:

| $ elixir spawn1.exs
| In process #PID<0.36.0>
| Spawned process #PID<0.36.0>

It’s up to the process scheduler whether the spawned process runs before the main process continues.

There’s another way of looking at spawn—it’s a function that transforms another function into a process. And, indeed, you can write the code that way.

| process_id = fn -> IO.puts "In process #{inspect self}" end |> spawn
| IO.puts "Spawned process #{inspect process_id}"

Messages

The expression:

| pid <- expression

sends the value of the expression to the process with the given pid. This value is queued in a per-process mailbox, and the sender continues to run.

When a process want to retrieve a message from its mailbox, it calls receive. This takes a block containing one or more clauses. Each clause consists of a pattern and an expression. Elixir looks at each message in the process’ mailbox in turn. For each message, it tries to match it against the patterns in the receive block. Once a match is found, it runs the corresponding code. If no match is found, Elixir waits until a new message arrives in the mailbox, and tests it. (This process means that any particular execution of receive will only inspect any message in the mailbox one time. However, if you call receive a second time, it will once again scan the entire mailbox.)

Let’s extend out an example, and have the spawned function wait for a message. When it receives it, it will write it in a cheery message.

| cheery = fn ->
| IO.puts "In spawned process #{inspect self}"
| receive do
| msg -> IO.puts "Well, hello #{msg}"
| end
| end
| pid = cheery |> spawn
| IO.puts "Spawned process #{inspect pid}"
| pid <- "world!"
| IO.puts "Message sent"

Running this, you may see output like:

| $ elixir spawn3.exs
| Spawned process #PID<0.36.0>
| In spawned process #PID<0.36.0>
| Message sent
| Well, hello world!

How would we get our process to send the message back to us, rather than just print it? The trick is to pass it our pid as part of the message we send it:

| cheery = fn ->
| IO.puts "In spawned process #{inspect self}"
| receive do
| { from_pid, msg } ->
| from_pid <- "Well, hello #{msg}"
| end
| end
| pid = cheery |> spawn
| IO.puts "Spawned process #{inspect pid}"
| pid <- { self, "world!" }
| IO.puts "Message sent"
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end

See how we defined a simple protocol for our message? Rather than passing a single value, we now pass a tuple containing our pid and message. The receiver uses pattern matching to break this message apart, allowing it to send the response back to the original process.

We can take this further. Let’s support different languages. We’ll add a language code to our protocol, and match multiple clauses in the receiving process.

| cheery = fn ->
| IO.puts "In spawned process #{inspect self}"
| receive do
| { from_pid, :en, msg } ->
| from_pid <- "Well, hello #{msg}"
| { from_pid, :fr, msg } ->
| from_pid <- "Bonjour #{msg}"
| { from_pid, :es, msg } ->
| from_pid <- "Hola #{msg}"
| end
| end
| pid = cheery |> spawn
| IO.puts "Spawned process #{inspect pid}"
| pid <- { self, :cn, "mundo!" }
| IO.puts "Message sent"
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end

| $ elixir spawn5.exs
| Spawned process #PID<0.36.0>
| In spawned process #PID<0.36.0>
| Message sent
| The process said: "Hola mundo!"

We’re on a roll, so let’s try sending a second message. (We’ll also remove some of the tracing.)

| cheery = fn ->
| receive do
| { from_pid, :en, msg } ->
| from_pid <- "Well, hello #{msg}"
| { from_pid, :fr, msg } ->
| from_pid <- "Bonjour #{msg}"
| { from_pid, :es, msg } ->
| from_pid <- "Hola #{msg}"
| end
| end
| pid = cheery |> spawn
| pid <- { self, :es, "mundo!" }
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end
| pid <- { self, :fr, "world!" }
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end

When we run this, we get the Spanish message, but then our process just hangs forever (modulo your patience). The problem is that our subprocess is no more—once it received the first message, it left the receive block and terminated. There was nothing around to receive the second message we sent. We’ll need to make our greeter loop, receiving requests. The easiest way to do this is to make it into a named function in a module.

| defmodule Greeter do
| def loop do
| receive do
| { from_pid, :en, msg } ->
| from_pid <- "Well, hello #{msg}"
| { from_pid, :fr, msg } ->
| from_pid <- "Bonjour #{msg}"
| { from_pid, :es, msg } ->
| from_pid <- "Hola #{msg}"
| end
| loop
| end
| end
| pid = spawn(function(Greeter, :loop, 0))
| pid <- { self, :es, "mundo!" }
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end
| pid <- { self, :fr, "tout le monde!" }
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end

The key here is the function loop. It handles a message, and then immediately calls itself, recursively, so it can handle the next message. Because Elixir turns tail recursion into a simple jump, this function can recurse forever without using any process stack.

Our code here is pretty poorly structured, though. Let’s give our module a proper API, and hide the implementation details (the loop) by making it private.

| defmodule Greeter do
| def loop do
| receive do
| { from_pid, :en, msg } ->
| from_pid <- "Well, hello #{msg}"
| { from_pid, :fr, msg } ->
| from_pid <- "Bonjour #{msg}"
| { from_pid, :es, msg } ->
| from_pid <- "Hola #{msg}"
| end
| loop
| end
| end
| pid = spawn(function(Greeter, :loop, 0))
| pid <- { self, :es, "mundo!" }
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end
| pid <- { self, :fr, "tout le monde!" }
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end

One of the cool features of this code is in the get_response function. We pass it the language for which we want a response, and use that language in the receive block to match only a message with that language code. Putting an uparrow in the pattern (^language) makes the pattern use the value in the variable, rather than assigning to it.

The main body of the code makes use of this. See how we submit requests for Spanish, French, and English (in that order), but we ask for the results in the order French, Spanish, English. And, sure enough, that’s what we see.

| $ elixir spawn8.exs
| Bonjour tout le monde!
| Hola mundo!
| Well, hello world!

Monitoring Your Processes

Remember the example where our child process didn’t loop and we sent it two messages. The first was processed, but the the child exited, and we never got a response to the second message.

We can handle this by linking our original process and the child process. If we do this, and one of the processes dies, the other will be send a message. Let’s go back to that example, and change our spawn call with spawn_link.

| cheery = fn ->
| receive do
| { from_pid, :en, msg } ->
| from_pid <- "Well, hello #{msg}"
| { from_pid, :fr, msg } ->
| from_pid <- "Bonjour #{msg}"
| { from_pid, :es, msg } ->
| from_pid <- "Hola #{msg}"
| end
| end
| pid = cheery |> spawn_link
| pid <- { self, :es, "mundo!" }
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end
| pid <- { self, :fr, "tout le monde!" }
| receive do
| msg -> IO.puts "The process said: #{inspect msg}"
| end

Run this, and you’ll see:

| $ elixir spawn9.exs
| The process said: "Hola mundo!"
| The process said: {:EXIT, #PID<0.36.0>, :normal}

Have a look at the second message we received. Rather than getting a French greeting, we get a tuple {:EXIT, pid, :normal}. This is Elixir sending us a message to notify us that the process exited normally.

In fact, all state information in Elixir is sent as messages. You can choose which processes you want to monitor, and you can create trees of these processes where each level monitors the processes below them. This is a large part of the ability of the Erlang VM to deliver 9-nines availability in highly concurrent systems.

A Final Example

Let’s use our knowledge of actors to write one final example program. This is a library that implements the map function. (The map function takes a collection and applies a function to each element, returning a new collection containing the results of each function call.) However, we’ll make this example more interesting by running each computation in a separate process. Without further ado, here’s the code:

| defmodule Parallel do
| import Enum, only: [map: 2]
| # Parallel map
| def pmap(collection, fun) do
| collection |> spawn_children(fun) |> collect_results
| end
| defp spawn_children(collection, fun), do: collection |> map(&spawn_child(&1, fun))
| def spawn_child(item, fun), do: spawn(__MODULE__, :child, [item, fun, self])
| def child(item, fun, parent), do: parent <- { self, fun.(item) }
| defp collect_results(pids), do: pids |> map(&collect_result_for_pid/1)
| defp collect_result_for_pid(pid) do
| receive do
| { ^pid, value } -> value
| end
| end
| end

The basic idea is a really nice example of the way functional languages express code. The main pmap function takes a collection, transforms it into a set of processes, and then transforms those processes into their results. The only new thing in this code is the use of spawn in the spawn_child function. Because we want to pass arguments, we use the variant of spawn that lets us specify a module, function, and argument list.

To give our code something to work on, we’ll write a deliberately inefficient function to calculate Fibonacci numbers.

| defmodule Sequence do
| def fib(0), do: 0
| def fib(1), do: 1
| def fib(n), do: fib(n-1) + fib(n-2)
| end

Some simple unit tests verify that it works:

| ExUnit.start
| defmodule MyTest do
| use ExUnit.Case
| import Parallel
| import Sequence
| test "basic fib works" do
| assert fib(10) == 55
| assert fib(30) == 832040
| end
| test "parallel map works" do
| assert pmap([1,2,3,4,5,6], &fib/1) == [1, 1, 2, 3, 5, 8]
| end
| end

But that isn’t the whole story. Let’s see if we can watch the parallelism at work. I’ll calculate some Fibonacci numbers. The graph shows the CPU utilization on my 4 core laptop.

| iex> c "pmap.exs"
| iex> Parallel.pmap [42,41,40,39,38,42,41,39,38], Sequence.fib(&1)
| [267914296, 165580141, 102334155, 63245986, 39088169, 267914296,
| 165580141, 63245986, 39088169]

We’re pegging all four cores until the end, when the code runs out of work and ends up finishing up a calculation using just one core.

Concurrency is the Core of Elixir

The functional aspects of Elixir are the most obvious external features. And they are what drew me to the language. But, after a while, you realize that they are really there to support the inherent concurrency in the environment.

And, for me, that’s been both the most difficult and the most rewarding aspect of adopting Elixir. Now, when I write code, I try to think of a decomposition not into objects, but rather into actors, or services. And given that Elixir can run literally millions of processes on even a modest laptop, these actors can be very fine-grained.

It’s an exciting time to be writing code, and Elixir is a gratifying language to use to write that code.

Dave Thomas is a programmer who likes to evangelize cool stuff. He cowrote The Pragmatic Programmer, and was one of the creators of the Agile Manifesto. His book Programming Ruby introduced the Ruby language to the world, and Agile Web Development with Rails helped kickstart the Rails revolution.

Send the author your feedback or discuss the article in the magazine forum.

Identity, Value, and State in Clojure

Clojure’s Answer to Unconstrained Mutability

by Michael Bevilacqua-Linn

Michael looks at Clojure’s novel philosophy on identity, value, and state.

Welcome back to a Pragmatic series on Clojure!

Last time we we took a look at one language feature that makes Clojure special, its Lisp-style macro system and the syntax that enables it. This code as data style is a very old characteristic of languages in the Lisp family, and Clojure is one modern instance of it.

Clojure’s Lisp-style macro system isn’t the only trick its got up its sleeve. In this article we’ll take a look at another one of its defining characteristics. Clojure embodies a novel philosophy on identity, value, and state. While this philosophy makes Clojure uniquely suitable for concurrent programming, I’ve found that it also reduces the complexity of non-concurrent programs.

In this article, we’ll explore Clojure’s philosophy on identity, value, and state. Then we’ll take a look at what this philosophy means for programs without a lot of concurrency, and some of the technology that makes it feasible. Next month, we’ll take a look at concurrent programming.

Clojure’s approach to identity, value, and state represents a fresh take on some very old computing concepts. Let’s start by taking a look at the model Clojure is trying to improve on, the traditional object-oriented approach to state. In one very important way, this approach has a lot more to do with the way computer memory works than with how humans think and model problems.

To understand why, let’s take a look at simplified computer memory. It consists of a series of eight memory cells, each which hold a bit of information. Every cell has an address, which represents a place. You can reach into that place and change what’s there, but once you change it, any notion of what was there in the past is lost.

Simplified Computer Memory

To change a value, the bit at a particular location is flipped. For instance, in the following example we change the value at memory location 8 from 0 to 1.

Changing a Value: a Dramatic Re-enactment

This simplified model is similar to the model presented by even modern computer memory. Modern memory holds many more bits, and the addressing schemes it uses are more complex, but the core model of memory as a series of cells with an address still holds.

One core abstraction we work with as developers in popular object-oriented languages are mutable object references to mutable objects. This makes memory friendlier to work with in many ways. However, in one important respect the model that they provide is much like the computer memory we just looked at.

Traditional object-oriented models encourage us to program as if we were using places to store our data, just like the cells in our simple memory model. Many object references may refer to the same place, however, if we modify an object through any of those references, all of the references may eventually see the modification.

For instance, the following diagram is a simple data model of a person which represents your humble author. Here, I’m modeled as an object with three references pointing at it, which stand in for Java-style object references. For the sake of the model, let’s say I commit a heinous crime involving three yaks, a razor, and an industrial-sized can of shaving cream.

On the run from the law, I need to change my name and address. Doing so using standard object-oriented tools would mutate the object that represents me. This means that my old name and address would be lost, as the following diagram demonstrates!

A Person as a Place: on the Run from the Law

This is not how human memory works. Just because I changed my name now, doesn’t mean that my past name is just wiped away. We’re used to computers working this way, so the strangeness inherent in this model isn’t readily apparent to the seasoned developer.

This model can have surprising effects. For instance, let’s say that we modified the object through the reference labeled me. What do the other references see?

In the simple case our program is single threaded and we might reasonably expect that once the person is modified, all subsequent attempts to view it through any of the references will see the modified value. Even this simple case can be fairly complex, especially for large software systems. Two pieces of code far apart in the codebase may modify the same data without realizing it, which can cause errors that are difficult to debug.

Worst of all is when the references reside in separate threads. In this case, two threads may attempt to modify the person concurrently. This causes all sorts of problems; we'll discuss how Clojure helps solve them next month. The traditional solution to these problems involves locks. Programming with locks is notoriously difficult, and has driven many a noble developer to the brink of insanity, or at least a career in marketing.

Clojure provides a different model. To explain it, let’s revisit the terms value, state, and identity and give them definitions that might be a bit different from the ones you’re used to.

In Clojure’s world, a value is an immutable piece of data, or a composite of immutable pieces of data. This differs from what we might consider a value in most object-oriented languages, where we’d consider both immutable data, such as integers and strings, and mutable data, such as most objects, to be values.

Next up is the notion of an identity. An identity is an entity that’s associated with a series of values over time. An identity is not a place, like a mutable memory cell, or the mutable objects we saw earlier. Neither is an identify a name, though it may be named, or have several names. For instance, I might refer to myself as “me,” but you, gentle reader, might call me “Mike.”

Finally, Clojure’s idea of state: In Clojure’s model, state is just the value of an identity at a point in time. This may seem like a strange definition of state at first blush, but it’s fairly intuitive. The state of a system is whatever value the system holds when it’s queried.

The following figure lays out our first cut at representing my run from the law in our new model. We’d make our person immutable. To change the name and address, we create a new immutable person that starts with the old one as a base. Then when we create a new person, we’d assign our reference to the new person.

A Person as a Series of Values: Still on the Run

This allows us to have some references refer to the new person and some to the old person, so if one thread is using the data while another thread attempts to modify it, it can simply use the old value.

The first bits of technology that help make this style of programming a reality are Clojure’s persistent data structures, maps, vectors, and sets. A persistent data structure is a data structure that preserves its previous version when it’s modified, making it appear to be immutable.

A deep explanation of how Clojure’s persistent data structures work is beyond the scope of this article, but let’s dig into them just a bit. Closure’s persistent data structures use a technique called structural sharing to avoid copying all of the data in the data structure. In addition, behind the scenes they’re built on top of trees with very high branching factors.

The upshot of this is that while there is a slight memory and performance penalty associated with Clojure’s persistent data structures, they’re both memory efficient enough and performant enough for most uses.

Let’s take a look at defining a persistent map to represent a person. It’s common in Clojure to use a map to carry data, rather than defining a new type as we would in an object-oriented language. We can create a map by putting a series of key-value pairs between curly braces, as we do so in the following code snippet.

| (def mike
| {:name "Michael"
| :address "123 Fake St."
| :age 30})

Since the map referred to by mike is immutable, we can’t modify it directly. Instead, we can use assoc to create a new, immutable map. The assoc function takes a map-like data structure and a variable number of key-value pairs, and associates them with the map. If the keys have values existing in the map, they are overwritten.

In the following code snippet, we use assoc to create a new map based on the one referred to by mike, and we create a new reference to mike. As we can see, the existing new-mike is not affected.

| => (def new-mike (assoc mike :name "Rollo"
| :address "4 Hideway Dr."))
| #'examples/new-mike
| => mike
| {:age 30, :name "Michael", :address "123 Fake St."}
| => new-mike
| {:age 30, :name "Rollo", :address "4 Hideway Dr."}

In addition to the persistent maps we just saw, Clojure has several other persistent data structures. Most notably, vectors, which give us fast lookup by index and fast appends all while remaining immutable, and sets, which are just immutable versions of normal sets. We’ll touch on these in future articles.

Now that we’ve seen how Clojure handles immutable data, let’s revisit the simple model of personasvalue. While it gets at the intuition behind Clojure’s approach to state, value, and identity, it’s got a fairly large problem. What is responsible for ensuring that the references are updated when the person takes on a new value, or ensures that the reference isn’t updated if a thread is working with the current value?

As with all problems in computer science, this can be solved with an additional layer of indirection. Enter the second bit of Clojure technology we’ll see: Clojure’s managed reference types. Clojure’s managed references can sit between an immutable data structure and the references to it and manage state changes. This ensures that all of the references being managed point at the latest value for a given identity.

To see a value, the reference that points to it must be de-referenced. De-referencing returns the state that the reference refers to, which captures it at the point in time it was de-referenced.

The following diagram shows one last model of our person. Here, we’ve got a managed reference through which our person is accessed. As we can see, this adds an additional layer of indirection. Both the personOne and aGuy references go through the managed references, while the me reference does not.

A Person as a Managed Reference: Immutably on the Run

Clojure has several different types of managed references, which are useful for handling different types of concurrency problems. The three types of references we’ll go over are atoms, agents, and refs. Atoms are generally all you need when you’re not dealing with a highly concurrent problem, so we’ll cover them in detail now and we’ll go over agents and refs next time.

Atoms provide a way to manage independent state synchronously. This means that they’re good for managing state that only involves a single managed reference, and where the calling thread wants the changes to happen synchronously.

To create a new atom we use the atom. To modify the reference and have it so it points at a new value, we can use swap!. The swap! function takes an atom, a function to call on the value it refers to, and additional arguments for the function.

The swap! function then runs the passed-in function on the value of the atom and the additional arguments, returning the new value. This is easiest to see by example. Here, we create an add-to-int-atom which adds an integer to the atom we just defined.

| (def int-atom (atom 0))
| (defn add-to-int-atom [value]
| (swap! int-atom + value))

Now we can use add-to-int-atom to add the number 42 to the number that the atom refers to.

| => (add-to-int-atom 42)
| 42

To de-reference the atom, we use the @ symbol. Here, we de-reference it into int-atom-snapshot.

| => (def int-atom-snapshot @int-atom)
| #'cim-clojure-class.2-state/int-atom-snapshot
| => int-atom-snapshot
| 42

Now we can create another reference, another-int-atom-reference, which refers to the same managed reference that int-atom does.

| => (def another-int-atom-reference int-atom)
| #'cim-clojure-class.2-state/another-int-atom-reference
| => @another-int-atom-reference
| 42

Now we’ll be able to see the results of the addition when we de-reference both int-atom and another-int-atom-reference.

| => (add-to-int-atom 68)
| 110
| => @int-atom
| 110
| => @another-int-atom-reference
| 110

However, the snapshot we took earlier remains unchanged.

| => int-atom-snapshot
| 42

This gives developers an additional degree of control and certainty even in programs without a high degree of concurrency. If you’ve de-referenced a piece of data from an atom, you can be sure it won’t change out from under you. This can be important even in a single threaded application.

This is because traditional object-oriented programs consist of a graph of mutable data structures. Making a modification at one node in the graph can mutate a piece of data many hops away. Even though these mutations are done through a well-defined interface, this can still lead to the sort of spooky action at a distance most commonly associated with global variables and quantum physics, two fairly obtuse phenomena.

Since Clojure’s approach gives us a way to ensure that this action at a distance can’t take place, it helps with the non-concurrent case as well as the concurrent one. That said, Clojure’s take on value, state, and identity was designed to give the programmer a better model for solving concurrent problems, so next month we’ll take a closer look at them.

We’ll dig deeper into atoms, to see how they behave in the face of concurrency, and we’ll explore Clojure’s other managed reference types. We’ll also introduce the software transactional memory system that they work in concert with. Together, these form a powerful model for concurrent programming.

Thanks for reading, and see you next month!

Michael Bevilacqua-Linn has been programming computers ever since he dragged an Apple IIGS that his parents got for opening a bank account into his fifth grade class to explain loops and variables to a bunch of pre-teenagers. He currently works for Comcast, where he builds distributed systems that power infrastructure for their next generation services, and wrote Functional Programming Patterns for the Pragmatic Bookshelf. He tweets occasionally at @NovusTiro.

Send the author your feedback or discuss the article in the magazine forum.

Finding the Joy in Legacy Code

It’s Out There

by Jeff Foster

The people who wrote that code knew more than you do about the problem they were solving.

Does anyone like working with legacy code?  Even if we don’t exactly agree on a precise definition of legacy code we all know it when we see it.  One of my favorite examples had the following code in its infamous supportutils library.

|     if (!fileExists("c:/some/path/")) {
|         // the file exists?
|         printf("WAT");
|     }

|     #define FILE_EXIST_SUCCESS 0

|     if (FILE_EXIST_SUCCESS == fileExists("c:/some/path")) {
|         printf("Sanity restored");
|     }

We’ll never know why this code exists, but we can probably agree it’s confusing and it makes associated code much more difficult to change with confidence.  This brings us nicely to my definition of legacy code:

Legacy code is code you are scared to change.

Why on earth would anyone want to work with a legacy code base?  

Most of us do work with legacy code but probably not through choice.  The majority of software engineering jobs involve the maintenance of code, and it’s rarely the clear, concise code that we’d all produce (Obviously. Of course. Without a doubt).  I’m sure that most of us have felt like a poor dung beetle at some point; all we do is roll the code around making the problem bigger.

How do you snap out of this way of working?

Confronting a legacy code base is one of the biggest challenges of software engineering. Entropy is a huge force to be reckoned with; adding another line to a method always feels easier (and less risky!) than splitting the method in two.  

Perhaps you have convinced yourself that fixing those pesky compiler warnings or adhering to naming conventions is improving your code base? After all, fixing coding standards issues is a much less risky change than refactoring a method.  Unfortunately, it’s much less risky because it produces little to no benefit.  Ask yourself whether removing all instances of Hungarian notation from your Java code base (or making all the parameters final) will really make new features easier to add? Be honest.

Maintaining the status quo is not the way to find joy in legacy code. If you’re on a legacy code project for some time and you want to quicken the pace of change on the project then you need to invest some time. I’m not going to lie, working with legacy code isn’t easy and it’s a hard battle to fight, but for me there’s no better way to learn software engineering than to roll your sleeves up and take the fight to the code.

So how do you set yourself up for success?  

It’s all too easy to go in with all well-intentioned “refactoring” guns blazing and tear that code base up.  Unfortunately, if you’re anything like me, this then ends up with a broken product.  Worse still, it often ends up with a subtlety broken product where the customer is finding the problem rather than you. If you’re working in a legacy code base, you’re going to need to take some care and work in a disciplined, incremental way.

The basic tools you’ll want in place are very simple and they are all based around the theme of giving you confidence and quicker feedback:

Continuous Build: Each time you make a commit to the code base, a build cycle runs giving you early alerting of any big problems.

Automated Tests: You’ll want (even though you might not have them yet!) a suite of tests that give you confidence that the product isn’t broken.

Whole Team Support: ’nuff said.

Putting a continuous build and test setup into a legacy code base is often a huge challenge in itself. In the worst case, the code may only compile on a developer machine. The effort of setting up a continuous build might be obscenely high, but until you’ve invested that time you’ll be destined to forever be pushing the dung around.

The problems you’ll face aren’t just technical. If you want to tackle legacy code head-on, then you’ll need the support of the whole team. Arguing the case for tackling legacy debt is hard, and any developer arguing that “they want to tidy up the persistence code” is going to have a hard time selling the idea. Instead, link refactoring work to stories in the backlog, and try to demonstrate that tackling the technical debt will lessen the time needed to implement those features.  

So, given all the work it causes, what is there to love about legacy code?  

Believe it or not, legacy code captures the wisdom and knowledge of engineers over the lifetime of the project. Given the state of most code, that might be hard to believe, and you’d be forgiven for raising a skeptical eyebrow. Instead, try to think that each engineer involved in the implementation was making the best decision based on the evidence they had at the time. Carefully analyzing the history of code can often reveal issues and business needs that aren’t immediately apparent.

Of course, you have more information now, so how do you control the urge to rewrite? It’s often said that one of the biggest mistakes engineers can make (as Joel Spolsky argues) is rewriting code. I completely agree with this point of view. Instead of rewriting, think of working with legacy code as software archaeology. Digging through the history of a particular function can reveal hidden requirements that rewriting will forget. The code is not a liability; it’s your biggest asset—if you treat it correctly.

The rate of change of code within a project is often revealing in itself.  Which files have changed the most? Which have changed the least? This can help guide important decisions, such as which parts of the system are worth getting under test. Chances are if code hasn’t been edited for 10 years, it probably works pretty well and doesn’t require as much testing as the code that’s changed every 5 minutes to handle a neverending stream of edge cases.

Technologies will come and go, but there will always be legacy code. If you can get into the right mindset (or at least a positive, explorative mindset) when you work with legacy code, then you’ll be rewarded with skills that will last a lifetime.

Jeff has been writing code for decades and enjoys the never-ending challenge of developing software. He currently works as Head of Software Engineering at Red Gate. He tweets about software development from @fffej.

Send the author your feedback or discuss the article in the magazine forum.

Big Problems

...and Data Wasn’t the Only One

by Jesse Anderson

A cautionary tale.

I worked at a startup once. It had Big Data problems or it would have if the company had been successful. I would sit there and wonder how we were going to solve them. I’d think about how many PhDs I’d have to hire to write the software the company needed to handle the Big Data problems. I’d wonder how long it would take for all of those PhDs to create something that was stable and usable.

The Airing of Problems

I wrote a program to analyze and graph the company’s metrics. It analyzed things like number of users and hits per time of day. One of the more difficult algorithms was a moving average of unique users per day that was stratified by the activity of a given user. The entire time I was writing the algorithm, I knew it was inefficient and wouldn’t scale. It was that algorithm’s continued efficiency that told me the company’s user base wasn’t coming back. It was funny that the algorithm I wrote to create a graph about the company’s health, told me the health without having to see the graph.

We needed temporal, geospatial queries. That meant that we needed to have queries that took place within a time range and that physically occurred within the radius of a given latitude and longitude. Each query’s latitude, longitude and radius would be different as well as the time range. Given the company’s plans for world domination, that was a big dataset that extended throughout the world. We had ideas about sharding databases, but there was always that edge case of how to deal with those queries that spanned databases.

Then there were the user recommendations. This problem could be split into the algorithm for creating the user recommendations and the distributed system for running those recommendations. Neither of these problems was easy. The company’s recommendations use case led to exponential growth every time another user signed up. That required a distributed system that could scale to handle the recommendation system’s ever increasing load.

There’s a Distributed System for That

It just so happens that Hadoop can handle all three of these problems. You’ll have to do some coding; but like all frameworks, you won’t have to start from scratch.

As I worried, starting from scratch on a distributed system means a great deal of work, lots of PhDs, and time. Time in startup-land is at a premium.

The first problem was handling click-stream data. All of the data could have been stored in HDFS and processed using MapReduce. Using MapReduce to calculate moving averages can scale to the levels that the company hoped for. Each of the graphs that the executives came to rely on could have been produced at scale using MapReduce.

We move on to our database problem and handling temporal, geospatial data.

In the Big Data world, we go straight for HBase. HBase sits on top of Hadoop. It handles the scaling or sharding automatically for us. It could hold all of the company’s data without having to start sharding our data in odd ways. Using geohashing, we could query for very specific areas of the world without having to look at the entire world’s dataset.

You can kill two birds with one stone using another ecosystem project called Mahout. Mahout has implementations of various recommendation engines and other machine-learning algorithms. Better yet, many of these algorithms, including the recommendation-engine algorithm, are written to use MapReduce. This way the load of new data could scale using the cluster’s resources with MapReduce. Using Mahout, we could have given our users’ recommendations and scaled as the demand increased.

Conclusion

When starting a company or a new project, you look at the various requirements. Not all companies have true Big Data problems or start out with Big Data problems. Some companies opt to ignore the future pain of re-engineering in favor of current glory. It requires an extra investment upfront to handle Big Data problems from the very beginning.

Even though there are other distributed systems like Hadoop, they don’t negate the need for high-quality engineers. Hadoop and its ecosystem still require a good programmer. However, it will give you a significant head start and save you money. You’ll be able to focus on the software that makes your company special and not the software infrastructure behind the scenes. In the end, that’s what your customers care about.

Jesse Anderson is a Creative Engineer in Reno with many years of experience in creating products and helping companies improve their software engineering. He works at Cloudera on the Educational Services team as a Curriculum Developer and Instructor. Jesse works on both professional and personal projects. A recent personal project, Million Monkeys Shakespeare project, went viral and gained international notoriety. Jesse’s interviews with national media outlets including the Wall Street Journal and Fox News. He volunteers his time as the President of the Northern Nevada Software Developers Group and he sits on the Technology Advisory Committee at Morrison University.

His blog and website is jesse-anderson.com and his Github account contains other example MapReduce programs.

Send the author your feedback or discuss the article in the magazine forum.

3D Graphics with SceneKit

Scaling Complexity

by David Rönnqvist

The ramp from 2D to 3D is steep, but a really good framework helps a lot.

Rich interfaces and interactions are key components to applications on both OS X and iOS that users have come to expect. Powerful frameworks like Core Graphics, AppKit, UIKit, Core Animation, and Core Image enable you both to get up and running quickly and to experiment and iterate on a design. It is what makes us able to write great 2D interfaces every day.

But the step from 2D graphics to 3D graphics and OpenGL (or OpenGL ES) is very steep and doesn’t come naturally. Not only are there new concepts and a completely new API to learn but we also have to say goodbye to all the frameworks we’ve come to learn as Cocoa developers and re-implement our own low-level animations, image processing, etc.

I’ve been wanting to get more people interested in 3D graphics but it has been too hard to get past the first hurdles—and 3D just hasn’t been approachable for everyone. Last year at WWDC, Apple did something to remedy this when they introduced SceneKit for OS X.

SceneKit is a high-level 3D framework for Mountain Lion that integrates with other UI frameworks like AppKit and Core Animation. Images and colors are regular NSImage and NSColor objects, the SCNView supports hit testing of 3D objects from a 2D point and animating a property is as simple as adding a CAAnimation to an object in the scene. It’s all the things we are used to but in 3D.

The world in SceneKit is called a scene. It contains lights, cameras, and pieces of geometry, each attached to a node that controls the position, scale, and rotation of the object attached to it. Just like views and layers have sublayers, nodes have child nodes that are managed in much the same way.

To get something 3D on the screen, you create a light, a camera, and perhaps one of the built-in geometry types, like a box, and attach them all to a node. The nodes are then positioned and added to the root node of the scene. You can animate the rotation of the box by adding a basic animation to its node.

| // A camera
| SCNNode *cameraNode = [SCNNode node];
| cameraNode.camera = [SCNCamera camera];
| cameraNode.position = SCNVector3Make(0, 0, 30);
| [scene.rootNode addChildNode:cameraNode];

| // A spotlight
| SCNLight *spotLight = [SCNLight light];
| spotLight.type = SCNLightTypeSpot;
| SCNNode *spotLightNode = [SCNNode node];
| spotLightNode.light = spotLight;
| [cameraNode addChildNode:spotLightNode];

| // A square box
| CGFloat boxSide = 15.0;
| SCNBox *box = [SCNBox boxWithWidth:boxSide
| height:boxSide
| length:boxSide
| chamferRadius:0];
| box.firstMaterial.diffuse.contents = [NSImage imageNamed:@"logo"];
| SCNNode *boxNode = [SCNNode nodeWithGeometry:box];
| [scene.rootNode addChildNode:boxNode];

| // Rotating the box
| CABasicAnimation *boxRotation =
| [CABasicAnimation animationWithKeyPath:@"rotation"];
| boxRotation.byValue =
| [NSValue valueWithSCNVector4:SCNVector4Make(1, 1, 0,
| 2.0*M_PI)];
| boxRotation.timingFunction =
| [CAMediaTimingFunction functionWithName:kCAMediaTimingFunctionLinear];
| boxRotation.repeatCount = INFINITY;
| boxRotation.duration = 5.0;

| [boxNode addAnimation:boxRotation
| forKey:@"RotateTheBox"];

From there you can experiment with properties on the geometry’s material, the lights and the camera to play around with lighting, shadows, textures, animations, and many other things. You can extend the scene with more geometry, more lights, multiple cameras, simply by adding more nodes to the hierarchy. If you want a more detailed introduction to SceneKit you can find one by Jeff LaMarche and one by me, but neither should be necessary to follow along in this article.

Scaling Complexity for Features and Learning

Even though SceneKit is a high-level framework, it is built on top of OpenGL and allows you to access the lower-level technologies where needed while using the higher-level technologies everywhere else. If you want to learn more or have special requirements for rendering or geometry then you can focus your efforts on only those areas.

Custom Rendering

Rendering in OpenGL is done by writing shader programs in GLSL, the OpenGL Shading Language. The two main shader types are called vertex shaders and fragment shaders. Vertex shaders operate on the vertex data (position, surface normal, etc.) to move or transform the geometry. The transformed vertices are then assembled into primitives like lines and triangles. These are rasterized and the fragment shader is used to calculate the color for each fragment (pixel).

Any geometry object can be rendered completely custom by supplying your own vertex and fragment shader in a SCNProgram. Creating a program from the two shaders is as easy as reading the files and assigning their content to the vertexShader and fragmentShader properties on the program.

| SCNProgram *program = [SCNProgram program];
| NSURL *vertexShaderURL = [[NSBundle mainBundle]
| URLForResource:@"shader" withExtension:@"vert"];
| NSURL *fragmentShaderURL = [[NSBundle mainBundle]
| URLForResource:@"shader" withExtension:@"frag"];
| NSString *vertexShader =
| [[NSString alloc] initWithContentsOfURL:vertexShaderURL

| encoding:NSUTF8StringEncoding
| error:NULL];
| NSString *fragmentShader =
| [[NSString alloc] initWithContentsOfURL:fragmentShaderURL

| encoding:NSUTF8StringEncoding

| error:NULL];

| program.vertexShader = vertexShader;
| program.fragmentShader = fragmentShader;

Common things like vertices, surface normals, or the model-view projection matrix can be bound to a variable name in the shaders by calling setSemantic:forSymbol:options for each semantic-variable pair.

| [program setSemantic:SCNGeometrySourceSemanticVertex
| forSymbol:@"vertex"
| options:nil];
| [program setSemantic:SCNModelViewProjectionTransform
| forSymbol:@"modelViewProjection"
| options:nil];

This can be a good way of starting out with the GLSL. You can swap out the geometry or move the node around the scene and have the changes automatically reflected in the shader inputs. You can even continue animating properties in the scene.

Custom Geometry

Another area to dive deeper into is the geometry in the scene. If the main purpose is to render a complex scene with custom geometry then maybe generating the geometry data yourself isn’t the best way to go. Instead you should let a 3D designer create a collection of objects or perhaps a full scene using 3D modeling software and import it as a DAE file. If you want to broaden your 3D skills then you can download Blender and create some 3D scenes yourself. Blender Cookie has a good introductory video series and if you have further questions, why not try asking them over at the Blender Stack Exchange site.

While 3D modeling software is the more efficient way of dealing with most complex scenes, there are still two good reasons to generate the geometry data yourself: working with dynamic data that can’t be modeled beforehand and learning how geometry data is constructed behind it all.

To create a geometry object that works with lights, materials, and textures in SceneKit you need to generate a list of vertices, surface normals, and texture coordinates. Generating the actual data is probably going to be harder than creating the object. GLKit will help you with the linear algebra to calculate surface normals and you can translate GLKVector3s to SCNVector3s. If you are interested in more details about how vertices come together to form surfaces made out of triangles then you can read more about it here.

With all the data generated you can create geometry sources for the vertices, normals, and texture coordinates and a geometry element for the indices that is used to assemble the vertices into triangles.

| SCNVector3 vertices[pointCount] = { /* a list of vectors */ };
| SCNVector3 normals[pointCount] = { /* a list of vectors */ };
| CGPoint textures[pointCount] = { /* a list of points */ };
| int indices[numberOfIndices] = { /* a list of indices */ };

| SCNGeometrySource *vertexSource =
| [SCNGeometrySource geometrySourceWithVertices:vertices
| count:pointCount];
| SCNGeometrySource *normalSource =
| [SCNGeometrySource geometrySourceWithNormals:normals
| count:pointCount];
| SCNGeometrySource *textureSource =
| [SCNGeometrySource geometrySourceWithTextureCoordinates:textures
| count:pointCount];

| NSData *indexData = [NSData dataWithBytes:indices
| length:sizeof(indices)];
| SCNGeometryElement *element =
| [SCNGeometryElement geometryElementWithData:indexData
| primitiveType:SCNGeometryPrimitiveTypeTriangleStrip
| primitiveCount:numberOfIndices
| bytesPerIndex:sizeof(int)];

| SCNGeometry *geometry =
| [SCNGeometry geometryWithSources:@[vertexSource,
| normalSource,
| textureSource]
| elements:@[element]];

From geometry sources and geometry elements you can create your own custom geometry object. It works just like any other geometry object. You can assign materials to it, add it to nodes, and hit test it just as if it were one of the built-in geometry classes.

Custom, Custom, Custom

These two things can of course be combined to both generate and render your own geometry, at which point it almost feels like you are at the level of OpenGL. In reality, there is still convenience from the SCNProgram and the SCNGeometry object that take care of compiling the shaders, binding the uniforms, and dealing with vertex attribute data. If, however, you have a reason to do these things yourself, you can drop down to use the OpenGL C API and render the node completely custom. This is done by becoming the render delegate of the node.

The SCNNodeRendererDelegate protocol contains only one method to implement: renderNode:renderer:arguments. When it is time to render the node, the delegate method is invoked with the OpenGL context already set up and all the necessary transforms like the model view projection transform passed in the arguments dictionary. An explanation of any actual OpenGL code is outside the scope of this article.

Conclusion

If you’ve tried to learn about 3D graphics before but never got around to doing it or have worked with 3D graphics and thought “if I could animate this with a CAAnimation then that would be really cool,” then maybe you should take a look at SceneKit.

David Rönnqvist is an iOS developer from Sweden with a passion for graphics and animations. On his spare time he plays around with Mac and iOS code, spends time on Stack Overflow, teaches kung fu to kids and writes about programming (and sometimes about its underlying math) on his site. Occasionally he tweets about some of this as @davidronnqvist.

Send the author your feedback or discuss the article in the magazine forum.

Puzzle

A Sudoku-Anagram Mashup

by Michael Swaine

A short session in mental calisthenics.

Here’s another programming-related sudoku puzzle.

It uses letters rather than numbers, but otherwise it works just the way you expect. Your task is to fill in the empty cells so that every row, column, and each of the 3x3 squares contains one each of the nine letters you’ll find in the cells already filled in.

Oh, one more thing. When you finish the sudoku, rearrange those nine letters to spell a name shared by two characters in a television show that should be familiar to every programmer.

R
X
X

X
A
X

X
S
X

X
A
X

X
X
N

X
X
X

X
X
X

W
H
X

X
X
X

S
X
X

X
H
N

X
X
X

R
E
X

X
X
X

X
S
X

B
X
X

R
E
X

X
A
X

E
X
X

X
X
X

X
B
X

X
X
R

X
H
B

X
U
X

X
X
X

U
X
X

X
X
X

And the name is: _ _ _ _ _ _ _ _ _ .

Solution right here next issue. Or on Twitter right now, no doubt.

Solution to Last Month’s Puzzle

U
C
W

A
H
Y

J
L
M

M
H
Y

L
C
J

A
W
U

A
L
J

W
M
U

Y
C
H

H
Y
C

W
U
A

L
M
J

U
L
A

J
M
H

W
Y
C

M
J
W

L
Y
C

U
H
A

Y
A
L

M
J
U

C
W
H

H
J
W

C
A
L

Y
U
M

C
U
M

H
W
Y

J
A
L

And the secret word is: JWMAUCHLY, or John Mauchly, co-inventor of ENIAC.

Calendar

Want to meet one of the Pragmatic Bookshelf authors face-to-face? Here’s where they’ll be in the coming months.

2013-07-25, OSCON, Portland

Getting Started with 3D Programming in Three.js

Chris Strom (author of The SPDY Book , Dart for Hipsters , and 3D Game Programming for Kids )

2013-08-05, Agile 2013, Nashville, Tennessee, USA

Chairing this years Agile 2013 UX Stage with Will Evans

Eewei Chen (author of 101 Design Ingredients to Solve Big Tech Problems )

2013-08-06, Agile 2013

Agile Teams and Collaboration: What's New About Agile?

Johanna Rothman (author of Behind Closed Doors , Manage It! , Manage Your Project Portfolio , and Hiring Geeks That Fit )

2013-08-07, Agile 2013

Transparent Decisions: Managing the Project Portfolio

Johanna Rothman

2013-08-08, Agile 2013

The Latest in Mocks for [Legacy Embedded] C

James Grenning (author of Test Driven Development for Embedded C )

2013-08-10, Berlin, Germany

Master of Ceremonies for an unusual Ruby Conference

Chad Fowler (author of The Passionate Programmer (2nd edition) and Rails Recipes )

2013-08-12, The Unix Fu Workshop, Online

Unix systems programming for Ruby developers

Jesse Storimer (author of Working with Unix Processes and Working with TCP Sockets )

2013-08-12, That Conference, Wisconsin Dells, WI

Keynote and smores

Dave Thomas (author of Programming Ruby (2nd edition) , Agile Web Development with Rails (3rd edition) , The Ruby Object Model and Metaprogramming , Agile Web Development with Rails 4 , Programming Ruby 1.9 & 2.0 (4th edition) , Agile Web Development with Rails 3.2 , and Programming Elixir )

2013-08-12, That Conference (Wisconsin Dells)

Building Native iOS Apps With RubyMotion

Jonathan Penn (author of Test iOS Apps with UI Automation )

2013-08-13, ThatConference, Wisconsin Dells

Resumes, Red Flags, and Rock Stars: How to Hire a Great Development: What really determines the success of a development team? Put scrum aside for a second. This talk is about how to hire and cultivate a great development team.

Ka Wai Cheung (author of The Developer's Code and The Developer's Code (audio book) )

2013-08-13, ThatConference, Wisconsin Dells, Wisconsin

Testing Client Side Code with Jasmine and CoffeeScript With more logic moving from the client-side to the server-side, we've got to get better at testing code. In this talk you'll learn about writing tests for client-side code that help drive the d

Brian P. Hogan (author of HTML5 and CSS3 , Programming Concurrency on the JVM , The Rails View , Crafting Rails Applications , Deploying Rails , iOS SDK Development , Web Development Recipes , The Developer's Code , Deploying with JRuby , tmux , Programming Groovy 2 , Test iOS Apps with UI Automation , The Healthy Programmer , HTML5 and CSS3 (2nd edition) , and Crafting Rails 4 Applications )

2013-08-14, CocoaConf Portland

iOS 2D Game Development Workshop

Jonathan Penn

2013-08-14, CocoaConf Portland

iPad Productivity APIs workshop

Chris Adamson (author of iOS SDK Development )

2013-08-15, CocoaConf Portland

NSIncrementalStore - Bet you didn't think Core Data could do *that*!

Jonathan Penn

2013-08-15, CocoaConf Portland

Get on the Audiobus

Chris Adamson

2013-08-16, CocoaConf Portland

Glitch-Free A/V Encoding

Chris Adamson

2013-08-16, CocoaConf Portland

Testing iOS Apps With UI Automation

Jonathan Penn

2013-08-17, Eurucamp Berlin

Modelling State Machines with Ragel

Drew Neil (author of Practical Vim )

2013-08-18, Problem Solving Leadership Workshop, Albuquerque, NM

5.5 days of leadership training for people who want to learn to be problem solving leaders.

Johanna Rothman

2013-08-29, devLink, Chattanooga, TN

How to be a Healthier Programmer: Your health affects your memory and creativity--skills critical to doing your job well. In this talk you'll learn to change habits, boost brainpower, and master exercises that make working at a computer more comfortable.

Joe Kutner (author of Deploying with JRuby and The Healthy Programmer )

2013-09-05, Agile on the Beach, Falmouth, UK

Workshop: The Art of Slicing and Dicing User Stories

Rachel Davies (author of Agile Coaching )

2013-09-10, 360|iDev - Denver, CO

Dynamic Audio for Apps and Games - In this talk you'll get a brief introduction to the field of sound design and different techniques that can be used to create sound. You'll then be introduced to PureData, an application and platform for working with

Tony Hillerson

2013-09-11, BDD Kickstart, Barcelona

Get a flying start with BDD, the collaborative process that's changing the face of software development.

Matt Wynne (author of The Cucumber Book and Cucumber Recipes )

2013-09-14, Barcelona Ruby Conference

Hexagonal rails

Matt Wynne

2013-09-14, Baruco

TBD

David Chelimsky (author of The RSpec Book )

2013-09-17, CukeUp! NYC

A 1-day conference dedicated to Cucumber, Specification by Example and BDD.

Aslak Hellesøy (author of The RSpec Book , The Cucumber Book , and Cucumber Recipes )

2013-09-19, Hamburg, Germany

Speaking about something fascinating!

Chad Fowler

2013-09-24, Cambridge, UK

Workshop: Slicing and Dicing User Stories and Experience Report: XP at Unruly.

Rachel Davies

2013-09-26, CocoaConf Columbus

iPad Productivity APIs workshop

Chris Adamson

2013-09-27, CocoaConf Columbus

Intro to iOS Gesture Recognizers

Jonathan Penn

2013-09-27, CocoaConf Columbus

NSIncrementalStore - Bet you didn't think Core Data could do *that*!

Jonathan Penn

2013-09-27, Amsterdam, Netherlands

Speaking about something fascinating!

Chad Fowler

2013-09-27, ISVCON 2013 - Reno

I'll be covering how ISVs can use The Cloud to lower costs and improve customer satisfaction.

Jesse Anderson (author of The Cloud and Amazon Web Services and Processing Big Data with MapReduce )

2013-09-30, Aarhus, Denmark

Keynote: The Passionate Programmer

Chad Fowler

2013-10-09, Tel Aviv, Israel

Disposable Components

Chad Fowler

2013-10-17, GOTO Berlin

Keynote speech and case study on how we apply eXtreme Programming to product development at Unruly.

Rachel Davies

2013-10-25, Rakuten Technology Conference, Tokyo

Talk (and possibly a tutorial)

Dave Thomas

2013-10-27, RubyConf China, Beijing

Keynote

Dave Thomas

2013-11-01, London, UK

The Emperor's New Clothes: Meaningful Interactions in Stressful Situations

Portia Tung

2013-11-08, Software Craftsmanship North America, Chicago

Talk, to be determined

Dave Thomas

2013-11-18, BITS - Software Developer Conference, Denmark

I'll be speaking on TDD, and have a hands-on workshop. We'll also offer a 2-day public training class for TDD and embedded C/C++

James Grenning

Bookshelf

Pragmatic Bookshelf News

Here’s what’s new and what’s hot from the Pragmatic Bookshelf.

What’s New

Here’s what’s coming next to the Bookshelf:

Seven Web Frameworks in Seven Weeks by Jack Moffitt and Fred Daoud, Processing Big Data with MapReduce Screencast by Jesse Anderson, and Programming Erlang, 2nd Edition by Joe Armstrong (in print).

What’s Hot

Top-Ten lists are passé—ours goes to 11. These are the top titles that folks are interested in currently, along with their rank from last month. This is based solely on direct sales from our online store.

1^ | 10 | The Healthy Programmer
2^ | 3 | Agile Web Development with Rails 4
3v | 2 | Programming Ruby 1.9 & 2.0
4v | 1 | Programming Elixir
5 | 5 | Crafting Rails 4 Applications
6^ | NEW | Modern C++ Programming with Test-Driven Development
7^ | NEW | Practical Programming
8v | 7 | The Definitive ANTLR 4 Reference
9^ | NEW | Programming Groovy 2
10v | 6 | Programming Erlang
11v | 4 | HTML5 and CSS3

What’s Happening

But to really be in the know, you need to subscribe to our weekly newsletter. It’ll keep you in the loop, it’s a fun read, and it’s free. All you need to do is create an account on pragprog.com (email address and password is all it takes) and select the checkbox to receive newsletters.

Shady Illuminations

Implementing Alistair Cockburn

by John Shade

John embraces the metaphor of the software developer as a downhill skier, and encourages developers to embrace the trees they’ll encounter on their downhill run.

I’ve never met Alistair Cockburn. It’s probably for the best. He seems like the sort of guy who needs a lot of stimulation. I’m not that stimulating.

Tother Alistair

Cockburn is one of the spark plugs of the agile software development movement. You probably knew that. He’s too much of an over-achiever to merely have created a software methodology. He’s created a whole family of software methodologies. Like Friedrich Mohs, he has an eponymous hardness scale. He writes, he speaks, he consults, he codes. He’s a TED Talker, a big Agile cheese, a lightweight methodology heavyweight.

Cockburn also jumps out of airplanes and off cliffs, plays with sting rays, and explains ontology. He claims to like the “texture” of sitting still, but only underwater. His motto used to be “stasis is stagnation.” So yeah, not a stasis kind of guy. Later he changed it to “lean downhill,” which sounds sort of lazy until you realize that the metaphor is from skiing. Leaning into danger yields maximum dynamic stability. Something like that. He says he considers boredom the lowest depth of misery. And that he hasn’t experienced much of it.

His Facebook page describes Cockburn as chaotic. Says he “wreaks havoc on the unsuspecting.” Since he presumably wrote that description himself, that would seem to be how he chooses to be seen by the world. Or by Facebook visitors, anyway. He’s apparently a big Calvin and Hobbes fan, too. I’m guessing more Calvin than Hobbes.

Cockburn is full of ideas. He says we should look at software development as a collaborative game. And at open-source software development as carpet-wrestling. I had to look that up. Another thing I learned from Cockburn is that Agile causes Alzheimer’s. It’s somebody else’s theory, but Cockburn passed it along. Good to know.

Back to How Software Sucks

We have a software crisis today. We’ve always had a software crisis, because software is inherently crisis-prone. A tiny error in logic or typing and it crashes. It doesn’t just not work as well, it usually ceases to be any kind of solution at all. At least the way we’ve been building it.

So I was thinking, maybe we should be building software more like Alistair Cockburn.

Not like Alistair Cockburn says we should develop software so much as like Alistair Cockburn himself. Software that needs stimulation, hates stagnation, leans downhill. We’ve already got the “wreaks havoc on the unsuspecting” part. We just need our software to thrive on chaos. Then it won’t be so fragile. Maximum dynamic stability, like a skier leaning downhill.

We have always been in a situation where a small random change will wreck the system. Like alpha rays causing cancer. We need to be in a situation where a small random change will most likely make things better. Like cosmic rays causing the Fantastic Four. We need software that embraces randomness, error, chaos. If that means that it bursts into flame when annoyed, it’s a small price to pay if you also get strechiness, invisibility, and super strength. Still going with the Fantastic Four analogy because there’s no cooler cultural reference than a comic book that’s older than anyone you’re talking to.

Just Do It

I’m going to admit right now that this is one of those “implementation details left as an exercise for the reader” things. I have no idea how you’d accomplish this chaos-loving software.

Algorithms built to exploit simulated annealing, maybe. You know about simulated annealing? The technique of climbing hills by leaning downhill? Seems kind of Cockburnesque. A little random wrong-direction heuristic thrown in to keep your algorithm from getting too smug.

Anyway, somebody out there surely can figure out how to do it. Discover and implement solution surfaces where every path works but some work better than others. Instead of what we have now, which is fragile solution paths threading through vast fields of fail.

Well, now that I’ve explained the idea so clearly, I’m sure you can implement it. Go create some chaos-loving, reckless software. That leans downhill. Make Alistair Cockburn proud. Or nervous, I’d be OK with that, too.

John Shade was born under a cloud in Montreux, Switzerland, in 1962. Subsequent internment in a series of obscure institutions of ostensibly higher learning did nothing to brighten his outlook. He claims that his skill in not jumping out of airplanes is a thing of legend. Follow John on Twitter, send him your feedback, or discuss the article in the magazine forum.

Rear Window

A Parting Shot

What Ed Roberts, founder of MITS, did after creating the personal computer. Photo by Paul Freiberger.

What do you do after you’ve invented the personal computer and launched multiple industries? You go to medical school, become a doctor, and move to rural Georgia to work as a small-town general practitioner, of course.

Ed Roberts, with Altair 8800b and white coat, 1984.