PragPub 2014-02: Issue #56

Table of Contents

Features

A Closer Look At Clojure Sequences

by Michael Bevilacqua-Linn

Clojure has come a long way from its origins in Lisp, but it manages to address a rich collection of data structures with a common Lisplike interface.

Web-Based Intrusion Prevention with Repsheet

by Aaron Bedra

Your web app is under attack. What are you doing about it?

Core Data As Data Structure Service

by Joshua Smith

Core Data makes all the hard parts go away.

Lego League: Lessons Learned

by Seb Rose

Our series on teaching kids to code continues with lessons learned from coaching and judging a coding competition.

Departments

On Tap

by Michael Swaine

In which we introduce not only the issue before you but a special issue as well.

Choice Bits

A month recaptured in tweets.

Mac & Me

by Michael Swaine

Michael reflects on 30 years of the Mac.

Rothman and Lester

by Johanna Rothman and Andy Lester

Practical advice for managing your manager.

Pragmatic Bookstuff

Want to meet one of the Pragmatic Bookshelf authors face-to-face? Here’s where they’ll be in the coming months. Also, find out what are the top-selling Pragmatic Bookshelf books and what new books are coming out.

Antonio on Books

by Antonio Cangiano

All the new tech books of note.

Shady Illuminations

by John Shade

John shares a parable.

Rear Window

Happy 30th birthday, Mac!

Except where otherwise indicated, entire contents copyright © 2014 The Pragmatic Programmers.

You may not sell this magazine or its content, nor extract and use more than a paragraph of content in some other publication without our permission.

Published monthly in PDF, mobi, and epub formats by The Pragmatic Programmers, LLC, Dallas, TX, and Raleigh, NC. E-Mail webmaster@swaine.com. The editor is Michael Swaine (mailto:michael@pragprog.com). Visit us at http://pragprog.com

ISSN: 1948-3562

On Tap

A Couple of Issues

by Michael Swaine

Usually in this space I introduce the current issue of PragPub, and I’ll do that this month as well. But I have an extra introduction to make, to a special issue of PragPub.

We’re running a series of articles all this year on teaching kids to code. We think the topic is important and we want the articles to get the widest possible readership, so we decided to also publish them separately from the regular issues of the magazine in which they appear. So we are gathering them all up and publishing them as a special issue of PragPub totally dedicated to teaching kids to code.

And giving it away for free.

The only problem with this plan was that the special issue wouldn’t be complete until all the articles were published, a year from now. We wanted to get it out there right now, and have it promote the idea of teaching kids to code all year. We didn’t want to wait.

So we didn’t.

The special issue of PragPub on teaching kids to code is available right now, and will be updated with new articles all year long. You can download it via our website and you can sign up to get a ping when new content is added.

As for the issue before you, we have a nice variety of articles.

Michael Bevilacqua-Linn is winding down his popular series on the Clojure language. In what may be the penultimate article in the series, he delves into Clojure sequences. Rooted in the linked lists of ancient Lisp, they offer far more power and flexibility.

Aaron Bedra has been focusing a lot of his attention lately on how to battle intruders in web app space. He has both some sage advice and a nifty tool that can save you a lot of grief in dealing with the bad guys.

Joshua Smith checks in with an engaging article on Core Data. You won’t think about this Apple technology the same way after you’ve read it.

Plus: Choice Bits, Rothman and Lester on managing the managers, your editor on thirty years of Macintosh, Seb Rose on First Lego League, John Shade on dumb data, and more. I hope you enjoy the issue.

Cover, istockphoto.com/evilclown; p. 13, istockphoto.com/GreenStock; p. 16, istockphoto.com/Agustinc; p. 23, istockphoto.com/Calamusdesign; p. 27, istockphoto.com/stevecoleimages.

Choice Bits

Bursts from the Twitterhose

A month recaptured in tweets.

Now that the excitement of Superbowl Sunday is over we can all get back to waiting breathlessly for the next revision of Windows.

This Coding Life

I worry that by the time I die, I will have tried every wysiwig HTML editing library out there and eventually hated all of them. — @pamelafox

@pamelafox 1. WYSIWYG is a lie, has always been a lie, will always be a lie. 2. Emacs. — @timbray

Nothing is as relaxing as going into the final 2 hours before launch having pre-sold 50% of your target. time to play some mario, and wait! — @amyhoy

Real men use Android. Evidence is in from CES. — @GuyKawasaki

Vacation update: I just taught my three year old daughter how to play Robot Turtles. #trainthemyoung — @JEG2

Holy cow, www.webcrawler.com still exists. Who knew? — @petdance

I am pretending the 1000% + job recruitment on LinkedIn is NOT because I changed my profile name from “Susanne” to “S”. #womenintech — @girlveteran

On average, #adhoc teams are just as successful as #traditional teams. — @scottwambler

Any seemingly good set of practices may yield better results when used contextually instead of prescriptively. — @venkat_s

The Electricity-Generating Bicycle Desk That Would Power the World: http://snipe.ly/Le5bJP Can’t wait until this tech is practical. — @snipeyhead

The “inventor” of Twitter has checked out of the service. Is this a problem? — @mims

If Unicode can embrace the Wingdings Rude Boy Exclamation Point we should start lobbying for Clarus the Dogcow. — @cdespinosa

Bacon Is the New Cats

Cup of tea, cup of tea, beat myself about the head with “Good Math”, cup of tea. — @rit

McDonald’s says it will woo the French from their breakfast croissants by selling them its excellent coffee. — @mims

Coding beverage of choice has moved from Coke Zero to @breweryvivant Triomphe IPA for the rest of the afternoon. — @invalidname

Pouring Cheese on Icy Roads in (Where Else?) Wisconsin. http://nyti.ms/1laKwRm — @invalidname

Holding Up Half This Sky

11 Reasons 2014 Will Be A Breakout Year For Women Entrepreneurs; http://ow.ly/srBMh (via @forbes) #womenintech — @KVCJazz

Tech women are busy building their own networks. There are women in tech! http://wapo.st/1cPSKcCvia #WomenInTech — @washingtonpost

The Twitter list of women who code is up to 454 members! Follow them and improve your Twitter experience! — @jeresig

Setting up for the #CornellCEN #womenintech event tonight! Sold out crowd! Can’t wait! pic.twitter.com/IPS4NLbEDG — @CornellNetwork

Check out @CodeFirstGirls http://www.codefirstgirls.org.uk/about-us.htmlLaunching in Edinburgh & St Andrews campuses this semester. #coding #womenintech — @InformedScot

Line for the men’s room, no line for the ladies: http://4sq.com/1cH7Nsp — @SavIsSavvy

Through a Glass Snarkily

The Simpsons’ take on Google Glass is truly dystopian. — @mims

Rumor: Google Glass to be followed by Google Earbuds, Google Noseplugs, and Google Emotion-Simulating Facemask. — @mims

@mims I’m waiting for the Google Neck Brace. — @andrewmseaman

Google Goes Home

Google just bought high tech thermostat and fire alarm maker Nest for $3.2 billion in cash. — @HuffPostTech

Google’s Top 10 Acquisitions & where Nest falls on the list (hint: #2). — @dannysullivan

Pretty sure if Google or Apple wants smart homes to go mainstream they’re going to have to start selling homes. — @mims

Who Are Those Guys?

First, they’re not all guys. Second, we have to confess that we cleaned up the punctuation and stuff in the tweets. Here’s the list of folks we followed this month: Chris Adamson, Scott W. Ambler, Tim Bray, Cornell Network, Chris Espinosa, Pamela Fox, James Edward Gray II, Amy Hoy, HuffPost Tech, Informed Scotland, Kirsty Jarvis, Guy Kawasaki, Andy Lester, Brendan McAdams, Christopher Mims, Savannah Peterson, John Resig, Susanne Rossignol, Andrew M. Seaman, snipe, Venkat Subramaniam, and Danny Sullivan. Also, we met the Washington Post in a parking garage at midnight.

Fair’s fair. You can follow us at www.twitter.com/pragpub.

Mac & Me

Thirty Years Before the Mouse

by Michael Swaine

Like Captain Queeg, I can only cover these things from memory. If I’ve left anything out, why, just ask me specific questions and I’ll make something up.

Because a lot of my friends published reminscences on the occasion of the Macintosh’s 30th birthday in January, I decided that I should dig around in the closet and in my memory and put down some thoughts about the past thirty years and the Mac. Because you can easily find the general Mac history for yourself, I’m reflecting here on how the Mac intersected my own life. Because people are people, some part of it may strike a familiar note for you—or it may trigger you to think, “That’s not what was important in 1996, the important thing in 1996 was....”

And that would be fine, too.

Before the Mac

Apple was already a big deal in the personal computer industry before the Mac. Of course, it wasn’t much of an industry, not by today’s standards. But IBM’s entry into the arena in 1981 had legitimized the concept of a personal computer, and in the early ’80s Apple and IBM were the hardware companies to watch.

I was working at the newsweekly InfoWorld, researching and writing about the more technical topics. John Markoff and Paul Freiberger covered IBM and Apple, respectively. Paul and I got the idea to write a book about these crazy entrepreneurs who were changing the world, and we started doubling up on interviews. We’d talk to Gates or Jobs—or Lee Felsenstein or Bill Godbout or Adam Osborne or Captain Crunch—for InfoWorld and then we’d put on our author hats and ask them for the backstory and they’d let their hair down and tell us all the good stuff. For two young journalists, it was great fun.

Because Paul was covering Apple, he knew all about this new computer Apple was coming out with. Steve wasn’t happy about that at all. Of course we had to put it on the cover of our book.

1984

Apple launched the Macintosh in January of 1984 and everybody watching the Superbowl that year learned why 1984 wouldn’t be like 1984. I was pretty busy that winter. Our book came out. I began a romance that would last thirty years and counting. And I left InfoWorld and took over editing the programmers’ magazine Dr. Dobb’s Journal.

Dr. Dobb’s, which launched early in the personal computer era in 1976, was my favorite computer magazine by far, but it had was languishing, focused too much on CP/M and other fading technologies. I put a Mac on my desk as soon as I could and immediately started commissioning Mac articles for DDJ.

They weren’t all programming articles. The very first Mac article we did, by Tom Lafleur and Susan Raab, explained how to void your warranty and fatten your Mac with more memory. It involved a hacksaw as I recall.

I started writing code for the Mac and briefly became part of a Mac game software company with a couple of friends. But I quickly decided that, when it came to the Mac, I was more of a Saturday-morning hacker. The Mac was a challenge to develop for. It was said that the presrequisite for reading the developers’ documentation, Inside Macintosh, was a thorough mastery of all of Inside Macintosh.

HyperCard

Bill Atkinson’s HyperCard was an astonishingly good product. It created a machine within the machine that non-programmers could program, but included a scripting language that could be learned a little at a time, as needed. It not only made it possible for ordinary users to become power users, for enthusiasts to learn the elements of programming, and for developers to create useful utilities, all with great ease and charm. It made me rethink my no-programming decision.

I ended up writing a book about the scripting language, writing a HyperCard column for MacUser, and, with Nancy Groth, publishing a HyperCard magazine. And I wrote software, a bunch of it, even if the programs were called stacks and they were all utilities of use only to me.

System 7

I ended up writing four different Mac-related columns for MacUser and a couple for other mags. One column, called “Beating the System,” was inspired by Mac System 7. Was it the most significant operating system rev for Apple between the original Mac OS and X? I think so.

In retrospect the following decade seems both dull and grim for Apple. Jobs was elsewhere, and the kind of excitement that Apple had once generated had left with him, it would seem. But at the time those of us following Apple were having a ball. Literally. Nancy was managing editor of MacUser, and we found ourselves going to award shows that rivalled the Oscars. I had to wear a tux.

And then there was the Newton. Not a Mac, of course, but it did provide some excitement, as well as some good material for Gary Trudeau.

I went all in on Newton. I bought multiple models and plunked down my money for the developer’s kit. I recall that I spent many happy hours working on a wild-mushroom-identification program. It’s a good thing I never finished it, probably. I can just imagine the liability issues. The best thing I ever used my MessagePad for was to get Paula Poundstone’s autograph.

The Return of Steve Jobs

Things were really grim for Apple in 1996. The board had brought in Gil Amelio, a fixer, to save it or sell it. Unfortunately, the window of opportunity for selling had passed.

When Amelio announced that they were buying NeXT and would use its operating system to reboot the Mac, I was optimistic. Not so much because that operating system was so wonderful—there were other options on the table, including BeOS from Jean-Louis Gassée’s company. I was optimistic because for the first time in a decade or so, Apple was willing to see the need for radical action.

Of course, the journalist in me was fascinated by the drama of the situation. Which ratcheted up twenty notches when Amelio announced that Steve himself would be returning to the company. Radical.

Things were moving quickly now, and culminated in a matter of weeks in a Macworld Expo presentation simmering with drama that even an interminable speech by Amelio couldn’t squelch. When Steve Jobs took the stage—well, you hadda be there. I was.

Soon after, the coup was complete. Jobs installed his chosen lieutenants in software and hardware, Amelio was bumped aside, and a new board of directors took their chairs and handed unquestioned power to Steve.

Jobs acted decisively, killing off product lines, refocusing the company. But when you looked closely, you saw the operating system strategy changing by the week. When you kept your ears open, you heard that somebody heard Steve say he thought Apple was toast. When you looked at how he sold off stock, you could hardly think Apple’s problems were all behind them.

Long story short, OS X and Jobs’s decisions saved the company. And the new Macs. Looking back, the see-through shell and trippy colors seem a little silly. But somehow it was exactly the right thing.

Worked for us, anyway. Nancy and I had moved to Oregon, where she started a business. We outfitted the entire office with a fleet of blueberry iMacs and one blueberry iBook. I still have the iBook. Most of it.

Today

Today the universe is moving away from personal computers. My phones and my tablet can do pretty much anything I formerly needed a computer for. But I haven’t given up on Macs. Nancy and I both work at home now. Or just as often not at home. But wherever we are, you’ll find us sitting across from one another, typing away on identical MacBook Pros. We’re Mac people.

Michael Swaine is the editor of PragPub.

Rothman and Lester

How to Manage Your Boss

by Johanna Rothman and Andy Lester

You say your boss is driving you crazy with questions and micromanagement? Try these insights for getting the boss off your back.

Johanna: Have you ever been in a situation where you thought your boss was micromanaging you? Or you thought you told your project manager where you were in the project, but you still felt as if you weren’t trusted? That’s the topic of this month’s column.

Let’s begin by turning this around. Instead of being the victim, try asking, “What would it take for my manager or project manager or boss to feel happy?” Ask yourself how you can take charge of the situation.

Andy: It sounds overly simple, but in the times where I’ve had my boss on my back about something, it’s because he wasn’t getting something from me that he needed. Sometimes that may have been a certain deliverable in a certain way, but often it seemed to be an issue of the boss feeling uncertain or wary about how things were going with me.

So for me, if I feel like something isn’t going right and I’m feeling nagged, the first thing I do is to tell myself not to take it personally.

Johanna: Yes. It sure feels personal. But the real problem is, your boss isn’t getting something. So let’s put on our detective hats, and consider some alternatives. There are some common problems that make bosses get nervous and start to feel uncertain.

One common problem is when you take longer to deliver something than you expected. It’s one thing if your original estimate was off. I certainly don’t estimate all that well. Our industry is built on bad estimation. But the more insidious problem that arises with estimates that are off comes when you don’t tell your boss where you are, and what you will do about it.

Andy: Right, bosses and project managers can get very concerned when deadlines slip, or even when they fear that they might slip. So if you’re feeling that you’re getting watched or “micromanaged” (which is a word I think is often used inappropriately), then be proactive and talk to your boss or project manager about it.

You might start with “Hey, Susan, here are those revised estimates you asked for. I’m getting a feeling that you’re concerned about the timeline of the project (or the number of bugs in the queue or whatever). Is there anything I can do to help alleviate those concerns?”

You’re not accusing Susan. You’re not saying “Get off my back.” In fact, quite the opposite. You’re recognizing the situation and asking to get involved. This helps the project, yes, AND it’s also self-serving because ultimately it will get Susan off your back.

Johanna: You’re recognizing the risk and acting appropriately. Not acting like an ostrich is key.

Andy: And you’re ASKING. You’re not assuming you know what’s what.

Susan may come back and say “Well, you know we missed the phase one deadline by a week, and so I’m concerned about the rest of the schedule slipping,” but she might also come back with “I’m sorry to be a pest, but you know this is high profile, and Dave the VP is taking a specific interest in the project. He’s asking for status updates every day.”

The same behavior can have two very different causes, and that means two different ways you can help. Maybe the answer in the second case is “If Dave is pushing for status reports, would it help if I just mailed you updates every day before I go home? Then you wouldn’t have to ask me for them. Would that help?”

Johanna: Or, it could be that the project dashboard doesn’t have exactly the right data on it. You could ask, “What data would be the most helpful to you?” See Manage It! for ideas.

Or, you could suggest, “It looks as if you want this trend or that trend. How about if I generate a daily report and post it here on the wall and email it to you every day at 4 p.m.?” That might be helpful too.

Andy: The key here is that Susan is not your enemy. Susan may be annoying, and Susan may not do things the way you like them, but she’s still on your team, and you both ultimately want the projects and team to succeed.

The other key is to seek first to understand. It’s easy when you’re asked “Hey how’s the project going?” three times a day to think “Gaaaah! Just leave me alone and let me work!” but that’s just thinking from your point of view.

In fact, look at it from a scientific point of view. Why is Susan asking you three times a day how the project is going? She’s not doing it because she’s bored. She’s doing it because some need on the project is not getting met. So approach it from that point of view.

Johanna: That’s a project example. Let me give you a personal example, too.

I was once a developer fixing a problem for a release. We had many problems in that release. I was, of course, the last one done. As soon as I was done, the release could go. I was under a lot of pressure to finish. My boss was cool, but there must have been 50 people (okay, not 50, but it felt like it), coming to my office every hour, asking, “When will you be done?”

I finally blew up at somebody. I’m sure I yelled, “I’ll be done a lot faster if no one asks me when I’ll be done for at least 15 minutes!”

I finally realized they needed status updates. I decided if I posted a status update on my door every two hours, that would work. So that’s what I did. It was a total pain in the tush, but it gave me breathing room, and I finally finished my fixing in two days.

I actually made copies of my engineering notebook, too, so they could see my debugging process. Only the other engineers thought that was interesting, but it was the transparency that really helped.

I’m not sure a status update every two hours is the right thing. And I think having someone to run interference for a technical person is a better idea. But the other people really needed to know what was going on. We had significant revenue riding on this release.

That’s why they all needed to know.

Andy: And that’s the core of the problem. If something is making noise, it indicates a problem. Same thing with people making noise.

Johanna: Listen to the noise. Be transparent with your status and your solution.

Andy: One of the frustrations I have is hearing people talk about “micromanaging.” Usually I hear that term thrown around in response to the boss who wants status updates all the time. But sometimes micromanaging is used in the more correct sense, when your boss will not let you make decisions.

Johanna: Sometimes it’s about how to do the work, too.

Andy: And that’s a trickier area to handle.

Johanna: Yes, because then it’s about trust.

Andy: But the core answer is still probably the same: Figure out what is missing that is driving the behavior, and fix it.

For instance, a friend of mine (I’ll call him Steve) had a boss who he found would always stick his fingers in Steve’s code. Thing is, once Steve got past the annoyance and feeling attacked, he was able to suss out what the boss wanted. The boss was mostly into how things were named, and worrying about decisions that he saw affecting performance.

So, rather than fighting it, Steve went the other way. When he’d come up with table schemas for a new project, and he knew his boss would put his fingers in it, he’d proactively go to boss and say “Here are the table layouts. Anything you’d change?” Same thing with his high-level architecture. He’d sketch out ideas and then say “Anything you’d change?”

Now, I know a lot of my geek brethren will say “But Steve shouldn’t have to do that! His boss should trust his decisions!” And there’s validity to that. However, the reality of this particular situation was that Steve figured out that it was easier and saved more time to head off the meddling at the beginning.

Your mileage may vary. You may feel like it’s an onerous burden and that you shouldn’t have to do that and you can’t live like that. And if that’s the case, you’ll probably want to look for another job, because with things like that, where your boss has certain pet topics that he has to have dominion over, you’re never going to change it.

Johanna: See, if I had been in Steve’s position, I would have asked my boss why he did this the first time. I would have asked him to coach me. And the second time he did it without coaching me? I’d have had my résumé on the street. I don’t have the patience for that. But if the boss had coached me? I’d have been all over that.

Andy: Go on.

Johanna: I think it’s pretty smart of Steve to have figured this out.

Andy: I see. I thought you were saying that your way is a preferred way.

Johanna: No. It’s just an example of what one can handle as a person.

There are plenty of times I want the coaching. There are tons of things I don’t know. But if I sign up for the coaching, and then I don’t get it, I feel cheated.

Andy: Agreed, that sort of guidance is good. On the other hand, sometimes there are things that are just the way the boss wants it, and it’s useless to fight it. And saying “It’s useless to fight it” isn’t defeatist. It’s just saying that there’s such a thing as diminishing returns, especially when in many of these cases it boils down to “I’m right, he’s wrong” and it’s your ego at stake.

Johanna: One reframe I have found useful is to consider the “helpful" rule. No matter what it looks like, the other person is trying to be helpful. The other person might not be trying to be helpful to you. The person might be trying to be helpful to his or her boss, which is why it might look like micromanagement to you. But the other person probably is trying to be helpful. Understanding that can help you deal with, and change, the behavior.

Your job is to figure out what the source of the noise is, as Andy said before. Listen, and figure out what you have to do to silence that noise.

Next month, we’ll discuss how to transition to a new role and leave your old role behind, whether it’s a new position in the same company, or you’re making a jump to a new company.

Johanna Rothman helps leaders solve problems and seize opportunities. She consults, speaks, and writes on managing high-technology product development. She enables managers, teams, and organizations to become more effective by applying her pragmatic approaches to the issues of project management, risk management, and people management. She writes the Pragmatic Manager email newsletter and two blogs on www.jrothman.com.

Andy Lester has developed software for more than twenty years in the business world and on the Web in the open source community. Years of sifting through résumés, interviewing unprepared candidates, and even some unwise career choices of his own spurred him to write his nontraditional book on the new guidelines for tech job hunting. Andy is an active member of the open source community, and lives in the Chicago area. He blogs at petdance.com, and can be reached by email at andy@petdance.com.

Send the authors your feedback or discuss the article in the magazine forum.

A Closer Look At Clojure Sequences

With A Dash Of Light Lisp History And A Side Of Laziness

by Michael Bevilacqua-Linn

Clojure has come a long way from its origins in Lisp, but it manages to address a rich collection of data structures with a common Lisplike interface.

Welcome back once more to this series on the Clojure language.

This time, we’ll examine Clojure’s sequences and sequence library in greater detail. We’ll start by taking a look at the history that inspired Clojure’s sequence abstraction. In the past, the core abstraction of many Lisps has been the humble, singly linked list. The name Lisp, after all, started out in all caps as LISP—a rather strangely constructed acronym for LISt Processing language.

Below is a representation of a singly linked list containing the integers 1, 2, 3, and 4:

Since the list is singly linked and only a reference to the head of the list is maintained, the number of operations that can be performed efficiently on it are limited. We can add an item to the head of the list in constant time, but to add an item to the tail of the list would require time proportional to the number of items in it, since we’d need to traverse the whole thing, starting from the head.

Similarly, we can retrieve the first item in the list in constant time. We can also retrieve a pointer to the rest of the list, that is, everything other than the first item, in constant time. Lisp originally relied on these three operations, adding to the head of a list, a function known as cons, retrieving the first item in a list, a function known as first, and retrieving the rest of the items in the list, or the rest function.

For obscure historical reasons involving the 1954 IBM 704, on which the original Lisp was implemented, first was known as car and rest as cdr in many previous Lisps. Thankfully, Clojure has dispensed with this bit of historical fetishism.

Way back in the 50s, when Lisp was originally invented, singly linked lists were quite sophisticated data structures. Over the years, fancier and more capable data structures were invented, but the core data structure of Lisp remained the singly linked list.

In Clojure, there are several other built-in data structures that are equally important: vectors, sets, and maps. Clojure lets us manipulate these data structures using a common sequence interface. To do so, Clojure pulls apart the concrete implementation of a singly linked list from classic Lisp, and the interface that it presents: cons, first, and rest. It does this using the sequence interface, ISeq.

The fancier data structures can all be turned into sequences and manipulated using cons, first, and rest. Let’s take a look at a few examples. Here, we define a list with three elements:

(def a-list (list :foo :bar :baz))

The first and rest functions work as advertised.

| => (first a-list)
| :foo
| => (rest a-list)
| (:bar :baz)

The cons function take an existing sequence and prepends an item to it, returning a new sequence. The existing sequence is not modified. For more information on Clojure’s immutable data structures themselves, check out “Concurrent Programming In Clojure” in the September issue.

| => (cons a-list :quz)
| (:quz :foo :bar :baz)
| => a-list
| (:foo :bar :baz)

We can manipulate other Clojure data structures using the same functions. For example, here we create a vector and manipulate it using the sequence functions.

| => (def a-vector [:foo :bar :baz])
| #'Sequences/a-vector
| => (first a-vector)
| :foo
| => (rest a-vector)
| (:bar :baz)
| => (cons :quz a-vector)
| (:quz :foo :bar :baz)

Here we pull the first key value pair out of a map using first.

| => (def a-map {:foo :bar :baz :quz})
| #'Sequences/a-map
| => (first a-map)
| [:foo :bar]

When we get the rest of a map or vector, Clojure automatically converts the map or vector into a type that implements the ISeq interface. To convert back into the original type, we can use the into function, which pours a sequence into some data structure. Here, we convert the rest of our map and vector, which is now just a sequence, back into a map and vector.

| => (into [] (rest a-vector))
| [:bar :baz]
| => (into {} (rest a-map))
| {:baz :quz}

The upshot of this is that Clojure’s powerful sequence library can be used across a range of built-in data types, which is very convenient. As a simple example, here we use filter on a list, vector, and map to obtain only even numbers. We pour the latter two back into their original data structures.

| => (filter even? (list 1 2 3 4))
| (2 4)
| => (into [] (filter even? [1 2 3 4]))
| [2 4]
| => (into {}
| (filter (fn [[key value]] (even? value))
| {:foo 1 :bar 2 :baz 3 :quz 4}))
| {:bar 2, :quz 4}

Clojure has dozens of functions in its sequence libraries, all of which can be used across Clojure’s built-in data structures. For a full list, take a look at clojure.org/sequences.

Let’s finish up with a bit of a closer look at what happens when we get the rest of a Clojure data structure. Here, we take a look at the type of the item returned when we get the rest of our vector.

| => (type (rest a-vector))
| clojure.lang.PersistentVector$ChunkedSeq

As we can see, it’s the somewhat odd looking

clojure.lang.PersistentVector$ChunkedSeq.

What is a ChunkedSeq? (And does it love Sloth?)

The answer is that sequences in Clojure are evaluated lazily. Lazy evaluation of a sequence means that an item in a sequence is only created when it’s needed, and not before. For instance, in the following code snippet we use the range function to create an infinitely long sequence, and then take the first ten items from it.

| => (take 10 (range))
| (0 1 2 3 4 5 6 7 8 9)

There’s some overhead to creating an item in the sequence, so for the sake of efficiency, Clojure will only produce sequence items in chunks of 32. So with our range example above, the range function would create the first 32 integers, and take would take the first ten. This means that only one chunk would end up getting created. If we asked for 33 integers, two chunks would be created, and so on. The clojure.lang.PersistentVector$ChunkedSeq we saw above acts similarly, producing chunks of 32 items at a time out of the source vector.

Thanks once more for reading! Join us again next time as we wrap up this series with a deeper dive into Clojure’s macro system.

Michael Bevilacqua-Linn has been programming computers ever since he dragged an Apple IIGS that his parents got for opening a bank account into his fifth grade class to explain loops and variables to a bunch of pre-teenagers. He currently works for Comcast, where he builds distributed systems that power infrastructure for their next generation services, and wrote Functional Programming Patterns. He tweets occasionally at @NovusTiro.

Web-Based Intrusion Prevention with Repsheet

Dealing with the Bad Guys

by Aaron Bedra

Your web app is under attack. What are you doing about it?

These days, hooking a web application up to the fire hose that is the internet can be a dangerous thing to do. Within minutes robots good and bad will begin to explore the depths of your site. Just like the monitoring you put in place to keep track of the health and performance of your application, it is good practice to put in place monitors that tell you about who is acting on your site, and if they are acting in a malicious manner. The goal is to stop bad actors in progress or potentially before they even attempt anything malicious.

You Are Being Attacked

This shouldn’t come as a surprise. If you are surprised get ready for a lot more surprises! All day there are bot-nets aplenty scouring the internet trying every trick they have to exploit vulnerabilities on web applications. At first they were only marginally effective. Like every technology, though, they have begun to get more and more advanced and have become a real threat. A quick look through your logs will almost certainly show signs of their activity.

Different Types of Web Attacks

Attack and abuse of web applications comes in a variety of forms. There are standard attacks designed to exploit known vulnerabilities like Injection, Cross-Site Scripting (XSS), or one of these other top web vulnerabilities. Attacks targeting these vulnerabilities are usually pretty straightforward and are present on an astounding number of applications. These vulnerabilities are normally easily fixed once found.

The other class of attacks are centered around abuse of the application. This comes in the form of design flaws with the way users communicate with the application or even the details of particular common actions like credit card verification. These attacks are much more focused and are usually much harder to isolate while they are in progress since they are specific to your application and traditionally don’t fit any well-known patterns.

Analyzing an attack

Spotting attacks against the common known web application vulnerabilities isn’t trivial, but there are many projects attempting to solve this problem. The ModSecurity Web Application Firewall project is an open source example of this, offering a nice set of tools to get started detecting and preventing attacks. While it is a good start, the high rate of false positives makes it hard to trust. Because it has the potential to block traffic, this combination can lead to blocking legitimate traffic. This is why most prefer to run ModSecurity in passive mode. This simply records the presence of triggered rules and allows for an additional chain of controls that eventually determine if an actor should be stopped.

To dive a bit deeper, let’s make the discussion more concrete by looking at a few requests from an attack in progress. These are the first four requests made by this actor:

| ```
| 10.20.253.8 - - [23/Apr/2013:14:20:21 +0000]
| "POST /login HTTP/1.1" 200 267"-"
| "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0)
| Gecko/20100101 Firefox/8.0"
| "77.77.165.233"
| ```
| ```
| 10.20.253.8 - - [23/Apr/2013:14:20:22 +0000]
| "POST /users/king-roland/credit_cards
| HTTP/1.1" 302 2085 "-" "Mozilla/5.0
| (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101
| Firefox/8.0" "77.77.165.233"
| ```
| ```
| 10.20.253.8 - - [23/Apr/2013:14:20:23 +0000]
| "POST /users/king-roland/credit_cards
| HTTP/1.1" 302 2083 "-" "Mozilla/5.0
| (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101
| Firefox/8.0" "77.77.165.233"
| ```
| ```
| 10.20.253.8 - - [23/Apr/2013:14:20:24 +0000]
| "POST /users/king-roland/credit_car’ds
| HTTP/1.1" 302 2085 "-" "Mozilla/5.0
| (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101
| Firefox/8.0" "77.77.165.233"
| ```

If you take a look at these requests you will notice quite a few things that don’t add up. Yet these requests on their own don’t set off any standard web application firewall rules or raise alarms in any other significant way. Let’s highlight all of the strange behavior here and try to derive the intent of the actor.

The first request was a POST, not a GET request to the login form.

The second request was again a POST without a GET.

The credit_cards POST requests returned a 302 which was not followed.

The requests were all 1 second apart.

The user agent doesn’t make sense (Windows 7 running Firefox 8.0).

The location of the actor is Plovdiv Bulgaria.

From looking at all of the information in these requests we can determine with pretty decent accuracy that the actor probably isn’t legitimate. More likely, the actor is actually a script.

We haven’t reached the point of the exercise though, and that is to figure out the intent of the script.

Thinking more about the URI of the last three POST requests, we see that it points to a resource labeled credit_cards. If we were to dig deeper we would realize that this end point is responsible for adding or changing credit card information for a user. This type of activity done in rapid succession is an indicator of something called a Carding Attack. It is a method used to verify stolen credit card information. While it might not seem immediately relevant, it is important to remember that if your site takes credit cards these carding attacks take up resources and can potentially cost you money in transaction fees as well as charge-backs if the stolen cards are used to purchase items from your site.

And this is just a simple example of the types of things that a web application firewall doesn’t protect against.

Acting on an Attacker

Once you identify that an actor has malicious intent, you now have a decision to make. What do you do with the actor?

Do you block them?

At the firewall?

At the web server?

At the application?

Somewhere upstream (Akamai, CloudFlare, etc)?

There are a lot of options here, and blocking isn’t always the answer. Of course, blocking is easy and makes sense for a lot of situations, but once you block an actor you are tipping your hand to the fact that you have caught them and that you want them to go away. The problem with blocking is that most attackers have more IP addresses than you have time, and playing whack-a-mole isn’t really in your job description. There are other options for dealing with these types of attacks once they are identified:

Disconnect the backend (allow the actor to continue the attack, just don’t actually do anything).

Send them to a Honeypot and study the attack in greater detail.

Send back nonstandard responses in an attempt to confuse or crash the script.

Call the authorities to help you deal with the attack.

These may be better options. Specifically when dealing with carding attacks, it is sometimes a good idea not to block the actors if you can handle the extra load on your servers. It will let the scripts think that they are successful and they will go on gathering potentially false data and not actually achieving their nefarious goal. This is a lot to think about, though, and spending the additional time to implement these controls can be costly. In order to handle the whack-a-mole situation, you need an automated actor of your own.

Dealing with State across Web Servers

It is highly likely that you have more than one web server. If you don’t now and your application continues to grow, you will eventually get there. Since each web server only has knowledge of what requests were made directly against it, you have to find a way to share threat information across all of your servers. Each server has to know which actors to consider a threat, and that information needs to be decided from the information gathered across all of your servers. You could, of course, do aggregated log analysis, but that doesn’t leave you with a way to tell the server that the actor is misbehaving.

This problem isn’t exactly an easy one to solve. You have to ask a question about the actor every time they make a request. You can store it in the web server’s allocated memory, but if the information grows too large, you could have issues. You don’t want to tie up the request making expensive calls to a database either. A global cache seems ideal in this situation. You just have to be aware that the caching system you use needs to be able to support your request volume to prevent performance issues.

Luckily, Repsheet provides a mechanism for dealing with the global state across web servers.

Introducing Repsheet

Repsheet is a reputation engine. It consists of a web server module, a backend component, and a visualizer. We will go through each of the pieces in more detail, but here is a high-level diagram of a system that uses Repsheet:

The Repsheet module uses the cache to look up an actor when a request is made. If that actor has been blacklisted, the web server will return a 403 and the request will be denied. If the actor checks out, the Repsheet module will record some details about the request including any ModSecurity findings. The Repsheet Backend will continuously look at the data in the cache to see if ModSecurity thresholds have been exceeded. If so, it will automatically blacklist the actor. The Backend will also ask for a risk score from the OFDEP fraud detection API. This score is an indicator of the reputation of the IP address across the internet. If this score exceeds the threshold, it will automatically blacklist the IP. We will talk more about the visualizer shortly. First we will look at how to use ModRepsheet and Repsheet Backend.

Mod Repsheet

The web server module is the core of Repsheet. The main function is lookup and act.This looks for the IP address in the cache to see if the actor has been flagged. During this evaluation ModRepsheet looks at the results from ModSecurity and extracts the rules that were triggered. It keeps track of how many times an IP triggers a particular rule so that it can be further evaluated by the Backend. An actor can be flagged in two ways:

The actor can be placed on the Repsheet. In this case the actor has triggered a ModSecurity rule. It doesn’t necessarily mean that the actor is bad, it just raises some suspicion. In this case ModRepsheet will add a header to the downstream application letting it know that the actor has been flagged. This header is X-Repsheet: true. This way the downstream application can potentially modify its response in order to prevent abuse or attack.

An actor can also be placed on the blacklist. If this occurs, all requests from that actor will be denied with a 403 response. This can be done automatically via the Backend, or manually via the Visualizer.

The module also records specific details about the request. This helps to manually review what the actor is doing and potentially make a decision to blacklist if other indicators don’t trigger the automatic blacklisting.

At the moment, this module only works with Apache, but an nginx port is in development. You can read the detailed setup instructions on the Repsheet website. There are packages for Debian- and RedHat-based systems, as well as the ability to compile and install the module manually.

Repsheet Backend

The Repsheet Backend is the starting point for automation. It provides the ability to automatically blacklist actors. It looks at the total number of ModSecurity rules triggered, as well as reaching out to OFDEP to determine a risk score for the actor. If either of the two is above the provided threshold, the actor is blacklisted. The Backend also keeps track of any actors that have been blacklisted. If an actor returns and triggers any ModSecurity rules, they are immediately blacklisted again. The Backend also helps organize the data for presentation via the Visualizer.

The Backend is designed to run via a cron task. Most setups run the Backend every minute. You can read the detailed setup instructions on the Repsheet website. As with the webserver module, there are packages for Debian- and RedHat-based systems, as well as the ability to compile and install the module manually.

Repsheet Visualizer

The Visualizer works as advertised. It provides details on actors that are currently suspect (triggered ModSecurity or OFDEP scores), actors that have been blacklisted, and actors that have been whitelisted. The Visualizer also provides a way to manually whitelist or blacklist actors upon inspection of their behavior. The dashboard view shows a high-level view of the current landscape:

For any actor, you can dive into their profile and get a better understanding of their activity:

This actor has already been blacklisted. If you took a look at the request information you would find that the actor was attempting to run a vulnerability analysis to determine if the server was able to be exploited. For this particular pattern, the intent is to turn the server into a bitcoin mining agent.

Taking It to the Next Level

One of the design goals of Repsheet is to keep the cache accessible and open. This means that you can use the Repsheet framework with any of your data. Let’s say you have one of the popular log aggregation systems set up in your environment (Splunk, LogStash, SumoLogic, etc.) and you already have queries that look for interesting behavior. You can just plumb the data from your engine of choice into the Repsheet cache and let it do the heavy lifting. Basically, if you can dream up a query or behavioral analysis for your application, you can tell Repsheet to deal with the bad actors on your behalf.

Blocking bad actors is really only the start. Modifying your controls based on how an actor behaves can be very compelling. You get to reward good behavior by reducing controls, and increasingly thwart bad behavior by increasing them. The goal is that your users shouldn’t have to suffer static controls because you don’t know who is good and who isn’t.

Wrap Up

This has been a sort of whirlwind tour of some of the ideas around defending web applications and integrating Repsheet into your system. This is just the beginning. There are a lot more ideas to explore, and more ways to increase your web application defense. If you want to read more about this subject, Web Application Defender’s Cookbook is a great place to start. But it is important to remember that none of these ideas is a substitute for good defensive programming and security-focused review of applications. The ideas here are meant to enhance an already secure application.

Aaron is a Principal Consultant at Cigital where he helps drive secure programming practices and better understanding of software security. Aaron is the creator of Repsheet, an open source framework for web application attack prevention. He is a co-author of Programming Clojure.

Send the author your feedback or discuss the article in the magazine forum.

Core Data As Data Structure Service

Persistence Conquers All

by Joshua Smith

Core Data makes all the hard parts go away.

On the first day at your new job your boss walks into your office and says the project needs a priority queue. She tells you it shouldn’t take too long to write and then leaves you to get it done.

Just to make sure you understand fully what she asked for, you google Objective-C and Priority Queue. The first thing you find out is that there isn’t a generic priority queue in Objective-C. Then you learn that a priority queue is an abstract data type like a regular queue or a stack, but each element has a priority and the order of the elements in a priority queue is determined by this priority (hence the name). Additionally, Wikipedia notes that a priority queue can be implemented using a list.

Now confident that you understand the problem, you start typing furiously, because we all know there is no problem that cannot be solved by typing faster.

What you come up with is a decent, though naive, priority queue. You have chosen to use a mutable list that you sort every time you add an element:

| @interface NaivePrioQueue : NSEnumerator
| - (instancetype) initWithComparator:(NSComparator) cmp;
| - (NSUInteger) count;
| - (id) pop;
| - (id) front;
| - (void) push:(id) obj;
| @end

It’s not a lot of code, and it works. The code is testable and clean. So you show it to your boss and she says it’s okay, but maybe it would be better to implement it using a CFBinaryHeap. You had no idea such a thing existed. A quick glance at NSHipster reveals that it does exists and that you can implement a priority queue in it. So you do.

The first thing you set up is a protocol for objects you add to this priority queue to manage the comparators:

| @protocol RCWPrioCanCompare <NSObject>
| - (CFComparisonResult) compareWith:(id<RCWPrioCanCompare>) otherObj;
| - (NSString *) comparisonValue;
| @end

Then the interface looks like this:

| @interface RCWPrioQueue : NSEnumerator
| - (NSUInteger) count;
| - (void) push:(id<RCWPrioCanCompare>) obj;
| - (id) pop;
| - (id) front;
| @end

This version is pretty awesome, you think. It’s faster and cleaner than the first version. You proudly show it to the tech lead and she sighs. You know this is a bad sign.

The problem is, she tells you, the project requires that the queue be persistent. The queue has to be stored in a such a way that it can recover if the phone is rebooted, the app is killed, or something else. You stop and check your shirt to see if it’s red because this seems impossible. Serializing can be done with the array, sure, but only if it’s smaller than memory. Serializing a CFBinaryHeap sounds pretty hard. So what to do?

The tech lead nods her head toward an office you know you are not supposed to approach. Opposite the door hangs a partially disassembled corpse of something called a MicroVAX in the shape of a voodoo skull. But you shudder and approach Sharon’s office. Her gray hair and insistent use of something called Emacs has made her the terror of the intern pool.

You knock gently. She turns and stares as you as you stammer out the requirements. She leans back for a minute, obviously thinking, and then says, “Use Core Data. A UIManagedDocument subclass can encapsulate it all for you.” And then she turns back to her terminal and you are dismissed.

You go back to you desk and start typing again, this time entering a new domain of Cocoa Development and wondering just how far down this rabbit hole can go.

You can use the @import syntax to pull Core Data into your project with one line:

@import CoreData;

Then the interface looks like this:

| @interface PrioQueueProject : UIManagedDocument
| + (instancetype) queue;
| + (NSURL *) fileURL;
| - (NSUInteger) count;
| - (id) front;
| - (id) pop;
| - (void) push:(id<NSCoding>) obj withPriority:(NSUInteger) prio;
| @end

A big change is the fileURL and queue class methods. The queue class is an alloc/init combo similar to the array method on NSArray. The fileURL is how you can change the location of the persistent store in subclasses. Because this is a base class and we’re not assuming that a model exists, the class creates a model in code. The model is pretty simple. It includes a single entity that has two attributes. The priority attribute is used for ordering, and the target attribute is a serialized version (via NSCoding) of the object in the queue.

| - (NSManagedObjectModel *) managedObjectModel {
| if (self.privateModel == nil) {
| NSManagedObjectModel *model =
| [[NSManagedObjectModel alloc] init];
| NSEntityDescription *entity =
| [[NSEntityDescription alloc] init];
| entity.name = NSStringFromClass([self class]);
| NSAttributeDescription *prio_attr =
| [[NSAttributeDescription alloc] init];
| prio_attr.name = PQP_PRIO_KEY;
| prio_attr.attributeType = NSInteger64AttributeType;
| NSAttributeDescription *target_attr =
| [[NSAttributeDescription alloc] init];
| target_attr.name = PQP_OBJECT_KEY;
| target_attr.attributeType =
| NSTransformableAttributeType;
| entity.properties = @[ prio_attr, target_attr];
| [model setEntities:@[entity]];
| self.privateModel = model;
| }
| return self.privateModel;
| }

All of the hard work happens in using the fetch request.

| - (void) setupFetchRequest {
| self.fetch =
| [NSFetchRequest
| fetchRequestWithEntityName:NSStringFromClass([self class])];
| NSSortDescriptor *sortDesc =
| [NSSortDescriptor sortDescriptorWithKey:PQP_PRIO_KEY
| ascending:NO];
| self.fetch.fetchLimit = 1;
| self.fetch.sortDescriptors = @[ sortDesc ];
| }

Because we only care about the first element, we can set the fetchLimit to 1. This fetch request is fired every time an element is added to or removed from the queue, and the result of the fetch request is cached for use in the front method. The power of Core Data as a data structure service saves the day.

Epilogue

The thing to remember is that Core Data isn’t just some fancy-pants ORM. It is really a data-structure service that provides the ability to persist when you need it. You don’t have to persist anything, and if you use the in-memory store, it won’t. You don’t need an existing model definition file to work with since you can just create the NSManagedObjectModel in code. You don’t even have to manage the Core Data Stack because the UIManagedDocument class does that for you.

The full code can be found on github.

Josh Smith is a writer and consultant with Rubber City Wizards. They are a small consultancy focused on innovative mobile systems. He has 18 years of professional IT experience and has been a sysadmin (and still kinda is), programmer in various languages, consultant, employee, and guy who makes sure the pop machine is full. He lives and works in Pittsburgh, PA and tries to use his powers for good.

Send the author your feedback or discuss the article in the magazine forum.

Lego League: Lessons Learned

Teaching Kids to Code

by Seb Rose

Choose the right sort of problem, focus on strategies for solving the problem, and the code will come easily.

Some approaches to teaching kids to code work better than others.

I want to tell you a story. Two stories, actually.

A few years ago I found myself in one of those huge, soulless, air-conditioned conference halls, listening to one of those keynote speeches on a subject that had no discernible relevance to my work. But despite my preconceptions, I found myself surprisingly moved by the presentation. The speaker was inventor Dean Kamen, and he was talking about something called First Lego League.

I was inspired. Returning to the UK, I chased down our local FLL and registered. I coerced my children, aged ten and twelve, into participating. I put notices around the local junior schools, gathering another eight local offspring. I booked a room in a local hall one night a week for the rest of the year, bought the lumber needed to build the competition table, and waited for the kit to arrive.

The kit, right. I see I’m getting ahead of myself. I’d better back up and explain a bit about the FLL.

The First Lego League

The First Lego League is an annual, international competition for teams of 9-to 16-year-olds that involves programming Lego MindStorms to solve challenges, doing a piece of original research, and presenting the results of that research. You can read more about it at the website. It’s interesting reading, really.

Anyway, about the kit. Each year FLL chooses a theme of global significance (in 2010 it was Biomedical Engineering) and devises a number of robotic challenges around it. And they send you this kit to get you started. The competition board is 8' x 4' and they provide a roll-out map to use as the base for your constructions, as well as a box of Lego bits and pages of instructions about how to put together the Lego challenges. Just building all of the challenges took our group 2 whole sessions.

Once the challenges are assembled and laid out on the board, it’s time to start designing and programming your robot. The robot construction rules are strict—only Lego-manufactured material allowed, no more than 4 motors, no more than 1 of each sensor, no non-standard software environments—and you only get 2-1/2 minutes to complete as many challenges as you can. Whenever the robot returns to base, a small area at one corner of the board, you’re allowed to touch it, modify its attachments, start a different program. At all other times it must be completely autonomous (NOT remote controlled) and any interference with it leads to penalty points.

So there I was, with ten 9- to 12-year-olds with no engineering or programming experience. Mercifully, the role of coach in FLL, they explain, is not to teach, but to enable. Good. So I printed out the challenges, loaded the Lego programming environment onto some school laptops, and asked them to pick a few challenges to have a go at. There were 17 to choose from, and no way to accomplish all of them in the short time available.

There was teaching involved, but once they had selected the challenges, it was pretty straightforward. We started with the engineering, because that was a more tractable problem, and you need to have an idea of the robot’s physical characteristics before trying to control it with software. Then came the challenge of introducing them to software. The Lego programming environment is fully graphical—you drag action, sensor, conditional, and branching “blocks” around, linking them together to achieve the required behavior. We started with simple, dead-reckoning approaches—“turn both wheels for 8 revolutions at 5 revolutions per minute,” “turn left 90 degrees,” “lift arm 45 degrees”—and just that is enough to get a lot of the challenges done.

What the team noticed, though, was that this style of control wasn’t very robust. It was susceptible to small variations in the robot’s initial position. The state of the batteries affected the distance the robot travelled. Even with identical starting conditions, the end position of the robot could differ significantly. This is when I introduced them to the sensors.

The most useful sensor was the light sensor. They used this to detect color changes on the map and trigger a change in behavior of the robot. Different groups of children would work on different parts of the robot’s route, and these would then be joined together by “blocks” switched by the sensors. This was particularly effective for the “pill dispenser” challenge, where the robot needed to push a panel to dispense 2 Lego “pills,” but leave the third “pill” in the dispenser. There were handy black markings on the map that could be used to count how far the robot had traveled, and hence how many “pills” had been dispensed.

Another useful sensor was the push button, which was useful to ensure the robot kept the right distance from obstacles. We never found a use for the ultrasonic sensor—it was just too unreliable. And the microphone would probably have been thought of as a remote control, so we never even tried that.

What interested me about the whole experience was that we rarely talked about programming. The team was always trying to solve a problem—sometimes the solution was a new attachment for the robot, sometimes it was a new behavior. They quickly picked up control and conditionals. Looping was harder, and I don’t think they ever really got there —they were much happier copying and pasting sections of the program to get the number of iterations they wanted.

Second Story

The local group lost its funding the next year and FLL in Scotland lapsed until this year, when a local company, Lamda Jam brought it back. This year, as part of the local British Computer Society branch I was asked to judge the robot competition. There were 7 schools competing—two high schools and five junior schools.

One difference in this competition was that, although there was a lot of inventive engineering, there was no use of sensors at all. Even the best teams were controlling their robots using dead reckoning alone. One of the side effects of this was that scores for the same team, using the same programs, varied wildly. Each team played 3 heats, and the winning team’s scores varied between 125 and 348.

So what did I learn from these two experiences in teaching kids to code?

Having seen FLL from both the participant and judging sides I’m confident that the FLL can be an excellent vehicle for getting children to learn to code. Because there are so many different angles to the challenge, it’s easy for the team to do a little bit of everything without getting overwhelmed by any one aspect. The programming part of the challenge does introduce them to simple constructs, and with the right coach this can go a lot further. I can easily see a team building on its experience over a number of years, with some of the members eventually getting to be quite sophisticated practitioners.

But the counterintuitive insight was that it’s not so much about the code.

A comment in a recent thread on the accu-general mailing list with the subject line “Programming for teenagers” captured nicely what I learned from my experiences:

“I think that finding an interesting and reasonable-sized project that can be expanded upon is more important than the choice of tools and environment. That is the hook to keep her interested and search for ways to learn more, and I also think the very first visible result must be reached quickly.”

A good project, room to grow, and getting visible results quickly. Those are key. And one more thing: a good mentor. That, of course, would be you.

I encourage anyone with even the slightest interest in introducing children to programming to take a look at what FLL has to offer. You may end up doing more Lego than programming, but you’ll be giving the children exactly the right role model—us. Because we’re people who solve problems and get results quickly.

Seb wrote his first commercial software in the early eighties on an Apple II. He went on to graduate from the University of Edinburgh with a 1st Class Joint Honours in Computer Science and Electronics in 1987. Since then he has had a varied career working with companies, both large and small, in roles that cover the complete technical spectrum.

Over the past 6 years, Seb has focused on helping teams adopt and refine their agile practices. He wrote internal training courses for IBM’s Quality Software Engineering department (QSE) and went on to develop his own courses that he runs for clients throughout Europe He speaks regularly at international conferences, specializing in topics such as Unit Testing, Test Driven Development (TDD), Behaviour Driven Development (BDD) and Acceptance Test Driven Development (ATDD). He is a popular blogger and a regular contributor to technical journals, and is a joint author of the forthcoming The Cucumber-JVM Book from the Pragmatic Bookshelf.

Send the author your feedback or discuss the article in the magazine forum.

Pragmatic Bookstuff

Here’s what’s up with the Pragmatic Bookshelf and its authors.

New books from The Pragmatic Bookshelf.

Recently Released

Programming Sound with Pure Data

Dart for Hipsters, 1.0

Seven Web Frameworks in Seven Weeks

Web Development with Clojure

Grails 2: A Quick-Start Guide

Coming Up Next

Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions, in print

iPad and iPhone Kung Fu: Tips, Tricks, Hints, and Hacks for iOS 7, in print

Raspberry Pi: A Quick-Start Guide, in print

What’s Hot

Most lists only go to 10. Ours goes to 11!

1^ | 2 | Agile Web Development with Rails 4
2^ | 6 | Programming Ruby 1.9 & 2.0
3^ | NEW | Web Development with Clojure
4v | 3 | Crafting Rails 4 Applications
5^ | NEW | Seven Web Frameworks in Seven Weeks
6^ | NEW | Practical Programming
7v | 1 | Learn to Program with Minecraft Plugins
8v | 7 | Programming Elixir
9^ | NEW | Grails 2: A Quick-Start Guide
10^ | NEW | The Definitive ANTLR 4 Reference
11^ | NEW | Practical Vim

Who’s Where When

2014-02-26, Agile India, Bangalore

Keynote on Tacit Knowledge

Dave Thomas (author of Programming Ruby (2nd edition) , Agile Web Development with Rails (3rd edition) , The Ruby Object Model and Metaprogramming , Agile Web Development with Rails 4 , Programming Ruby 1.9 & 2.0 (4th edition) , Agile Web Development with Rails 3.2 , and Programming Elixir )

2014-02-28, NorDevCon, UK

XP at Unruly: a talk about how we use XP and where we've had to make adjustments to adapt to scale.

Rachel Davies (author of Agile Coaching )

2014-03-06, CocoaConf Chicago

AV Foundation Film School

Chris Adamson (author of iOS SDK Development )

2014-03-07, Joy of Coding, Rotterdam, Netherlands

Mutation testing for fun and profit

Seb Rose

2014-03-08, CocoaConf Chicago

Stupid Video Tricks

Chris Adamson

2014-03-27, CocoaConf DC

AV Foundation Film School

Chris Adamson

2014-03-28, CocoaConf DC

Stupid Video Tricks

Chris Adamson

2014-04-03, DevWeek, London, UK

So long, and thanks for all the tests

Seb Rose

2014-04-03, DevWeek, London, UK

Mutation testing - better code by making bugs

Seb Rose

2014-04-06, Agile Adria, Croatia

Giving a talk about coaching agile teams.

Rachel Davies

2014-04-24, CocoaConf San Jose

AV Foundation Film School

Chris Adamson

2014-04-26, CocoaConf San Jose

Stupid Video Tricks

Chris Adamson

2014-04-27, FITC Toronto 2014

Getting Started with 3D Game Programming in Three.js

Chris Strom (author of The SPDY Book , Dart for Hipsters , and 3D Game Programming for Kids )

2014-04-27, FITC Toronto 2014

Why Your Next Application Should Be Written in Dart

Chris Strom

2014-05-08, ACCU 2014, Brisol, UK

* Mock that Legacy C * SOLID Design for C

James Grenning (author of Test Driven Development for Embedded C )

2014-05-27, Agile Dev Practices, Potsdam, Germany

Introduction to Cucumber

Seb Rose

2014-06-19, Test Automation Day, Rotterdam, Netherlands

Don't you trust me? (A technical approach to building trust and consensus)

Seb Rose

What’s Happening

But to really be in the know, you need to subscribe to the weekly newsletter. It’ll keep you in the loop, it’s a fun read, and it’s free. All you need to do is create an account on pragprog.com (email address and password is all it takes) and select the checkbox to receive newsletters.

Antonio on Books

Don’t Make Me Think (and Other Excellent Advice)

by Antonio Cangiano

Antonio Cangiano is the author of the excellent Technical Blogging and you can subscribe to his reports on new books in technology and other fields here.

New Riders just published the third edition of their iconic Don’t Make Me Think web usability book. It’s not a coding book, as many of you will know, but instead is a quick, fun, and notably important read for anyone who writes code for a living.

Two other books that caught my eye this month, both of which are bona fide programming references, are XML and Web Technologies for Data Sciences with R (no worries, JSON is covered too) and Web Development with Clojure: Build Bulletproof Web Apps with Less Code.

And to wrap up, you’ll also notice that there has been an emergence of titles dedicated to Raspberry Pi as of late. All three presented here appear to be promising for the interested tinkerer.

Antonio’s Pick: Don't Make Me Think, Revisited: A Common Sense Approach to Web Usability (3rd Edition) By Steve Krug • ISBN: 0321965515 • Publisher: New Riders • Publication date: January 3, 2014 • Binding: Paperback • Estimated price: $26.46

XML and Web Technologies for Data Sciences with R By Deborah Nolan, Duncan Temple Lang • ISBN: 1461478995 • Publisher: Springer • Publication date: January 28, 2014 • Binding: Paperback • Estimated price: $55.73

The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies By Erik Brynjolfsson, Andrew McAfee • ISBN: 0393239357 • Publisher: W. W. Norton & Company • Publication date: January 20, 2014 • Binding: Hardcover • Estimated price: $12.99

A Concise Introduction to Logic By Patrick J. Hurley • ISBN: 1285196546 • Publisher: Cengage Learning • Publication date: January 1, 2014 • Binding: Hardcover • Estimated price: $125.98

Statistics, Data Mining, and Machine Learning in Astronomy: A Practical Python Guide for the Analysis of Survey Data By Zeljko Ivezic, Andrew J. Connolly, Jacob T VanderPlas, Alexander Gray • ISBN: 0691151687 • Publisher: Princeton University Press • Publication date: January 12, 2014 • Binding: Hardcover • Estimated price: $62.00

iOS 7 Application Development in 24 Hours, Sams Teach Yourself (5th Edition) By John Ray • ISBN: 0672337061 • Publisher: Sams Publishing • Publication date: January 24, 2014 • Binding: Paperback • Estimated price: $28.42

Android for Programmers: An App-Driven Approach (2nd Edition) By Paul Deitel, Harvey Deitel, Abbey Deitel • ISBN: 0133570924 • Publisher: Prentice Hall • Publication date: January 6, 2014 • Binding: Paperback • Estimated price: $24.28

Adventures In Raspberry Pi By Carrie Anne Philbin • ISBN: 1118751256 • Publisher: Wiley • Publication date: January 7, 2014 • Binding: Paperback • Estimated price: $13.87

Raspberry Pi Cookbook By Simon Monk • ISBN: 1449365221 • Publisher: O'Reilly Media • Publication date: January 7, 2014 • Binding: Paperback • Estimated price: $23.03

Raspberry Pi Hacks: Tips & Tools for Making Things with the Inexpensive Linux Computer By Ruth Suehle, Tom Callaway • ISBN: 1449362346 • Publisher: Maker Media, Inc • Publication date: January 5, 2014 • Binding: Paperback • Estimated price: $13.35

iOS 7 Programming Pushing the Limits By Rob Napier, Mugunth Kumar • ISBN: 1118818342 • Publisher: Wiley • Publication date: January 28, 2014 • Binding: Paperback • Estimated price: $30.10

Head First iPhone and iPad Development: A Learner's Guide to Creating Objective-C Applications for the iPhone and iPad By Dan Pilone, Tracey Pilone • ISBN: 1449316573 • Publisher: O'Reilly Media • Publication date: January 6, 2014 • Binding: Paperback • Estimated price: $18.98

jQuery and JavaScript in 24 Hours, Sams Teach Yourself By Brad Dayley • ISBN: 0672337347 • Publisher: Sams Publishing • Publication date: January 6, 2014 • Binding: Paperback • Estimated price: $19.14

HTML5 and CSS3 All-in-One For Dummies By Andy Harris • ISBN: 1118289382 • Publisher: For Dummies • Publication date: January 7, 2014 • Binding: Paperback • Estimated price: $22.48

Windows PowerShell Best Practices By Ed Wilson • ISBN: 0735666490 • Publisher: Microsoft Press • Publication date: January 6, 2014 • Binding: Paperback • Estimated price: $36.02

Intro to Java Programming, Comprehensive Version (10th Edition) By Y. Daniel Liang • ISBN: 0133761312 • Publisher: Prentice Hall • Publication date: January 6, 2014 • Binding: Paperback • Estimated price: $125.47

Scratch 2.0 Programming for Teens By Jr. Jerry Lee Ford • ISBN: 1305075196 • Publisher: Cengage Learning PTR • Publication date: January 1, 2014 • Binding: Paperback • Estimated price: $18.48

Linked Data By David Wood, Marsha Zaidman, Luke Ruth, Michael Hausenblas • ISBN: 1617290394 • Publisher: Manning Publications • Publication date: January 24, 2014 • Binding: Paperback • Estimated price: $28.96

Web Development with Clojure: Build Bulletproof Web Apps with Less Code By Dmitri Sotnikov • ISBN: 1937785645 • Publisher: Pragmatic Bookshelf • Publication date: January 12, 2014 • Binding: Paperback • Estimated price: $29.34

Seven Web Frameworks in Seven Weeks: Adventures in Better Web Apps By Jack Moffitt, Frederic Daoud • ISBN: 1937785637 • Publisher: Pragmatic Bookshelf • Publication date: January 10, 2014 • Binding: Paperback • Estimated price: $23.54

Shady Illuminations

The Parable of the Streetlight

by John Shade

John shares a parable.

Last month I think I mixed up Neptune and Saturn. My apologies to both. Anyway, this month I want to share my version of a 1924 Boston version of a timeless parable:

The Parable of the Streetlight

In the wee hours of a Boston morning, before the sun had found its way to Copley Square, an earnest young police officer of the Hibernian tribe was walking the traditional beat assigned to those eager officers of the most tender tenure. And as will happen in the natural order of things, this beardless representative of Bay State law and order came upon a fellow Bostonian who was as clearly in his cups as he was clearly crawling on his hands and knees under a streetlight. The officer, with a quick intelligence that spoke well of his upbringing, deduced at once that the man was looking for something.

“Faith and begorrah,” the officer said, “And what might ye be doin’ down there on the Copley Square pavement at this ungodly hour of the mahrnin’?” Because that’s the best I can do for an Irish cop in 1924 Boston.

“Evenin’, officer,” rejoined the inebriate with exaggerated dignity, tipping his hat promptly into the Copley Square gutter, “I lost a two-dollar bill, and I need it to pay my rent.”

“To pay your rent, to be sure,” the officer replied skeptically as he handed the hat back to the unfortunate tippler. But he was a kindly sort, and remembered what his dear old mother had told him that very week about the responsibility laid upon him, by his position, to help those in need. And our young Patrick or Michael as the case may be got down on his hands and knees to help this victim of the demon rum search for his two-dollar bill.

They searched, and they searched with an intensity and thoroughness that would have gladdened your heart had you been there to witness it. While they were about their searching, hints of sunlight to come began to flicker into the dim corners of the Square. Time, as is its wont, advanced, but the search, alas, did not. In the end, all their efforts proved fruitless, bootless, and decidedly two-dollar-bill-less.

At last, rising and brushing the ancient dust of the Cabots and the Lodges from his police offficer trouser knees, the policeman sighed and asked, “Are ye absolootely sure ye lost the currency right here on this here corner?”

“Nah,” the toper replied with aplomb, “I dropped it in an alley just off Atlantic Avenue.”

Retaining his dignity by recalling another nugget of his sainted mother’s advice, the police officer nevertheless sputtered incoherently for a moment before inquiring, “What’s that ye say, ye pickle-livered reprobate? Ye say ye lost this benighted two-dollar bill down by Atlantic Avenue? Is that what ye are sayin’? Then why in the name of all that’s holy are ye scrabblin’ round here on the pavement in Copley Square?”

“Because,” said the man with a superior sniff, “the light’s better up here.”

Lightning Bugs

Yeah, so that’s the Parable of the Streetlight. It’s one of my favorites. Right up there with “We need the eggs.” You can apply it to all kinds of situations.

But you can also apply it to a not-quite-right situation. The mark of a good parable is that it can be applied to many different situations, and the best stories have many morals. Often contradictory. Which is why it’s really annoying when somebody with a tin ear for analogy applies a parable to a not-quite-right situation. It’s like the Mark Twain lightning bug thing. You know:

“The difference between the right word and the almost-right word is the difference between lightning and a lightning bug.”–Mark Twain

The almost-right word or the not-quite-right application of a parable undermines the impact of the right word or the right application. Like all those people saying “three time more” when they really mean “three times as much” make it impossible for those who know that “three times more” means “four times as much” to use the expression effectively.

Bad communication is not just bad in itself, it can sometimes undermine good communication.

Here’s a not-quite-right way to apply the Streetlight Parable:

Look around on the internet for “streetlight joke” (for some reason nobody else seems to refer to it as a parable, so you have to search this way) and you’ll find a lot of discussion about poorly-designed research studies and theories that were eventually overthrown by looking on a different street corner. The implication of many of these discussions is that science goes askew when scientists look at the convenient variables rather than the hard-to-measure ones. Well, sure. Hard to argue with that. But this, if you follow the implication of these discussions, is the Streetlight Parable applied. I’ll argue with that.

I claim that this is a not-quite-right application of the Streetlight Parable. I say this because the parable applies far more interestingly to all scientific research. Applying it only to bad scientific research undermines that more significant application.

Here’s what I’m talking about: In the design of scientific experiments, the thing you measure is never the thing you’re theorizing about. It’s always a convenient proxy for the thing you’re theorizing about. Your theory makes an assertion about intelligence or the mass of any object; your experiment records the results of a particular group of subjects on a particular test in a particular setting or the reading on a scale in your lab. The thing you’re theorizing about is always either too general or too abstract or too remote or just too inconvenient to examine directly. So you look over here where the light is better. Doing the streetlight thing is a necessary element of the scientific method. It’s how you make the leap from a concrete result in a lab to a general universal principle.

Applying the Streetlight Parable to the quality of research is not wrong, it’s just lame. Like criticizing Nazis because their uniforms weren’t cool.

Dumb as Dirt

Here’s a different way of using the Streetlight Parable that is sufficiently different that it’s OK. Anyway that’s what I claim.

When we search the internet, we only look where the light is good. The thing we’re looking for may be over there, but we’re going to search for it in the cone of light cast by the tools we have. Most of the internet will fall outside that cone. Some examples of what we’re looking for will surely fall outside the cone. Much of what we’re searching for will fall inside the cone, but how much? Can we even quantify it? Search tools score a lot of hits for us, but how much do we know about the misses?

The obvious tool for search is the search engine, and a lot of effort goes into making search engines effective. But there are other tools. There is also the data itself. The intelligence of the data is part of the toolkit for searching. And most data is dumb as dirt.

I say we need better data. Smarter data. Data that has some self-knowledge, some introspection.

Currently, we have something that passes itself off as introspection, in the form of hand-crafted metadata. Which doesn’t work. Not really.

If I’m interested in gritty World War II action movies featuring a strong female lead, I am restricted to metadata keyed in by people explicitly because they assumed that enough people would be searching on these terms to make them worth keying in. Is Cate Blanchette a strong female lead in Blue Dahlia? Is Catch-22 a gritty World War II action movie?

Basically, I am at the mercy of some anonymous person’s opinions about who is a strong female lead, and I am restricted to searching for aspects of movies that many other people are also searching for. So I get dumbed-down results and all the really interesting questions are ignored.

What goes for movie searches applies to everything you might want to search for. Search will remain under the streetlight until we have truly introspective data. Not externally-attached tags of metadata stapled on as an afterthought, but self-knowledge that comes from deep reflection. Data needs to look at itself and discover what’s interesting about itself. It probably needs to learn to meditate.

I don't know how to teach data to meditate. Nobody does. It's an opportunity. You're welcome.

John Shade was born under a cloud in Montreux, Switzerland, in 1962. Subsequent internment in a series of obscure institutions of ostensibly higher learning did nothing to brighten his outlook. Today he wanders the earth in search of a better streetlight. Follow John on Twitter, send him your feedback, or discuss the article in the magazine forum.

Rear Window

A Parting Shot

Happy 30th birthday, Mac!

Thirty years ago, in January 1984, Apple released the Macintosh. Here’s a baby picture of the birthday box.

Photo courtesy of Apple Computer.