Node: Up and Running

Tom Hughes-Croucher

Editor

Simon St. Laurent

Copyright © 2011 Yahoo!, Inc.

O'Reilly Media

Author's Note

When Simon, my editor, and I were initially discussing this project it was obvious how vibrant the Node.js community is. We felt that it was important that we engaged with the community as we worked on this manuscript. In order to do that we decided to release the book in parts as I wrote it. What you are reading now is one of those partial releases.

What you'll find within this first release is not necessarily the final work that we will publish. We hope by making this book available as it's written we'll get your feedback, ideas and thoughts on what I've already written and what else we should be covering. Not only that, but Node is growing almost faster than I can keep up with it. Not to mention the community modules which are growing even faster than that. Before we go to press we'll make sure that the information in the text is up to date with the latest version of Node, and that we haven't missed any critical contributions from the community that need to be covered.

--Tom

Preface

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This icon signifies a tip, suggestion, or general note.

Caution

This icon indicates a warning or caution.

Using Code Examples

This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Book Title by Some Author (O’Reilly). Copyright 2011 Some Copyright Holder, 978-0-596-xxxx-x.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.

Safari® Books Online

Note

Safari Books Online is an on-demand digital library that lets you easily search over 7,500 technology and creative reference books and videos to find the answers you need quickly.

With a subscription, you can read any page and watch any video from our library online. Read books on your cell phone and mobile devices. Access new titles before they are available for print, and get exclusive access to manuscripts in development and post feedback for the authors. Copy and paste code samples, organize your favorites, download chapters, bookmark key sections, create notes, print out pages, and benefit from tons of other time-saving features.

O’Reilly Media has uploaded this book to the Safari Books Online service. To have full digital access to this book and others on similar topics from O’Reilly and other publishers, sign up for free at http://my.safaribooksonline.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at:

http://www.oreilly.com/catalog/<catalog page>

To comment or ask technical questions about this book, send email to:

bookquestions@oreilly.com

For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see our website at:

http://www.oreilly.com

Part I. Core Concepts

Chapter 1. Introduction

A very brief introduction to Node.js

Node.js is many things, but mostly it's a way of running JavaScript outside the web browser. This book will cover why that's important, and what benefits Node.js provides. This introduction attempts to sum up that explanation in a few paragraphs, rather than a few hundred pages.

Many people use the JavaScript programming languages extensively for programming the interfaces of Web sites. Node.js allows this popular programming language to be applied in many more contexts, in particular on Web servers. There are several notable features about Node.js that make it worthy of interest.

Node is a wrapper around the high-performance V8 JavaScript runtime from the Google Chrome browser. Node tunes V8 to work better in contexts other than the browser, mostly by providing additional APIs which are optimized for specific use cases. For example, in a server context manipulation of binary data is often necessary. This is poorly supported by the JavaScript language and as a result, V8. Node's Buffer class provides easy manipulation of binary data. As such, Node doesn't just provide direct access to the V8 JavaScript runtime. It also makes JavaScript more useful for the contexts in which people use Node.

V8 itself uses some of the newest techniques in compliler technology. This often allows code written in a high-level langauge like JavaScript to perform similarly to code written in much lower level language, like C, with a fraction of the development cost. This focus on performance is a key aspect of Node.

JavaScript is an event driven language, and Node uses this to its advantage to produce highly scalable servers. Using an architecture called an event loop, Node makes programming high scalable servers both easy and safe. There are various strategies that are used to make servers performant. Node has chosen an architecture that performs very well but also reduces the complexity for the application developer. This is an extremely important feature. Programming concurrency is hard, and frought with dangers. Node side-steps these while still offering impressive performance. As always any approach still has trade-offs that are discussed in detail later in the book.

To support the event-loop approach, Node supplies a set of "non-blocking" libraries. In essence these are interfaces to things like the filesystem or databases which operate in an event-driven way. When you make a request to the file system, rather than requiring Node to wait for the hard drive to spin up and retrive the file the non-blocking interface simply notifies Node when it has access in the same way that web browsers notifies your code about an onclick event. This model simplifies access to slow resources in a scalable way that is intuitive to JavaScript programmers and easy to learn for everyone else.

While not unique to Node, supporting JavaScript on the server is also a powerful feature. Whether we like it or not the browser environment gives us little choice of programming languages. Certainly if we would like our code to work in any reasonable percentage of browsers JavaScript is the only choice. To acheive any asperations of sharing code between the server and the brower we must use JavaScript. Due to the increasingly complexity of client applications we are building in the browser using JavaScript (such as GMail) the more code we can share between the browser and the server the more we can reduce the cost of creating rich web applications. Since we must rely on JavaScript in the browser having a server-side environment that use JavaScript opens the door to code sharing in a way which is not possible using other server-side languages such as PHP, Java, Ruby or Python. While there are other platforms which support programming web servers with JavaScript, Node is quickly becoming the dominent platform in the space.

Aside from what you can build with Node, one extremely pleasing aspect is how much you can build for Node. Node is extremely extensible with a large volume of community modules having been built in the relatively short time the project has been running. Many of these are drivers to connect with databases or other software, but many are also useful software applications in their own right.

The last but certainly not least reason to celebrate Node is its community. The Node project is still very young, and yet rarely has the author seen such fervor around a project. Both novices and experts have coalesced around the project to use and contribute to Node, making it both a pleasure to explore and a supportive place to share and get advice.

Chapter 2. Why Node

In writing this book I've been acutely aware of how new Node.js is. Many platforms take years to find adoption, and yet I've found a level of excitement around Node.js that I've never seen before in such a young platform. I hope that by looking at the reasons other people are getting so excited about Node.js I will find reasons that also resonate for you. By looking at Node.js' strengths we can find the the places where it is most applicable. This chapter will look at the factors that have come together to create a space for Node.js and look at the reasons why it's become so popular in such a short time.

2.1 Professionalism in JavaScript

JavaScript was created by Brendan Eich in 1995 to be a simple scripting language for use in web pages on the Netscape browser platform. Surprisingly almost since its inception JavaScript has been used in non-browser settings. Some of the early Netscape server products supported JavaScript (known then as LiveScript) as a server-side scripting language. While server-side JavaScript didn't really catch on then, that certainly wasn't true for the exploding browser market. On the Web JavaScript competed with Microsoft's VBScript to provide programming functionality in Web pages. It's hard to say why JavaScript won, perhaps Microsoft allowing JavaScript in Internet Explorer[1] did it. Perhaps it was the JavaScript language itself, but win it did. This meant by the early 2000s JavaScript had emerged as the Web language. Not the first choice, but the only choice for programming with HTML in browsers.

What does this have to do with Node.js? Well the important thing to remember is that when the AJAX revolution happened and the Web became big business (think Yahoo, Amazon, Google, etc) the only choice for the "J" in AJAX was JavaScript there simply wasn't an alternative. As a result a whole industry needed an awful lot of JavaScript programmers, really good ones at that, rather fast. The emergence of the Web as a serious platform and JavaScript as its programming language meant that we, as JavaScript programmers needed to shape up. We can equate the change in JavaScript as the second or third programming language of a programmer to the change in perception of its importance. We started to get emerging experts who lead the charge in making JavaScript respectable.

Arguably at the head of this movement was Douglas Crockford. His popular articles and videos on JavaScript have helped many programmers discover that inside a language much maligned there is a lot of inner beauty. Most programmers working with JavaScript had spent the majority of their time working with the browser implementation of the W3C DOM API for manipulating HTML or XML documents. Unfortunately, the DOM is probably not the prettiest API ever conceived, but worse its various implementations in the browsers are inconsistent and incomplete. No wonder that for a decade after its release JavaScript was not thought of as a "proper" language by so many programmers. More recently Douglas' work on "the good parts" of JavaScript have helped create a movement of advocates of the language which recognize that it has a lot going for it despite the warts.

In 2010 we now have a proliferation of JavaScript experts advocating well written, performant, maintainable JavaScript code. People such as Douglas Crockford, Dion Almaer, Peter Paul Koch (PPK), John Resig, Alex Russell, Thomas Fuchs, and many more have provided research, advice, tools, and primarily libraries that have allowed thousands of professional JavaScript programmers worldwide to practice their trade with a spirit of excellence. Libraries like jQuery, YUI, Dojo, Prototype, Mootools, Sencha and many others are now used daily by thousands of people and deployed on millions of Web sites. It is in this environment where JavaScript is not only accepted, but widely used and celebrated that a platform larger than the web makes sense. When so many programmers know JavaScript its ubiquity has become a distinct advantage.

When I speak at conferences I can ask a room full of Web programmers what languages they use. Java and PHP are very popular, Ruby is probably next most popular these days or at least closely tied with Python and Perl still has a huge following. However, almost without exception anyone who does any programming for the web has programmed in JavaScript. While backend languages are fractured in browser programming is united by the necessities of deployment. Various browsers and browser plugins allow the use of other languages, but they simply aren't universal enough for the web. So here we are with a single universal web language. How can we get it on the server?

[1] Internet Explorer doesn't actually support JavaScript or ECMAScript, it supports a language varienty called JScript. In recent years JScript has fully supported the ECMAScript 3standard and has some ECMAScript 5 supoort. However it also implements proprietary extensions in the same way Mozilla JavaScript has features that ECMAScript does not.

2.2 Browser Wars 2.0

Fairly early in the days of the Web we had the infamous browser wars. Internet Explorer and Netscape competed viciously on Web features, adding various incompatible programmatic features to their browsers and not supporting the features in the other browser. For those of us who programmed the web this was the cause of much anguish because it made Web programming really tiresome. Internet Explorer more or less emerged the winner of that round and became the dominant browser. Fast forward a few years, Internet Explorer has been languishing at version 6 and a new contender, Firefox emerges from the remnants of Netscape. Firefox kicks off a new resurgence in browsers being followed by Webkit (Safari) and then Chrome. Most interesting about this current trend is the resurgence of competition into the browser market.

Unlike the first iteration of the browser wars today's browser compete on two fronts, adhering to the standards that emerged after the previous browser war and performance. As Web sites have become more complex users want the fastest experience possible. This has meant that browsers not only need to support the Web standards well, allowing developers to optimize, but also to do a little optimization of their own. JavaScript being a core component of Web 2.0, AJAX web sites has become part of the battleground.

Each browser has their own JavaScript runtimes: Spider Monkey for Firefox, Squirrel Fish Extreme for Safari, Karakan for Opera, and finally V8 for Chrome. As these runtimes compete on performance it creates an environment of innovation for JavaScript. In order to differentiate their browsers vendors are going to great lengths to make them as fast as possible.

Chapter 3. Understanding Node.js

To make the most of the ServerSide JavaScript environment, it's important to understand some core concepts behind the design choices that were made for Node.js and JavaScript in general. Understanding the decisions and tradeoffs will make it easier for you to write great code and architect your systems. It will also help you explain to other people why Node.js is different from other systems they've used and where the performance gains come from. No engineer likes unknowns in their systems. "Magic" is not an acceptable answer, so it helps to be able to explain why a particular architeture is benificial and under what circumstances.

The Event Loop

A fundamental part of Node is the event loop, a concept underlying the behavior of JavaScript as well as most other interactive systems. In many languages, event models are bolted onto the side, but JavaScript events have always been a core part of the language. This is because JavaScript has always dealt with user interaction. Anyone who has used a modern Web brower is used to Web pages that do things "onclick," "onmouseover," etc. These events are so common that we hardly think about them when writing Web page interaction, but having this event support in the language is incredibly powerful. On the server, instead of the limited set of events based on the user-driven interaction with the Web page's DOM, we have an infinite variety of events based on what's happening in the server software we use. For example, the HTTP server module provides an event called "request," emitted when a user sends the Web server a request.

The event loop is the system that JavaScript uses to deal with these incoming request from various parts of the system in a sane manner. There are a number of ways people deal with "real-time" or "parallel" issues in computing. Most of them are fairly complex and frankly make my brain hurt. JavaScript takes a simple approach that makes the process much more understandable but does introduce a few constraints. By having a grasp of how the event loop works, you'll be able to use it to it's full advantage and avoid the pitfalls of this approach.

Node takes the approach that all I/O activities should be non-blocking (for reasons we'll explain more later). This means that HTTP requests, database queries, file I/O, etc. do not halt execution until they return. Instead, they run independently, and then emit an event when their data is available.

This means that programming in Node.js has lots of callbacks dealing with all kinds of I/O, many callbacks initiating other callbacks in a cascading fashion. This is a very different from browser programming. There is still a certain amount of linear setup, but the bulk of the code involves dealing with callbacks.

Because of this somewhat unfamiliar programming style, we need to look for patterns to help us effectively program on the server. That starts with the event loop. I think that most people intuitively get event driven programming because it's like everyday life. Imagine you are cooking. You are chopping a bell pepper and a pot starts to boil over (Figure 3-1). You finish the slice you are doing, and then turn down the stove.

Figure 3-1. Event Driven People

In every day life we are used to having all sorts of internal callbacks for dealing with events, and yet, like JavaScript, we always do just one thing at once. Yes, yes, I can see you are rubbing your tummy and patting your head at the same time, well done. But, if you try to do any serious activities at the same time, it goes wrong pretty quick. This is like JavaScript. It's great at letting events drive the action, but it's "single-threaded" so that only one thing happens at once.

This single-threaded concept is really important. One of the criticisms leveled at Node.js fairly often is its lack of "concurrency." That is, it doesn't use all of the CPUs on a machine to run the JavaScript. The problem with running code on multiple CPUs at once is that it requires co-ordination between multiple "threads" of execution. In order for multiple CPUs to effectively split up work, they would have to be able to talk to each other about the current state of the program, what work they'd each done, etc. While this is possible, it's a more complex model that requires more complexity from both the programmer and the system. JavaScript's approach is simple: there is only one thing happening at once. Since everything that Node does is non-blocking, the time between an event being emited and Node being able to act on that event is very small, because it's not waiting on things like disk I/O.

Another way to think about the event loop isto compare it to a postman (mailman). To our event loop postman, each letter is an event. He has a stack of events to deliver in order. For each letter (event) the postman gets, he walks to the route to deliver the letter (Figure 3-2). The route is the callback function assigned to that event (sometimes more than one). However, critically, since our postman only has a single set of legs, he can walk only a single code path at once.

Figure 3-2. The Event Loop Postman

Sometimes, while the postman is walking a code route, someone will give him another letter. This is the callback function he is visiting at the moment. In this case, the postman delivers the new message immediately (after all, someone gave it to him directly instead of going via the post office, so it must be urgent). The postman will diverge from his current code path and walk the proper code path to deliver the new event. He then carries on walking the original event that emitted the event he just walked.

Let's look at the behaviour of our postman in a typical program by picking something really simple. Supose we have a Web (HTTP) server that get requests, retrieves some data from a database, and returns it to the user. In this scenario we have a few events to deal with. First (as in most cases) comes the request event from the user asking the Web server for a Web page. The callback that deals with the initial request (let's call it callback A) looks at the request object and figures out what data it needs from the database. It then makes a request to the database for that data, passing another function, callback B, to be called on the response event. Having handled the request, callback A returns. When the database has found the data, it issues the responseevent. The event loop then calls callback B, which sends the data back to the user.

This seems fairly straight forward. The obvious things to note here are the "break" in the code, which you wouldn't get in a proceedural system. Since Node.js is non-blocking system, when we get to the database call that would make us wait, we instead issue a callback. This means that different fundtions must start handling the request and finish handling it when the data is ready to return. So we need to make sure that we pass any state we need to the callback, or make it available in some other way. JavaScript programming typically does it through closures. We'll discuss that in more detail later.

Let's look at another example. We've given the postman a letter to deliver that requires a gate to be opened. He gets there and the gate is closed, so he simply waits and tries again, and again. He's trapped in an endless loop waiting for the gate to open (Figure 3-3). But there is a letter on the stack that will ask someone to open the gate so the postman can get through. Surely that will solve things, right? Unfortunately it won't unless the postman gets to deliver the letter, and currently he's stuck waiting endlessly for the gate to open. This is because the event that opens the gate is external to the current event callback. If we emit the event from within a callback, we already know our postman will go and deliver that letter before carrying on, but when events are emitted outside the currently executing piece of code, they will not be called until that piece of code has been fully evaluated to its conclusion.

Figure 3-3. Blocking the event loop

As an example, the following code creates a loop that Node.js (or a browser) will never break out of:

Example 3-1. Event loop blocking code

                EE = require('events').EventEmitter;
ee = new EE();

die = false;

ee.on('die', function() {
    die = true;
});

setTimeout(function() {
    ee.emit('die');
}, 100);

while(!die) {
}

console.log('done');

In this example, console.log will never be called because the while loop stops Node from ever getting a chance to callback the timeout and emit the die event. Although it's unlikely we'd program a loop like this that relies on an external condition to exit, it illustrates how Node.js can only do one thing at once, and getting a fly in the ointment can really screw up the whole server.

Given a basic understanding of the event loop, let's look at the standard Node.js code for creating an HTTP server:

Example 3-2. A basic HTTP server

                var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(8124, "127.0.0.1");
console.log('Server running at http://127.0.0.1:8124/'

This code is the most basic example from the Node.js Web site (but as we'll see soon, it's not the ideal way to code). The example creates an HTTP server using a factory method in the http library. The factory method creates a new HTTP server and attaches a callback to the request event. The callback is specified as the argument to the createServer method. What's interesting here is what happens when this code is run. The first thing Node.js does is run the code above from top to bottom. This can be considered the 'setup' phase of Node programming. Since we attached some event listeners, Node.js doesn't exit, but waits for an event to be fired. If we didn't attach any events, Node.js would exit as soon as it had run the code.

So what happens when the server gets an HTTP request? Node.js emits the request event, which causes the callbacks attached to that event to be run in order. In this case, there is only one callback, the anonymous function we passed as an argument to createServer. Let's assume it's the first request the server has had since setup. Since there is no other code running, the request event is handled immediately and the callback is run. It's a very simple callback and it runs pretty fast.

Let's assume that our site gets really popular and we get lots of requests. If, for the sake of argument, our callback takes 1 second and we get a second request shortly after the first one, and the second request isn't going to be acted on for another second or so. Obviously, a second is a really long time, and as we look at the requirements of real world applications, the problem of blocking the event loop becomes more damaging to the user experience. The operating system kernel actually handles the TCP connections to clients for the HTTP server, so there isn't a risk of rejecting new connections, but there is a real danger of not acting on them. The upshot of this is that we want to keep Node.js as event-driven and non-blocking as possible. In the same way that a slow I/O event should use callbacks to indicate the presence of data Node.js can act on, the Node.js program itself should be written in such a way that no single callback ties up the event loop for extended pieces of time.

This means that you should follow two strategies when writing a Node.js server:

Once set up has been completed, make all actions event driven.

If Node.js is required to process something that will take a long time consider deligating it to Web workers.

Taking the event driven approach works effectively with the event loop (the name is a hint it would), but it's also important to write event-driven code in a way that is easy to read and understand. In the previous example, we used an anonymous function as the event callback, which makes things hard in a couple of ways.

First, we have no control over where the code lives: the anonymous function must live where it is attached to the event either via a factory method or the on method of an EventEmitter. The second issue is debugging. If everything is an anonymous event, it can sometimes be hard to distinguish similiar callbacks from each other when an exception occurs.

Chapter 4. Event Driven Programming Style

Patterns

The I/O Problem Space

Event driven programming is different from proceeedural programming. Before we launch into patterns let's take a look at what is really happening in various programming styles to give some context to the patterns. Firstly as we discussed in the previous chapter event-driven programming is focused on solving an I/O problem. When we are working with data in memory that doesn't require I/O Node can be completely proceedural. As such most of this chapter will focus on I/O, although we will also look at some JavaScript patterns that will help you write good code.

Let's start by defining various types of I/O requirements by systems. We can use these as the basis of our patterns. The first obvious thing to look it is serial and parallel I/O. Serial is obvious. Do this I/O, when it is finished do that I/O. Serial is also easy to understand do this I/O and that I/O at the same time.

Serial and parallel work can also be combined as units. For example paralell requests in serial: do this and that then do other and another.

An important point here though, ordering is normally considered implicit in serial tasks. However the same is not true for parallel tasks. Paralell tasks could return in any order.

In Node we assume that all I/O has unbounded latency. That means that any I/O tasks could take from 0 to infinity time. We don't know, and can't assume how long these tasks take so instead of waiting for them we use placeholders (events) which then fire callbacks as the I/O happens. Since we have assumed unbounded latency exists then it's easy to to perform parallel tasks. You simply make a number of calls for various I/O tasks. They will return whenever they are ready in whatever order that happens to be. Ordered serial requests are also easy by nesting or referencing callbacks together so that the first callback will initiate the second I/O request, the second callback will initiate the third, etc. So while every request is asyncronous and doesn't block the event loop the requests are only made in serial.

So far we have two ways to do I/O. Ordered serial requests, and unordered parallel requests. We can also consider ordered parallel requests as a useful pattern. We want to do the I/O request themselves in parallel but deal with the requests in a particular sequence. Unordered serial is less useful because it's just like unordered parallel except less request are happening together, and so it's slower. We will want to make ordered serial requests when we require the results of one piece of I/O to inform the detuals of the next piece. We have 3 main classes of I/O now. Ordered serial, unordered parallel and order parallel. Don't forget we can also mix and match them.

Ordered serial I/O

In this pattern we want to do some I/O (unbounded latency) tasks in sequence. Each previous task must be completed before the next task is started. In Node's structure of callbacks this means nesting callbacks so that as each task completes and issues a callback the next is started.

Example 4-1. Nesting callbacks to produce serial requests

            server.on('request', function(req, res) {
  //get session information from memcached
  memcached.getSession(req, function(session) {
    //get information from db
    db.get(session.user, function(userData) {
      //some other web service call
      ws.get(req, function(wsData) {
        //render page
        page = pageRender(req, session, userData, wsData);
        //output the response
        res.write(page);
      });
    });
  });
});

While nesting callbacks allows easy creation of ordered serial I/O is also creates so called 'pyramid' code[2]. This code can be hard to read and understand, and as a consiquence hard to maintain. There are a few ways to make this code more useful without breaking the fundamental ordered serial pattern.

Firstly we can continue to use inline function declarations but we can name them. This makes debugging a lot easier as well as giving an indication of what the callback is going to do.

Example 4-2. Naming function calls in callbacks

            server.on('request', getMemCached(req, res) {
  memcached.getSession(req, getDbInfo(session) {
    db.get(session.user, getWsInfo(userData) {
      ws.get(req, render(wsData) {
        //render page
        page = pageRender(req, session, userData, wsData);
        //output the response
        res.write(page);
      });
    });
  });
});

Another approach that changes the style of code is to use declared functions instead of just anonymous or named ones. This removes the natural pyramid seen in the other approaches, which shows the order of execution, but it also breaks the code out into more managable chunks.

Example 4-3. Using declared functions to seperate out code

        var render = function(wsData) {
  page = pageRender(req, session, userData, wsData);
};

var getWsInfo = function(userData) {

};

var getDbInfo = function(session) {

};

var getMemCached = function(req, res) {

};

[2] This term was coined by Tim Caswell

Part II. Writing Code with Node.js

Chapter 5. Getting Started

Installing Node.js

Installing Node.js is fairly simple, but currently requires a POSIX-compliant operating system. So if you are using Windows, you will need to install Node.js on a virtual machine running Linux or some other POSIX OS. Node.js is available from two primary locations: the project's web site or the Github repository. You're probably better off with the Node web site because it contains the official releases. The latest cutting edge features are hosted on Github for the core development team and anyone else who wants a copy. While these features are new and often intriguing, they are also less stable than those in a release.

Let's get started by installing Node.js on a Linux machine; then we'll explore some other operating systems. The first thing to do is download Node.js from the web site. So let's go there and find the latest release. From the Node homepage find the download link:

Figure 5-1. Downloading the Node.js source code

The current release at the time of print is 0.4.1, which is a stable release. Let's download that with curl into our home directory:

Example 5-1. Getting a copy of the Node source

Enki:~ $ curl -O http://nodejs.org/dist/node-v0.4.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4741k  100 4741k    0     0  1218k      0  0:00:03  0:00:03 --:--:-- 1230k
Enki:~ $

Note

Node.js version numbers follow the C convention of major.midi.minor. Stable versions of Node.js have an even midi version number, development versions have an odd midi version number. It's unclear when Node will start using the version numbers, but it's a fair assumption that it will only be for the first "production" release.

Once you have the code, you'll need to unpack it. The tar command does this using the flags xzf. The x stands for extract (rather than compress), z tells tar to also decompress using the GZIP algorithm, and f indicates we are unpacking the filename given as the final argument:

Example 5-2. Unpacking the code

Enki:~ $ tar xzf node-v0.4.1.tar.gz
Enki:~ $ cd node-v0.4.1
Enki:~/node-v0.4.1 $ ls
AUTHORS			README.md		doc
CMakeLists.txt		TODO			lib
CTestConfig.cmake	TODO.win32		src
ChangeLog		benchmark		test
LICENSE			cmake			tools
Makefile		config.h.cmake		wscript
Makefile.cmake		configure
README.cmake		deps
Enki:~/node-v0.4.1 $

The next step is to configure the code for your system. Node.js uses the configure/make system for its installation. The configure script looks at your system and finds the paths Node needs to use for the dependancies it needs. Node generally has very few dependancies. The installer requires Python 2.4 or greater, and if you wish to use TLS or cryptology (such as SHA1), Node needs the OpenSSL development libraries. Running configure will let you know whether any of these dependancies are missing. I've included some more specific instructions for a few platforms in see Appendix A for readers who have never done this kind of install before.

Example 5-3. Configuring the Node install

Enki:~/node-v0.4.1 $ ./configure
Checking for program g++ or c++          : /usr/bin/g++
Checking for program cpp                 : /usr/bin/cpp
Checking for program ar                  : /usr/bin/ar
Checking for program ranlib              : /usr/bin/ranlib
Checking for g++                         : ok

...

Checking for function nanosleep          : yes
Checking for function ceil               : yes
Checking for fdatasync(2) with c++       : no
'configure' finished successfully (2.351s)
Enki:~/node-v0.4.1 $

The next installation step is to make the project. This compiles Node and builds the binary version of the project that you will use into a build directory of the source directory we've been using. Node numbers each of the build steps it needs to do so you can follow the progress it makes during the compile.

Example 5-4. Compiling Node with the make command

Enki:~/node-v0.4.1 $ make
Waf: Entering directory `/Users/croucher/node-v0.4.1/build'
DEST_OS: darwin
DEST_CPU: x64
Parallel Jobs: 1
Product type: program
[ 1/74] cc: deps/libeio/eio.c -> build/default/deps/libeio/eio_1.o
/usr/bin/gcc -rdynamic -D_GNU_SOURCE -DHAVE_CONFIG_H=1 -pthread -arch x86_64 -g -O3 -DHAVE_OPENSSL=1 -DEV_FORK_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_MULTIPLICITY=0 -DX_STACKSIZE=65536 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DEV_MULTIPLICITY=0 -DHAVE_FDATASYNC=0 -DPLATFORM="darwin" -D__POSIX__=1 -Wno-unused-parameter -D_FORTIFY_SOURCE=2 -DNDEBUG -Idefault/deps/libeio -I../deps/libeio -Idefault -I.. ../deps/libeio/eio.c -c -o default/deps/libeio/eio_1.o
In file included from ../deps/libeio/eio.c:99:
default/config.h:11:1: warning: "HAVE_FDATASYNC" redefined
<command-line>: warning: this is the location of the previous definition
[ 2/74] cc: deps/libev/ev.c -> build/default/deps/libev/ev_1.o
/usr/bin/gcc -rdynamic -D_GNU_SOURCE -DHAVE_CONFIG_H=1 -pthread -arch x86_64 -g -O3 -DHAVE_OPENSSL=1 -DEV_FORK_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_MULTIPLICITY=0 -DX_STACKSIZE=65536 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DEV_MULTIPLICITY=0 -DHAVE_FDATASYNC=0 -DPLATFORM="darwin" -D__POSIX__=1 -Wno-unused-parameter -D_FORTIFY_SOURCE=2 -DNDEBUG -Idefault/deps/libev -I../deps/libev -Idefault -I.. ../deps/libev/ev.c -c -o default/deps/libev/ev_1.o
In file included from ../deps/libev/ev.c:45:
default/config.h:11:1: warning: "HAVE_FDATASYNC" redefined
<command-line>: warning: this is the location of the previous definition
[ 3/74] cc: deps/c-ares/ares_strcasecmp.c -> build/default/deps/c-ares/ares_strcasecmp_1.o
/usr/bin/gcc -rdynamic -D_GNU_SOURCE -DHAVE_CONFIG_H=1 -pthread -arch x86_64 -g -O3 -DHAVE_OPENSSL=1 -DEV_FORK_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_MULTIPLICITY=0 -DX_STACKSIZE=65536 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DEV_MULTIPLICITY=0 -DHAVE_FDATASYNC=0 -DPLATFORM="darwin" -D__POSIX__=1 -Wno-unused-parameter -D_FORTIFY_SOURCE=2 -DNDEBUG -Idefault/deps/c-ares -I../deps/c-ares -Idefault/deps/c-ares/darwin-x64 -I../deps/c-ares/darwin-x64 ../deps/c-ares/ares_strcasecmp.c -c -o default/deps/c-ares/ares_strcasecmp_1.o

...

[72/74] cxx: src/platform_darwin.cc -> build/default/src/platform_darwin_4.o
/usr/bin/g++ -pthread -arch x86_64 -g -O3 -DHAVE_OPENSSL=1 -DEV_FORK_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_MULTIPLICITY=0 -DX_STACKSIZE=65536 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DEV_MULTIPLICITY=0 -DHAVE_FDATASYNC=0 -DPLATFORM="darwin" -D__POSIX__=1 -Wno-unused-parameter -D_FORTIFY_SOURCE=2 -DNDEBUG -Idefault/src -I../src -Idefault/deps/libeio -I../deps/libeio -Idefault/deps/http_parser -I../deps/http_parser -Idefault/deps/v8/include -I../deps/v8/include -Idefault/deps/libev -I../deps/libev -Idefault/deps/c-ares -I../deps/c-ares -Idefault/deps/c-ares/darwin-x64 -I../deps/c-ares/darwin-x64 -Ideps/v8/include ../src/platform_darwin.cc -c -o default/src/platform_darwin_4.o
[73/74] cxx: src/node_crypto.cc -> build/default/src/node_crypto_4.o
/usr/bin/g++ -pthread -arch x86_64 -g -O3 -DHAVE_OPENSSL=1 -DEV_FORK_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_MULTIPLICITY=0 -DX_STACKSIZE=65536 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DEV_MULTIPLICITY=0 -DHAVE_FDATASYNC=0 -DPLATFORM="darwin" -D__POSIX__=1 -Wno-unused-parameter -D_FORTIFY_SOURCE=2 -DNDEBUG -Idefault/src -I../src -Idefault/deps/libeio -I../deps/libeio -Idefault/deps/http_parser -I../deps/http_parser -Idefault/deps/v8/include -I../deps/v8/include -Idefault/deps/libev -I../deps/libev -Idefault/deps/c-ares -I../deps/c-ares -Idefault/deps/c-ares/darwin-x64 -I../deps/c-ares/darwin-x64 -Ideps/v8/include ../src/node_crypto.cc -c -o default/src/node_crypto_4.o
[74/74] cxx_link: build/default/src/node_main_4.o build/default/src/node_4.o build/default/src/node_buffer_4.o build/default/src/node_javascript_4.o build/default/src/node_extensions_4.o build/default/src/node_http_parser_4.o build/default/src/node_net_4.o build/default/src/node_io_watcher_4.o build/default/src/node_constants_4.o build/default/src/node_cares_4.o build/default/src/node_events_4.o build/default/src/node_file_4.o build/default/src/node_signal_watcher_4.o build/default/src/node_stat_watcher_4.o build/default/src/node_timer_4.o build/default/src/node_script_4.o build/default/src/node_os_4.o build/default/src/node_dtrace_4.o build/default/src/node_stdio_4.o build/default/src/node_child_process_4.o build/default/src/platform_darwin_4.o build/default/src/node_crypto_4.o build/default/deps/libeio/eio_1.o build/default/deps/http_parser/http_parser_2.o build/default/deps/libev/ev_1.o build/default/deps/c-ares/ares_strcasecmp_1.o build/default/deps/c-ares/ares_free_string_1.o build/default/deps/c-ares/ares_options_1.o build/default/deps/c-ares/ares_send_1.o build/default/deps/c-ares/ares_parse_txt_reply_1.o build/default/deps/c-ares/ares_parse_ptr_reply_1.o build/default/deps/c-ares/ares_nowarn_1.o build/default/deps/c-ares/ares_search_1.o build/default/deps/c-ares/ares_gethostbyname_1.o build/default/deps/c-ares/ares_getsock_1.o build/default/deps/c-ares/ares__timeval_1.o build/default/deps/c-ares/inet_ntop_1.o build/default/deps/c-ares/ares_parse_a_reply_1.o build/default/deps/c-ares/ares_getopt_1.o build/default/deps/c-ares/ares__close_sockets_1.o build/default/deps/c-ares/ares_expand_string_1.o build/default/deps/c-ares/ares_destroy_1.o build/default/deps/c-ares/ares_cancel_1.o build/default/deps/c-ares/ares_parse_aaaa_reply_1.o build/default/deps/c-ares/ares_parse_ns_reply_1.o build/default/deps/c-ares/ares_version_1.o build/default/deps/c-ares/ares__get_hostent_1.o build/default/deps/c-ares/ares_writev_1.o build/default/deps/c-ares/ares_expand_name_1.o build/default/deps/c-ares/ares_free_hostent_1.o build/default/deps/c-ares/ares_parse_mx_reply_1.o build/default/deps/c-ares/ares_gethostbyaddr_1.o build/default/deps/c-ares/ares_query_1.o build/default/deps/c-ares/ares_data_1.o build/default/deps/c-ares/ares_init_1.o build/default/deps/c-ares/ares_fds_1.o build/default/deps/c-ares/inet_net_pton_1.o build/default/deps/c-ares/ares_strerror_1.o build/default/deps/c-ares/bitncmp_1.o build/default/deps/c-ares/ares_getnameinfo_1.o build/default/deps/c-ares/ares_library_init_1.o build/default/deps/c-ares/ares_mkquery_1.o build/default/deps/c-ares/ares_process_1.o build/default/deps/c-ares/windows_port_1.o build/default/deps/c-ares/ares_parse_srv_reply_1.o build/default/deps/c-ares/ares_llist_1.o build/default/deps/c-ares/ares_timeout_1.o build/default/deps/c-ares/ares__read_line_1.o build/default/deps/c-ares/ares_strdup_1.o -> build/default/node
/usr/bin/g++ default/src/node_main_4.o default/src/node_4.o default/src/node_buffer_4.o default/src/node_javascript_4.o default/src/node_extensions_4.o default/src/node_http_parser_4.o default/src/node_net_4.o default/src/node_io_watcher_4.o default/src/node_constants_4.o default/src/node_cares_4.o default/src/node_events_4.o default/src/node_file_4.o default/src/node_signal_watcher_4.o default/src/node_stat_watcher_4.o default/src/node_timer_4.o default/src/node_script_4.o default/src/node_os_4.o default/src/node_dtrace_4.o default/src/node_stdio_4.o default/src/node_child_process_4.o default/src/platform_darwin_4.o default/src/node_crypto_4.o default/deps/libeio/eio_1.o default/deps/http_parser/http_parser_2.o default/deps/libev/ev_1.o default/deps/c-ares/ares_strcasecmp_1.o default/deps/c-ares/ares_free_string_1.o default/deps/c-ares/ares_options_1.o default/deps/c-ares/ares_send_1.o default/deps/c-ares/ares_parse_txt_reply_1.o default/deps/c-ares/ares_parse_ptr_reply_1.o default/deps/c-ares/ares_nowarn_1.o default/deps/c-ares/ares_search_1.o default/deps/c-ares/ares_gethostbyname_1.o default/deps/c-ares/ares_getsock_1.o default/deps/c-ares/ares__timeval_1.o default/deps/c-ares/inet_ntop_1.o default/deps/c-ares/ares_parse_a_reply_1.o default/deps/c-ares/ares_getopt_1.o default/deps/c-ares/ares__close_sockets_1.o default/deps/c-ares/ares_expand_string_1.o default/deps/c-ares/ares_destroy_1.o default/deps/c-ares/ares_cancel_1.o default/deps/c-ares/ares_parse_aaaa_reply_1.o default/deps/c-ares/ares_parse_ns_reply_1.o default/deps/c-ares/ares_version_1.o default/deps/c-ares/ares__get_hostent_1.o default/deps/c-ares/ares_writev_1.o default/deps/c-ares/ares_expand_name_1.o default/deps/c-ares/ares_free_hostent_1.o default/deps/c-ares/ares_parse_mx_reply_1.o default/deps/c-ares/ares_gethostbyaddr_1.o default/deps/c-ares/ares_query_1.o default/deps/c-ares/ares_data_1.o default/deps/c-ares/ares_init_1.o default/deps/c-ares/ares_fds_1.o default/deps/c-ares/inet_net_pton_1.o default/deps/c-ares/ares_strerror_1.o default/deps/c-ares/bitncmp_1.o default/deps/c-ares/ares_getnameinfo_1.o default/deps/c-ares/ares_library_init_1.o default/deps/c-ares/ares_mkquery_1.o default/deps/c-ares/ares_process_1.o default/deps/c-ares/windows_port_1.o default/deps/c-ares/ares_parse_srv_reply_1.o default/deps/c-ares/ares_llist_1.o default/deps/c-ares/ares_timeout_1.o default/deps/c-ares/ares__read_line_1.o default/deps/c-ares/ares_strdup_1.o -o /Users/croucher/node-v0.4.1/build/default/node -pthread -framework Carbon -arch x86_64 -rdynamic /Users/croucher/node-v0.4.1/build/default/libv8.a -L/usr/lib -L/usr/local/lib -L/opt/local/lib -L/usr/sfw/lib -lssl -lcrypto -ldl -lutil
ld: warning: directory '/usr/sfw/lib' following -L not found
Waf: Leaving directory `/Users/croucher/node-v0.4.1/build'
'build' finished successfully (5m22.040s)
Enki:~/node-v0.4.1 $

The final step is to use make to install Node. First I'm going to show how to install Node globally for the whole system. This requires you either to have access to the root user or to have sudo privileges that let you act as root.

Example 5-5. Installing Node for the whole system

Enki:~/node-v0.4.1 $ sudo make install
Password:
Waf: Entering directory `/Users/croucher/node-v0.4.1/build'
DEST_OS: darwin
DEST_CPU: x64
Parallel Jobs: 1
Product type: program
* installing deps/libeio/eio.h as /usr/local/include/node/eio.h
* installing deps/v8/include/v8-debug.h as /usr/local/include/node/v8-debug.h

...

* installing build/default/src/node_config.h as /usr/local/include/node/node_config.h
Waf: Leaving directory `/Users/croucher/node-v0.4.1/build'
'install' finished successfully (0.322s)
Enki:~/node-v0.4.1 $

If you want to install only for the local user, and avoid using the sudo command, you need to run the configure script with the --prefix argument in to tell Node to install somewhere other than the default.

Example 5-6. Installing Node for a local user

Enki:~/Downloads/node-v0.4.1 $ mkdir ~/local
Enki:~/Downloads/node-v0.4.1 $ ./configure --prefix=~/local
Checking for program g++ or c++          : /usr/bin/g++
Checking for program cpp                 : /usr/bin/cpp

...

Checking for function nanosleep          : yes
Checking for function ceil               : yes
Checking for fdatasync(2) with c++       : no
'configure' finished successfully (3.248s)
Enki:~/Downloads/node-v0.4.1 $ make
Waf: Entering directory `/Users/croucher/Downloads/node-v0.4.1/build'
DEST_OS: darwin
DEST_CPU: x86
Parallel Jobs: 1

...

Waf: Leaving directory `/Users/croucher/Downloads/node-v0.4.1/build'
'build' finished successfully (5.943s)
Enki:~/Downloads/node-v0.4.1 $ make install
Waf: Entering directory `/Users/croucher/Downloads/node-v0.4.1/build'
DEST_OS: darwin
DEST_CPU: x86
Parallel Jobs: 1
* installing deps/libeio/eio.h as /Users/croucher/local/include/node/eio.h
* installing deps/v8/include/v8-debug.h as /Users/croucher/local/include/node/v8-debug.h

...

* installing build/default/node as /Users/croucher/local/bin/node
* installing build/default/src/node_config.h as /Users/croucher/local/include/node/node_config.h
Waf: Leaving directory `/Users/croucher/Downloads/node-v0.4.1/build'
'install' finished successfully (0.253s)
Enki:~/Downloads/node-v0.4.1 $ cd ~/local
Enki:~/local $ ls
bin     include lib     share
Enki:~/local $ cd bin
Enki:~/local/bin $ ls
node      node-repl node-waf
Enki:~/local/bin $

First Steps in Code

Node REPL

One of the things that's often hard to unerstand about Node.js is that, in addition to being a server, it's also a runtime environment in the same way that Perl, Python and Ruby are. As such, while we often refer to Node.js as "server-side JavaScript," that doesn't really accurately describe what Node.js does. One of the best ways to get to grips with Node.js is to use Node REPL ("Read-Evaluate-Print-Loop"), an interactive Node.js programming environment. It's great for testing out and learning about Node.js. You can try out any of the snippets in this book using Node REPL. In addition, because Node is a wrapper around V8, Node REPL is an ideal place to easily try out JavaScript.

Let's launch Node REPL and try out a few bits of JavaScript to warm up. Open up a console on your system. The system I'm using is a Mac with a custom command prompt, so your system might look a little different, but the commands should be the same:

Example 5-7. Starting Node REPL and trying some JavaScript

$Enki:~ $ node
> 3 > 2 > 1
false
> true == 1
true
> true === 1
false

Note

The first line, which evaluates to false, is from http://wtfjs.com, a collection of weird and amusing things about JavaScript. If you want to see more weird things about JavaScript from wtfjs, with explanations, see Appendix B.

Having a live programming environment is a really great learning tool, but you should know a few helpful features of Node REPL to make the most of it. It offers meta-commands, which all start with a period (.). Thus, .help shows the help menu, .clear clears the current context, and .exit quits Node REPL. The most useful command is .clear, which wipes out any variables or closures you have in memory without the need to restart the REPL.

Example 5-8. Using the meta-features in Node REPL

> console.log('Hello World');
Hello World
> .help
.clear  Break, and also clear the local context.
.exit   Exit the prompt
.help   Show repl options
> .clear
Clearing context...
> .exit
Enki:~ $

When using REPL, simply typing the name of a variable will enumerate it in the shell. Node tries to do this intelligently so a complex object won't just be represented as simple Object, but actually through a description that reflects what's in the object. The main exception to this involves functions. It's not that REPL doesn't have a way to enumerate functions, it's that functions have the tendency to be very large. If REPL enumerated functions, a lot of output could scroll by.

Example 5-9. Setting and enumerating objects with REPL

Enki:~ $ node
> myObj = {};
{}
> myObj.list = ["a", "b", "c"];
[ 'a', 'b', 'c' ]
> myObj.doThat = function(first, second, third) { console.log(first); };
[Function]
> myObj
{ list: [ 'a', 'b', 'c' ]
, doThat: [Function]
}
>

A First Server

While REPL gives us a great tool for learning and experimentation, the main application of Node.js is as a server. One of the specific design goals of Node.js is to provide a highly scalable server environment. This is an area where Node differs from V8, which I described in Chapter 1. Although the V8 runtime is used in Node.js to interpret the JavaScript, Node.js also uses a number of highly optimized libraries to make the server efficient. In particular, the HTTP module was written from scratch in C to provide a very fast non-blocking implementation of HTTP. Let's take a look at the canonical Node "Hello World" example using an HTTP server.

Example 5-10. A Hello World Node.js Web Server

var http = require('http');
http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('Hello World\n');
}).listen(8124, "127.0.0.1");
console.log('Server running at http://127.0.0.1:8124/');

The first thing that this code does is use require to include the HTTP library into the program. This concept is used in many languages, but Node uses the CommonJS module format, which we'll talk about more later in the chapter. The main thing to know at this point is that the functionality in the HTTP library is now assigned to the http object.

Next we need an HTTP server. Unlike some languages such as PHP, which run inside a server such as Apache, Node itself acts as the Web server. However, that also means we have to create it. The next line calls a factory method from the HTTP module that creates new HTTP servers. The new HTTP server isn't assigned to a variable, it's simply going to be an anonymous object in the global scope. Instead we use chaining to initialize the server and tell it to listen on port 8124.

When calling createServer, we passed an anonymous function as an argument. This function is attached to the new server's event listener for the request event. Events are central to both JavaScript and Node. In this case, whenever there is a new request to the Web server, it will call the method we've passed to deal with the request. We call these kinds of methods callbacks. That's because whenever an event happens, we "call back" all the methods listening for that event.

Perhaps a good analogy would be ordering a book from a bookshop. When your book is in stock they call back to let you know you can come and collect it. This specific callback takes the arguments req for the request object and res for the response object.

Inside the function we created for the callback, we call a couple of methods on the res object. These calls modify the response. This example doesn't use the request, but typically you would use both the request and response objects.

The first thing we must do is set the HTTP response header. We can't send any actual response to the client without it. the res.writeHead method does this. We set the values 200 (for the HTTP status code 200 OK) and pass a list of HTTP headers. In this case the only header we specify is Content-type.

After we've written the HTTP header to the client, we can write the HTTP body. In this case we use a single method to both write the body and close the connection. The end method closes the HTTP connection, but since we also passed it a string, it will send that to the client before it closes the connection.

Finally, the last line of our example uses the console.log. This simply prints information to STDOUT, much like the browser counterpart supported by Firebug and Web Inspector.

Let's run this with Node.js on the console and see what we get:

Example 5-11. Running the Hello World example

Enki:~ $ node
node> var http = require('http');
node> http.createServer(function (req, res) {
...     res.writeHead(200, {'Content-Type': 'text/plain'});
...     res.end('Hello World\n');
...   }).listen(8124, "127.0.0.1");
node> console.log('Server running at http://127.0.0.1:8124/');
Server running at http://127.0.0.1:8124/
node>

Here I start a Node REPL and type in the code from the sample (I'll forgive you for pasting it from the web site). Node REPL accepts the code using ... to indicate that you haven't completed the statement and should continue entering it. When we run the console.log line, Node REPL prints out Server running at http://127.0.0.1:8124/. Now we are ready to call our Hello World example in a web browser (Figure 5-2).

Figure 5-2. Viewing the Hello World Web Server from a browser

It works! While this isn't exactly a stunning demo, it is notable that we got "Hello World" working in 6 lines of code. Not that I would recommend that style of coding, but we are starting to get some where.

Chapter 6. Core APIs

Events

The first API we are going to look at is the Events API. This is because, while abstract, it is a fundamental piece of making every other API work. By having a good grip on this API, you'll be able to use all the other APIs effectively.

If you've ever programmed JavaScript in the browser you'll have used events before. However, the event model used in the browser comes from the DOM rather than JavaScript itself. A lot of the concepts in the DOM don't necessarily make sense out of that context. Let's look at the DOM model of events and compare it to the implementation in Node.

The DOM has a user-driven event model based on user interaction with a set of interface elements arranged in a tree structure (HTML, XML, etc). This means when a user interacts with a particular part of the interface there is an event and a context, which is the HTML/XML element on which the click or other activity took place. That context has a parent and potentially children. Since the context is within a tree, the model includes the concepts of bubbling and capturing. This allows elements either up or down the tree to receive the event that was called.

For example, in an HTML list, a click event on an <li> can be captured by a listening on the <ul> that is its parent. Conversely a click on the <ul> can be bubbled down to a listener on the <li>. Since JavaScript objects don't have this kind of tree structure the model in Node is much simpler.

EventEmitter

Because the event model is tied to the DOM in browsers, Node created the EventEmitter class in order to provide some basic event functionality. All event functionality in Node revolves around EventEmitter, because it is also designed to be an interface class for other classes to extend. It would be unusual to call an EventEmiter instance directly.

EventEmitter has a handful of methods, the main two being on and emit. The class provides these methods to other classes to use. The on method creates an event listener for an event:

Example 6-1. Listening for an event with the on method

server.on('event', function(a, b, c) {
  //do things
});

The on method takes two parameters: the name of the event to listen for and the function to call when that event is emitted. Since EventEmitter is an interface pseudo-class the class that inherits from EventEmitter is expected to be innovated with the new keyword. Let's look at how we create a new class as a listener.

Example 6-2. Creating a new class that supports events with EventEmitter

var utils = require('utils'),
    EventEmitter = require('events').EventEmitter;

var Server = function() {
  console.log('init');
};

utils.inherits(Server, EventEmitter);

var s = new Server();

s.on('abc', function() {
  console.log('abc');
});

This example begins by including the utils module so we can use the inherits method. We are going to look at utils in detail later in the chapter, so don't worry too much about how inherits works right now.

We then include the events module. However, we want to access just the specific EventEmitter class inside that module. Note how EventEmitter is capitalized to show it is a class. We didn't use a createEventEmitter method because we aren't planning to use an EventEmitter directly. We simply want to attach its methods to the server class we are going to make.

Once we have included the modules we need, the next step is to create our basic Server class. This offers just one simple function, which logs a message when it is initialized. In a real implementation, we would decorate the Server class prototype with the functions that the class would use. For the sake of simplicity, we've skipped that. The important step is to use sys.inherits to add EventEmitter as a superclass of our Server class.

When we want to use the Server class, we instantiate it with new Server(). This instance of Server will have access to the methods in the superclass (EventEmitter). That means we can add a listener to our instance using the on method.

Right now, however, the event listener we added will never be called, because the abc event isn't fired. We can fix this by adding some code to emit the event.

Example 6-3. Emitting an event

s.emit('abc');

Firing the event listener is as simple as calling the emit method that the Server instance inherited from EventEmitter. It's important to note that these events are instance based. There are no global events. When you call the on method, you attach to a specific EventEmitter-based object. Even the various instances of the Server class don't share events. s from the code sample will not share the same events as another Server instance, such as created by var z = new Server();.

Callback syntax

An important part of using events is dealing with callbacks. Chapter 4 looks at best practices in much more depth, but we'll look here at the mechanics of callbacks in Node. They use a few standard patterns, but first let's discuss what is possible.

Example 6-4. Passing parameters when emitting an event

s.emit('abc', a, b, c);

When calling emit, in addition to the event name, you can also pass an arbitrary list of parameters. The previous example included three such parameters. These will be passed to the function listening to the event. When you receive a request event from the http Server, for example, you receive two parameters: req and res. When the request event was emitted, those parameters were passed as the second and third arguments to the emit.

It is important to understand how Node calls the event listeners, because it will affect your programming style. When emit() is called, the following code is used to call each event listener.

Example 6-5. Calling event listeners from emit

if (arguments.length <= 3) {
  // fast case
  handler.call(this, arguments[1], arguments[2]);
} else {
  // slower
  var args = Array.prototype.slice.call(arguments, 1);
  handler.apply(this, args);
}

This code uses both of the JavaScript methods for calling a function from code. If emit() was passed with three or fewer arguments, the method takes a shortcut and uses call. Otherwise it use the slower apply to pass all the arguments as an array. The important thing to recognize here, though, is that Node makes both of these calls using the this argument directly. This means that the context in which the event listeners being called is the context of EventEmitter耒not their original context. Using Node REPL, you can see what is happening when things get called EventEmitter.

Example 6-6. The changes in context caused by EventEmitter

> var EventEmitter = require('events').EventEmitter,
...     util = require('util');
>
> var Server = function() {};
> util.inherits(Server, EventEmitter);
> Server.prototype.outputThis= function(output) {
...   console.log(this);
...   console.log(output);
... };
[Function]
>
> Server.prototype.emitOutput = function(input) {
...   this.emit('output', input);
... };
[Function]
>
> Server.prototype.callEmitOutput = function() {
...   this.emitOutput('innerEmitOutput');
... };
[Function]
>
> var s = new Server();
> s.on('output', s.outputThis);
{ _events: { output: [Function] } }
> s.emitOutput('outerEmitOutput');
{ _events: { output: [Function] } }
outerEmitOutput
> s.callEmitOutput();
{ _events: { output: [Function] } }
innerEmitOutput
> s.emit('output', 'Direct');
{ _events: { output: [Function] } }
Direct
true
>

The sample output first sets up a Server class. It includes functions to emit the output event. The outputThis method is attached to output event as an event listener. When we emit the output event from various contexts, we stay within the scope of the EventEmitter object, so the value of this that s.outputThis has access to is the one belonging to the EventEmitter. This means the this variable must be passed in as a parameter and assigned to, if we wish to create event listeners that make use of it.

HTTP

One of the core tasks of Node.js is to act as a web server. This is such a key part of the system that when Ryan Dahl started the project, he rewrote the HTTP stack for V8 to make it non-blocking. Although both the API and the internals for the original HTTP implementation has morphed a lot since it was created, the core activities are still the same. The Node implementation of HTTP is non-blocking and fast. Much of the code has moved from C into JavaScript.

HTTP uses a pattern that is common in Node. Pseudo-class[3] factories provide an easy way to create a new server. The http.createServer() method provides us with a new instance of the HTTP Server class. It is this server class that we use to define the actions taken when Node receives incoming HTTP requests. There are a few other main pieces of the HTTP module, and other Node modules in general. These are the events the Server class fires and the data structures that are passed to the callbacks. Knowing about these three types of Class allow you use the HTTP module well.

HTTP Servers

Acting as an HTTP server is probably the commonest current use case for Node. In the previous chapter, we set up an HTTP server and used it to serve a very simple request. However, HTTP is a lot more multifaceted than that. The server component of the HTTP module provides the raw tools to build complex and comprehensive Web servers. In this chapter, we are going to explore the mechanics of dealing with requests and issuing responses. Even if you end up using a higher level server such as Express, many of the concepts it uses are extensions of those defined here.

As we've already, seen the first step in using HTTP servers is to create a new server using the http.createServer() method. This returns a new instance of a the Server class. This Class has only a few methods, because most of the functionality is going to be provided using events. The http server Class has six events and three methods. The other thing to notice is how most of the methods are used to initialize the server, whereas events are used during its operation.

Let's start by creating the smallest basic HTTP server code we can.

Example 6-7. A simple, but very short, HTTP server

require('http').createServer(function(req,res){res.writeHead(200, {}); res.end('hello world');}).listen(8125);

This example is not good code. However, it illustrates some important points. We'll fix the style shortly. The first thing we do is require the http module. Notice how I can chain methods in order to access the module without first assigning it to a variable. Many things in Node return a function,[4] which allows us to invoke those functions immediately. From the included http module we call createServer. This doesn't have to take any arguments, but we pass it a function to attach to the request event. Finally, we tell the server created with createServer to listen on port 8125.

I hope you never write code like this in real situations, but it does show the flexibility of the syntax and the potential brevity of the language. Let's be a lot more explicit about our code. The following rewrite should make it a lot easier to understand and maintain.

Example 6-8. A simple, but more descriptive, HTTP server

var http = require('http');
var server = http.createServer();
var handleReq = function(req,res){
  res.writeHead(200, {});
  res.end('hello world');
};
server.on('request', handleReq);
server.listen(8125);

This example implements the minimal web server again. However, we've started assigning things to named variables. This not only makes the code easier to read than when it's chained, but also means you can reuse it. For example, it's not uncommon to use http more than once in a file. You want to have both an HTTP server and an HTTP client, so reusing the module object is really helpful. While JavaScript doesn't force you to think about memory, it doesn't mean you should thoughtlessly litter unnecessary objects everywhere. So rather than use an anonymous callback, I've named the function that handles the request event. This is less about memory usage and more about readability. I'm not saying you shouldn't use anonymous functions, but if you can lay out your code so it's easy to find, that helps a lot when maintaining it.

Note

Remember to look at Part I of the book for more help with programming style. Chapters 3 and 4 deal with programming style in particular.

Because we didn't pass the request event listener as part of the factory method for the http Server object, we need to add an event listener explicitly. Calling the on method from EventEmitter does this. Finally, as with the previous example, we call the listen method with the port we want to listen on. The http Class provides other functions, but this example illustrates the most important ones.

The http server supports a number of events, which are associated with either the TCP or HTTP connection to the client. The connection and close events indicate the build-up or tear-down of a TCP connection to a client. It's important to remember that some clients will be using HTTP 1.1, which supports keep-alive. This means that their TCP connections may remain open across multiple HTTP requests.

The request, checkContinue, upgrade, and clientError events are associated with HTTP requests. We've already used the request event, which signals a new HTTP request.

The checkContinue event indicates a special event. It allows you to take more direct control of an HTTP request in which the client streams chunks of data to the server. As the client sends data to the server, it will check whether it can continue, at which point this event will fire. If an event handler is created for this event, the request event will not be emitted.

The upgrade event is emitted when a client asks for a protocol upgrade. The http server will deny HTTP upgrade requests unless there is an event handler for this event.

Finally, the clientError event passes on error events sent by the client.

The HTTP server can throw a few events. The most common one is request but you can also get events associated with the TCP connection for the request as well as other parts of the request life-cycle.

When a new TCP stream is created for a request, a connection event is emitted. This event passes the TCP stream for the request as a parameter. The stream is also available as a request.connection variable for each request that happens through it. However, only one connection event will be emitted for each stream. This means tat many requests can happen from a client with only one connection event.

HTTP Clients

Node is also great when you want to make outgoing HTTP connections. This is useful in many contexts, such as using Web Services, connecting to document store databases, or just scraping web sites. You can use the same http module when doing HTTP requests, but should use the http.ClientRequest class. There are two factory methods for this class: a general purpose one and a convenience method. Let's take a look at the general purpose case:

Example 6-9. Creating an HTTP request

      var http = require('http');

var opts = {
  host: 'www.google.com'
  port: 80,
  path: '/',
  method: 'GET'
};

var req = http.request(opts, function(res) {
  console.log(res);
  res.on('data', function(data) {
    console.log(data);
  });
});

req.end();

The first thing you can see is that an options object defines a lot of the functionality of the request. We must provide the host name (although an IP address is also acceptable), the port, and the path. The method is optional and defaults to a value of GET if none is specified. In essence, the example is specifying that the request should be an HTTP GET request to http://www.google.com/ on port 80.

The next thing we do is use the options object to construct an instance of http.ClientRequest using the factory method http.request(). This method takes an options object and an optional callback argument. The passed callback listens to the response event. When a response event is received, we can process the results of the request. In the previous example, we simply output the response object to the console. However, it's important to notice that the body of the HTTP request is actually received via a stream in the response object. As such, you can subscribe to the data event of the response object to get the data as it becomes available.

The final important point to notice is that we had to end() the request. Since this was a GET request, we didn't write any data to the server, but for other HTTP methods, such as PUT or POST, you may need to. Until we call the end() method, request won't initiate the HTTP request because it doesn't know whether it should still be waiting for us to send data.

Making HTTP GET requests

Since GET is such a common HTTP use case, there is a special factory method to support it in a more convenient way:

Example 6-10. Simple HTTP GET requests

        var http = require('http');

var opts = {
  host: 'www.google.com'
  port: 80,
  path: '/',
};

var req = http.get(opts, function(res) {
  console.log(res);
  res.on('data', function(data) {
    console.log(data);
  });
});

This example of http.get() does exactly the same thing as the previous example, but it's slightly more concise. We've lost the method attribute of the config object and we didn't have to call request.end() because it's implied.

If you run the previous two examples, you are going to get back raw Buffer objects. As described later in this chapter, a Buffer is a special class defined in Node to support the storage of arbitrary, binary data. Although it's certainly possible to work with these, you often want a specific encoding, such as UTF-8 (an encoding for UNICODE characters). You can specify this with the response.setEncoding() method:

Example 6-11. Comparing raw Buffer output to output with a specified encoding

        > var http = require('http');
> var req = http.get({host:'www.google.com', port:80, path:'/'}, function(res) { console.log(res); res.on('data', function(c) { console.log(c); }); });
> <Buffer 3c 21 64 6f 63 74 79 70

...

65 2e 73 74>
<Buffer 61 72 74 54 69

...

69 70 74 3e>

>
> var req = http.get({host:'www.google.com', port:80, path:'/'}, function(res) { res.setEncoding('utf8'); res.on('data', function(c) { console.log(c); }); });
> <!doctype html><html><head><meta http-equiv="content-type

...

load.t.prt=(f=(new Date).getTime());
})();
</script>

>

In the first case, we do not pass ClientResponse.setEncoding(), and we get chunks of data in Buffers. Although the output is abridged in the previous print-out, you can see that it isn't just a single Buffer, but that several Buffers have been returned with data. In the second example, the data is returned as UTF-8 because we specified res.setEncoding('utf8'). The chunks of data returned from the server are still the same, but are given to the program as strings in the correct encoding rather than as raw Buffers. Although the print-out may not make this clear, there is one string for each of the original Buffers.

Uploading Data for HTTP POST and PUT

Not all HTTP is GET. You might also need to call POST, PUT, and other HTTP methods that alter data on the other end. This is functionally the same as making a GET request, except you are going to write some data upstream:

Example 6-12. Writing data to an upstream service

        var options = {
  host: 'www.example.com',
  port: 80,
  path: '/submit',
  method: 'POST'
};

var req = http.request(options, function(res) {
  res.setEncoding('utf8');
  res.on('data', function (chunk) {
    console.log('BODY: ' + chunk);
  });
});

req.write("my data");
req.write("more of my data");

req.end();

This example is very similar to the previous example, but uses the http.ClientRequest.write() method. This method allows you to send data upstream. And as explained earlier, it requires you to explicitly call http.ClientRequest.end() to indicate you're finished sending data. Whenever ClientRequest.write() is called, the data is sent upstream (it isn't buffered), but the server will not respond until ClientRequest.end() is called.

You can stream data to a server using ClientRequest.write() by coupling the writes to the data event of a Stream. This is ideal if you need to, for example, send a file from disk to a remote server over HTTP.

The ClientResponse Object

The ClientResponse object stores a variety of information about the request. In general, it is pretty intuitive. Obvious things that are often useful include statusCode (which contains the HTTP status) and headers (which is the response header object). Also hung off of ClientResponse are various streams and properties, which you may or may not want to interact with directly.

URL

The URL module provides tools for easily parsing and dealing with URL strings. It's extremely useful if you have to deal with URLs. The module offers three methods: parse, format, and resolve. Let's start by looking at an example of parse using Node REPL.

Example 6-13. Parsing a URL using the URL module

> var URL = require('url');
> var myUrl = "http://www.nodejs.org/some/url/?with=query&param=that&are=awesome#alsoahash";
> myUrl
'http://www.nodejs.org/some/url/?with=query&param=that&are=awesome#alsoahash'
> parsedUrl = URL.parse(myUrl);
{ href: 'http://www.nodejs.org/some/url/?with=query&param=that&are=awesome#alsoahash'
, protocol: 'http:'
, slashes: true
, host: 'www.nodejs.org'
, hostname: 'www.nodejs.org'
, hash: '#alsoahash'
, search: '?with=query&param=that&are=awesome'
, query: 'with=query&param=that&are=awesome'
, pathname: '/some/url/'
}
> parsedUrl = URL.parse(myUrl, true);
{ href: 'http://www.nodejs.org/some/url/?with=query&param=that&are=awesome#alsoahash'
, protocol: 'http:'
, slashes: true
, host: 'www.nodejs.org'
, hostname: 'www.nodejs.org'
, hash: '#alsoahash'
, search: '?with=query&param=that&are=awesome'
, query:
   { with: 'query'
   , param: 'that'
   , are: 'awesome'
   }
, pathname: '/some/url/'
}
>

The first thing we do, of course, is to require the URL module. Note that the names of modules are always lowercase. I've created a url as a string containing all the parts that will be parsed out. Parsing is really easy: we just call the parse method from the URL module on the string. It returns a data structure representing the parts of the parsed URL. The components it produces are:

href

protocol

host

auth

hostname

port

pathname

search

query

hash

The href is the full URL that was originally passed to parse. The protocol is the protocol used in the URL: e.g., http://, https://, ftp://, etc. host is the fully qualified hostname of the URL. This could be as simple as hostname for a local server, such as print server, or a fully qualified domain name such as www.google.com. It might also include a port number, such as 8080, or username and password credentials like un:pw@ftpserver.com. The various parts of the hostname are broken down further into auth, containing just the user credentials, port containing just the port, and hostname containing the hostname portion of the URL. An important thing to know about hostname is that it is still the full hostname including the top level domain (TLD, e.g. .com, .net, etc) and the specific server. If the URL was http://sport.yahoo.com/nhl, hostname would not give you just the TLD (yahoo.com) or just the host sport) but the entire hostname sport.yahoo.com. The url module doesn't have the capability to split the hostname down into its components such as domain or TLD.

The next set of components of the URL relate to everything after the host. The pathname is the entire file path after the host. In http://sports.yahoo.com/nhl, it would be /nhl. The next component is the search component. It stores the HTTP GET parameters in the URL. For example, if the URL was http://mydomain.com/?foo=bar&baz=qux, the search component would be ?foo=bar&baz=qux. Note the inclusion of the ?. The query parameter is similar to the search component. It contains one of two things, depending how parse was called.

parse takes two arguments: the url string and an optional Boolean that determines whether the queryString should be parsed using the querystring module, discussed in the next section. If the second argument is false, query will just contain a string similar to that of search but without the leading ?. If you don't pass anything for the second argument, it defaults to false.

The final component is the fragment portion of the URL. This is the part of the URL after the #. Commonly, this is used to refer to named anchors in HTML pages. For instance, http://abook.com/#chapter2 might refer to the second chapter on a web page hosting a whole book. The hash component in this case would contain #chapter2. Again note the included # in the string. Some sites, such as http://twitter.com, use more complex fragments for AJAX applications, but the same rules apply. So the URL for the Twitter mentions account, http://twitter.com/#!/mentions would have a pathname of / but a hash of #!/mentions.

querystring

The querystring module is a very simple helper module to deal with query strings. As discussed in the previous section, query strings are the parameters encoded at the end of a URL. However, when reported back as just a JavaScript string, the parameters are fiddly to deal with. The querystring module provides an easy way to create objects from the query strings. The main methods it offers are parse and decode, but some internal helper functions such as escape, unescape, unescapeBuffer, encode, and stringify are also exposed. If you have a query string, you can use parse to turn it into an object:

Example 6-14. Parsing a query string with the querystring module in Node-REPL

> var qs = require('querystring');
> qs.parse('a=1&b=2&c=d');
{ a: '1', b: '2', c: 'd' }
>

Here, the class's parse function turns the querystring into an object in which the properties are the keys and the values correspond to the ones in the query string. You should notice a few things, though. First, the numbers are returned as strings, not numbers. Since JavaScript is loosely typed and will coerce a string into a number in a numerical operation, this works pretty well. However, it's worth bearing in mind for those times when that coercion doesn't work.

Another thing it's important to be aware of is that you must pass the query string without the leading ? that demarks it in the URL. A typical URL might look like http://www.bobsdiscount.com/?item=304&location=san+francisco. The querystring starts with a ? to indicate where the filepath ends, but if you include the ? in the string you pass to parse, the first key will start with a ?, which is almost certainly not what you want.

This library is really useful in a bunch of contexts. Query strings are used outside of URLs. When you get content from an HTTP POST that is x-form-encoded, it will also be in query string form. All the browser manufacturers have standardized around this approach. By default, forms in HTML will send data to the server in this way also.

The querystring module is also used as a helper module to the url module. Specifically when decoding URLs you can ask URL to turn the querystring into an object for you rather than just a string. That's described in more detail in the previous section, but the parsing that is done is using the parse method from querystring.

Another important part of querystring is encode. This function takes a queryString key/value pair object and stringifies it. This is really useful when working with HTTP requests, especially POST data. It makes it easy to work with a JavaScript object until you need to send the data over the wire and then simply to encode it at that point. Any JavaScript object can be used however you should ideally use an object which only has the data in it you want because the encode method will add all properties of the object. However if the property value isn't a string, boolean or number won't serialized and the key will just be included with an empty value.

Example 6-15. Encoding an object into a querystring

> var myObj = {'a':1, 'b':5, 'c':'cats', 'func': function(){console.log('dogs')}}
> qs.encode(myObj);
'a=1&b=5&c=cats&func='
>

[3] When I talk about Pseudo-class I am referring to the definition found in Douglas Crockford's JavaScript: The Good Parts (O'Reilly). From now I will use Class to refer to Pseudo-class.

[4] This works in JavaScript because it supports first class functions. See Appendix B for more information

I/O

Streams

Many components in Node provide continuous output or can process continuous input. To make these components act in a consistent way, the stream API provides an abstract interface for them. This API provides common methods and properties that are available in specific implementation of streams. Streams can be readable, writable, or both. All streams are EventEmitter instances, allowing them to emit events.

Readable Streams

The readable stream API is a set of methods and events that provides access to chunks of data as they are sent by an underlying data source. Fundamentally, readable streams are about emitting data events. These events represent the stream of data as a stream of events. In order to make this manageable, streams have a number of features that allow you to configure how much data you get and how fast.

Example 6-16. Creating a readable file stream

var fs = require('fs');
var filehandle = fs.readFile('data.txt', function(err, data) {
  console.log(data)
});

The basic stream in the example simply reads data from a file in chunks. Every time a new chunk is made available, it is exposed to a callback in the variable called data. In this example, we simply log the data to the console. However, in real use cases you might either stream the data somewhere else or spool it into bigger pieces before you work on it. In essence, the data event simply provides access to the data, while you have to figure out what to do with each chunk.

Let's look in more detail at one of the common patterns used in dealing with streams. The spooling pattern is used when we need an entire resource available before we deal with it. We know it's important not to block the event loop for Node to perform well, so even though we don't want to perform the next action on this data until we've received all of it, we don't want to block the event loop. In this scenario we use a stream to get the data but use the data only when enough is available. Typically this means when the stream ends, but it could be another event or condition.

Example 6-17. Using the spooling pattern to read a complete stream

          //abstract stream
var spool = "";
stream.on('data', function(data) {
  spool += data;
});
stream.on('end', function() {
  console.log(spool);
});

FileSystem

Filesystem is obviously a very helpful module because you need it in order to access files on disk. It closely mimics the POSIX style of file I/O. It is a somewhat unique module in that all of the methods have both asynchronous and synchronous versions. However, I strongly recommend that you use the asynchronous methods unless you are building command line scripts with Node. Even then, it is often much better to use the async versions, even though doing so adds a little extra code, so that you can access multiple files in parallel and reduce the running time of your script.

The main issue that people face while dealing with asynchronous calls is ordering, and this is especially true with file I/O. It's common to want to do a number of moves, renames, copies, reads, or writes at one time. However, if one of the operations depends on another then this can create issues because return order is not guaranteed. This means that the first operation in the code happens after the second operation in the code. Patterns exist to deal make ordering easy. They are talked about in detail in chapter 4 but we'll recap here as well.

An example of this might be reading and then deleting a file. If the delete (unlink) happens before the read, it will be impossible to read the content of the file.

Example 6-18. Reading and deleting a file asynchronously耒but all wrong

var fs = require('fs');

fs.readFile('warandpeace.txt', function(e, data) {
  console.log('War and Peace: ' + data);
});

fs.unlink('warandpeace.txt');

Notice that we are using the asynchronous methods, and while we have created callbacks we haven't written any code that defines which order they get called in. This often becomes a problem for programmers not used to programming in event loops. This code looks OK on the surface and sometimes it will work at runtime, but sometimes it won't. Instead, we need to use a pattern in which we specify the ordering we want for the calls. There are a few approaches. One common approach is to use nested callbacks. In the following example the asynchronous call to delete the file is nested within the callback to the asynchronous function that reads the file.

Example 6-19. Reading and deleting a file asynchronously using nested callbacks

var fs = require('fs');

fs.readFile('warandpeace.txt', function(e, data) {
  console.log('War and Peace: ' + data);
  fs.unlink('warandpeace.txt');
});

This approach is often a very effective one for discrete sets of operations. In our example with just two operations, it's easy to read and understand. This pattern can get out of control though. Another pattern to consider is

Buffers

Although Node is JavaScript, it is JavaScript out its usual environment. For instance, the browser requires JavaScript to perform many functions, but manipulating binary data is rarely one of them. While JavaScript does support bitwise operations, it doesn't have a native representation of binary data. This is especially troublesome when one also considers the limitations of the number type system in JavaScript, which might otherwise lend itself to binary representation. Node introduces the Buffer class to make up for this shortfall when working with binary data is often essential.

Buffers are an extension to the V8 engine, which means that they have their own set of pitfalls. Buffers are actually a direct allocation of memory, which may mean a little or a lot depending on your experience with lower level computer languages. Unlike the data types in JavaScript, which abstract some of the ugliness of storing data, Buffer provides specific functionality warts and all. Once a Buffer is created, it is a fixed size. If you want to add more data, you must clone the Buffer into a larger Buffer. Although some of these features may seem frustrating, they allow Buffer to perform at the speed necessary for many data operations on the server. This was a conscious design choice, to trade off some programmer convenience for performance.

A quick primer on binary

I thought it was important to include this quick primer on working with binary data for those who haven't done much of it, or as a refresher for those of us that hadn't in a long time (which was true for me when I started working with Node). Computers, as almost everyone knows work by manipulating states of 'on' and 'off'. We call this a binary state, because there are only two possibilities. Everything thing in computers is built on top of this, which means that working directly binary can often be the fastest method on the computer. In order to do more complex things, we collect "bits" (each representing a single binary state) into groups of eights, often called an octet or more commonly byte[5]. This allows us to represent bigger numbers than just 0 or 1.

By creating sets of 8 bits we are able to represent any number from 0 to 255. The rightmost bit represents one, but then we double the value of the number represented by each bit as we move left. In order to find out what number it represents we simply sum the numbers in column headers (Figure 6-1).

128 64 32 16 8 4 2 1
--- -- -- -- - - - -
0   0  0  0  0 0 0 0 = 0

128 64 32 16 8 4 2 1
--- -- -- -- - - - -
1   1  1  1  1 1 1 1 = 255

128 64 32 16 8 4 2 1
--- -- -- -- - - - -
1   0  0  1  0 1 0 1 = 149

Figure 6-1. Representing 0 through 255 in a byte

You'll also see the use of hexadecimal notation or "hex" a lot. Since bytes need to be easily described and a string of eight 0s and 1s isn't very convenient, hex notation has become popular. Binary notation is base 2, in that there are only two possible states per digit (0 or 1). Hex uses bases 16, and each digit in hex can have a value from 0 to F, where the letters A through F (or their lowercase equivalents stand for 10 through 15 respectively. What's very convenient about hex is that with two digits we can represent a whole byte. The right digit represents 1s and the left digit represents 16s. If we wanted to represent decimal 149 it is (16 x 9) + (5 x 1) or the hex 95.

Hex to Decimal:

0 1 2 3 4 5 6 7 8 9 A  B  C  D  E  F
- - - - - - - - - - -- -- -- -- -- --
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Counting in hex:

16 1
-- -
0  0 = 0

16 1
-- -
F  F = 255

16 1
-- -
9  5 = 149

Figure 6-2. Representing 0 through 255 with hex notation

In JavaScript, you can create a number from a hex value using the notation 0x in front of the hex value. For instance, 0x95 is decimal 149. More commonly, in Node, you'll see Buffers represented by hex values in console.log() output or Node-REPL:

Example 6-20. A creating a 3 byte buffer from an array of octets

> new Buffer([255,0,149]);
<Buffer ff 00 95>
>

So how does binary relate to other kinds of data? Well we've seen how binary can represent numbers. In network protocols, it's common to specify a certain number of bytes to convey some information, using particular bits in fixed places to indicate specific things. For example, when constructing a DNS request, the first 2 bytes are used as a number for a transaction ID, while the next byte is treated as individual bits, each used to indicate whether a specific feature of DNS is being used in this request.

The other extremely common use of binary is to represent strings. The two most common 'encoding' formats for strings are ASCII and UTF (typically UTF-8). These encodings define how the bits should be converted into characters. I'm not going to go into too much of the gory detail. Essentially, encodings work by having a look-up table of the character to a specific number represented in bytes. In order to convert the encoding, the computer has to simply convert from the number to the character by looking it up in a conversation table.

ASCII characters (some of which are non-visible "control characters", such as Return) are always exactly seven bits each, so they can be represented by the values from 0 to 127. The eight bit in a byte is often used to extend the character set to represent various choices of international characters (such as ȳ or ȱ).

UTF is a little more complex. Its character set has a lot more characters, including many international ones. Each character in UTF-8 is represented by at least 1 byte, but sometimes up to 4. Essentially, the first 128 values are good old ASCII, whereas the others are pushed further down in the map and represented by higher numbers. When a less common character is referenced, the first byte uses a number that tells the computer to check out the next byte to find the real address of the character starting on the second sheet of its map. If the character isn't on the second sheet of the map, the second byte tells the computer to look at the third, and so on. This means that in UTF-8, the length of a string measured in characters isn't necessarily the same as its length in bytes, as is always true with ASCII.

Binary and Strings

It is important to remember is that once you copy things to a Buffer, they will be stored as their binary representations. You can always convert the binary representation in the buffer back into other things such as strings later. So a Buffer is defined only by its size, not by the encoding or any other indication of its meaning.

Given that Buffer is opaque, how big does it ned to be in order to store a particular string of input? As we've said, a UTF character can occupy up to 4 bytes, so to be safe, you should define a Buffer to be 4 times the size of the largest input you can accept, measured in UTF characters. There may be ways you can reduce this burden; for instance, if you limit your input to European languages, you'll know there will be at most 2 bytes per character.

Using Buffers

Buffers can be created using three possible parameters: the length of the buffer in bytes, an array of bytes to copy into the buffer, or a string to copy into the buffer. The first and last methods are by far the most common. There aren't too many instances where you are likely to have a JavaScript array of bytes.[6]

Creating a Buffer of a particular size is a very common scenario and easy to deal with. Simply put, you specify the number of bytes as your argument when creating the Buffer.

Example 6-21. Creating a Buffer using byte length

> new Buffer(10);
<Buffer e1 43 17 05 01 00 00 00 41 90>
>

As you can see from the previous example, when we create a Buffer we get a matching number of bytes. However, since the Buffer is just getting an allocation of memory directly, it is uninitialized and the contents are left over from whatever happened to occupy them before. This is unlike all the native JavaScript types, which initialize all memory so that when you create a new primitive or object it doesn't assign whatever was already in the memory space to the primitive or object you just created. Here is a good way to think about it. If you go to a busy cafe and you want a table, the fastest way to get one would be to sit down as soon as some other people vacate one. However, while it's fast, you are left with all their dirty dishes and the detritus from their meals. You might prefer to wait for one of the staff to clear the table and wipe it down before you sit. This is a lot like Buffers versus native types. Buffers do very little to make things easy for you, but they do give you direct access to memory, fast. If you want to have a nicely zeroed set of bits, you'll need to do it yourself (or find a helper library).

Creating a Buffer using byte length is most common when you are working with things like network transport protocols that have very specifically defined structures. When you know exactly how big the data is going to be or you know exactly how big it could be and you want to allocate and reuse a Buffer for performance reasons, this is the way to go.

Probably the most common way to use a Buffer is to create it with a string of either ASCII or UTF-8 characters. Although a Buffer can hold any data, it is particularly useful for I/O with character data, because the constraints we've already seen on Buffer can make their operations much faster than operations on regular strings. So when you are building really highly scalable apps it's often worth using Buffers to hold strings. This is especially true if you are just shunting the strings around the application without modifying them. Therefore, even though strings exist as primitives in JavaScript, it's still very common to keep strings in Buffers in Node.

Example 6-22. Creating Buffers using strings

> new Buffer('foobarbaz');
<Buffer 66 6f 6f 62 61 72 62 61 7a>
> new Buffer('foobarbaz', 'ascii');
<Buffer 66 6f 6f 62 61 72 62 61 7a>
> new Buffer('foobarbaz', 'utf8');
<Buffer 66 6f 6f 62 61 72 62 61 7a>
> new Buffer('é');
<Buffer c3 a9>
> new Buffer('é', 'utf8');
<Buffer c3 a9>
> new Buffer('é', 'ascii');
<Buffer e9>
>

When we create a Buffer with a string, it defaults to UTF-8. That is, if you don't specify an encoding, it will be considered a UTF-8 string. That is not to say Buffer pads the string to fit any UNICODE character (blindly allocating 4 bytes per character), rather that it will not truncate characters. In the previous example, we can see that when taking a string with just lowercase alpha characters, the Buffer uses the same byte structure whatever the encoding because they all fall in the same range. However, when we have an é it's encoded as 2 bytes in the default UTF-8 case or if we specify UTF-8 explicitly. If we specify ASCII, the character gets truncated to a single byte.

Working with Strings

Node offers a number of operations to simplify working with strings and Buffers. First, you don't need to compute the length of a string before creating a Buffer to hold it, just assign the string as the argument when creating the Buffer. Alternatively, you can use the Buffer.byteLength() method. This method takes a string and an encoding and returns the length in bytes of that string, rather than the length in characters as String.length does.

You can also write a string to an existing Buffer. The Buffer.write() method writes a string to a specific index of a Buffer. If there is room in the Buffer starting from the specified offset, the entire string will be written. Otherwise, characters are truncated from the end of the string to fit the buffer. In either case, Buffer.write() will return the number of bytes that were written. In the case of UTF-8 strings, if a whole character can't be written to the Buffer, none of the bytes for that character will be written. In the following example, because the buffer is too small for even one non-ASCII character, it ends up empty.

Example 6-23. Buffer.write() and partial characters

          > var b = new Buffer(1);
> b
<Buffer 00>
> b.write('a');
1
> b
<Buffer 61>
> b.write('é');
0
> b
<Buffer 61>
>

In a single-byte Buffer it's possible to write an 'a' character, and doing so returns 1 indicating that 1 byte was written. However trying to write a 'é' character fails because it requires 2 bytes and the method returns a 0 because nothing was written.

There is a little more complexity to Buffer.write(), though. If possible, when writing UTF-8, Buffer.write() will terminate the character string with a NUL character.[7] This is much more significant when writing into the middle of a larger Buffer.

Example 6-24. Writing a string into a Buffer including a terminator

          > var b = new Buffer(5);
> b.write('fffff');
5
> b
<Buffer 66 66 66 66 66>
> b.write('ab', 1);
2
> b
<Buffer 66 61 62 00 66>
>

After creating a Buffer 5 bytes long (which could have been done directly using the string), we write the character f to the entire Buffer. f is the character code 0x66 (102 in decimal). This makes it easy to see what happens when we write the characters 'ab' to the Buffer starting with an offset of 1. The zeroeth character is left as f. At positions 1 and 2, the characters themselves are written, 61 followed by 62. Then Buffer.write() inserts a terminator, in this case a null character of 0x00.

Console.log

Borrowed from the Firebug debugger in Firefox, this simple command allows you to easily output to STDOUT without using any modules. It also does some pretty printing functionality to help enumerate objects.

Example 6-25. Outputting with console.log

> foo = {};
{}
> foo.bar = function() {1+1};
[Function]
> console.log(foo);
{ bar: [Function] }
>

[5] There is no "standard" size of byte, but the de facto size that virtually everyone uses nowadays is 8 bits. As such, octets and bytes are equivalent, and I'll be using the more common term byte to mean specifically an octet.

[6] It's very memory-inefficient, for one thing. If you store each byte as a number, for instance, you are using a 64-bit memory space to represent 8 bits.

[7] This generally just means a binary 0

Chapter 7. Helper APIs

Helpers

DNS

It is necessary to be able to refer to things by their domain names rather than their IPs. The DNS module provides this look up facility to you, but it is also used under the hood whenever you are able to use a domain name, for example in HTTP clients.

There are two main methods of dns and a number of convenience methods. The two primary method are resolve() and reverse(). These turn a domain into DNS records and IP addresses into domains. All of the other methods in the dns module are more specialised forms of these methods.

dns.resolve() takes three arguments the domain to be resolved the type of records being requested and a callback. The domain is pretty obvious, although it should be noted that 'domain' can include hostnames. For example 'www.yahoo.com' is what we consider a domain in this context. 'www' is technically a hostname but the system will resolve it for you automatically. Simply put, just request the domain you want and the API will figure out the rest. The record type requires a little more understanding of DNS. Most people are familiar with the 'address' or 'A' record type. This type of record maps an IPv4 domain to a domain name (as defined above). There are a few other types, 'canonical name' or 'CNAME' records allow you to create an alias of an A record or another CNAME. For example, 'www.example.com' might be a CNAME of the A record at 'example.com'. 'MX' records point to the mail server for a domain for the purposes of SMTP. When you email person@domain.com then the MX record for 'domain.com' tells your email server where to send their mail. Text records or 'TXT' are notes attached to domain. They have been used for all kinds of functions. The final type supported by this library is the Service or 'SRV' records. These records provide information on services that are available at a particular domain.

Calling dns.resolve() is easy, although the callback may be slightly different to other callbacks you've have used so far.

Example 7-1. Calling dns.resolve()

dns.resolve('yahoo.com', 'A', function(e,r) {
  if (e) {
    console.log(e);
  }
  console.log(r);
} );

We simply called dns.resolve() with the domain and a record type of 'A' we also passed a callback. Notice the first arguemnt of the callback is actually an error. If an error occurs we get an error object we can use to see what went wrong. If there isn't an error then the object will be null instead. The second argument is the record(s) returned by the query. All of the types of record listed above have shorthands in the API. For example, rather than calling resolve('example.com', 'MX', callback) you can call resolveMx('example.com', callback) instead. These shorthands make it easy to quickly call a particular type of record. There are also resolve4() and resolve6() which resolve IPv4 and IPv6 address records respectively.

Example 7-2. Using resolve() vs resolveMx()

var dns = require('dns');

dns.resolve('example.com', 'MX', function(e, r) {
  if (e) {
    console.log(e);
  }
  console.log(r);
});

dns.resolveMx('example.com', function(e, r) {
  if (e) {
    console.log(e);
  }
  console.log(r);
});

Since resolve() mostly returns a list of many IP addresses so there is also a convenience method to just get one IP from an A record query. The dns.lookup() method takes a domain an IP family (4 or 6) and a callback. However, unlike resolve it only ever returns one address. If you don't pass an address it defaults to the network interface's current setting.

Example 7-3. Looking up a single A record with lookup()

var dns = require('dns');

dns.lookup('google.com', 4, function(e, a) {
  console.log(a);
});

Crypto

Cryptography is used in lots of places for a variety of tasks. Node uses the OpenSSL library as the basis of its cryptography. This is because OpenSSL is already a well tested, hardened implementation of the cryptographic algorithms. This means that if you didn't compile Node with OpenSSL support to installation appendix the crypto module won't work.

The cryptography module enables a number of different tasks. Firstly it powers the SSL/TLS parts of Node. Secondly, it contains the hashing algorithms like MD5 or SHA-1 that you might want to use in your application. Thirdly, it allows you to use HMAC[8]. There are some encryption methods to cipher data to it is encrypted. Finally, it contains other public key cryptographic functions to sign data and verify signatures.

Each of the functions that cryptography does is contained within a Class (or Classes).

Hashing

Hashes are used for a few important functions such as obfuscating data in a way that allows it to be validated or providing a small checksum for a much larger piece of data. In order to use hashes in Node you should create a hash object using the factory method crypto.createHash(). This returns a new Hash instance using a specified hashing algorithm. Most popular algorithms are available, the exact ones depend on your version of OpenSSL but common ones are:

md5

sha1

sha256

sha512

ripemd160

These algorithms all have different advantages and disadvantages. MD5, for example, is used in many applications but has a number of known flaws including collision issues[9]. Depending on your application you will probably want to pick either a widely deployed algorithm like MD5, or preferably SHA1 or a more hardened algorithm like RIPEMD or SHA256 or SHA512.

Once you have data in the hash then you can use it to create a digest. This is done by calling hash.update() with some data. You can keep updating a hash with data until you want to output it. To output the hash of the data call the hash.digest() method. This will output the digest of the data inputted into the hash with hash.update().

Example 7-4. Creating a digest using hash

> var crypto = require('crypto');
> var md5 = crypto.createHash('md5');
> md5.update('foo');
{}
> md5.digest();
'¬½\u0018ÛLÂø\\íïeOÌÄ¤Ø'
>

Notice the output of the digest is a bit weird. That's because that's the binary representation, it's most common to represent a digest as hex. We can do that by adding 'hex' as a parameter to hash.digest:

Example 7-5. The lifespan of hashes and getting hex output

> var md5 = crypto.createHash('md5');
> md5.update('foo');
{}
> md5.digest();
'¬½\u0018ÛLÂø\\íïeOÌÄ¤Ø'
> md5.digest('hex');
Error: Not initialized
    at [object Context]:1:5
    at Interface.<anonymous> (repl.js:147:22)
    at Interface.emit (events.js:42:17)
    at Interface._onLine (readline.js:132:10)
    at Interface._line (readline.js:387:8)
    at Interface._ttyWrite (readline.js:564:14)
    at ReadStream.<anonymous> (readline.js:52:12)
    at ReadStream.emit (events.js:59:20)
    at ReadStream._emitKey (tty_posix.js:280:10)
    at ReadStream.onData (tty_posix.js:43:12)
> var md5 = crypto.createHash('md5');
> md5.update('foo');
{}
> md5.digest('hex');
'acbd18db4cc2f85cedef654fccc4a4d8'
>

When we call hash.digest() again we get an error. This is because once hash.digest() is called the hash object is finalised and cannot be reused. We need to create a new instance of Hash and use that instead. However this time we get the hex output that is often more useful. The options for hash.digest() output are binary (default), hex and base64.

Another thing worth noting is that all the data from each call of hash.update() will be concatenated. In other words the following examples are identical:

Example 7-6. Looking at how hash update concatinates input

> var sha1 = crypto.createHash('sha1');
> sha1.update('foo');
{}
> sha1.update('bar');
{}
> sha1.digest('hex');
'8843d7f92416211de9ebb963ff4ce28125932878'
> var sha1 = crypto.createHash('sha1');
> sha1.update('foobar');
{}
> sha1.digest('hex');
'8843d7f92416211de9ebb963ff4ce28125932878'
>

It is also important to know that while hash.update() look a lot like a stream it isn't really. You can easily hook a stream to hash.update() but you can't use stream.pipe().

HMAC

HMAC combines the hashing algorithms with a cryptographic key in order to stop a number of attacks to the integrity of the signature. This means that HMAC uses both a hashing algorithm (such as the ones discussed in the previous section) and an encryption key. The HMAC API in Node is virtually identical to the Hash API. The only difference is that the creation of an hmac object requires a key as well as a hash algorithm.

crypto.createHmac() returns an instance of Hmac which has both hmac.update() and hmac.digest() which are identical to hash.update() and hash.digest().

The key required to create an hmac object is a PEM encoded key as a string. It is easy to create a key on the command line using openssl:

Example 7-7. Creating a PEM encoded key

Enki:~ $ openssl genrsa -out key.pem 1024
Generating RSA private key, 1024 bit long modulus
...++++++
............................++++++
e is 65537 (0x10001)
Enki:~ $

It's really easy to create an RSA in PEM format and put it into a file, in this case called key.pem. We could also have called this directly from Node using the process module if we omitted the -out key.pem option to get it on a STDOUT stream. Instead we are going to import a key from file and use it to create an hmac object and create a digest:

Example 7-8. Creating an hmac digest

> var crypto = require('crypto');
> var fs = require('fs');
>
> var pem = fs.readFileSync('key.pem');
> var key = pem.toString('ascii');
>
> var hmac = crypto.createHmac('sha1', key);
>
> hmac.update('foo');
{}
> hmac.digest('hex');
'7b058f2f33ca28da3ff3c6506c978825718c7d42'
>

In this example I'm using fs.readFileSync() since a lot of the time loading keys will be a server setup task. As such it's fine to do them synchronously (it might slow down server start up time) because you aren't serving clients yet so blocking the event loop is ok. In general, other than the use of the encryption key this is exactly like Hash.

Public Key Cryptography

The public key cryptography functions are split into four Classes. Cipher, Decipher, Sign and Verify. Like all the other Classes in crypto they have factory methods. Cipher encrypts data, Decipher decrypts data, Sign creates a cryptographic signature for data and Verify validates cryptographic signatures.

For the HMAC operations we used a private key. For these operations we are going to use both the public and the private keys. Public key cryptography has matched sets of keys. One key, the private key, is kept by the owner and it is used to decrypt and sign data. The other key, the public key, is made available to other parties. The public key can be used to encrypt data that only the private key owner can read, or to verify the signature of data signed with the private key. Let's extract the public key of the private key we generated to do the HMAC digests. Node expects public keys in certificate format which requires additional 'information' you can leave all the information blank if you like:

Example 7-9. Extracting a public key certificate from a private key

Enki:~ $ openssl req -key key.pem -new -x509 -out cert.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:
Enki:~ $ ls cert.pem
cert.pem
Enki:~ $

We simply ask openssl to read the private key in and then output the public key into a new file called cert.pem in X509 certificate format. All of the operations in crypto expect keys in PEM format.

Encrypting with Cipher

The Cipher Class provides a wrapper for encrypting data using a private key. The factory method to create a cipher takes an algorithm and the private key. The algorithms supported come from those compiled into your OpenSSL implementation.

blowfish

aes192

Many modern cryptographic algorithms use block ciphers. This means that the output is always in standard sized 'blocks'. The block sizes vary between algorithms blowfish, for example, uses 40byte blocks . This is significant when using the Cipher API because the API will always output fixed sized blocks. This is to stop information being leaked to an attacker about the data being encrypted or the specific key being used to do the encryption. Like Hash and Hmac the Cipher API also uses the update() method to input data. However, update works differently when used in a cipher. Firstly cipher.update() returns a block of a encytped data if it can. This is where block size becomes important. If the amount of data in the cipher plus the amount of data passed to cipher.update() is enough to create a block then the encrypted data will be returned. If there isn't enough to form a block then it will be stored in the cipher. Cipher also has a new method Cipher.final() which replaces the digest() method. When cipher.final() is called any remaining data in the cipher will be returned encrypted but with enough padding to make sure the block size is reached.

Example 7-10. Ciphers and block size

> var crypto = require('crypto');
> var fs = require('fs');
>
> var pem = fs.readFileSync('key.pem');
> var key = pem.toString('ascii');
>
> var cipher = crypto.createCipher('blowfish', key);
>
> cipher.update(new Buffer(4), 'binary', 'hex');
''
> cipher.update(new Buffer(4), 'binary', 'hex');
'ff57e5f742689c85'
> cipher.update(new Buffer(4), 'binary', 'hex');
''
> cipher.final('hex')
'96576b47fe130547'
>

In order to make it easier to read I specified the input and output formats. In this case binary in because I'm passing a new Buffer (containing whatever random crap was in memory) and hex to give us a nice easy to read output. You can see that the first time I call cipher.update() with 4 bytes of data I get back an empty string. The second time because we have enough data to generate a block we get the encrypted data back as hex. When we call cipher.final() there isn't enough data to create a full block so the output is padded and a full (and final) block is returned. Since Cipher.final() only outputs no input format was included. The input and output format are both optional and will be assumed to be binary unless specificed.

Decrypting with Decipher

The Decipher class is almost exactly the same as the Cipher class. You can pass encrypted data to a decipher object using decipher.update(). It will stream it into blocks until it can output the unencrypted data. You might think that since cipher.update() and cipher.final() always give fixed length blocks you would have to give perfect blocks to decipher, however it will happily buffer the data so you can pass it data you got off some other I/O transport such as disk or network which might give you block sizes different from those used by the encryption algorithm.

Let's take a look an example of encrypting data and then decrypting it:

Example 7-11. Encrypting and decrypting text

> var crypto = require('crypto');
> var fs = require('fs');
>
> var pem = fs.readFileSync('key.pem');
> var key = pem.toString('ascii');
>
> var plaintext = new Buffer('abcdefghijklmnopqrstuv');
> var encrypted = "";
> var cipher = crypto.createCipher('blowfish', key);
>
> encrypted += cipher.update(plaintext, 'binary', 'hex');
> encrypted += cipher.final('hex');
>
> var decrypted = "";
> var decipher = crypto.createDecipher('blowfish', key);
> decrypted += decipher.update(encrypted, 'hex', 'binary');
> decrypted += decipher.final('binary');
>
> var output = new Buffer(decrypted);
>
> output
<Buffer 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76>
> plaintext
<Buffer 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76>
>

It is important to make sure the both the input and output formats match up for both the plaintext and the encrypted data. It's also worth noting that in order to get a buffer you'll have to make one from the strings returned by cipher and decipher.

Creating signatures using Sign

Signatures verify that some data has been authenticated by the signer using the private key. However, unlike HMAC, the public key can be used to authenticate the signature. The API for Sign is nearly identical to that for HMAC. crypto.createSign() is used to make a sign object, it only take the signing algorithm. sign.update() allows you to add data to the sign object. When you want to create the signature you can call sign.sign() with a private key to sign the data.

Example 7-12. Signing data with sign

> var sign = crypto.createSign('RSA-SHA256');
> sign.update('abcdef');
{}
> sig = sign.sign(key, 'hex');
'35eb47af5260a00c7bad26edfbe7732a897a3a03290963e3d17f48331a42d48cc0753621e95c374c917424574be237e112c2be2a5eff74c200697b58e275a846f46da5e6f53ba69a0532b1db7e93aba49e3a463620f53461e7215189ae4a1658f4a57720174dddc5a9eaffbd34982ca4a3eefc03886f3caa53e013c3f03aa81b'
>

Verifying signatures with Verify

The Verify API works uses the same verify.update() format to add data. When you have added all the data to be verified against the signature verify.verify() validates the signature. It takes the cert (the public key), the signature and the format of the signature.

Example 7-13. Verifying signatures

> var crypto = require('crypto');
> var fs = require('fs');
>
> var privatePem = fs.readFileSync('key.pem');
> var publicPem = fs.readFileSync('cert.pem');
> var key = privatePem.toString();
> var pubkey = publicPem.toString();
>
> var data = "abcdef"
>
> var sign = crypto.createSign('RSA-SHA256');
> sign.update(data);
{}
> var sig = sign.sign(key, 'hex');
>
> var verify = crypto.createVerify('RSA-SHA256');
> verify.update(data);
{}
> verify.verify(pubkey, sig, 'hex');
1

[8] Hash-based Message Authentication Code is a cryptographic way of verifying data. It is often used like hashing algorithms to verify that two pieces of data match, but it also verifies that the data hasn't been tampered with.

[9] It's possible to deliberately make two pieces of data with the same MD5 checksum which for some purposes can make the algorithm less desirable more modern algorithms are less prone to this although people are finding similar problems with SHA1 now.

Processes

While Node as a platform abstracts a lot of things from the OS, at the same time you are still running in an OS and there are often time when you want to interact more directly with the OS. Node allows you to interact with system processes that already exist as well as creating new child processes in do work of various kinds. While Node itself is generally a 'fat' thread with a single event loop it doesn't mean you shouldn't start other processes (threads) to do work out of the event loop.

Process

The process module gives you the ability to get information about and change the settings of the current Node process. Unlike most modules the process module is a global and is always available as the variable process.

Process events

process is an instance of EventEmitter so it provides events based on systems calls to the Node process. The exit event provides a final hook before the Node process exits. Importantly the event loop will not be called after the exit event so only code without callbacks will be executed:

Example 7-14. Calling code when Node is exiting

process.on('exit', function () {
  setTimeout(function () {
   console.log('This will not run');
  }, 100);
  console.log('Bye.');
});

It's important to understand the event loop in this context. Since the loop isn't going to run again the setTimeout() code will never be evaluated.

An extremely useful event provided by process is uncaughtException. After you've spent any time with Node you'll find that exceptions that hit the main event loop will kill your Node process. In many use cases, especially servers, that are expected to never be down this is unacceptable. The uncaughtException event provides an extremely brute force way of catching these exceptions. It's really a last line of defense but it's extremely useful for that purpose.

Example 7-15. Trapping an exception with the uncaughtException event

process.on('uncaughtException', function (err) {
  console.log('Caught exception: ' + err);
});

setTimeout(function () {
  console.log('This will still run.');
}, 500);

// Intentionally cause an exception, but don't catch it.
nonexistentFunc();
console.log('This will not run.');

Let's break down what's happening. Firstly we create an event listener for uncaughtException. This is not a smart handler, it simply outputs to STDOUT the exception. If this Node script were running as a server STDOUT could easily be used to log into a file and capture these errors. However by capturing this event Node will not exit. However we should also look at how this code executes. We know that all the JavaScript be run once and then any callbacks will be run each time their event listener emits. In this scenario since nonexistentFunc() will throw an exception all the code after it will not be called. However, any code that has already been run will continue to run. This means the setTimeout() will still call. This is significant when writing servers. Let's consider some more code in this area:

Example 7-16. The affect on callbacks of catching exceptions

var http = require('http');
var server = http.createServer(function(req,res) {
  res.writeHead(200, {});
  res.end('response');
  badLoggingCall('sent response');
  console.log('sent response');
});

process.on('uncaughtException', function(e) {
  console.log(e);
});

server.listen(8080);

This code creates a simple HTTP server and then listens for any uncaught exceptions at the process level. In our HTTP server the callback deliberately calls a bad function after we've sent the HTTP response. Let's look at the console output for this script.

Example 7-17. Output of exception handling an HTTP server

Enki:~ $ node ex-test.js
{ stack: [Getter/Setter],
  arguments: [ 'badLoggingCall' ],
  type: 'not_defined',
  message: [Getter/Setter] }
{ stack: [Getter/Setter],
  arguments: [ 'badLoggingCall' ],
  type: 'not_defined',
  message: [Getter/Setter] }
{ stack: [Getter/Setter],
  arguments: [ 'badLoggingCall' ],
  type: 'not_defined',
  message: [Getter/Setter] }
{ stack: [Getter/Setter],
  arguments: [ 'badLoggingCall' ],
  type: 'not_defined',
  message: [Getter/Setter] }

When we start the example script the server is available and I made a number of HTTP requests to it. Notice that the server doesn't shut down at any point. Instead the errors are logged using the function attached to the uncaughtException event. However, we are still serving complete HTTP requests. Why? Well the deliberate error happened after the HTTP request finished, so it was unaffected but we were never able to call the console.log(). Like the previous example the code path stopped before completion, however in this case it's important to remember that the server kept running, any other code was unaffected by the exception encapsulated in one specific code path. It's important to understand the way that listeners are implemented in Node.

Example 7-18. The abreviated listener code for EventEmitter

EventEmitter.prototype.emit = function(type) {

...

  var handler = this._events[type];

...

  } else if (isArray(handler)) {
    var args = Array.prototype.slice.call(arguments, 1);

    var listeners = handler.slice();
    for (var i = 0, l = listeners.length; i < l; i++) {
      listeners[i].apply(this, args);
    }
    return true;

...

};

In the emit code, when an event is emitted one of the checks is to see if there is an array of listeners. If there is more than one listener the way the listeners are called is by looping through the array in order. This means the first attached listener will be called first with apply() , then the second, etc. What's important to note here is that all all listeners on the same event are the same code path. That is, to the event loop it calls emit() and then emit runs a bunch of JavaScript which happens to call your callbacks in order. That's significant, because while an exception in one instance of an event won't affect other events an exception in one callback will stop execution for all other callbacks on the same event.

We also get access to a number of systems events through process. When the process gets a signal those are exposed to Node via events emitted by process. There are a lot of POSIX system events that can be found in the sigaction(2) man page. Really common ones including things like SIGINT, the interupt signal. Typically a SIGINT is what happens when you press ctl+c in the terminal on a running process. Unless you handle the signal events via process Node will just perform the default action, in the case of a SIGINT this would be to kill the process.

Example 7-19. Catching signals to the Node process

// Start reading from stdin so we don't exit.
process.stdin.resume();

process.on('SIGINT', function () {
  console.log('Got SIGINT.  Press Control-D to exit.');
});

In order to make sure Node doesn't exit on it's own we read from STDIN so the Node process continues to run. If you ctl+c the program while it's running the OS will send a SIGINT to Node which will be caught by the SIGINT event handler and instead of exiting we'll log to console instead. Any of the standard POSIX signals have an event in process by default.

Interacting with the current Node process

Process contains a lot of meta-information about the Node process this can be really helpful when you need to manage your Node environment from within the process. There are a number of properties which contain immutable information about the Node such as process.version or process.installPrefix which contain the compiled Node version and the install path (/usr/local, ~/local, etc) that was used during installation respectively. process.platform lists the platform on which the Node is currently running. The output will specify the kernel (linux2, darwin, etc) rather than 'Redhat ES3', 'Windows 7', 'OSX 10.7', etc. The process.uptime() method provide the number of seconds the process has been running.

There are also a number of things that you can get and set about the Node process itself. When the process runs it does so with a particular user and group. You can get these and set them with process.getgid(), process.setgid(), process.getuid() and process.setuid(). These can be very useful for making sure that Node is running in a secure way. It's worth noting that the set methods take either the numerical ID of group/username or the group/username itself. However, if you pass the group/username the methods do a blocking lookup to turn the group/username into an ID.

The Process ID or PID is also available as the process.pid property. This is the ID the system has assigned this specific Node instance. You can also set the title that Node displays to the system using the process.title property. Whatever is set in this property will be displayed in the ps command. This can be extremely useful when using multiple Node process in a production environment. Instead of having a lot of processes called "node" or possibly "node app.js" you can set them intelligently to make them easy to refer to. When one process is hogging CPU or RAM it's great to have a quick idea of what it is doing immediately.

Other available information includes the process.execPath which shows the execution path of the current Node binary (e.g. /usr/local/bin/node). The current working directory (to which all files opened will be relative) is accessible with process.cwd(). The working directory is the directory you were in when Node was started. You can change it using process.chdir(). This will throw an exception if the directory is unreadable or doesn't exist. You can also get the current memory usage of the current Node process using process.memoryUsage(). This returns an object specifying the size of the memory usage in a couple of ways: rss show how much RAM is being used and vsize shows the total memory used including both RAM and swap. You'll also get some V8 stats: heapTotal and heapUsed which show how much memory V8 has allocated and how much it is actively using.

OS Input/Output

There are a number of places you can touch interact with the OS (other than fiddling with the process) from process. One of the main ones is having access to the standard OS IO streams. STDIN is an input stream to the process, STDOUT is the output stream and STDERR is an error stream from the process. These are exposed with process.stdin, process.stdout and process.stderr respectively. process.stdin is a readable stream, process.stdout and process.stderr are both writable streams.

process.stdin

STDIN is a really useful device for interprocess communication. It's used to facilitate things like piping in the shell. When I do cat file.txt | node program.js it will be the STDIN stream that receives the data from the cat command.

Since process is always available the process.stdin stream is also always initialised in any Node process. For this reason it is in a paused state until you resume() it. This means that Node will only fill the read buffer for the stream and then stop until you are ready to deal with it. This approach avoids data loss.

Example 7-20. Writing STDIN to STDOUT

process.stdin.resume();
process.stdin.setEncoding('utf8');

process.stdin.on('data', function (chunk) {
  process.stdout.write('data: ' + chunk);
});

process.stdin.on('end', function () {
  process.stdout.write('end');
});

We ask process.stdin to resume() set the encoding to UTF-8 and then set a listener to push any data sent to process.stdout. When the process.stdin sends the end event we pass that on to the process.stdout stream. We could also easily do this with the stream pipe() method since these are both real streams.

Example 7-21. Writing STDIN to STDOUT using Pipe

process.stdin.resume();
process.stdin.pipe(process.stdout);

This is the most elegant way of connecting two streams together and it works really well because in this case they are both real streams.

process.stderr

STDERR is used to output exceptions and problems with program execution. In POSIX systems because it is a separate stream output logs and error logs can be easily separated into different destinations. This can be very desirable. However, that means it has a couple of caveats in Node. When you write to STDERR Node guarantees that the write will happen. However unlike a regular stream this is done as a blocking call. Typically calls to Steam.write() return a boolean of if Node was able to write to the kernel buffer. With process.stderr this will always be true, but it might take a while to return unlike the regular write(). Typically it will be very fast but it is possible the kernel buffer is full and it might not be. This means that it is generally inadvisable to write a lot to STDERR in a production system because it may block real work.

One final thing is that process.stderr is always a UTF-8 stream.

ARGV

Another place where we often touch the OS is the passed arguments when the program is started. ARGV is an array containing the command line arguments. This also includes the node command itself.

Example 7-22. A simple script outputting argv

console.log(process.argv);

Example 7-23. Running a script to output argv

Enki:~ $ node argv.js -t 3 -c "abc def" -erf       foo.js
[ 'node',
  '/Users/croucher/argv.js',
  '-t',
  '3',
  '-c',
  'abc def',
  '-erf',
  'foo.js' ]
Enki:~ $

There are few things to notice here. Firstly the process.argv array is simply a split of the command line based on whitespace. It's not simply single spaces used for the split if there are many pieces of whitespace they only count as a single split. It's \s+ in regex. This doesn't count for white space in quotes however. Quotes can be used to keep tokens together. Also, notice how the first file argument is expanded. On the command line I can pass a relative file argument and it will be expanded. This is also true for special characters like ~ to refer to the home directory. However, file arguments which are not the first argument will not be expanded.

ARGV is extremely helpful for writing command line scripts, but it's pretty raw. There are a number of community projects which extend support to help you easily write command line applications with support for automatically enabling features, writing in-line help systems and other more advanced features.

Event loop and tickers

If you've done work with JavaScript in browsers you'll be familiar with setTimeout(). In Node we have a much more direct way to access the event loop and defer work which is extremely useful. process.nextTick() create a callback to be executed on the next "tick" or loop of the event loop. While it is implemented as a queue it will supersede other events. Let's explore that a little bit:

Example 7-24. Using process.nextTick() to insert callbacks into the event loop

> var http = require('http');
> var s = http.createServer(function(req, res) {
... res.writeHead(200, {});
... res.end('foo');
... console.log('http response');
... process.nextTick(function(){console.log('tick')});
... });
> s.listen(8000);
>
> http response
tick
http response
tick

In this example I create an HTTP server with a callback that creates a callback on process.nextTick(). No matter how many requests I make to the HTTP server the "tick" will always occur on the next pass of the event loop. Unlike other callbacks nextTick() callbacks are not a single event so they are not subject to the usual callback exception brittleness:

Example 7-25. nextTick()s don't fail because of exceptions

process.on('uncaughtException', function(e) {
  console.log(e);
});

process.nextTick(function() {
  console.log('tick');
});
process.nextTick(function() {
  iAmAMistake();
  console.log('tock');
});
process.nextTick(function() {
  console.log('tick tock');
});
console.log('End of 1st loop');

Example 7-26. Results of previous code

Enki:~ $ node process-next-tick.js
End of 1st loop
tick
{ stack: [Getter/Setter],
  arguments: [ 'iAmAMistake' ],
  type: 'not_defined',
  message: [Getter/Setter] }
tick tock
Enki:~ $

Despite the deliberate error each of the ticks is isolated. Notice how there is code to run the tick events only happen after Node has read the main body of code.

Child Process

The child_process module allows you to create child processes of your main Node process. Since Node has only one event loop in a single process sometimes it is helpful to create child processes. This can be do make use of more cores of your CPU since a single Node process can only use one of the cores. child_process can also be used to launch other programs and let Node interact with them. This extremely helpful when writing command line scripts.

There are two main methods in child_process the spawn() method creates a child process with its own STDIN, STDOUT and STDERR file descriptors, the exec() method creates a child process and returns the result as a callback when the process is complete. This provides one extremely versatile way to create child processes and one way that is still non-blocking but doesn't require you to write extra code in order to steam.

Every Child_Process has some common properties. They all contain properties for stdin, stdout and stderr which are the streams for the three ways of interacting with a process. There is also a pid property which contains the OS process ID of the process. They also emit the exit event when they exit. Obviously other data events are available via the stream methods of child_process.stdin, child_process.stdout and child_process.stderr

child_process.exec()

Let's start with exec() as the more straight-forward use case. Using exec() you can create a process which will run some program (possibly another Node program) and then return the results for you in a callback:

Example 7-27. Calling ls with exec()

var cp = require('child_process');

cp.exec('ls -l', function(e, stdout, stderr) {
  if(!e) {
    console.log(stdout);
    console.log(stderr);
  }
});

When we call exec() can pass a command to call. Note that the entire command is a string. If you need to pass arguments they should be constructed into the string. In the example you can see I passed the -l argument to get the long form output. You can also include things like | in order to chain commands together. Node will return the agregate results. The callback function receive three arguments: an error, the result of STDOUT and the result of STDERR. Notice that just calling ls will run it in the current working directory of Node accessible from process.cwd().

It's important to understand the difference between the first and third arguments. The error object returned will be null unless an error status code is returned from the child process or there was another exception. When the child process exits it passes a status up to the parent process. In Unix, for example, this is 0 for success and an 8bit number greater than 0 for an error. When an error code is returned from the child process the error object will contain the error code and STDERR. However, when a process is successful there may still be data on STDERR. The error object is also used when the command called doesn't meet the constraints placed on it by Node. exec() takes an optional second argument with an options object. By default this object contains the following:

Example 7-28. Default options object for child_process.exec()

var options = { encoding: 'utf8',
                timeout: 0,
                maxBuffer: 200 * 1024,
                killSignal: 'SIGTERM',
                setsid: false,
                cwd: null,
                env: null };

The encoding property is obvious, timeout specifies the number of milliseconds the process can run before Node kills it. The killSignal property specifies the type of kill signal to use to terminal the process in the case of a time or buffer size overrun. The maxBuffer property is the maximum number of bytes that the STDERR and STDOUT may each grow to. setsid creates a new session for the process, cwd changes the working directory and env sets the environment variables.

Let's set some of the options to restrict the constraint on a process. First let's try restricting the Buffer size of the response:

Example 7-29. Restricting the Buffer size on child_process.exec() calls

> var child = cp.exec('ls', {maxBuffer:1}, function(e, stdout, stderr) {
... console.log(e);
... }
... );
> { stack: [Getter/Setter],
  arguments: undefined,
  type: undefined,
  message: 'maxBuffer exceeded.' }

In this example you can see that when we set a tiny maxBuffer (1kb) running ls quickly exhausted the available space and an error is thrown. It's important to check for errors so that you can deal with them in a sensible way rather than causing an actual exception by trying to access resources that are unavailable because the error occurred. It's also possible to stop a Child after a set amount of time:

Example 7-30. Setting a timeout on process.exec() calls

> var child = cp.exec('for i in {1..100000};do echo $i;done',
... {timeout:500, killSignal:'SIGKILL'},
... function(e, stdout, stderr) {
...   console.log(e);
... });
> { stack: [Getter/Setter], arguments: undefined, type: undefined, message: 'Command failed: ', killed: true, code: null, signal: 'SIGKILL' }

I define a deliberately long running process (counting from 1 to 100,000 in shell script), however I also set a timeout. Notice that I also specified a killSignal, by default the kill signal is SIGTERM but I used SIGKILL[10] to show the feature. When we get the error back notice there is a killed property which tells us that Node killed the process it didn't exit. This is also true for the previous example. Since it didn't exit on its own there isn't a code property or some of the other properties of a system error.

child_process.spawn()

spawn() is very similar to exec() however it is a more general purpose method that requires you to deal with streams and their callbacks yourself. This makes it a lot more powerful and flexible but it also means more code is required to do the kind of one shot system calls we did with exec(). This means that spawn() is most commonly used in server contexts to create subcomponents of a server. This is the most common way people make Node work with multiple cores on a single machine.

While it performs the same function as exec() the API for spawn() is slightly different. The first argument is still the command to start the process with however, unlike exec() it is not a command string, but just the executable. The process' arguments are passed in an array as the (optional) second argument to spawn(). It's like an inverse of process.argv instead of the command being split() across spaces you provide an array to be join()ed with spaces.

Finally, spawn() also takes an options array as the final argument. Some of these options are the same as exec() but we'll cover that in more detail shortly.

Example 7-31. Starting child processes using spawn()

var cp = require('child_process');

var cat = cp.spawn('cat');

cat.stdout.on('data', function(d) {
  console.log(d.toString());
});
cat.on('exit', function() {
  console.log('kthxbai');
});

cat.stdin.write('meow');
cat.stdin.end();

Example 7-32. Results of previous example

Enki:~ $ node cat.js
meow
kthxbai
Enki:~ $

In this example I'm using the UNIX program cat. Cat simply echoes back whatever input it gets. You can see unlike exec() we don't issue a callback to spawn() directly. That's because we are expecting to use the Streams provided by the Child Class to get and send data. I named the variable with the instance of Child "cat". I can access cat.stdout to set events on the STDOUT stream of the child process. I set a listener on cat.stdout to watch for any data events, and I set a listener on the child itself in order to watch for the exit event. I can send my new child data using STDIN by accessing its child.stdin stream. This is just a regular writable stream. However, as a behaviour of the cat program when I close STDIN the process exits. This might not be true for all processes but it is true for cat, which only exits to echo back data.

The options that can be passed to spawn() aren't exactly the same as exec(). This is because you are expected to manage more things by hand with spawn(). The env, setsid and cwd properties are all options for spawn() as are uid and gid to set the user ID and the group ID respectively. Like process setting the uid or the gid to a username or a group name will block briefly while the user or group is looked up. There is one more option for spawn() which doesn't exist for exec(). You can set custom file descriptors that will be given to the new child process. Let's take a little bit of time to cover this topic because it's a little complex.

A file descriptor in UNIX is a way of keeping track of which programs are doing what with which files. Since UNIX lets many programs run at the same time there needs to be a way to make sure that when they interact with the file system they don't accidentally overwrite someone else's changes. The file descriptor table keeps track of all the file which a process wants to access. The kernel might lock a particular file to stop two programs writing to the file at the same time, as well as other management functions. A process will look at its file descriptor table to find the file descriptor representing a particular file and pass that to the kernel to access the file. The file descriptor is simply an integer.

The important thing is that not just pure files are represented by file descriptors. The name is a little deceptive because network and other sockets are also allocated file descriptors. UNIX has inter-process communications (IPC) sockets which lets processes talk to each other. We've been calling them STDIN, STDOUT and STDERR. This is interesting because spawn() let's us specify specific file descriptors when starting a new child process. This means instead of the OS assigning a new file descriptor we can ask a child processes to share an existing file descriptor with the parent process. That file descriptor might be a network socket to the Internet or just the parent's STDIN, the point is that we have a powerful way of delegating work to child processes.

How does this work in practice? When passing the options object to spawn() we can specify customFds to pass our own three file descriptors to the child instead of them creating a STDIN, STDOUT and STDERR file descriptor.

Example 7-33. Passing STDIN, STDOUT and STDERR to a child process

var cp = require('child_process');

var child = cp.spawn('cat', [], {customFds:[0, 1, 2]});

Example 7-34. Running previous example and in data to STDIN

Enki:~ $ echo "foo"
foo
Enki:~ $ echo "foo" | node

readline.js:80
    tty.setRawMode(true);
        ^
Error: ENOTTY, Inappropriate ioctl for device
    at new Interface (readline.js:80:9)
    at Object.createInterface (readline.js:38:10)
    at new REPLServer (repl.js:102:16)
    at Object.start (repl.js:218:10)
    at Function.runRepl (node.js:365:26)
    at startup (node.js:61:13)
    at node.js:443:3
Enki:~ $ echo "foo" | cat
foo
Enki:~ $ echo "foo" | node fds.js
foo
Enki:~ $

The file descriptors 0, 1 and 2 represent STDIN, STDOUT and STDERR respectively. In this example we create a child and pass it STDIN, STDOUT and STDERR from the parent Node process. We can test this wiring using the command line. The echo command outputs a string we did it. If we pass that directly to node with a pipe (STDOUT to STDIN), we get an error. We can however pass it to the cat command which echoes it back. If we pipe to the Node process running our script it echoes back. This is because we've hooked up the STDIN, STDOUT and STDERR of the Node process directly to the cat command in our child process. When the main Node process gets data on STDIN then it get passed to the cat child process which echoes it back on the shared STDOUT. One thing to note is that once you wire up Node process this way the child process loses it's child.stdin, child.stdout and child.stderr file descriptor references. This is because once you pass the file descriptors to the process they are duplicated and the kernel handles the data passing. This means that Node isn't in between the process and the file descriptors so you cannot add events to those streams.

Example 7-35. Tying to access file descriptor streams fails when custom FDs are passed

var cp = require('child_process');
var child = cp.spawn('cat', [], {customFds:[0, 1, 2]});
child.stdout.on('data', function(d) {
  console.log('data out');
});

Example 7-36. Results of test

Enki:~ $ echo "foo" | node fds.js

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
 foo
       ^
TypeError: Cannot call method 'on' of null
    at Object.<anonymous> (/Users/croucher/fds.js:3:14)
    at Module._compile (module.js:404:26)
    at Object..js (module.js:410:10)
    at Module.load (module.js:336:31)
    at Function._load (module.js:297:12)
    at Array.<anonymous> (module.js:423:10)
    at EventEmitter._tickCallback (node.js:126:26)
Enki:~ $

When custom file descriptors are specified the streams are literally set to null and are completely inaccessible from the parent. It is still preferable in many cases though because the kernel is much faster at routing than if we did something like stream.pipe() with Node to connect the streams together. However STDIN, STDOUT and STDERR aren't the only file descriptors worth connecting to child processes. A very common use case is connecting network sockets to a number children which allows multi-core utilization.

Say we are creating a web site, or a game server or anything that has to deal with a bunch of traffic. We have this great server which has a bunch of processors each of which has 2 or 4 cores. If we just started a Node process running our code we'd have just one core being used. While CPU isn't always the critical factor for Node we want to make sure we get as close to CPU bound as we can. We could start a bunch of Node processes with different ports and load balance them with Nginx or Apache Traffic Server. However that's inelegant and requires use to use more software. We could create a Node process that creates a bunch of child processes and routes all the requests to them. This is a bit closer to our optimal solution, but we just created a single point of failure. There is one Node process which routes all the traffic. This isn't ideal. This is where passing custom FDs comes into it's own. In the same way we can pass the STDIN, STDOUT, STDERR of a master process we can create other sockets and pass those in to child processes. However, because we are passing file descriptors instead of messages the kernel will deal with the routing. This means while the master Node process is still required it isn't bearing the load for all the traffic.

[10] SIGKILL is kill -9

Testing

Assert

Assert is a core library which provides the basis for testing code. Assert does pretty much what you would expect. It allows you to make assertions about objects and function calls so that you can test code. Node's own tests are written with Assert.

Most assert methods are pairs. One method provides the positive test the other, the negative. They are really easy to get started with and provide a great way to unit test your code features.

Example 7-37. Basic assertions

> var assert = require('assert');
> assert.equal(1, true, 'Truthy');
> assert.notEqual(1, true, 'Truthy');
AssertionError: Truthy
    at [object Context]:1:8
    at Interface.<anonymous> (repl.js:171:22)
    at Interface.emit (events.js:64:17)
    at Interface._onLine (readline.js:153:10)
    at Interface._line (readline.js:408:8)
    at Interface._ttyWrite (readline.js:585:14)
    at ReadStream.<anonymous> (readline.js:73:12)
    at ReadStream.emit (events.js:81:20)
    at ReadStream._emitKey (tty_posix.js:307:10)
    at ReadStream.onData (tty_posix.js:70:12)
>

The most obvious thing here is that when an assert method doesn't pass then it throws an exception. This is a fundamental principle in the test suites. When a test suite runs it should just run, without throwing. If that is the case then it is successful. You will also notice the methods take two values, one is the expected value the second is the actual value. This is likely to be the return value from a function call.

There aren't too many assertions but they each serve a different purpose. The equal() and notEqual() check for the == equality and != inequality operators. This means they test weakly for truthy and falsy values as Crockford coined them. In brief, when tested as a boolean falsy values are false, 0, empty strings (e.g. ""), null, undefined and NaN. All other values are truthy. A string such as "false" is truthy. A string containing "0" is also truthy. As such equal() and notEqual() are fine to compare simple values (strings, numbers, etc) with each other, but you should be careful checking against booleans to ensure you got the result you wanted. The stringEqual() and notStrictEqual() methods provide a way for testing equality with === and !== which will ensure that only actualy values of true and false are treated as true and false respectively. The ok() method is a shorthand for testing something is truthy by comparing the value with true using ==.

Example 7-38. Testing something is truthy with assert.ok()

> assert.ok('This is a string', 'Strings that are not empty are truthy');
> assert.ok(0, 'Zero is not truthy');
AssertionError: Zero is not truthy
    at [object Context]:1:8
    at Interface.<anonymous> (repl.js:171:22)
    at Interface.emit (events.js:64:17)
    at Interface._onLine (readline.js:153:10)
    at Interface._line (readline.js:408:8)
    at Interface._ttyWrite (readline.js:585:14)
    at ReadStream.<anonymous> (readline.js:73:12)
    at ReadStream.emit (events.js:81:20)
    at ReadStream._emitKey (tty_posix.js:307:10)
    at ReadStream.onData (tty_posix.js:70:12)
>

Often the things you want to compare aren't simple values but objects. JavaScript doesn't have a way to let objects define equality operators on themselves, and even if it did often people wouldn't. The deepEqual() and notDeepEqual() methods provide a way of deeply comparing object values. Without going into too much of the gory details these methods perform a few checks. Depending on each step the test may pass or fail at each test in which case it will return immediately. The first test checks that the values don't simply match with the === operator. Next the values are checked to see if they are Buffers and if so are checked for their length, and then checked byte by byte. Next if the object types don't match with the == operator they can't be equal. Finally if it is an object a more extensive test is done. The final tests compare the prototypes of the two objects, the number of properties and recursively perform deepEqual() on each property.

The important point is here is that deepEqual() and notDeepEqual() are extremely helpful and thorough but also really potentially expensive. You should really try and use them only when needed. While they will attempt to do the most efficient tests first it can still take a longer to find an inequality. If you can provide a more specific reference such as the property of an object rather than the whole object you can improve the performance of your tests a lot.

The next assert methods are throws() and doesNotThrow(). These check if a particular block of code throws (or doesn't) any exception or a specific exception. They are pretty straight-forward to use but have a few options that are worth reviewing. It might be easy to overlook these tests but handling exceptions is an essential part of writing robust JavaScript code so you should use these tests to make sure the code you write throws in all the correct places. There is more on how to deal with exceptions well in Chapter 4. In order to pass blocks of code to throws() and doesNotThrow() we wrap them in functions that take no arguments. The exception being tested for is optional. If one isn't passed throws() will just check if any exception happened and doesNotThrow() will ensure one doesn't. If a specific error is passed then throws() will check that the specified exception and only that exception was thrown. If any other exceptions are thrown or the exception isn't thrown the test will not pass. For doesNotThrow() when an error is specified it will throw the message other exceptions and ignore them. If an exception matching the specified error is thrown it will cause the test to fail.

Example 7-39. Using assert.throws() and assert.doesNotThrow() to check for exception handling

> assert.throws(
... function() {
...   throw new Error("Seven Fingers. Ten is too mainstream.");
... });
> assert.doesNotThrow(
... function() {
...   throw new Error("I lived in the ocean way before Nemo");
... });
AssertionError:   "Got unwanted exception (Error).."
    at Object._throws (assert.js:281:5)
    at Object.doesNotThrow (assert.js:299:11)
    at [object Context]:1:8
    at Interface.<anonymous> (repl.js:171:22)
    at Interface.emit (events.js:64:17)
    at Interface._onLine (readline.js:153:10)
    at Interface._line (readline.js:408:8)
    at Interface._ttyWrite (readline.js:585:14)
    at ReadStream.<anonymous> (readline.js:73:12)
    at ReadStream.emit (events.js:81:20)
>

There are four ways to specify the type of error to look for or avoid. You can pass a comparison function, a regular expression, a string or an Object constructor. If you pass a function it should take a single argument, which will be the exception error. It will be used to compare the exception to see if there is a match. It should return true or false. If you pass a regex it will be run on the error message to find a match using the regex.test() method in JavaScript. If you pass a string it will be directly compared with the error message. Finally, if you pass a constructor it will be used to perform a typeof test on the exception. If the error that is thrown is a typeof the constructor then the exception matches. This can be used to make throws() and doesNotThrow() very flexible.

VM

The vm module allows you to run arbitary chunks of code and get a result back. It has a number of features that allow you to change the context which the code runs in. This can be useful to act as a kind of faux sandbox. However, it still running in the same Node process so you should be catious of what code you run. vm is similar to eval() but it offers some more features and a better API for managing code. It doesn't have the ability to interact with the local scope in the way that eval() does, however.

There are two ways to run code with vm the first is to run the code "inline" this is similar to using eval() the second is to precompile it into a vm.Script object. Let's have a look at running code inline using vm.

Example 7-40. Using vm to run code

    > var vm = require('vm');
> vm.runInThisContext("1+1");
2

So far vm looks a lot like eval() we pass some code to it and we get a result back. However, vm doesn't interact with local scope in the same way that eval() does. Code run with eval() will behave as is it was truly inline in place of the eval() method call however calls to vm methods will not interact with the local scope.

Example 7-41. Accessing the local scope the differences between vm and eval()

    > var vm = require('vm'),
... e = 0,
... v = 0;
> eval(e=e+1);
1
> e
1
> vm.runInThisContext('v=v+1');
ReferenceError: v is not defined
    at evalmachine.<anonymous>:1:1
    at [object Context]:1:4
    at Interface.<anonymous> (repl.js:171:22)
    at Interface.emit (events.js:64:17)
    at Interface._onLine (readline.js:153:10)
    at Interface._line (readline.js:408:8)
    at Interface._ttyWrite (readline.js:585:14)
    at ReadStream.<anonymous> (readline.js:73:12)
    at ReadStream.emit (events.js:81:20)
    at ReadStream._emitKey (tty_posix.js:307:10)
>
> vm.runInThisContext('v=0');
0
> vm.runInThisContext('v=v+1');
1
>
0

I've created two variables e and v. When I use the e variable with eval() the end result of the statment applies back to the main context. However when I try it with v and vm.runInThisContext() we get an acception because v is not defined. While eval() runs as the local scope vm does not. However, if we create v within the scope of the vm then it is available subsiquently in the state we left it. Finally we check that v from the vm has no impact on v in the local scope of the main event loop, which is does not.

It's also possible to pass a pre-existing context to vm.

Part III. API Documentation

Chapter 8. API Documentation

TTY

Use require('tty') to access this module.

tty.open(path, args=[])

Spawns a new process with the executable pointed to by path as the session leader to a new pseudo terminal.

Returns an array [slaveFD, childProcess]. slaveFD is the file descriptor of the slave end of the pseudo terminal. childProcess is a child process object.

tty.isatty(fd)

Returns true or false depending on if the fd is associated with a terminal.

tty.setRawMode(mode)

mode should be true or false. This sets the properies of the current process's stdin fd to act either as a raw device or default.

tty.setWindowSize(fd, row, col)

ioctls the window size settings to the file descriptor.

tty.getWindowSize(fd)

Returns [row, col] for the TTY associated with the file descriptor.

Debugger

V8 comes with an extensive debugger which is accessable out-of-process via a simple TCP protocol. Node has a built-in client for this debugger. To use this, start Node with the debug argument; a prompt will appear:

Example 8-1.  

% node debug myscript.js
debug>

At this point myscript.js is not yet running. To start the script, enter the command run. If everything works okay, the output should look like this:

Example 8-2.  

% node debug myscript.js
debug> run
debugger listening on port 5858
connecting...ok

Node's debugger client doesn't support the full range of commands, but simple step and inspection is possible. By putting the statement debugger; into the source code of your script, you will enable a breakpoint.

For example, suppose myscript.js looked like this:

Example 8-3.  

// myscript.js
x = 5;
setTimeout(function () {
  debugger;
  console.log("world");
}, 1000);
console.log("hello");

Then once the debugger is run, it will break on line 4.

Example 8-4.  

% ./node debug myscript.js
debug> run
debugger listening on port 5858
connecting...ok
hello
break in #<an Object>._onTimeout(), myscript.js:4
  debugger;
  ^
debug> next
break in #<an Object>._onTimeout(), myscript.js:5
  console.log("world");
  ^
debug> print x
5
debug> print 2+2
4
debug> next
world
break in #<an Object>._onTimeout() returning undefined, myscript.js:6
}, 1000);
^
debug> quit
A debugging session is active. Quit anyway? (y or n) y
%

The print command allows you to evaluate variables. The next command steps over to the next line. There are a few other commands available and more to come type help to see others.

Advanced Usage

The V8 debugger can be enabled and accessed either by starting Node with the --debug command-line flag or by signaling an existing Node process with SIGUSR1.

Assert

This module is used for writing unit tests for your applications, you can access it with require('assert').

assert.fail(actual, expected, message, operator)

Tests if actual is equal to expected using the operator provided.

assert.ok(value, [message])

Tests if value is a true value, it is equivalent to assert.equal(true, value, message);

assert.equal(actual, expected, [message])

Tests shallow, coercive equality with the equal comparison operator ( == ).

assert.notEqual(actual, expected, [message])

Tests shallow, coercive non-equality with the not equal comparison operator ( != ).

assert.deepEqual(actual, expected, [message])

Tests for deep equality.

assert.notDeepEqual(actual, expected, [message])

Tests for any deep inequality.

assert.strictEqual(actual, expected, [message])

Tests strict equality, as determined by the strict equality operator ( === )

assert.notStrictEqual(actual, expected, [message])

Tests strict non-equality, as determined by the strict not equal operator ( !== )

assert.throws(block, [error], [message])

Expects block to throw an error. error can be constructor, regexp or validation function.

Validate instanceof using constructor:

Example 8-5.  

assert.throws(
  function() {
    throw new Error("Wrong value");
  },
  Error
);

Validate error message using RegExp:

Example 8-6.  

assert.throws(
  function() {
    throw new Error("Wrong value");
  },
  /value/
);

Custom error validation:

Example 8-7.  

assert.throws(
  function() {
    throw new Error("Wrong value");
  },
  function(err) {
    if ( (err instanceof Error) && /value/.test(err) ) {
      return true;
    }
  },
  "unexpected error"
);

assert.doesNotThrow(block, [error], [message])

Expects block not to throw an error, see assert.throws for details.

assert.ifError(value)

Tests if value is not a false value, throws if it is a true value. Useful when testing the first argument, error in callbacks.

Child Processes

Node provides a tri-directional popen(3) facility through the ChildProcess class.

It is possible to stream data through the child's stdin, stdout, and stderr in a fully non-blocking way.

To create a child process use require('child_process').spawn().

Child processes always have three streams associated with them. child.stdin, child.stdout, and child.stderr.

ChildProcess is an EventEmitter.

Event: 'exit'

function (code, signal) {}

This event is emitted after the child process ends. If the process terminated normally, code is the final exit code of the process, otherwise null. If the process terminated due to receipt of a signal, signal is the string name of the signal, otherwise null.

See waitpid(2).

child.stdin

A Writable Stream that represents the child process's stdin. Closing this stream via end() often causes the child process to terminate.

child.stdout

A Readable Stream that represents the child process's stdout.

child.stderr

A Readable Stream that represents the child process's stderr.

child.pid

The PID of the child process.

Example:

Example 8-8.  

var spawn = require('child_process').spawn,
    grep  = spawn('grep', ['ssh']);

console.log('Spawned child pid: ' + grep.pid);
grep.stdin.end();

child_process.spawn(command, args=[], [options])

Launches a new process with the given command, with command line arguments in args. If omitted, args defaults to an empty Array.

The third argument is used to specify additional options, which defaults to:

Example 8-9.  

{ cwd: undefined,
  env: process.env,
  customFds: [-1, -1, -1],
  setsid: false
}

cwd allows you to specify the working directory from which the process is spawned. Use env to specify environment variables that will be visible to the new process. With customFds it is possible to hook up the new process' [stdin, stout, stderr] to existing streams; -1 means that a new stream should be created. setsid, if set true, will cause the subprocess to be run in a new session.

Example of running ls -lh /usr, capturing stdout, stderr, and the exit code:

Example 8-10.  

var util   = require('util'),
    spawn = require('child_process').spawn,
    ls    = spawn('ls', ['-lh', '/usr']);

ls.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

ls.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

ls.on('exit', function (code) {
  console.log('child process exited with code ' + code);
});

Example: A very elaborate way to run 'ps ax | grep ssh'

Example 8-11.  

var util   = require('util'),
    spawn = require('child_process').spawn,
    ps    = spawn('ps', ['ax']),
    grep  = spawn('grep', ['ssh']);

ps.stdout.on('data', function (data) {
  grep.stdin.write(data);
});

ps.stderr.on('data', function (data) {
  console.log('ps stderr: ' + data);
});

ps.on('exit', function (code) {
  if (code !== 0) {
    console.log('ps process exited with code ' + code);
  }
  grep.stdin.end();
});

grep.stdout.on('data', function (data) {
  console.log(data);
});

grep.stderr.on('data', function (data) {
  console.log('grep stderr: ' + data);
});

grep.on('exit', function (code) {
  if (code !== 0) {
    console.log('grep process exited with code ' + code);
  }
});

Example of checking for failed exec:

Example 8-12.  

var spawn = require('child_process').spawn,
    child = spawn('bad_command');

child.stderr.on('data', function (data) {
  if (/^execvp\(\)/.test(data.asciiSlice(0,data.length))) {
    console.log('Failed to start child process.');
  }
});

See also: child_process.exec()

child_process.exec(command, [options], callback)

High-level way to execute a command as a child process, buffer the output, and return it all in a callback.

Example 8-13.  

var util   = require('util'),
    exec  = require('child_process').exec,
    child;

child = exec('cat *.js bad_file | wc -l',
  function (error, stdout, stderr) {
    console.log('stdout: ' + stdout);
    console.log('stderr: ' + stderr);
    if (error !== null) {
      console.log('exec error: ' + error);
    }
});

The callback gets the arguments (error, stdout, stderr). On success, error will be null. On error, error will be an instance of Error and err.code will be the exit code of the child process, and err.signal will be set to the signal that terminated the process.

There is a second optional argument to specify several options. The default options are

Example 8-14.  

{ encoding: 'utf8',
  timeout: 0,
  maxBuffer: 200*1024,
  killSignal: 'SIGTERM',
  cwd: null,
  env: null }

If timeout is greater than 0, then it will kill the child process if it runs longer than timeout milliseconds. The child process is killed with killSignal (default: 'SIGTERM'). maxBuffer specifies the largest amount of data allowed on stdout or stderr - if this value is exceeded then the child process is killed.

child.kill(signal='SIGTERM')

Send a signal to the child process. If no argument is given, the process will be sent 'SIGTERM'. See signal(7) for a list of available signals.

Example 8-15.  

var spawn = require('child_process').spawn,
    grep  = spawn('grep', ['ssh']);

grep.on('exit', function (code, signal) {
  console.log('child process terminated due to receipt of signal '+signal);
});

// send SIGHUP to process
grep.kill('SIGHUP');

Note that while the function is called kill, the signal delivered to the child process may not actually kill it. kill really just sends a signal to a process.

See kill(2)

REPL

A Read-Eval-Print-Loop (REPL) is available both as a standalone program and easily includable in other programs. REPL provides a way to interactively run JavaScript and see the results. It can be used for debugging, testing, or just trying things out.

By executing node without any arguments from the command-line you will be dropped into the REPL. It has simplistic emacs line-editing.

Example 8-16.  

mjr:~$ node
Type '.help' for options.
> a = [ 1, 2, 3];
[ 1, 2, 3 ]
> a.forEach(function (v) {
...   console.log(v);
...   });
1
2
3

For advanced line-editors, start node with the environmental variable NODE_NO_READLINE=1. This will start the REPL in canonical terminal settings which will allow you to use with rlwrap.

For example, you could add this to your bashrc file:

Example 8-17.  

alias node="env NODE_NO_READLINE=1 rlwrap node"

repl.start(prompt='> ', stream=process.stdin)

Starts a REPL with prompt as the prompt and stream for all I/O. prompt is optional and defaults to >. stream is optional and defaults to process.stdin.

Multiple REPLs may be started against the same running instance of node. Each will share the same global object but will have unique I/O.

Here is an example that starts a REPL on stdin, a Unix socket, and a TCP socket:

Example 8-18.  

var net = require("net"),
    repl = require("repl");

connections = 0;

repl.start("node via stdin> ");

net.createServer(function (socket) {
  connections += 1;
  repl.start("node via Unix socket> ", socket);
}).listen("/tmp/node-repl-sock");

net.createServer(function (socket) {
  connections += 1;
  repl.start("node via TCP socket> ", socket);
}).listen(5001);

Running this program from the command line will start a REPL on stdin. Other REPL clients may connect through the Unix socket or TCP socket. telnet is useful for connecting to TCP sockets, and socat can be used to connect to both Unix and TCP sockets.

By starting a REPL from a Unix socket-based server instead of stdin, you can connect to a long-running node process without restarting it.

REPL Features

Inside the REPL, Control+D will exit. Multi-line expressions can be input.

The special variable _ (underscore) contains the result of the last expression.

Example 8-19.  

> [ "a", "b", "c" ]
[ 'a', 'b', 'c' ]
> _.length
3
> _ += 1
4

The REPL provides access to any variables in the global scope. You can expose a variable to the REPL explicitly by assigning it to the context object associated with each REPLServer. For example:

Example 8-20.  

// repl_test.js
var repl = require("repl"),
    msg = "message";

repl.start().context.m = msg;

Things in the context object appear as local within the REPL:

Example 8-21.  

mjr:~$ node repl_test.js
> m
'message'

There are a few special REPL commands:

.break - While inputting a multi-line expression, sometimes you get lost or just don't care about completing it. .break will start over.

.clear - Resets the context object to an empty object and clears any multi-line expression.

.exit - Close the I/O stream, which will cause the REPL to exit.

.help - Show this list of special commands.

Query String

This module provides utilities for dealing with query strings. It provides the following methods:

querystring.stringify(obj, sep='&', eq='=')

Serialize an object to a query string. Optionally override the default separator and assignment characters.

Example:

Example 8-22.  

querystring.stringify({foo: 'bar'})
// returns
'foo=bar'

querystring.stringify({foo: 'bar', baz: 'bob'}, ';', ':')
// returns
'foo:bar;baz:bob'

querystring.parse(str, sep='&', eq='=')

Deserialize a query string to an object. Optionally override the default separator and assignment characters.

Example:

Example 8-23.  

querystring.parse('a=b&b=c')
// returns
{ a: 'b', b: 'c' }

querystring.escape

The escape function used by querystring.stringify, provided so that it could be overridden if necessary.

querystring.unescape

The unescape function used by querystring.parse, provided so that it could be overridden if necessary.

URL

This module has utilities for URL resolution and parsing. Call require('url') to use it.

Parsed URL objects have some or all of the following fields, depending on whether or not they exist in the URL string. Any parts that are not in the URL string will not be in the parsed object. Examples are shown for the URL

'http://user:pass@host.com:8080/p/a/t/h?query=string#hash'

href: The full URL that was originally parsed.

Example: 'http://user:pass@host.com:8080/p/a/t/h?query=string#hash' * protocol: The request protocol.

Example: 'http:' * host: The full host portion of the URL, including port and authentication information.

Example: 'user:pass@host.com:8080' * auth: The authentication information portion of a URL.

Example: 'user:pass' * hostname: Just the hostname portion of the host.

Example: 'host.com' * port: The port number portion of the host.

Example: '8080' * pathname: The path section of the URL, that comes after the host and before the query, including the initial slash if present.

Example: '/p/a/t/h' * search: The 'query string' portion of the URL, including the leading question mark.

Example: '?query=string' * query: Either the 'params' portion of the query string, or a querystring-parsed object.

Example: 'query=string' or {'query':'string'} * hash: The 'fragment' portion of the URL including the pound-sign.

Example: '#hash'

The following methods are provided by the URL module:

url.parse(urlStr, parseQueryString=false)

Take a URL string, and return an object. Pass true as the second argument to also parse the query string using the querystring module.

url.format(urlObj)

Take a parsed URL object, and return a formatted URL string.

url.resolve(from, to)

Take a base URL, and a href URL, and resolve them as a browser would for an anchor tag.

HTTPS

HTTPS is the HTTP protocol over TLS/SSL. In Node this is implemented as a separate module.

https.Server

https.createServer

Example:

Example 8-24.  

// curl -k https://localhost:8000/
var https = require('https');
var fs = require('fs');

var options = {
  key: fs.readFileSync('test/fixtures/keys/agent2-key.pem'),
  cert: fs.readFileSync('test/fixtures/keys/agent2-cert.pem')
};

https.createServer(options, function (req, res) {
  res.writeHead(200);
  res.end("hello world\n");
}).listen(8000);

https.request(options, callback)

Makes a request to a secure web server. Similar options to http.request().

Example:

Example 8-25.  

var https = require('https');

var options = {
  host: 'encrypted.google.com',
  port: 443,
  path: '/',
  method: 'GET'
};

var req = https.request(options, function(res) {
  console.log("statusCode: ", res.statusCode);
  console.log("headers: ", res.headers);

  res.on('data', function(d) {
    process.stdout.write(d);
  });
});
req.end();

req.on('error', function(e) {
  console.error(e);
});

https.get(options, callback)

Like http.get() but for HTTPS.

Example:

Example 8-26.  

var https = require('https');

https.get({ host: 'encrypted.google.com', path: '/' }, function(res) {
  console.log("statusCode: ", res.statusCode);
  console.log("headers: ", res.headers);

  res.on('data', function(d) {
    process.stdout.write(d);
  });

}).on('error', function(e) {
  console.error(e);
});

UDP / Datagram Sockets

Datagram sockets are available through require('dgram'). Datagrams are most commonly handled as IP/UDP messages but they can also be used over Unix domain sockets.

Event: 'message'

function (msg, rinfo) { }

Emitted when a new datagram is available on a socket. msg is a Buffer and rinfo is an object with the sender's address information and the number of bytes in the datagram.

Event: 'listening'

function () { }

Emitted when a socket starts listening for datagrams. This happens as soon as UDP sockets are created. Unix domain sockets do not start listening until calling bind() on them.

Event: 'close'

function () { }

Emitted when a socket is closed with close(). No new message events will be emitted on this socket.

dgram.createSocket(type, [callback])

Creates a datagram socket of the specified types. Valid types are: udp4, udp6, and unix_dgram.

Takes an optional callback which is added as a listener for message events.

dgram.send(buf, offset, length, path, [callback])

For Unix domain datagram sockets, the destination address is a pathname in the filesystem. An optional callback may be supplied that is invoked after the sendto call is completed by the OS. It is not safe to re-use buf until the callback is invoked. Note that unless the socket is bound to a pathname with bind() there is no way to receive messages on this socket.

Example of sending a message to syslogd on OSX via Unix domain socket /var/run/syslog:

Example 8-27.  

var dgram = require('dgram');
var message = new Buffer("A message to log.");
var client = dgram.createSocket("unix_dgram");
client.send(message, 0, message.length, "/var/run/syslog",
  function (err, bytes) {
    if (err) {
      throw err;
    }
    console.log("Wrote " + bytes + " bytes to socket.");
});

dgram.send(buf, offset, length, port, address, [callback])

For UDP sockets, the destination port and IP address must be specified. A string may be supplied for the address parameter, and it will be resolved with DNS. An optional callback may be specified to detect any DNS errors and when buf may be re-used. Note that DNS lookups will delay the time that a send takes place, at least until the next tick. The only way to know for sure that a send has taken place is to use the callback.

Example of sending a UDP packet to a random port on localhost;

Example 8-28.  

var dgram = require('dgram');
var message = new Buffer("Some bytes");
var client = dgram.createSocket("udp4");
client.send(message, 0, message.length, 41234, "localhost");
client.close();

dgram.bind(path)

For Unix domain datagram sockets, start listening for incoming datagrams on a socket specified by path. Note that clients may send() without bind(), but no datagrams will be received without a bind().

Example of a Unix domain datagram server that echoes back all messages it receives:

Example 8-29.  

var dgram = require("dgram");
var serverPath = "/tmp/dgram_server_sock";
var server = dgram.createSocket("unix_dgram");

server.on("message", function (msg, rinfo) {
  console.log("got: " + msg + " from " + rinfo.address);
  server.send(msg, 0, msg.length, rinfo.address);
});

server.on("listening", function () {
  console.log("server listening " + server.address().address);
})

server.bind(serverPath);

Example of a Unix domain datagram client that talks to this server:

Example 8-30.  

var dgram = require("dgram");
var serverPath = "/tmp/dgram_server_sock";
var clientPath = "/tmp/dgram_client_sock";

var message = new Buffer("A message at " + (new Date()));

var client = dgram.createSocket("unix_dgram");

client.on("message", function (msg, rinfo) {
  console.log("got: " + msg + " from " + rinfo.address);
});

client.on("listening", function () {
  console.log("client listening " + client.address().address);
  client.send(message, 0, message.length, serverPath);
});

client.bind(clientPath);

dgram.bind(port, [address])

For UDP sockets, listen for datagrams on a named port and optional address. If address is not specified, the OS will try to listen on all addresses.

Example of a UDP server listening on port 41234:

Example 8-31.  

var dgram = require("dgram");

var server = dgram.createSocket("udp4");
var messageToSend = new Buffer("A message to send");

server.on("message", function (msg, rinfo) {
  console.log("server got: " + msg + " from " +
    rinfo.address + ":" + rinfo.port);
});

server.on("listening", function () {
  var address = server.address();
  console.log("server listening " +
      address.address + ":" + address.port);
});

server.bind(41234);
// server listening 0.0.0.0:41234

dgram.close()

Close the underlying socket and stop listening for data on it. UDP sockets automatically listen for messages, even if they did not call bind().

dgram.address()

Returns an object containing the address information for a socket. For UDP sockets, this object will contain address and port. For Unix domain sockets, it will contain only address.

dgram.setBroadcast(flag)

Sets or clears the SO_BROADCAST socket option. When this option is set, UDP packets may be sent to a local interface's broadcast address.

dgram.setTTL(ttl)

Sets the IP_TTL socket option. TTL stands for "Time to Live," but in this context it specifies the number of IP hops that a packet is allowed to go through. Each router or gateway that forwards a packet decrements the TTL. If the TTL is decremented to 0 by a router, it will not be forwarded. Changing TTL values is typically done for network probes or when multicasting.

The argument to setTTL() is a number of hops between 1 and 255. The default on most systems is 64.

dgram.setMulticastTTL(ttl)

Sets the IP_MULTICAST_TTL socket option. TTL stands for "Time to Live," but in this context it specifies the number of IP hops that a packet is allowed to go through, specifically for multicast traffic. Each router or gateway that forwards a packet decrements the TTL. If the TTL is decremented to 0 by a router, it will not be forwarded.

The argument to setMulticastTTL() is a number of hops between 0 and 255. The default on most systems is 64.

dgram.setMulticastLoopback(flag)

Sets or clears the IP_MULTICAST_LOOP socket option. When this option is set, multicast packets will also be received on the local interface.

dgram.addMembership(multicastAddress, [multicastInterface])

Tells the kernel to join a multicast group with IP_ADD_MEMBERSHIP socket option.

If multicastAddress is not specified, the OS will try to add membership to all valid interfaces.

dgram.dropMembership(multicastAddress, [multicastInterface])

Opposite of addMembership - tells the kernel to leave a multicast group with IP_DROP_MEMBERSHIP socket option. This is automatically called by the kernel when the socket is closed or process terminates, so most apps will never need to call this.

If multicastAddress is not specified, the OS will try to drop membership to all valid interfaces.

net

The net module provides you with an asynchronous network wrapper. It contains methods for creating both servers and clients (called streams). You can include this module with require("net");

net.createServer(connectionListener)

Creates a new TCP server. The connectionListener argument is automatically set as a listener for the 'connection' event.

net.createConnection(arguments...)

Construct a new socket object and opens a socket to the given location. When the socket is established the 'connect' event will be emitted.

The arguments for this method change the type of connection:

net.createConnection(port, [host])

Creates a TCP connection to port on host. If host is omitted, localhost will be assumed.

net.createConnection(path)

Creates unix socket connection to path

net.Server

This class is used to create a TCP or UNIX server.

Here is an example of a echo server which listens for connections on port 8124:

Example 8-32.  

var net = require('net');
var server = net.createServer(function (c) {
  c.write('hello\r\n');
  c.pipe(c);
});
server.listen(8124, 'localhost');

Test this by using telnet:

Example 8-33.  

telnet localhost 8124

To listen on the socket /tmp/echo.sock the last line would just be changed to

Example 8-34.  

server.listen('/tmp/echo.sock');

Use nc to connect to a UNIX domain socket server:

Example 8-35.  

nc -U /tmp/echo.sock

net.Server is an EventEmitter with the following events:

server.listen(port, [host], [callback])

Begin accepting connections on the specified port and host. If the host is omitted, the server will accept connections directed to any IPv4 address (INADDR_ANY).

This function is asynchronous. The last parameter callback will be called when the server has been bound.

One issue some users run into is getting EADDRINUSE errors. Meaning another server is already running on the requested port. One way of handling this would be to wait a second and the try again. This can be done with

Example 8-36.  

server.on('error', function (e) {
  if (e.code == 'EADDRINUSE') {
    console.log('Address in use, retrying...');
    setTimeout(function () {
      server.close();
      server.listen(PORT, HOST);
    }, 1000);
  }
});

(Note: All sockets in Node are set SO_REUSEADDR already)

server.listen(path, [callback])

Start a UNIX socket server listening for connections on the given path.

This function is asynchronous. The last parameter callback will be called when the server has been bound.

server.listenFD(fd)

Start a server listening for connections on the given file descriptor.

This file descriptor must have already had the bind(2) and listen(2) system calls invoked on it.

server.close()

Stops the server from accepting new connections. This function is asynchronous, the server is finally closed when the server emits a 'close' event.

server.address()

Returns the bound address of the server as seen by the operating system. Useful to find which port was assigned when giving getting an OS-assigned address

Example:

Example 8-37.  

var server = net.createServer(function (socket) {
  socket.end("goodbye\n");
});

// grab a random port.
server.listen(function() {
  address = server.address();
  console.log("opened server on undefined", address);
});

server.maxConnections

Set this property to reject connections when the server's connection count gets high.

server.connections

The number of concurrent connections on the server.

Event: 'connection'

function (socket) {}

Emitted when a new connection is made. socket is an instance of net.Socket.

Event: 'close'

function () {}

Emitted when the server closes.

net.Socket

This object is an abstraction of of a TCP or UNIX socket. net.Socket instances implement a duplex Stream interface. They can be created by the user and used as a client (with connect()) or they can be created by Node and passed to the user through the 'connection' event of a server.

net.Socket instances are EventEmitters with the following events:

socket.connect(port, [host], [callback])

socket.connect(path, [callback])

Opens the connection for a given socket. If port and host are given, then the socket will be opened as a TCP socket, if host is omitted, localhost will be assumed. If a path is given, the socket will be opened as a unix socket to that path.

Normally this method is not needed, as net.createConnection opens the socket. Use this only if you are implementing a custom Socket or if a Socket is closed and you want to reuse it to connect to another server.

This function is asynchronous. When the 'connect' event is emitted the socket is established. If there is a problem connecting, the 'connect' event will not be emitted, the 'error' event will be emitted with the exception.

The callback paramenter will be added as an listener for the 'connect' event.

socket.bufferSize

net.Socket has the property that socket.write() always works. This is to help users get up an running quickly. The computer cannot necessarily keep up with the amount of data that is written to a socket - the network connection simply might be too slow. Node will internally queue up the data written to a socket and send it out over the wire when it is possible. (Internally it is polling on the socket's file descriptor for being writable).

The consequence of this internal buffering is that memory may grow. This property shows the number of characters currently buffered to be written. (Number of characters is approximately equal to the number of bytes to be written, but the buffer may contain strings, and the strings are lazily encoded, so the exact number of bytes is not known.)

Users who experience large or growing bufferSize should attempt to "throttle" the data flows in their program with pause() and resume()`.

socket.setEncoding(encoding=null)

Sets the encoding (either 'ascii', 'utf8', or 'base64') for data that is received.

socket.setSecure()

This function has been removed in v0.3. It used to upgrade the connection to SSL/TLS. See the TLS for the new API.

socket.write(data, [encoding], [callback])

Sends data on the socket. The second parameter specifies the encoding in the case of a string--it defaults to UTF8 encoding.

Returns true if the entire data was flushed successfully to the kernel buffer. Returns false if all or part of the data was queued in user memory. 'drain' will be emitted when the buffer is again free.

The optional callback parameter will be executed when the data is finally written out - this may not be immediately.

socket.write(data, [encoding], [fileDescriptor], [callback])

For UNIX sockets, it is possible to send a file descriptor through the socket. Simply add the fileDescriptor argument and listen for the 'fd' event on the other end.

socket.end([data], [encoding])

Half-closes the socket. I.E., it sends a FIN packet. It is possible the server will still send some data.

If data is specified, it is equivalent to calling socket.write(data, encoding) followed by socket.end().

socket.destroy()

Ensures that no more I/O activity happens on this socket. Only necessary in case of errors (parse error or so).

socket.pause()

Pauses the reading of data. That is, 'data' events will not be emitted. Useful to throttle back an upload.

socket.resume()

Resumes reading after a call to pause().

socket.setTimeout(timeout, [callback])

Sets the socket to timeout after timeout milliseconds of inactivity on the socket. By default net.Socket do not have a timeout.

When an idle timeout is triggered the socket will receive a 'timeout' event but the connection will not be severed. The user must manually end() or destroy() the socket.

If timeout is 0, then the existing idle timeout is disabled.

The optional callback parameter will be added as a one time listener for the 'timeout' event.

socket.setNoDelay(noDelay=true)

Disables the Nagle algorithm. By default TCP connections use the Nagle algorithm, they buffer data before sending it off. Setting noDelay will immediately fire off data each time socket.write() is called.

socket.setKeepAlive(enable=false, [initialDelay])

Enable/disable keep-alive functionality, and optionally set the initial delay before the first keepalive probe is sent on an idle socket. Set initialDelay (in milliseconds) to set the delay between the last data packet received and the first keepalive probe. Setting 0 for initialDelay will leave the value unchanged from the default (or previous) setting.

socket.remoteAddress

The string representation of the remote IP address. For example, '74.125.127.100' or '2001:4860:a005::68'.

This member is only present in server-side connections.

Event: 'connect'

function () { }

Emitted when a socket connection successfully is established. See connect().

Event: 'data'

function (data) { }

Emitted when data is received. The argument data will be a Buffer or String. Encoding of data is set by socket.setEncoding(). (See the section on Readable Socket for more information.)

Event: 'end'

function () { }

Emitted when the other end of the socket sends a FIN packet.

By default (allowHalfOpen == false) the socket will destroy its file descriptor once it has written out its pending write queue. However, by setting allowHalfOpen == true the socket will not automatically end() its side allowing the user to write arbitrary amounts of data, with the caveat that the user is required to end() their side now.

Event: 'timeout'

function () { }

Emitted if the socket times out from inactivity. This is only to notify that the socket has been idle. The user must manually close the connection.

See also: socket.setTimeout()

Event: 'drain'

function () { }

Emitted when the write buffer becomes empty. Can be used to throttle uploads.

Event: 'error'

function (exception) { }

Emitted when an error occurs. The 'close' event will be called directly following this event.

Event: 'close'

function (had_error) { }

Emitted once the socket is fully closed. The argument had_error is a boolean which says if the socket was closed due to a transmission error.

net.isIP

net.isIP(input)

Tests if input is an IP address. Returns 0 for invalid strings, returns 4 for IP version 4 addresses, and returns 6 for IP version 6 addresses.

net.isIPv4(input)

Returns true if input is a version 4 IP address, otherwise returns false.

net.isIPv6(input)

Returns true if input is a version 6 IP address, otherwise returns false.

Path

This module contains utilities for dealing with file paths. Use require('path') to use it. It provides the following methods:

path.normalize(p)

Normalize a string path, taking care of '..' and '.' parts.

When multiple slashes are found, they're replaces by a single one; when the path contains a trailing slash, it is preserved. On windows backslashes are used.

Example:

Example 8-38.  

path.normalize('/foo/bar//baz/asdf/quux/..')
// returns
'/foo/bar/baz/asdf'

path.join([path1], [path2], [...])

Join all arguments together and normalize the resulting path.

Example:

Example 8-39.  

node> require('path').join(
...   '/foo', 'bar', 'baz/asdf', 'quux', '..')
'/foo/bar/baz/asdf'

path.resolve([from ...], to)

Resolves to to an absolute path.

If to isn't already absolute from arguments are prepended in right to left order, until an absolute path is found. If after using all from paths still no absolute path is found, the current working directory is used as well. The resulting path is normalized, and trailing slashes are removed unless the path gets resolved to the root directory.

Another way to think of it is as a sequence of cd commands in a shell.

Example 8-40.  

path.resolve('foo/bar', '/tmp/file/', '..', 'a/../subfile')

Is similar to:

Example 8-41.  

cd foo/bar
cd /tmp/file/
cd ..
cd a/../subfile
pwd

The difference is that the different paths don't need to exist and may also be files.

Examples:

Example 8-42.  

path.resolve('/foo/bar', './baz')
// returns
'/foo/bar/baz'

path.resolve('/foo/bar', '/tmp/file/')
// returns
'/tmp/file'

path.resolve('wwwroot', 'static_files/png/', '../gif/image.gif')
// if currently in /home/myself/node, it returns
'/home/myself/node/wwwroot/static_files/gif/image.gif'

path.dirname(p)

Return the directory name of a path. Similar to the Unix dirname command.

Example:

Example 8-43.  

path.dirname('/foo/bar/baz/asdf/quux')
// returns
'/foo/bar/baz/asdf'

path.basename(p, [ext])

Return the last portion of a path. Similar to the Unix basename command.

Example:

Example 8-44.  

path.basename('/foo/bar/baz/asdf/quux.html')
// returns
'quux.html'

path.basename('/foo/bar/baz/asdf/quux.html', '.html')
// returns
'quux'

path.extname(p)

Return the extension of the path. Everything after the last '.' in the last portion of the path. If there is no '.' in the last portion of the path or the only '.' is the first character, then it returns an empty string. Examples:

Example 8-45.  

path.extname('index.html')
// returns
'.html'

path.extname('index')
// returns
''

path.exists(p, [callback])

Test whether or not the given path exists. Then, call the callback argument with either true or false. Example:

Example 8-46.  

path.exists('/etc/passwd', function (exists) {
  util.debug(exists ? "it's there" : "no passwd!");
});

os Module

Use require('os') to access this module.

os.hostname()

Returns the hostname of the operating system.

os.type()

Returns the operating system name.

os.release()

Returns the operating system release.

os.uptime()

Returns the system uptime in seconds.

os.loadavg()

Returns an array containing the 1, 5, and 15 minute load averages.

os.totalmem()

Returns the total amount of system memory in bytes.

os.freemem()

Returns the amount of free system memory in bytes.

os.cpus()

Returns an array of objects containing information about each CPU/core installed: model, speed (in MHz), and times (an object containing the number of CPU ticks spent in: user, nice, sys, idle, and irq).

Example inspection of os.cpus:

Example 8-47.  

[ { model: 'Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz',
    speed: 2926,
    times:
     { user: 252020,
       nice: 0,
       sys: 30340,
       idle: 1070356870,
       irq: 0 } },
  { model: 'Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz',
    speed: 2926,
    times:
     { user: 306960,
       nice: 0,
       sys: 26980,
       idle: 1071569080,
       irq: 0 } },
  { model: 'Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz',
    speed: 2926,
    times:
     { user: 248450,
       nice: 0,
       sys: 21750,
       idle: 1070919370,
       irq: 0 } },
  { model: 'Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz',
    speed: 2926,
    times:
     { user: 256880,
       nice: 0,
       sys: 19430,
       idle: 1070905480,
       irq: 20 } },
  { model: 'Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz',
    speed: 2926,
    times:
     { user: 511580,
       nice: 20,
       sys: 40900,
       idle: 1070842510,
       irq: 0 } },
  { model: 'Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz',
    speed: 2926,
    times:
     { user: 291660,
       nice: 0,
       sys: 34360,
       idle: 1070888000,
       irq: 10 } },
  { model: 'Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz',
    speed: 2926,
    times:
     { user: 308260,
       nice: 0,
       sys: 55410,
       idle: 1071129970,
       irq: 880 } },
  { model: 'Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz',
    speed: 2926,
    times:
     { user: 266450,
       nice: 1480,
       sys: 34920,
       idle: 1072572010,
       irq: 30 } } ]

TLS (SSL)

Use require('tls') to access this module.

The tls module uses OpenSSL to provide Transport Layer Security and/or Secure Socket Layer: encrypted stream communication.

TLS/SSL is a public/private key infrastructure. Each client and each server must have a private key. A private key is created like this

Example 8-48.  

openssl genrsa -out ryans-key.pem 1024

All severs and some clients need to have a certificate. Certificates are public keys signed by a Certificate Authority or self-signed. The first step to getting a certificate is to create a "Certificate Signing Request" (CSR) file. This is done with:

Example 8-49.  

openssl req -new -key ryans-key.pem -out ryans-csr.pem

To create a self-signed certificate with the CSR, do this:

Example 8-50.  

openssl x509 -req -in ryans-csr.pem -signkey ryans-key.pem -out ryans-cert.pem

Alternatively you can send the CSR to a Certificate Authority for signing.

(TODO: docs on creating a CA, for now interested users should just look at test/fixtures/keys/Makefile in the Node source code)

s = tls.connect(port, [host], [options], callback)

Creates a new client connection to the given port and host. (If host defaults to localhost.) options should be an object which specifies

key: A string or Buffer containing the private key of the server in PEM format. (Required)

cert: A string or Buffer containing the certificate key of the server in PEM format.

ca: An array of strings or Buffers of trusted certificates. If this is omitted several well known "root" CAs will be used, like VeriSign. These are used to authorize connections.

tls.connect() returns a cleartext CryptoStream object.

After the TLS/SSL handshake the callback is called. The callback will be called no matter if the server's certificate was authorized or not. It is up to the user to test s.authorized to see if the server certificate was signed by one of the specified CAs. If s.authorized === false then the error can be found in s.authorizationError.

tls.Server

This class is a subclass of net.Server and has the same methods on it. Instead of accepting just raw TCP connections, this accepts encrypted connections using TLS or SSL.

Here is a simple example echo server:

Example 8-51.  

var tls = require('tls');
var fs = require('fs');

var options = {
  key: fs.readFileSync('server-key.pem'),
  cert: fs.readFileSync('server-cert.pem')
};

tls.createServer(options, function (s) {
  s.write("welcome!\n");
  s.pipe(s);
}).listen(8000);

You can test this server by connecting to it with openssl s_client:

Example 8-52.  

openssl s_client -connect 127.0.0.1:8000

tls.createServer(options, secureConnectionListener)

This is a constructor for the tls.Server class. The options object has these possibilities:

key: A string or Buffer containing the private key of the server in PEM format. (Required)

cert: A string or Buffer containing the certificate key of the server in PEM format. (Required)

ca: An array of strings or Buffers of trusted certificates. If this is omitted several well known "root" CAs will be used, like VeriSign. These are used to authorize connections.

requestCert: If true the server will request a certificate from clients that connect and attempt to verify that certificate. Default: false.

rejectUnauthorized: If true the server will reject any connection which is not authorized with the list of supplied CAs. This option only has an effect if requestCert is true. Default: false.

Event: 'secureConnection'

function (cleartextStream) {}

This event is emitted after a new connection has been successfully handshaked. The argument is a duplex instance of stream.Stream. It has all the common stream methods and events.

cleartextStream.authorized is a boolean value which indicates if the client has verified by one of the supplied cerificate authorities for the server. If cleartextStream.authorized is false, then cleartextStream.authorizationError is set to describe how authorization failed. Implied but worth mentioning: depending on the settings of the TLS server, you unauthorized connections may be accepted.

server.listen(port, [host], [callback])

Begin accepting connections on the specified port and host. If the host is omitted, the server will accept connections directed to any IPv4 address (INADDR_ANY).

This function is asynchronous. The last parameter callback will be called when the server has been bound.

See net.Server for more information.

server.close()

Stops the server from accepting new connections. This function is asynchronous, the server is finally closed when the server emits a 'close' event.

server.maxConnections

Set this property to reject connections when the server's connection count gets high.

server.connections

The number of concurrent connections on the server.

Crypto

Use require('crypto') to access this module.

The crypto module requires OpenSSL to be available on the underlying platform. It offers a way of encapsulating secure credentials to be used as part of a secure HTTPS net or http connection.

It also offers a set of wrappers for OpenSSL's hash, hmac, cipher, decipher, sign and verify methods.

crypto.createCredentials(details)

Creates a credentials object, with the optional details being a dictionary with keys:

key : a string holding the PEM encoded private key

cert : a string holding the PEM encoded certificate

ca : either a string or list of strings of PEM encoded CA certificates to trust.

If no 'ca' details are given, then node.js will use the default publicly trusted list of CAs as given in http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt.

crypto.createHash(algorithm)

Creates and returns a hash object, a cryptographic hash with the given algorithm which can be used to generate hash digests.

algorithm is dependent on the available algorithms supported by the version of OpenSSL on the platform. Examples are 'sha1', 'md5', 'sha256', 'sha512', etc. On recent releases, openssl list-message-digest-algorithms will display the available digest algorithms.

hash.update(data)

Updates the hash content with the given data. This can be called many times with new data as it is streamed.

hash.digest(encoding='binary')

Calculates the digest of all of the passed data to be hashed. The encoding can be 'hex', 'binary' or 'base64'.

crypto.createHmac(algorithm, key)

Creates and returns a hmac object, a cryptographic hmac with the given algorithm and key.

algorithm is dependent on the available algorithms supported by OpenSSL - see createHash above. key is the hmac key to be used.

hmac.update(data)

Update the hmac content with the given data. This can be called many times with new data as it is streamed.

hmac.digest(encoding='binary')

Calculates the digest of all of the passed data to the hmac. The encoding can be 'hex', 'binary' or 'base64'.

crypto.createCipher(algorithm, key)

Creates and returns a cipher object, with the given algorithm and key.

algorithm is dependent on OpenSSL, examples are 'aes192', etc. On recent releases, openssl list-cipher-algorithms will display the available cipher algorithms.

cipher.update(data, input_encoding='binary', output_encoding='binary')

Updates the cipher with data, the encoding of which is given in input_encoding and can be 'utf8', 'ascii' or 'binary'. The output_encoding specifies the output format of the enciphered data, and can be 'binary', 'base64' or 'hex'.

Returns the enciphered contents, and can be called many times with new data as it is streamed.

cipher.final(output_encoding='binary')

Returns any remaining enciphered contents, with output_encoding being one of: 'binary', 'ascii' or 'utf8'.

crypto.createDecipher(algorithm, key)

Creates and returns a decipher object, with the given algorithm and key. This is the mirror of the cipher object above.

decipher.update(data, input_encoding='binary', output_encoding='binary')

Updates the decipher with data, which is encoded in 'binary', 'base64' or 'hex'. The output_decoding specifies in what format to return the deciphered plaintext: 'binary', 'ascii' or 'utf8'.

decipher.final(output_encoding='binary')

Returns any remaining plaintext which is deciphered, with output_encoding' being one of:'binary','ascii'or'utf8'`.

crypto.createSign(algorithm)

Creates and returns a signing object, with the given algorithm. On recent OpenSSL releases, openssl list-public-key-algorithms will display the available signing algorithms. Examples are 'RSA-SHA256'.

signer.update(data)

Updates the signer object with data. This can be called many times with new data as it is streamed.

signer.sign(private_key, output_format='binary')

Calculates the signature on all the updated data passed through the signer. private_key is a string containing the PEM encoded private key for signing.

Returns the signature in output_format which can be 'binary', 'hex' or 'base64'.

crypto.createVerify(algorithm)

Creates and returns a verification object, with the given algorithm. This is the mirror of the signing object above.

verifier.update(data)

Updates the verifier object with data. This can be called many times with new data as it is streamed.

verifier.verify(cert, signature, signature_format='binary')

Verifies the signed data by using the cert which is a string containing the PEM encoded public key, and signature, which is the previously calculates signature for the data, in the signature_format which can be 'binary', 'hex' or 'base64'.

Returns true or false depending on the validity of the signature for the data and public key.

Streams

A stream is an abstract interface implemented by various objects in Node. For example a request to an HTTP server is a stream, as is stdout. Streams are readable, writable, or both. All streams are instances of EventEmitter.

Readable Stream

A Readable Stream has the following methods, members, and events.

Event: 'data'

function (data) { }

The 'data' event emits either a Buffer (by default) or a string if setEncoding() was used.

Event: 'end'

function () { }

Emitted when the stream has received an EOF (FIN in TCP terminology). Indicates that no more 'data' events will happen. If the stream is also writable, it may be possible to continue writing.

Event: 'error'

function (exception) { }

Emitted if there was an error receiving data.

Event: 'close'

function () { }

Emitted when the underlying file descriptor has been closed. Not all streams will emit this. (For example, an incoming HTTP request will not emit 'close'.)

Event: 'fd'

function (fd) { }

Emitted when a file descriptor is received on the stream. Only UNIX streams support this functionality; all others will simply never emit this event.

stream.readable

A boolean that is true by default, but turns false after an 'error' occured, the stream came to an 'end', or destroy() was called.

stream.setEncoding(encoding)

Makes the data event emit a string instead of a Buffer. encoding can be 'utf8', 'ascii', or 'base64'.

stream.pause()

Pauses the incoming 'data' events.

stream.resume()

Resumes the incoming 'data' events after a pause().

stream.destroy()

Closes the underlying file descriptor. Stream will not emit any more events.

stream.destroySoon()

After the write queue is drained, close the file descriptor.

stream.pipe(destination, [options])

This is a Stream.prototype method available on all Streams.

Connects this read stream to destination WriteStream. Incoming data on this stream gets written to destination. The destination and source streams are kept in sync by pausing and resuming as necessary.

Emulating the Unix cat command:

Example 8-53.  

process.stdin.resume();
process.stdin.pipe(process.stdout);

By default end() is called on the destination when the source stream emits end, so that destination is no longer writable. Pass { end: false } as options to keep the destination stream open.

This keeps process.stdout open so that "Goodbye" can be written at the end.

Example 8-54.  

process.stdin.resume();

process.stdin.pipe(process.stdout, { end: false });

process.stdin.on("end", function() {
  process.stdout.write("Goodbye\n");
});

NOTE: If the source stream does not support pause() and resume(), this function adds simple definitions which simply emit 'pause' and 'resume' events on the source stream.

Writable Stream

A Writable Stream has the following methods, members, and events.

Event: 'drain'

function () { }

Emitted after a write() method was called that returned false to indicate that it is safe to write again.

Event: 'error'

function (exception) { }

Emitted on error with the exception exception.

Event: 'close'

function () { }

Emitted when the underlying file descriptor has been closed.

Event: 'pipe'

function (src) { }

Emitted when the stream is passed to a readable stream's pipe method.

stream.writable

A boolean that is true by default, but turns false after an 'error' occurred or end() / destroy() was called.

stream.write(string, encoding='utf8', [fd])

Writes string with the given encoding to the stream. Returns true if the string has been flushed to the kernel buffer. Returns false to indicate that the kernel buffer is full, and the data will be sent out in the future. The 'drain' event will indicate when the kernel buffer is empty again. The encoding defaults to 'utf8'.

If the optional fd parameter is specified, it is interpreted as an integral file descriptor to be sent over the stream. This is only supported for UNIX streams, and is silently ignored otherwise. When writing a file descriptor in this manner, closing the descriptor before the stream drains risks sending an invalid (closed) FD.

stream.write(buffer)

Same as the above except with a raw buffer.

stream.end()

Terminates the stream with EOF or FIN.

stream.end(string, encoding)

Sends string with the given encoding and terminates the stream with EOF or FIN. This is useful to reduce the number of packets sent.

stream.end(buffer)

Same as above but with a buffer.

stream.destroy()

Closes the underlying file descriptor. Stream will not emit any more events.

Buffers

Pure Javascript is Unicode friendly but not nice to binary data. When dealing with TCP streams or the file system, it's necessary to handle octet streams. Node has several strategies for manipulating, creating, and consuming octet streams.

Raw data is stored in instances of the Buffer class. A Buffer is similar to an array of integers but corresponds to a raw memory allocation outside the V8 heap. A Buffer cannot be resized.

The Buffer object is global.

Converting between Buffers and JavaScript string objects requires an explicit encoding method. Here are the different string encodings;

'ascii' - for 7 bit ASCII data only. This encoding method is very fast, and will strip the high bit if set.

'utf8' - Unicode characters. Many web pages and other document formats use UTF-8.

'base64' - Base64 string encoding.

'binary' - A way of encoding raw binary data into strings by using only the first 8 bits of each character. This encoding method is depreciated and should be avoided in favor of Buffer objects where possible. This encoding will be removed in future versions of Node.

'hex' - Encode each byte as two hexidecimal characters.

new Buffer(size)

Allocates a new buffer of size octets.

new Buffer(array)

Allocates a new buffer using an array of octets.

new Buffer(str, encoding='utf8')

Allocates a new buffer containing the given str.

buffer.write(string, offset=0, encoding='utf8')

Writes string to the buffer at offset using the given encoding. Returns number of octets written. If buffer did not contain enough space to fit the entire string, it will write a partial amount of the string. In the case of 'utf8' encoding, the method will not write partial characters.

Example: write a utf8 string into a buffer, then print it

Example 8-55.  

buf = new Buffer(256);
len = buf.write('\u00bd + \u00bc = \u00be', 0);
console.log(len + " bytes: " + buf.toString('utf8', 0, len));

// 12 bytes: ½ + ¼ = ¾

buffer.toString(encoding, start=0, end=buffer.length)

Decodes and returns a string from buffer data encoded with encoding beginning at start and ending at end.

See buffer.write() example, above.

buffer[index]

Get and set the octet at index. The values refer to individual bytes, so the legal range is between 0x00 and 0xFF hex or 0 and 255.

Example: copy an ASCII string into a buffer, one byte at a time:

Example 8-56.  

str = "node.js";
buf = new Buffer(str.length);

for (var i = 0; i < str.length ; i++) {
  buf[i] = str.charCodeAt(i);
}

console.log(buf);

// node.js

Buffer.isBuffer(obj)

Tests if obj is a Buffer.

Buffer.byteLength(string, encoding='utf8')

Gives the actual byte length of a string. This is not the same as String.prototype.length since that returns the number of characters in a string.

Example:

Example 8-57.  

str = '\u00bd + \u00bc = \u00be';

console.log(str + ": " + str.length + " characters, " +
  Buffer.byteLength(str, 'utf8') + " bytes");

// ½ + ¼ = ¾: 9 characters, 12 bytes

buffer.length

The size of the buffer in bytes. Note that this is not necessarily the size of the contents. length refers to the amount of memory allocated for the buffer object. It does not change when the contents of the buffer are changed.

Example 8-58.  

buf = new Buffer(1234);

console.log(buf.length);
buf.write("some string", "ascii", 0);
console.log(buf.length);

// 1234
// 1234

buffer.copy(targetBuffer, targetStart=0, sourceStart=0, sourceEnd=buffer.length)

Does a memcpy() between buffers.

Example: build two Buffers, then copy buf1 from byte 16 through byte 19 into buf2, starting at the 8th byte in buf2.

Example 8-59.  

buf1 = new Buffer(26);
buf2 = new Buffer(26);

for (var i = 0 ; i < 26 ; i++) {
  buf1[i] = i + 97; // 97 is ASCII a
  buf2[i] = 33; // ASCII !
}

buf1.copy(buf2, 8, 16, 20);
console.log(buf2.toString('ascii', 0, 25));

// !!!!!!!!qrst!!!!!!!!!!!!!

buffer.slice(start, end=buffer.length)

Returns a new buffer which references the same memory as the old, but offset and cropped by the start and end indexes.

Modifying the new buffer slice will modify memory in the original buffer!

Example: build a Buffer with the ASCII alphabet, take a slice, then modify one byte from the original Buffer.

Example 8-60.  

var buf1 = new Buffer(26);

for (var i = 0 ; i < 26 ; i++) {
  buf1[i] = i + 97; // 97 is ASCII a
}

var buf2 = buf1.slice(0, 3);
console.log(buf2.toString('ascii', 0, buf2.length));
buf1[0] = 33;
console.log(buf2.toString('ascii', 0, buf2.length));

// abc
// !bc

Events

Many objects in Node emit events: a net.Server emits an event each time a peer connects to it, a fs.readStream emits an event when the file is opened. All objects which emit events are instances of events.EventEmitter. You can access this module by doing: require("events");

Typically, event names are represented by a camel-cased string, however, there aren't any strict restrictions on that, as any string will be accepted.

Functions can then be attached to objects, to be executed when an event is emitted. These functions are called listeners.

events.EventEmitter

To access the EventEmitter class, require('events').EventEmitter.

When an EventEmitter instance experiences an error, the typical action is to emit an 'error' event. Error events are treated as a special case in node. If there is no listener for it, then the default action is to print a stack trace and exit the program.

All EventEmitters emit the event 'newListener' when new listeners are added.

emitter.addListener(event, listener)

emitter.on(event, listener)

Adds a listener to the end of the listeners array for the specified event.

Example 8-61.  

server.on('connection', function (stream) {
  console.log('someone connected!');
});

emitter.once(event, listener)

Adds a one time listener for the event. The listener is invoked only the first time the event is fired, after which it is removed.

Example 8-62.  

server.once('connection', function (stream) {
  console.log('Ah, we have our first user!');
});

emitter.removeListener(event, listener)

Remove a listener from the listener array for the specified event. Caution: changes array indices in the listener array behind the listener.

Example 8-63.  

var callback = function(stream) {
  console.log('someone connected!');
};
server.on('connection', callback);
// ...
server.removeListener('connection', callback);

emitter.removeAllListeners(event)

Removes all listeners from the listener array for the specified event.

emitter.setMaxListeners(n)

By default EventEmitters will print a warning if more than 10 listeners are added to it. This is a useful default which helps finding memory leaks. Obviously not all Emitters should be limited to 10. This function allows that to be increased. Set to zero for unlimited.

emitter.listeners(event)

Returns an array of listeners for the specified event. This array can be manipulated, e.g. to remove listeners.

Example 8-64.  

server.on('connection', function (stream) {
  console.log('someone connected!');
});
console.log(util.inspect(server.listeners('connection')); // [ [Function] ]

emitter.emit(event, [arg1], [arg2], [...])

Execute each of the listeners in order with the supplied arguments.

Event: 'newListener'

function (event, listener) { }

This event is emitted any time someone adds a new listener.

Child Processes

Node provides a tri-directional popen(3) facility through the ChildProcess class.

It is possible to stream data through the child's stdin, stdout, and stderr in a fully non-blocking way.

To create a child process use require('child_process').spawn().

Child processes always have three streams associated with them. child.stdin, child.stdout, and child.stderr.

ChildProcess is an EventEmitter.

Event: 'exit'

function (code, signal) {}

This event is emitted after the child process ends. If the process terminated normally, code is the final exit code of the process, otherwise null. If the process terminated due to receipt of a signal, signal is the string name of the signal, otherwise null.

See waitpid(2).

child.stdin

A Writable Stream that represents the child process's stdin. Closing this stream via end() often causes the child process to terminate.

child.stdout

A Readable Stream that represents the child process's stdout.

child.stderr

A Readable Stream that represents the child process's stderr.

child.pid

The PID of the child process.

Example:

Example 8-65.  

var spawn = require('child_process').spawn,
    grep  = spawn('grep', ['ssh']);

console.log('Spawned child pid: ' + grep.pid);
grep.stdin.end();

child_process.spawn(command, args=[], [options])

Launches a new process with the given command, with command line arguments in args. If omitted, args defaults to an empty Array.

The third argument is used to specify additional options, which defaults to:

Example 8-66.  

{ cwd: undefined,
  env: process.env,
  customFds: [-1, -1, -1],
  setsid: false
}

cwd allows you to specify the working directory from which the process is spawned. Use env to specify environment variables that will be visible to the new process. With customFds it is possible to hook up the new process' [stdin, stout, stderr] to existing streams; -1 means that a new stream should be created. setsid, if set true, will cause the subprocess to be run in a new session.

Example of running ls -lh /usr, capturing stdout, stderr, and the exit code:

Example 8-67.  

var util   = require('util'),
    spawn = require('child_process').spawn,
    ls    = spawn('ls', ['-lh', '/usr']);

ls.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

ls.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

ls.on('exit', function (code) {
  console.log('child process exited with code ' + code);
});

Example: A very elaborate way to run 'ps ax | grep ssh'

Example 8-68.  

var util   = require('util'),
    spawn = require('child_process').spawn,
    ps    = spawn('ps', ['ax']),
    grep  = spawn('grep', ['ssh']);

ps.stdout.on('data', function (data) {
  grep.stdin.write(data);
});

ps.stderr.on('data', function (data) {
  console.log('ps stderr: ' + data);
});

ps.on('exit', function (code) {
  if (code !== 0) {
    console.log('ps process exited with code ' + code);
  }
  grep.stdin.end();
});

grep.stdout.on('data', function (data) {
  console.log(data);
});

grep.stderr.on('data', function (data) {
  console.log('grep stderr: ' + data);
});

grep.on('exit', function (code) {
  if (code !== 0) {
    console.log('grep process exited with code ' + code);
  }
});

Example of checking for failed exec:

Example 8-69.  

var spawn = require('child_process').spawn,
    child = spawn('bad_command');

child.stderr.on('data', function (data) {
  if (/^execvp\(\)/.test(data.asciiSlice(0,data.length))) {
    console.log('Failed to start child process.');
  }
});

See also: child_process.exec()

child_process.exec(command, [options], callback)

High-level way to execute a command as a child process, buffer the output, and return it all in a callback.

Example 8-70.  

var util   = require('util'),
    exec  = require('child_process').exec,
    child;

child = exec('cat *.js bad_file | wc -l',
  function (error, stdout, stderr) {
    console.log('stdout: ' + stdout);
    console.log('stderr: ' + stderr);
    if (error !== null) {
      console.log('exec error: ' + error);
    }
});

The callback gets the arguments (error, stdout, stderr). On success, error will be null. On error, error will be an instance of Error and err.code will be the exit code of the child process, and err.signal will be set to the signal that terminated the process.

There is a second optional argument to specify several options. The default options are

Example 8-71.  

{ encoding: 'utf8',
  timeout: 0,
  maxBuffer: 200*1024,
  killSignal: 'SIGTERM',
  cwd: null,
  env: null }

If timeout is greater than 0, then it will kill the child process if it runs longer than timeout milliseconds. The child process is killed with killSignal (default: 'SIGTERM'). maxBuffer specifies the largest amount of data allowed on stdout or stderr - if this value is exceeded then the child process is killed.

child.kill(signal='SIGTERM')

Send a signal to the child process. If no argument is given, the process will be sent 'SIGTERM'. See signal(7) for a list of available signals.

Example 8-72.  

var spawn = require('child_process').spawn,
    grep  = spawn('grep', ['ssh']);

grep.on('exit', function (code, signal) {
  console.log('child process terminated due to receipt of signal '+signal);
});

// send SIGHUP to process
grep.kill('SIGHUP');

Note that while the function is called kill, the signal delivered to the child process may not actually kill it. kill really just sends a signal to a process.

See kill(2)

Addons

Addons are dynamically linked shared objects. They can provide glue to C and C++ libraries. The API (at the moment) is rather complex, involving knowledge of several libraries:

V8 JavaScript, a C++ library. Used for interfacing with JavaScript: creating objects, calling functions, etc. Documented mostly in the v8.h header file (deps/v8/include/v8.h in the Node source tree).

libev, C event loop library. Anytime one needs to wait for a file descriptor to become readable, wait for a timer, or wait for a signal to received one will need to interface with libev. That is, if you perform any I/O, libev will need to be used. Node uses the EV_DEFAULT event loop. Documentation can be found here.

libeio, C thread pool library. Used to execute blocking POSIX system calls asynchronously. Mostly wrappers already exist for such calls, in src/file.cc so you will probably not need to use it. If you do need it, look at the header file deps/libeio/eio.h.

Internal Node libraries. Most importantly is the node::ObjectWrap class which you will likely want to derive from.

Others. Look in deps/ for what else is available.

Node statically compiles all its dependencies into the executable. When compiling your module, you don't need to worry about linking to any of these libraries.

To get started let's make a small Addon which does the following except in C++:

Example 8-73.  

exports.hello = 'world';

To get started we create a file hello.cc:

Example 8-74.  

#include <v8.h>

using namespace v8;

extern "C" void
init (Handle<Object> target)
{
  HandleScope scope;
  target->Set(String::New("hello"), String::New("world"));
}

This source code needs to be built into hello.node, the binary Addon. To do this we create a file called wscript which is python code and looks like this:

Example 8-75.  

srcdir = '.'
blddir = 'build'
VERSION = '0.0.1'

def set_options(opt):
  opt.tool_options('compiler_cxx')

def configure(conf):
  conf.check_tool('compiler_cxx')
  conf.check_tool('node_addon')

def build(bld):
  obj = bld.new_task_gen('cxx', 'shlib', 'node_addon')
  obj.target = 'hello'
  obj.source = 'hello.cc'

Running node-waf configure build will create a file build/default/hello.node which is our Addon.

node-waf is just WAF, the python-based build system. node-waf is provided for the ease of users.

All Node addons must export a function called init with this signature:

Example 8-76.  

extern 'C' void init (Handle<Object> target)

For the moment, that is all the documentation on addons. Please see http://github.com/ry/node_postgres for a real example.

Modules

Node uses the CommonJS module system.

Node has a simple module loading system. In Node, files and modules are in one-to-one correspondence. As an example, foo.js loads the module circle.js in the same directory.

The contents of foo.js:

Example 8-77.  

var circle = require('./circle.js');
console.log( 'The area of a circle of radius 4 is '
           + circle.area(4));

The contents of circle.js:

Example 8-78.  

var PI = Math.PI;

exports.area = function (r) {
  return PI * r * r;
};

exports.circumference = function (r) {
  return 2 * PI * r;
};

The module circle.js has exported the functions area() and circumference(). To export an object, add to the special exports object.

Variables local to the module will be private. In this example the variable PI is private to circle.js.

Core Modules

Node has several modules compiled into the binary. These modules are described in greater detail elsewhere in this documentation.

The core modules are defined in node's source in the lib/ folder.

Core modules are always preferentially loaded if their identifier is passed to require(). For instance, require('http') will always return the built in HTTP module, even if there is a file by that name.

File Modules

If the exact filename is not found, then node will attempt to load the required filename with the added extension of .js, and then .node.

.js files are interpreted as JavaScript text files, and .node files are interpreted as compiled addon modules loaded with dlopen.

A module prefixed with '/' is an absolute path to the file. For example, require('/home/marco/foo.js') will load the file at /home/marco/foo.js.

A module prefixed with './' is relative to the file calling require(). That is, circle.js must be in the same directory as foo.js for require('./circle') to find it.

Without a leading '/' or './' to indicate a file, the module is either a "core module" or is loaded from a node_modules folder.

Loading from node_modules Folders

If the module identifier passed to require() is not a native module, and does not begin with '/', '../', or './', then node starts at the parent directory of the current module, and adds /node_modules, and attempts to load the module from that location.

If it is not found there, then it moves to the parent directory, and so on, until either the module is found, or the root of the tree is reached.

For example, if the file at '/home/ry/projects/foo.js' called require('bar.js'), then node would look in the following locations, in this order:

/home/ry/projects/node_modules/bar.js

/home/ry/node_modules/bar.js

/home/node_modules/bar.js

/node_modules/bar.js

This allows programs to localize their dependencies, so that they do not clash.

Optimizations to the node_modules Lookup Process

When there are many levels of nested dependencies, it is possible for these file trees to get fairly long. The following optimizations are thus made to the process.

First, /node_modules is never appended to a folder already ending in /node_modules.

Second, if the file calling require() is already inside a node_modules heirarchy, then the top-most node_modules folder is treated as the root of the search tree.

For example, if the file at '/home/ry/projects/foo/node_modules/bar/node_modules/baz/quux.js' called require('asdf.js'), then node would search the following locations:

/home/ry/projects/foo/node_modules/bar/node_modules/baz/node_modules/asdf.js

/home/ry/projects/foo/node_modules/bar/node_modules/asdf.js

/home/ry/projects/foo/node_modules/asdf.js

Folders as Modules

It is convenient to organize programs and libraries into self-contained directories, and then provide a single entry point to that library. There are three ways in which a folder may be passed to require() as an argument.

The first is to create a package.json file in the root of the folder, which specifies a main module. An example package.json file might look like this:

Example 8-79.  

{ "name" : "some-library",
  "main" : "./lib/some-library.js" }

If this was in a folder at ./some-library, then require('./some-library') would attempt to load ./some-library/lib/some-library.js.

This is the extent of Node's awareness of package.json files.

If there is no package.json file present in the directory, then node will attempt to load an index.js or index.node file out of that directory. For example, if there was no package.json file in the above example, then require('./some-library') would attempt to load:

./some-library/index.js

./some-library/index.node

Caching

Modules are cached after the first time they are loaded. This means (among other things) that every call to require('foo') will get exactly the same object returned, if it would resolve to the same file.

All Together...

To get the exact filename that will be loaded when require() is called, use the require.resolve() function.

Putting together all of the above, here is the high-level algorithm in pseudocode of what require.resolve does:

Example 8-80.  

require(X)
1. If X is a core module,
   a. return the core module
   b. STOP
2. If X begins with `./` or `/`,
   a. LOAD_AS_FILE(Y + X)
   b. LOAD_AS_DIRECTORY(Y + X)
3. LOAD_NODE_MODULES(X, dirname(Y))
4. THROW "not found"

LOAD_AS_FILE(X)
1. If X is a file, load X as JavaScript text.  STOP
2. If X.js is a file, load X.js as JavaScript text.  STOP
3. If X.node is a file, load X.node as binary addon.  STOP

LOAD_AS_DIRECTORY(X)
1. If X/package.json is a file,
   a. Parse X/package.json, and look for "main" field.
   b. let M = X + (json main field)
   c. LOAD_AS_FILE(M)
2. LOAD_AS_FILE(X/index)

LOAD_NODE_MODULES(X, START)
1. let DIRS=NODE_MODULES_PATHS(START)
2. for each DIR in DIRS:
   a. LOAD_AS_FILE(DIR/X)
   b. LOAD_AS_DIRECTORY(DIR/X)

NODE_MODULES_PATHS(START)
1. let PARTS = path split(START)
2. let ROOT = index of first instance of "node_modules" in PARTS, or 0
3. let I = count of PARTS - 1
4. let DIRS = []
5. while I > ROOT,
   a. if PARTS[I] = "node_modules" CONTINUE
   c. DIR = path join(PARTS[0 .. I] + "node_modules")
   b. DIRS = DIRS + DIR
6. return DIRS

Loading from the require.paths Folders

In node, require.paths is an array of strings that represent paths to be searched for modules when they are not prefixed with '/', './', or '../'. For example, if require.paths were set to:

Example 8-81.  

[ '/home/micheil/.node_modules',
  '/usr/local/lib/node_modules' ]

Then calling require('bar/baz.js') would search the following locations:

1: '/home/micheil/.node_modules/bar/baz.js'

2: '/usr/local/lib/node_modules/bar/baz.js'

The require.paths array can be mutated at run time to alter this behavior.

It is set initially from the NODE_PATH environment variable, which is a colon-delimited list of absolute paths. In the previous example, the NODE_PATH environment variable might have been set to:

Example 8-82.  

/home/micheil/.node_modules:/usr/local/lib/node_modules

Note: Please Avoid Modifying require.paths

For compatibility reasons, require.paths is still given first priority in the module lookup process. However, it may disappear in a future release.

While it seemed like a good idea at the time, and enabled a lot of useful experimentation, in practice a mutable require.paths list is often a troublesome source of confusion and headaches.

Setting require.paths to some other value does nothing.

This does not do what one might expect:

Example 8-83.  

require.paths = [ '/usr/lib/node' ];

All that does is lose the reference to the actual node module lookup paths, and create a new reference to some other thing that isn't used for anything.

Putting relative paths in require.paths is... weird.

If you do this:

Example 8-84.  

require.paths.push('./lib');

then it does not add the full resolved path to where ./lib is on the filesystem. Instead, it literally adds './lib', meaning that if you do require('y.js') in /a/b/x.js, then it'll look in /a/b/lib/y.js. If you then did require('y.js') in /l/m/n/o/p.js, then it'd look in /l/m/n/o/lib/y.js.

In practice, people have used this as an ad hoc way to bundle dependencies, but this technique is brittle.

Zero Isolation

There is (by regrettable design), only one require.paths array used by all modules.

As a result, if one node program comes to rely on this behavior, it may permanently and subtly alter the behavior of all other node programs in the same process. As the application stack grows, we tend to assemble functionality, and it is a problem with those parts interact in ways that are difficult to predict.

Addenda: Package Manager Tips

The semantics of Node's require() function were designed to be general enough to support a number of sane directory structures. Package manager programs such as dpkg, rpm, and npm will hopefully find it possible to build native packages from Node modules without modification.

Below we give a suggested directory structure that could work:

Let's say that we wanted to have the folder at /usr/lib/node/<some-package>/<some-version> hold the contents of a specific version of a package.

Packages can depend on one another. In order to install package foo, you may have to install a specific version of package bar. The bar package may itself have dependencies, and in some cases, these dependencies may even collide or form cycles.

Since Node looks up the realpath of any modules it loads (that is, resolves symlinks), and then looks for their dependencies in the node_modules folders as described above, this situation is very simple to resolve with the following architecture:

/usr/lib/node/foo/1.2.3/ - Contents of the foo package, version 1.2.3.

/usr/lib/node/bar/4.3.2/ - Contents of the bar package that foo depends on.

/usr/lib/node/foo/1.2.3/node_modules/bar - Symbolic link to /usr/lib/node/bar/4.3.2/.

/usr/lib/node/bar/4.3.2/node_modules/* - Symbolic links to the packages that bar depends on.

Thus, even if a cycle is encountered, or if there are dependency conflicts, every module will be able to get a version of its dependency that it can use.

When the code in the foo package does require('bar'), it will get the version that is symlinked into /usr/lib/node/foo/1.2.3/node_modules/bar. Then, when the code in the bar package calls require('quux'), it'll get the version that is symlinked into /usr/lib/node/bar/4.3.2/node_modules/quux.

Furthermore, to make the module lookup process even more optimal, rather than putting packages directly in /usr/lib/node, we could put them in /usr/lib/node_modules/<name>/<version>. Then node will not bother looking for missing dependencies in /usr/node_modules or /node_modules.

In order to make modules available to the node REPL, it might be useful to also add the /usr/lib/node_modules folder to the $NODE_PATH environment variable. Since the module lookups using node_modules folders are all relative, and based on the real path of the files making the calls to require(), the packages themselves can be anywhere.

Timers

setTimeout(callback, delay, [arg], [...])

To schedule execution of callback after delay milliseconds. Returns a timeoutId for possible use with clearTimeout(). Optionally, you can also pass arguments to the callback.

clearTimeout(timeoutId)

Prevents a timeout from triggering.

setInterval(callback, delay, [arg], [...])

To schedule the repeated execution of callback every delay milliseconds. Returns a intervalId for possible use with clearInterval(). Optionally, you can also pass arguments to the callback.

clearInterval(intervalId)

Stops a interval from triggering.

Global Objects

These object are available in the global scope and can be accessed from anywhere.

global

The global namespace object.

In browsers, the top-level scope is the global scope. That means that in browsers if you're in the global scope var something will define a global variable. In Node this is different. The top-level scope is not the global scope; var something inside a Node module will be local to that module.

process

The process object. See the 'process object' section.

require()

To require modules. See the 'Modules' section.

require.resolve()

Use the internal require() machinery to look up the location of a module, but rather than loading the module, just return the resolved filename.

require.paths

An array of search paths for require(). This array can be modified to add custom paths.

Example: add a new path to the beginning of the search list

Example 8-85.  

require.paths.unshift('/usr/local/node');

__filename

The filename of the script being executed. This is the absolute path, and not necessarily the same filename passed in as a command line argument.

Example: running node example.js from /Users/mjr

Example 8-86.  

console.log(__filename);
// /Users/mjr/example.js

__dirname

The dirname of the script being executed.

Example: running node example.js from /Users/mjr

Example 8-87.  

console.log(__dirname);
// /Users/mjr

module

A reference to the current module. In particular module.exports is the same as the exports object. See src/node.js for more information.

Synopsis

An example of a web server written with Node which responds with 'Hello World':

Example 8-88.  

var http = require('http');

http.createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'});
  response.end('Hello World\n');
}).listen(8124);

console.log('Server running at http://127.0.0.1:8124/');

To run the server, put the code into a file called example.js and execute it with the node program

Example 8-89.  

> node example.js
Server running at http://127.0.0.1:8124/

All of the examples in the documentation can be run similarly.

DNS

Use require('dns') to access this module.

Here is an example which resolves 'www.google.com' then reverse resolves the IP addresses which are returned.

Example 8-90.  

var dns = require('dns');

dns.resolve4('www.google.com', function (err, addresses) {
  if (err) throw err;

  console.log('addresses: ' + JSON.stringify(addresses));

  addresses.forEach(function (a) {
    dns.reverse(a, function (err, domains) {
      if (err) {
        console.log('reverse for ' + a + ' failed: ' +
          err.message);
      } else {
        console.log('reverse for ' + a + ': ' +
          JSON.stringify(domains));
      }
    });
  });
});

dns.lookup(domain, family=null, callback)

Resolves a domain (e.g. 'google.com') into the first found A (IPv4) or AAAA (IPv6) record.

The callback has arguments (err, address, family). The address argument is a string representation of a IP v4 or v6 address. The family argument is either the integer 4 or 6 and denotes the family of address (not neccessarily the value initially passed to lookup).

dns.resolve(domain, rrtype='A', callback)

Resolves a domain (e.g. 'google.com') into an array of the record types specified by rrtype. Valid rrtypes are A (IPV4 addresses), AAAA (IPV6 addresses), MX (mail exchange records), TXT (text records), SRV (SRV records), and PTR (used for reverse IP lookups).

The callback has arguments (err, addresses). The type of each item in addresses is determined by the record type, and described in the documentation for the corresponding lookup methods below.

On error, err would be an instanceof Error object, where err.errno is one of the error codes listed below and err.message is a string describing the error in English.

dns.resolve4(domain, callback)

The same as dns.resolve(), but only for IPv4 queries (A records). addresses is an array of IPv4 addresses (e.g. ['74.125.79.104', '74.125.79.105', '74.125.79.106']).

dns.resolve6(domain, callback)

The same as dns.resolve4() except for IPv6 queries (an AAAA query).

dns.resolveMx(domain, callback)

The same as dns.resolve(), but only for mail exchange queries (MX records).

addresses is an array of MX records, each with a priority and an exchange attribute (e.g. [{'priority': 10, 'exchange': 'mx.example.com'},...]).

dns.resolveTxt(domain, callback)

The same as dns.resolve(), but only for text queries (TXT records). addresses is an array of the text records available for domain (e.g., ['v=spf1 ip4:0.0.0.0 ~all']).

dns.resolveSrv(domain, callback)

The same as dns.resolve(), but only for service records (SRV records). addresses is an array of the SRV records available for domain. Properties of SRV records are priority, weight, port, and name (e.g., [{'priority': 10, {'weight': 5, 'port': 21223, 'name': 'service.example.com'}, ...]).

dns.reverse(ip, callback)

Reverse resolves an ip address to an array of domain names.

The callback has arguments (err, domains).

If there an an error, err will be non-null and an instanceof the Error object.

Each DNS query can return an error code.

dns.TEMPFAIL: timeout, SERVFAIL or similar.

dns.PROTOCOL: got garbled reply.

dns.NXDOMAIN: domain does not exists.

dns.NODATA: domain exists but no data of reqd type.

dns.NOMEM: out of memory while processing.

dns.BADQUERY: the query is malformed.

File System

File I/O is provided by simple wrappers around standard POSIX functions. To use this module do require('fs'). All the methods have asynchronous and synchronous forms.

The asynchronous form always take a completion callback as its last argument. The arguments passed to the completion callback depend on the method, but the first argument is always reserved for an exception. If the operation was completed successfully, then the first argument will be null or undefined.

Here is an example of the asynchronous version:

Example 8-91.  

var fs = require('fs');

fs.unlink('/tmp/hello', function (err) {
  if (err) throw err;
  console.log('successfully deleted /tmp/hello');
});

Here is the synchronous version:

Example 8-92.  

var fs = require('fs');

fs.unlinkSync('/tmp/hello')
console.log('successfully deleted /tmp/hello');

With the asynchronous methods there is no guaranteed ordering. So the following is prone to error:

Example 8-93.  

fs.rename('/tmp/hello', '/tmp/world', function (err) {
  if (err) throw err;
  console.log('renamed complete');
});
fs.stat('/tmp/world', function (err, stats) {
  if (err) throw err;
  console.log('stats: ' + JSON.stringify(stats));
});

It could be that fs.stat is executed before fs.rename. The correct way to do this is to chain the callbacks.

Example 8-94.  

fs.rename('/tmp/hello', '/tmp/world', function (err) {
  if (err) throw err;
  fs.stat('/tmp/world', function (err, stats) {
    if (err) throw err;
    console.log('stats: ' + JSON.stringify(stats));
  });
});

In busy processes, the programmer is strongly encouraged to use the asynchronous versions of these calls. The synchronous versions will block the entire process until they complete--halting all connections.

fs.rename(path1, path2, [callback])

Asynchronous rename(2). No arguments other than a possible exception are given to the completion callback.

fs.renameSync(path1, path2)

Synchronous rename(2).

fs.truncate(fd, len, [callback])

Asynchronous ftruncate(2). No arguments other than a possible exception are given to the completion callback.

fs.truncateSync(fd, len)

Synchronous ftruncate(2).

fs.chmod(path, mode, [callback])

Asynchronous chmod(2). No arguments other than a possible exception are given to the completion callback.

fs.chmodSync(path, mode)

Synchronous chmod(2).

fs.stat(path, [callback])

Asynchronous stat(2). The callback gets two arguments (err, stats) where stats is a fs.Stats object. It looks like this:

Example 8-95.  

{ dev: 2049,
  ino: 305352,
  mode: 16877,
  nlink: 12,
  uid: 1000,
  gid: 1000,
  rdev: 0,
  size: 4096,
  blksize: 4096,
  blocks: 8,
  atime: '2009-06-29T11:11:55Z',
  mtime: '2009-06-29T11:11:40Z',
  ctime: '2009-06-29T11:11:40Z' }

See the fs.Stats section below for more information.

fs.lstat(path, [callback])

Asynchronous lstat(2). The callback gets two arguments (err, stats) where stats is a fs.Stats object. lstat() is identical to stat(), except that if path is a symbolic link, then the link itself is stat-ed, not the file that it refers to.

fs.fstat(fd, [callback])

Asynchronous fstat(2). The callback gets two arguments (err, stats) where stats is a fs.Stats object.

fs.statSync(path)

Synchronous stat(2). Returns an instance of fs.Stats.

fs.lstatSync(path)

Synchronous lstat(2). Returns an instance of fs.Stats.

fs.fstatSync(fd)

Synchronous fstat(2). Returns an instance of fs.Stats.

fs.link(srcpath, dstpath, [callback])

Asynchronous link(2). No arguments other than a possible exception are given to the completion callback.

fs.linkSync(srcpath, dstpath)

Synchronous link(2).

fs.symlink(linkdata, path, [callback])

Asynchronous symlink(2). No arguments other than a possible exception are given to the completion callback.

fs.symlinkSync(linkdata, path)

Synchronous symlink(2).

fs.readlink(path, [callback])

Asynchronous readlink(2). The callback gets two arguments (err, resolvedPath).

fs.readlinkSync(path)

Synchronous readlink(2). Returns the resolved path.

fs.realpath(path, [callback])

Asynchronous realpath(2). The callback gets two arguments (err, resolvedPath).

fs.realpathSync(path)

Synchronous realpath(2). Returns the resolved path.

fs.unlink(path, [callback])

Asynchronous unlink(2). No arguments other than a possible exception are given to the completion callback.

fs.unlinkSync(path)

Synchronous unlink(2).

fs.rmdir(path, [callback])

Asynchronous rmdir(2). No arguments other than a possible exception are given to the completion callback.

fs.rmdirSync(path)

Synchronous rmdir(2).

fs.mkdir(path, mode, [callback])

Asynchronous mkdir(2). No arguments other than a possible exception are given to the completion callback.

fs.mkdirSync(path, mode)

Synchronous mkdir(2).

fs.readdir(path, [callback])

Asynchronous readdir(3). Reads the contents of a directory. The callback gets two arguments (err, files) where files is an array of the names of the files in the directory excluding '.' and '..'.

fs.readdirSync(path)

Synchronous readdir(3). Returns an array of filenames excluding '.' and '..'.

fs.close(fd, [callback])

Asynchronous close(2). No arguments other than a possible exception are given to the completion callback.

fs.closeSync(fd)

Synchronous close(2).

fs.open(path, flags, [mode], [callback])

Asynchronous file open. See open(2). Flags can be 'r', 'r+', 'w', 'w+', 'a', or 'a+'. mode defaults to 0666. The callback gets two arguments (err, fd).

fs.openSync(path, flags, [mode])

Synchronous open(2).

fs.write(fd, buffer, offset, length, position, [callback])

Write buffer to the file specified by fd.

offset and length determine the part of the buffer to be written.

position refers to the offset from the beginning of the file where this data should be written. If position is null, the data will be written at the current position. See pwrite(2).

The callback will be given two arguments (err, written) where written specifies how many bytes were written.

fs.writeSync(fd, buffer, offset, length, position)

Synchronous version of buffer-based fs.write(). Returns the number of bytes written.

fs.writeSync(fd, str, position, encoding='utf8')

Synchronous version of string-based fs.write(). Returns the number of bytes written.

fs.read(fd, buffer, offset, length, position, [callback])

Read data from the file specified by fd.

buffer is the buffer that the data will be written to.

offset is offset within the buffer where writing will start.

length is an integer specifying the number of bytes to read.

position is an integer specifying where to begin reading from in the file. If position is null, data will be read from the current file position.

The callback is given the two arguments, (err, bytesRead).

fs.readSync(fd, buffer, offset, length, position)

Synchronous version of buffer-based fs.read. Returns the number of bytesRead.

fs.readSync(fd, length, position, encoding)

Synchronous version of string-based fs.read. Returns the number of bytesRead.

fs.readFile(filename, [encoding], [callback])

Asynchronously reads the entire contents of a file. Example:

Example 8-96.  

fs.readFile('/etc/passwd', function (err, data) {
  if (err) throw err;
  console.log(data);
});

The callback is passed two arguments (err, data), where data is the contents of the file.

If no encoding is specified, then the raw buffer is returned.

fs.readFileSync(filename, [encoding])

Synchronous version of fs.readFile. Returns the contents of the filename.

If encoding is specified then this function returns a string. Otherwise it returns a buffer.

fs.writeFile(filename, data, encoding='utf8', [callback])

Asynchronously writes data to a file. data can be a string or a buffer.

Example:

Example 8-97.  

fs.writeFile('message.txt', 'Hello Node', function (err) {
  if (err) throw err;
  console.log('It\'s saved!');
});

fs.writeFileSync(filename, data, encoding='utf8')

The synchronous version of fs.writeFile.

fs.watchFile(filename, [options], listener)

Watch for changes on filename. The callback listener will be called each time the file is accessed.

The second argument is optional. The options if provided should be an object containing two members a boolean, persistent, and interval, a polling value in milliseconds. The default is { persistent: true, interval: 0 }.

The listener gets two arguments the current stat object and the previous stat object:

Example 8-98.  

fs.watchFile(f, function (curr, prev) {
  console.log('the current mtime is: ' + curr.mtime);
  console.log('the previous mtime was: ' + prev.mtime);
});

These stat objects are instances of fs.Stat.

If you want to be notified when the file was modified, not just accessed you need to compare curr.mtime and `prev.mtime.

fs.unwatchFile(filename)

Stop watching for changes on filename.

fs.Stats

Objects returned from fs.stat() and fs.lstat() are of this type.

stats.isFile()

stats.isDirectory()

stats.isBlockDevice()

stats.isCharacterDevice()

stats.isSymbolicLink() (only valid with fs.lstat())

stats.isFIFO()

stats.isSocket()

fs.ReadStream

ReadStream is a Readable Stream.

fs.createReadStream(path, [options])

Returns a new ReadStream object (See Readable Stream).

options is an object with the following defaults:

Example 8-99.  

{ flags: 'r',
  encoding: null,
  mode: 0666,
  bufferSize: 4096 }

options can include start and end values to read a range of bytes from the file instead of the entire file. Both start and end are inclusive and start at 0. When used, both the limits must be specified always.

An example to read the last 10 bytes of a file which is 100 bytes long:

Example 8-100.  

fs.createReadStream('sample.txt', {start: 90, end: 99});

fs.WriteStream

WriteStream is a Writable Stream.

Event: 'open'

function (fd) { }

fd is the file descriptor used by the WriteStream.

fs.createWriteStream(path, [options])

Returns a new WriteStream object (See Writable Stream).

options is an object with the following defaults:

Example 8-101.  

{ flags: 'w',
  encoding: null,
  mode: 0666 }

util

These functions are in the module 'util'. Use require('util') to access them.

util.debug(string)

A synchronous output function. Will block the process and output string immediately to stderr.

Example 8-102.  

require('util').debug('message on stderr');

util.log(string)

Output with timestamp on stdout.

Example 8-103.  

require('util').log('Timestmaped message.');

util.inspect(object, showHidden=false, depth=2)

Return a string representation of object, which is useful for debugging.

If showHidden is true, then the object's non-enumerable properties will be shown too.

If depth is provided, it tells inspect how many times to recurse while formatting the object. This is useful for inspecting large complicated objects.

The default is to only recurse twice. To make it recurse indefinitely, pass in null for depth.

Example of inspecting all properties of the util object:

Example 8-104.  

var util = require('util');

console.log(util.inspect(util, true, null));

util.pump(readableStream, writableStream, [callback])

Experimental

Read the data from readableStream and send it to the writableStream. When writableStream.write(data) returns false readableStream will be paused until the drain event occurs on the writableStream. callback gets an error as its only argument and is called when writableStream is closed or when an error occurs.

util.inherits(constructor, superConstructor)

Inherit the prototype methods from one constructor into another. The prototype of constructor will be set to a new object created from superConstructor.

As an additional convenience, superConstructor will be accessible through the constructor.super_ property.

Example 8-105.  

var util = require("util");
var events = require("events");

function MyStream() {
    events.EventEmitter.call(this);
}

util.inherits(MyStream, events.EventEmitter);

MyStream.prototype.write = function(data) {
    this.emit("data", data);
}

var stream = new MyStream();

console.log(stream instanceof events.EventEmitter); // true
console.log(MyStream.super_ === events.EventEmitter); // true

stream.on("data", function(data) {
    console.log('Received data: "' + data + '"');
})
stream.write("It works!"); // Received data: "It works!"

Executing JavaScript

You can access this module with:

Example 8-106.  

var vm = require('vm');

JavaScript code can be compiled and run immediately or compiled, saved, and run later.

vm.runInThisContext(code, [filename])

vm.runInThisContext() compiles code as if it were loaded from filename, runs it and returns the result. Running code does not have access to local scope. filename is optional.

Example of using vm.runInThisContext and eval to run the same code:

Example 8-107.  

var localVar = 123,
    usingscript, evaled,
    vm = require('vm');

usingscript = vm.runInThisContext('localVar = 1;',
  'myfile.vm');
console.log('localVar: ' + localVar + ', usingscript: ' +
  usingscript);
evaled = eval('localVar = 1;');
console.log('localVar: ' + localVar + ', evaled: ' +
  evaled);

// localVar: 123, usingscript: 1
// localVar: 1, evaled: 1

vm.runInThisContext does not have access to the local scope, so localVar is unchanged. eval does have access to the local scope, so localVar is changed.

In case of syntax error in code, vm.runInThisContext emits the syntax error to stderr and throws.an exception.

vm.runInNewContext(code, [sandbox], [filename])

vm.runInNewContext compiles code to run in sandbox as if it were loaded from filename, then runs it and returns the result. Running code does not have access to local scope and the object sandbox will be used as the global object for code. sandbox and filename are optional.

Example: compile and execute code that increments a global variable and sets a new one. These globals are contained in the sandbox.

Example 8-108.  

var util = require('util'),
    vm = require('vm'),
    sandbox = {
      animal: 'cat',
      count: 2
    };

vm.runInNewContext('count += 1; name = "kitty"', sandbox, 'myfile.vm');
console.log(util.inspect(sandbox));

// { animal: 'cat', count: 3, name: 'kitty' }

Note that running untrusted code is a tricky business requiring great care. To prevent accidental global variable leakage, vm.runInNewContext is quite useful, but safely running untrusted code requires a separate process.

In case of syntax error in code, vm.runInThisContext emits the syntax error to stderr and throws an exception.

vm.createScript(code, [filename])

createScript compiles code as if it were loaded from filename, but does not run it. Instead, it returns a vm.Script object representing this compiled code. This script can be run later many times using methods below. The returned script is not bound to any global object. It is bound before each run, just for that run. filename is optional.

In case of syntax error in code, createScript prints the syntax error to stderr and throws an exception.

script.runInThisContext()

Similar to vm.runInThisContext but a method of a precompiled Script object. script.runInThisContext runs the code of script and returns the result. Running code does not have access to local scope, but does have access to the global object (v8: in actual context).

Example of using script.runInThisContext to compile code once and run it multiple times:

Example 8-109.  

var vm = require('vm');

globalVar = 0;

var script = vm.createScript('globalVar += 1', 'myfile.vm');

for (var i = 0; i < 1000 ; i += 1) {
  script.runInThisContext();
}

console.log(globalVar);

// 1000

script.runInNewContext([sandbox])

Similar to vm.runInNewContext a method of a precompiled Script object. script.runInNewContext runs the code of script with sandbox as the global object and returns the result. Running code does not have access to local scope. sandbox is optional.

Example: compile code that increments a global variable and sets one, then execute this code multiple times. These globals are contained in the sandbox.

Example 8-110.  

var util = require('util'),
    vm = require('vm'),
    sandbox = {
      animal: 'cat',
      count: 2
    };

var script = vm.createScript('count += 1; name = "kitty"', 'myfile.vm');

for (var i = 0; i < 10 ; i += 1) {
  script.runInNewContext(sandbox);
}

console.log(util.inspect(sandbox));

// { animal: 'cat', count: 12, name: 'kitty' }

Note that running untrusted code is a tricky business requiring great care. To prevent accidental global variable leakage, script.runInNewContext is quite useful, but safely running untrusted code requires a separate process.

HTTP

To use the HTTP server and client one must require('http').

The HTTP interfaces in Node are designed to support many features of the protocol which have been traditionally difficult to use. In particular, large, possibly chunk-encoded, messages. The interface is careful to never buffer entire requests or responses--the user is able to stream data.

HTTP message headers are represented by an object like this:

Example 8-111.  

{ 'content-length': '123',
  'content-type': 'text/plain',
  'connection': 'keep-alive',
  'accept': '*/*' }

Keys are lowercased. Values are not modified.

In order to support the full spectrum of possible HTTP applications, Node's HTTP API is very low-level. It deals with stream handling and message parsing only. It parses a message into headers and body but it does not parse the actual headers or the body.

http.Server

This is an EventEmitter with the following events:

Event: 'request'

function (request, response) { }

request is an instance of http.ServerRequest and response is an instance of http.ServerResponse

Event: 'connection'

function (stream) { }

When a new TCP stream is established. stream is an object of type net.Stream. Usually users will not want to access this event. The stream can also be accessed at request.connection.

Event: 'close'

function (errno) { }

Emitted when the server closes.

Event: 'request'

function (request, response) {}

Emitted each time there is request. Note that there may be multiple requests per connection (in the case of keep-alive connections).

Event: 'checkContinue'

function (request, response) {}

Emitted each time a request with an http Expect: 100-continue is received. If this event isn't listened for, the server will automatically respond with a 100 Continue as appropriate.

Handling this event involves calling response.writeContinue if the client should continue to send the request body, or generating an appropriate HTTP response (e.g., 400 Bad Request) if the client should not continue to send the request body.

Note that when this event is emitted and handled, the request event will not be emitted.

Event: 'upgrade'

function (request, socket, head)

Emitted each time a client requests a http upgrade. If this event isn't listened for, then clients requesting an upgrade will have their connections closed.

request is the arguments for the http request, as it is in the request event.

socket is the network socket between the server and client.

head is an instance of Buffer, the first packet of the upgraded stream, this may be empty.

After this event is emitted, the request's socket will not have a data event listener, meaning you will need to bind to it in order to handle data sent to the server on that socket.

Event: 'clientError'

function (exception) {}

If a client connection emits an 'error' event - it will forwarded here.

http.createServer(requestListener)

Returns a new web server object.

The requestListener is a function which is automatically added to the 'request' event.

server.listen(port, [hostname], [callback])

Begin accepting connections on the specified port and hostname. If the hostname is omitted, the server will accept connections directed to any IPv4 address (INADDR_ANY).

To listen to a unix socket, supply a filename instead of port and hostname.

This function is asynchronous. The last parameter callback will be called when the server has been bound to the port.

server.listen(path, [callback])

Start a UNIX socket server listening for connections on the given path.

This function is asynchronous. The last parameter callback will be called when the server has been bound.

server.close()

Stops the server from accepting new connections.

http.ServerRequest

This object is created internally by a HTTP server -- not by the user -- and passed as the first argument to a 'request' listener.

This is an EventEmitter with the following events:

Event: 'data'

function (chunk) { }

Emitted when a piece of the message body is received.

Example: A chunk of the body is given as the single argument. The transfer-encoding has been decoded. The body chunk is a string. The body encoding is set with request.setBodyEncoding().

Event: 'end'

function () { }

Emitted exactly once for each message. No arguments. After emitted no other events will be emitted on the request.

request.method

The request method as a string. Read only. Example: 'GET', 'DELETE'.

request.url

Request URL string. This contains only the URL that is present in the actual HTTP request. If the request is:

Example 8-112.  

GET /status?name=ryan HTTP/1.1\r\n
Accept: text/plain\r\n
\r\n

Then request.url will be:

Example 8-113.  

'/status?name=ryan'

If you would like to parse the URL into its parts, you can use require('url').parse(request.url). Example:

Example 8-114.  

node> require('url').parse('/status?name=ryan')
{ href: '/status?name=ryan',
  search: '?name=ryan',
  query: 'name=ryan',
  pathname: '/status' }

If you would like to extract the params from the query string, you can use the require('querystring').parse function, or pass true as the second argument to require('url').parse. Example:

Example 8-115.  

node> require('url').parse('/status?name=ryan', true)
{ href: '/status?name=ryan',
  search: '?name=ryan',
  query: { name: 'ryan' },
  pathname: '/status' }

request.headers

Read only.

request.trailers

Read only; HTTP trailers (if present). Only populated after the 'end' event.

request.httpVersion

The HTTP protocol version as a string. Read only. Examples: '1.1', '1.0'. Also request.httpVersionMajor is the first integer and request.httpVersionMinor is the second.

request.setEncoding(encoding=null)

Set the encoding for the request body. Either 'utf8' or 'binary'. Defaults to null, which means that the 'data' event will emit a Buffer object..

request.pause()

Pauses request from emitting events. Useful to throttle back an upload.

request.resume()

Resumes a paused request.

request.connection

The net.Stream object associated with the connection.

With HTTPS support, use request.connection.verifyPeer() and request.connection.getPeerCertificate() to obtain the client's authentication details.

http.ServerResponse

This object is created internally by a HTTP server--not by the user. It is passed as the second parameter to the 'request' event. It is a Writable Stream.

response.writeContinue()

Sends a HTTP/1.1 100 Continue message to the client, indicating that the request body should be sent. See the the checkContinue event on Server.

response.writeHead(statusCode, [reasonPhrase], [headers])

Sends a response header to the request. The status code is a 3-digit HTTP status code, like 404. The last argument, headers, are the response headers. Optionally one can give a human-readable reasonPhrase as the second argument.

Example:

Example 8-116.  

var body = 'hello world';
response.writeHead(200, {
  'Content-Length': body.length,
  'Content-Type': 'text/plain' });

This method must only be called once on a message and it must be called before response.end() is called.

If you call response.write() or response.end() before calling this, the implicit/mutable headers will be calculated and call this function for you.

response.statusCode

When using implicit headers (not calling response.writeHead() explicitly), this property controlls the status code that will be send to the client when the headers get flushed.

Example:

Example 8-117.  

response.statusCode = 404;

response.setHeader(name, value)

Sets a single header value for implicit headers. If this header already exists in the to-be-sent headers, it's value will be replaced. Use an array of strings here if you need to send multiple headers with the same name.

Example:

Example 8-118.  

response.setHeader("Content-Type", "text/html");

or

Example 8-119.  

response.setHeader("Set-Cookie", ["type=ninja", "language=javascript"]);

response.getHeader(name)

Reads out a header that's already been queued but not sent to the client. Note that the name is case insensitive. This can only be called before headers get implicitly flushed.

Example:

Example 8-120.  

var contentType = response.getHeader('content-type');

response.removeHeader(name)

Removes a header that's queued for implicit sending.

Example:

Example 8-121.  

response.removeHeader("Content-Encoding");

response.write(chunk, encoding='utf8')

If this method is called and response.writeHead() has not been called, it will switch to implicit header mode and flush the implicit headers.

This sends a chunk of the response body. This method may be called multiple times to provide successive parts of the body.

chunk can be a string or a buffer. If chunk is a string, the second parameter specifies how to encode it into a byte stream. By default the encoding is 'utf8'.

Note: This is the raw HTTP body and has nothing to do with higher-level multi-part body encodings that may be used.

The first time response.write() is called, it will send the buffered header information and the first body to the client. The second time response.write() is called, Node assumes you're going to be streaming data, and sends that separately. That is, the response is buffered up to the first chunk of body.

response.addTrailers(headers)

This method adds HTTP trailing headers (a header but at the end of the message) to the response.

Trailers will only be emitted if chunked encoding is used for the response; if it is not (e.g., if the request was HTTP/1.0), they will be silently discarded.

Note that HTTP requires the Trailer header to be sent if you intend to emit trailers, with a list of the header fields in its value. E.g.,

Example 8-122.  

response.writeHead(200, { 'Content-Type': 'text/plain',
                          'Trailer': 'TraceInfo' });
response.write(fileData);
response.addTrailers({'Content-MD5': "7895bf4b8828b55ceaf47747b4bca667"});
response.end();

response.end([data], [encoding])

This method signals to the server that all of the response headers and body has been sent; that server should consider this message complete. The method, response.end(), MUST be called on each response.

If data is specified, it is equivalent to calling response.write(data, encoding) followed by response.end().

http.request(options, callback)

Node maintains several connections per server to make HTTP requests. This function allows one to transparently issue requests.

Options:

host: A domain name or IP address of the server to issue the request to.

port: Port of remote server.

method: A string specifing the HTTP request method. Possible values: 'GET' (default), 'POST', 'PUT', and 'DELETE'.

path: Request path. Should include query string and fragments if any. E.G. '/index.html?page=12'

headers: An object containing request headers.

http.request() returns an instance of the http.ClientRequest class. The ClientRequest instance is a writable stream. If one needs to upload a file with a POST request, then write to the ClientRequest object.

Example:

Example 8-123.  

var options = {
  host: 'www.google.com',
  port: 80,
  path: '/upload',
  method: 'POST'
};

var req = http.request(options, function(res) {
  console.log('STATUS: ' + res.statusCode);
  console.log('HEADERS: ' + JSON.stringify(res.headers));
  res.setEncoding('utf8');
  res.on('data', function (chunk) {
    console.log('BODY: ' + chunk);
  });
});

// write data to request body
req.write('data\n');
req.write('data\n');
req.end();

Note that in the example req.end() was called. With http.request() one must always call req.end() to signify that you're done with the request - even if there is no data being written to the request body.

If any error is encountered during the request (be that with DNS resolution, TCP level errors, or actual HTTP parse errors) an 'error' event is emitted on the returned request object.

There are a few special headers that should be noted.

Sending a 'Connection: keep-alive' will notify Node that the connection to the server should be persisted until the next request.

Sending a 'Content-length' header will disable the default chunked encoding.

Sending an 'Expect' header will immediately send the request headers. Usually, when sending 'Expect: 100-continue', you should both set a timeout and listen for the continue event. See RFC2616 Section 8.2.3 for more information.

http.get(options, callback)

Since most requests are GET requests without bodies, Node provides this convience method. The only difference between this method and http.request() is that it sets the method to GET and calls req.end() automatically.

Example:

Example 8-124.  

var options = {
  host: 'www.google.com',
  port: 80,
  path: '/index.html'
};

http.get(options, function(res) {
  console.log("Got response: " + res.statusCode);
}).on('error', function(e) {
  console.log("Got error: " + e.message);
});

http.Agent

http.getAgent(host, port)

http.request() uses a special Agent for managing multiple connections to an HTTP server. Normally Agent instances should not be exposed to user code, however in certain situations it's useful to check the status of the agent. The http.getAgent() function allows you to access the agents.

Event: 'upgrade'

function (request, socket, head)

Emitted each time a server responds to a request with an upgrade. If this event isn't being listened for, clients receiving an upgrade header will have their connections closed.

See the description of the upgrade event for http.Server for further details.

Event: 'continue'

function ()

Emitted when the server sends a '100 Continue' HTTP response, usually because the request contained 'Expect: 100-continue'. This is an instruction that the client should send the request body.

agent.maxSockets

By default set to 5. Determines how many concurrent sockets the agent can have open.

agent.sockets

An array of sockets currently inuse by the Agent. Do not modify.

agent.queue

A queue of requests waiting to be sent to sockets.

http.ClientRequest

This object is created internally and returned from http.request(). It represents an in-progress request whose header has already been queued. The header is still mutable using the setHeader(name, value), getHeader(name), removeHeader(name) API. The actual header will be sent along with the first data chunk or when closing the connection.

To get the response, add a listener for 'response' to the request object. 'response' will be emitted from the request object when the response headers have been received. The 'response' event is executed with one argument which is an instance of http.ClientResponse.

During the 'response' event, one can add listeners to the response object; particularly to listen for the 'data' event. Note that the 'response' event is called before any part of the response body is received, so there is no need to worry about racing to catch the first part of the body. As long as a listener for 'data' is added during the 'response' event, the entire body will be caught.

Example 8-125.  

// Good
request.on('response', function (response) {
  response.on('data', function (chunk) {
    console.log('BODY: ' + chunk);
  });
});

// Bad - misses all or part of the body
request.on('response', function (response) {
  setTimeout(function () {
    response.on('data', function (chunk) {
      console.log('BODY: ' + chunk);
    });
  }, 10);
});

This is a Writable Stream.

This is an EventEmitter with the following events:

Event 'response'

function (response) { }

Emitted when a response is received to this request. This event is emitted only once. The response argument will be an instance of http.ClientResponse.

request.write(chunk, encoding='utf8')

Sends a chunk of the body. By calling this method many times, the user can stream a request body to a server--in that case it is suggested to use the ['Transfer-Encoding', 'chunked'] header line when creating the request.

The chunk argument should be an array of integers or a string.

The encoding argument is optional and only applies when chunk is a string.

request.end([data], [encoding])

Finishes sending the request. If any parts of the body are unsent, it will flush them to the stream. If the request is chunked, this will send the terminating '0\r\n\r\n'.

If data is specified, it is equivalent to calling request.write(data, encoding) followed by request.end().

request.abort()

Aborts a request. (New since v0.3.8.)

http.ClientResponse

This object is created when making a request with http.request(). It is passed to the 'response' event of the request object.

The response implements the Readable Stream interface.

Event: 'data'

function (chunk) {}

Emitted when a piece of the message body is received.

Event: 'end'

function () {}

Emitted exactly once for each message. No arguments. After emitted no other events will be emitted on the response.

response.statusCode

The 3-digit HTTP response status code. E.G. 404.

response.httpVersion

The HTTP version of the connected-to server. Probably either '1.1' or '1.0'. Also response.httpVersionMajor is the first integer and response.httpVersionMinor is the second.

response.headers

The response headers object.

response.trailers

The response trailers object. Only populated after the 'end' event.

response.setEncoding(encoding=null)

Set the encoding for the response body. Either 'utf8', 'ascii', or 'base64'. Defaults to null, which means that the 'data' event will emit a Buffer object..

response.pause()

Pauses response from emitting events. Useful to throttle back a download.

response.resume()

Resumes a paused response.

Appendix A. Installing Platform Dependencies

Foo bar baz qux

Appendix B. A Short Guide to JavaScript

Foo bar baz qux

Appendix C. WTFJS - Learning JavaScript's quirks

The http://wtfjs.com site provides some examples of the quirky excentricities of JavaScript. While amusing these snippets can help you squash those to find bugs when JavaScript just isn't behaving like you'd expect or learn a lot about how JavaScript really works. Thanks to Brian LeRoux the creator of WTFJS for curating these small insights into JavaScript.

General WTF

null is not an object

Example C-1. Classic null is not an Object.

typeof null // object
null instanceof Object // false

true has a value

@AtomFusion shows us that true sometimes has a value.

Example C-2. @AtomFusion shows us that true sometimes has a value.

(true + 1) === 2; // true
(true + true) === 2; // true
true === 2; // false
true === 1; // false

string is not string

Example C-3. When is a string, not a string? When it’s a duck - @rem

"string" instanceof String; // false.
// 'course it isn't not a string, it may look like a string
// but actually it's masquerading as a banana.

JavaScript Numbers WTF

magic increasing number

Example C-4. Look at me, I'm the magic increasing number! - @thomasfuchs

9999999999999999
//=> 10000000000000000

max vs the infinite

Example C-5. @pbakaus points out that Number.MAX_VALUE is close to infinity, but not too close.

Number.MAX_VALUE*1.0000000000000001 === (1/0) // false
Number.MAX_VALUE*1.0000000000000002 === (1/0) // true

isNaN

Example C-6. So, null is not an object though sometimes null is a Number.

isNaN( null ); // false
null === NaN; // false
null == NaN; // false

Example C-7. isNaN converts null to number. - @oleg008

Number( null ); // 0

min number treachery

Example C-8. It turns out that MIN_VALUE is the smallest number GREATER THAN ZERO, which of course totally makes sense.

Number.MIN_VALUE > 0;
//true? really? wtf.

not a number is not a not a number

Example C-9. Some argue this makes sense. Some ppl also like to sniff glue.

NaN === NaN // false

Appendix D. API documentation attribution

The API documentation from chapter 8 has been automatically generated from the Node.js project. It is licensed under the MIT license which is included below.

Thanks to the following people who have contributed to the API documentation: Ali Farhadi, Bert Belder, Brian White, Dan Søndergaard, Felix Geisendörfer, Joe Walnes, Kai Chen, Marco Rogers, Micheil Smith, Mikeal Rogers, Nikhil Marathe, Oleg Efimov, Oleg Slobodskoi, Ryan Dahl, Silas Sewell, Tim Caswell, Isaac Z. Schlueter, Koichi Kobayashi.

Node MIT License

Copyright Joyent, Inc. and other Node contributors. All rights reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Glossary

A glossary contains a collection of terms and brief descriptions or definitions of those terms. The basic markup follows.

Pseudo-class

A Psuedo-class is a way of creating an abtract object in JavaScript that is intended to be initialised into an object. Pseudo-classes should be turned into objects using the new keyword. Psuedo-class names start with a leading capital by convention to differentiate them from other kinds of objects. For example Server woudl be a psuedo-class and server might be an instance of that pseudo-class.

Class

See Pseudo-class

Method

A function which is a property of an object

Function

A unit of code that can be invoked with a set of variable parameters. It may pass a single return In JavaScript functions also have a context which defines the value of the reserved this variable. Functions in JavaScript are considered first class in that they can also be treated as variables or properties of objects.

About the Author

Tom Hughes-Croucher is a developer and technology evangelist. He’s worked for and with numerous well known brands, including Yahoo!, NASA, Tesco, Three Telecom, and UK Channel 4. Tom has contributed to a number of web standards for the World Wide Web Consortium (W3C) and the British Standards Institute (BSI).