justjs: node.js tutorials

New here? You might want to start at the beginning.

5/20 '12

Let's Post Something! (In which our blog becomes a blog, just barely)

Wait. What just happened?

In yesterday's installment, we did three very cool things:

1. We created a "db" module that stores and retrieves posts for us. That module happens to use MongoDB, but if we changed our minds, we could write a new module with the same methods and "require" that module instead, and the rest of our code would be none the wiser. That makes it much easier to maintain our code.

2. We learned how to do simple things simply. The async.series function lets us carry out a series of steps one at a time, even though the code that implements those features returns right away and doesn't invoke our callbacks until later.

3. We learned a lot about nested functions and closures, key features of JavaScript that make callback-driven programming (Node's defining characteristic) soooo much more pleasant than in most languages.

But there's one thing we didn't do: run our app! That's because we stopped short of exploring the last new module in this version: app.js.

The app.js module: controlling the action

In previous versions of the blog, our code for displaying blog posts was pretty much the same as our code for fetching them. (To be fair, we didn't have a database yet, so the code for fetching them was awfully simple.) Yesterday we busted out the code for fetching and creating blog posts to a new module.

In the new version, we'll break out the code that responds when a browser accesses a URL - the routes, in Express lingo - to a separate module as well. As we saw yesterday, this makes server.js very short indeed. Now server.js is just responsible for deciding what modules go into the project, initializing them, and starting up our webserver.

So let's take a look at app.js. But first we'll install a new npm module, justly famous in both the Node world and the browser-side [removed] underscore.

Introducing Underscore

Underscore fills many otherwise serious gaps in JavaScript- a beautiful language, but it has its faults. Underscore overcomes many of them by providing truly reliable ways to manipulate arrays and objects, especially when we're using objects like associative arrays in other languages.

We'll use Underscore for many things in this blog, but today we want it for the ability to easily pick just the properties we want from an object.

Use these commands at the terminal prompt to pull underscore into your project:

cd node-apps/blog-3
npm install underscore

Building the app module

Now let's see some code! The first two lines of app.js just summon the underscore and express modules:

var _ = require('underscore');
var express = require('express');

Now let's look at the way module.exports is structured in this module:

module.exports = {
  init: function(context, callback) {
    // Register all of our routes here, 
    // then invoke the callback...
    callback();
  }
};

"Hey, this could be simpler. Can't you just set module.exports to the init function and be done with it? And why have a callback at all if we're just going to call it right away?" 

Good eye. Yes, technically I could just write:

module.exports = function(context) { ... }

And expect server.js to know that requiring this module just returns a function, which I could call like this:

// Require the module and immediately invoke the function it returns
require('app.js')(context);

But there are two problems with this:

1. I'll probably want to add more methods to this module at some point. Once I do that, the cute trick of just returning the init() function doesn't work so well.

2. Sooner or later, init() will probably wind up calling something that returns immediately and invokes a callback later. By planning in advance for a callback function, I can avoid aggravation down the road.

Still, it's good that you realize modules won't always follow the same pattern I've shown here.

Let's dig into the init() function. app.init() starts by creating our Express server object and saving it in the context for use elsewhere. We also save a reference to it in a local variable for convenience:

    var app = context.app = express.createServer();

We're almost ready to set up routes. But first, since we're going to want to let the user write new posts, we need to be ready to deal with the POST HTTP method. As you know if you have written HTML forms before, a basic HTML form can submit to the server via one of two methods: GET and POST.

GET is handy for search engines and the like because the submitted fields wind up right in the URL. But since URLs have a limited length, and ideally should not be used as "verbs" that change things, we normally write:

<form method="POST" action="/new"> ... </form>

For a form that submits something big like a blog post.

"So wait, you're telling me Node can't handle forms?" Not at all- of course Node can handle POST. But Node doesn't automatically assume you want the form submission parsed for you in advance. For instance, if we were handling giant file uploads, we might prefer to handle it ourselves with a callback and display a progress meter while the upload is still going on.

For a form like this, though, parsing the form submission in advance sounds great. So let's add some middleware.

Express routes are very extensible. You can add "middleware" functions to any route, just by passing them as additional arguments before the final callback function.

Express includes a number of middleware functions, via a module called Connect that is baked into Express. One of these is express.bodyParser(), which returns a middleware function that can automatically parse POST requests and pop them into properties of the req.body object for us.

That lets us write:

req.body.title

We could install the body parser middleware for each route that requires it:

app.post('/new', express.bodyParser(), function (req, res) { 
  // use req.body like a boss 
});

But while this is handy, we have the even more convenient option of installing it for every route automatically. So we'll do it that way:

app.use(express.bodyParser());

"If express.bodyParser is a middleware function that the route calls later, why do we follow it with ()? Doesn't that call it now?" You're right. express.bodyParser is actually a function generator - a function that takes the options we pass as arguments and then returns an inline function that can still see those options in its scope later. In other words, it returns a closure. But don't worry, you don't have to worry about that today if your brain is already full.

Now that we've added the body parser, let's look at the "/" route that displays all the blog posts. Most of this function has not changed from the previous version. The big change is that we call context.db.posts.findAll() to fetch our posts, and we pass an inline function that receives the posts back when the database is ready.

Here's the entire function. Notice that the previous version of this route is pretty much intact, nested in the callback that receives the posts:

    app.get('/', function(req, res) {
      context.db.posts.findAll(function(err, posts) {
        if (err)
        {
          notFound(res);
          return;
        }
        var s = "<title>My Blog</title>\n";
        s += "<h1>My Blog</h1>\n";
        s += '<p><a href="/new">New Post</a></p>' + "\n";
        s += "<ul>\n";
        for (var slug in posts)
        {
          var post = posts[slug];
          s += '<li><a href="/posts/' + post.slug + '">' + post.title + '</a></li>' + "\n";
        }
        s += "</ul>\n";
        res.send(s);
      });
    });

Even though we now fetch real posts from a real database, our code didn't get much more complicated, apart from introducing a callback. And we're not dependent on MongoDB at all - as long as there's a context.db.posts object to talk to, we're happy. This simplicity is the big payoff of moving the database code into its own module.

The code to fetch and display a post hasn't changed too much either:

    app.get('/posts/:slug', function(req, res) {
      context.db.posts.findOneBySlug(req.params.slug, function(err, post) {
        if (err || (!post))
        {
          notFound(res);
          return;
        }
        var s = "<title>" + post.title + "</title>\n";
        s += "<h1><a href='/'>My Blog</a></h1>\n";
        s += "<h2>" + post.title + "</h2>\n";
        s += post.body;
        res.send(s);
      });
    });

We're still rocking the req.params object to access the named parameters in the route. The only change, again, is the introduction of a callback to receive the post from the server. If we don't actually get a post, we call notFound(), just as in the previous version.

Now let's move on to the cool bit: creating and saving new blog posts. For this, we have two different routes for the same URL: /new. The difference is that one responds to GET requests and the other to POST requests. We'll use the GET method version to display the form, and the POST method version to save it... or, when things don't go so well, to display it again with an error message.

The code for the actual form will be in a newPost() function shared by both routes. So the app.get('/new') route is very simple:

    app.get('/new', function(req, res) {
      newPost(res);
    });

The app.post('/new') route is only a little more complicated, because we already wrote a sweet context.db.posts.inert() method that takes are of generating a reasonable slug from the title and then saving the post. All we have to do is make sure we fetch the right properties from the user's form submission. And that's where underscore's _.pick() function comes in:

    app.post('/new', function(req, res) {
      var post = _.pick(req.body, 'title', 'body');
      context.db.posts.insert(post, function(err, post) {
        if (err)
        {
          newPost(res, "Make sure your title is unique.");
        }
        else
        {
          res.redirect('/posts/' + post.slug);
        }
      });
    });

In particular, check out this line:

      var post = _.pick(req.body, 'title', 'body');

Here we are interested in the "title" and "body" fields in the form submission. If any other fields were submitted, we're not interested, and we don't want them to clutter up our database. So we use _.pick(), a function that accepts an object and a series of property names, and returns a new object containing only the properties we asked for.

A note on security and user-submitted data

Our use of the pick() function here is a good first step toward validating the user's data. But it is only a baby step. It is important to realize that web browsers can submit anything... anything at all... no matter what our HTML and JavaScript code might say. That's because it is trivial for any halfway skilled hacker (or even your little brother) to gin up a script that ignores your markup and submits things you do not want on your website.

In the production version of justjs.com, I do more to validate form submissions (even though I'm currently the only poster to the blog). In future installments we'll explore tools like the _.defaults function, which is great for ensuring that no fields were skipped, and the validator npm module, which can be used to escape any HTML markup in submitted text, or just to validate submitted HTML to make sure it is not malicious.

On the other hand, there's a big area of security we don't have to worry so much about. Since we're using MongoDB's JavaScript-based API, not SQL, we don't have to worry about escaping queries to prevent SQL injection attacks. MongoDB's API was designed from the beginning to keep commands and data separate so the issue doesn't arise.

Inserting the Post

Once we have the post object the way we want it, we just call db.posts.insert() to insert it. If something goes wrong, we assume a conflict with another post that has the same title and call the newPost() function to redisplay the form. (Currently, we don't bother to redisplay it with the previous submission intact. We'll do more of that in future installments, in which we'll also introduce a nicer way of templating HTML markup.)

If everything works out great, we use res.redirect() to send the user to the URL of their shiny new post, just so they can have a special moment. Redirecting after a form submission is a good idea to prevent accidental resubmission of the same form if the user refreshes the page.

Displaying the Form

Finally, here's the newPost function that displays the form. I've kept the markup basic for now, since we'll be introducing nicer ways to handle HTML in future installments.

    // Send the "new post" page, with an error message if needed
    function newPost(res, message)
    {
      var s = "<title>New Post</title>\n";
      s += "<h1>My Blog</h1>\n";
      s += "<h2>New Post</h2>\n";
      if (message)
      {
        s += "<h3>" + message + "</h3>\n";
      }
      s += '<form method="POST" action="/new">' + "\n";
      s += 'Title: <input name="title" /> <br />' + "\n";
      s += '<textarea name="body"></textarea>' + "\n";
      s += '<input type="submit" value="Post It!" />' + "\n";
      s += "</form>\n";
      res.send(s);
    }

A few last details: notFound() and the callback

We still need our catch-all route to display a "404 not found" error for wacky, made-up URLs, and our notFound() option to render that. This code hasn't changed a bit since the previous version.

    app.get('*', function(req, res) {
      notFound(res);
    });

    function notFound(res)
    {
      res.send('<h1>Page not found.</h1>', 404);
    }

What's more interesting is the code that follows, at the very end of the init() method. As I mentioned earlier, we need to invoke the callback so that server.js knows our routes are ready:

    callback();

Again, doing things this way gives us the option of calling time-consuming functions that require a callback later.

Fire it up!

That's it- we've got a blog that's ready for posting! Let's fire up the server the usual way:

cd node-apps/blog-3
node server.js

Then connect to http://localhost:3000/ and enjoy. Lo and behold, the "New Post" link takes us to a form that successfully adds new posts to the blog. And they sort in newest-first order, as blog posts should.

Coming Up

So that's it right? Oh, not hardly. There are so many holes in this blog it ain't funny.

Things we haven't addressed yet (a partial list):

Templating. Currently we're writing HTML by concatenating JavaScript strings. This is not the easy-to-maintain way to go. It also represents a failure to separate concerns. Right now we have a nice "model" layer (the db module) and a nice "controller" layer (the app module). But the "view" layer is also shoved into the app module, as an afterthought.

Layout. Right now we have crummy bare-bones HTML pages that aren't very consistent. We need a way to wrap the markup for each route in an easily updated, attractive site-wide layout.

Access control. Everyone can post! That's how your blog works, right? Yeah, not so much. We need to log the user in, and decide what the user is allowed to do on the basis of who they are.

Session management. Part of logging people in is remembering that they are logged in.

Editing. We can post. But we can't edit or delete existing posts. We need to round out the feature set.

Better performance through web services. So far we've been assuming the web browser is a pretty dumb device, capable only of displaying web pages and filling out forms at the most. But in case you haven't noticed, it's 2012 and even Internet Explorer is pretty bright these days. gmail arrived a long time ago. We should be able to leverage the browser to speed up the user's experience and do more of the work on the browser side. At the same time, we need to do a rock-solid job of delivering the same content to dumb devices- and to Google and other search engines that are effectively equal to a web browser with JavaScript turned off. But we don't have all day to write duplicate code for two different devices. Fortunately, we don't have to - it's all JavaScript. We'll explore ways to leverage JavaScript's ubiquity to achieve good SEO and a modern user experience at the same time.

Looking forward to sharing more sweet code with you!

 

blog comments powered by Disqus