Rewriting the Tracking Tool using Backbone.js

My name is Ben Bradshaw and I’m a Senior Developer at Catalyst IT Ltd, working on the New Zealand Post website. One part of the site I focus on is the Beta Tracking tool. I’m going to take you through the story of the Beta and look at the decisions we made and lessons we learned along the way:

  • Why we chose to use Backbone.js instead of Spine.js and JavascriptMVC
  • Why we use Underscore.js for templating and how we deal with cross site scripting risks
  • The benefits of using Model-view-controller
  • The good and bad aspects of using a framework
  • How we’re using Google Analytics, the risk of using a CDN and why we were willing to throw out 1000+ lines of working jQuery and start again

I think it’s great that I’m able to track my packages from postage to delivery. I’ve ordered items from Hong Kong, America and other countries. Most items have had tracking and it’s nice to see my parcel moving closer and closer to my door. I also buy and sell things on Trade Me and I’m usually sent a tracking number for my item.

The current Tracking Tool lets me check the status of my tracked item online. It takes each tracking code individually and is limited to 5 at a time. You can’t name items or receive notifications. I don’t find it very user friendly but it offers the information I want at the time, so I use it and then leave.

With the Beta we wanted to make a tool that was much more usable and made it easier to not only track your items, but keep track of them. The Beta Tracking Tool lets you name your items so you don’t have to remember tracking numbers like EA123456789NZ. It has one input box that lets you put multiple numbers in at once. If you’re logged in your parcels are also remembered for when you return. You can also opt to receive email alerts when the status changes, share tracked item details with your friends and even link up with Trade Me and see your auction details right there on the page or share with the buyer.

 The original Beta – jQuery

We relied heavily on jQuery to build the original Beta. Using jQuery saved us a lot of time and it is used for loading the information, the drop down details and recognising when users click on parts of pages. jQuery is a swiss army knife for Javascript, but as we went along we found that while it was capable of doing what we wanted, it certainly wasn’t the best tool for the job.

By the end of development there were a total of 1576 lines of jQuery. Each row had to have a unique CSS id so that we could use jQuery selectors to get to it. Every input was watched in case it changed, particularly for the Select All checkbox. We coded our own GET and POST requests to the server and we were passing HTML or JSON back and forth. We had to disable all the options until all the rows had loaded because we couldn’t trust the state of the table. The code was becoming harder to maintain and it was clear that something needed to change.

In spite of all this messiness the Beta was receiving lots of positive feedback. Happy customers were using it and enjoying it. A flat or negative response would have stopped further development, but instead the decision was made to put time into fixing the tool. Discussions were had, research was done and the end decision was that we needed to move away from jQuery and find an alternative. We stuck with Javascript but we knew we needed structure.

The new Beta – Backbone.js

We knew that we had an interface that had been well received, so we didn’t try to change any of the features or the appearance of the Beta. Parts were tidied up and behaviour changed when necessary or helpful, but to end users this looked like a bit of a polish, not a rewrite. We focused on the underlying logic and data, deciding to pursue the Model-View-Controller pattern. This meant we could divide the tool into three logical sections and control how we represented the data.

There are a few libraries that offered this in Javascript – JavascriptMVC, Spine.js and Backbone.js made our short list. JavascriptMVC was too heavy for our needs as it’s targeted at websites that are totally Javascript based like Grooveshark. Spine and Backbone are better for single page tools like ours. In a close race Backbone came out ahead by a nose because it’s maintained by a company that uses it in their product. Since we released the Beta we’ve watched the progress of Backbone. It’s under active maintenance and keeps getting new features, which is great for us because it gives us more options and new users, which means it’s more likely to be maintained going forward.

If you want to learn Backbone, I recommend you start with the Todo example and annotated source code that I used to start on the Beta. You should also learn about the Underscore.js library it uses, there are many useful functions in there. While it’s true that “Backbone’s only hard dependency is Underscore.js.” a lot of the cool stuff requires jQuery.js 1.4.2 or later (or Zepto). You’re very likely to want to use jQuery for other things though, so it’s not a big problem.

Structure

The first thing I did was to structure the data we were working with – in this case, tracked items. There are different types of information we’re dealing with:

  • Parcel Details – Tracking code, Tracking status, Last update time, Scan history, etc.
  • User information – Trade Me auction details, Parcel name
  • Code information – Is the user editing this item? Is it expanded? It is selected?

Strictly speaking I shouldn’t have Code information in the Model since it’s not relevant to anything except how to display the item. The compromise is that this information is never permanently saved – as soon as you refresh the page it’s forgotten.

I use Backbone Collections to group all the tracked items together. They also keep items in order so they don’t move around in between visits. The Collection lets me get all the items on the page (or just the items that are ticked) easily. It also lets me change those items so I can very easily make a “select all” option. Here’s the code for the select all:

  window.TrackedItemList = Backbone.Collection.extend({

    // Reference to this collection's model.
    model: TrackedItem,

    url: function() {
      var base = '/tools/tracking/ajax/trackeditemlist';
      return base;
    },
    // Filter down the list of all TrackedItems that are selected
    checked: function() {
      return this.filter(function(tracked_item){ return tracked_item.get('checked'); });
    },
    // Used by select all
    toggleAll: function(on) {
      this.each(function(tracked_item) {
          tracked_item.set({checked: on});
      });
    },
  ...

Storing data

A new feature in the Beta was the ability to have your numbers remembered for next time, this is available to users who have an account on the website but we put a lot of effort into making sure registration was quick and easy.

We save your searched items, names and linkups on the website by defining a URL for Backbone to save to. This is a simplified example with no error checking to:

  window.TrackedItem = Backbone.Model.extend({
    // Default attributes for the TrackedItem
    defaults: {
      tracking_code: false,
      tracking_status: false,
      tracking_status_detailed: "",
      tracking_last_update: '',
      tracking_scan_history: '',
      name: "",
      fetching: false,
      valid: null, // unknown at creation time
      editing: false,
      expanded: null,
      tracked: true,
      checked: false,
      auction: false,
      auction_details: null
    },

    url: function() {
      var base = '/tools/tracking/ajax/trackeditem/';
      return base + this.get('tracking_code');
    },
  ...

Since you can only have a parcel in your list once, we use the tracking_code as the unique key. Backbone uses POST, GET, PUT and DELETE calls to the provided URL to Create, Read Update and Delete items. There are two URLs used on the Beta – one for a single tracked item which is used to track changed to single items and, one for the whole collection which is used to load all your saved items when the page loads.

Once Backbone has sent the necessary information to the URL it is received and processed by Drupal, which includes saving names or codes into the database and making queries to the Tracking API to get the status of a parcel or the TradeMe API to get the details of your linked auction. Once Drupal has finished it returns a JSON formatted object back to Backbone which is then used to update the page so the user can see the new information.

Rendering

Rendering the tracked items requires a total of 8 templates. They cover valid and invalid items, editing and viewing items and other views like items loading and the API being unavailable. The templates are not known about by the Model, instead a Backbone View is created for each Model. This View has a render() function which contains all the logic for picking the correct template and then displays it. In Backbone each View renders itself individually, so when you change a Model only the HTML for that View is changed.

Backbone supports different kinds of templates. At the time I looked at Moustache, Handlebars and the default Underscore template engines. I liked the look of all three and I tried out Handlebars, but in the end I chose to use Underscore.js templates because they had more flexibility. The flexibility helped balance out the structure that Backbone imposed on the data. We were able to reproduce all the existing templates in Underscore without too much trouble.

The biggest problem I found with Underscore templates is the fact they don’t filter their output. This means users could put script tags and raw HTML into parcel names. I turned to OWASP for an answer to this problem and found Reform. From their site, “The Reform library attempts to provide a solid set of functions for encoding output for the most common context targets in web applications (e.g. HTML, XML, JavaScript, etc).” It supports Java, PHP, .NET, Perl, ASP and Javascript to name a few.

I added the Reform htmlEncode function to the global window object and call it from the templates, safely sanitising outputted text.

window.htmlEncode = function HtmlEncode($str, $default) {
  if($str == null || $str.length == 0) {
    $str = ($default == null ? '' : $default);
  }
  $out = '';
  $len = $str.length;
  ...
}
  <th class="tracking-number">
    <%= htmlEncode(tracking_code) %>
  </th>

Serving templates

One thing we had to do was work out a way to supply templates to the page that Backbone could use for rendering. We didn’t want all the templates in a single static file (to make editing and change tracking easier) and we didn’t want to include multiple template scripts on the page like the Todo example. We wrote a Drupal callback that stores all the templates in window.Templates before Backbone loads.

  window.TrackedItemView = Backbone.View.extend({
    tagName:  "tbody",
    template_fetching: _.template(Templates['trackeditem-fetching.tmpl']),
    template_apidown: _.template(Templates['trackeditem-apidown.tmpl']),
    template_invalid: _.template(Templates['trackeditem-invalid.tmpl']),
    ...

    render: function() {
      if (this.model.get('fetching')) {
        $(this.el).html(this.template_fetching(this.model.toJSON()));
      }
      /*
       * Invalid tracking code
       */
      else if (this.model.get('valid') == false) {
        $(this.el).html(this.template_invalid(this.model.toJSON()));
      }
    }

Happy developer, unhappy designer

One thing missing in Backbone that we already had in the Beta was timed transitions and the other nice effects. The Todo example shows how Backbone cares more about changing the HTML and rendering the content than it does about transitions and fades. As a developer this doesn’t bother me, but it is a weakness when you’re trying to build attractive applications that are nice to use. To get the effects back we re-enlisted the faithful jQuery library and some creative CSS.

The slide down that gives you extra information about a tracked item is a good example of this. Without any effects clicking on Show detailed used to immediately show the extra info and Hide details used to immediately hide it. This happened by adding an .expanded class to the extra details div which was set to display: block;.

To get the full transition working, we change the .expanded class to display: none; and after Backbone had rendered the Template we called jQuery .slideDown(). In the opposite direction (sliding up) we use CSS to show the details (display: block;) and jQuery .slideUp() to hide it.

Mobile Tracking

Of course all this is great if you’re at a computer, but not if you’re out and about. We wanted to give you the ability to check your parcels from your phone so we made the original Mobile Tracking tool. This tool was all new code, written before we started looking at Backbone. It let you track parcels and remembered them on your phone (using HTML5 localStorage), so you didn’t have to keep typing them in every time you visited.

This was good, but we wanted to use Backbone where we could, so we ported the Beta tool to the mobile site. The Models and Collections were renamed but the structure of their data stayed the same. We wrote new Templates that were cut down to be mobile friendly and a new Controller that responded much better to touch devices.

It should be noted that while Backbone supports localStorage and remote storage (via URLs) it does not seem to like both at the same time. I had to rewrite bits of the localStorage plugin to handle the fact that Backbone had to request a URL to get the status of a parcel but should not try to save information anywhere except localStorage.

Because we like making things better, we decided to keep going. Soon you’ll be able to log in to the mobile site and the parcels you’re tracking in the Beta tool will be right there, complete with the ability to change their names.

 

What we’ve learned

Before I started on this piece of work I hadn’t worked with Backbone and my MVC experience was what I had learned at University. There was a learning curve and a lot of time spent figuring out how Backbone did things and how to use it. There are still things I’d like to improve and I’m sure that as we keep adding features we’ll clean up as we go.

Looking back on the upgrade, I’ve picked out some things we could have done better and some things I’d recommend doing.

Your code won’t be perfect (or if it is, you’re releasing too late)

Let’s face it, your code isn’t going to be perfect. It’s going to have compromises in it and things are going to be a bit rough around the edges – that’s okay. I’ve found that the principle of the 80-20 rule is one to remember – you don’t have to make something perfect for people to like it and use it – but you do have to make it well. The great thing about websites is that we can fix bugs, upgrade features and release an update rather easily and quickly. As a result, there will always be something I’d like to fix, but these are my top three.

The first thing to fix is the way we are displaying the Share, Subscribe and other modals on the page. They were all made at different times by different developers and there is no consistency in their implementation. We’re beginning to standardise on modalframe for modals and my plan is to replace all the custom modals with modalframe modals.

After that, I’d like to diagnose and fix my code so that it properly sends DELETE requests to the server. Right now the code watches for when a delete happens and uses jQuery to send the request, which is not a permanent solution.

Finally, the back end code could do with a clean up and better structuring. Code left over from the original Beta makes the file much harder to read and larger than it should be. Old code needs to be deleted and any dependencies need to be fixed and the current code would benefit from being broken up into smaller functions.

Plan before you code

The original Beta tool started life much simpler than it is today. jQuery was not a bad choice initially, but each new feature added more complexity and the freedom jQuery gives allowed us to build up a lot of technical debt that needed to be paid off.

The rewrite came at a time when the feature set was well rounded and so we thought about how to structure it. We also had discussions with the Business Owners to get a feel for which features they’d like to see in the future and used those to make sure we’d chosen a solution that would last.

You also need to know that the framework you choose supports the browsers you want to support. We support Internet Explorer 6+, Firefox, Chrome and Safari on Mac, so we picked a framework that supported them too.

Understand that there are no silver bullets

No framework is going to fix all your problems and make your site what you want it to be. The structure Backbone places on us is a bit limiting at times and requires compromises on occasion. The positive aspect of this is that our Templates are easy to update, our data is coherent and we don’t need to put unique IDs on each row anymore. To my mind, that’s a very worthwhile trade.

Frameworks can also be used poorly and Backbone is no exception, particularly if you’re starting fresh like I did. There are bits of the code I’d like to redo now that I’m more familiar and comfortable with Backbone, but we’re still miles ahead of the old tool and I plan to do tidy ups as I go.

Keep an eye on the ecosystem

New frameworks are being created all the time and released under Open Source licenses that allow them to be used by others at no charge. There has been a lot of interest and development in Javascript recently resulting in a lots of cool code we can use. It pays to keep an eye on what’s going on in the areas you work in.

Get involved too. Many frameworks have support channels via IRC or email/forums – don’t be afraid to join and see what people are discussing, ask your questions and help others where possible.

Backbone.js 0.1.0 was only released on the 14th October 2010, after the original Beta was released. When I started development the latest version was 0.3.3, when I had finished 0.5.1 had been released. Since we released, other examples of websites using Backbone have appeared all over, including Linkedin, Soundcloud, Trello and many more.

Make sure you have a CDN fallback

We originally used CDNjs to serve up Backbone.js, Underscore.js and json2.js and the Google CDN for jQuery. We took the time to add the versions we needed to CDNjs so they were available and we launched the tool.

We didn’t write any code to handle the files going missing or being served up incorrectly and sure enough, this happened. The result was the tool didn’t work and customers were not told why. As a result we moved to hosting files locally (except jQuery) and added code to inform users when the files weren’t available.

What we plan to do in a future release is have a graceful fallback like this one from html5 boilerplate:

<!-- Grab Google CDN's jQuery, with a protocol relative URL; fall back to local if necessary -->
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.js"></script>
<script>window.jQuery || document.write("<script src='js/libs/jquery-1.5.1.min.js'>\x3C/script>")</script>

It should be noted that we have not seen the Google CDN fail, but if it ever does we can make sure we’re covered.

Watch your users and learn

It’s common for websites to use web analytics to track which pages are most popular, but when your users only visit a single page it requires manual steps to still see what users are doing. We use Google Analytics events to get a broad view of which features are being used and which aren’t as total number of uses over a time period. We don’t get any unique data from users, but we do see which features are popular and that helps us decide where to put our efforts.

We also have multiple ways users can submit feedback. Suggestions like the ability to name items that haven’t been scanned yet and offering email alerts let us know what our users wanted and we built these features in to the Beta.

Final thoughts

Would I recommend Backbone.js to someone else? Yes I would, but I’d also tell them to shop around and look at Spine.js, which has also had improvements made to it since I last looked. We chose Backbone because it met our needs, your needs will be different.

The decision to rewrite the tool was not taken lightly. It took longer than expected and meant we threw out a lot of code that worked, but was very difficult to maintain. Sometimes you have to accept that what you have isn’t meeting your needs and it needs to be replaced.

Already we are reaping benefits from the rewrite. Mobile tracking is linked with the Beta now and it took less time than expected. We’re also going to save time and money in the future with changes to the Beta being easier, safer and quicker. This means we can be more confident in what we release and we can do it faster.

If you have any questions about the upgrade, Backbone or anything else I covered, leave a comment and I’ll do my best to answer them. The same goes for any features you’d like to see in the Beta – let us know what you think, we’re listening.

This entry was posted in Uncategorized and tagged . Bookmark the permalink.