Sunday, June 9, 2013

Building Modular Backbone Apps with RequireJS and Bower

One goal in application design is to properly architect separation of concerns by carefully compartmentalizing different functionality and avoiding coupling components such that its difficult to reuse or change them in future projects. While its easier to achieve this within a specific component by properly defining models and views, it becomes more challenging when different logical components need to interact. As a simple example, you can see that a products component can be separated from a customer component. However, an order component would clearly have an interest in consuming some of the functionality available in both the product and customer components. Defining clear public interfaces to those different components is all part of modular design and, as projects grow, this methodology sometimes breaks down in the interest of time or various other reasons.


An Idea



I wanted to try was to completely separate each component's development from the main application. Each component would be built as a library that could be used in any other project as needed. The final application would essentially load the required components, manage the navigation, and act as a mediator between components. I knew that RequireJS was probably a good candidate for managing the definition of modules and handling dynamic loading of components. However, I still needed something to help manage the project dependencies so all the required libraries could be easily refreshed as needed. It was less searching and more accidental that I found this tutorial on using Bower with RequireJS and Backbone to setup a project environment with all the dependencies automatically downloaded and available in the environment. Basically, its Ruby's Bundler but for Javascript projects which is exactly what I wanted. Of course, its still pretty new and not every library out there is using it. However, the basic Backbone requirements are available and, more importantly, it can be used to define my different Backbone-based component libraries.


Proof of Concept



A basic, useless proof of concept follows, but shows the principal ideas behind the layout of the different projects. I've already mentioned that I want different logical application components to be self-contained libraries physically separated from the main application project. Additionally, I want to ensure I can both test while developing and also generate an optimized build for production. Below is the project structure I'll be referring to through out the rest of the discussion. I've created a project on GitHub with this structure and all the files used to create the sample application/component.


projects
|
|- webapp1 # The main application
| |- bower.json # Bower package file. Includes component1 dependancy
| |
| |- app # Application code goes here
| | |- css
| | |- scripts
| | |- main.js # Main entry point. Requires app and starts
| | |- app.js # Defines navigation, mediates, requires components
| |
| |- build
| | |- build.js # Defines single optimized file
| | |- build.sh # Run r.js, copy dependencies to dist directory
| | |- index.html # Production main page. Loads the final optimized bundle.
| |
| |- components # Bower installs dependencies here
| |- vendor # Put non-bower things here
| |- tpl.js
| |
| |- dist # Target of the production build
| |
| |- test
| |- config.js # Loads everything with no optimization
| |- visual.html # Browse here to test
|
|- component1
|
|- bower.json # Defines the component and its dependencies
|- component1.js # Built component after r.js optimization
|- component1.min.js # Minified version
|
|- build # Create the stand-alone AMD compatible component
| |- almond.js
| |- build.js
| |- build.sh
| |- wrap.start
| |- wrap.end
|
|- components
|- vendor
| |- tpl.js
|
|- lib # Define Backbone component here main.js is the
| |- main.js # top level entry point for testing and building
| |- scripts # the final self-contained component. Each level
| |- models.js # builds the top-level Component1 object so the
| |- views.js # final component can be referenced globally
| |- models # or via RequireJS as component1
| | |- model1.js
| | |- model2.js
| |- views
| |- view1.js
| |- layout # The optimized version will use tpl to pre-compile
| |- view1.html # all the micro-templates and include them in the build
|
|- test # Test each variation of the component:
|- index-build.html # The build component1.js using RequireJS to load
|- build.js
|- index-lib.html # Not built, load all the individual files one-by-one
|- lib.js
|- index-global.html # Build, but do not use an AMD, just the global object



Creating a Component Library



Before writing the application, I'm going to need a component or two that will be used in the application. I'm using the word "component" here to not confuse it with module or package which are terms you'll see in RequireJS and Bower. A component could be really anything. I'm using concepts like customer, product, and order as examples, but the point is to create something that can stand on its own and provide meaningful functionality.

Define the Package

Before writing any code, I'm going to create my Bower definition. The package file actually works in two ways - one, when you're developing the package, it makes it easy to fetch and refresh all your dependencies, and second, once built, it can be referenced in other projects using Bower to manage external dependencies.

bower.json:

{
"name": "component1",
"version": "0.0.1",
"main": "component1.js",
"ignore": [
"components",
"test",
"build"
],
"dependencies": {
"jquery": "latest",
"backbone-amd": "latest",
"underscore-amd": "latest"
},
"devDependencies": {
"requirejs": "latest"
}
}


Saving this file in the root of my component project file and running bower install from the command line will cause bower to fetch the declared dependencies and and place them in the components directory. Since the dependencies I'm requesting are registered, I don't need to provide any location details. However, bower does allow specific locations to be declared. These can be git repositories or on the local filesystems. I'm going to leverage the latter feature when creating the application's bower definition. The other parts of the bower.json file are intended to define the package you're building. This information will be used when installing this package as part of another bower dependency map. I'm using the ignore directive to ensure only the lib directory is pulled into a dependent project. There's no need to have the other parts published since they are part of the development process.

Unfortunately, not everything has a Bower package file so you may still have to manually grab libraries you need to use. You can't put them in the components directory since its basically wiped out every time bower refreshes. I created a separate vendor directory to hold those non-bower libraries. For this example, I needed to use requirejs-tpl to handle loading view template files. Since this is not bower aware, I downloaded it and added it to this directory.

Write the Code

Once my dependencies are available, I can start developing the library. The normal Backbone project layout can be used here. The only difference is I'm going to wrap everything in RequireJS define() functions. Level-by-level, I'm going to declare the immediate lower level's definitions as dependencies so they are loaded and available in the definition function.

So, in the example component project, the main.js entry point would look like this:


define(['./scripts/models', './scripts/views'], function(models, views) {

Component1 = {
Models: models,
Views: views
};

return Component1;

});


main.js is in the library base directory (lib) so it will require anything in the directory immediately below it - scripts. In the scripts directory, the are two files - models.js and views.js. These will take care of the next level of directories. The resulting objects will be added to the main object definition Component1 and returned. Note the use of relative directories. This will become important when using this as a package while developing the application later.

I'll follow the views definition down all the way to one of the actual views as an example.

./scripts/views.js:

define(
[
'./views/View1'
],
function( View1 ) {

var Views = {
View1: View1
};

return Views;

});


You can see how that can get pretty lengthy if you have a lot of views. In the example project, I started playing with an idea to map all the arguments of the function to the final Views object. However, for simplicity, I left that out of this discussion.

./scripts/views/view1.js:

define(['jquery', 'underscore', 'backbone', 'tpl!./layout/view1.html'],
function( $, _, Backbone, template ) {

return Backbone.View.extend({

render: function () {

this.$el.html(template());

return this;
}

});

});


This is the first time I need to require any of the external libraries like jQuery and Backbone. Everything else has been just to build up the final component. I also made use of the requirejs-tpl plugin to load plain text Underscore templates from the layout directory and compile it before passing it to the definition function. Later, I'll use this to generate pre-compiled templates in the final build.

Make Sure it Works

Once there's some definitions in place, it might be nice to test them and ensure they work. In this setup, I've placed several alternative RequireJS configuration definitions in the test directory and created a few HTML pages to refer to them so I can test the various different uses of the component. For development purposes, I want to load all the files individually using RequireJS to solve the load order and ensure dependencies are loaded before loading the next module.

test/lib.js:

(function () {

require.config({

baseUrl: "../lib",

paths: {
"jquery": "../components/jquery/jquery",
"underscore": "../components/underscore-amd/underscore",
"backbone": "../components/backbone-amd/backbone",
"tpl": "../vendor/tpl"
}

});


require(["jquery", "main"], function( $, mod ) {

var view1 = new mod.Views.View1();


$(document.body).append(view1.render().$el);

});

})();



Then, in my HTML file, I just need to load RequireJS and set the data-main to "lib".

test/lib.js:

<script data-main="lib" src="../components/requirejs/require.js"></script>


The example project on GitHub also has definitions for testing the built component using RequireJS and as a global variable without using RequireJS.

Build and Optimize

Now, the final goal of the creating a component (ie order, customer, etc) is to have a single, AMD compliant file that can be consumed in a downstream component or application. It's similar to making a widget in jQuery UI. Once built, there should be a file in the root of the project with all the optimized code in one file excluding any external vendor's libraries. There is an example library in the RequireJS repository that illustrates how to setup a optimization build that packages only the library code while leaving the dependencies out and uses almond as a stand-in for RequireJS. I used that as a starting point for the optimizer build definition.

build.js:


{
baseUrl: "../lib",
include: ["../build/almond", "main"],
exclude: ["jquery", "underscore", "backbone"],
stubModules: ['tpl'],
out: "component1.js",

wrap: {
"startFile": "wrap.start",
"endFile": "wrap.end"
},

paths: {
"jquery": "../components/jquery/jquery",
"underscore": "../components/underscore-amd/underscore",
"backbone": "../components/backbone-amd/backbone",
"component1": "../component1",
"tpl": "../vendor/tpl"
},

optimize: "none"
}


Here, you can see it follows the example to use almond to emulate an AMD inside the components definition. This enables us to internally leverage the modular encapsulation built-in to the design of RequireJS without changing how any of our code is written or organized while providing the outside world with both an AMD-compliant definition or a global object if no AMD is detected. Additionally, we're not including jQuery, Underscore, or Backbone in the build because that will be handled by the final consumer of the library. It would be wasteful to include those libraries here and then have them potentially included somewhere else. Finally, since I'm using requirejs-tpl to load the underscore micro-templates, I can use the optimizer to load all the templates pre-compiled into the final component's build. This eliminates the need for the plugin so the stubModules: ['tpl'] can be used to exclude the definition but leave a stub in place to allow the dependencies to properly load.


An Application of Components



The next step is to create an application to use the component(s) we've built. Again, we'll start with the Bower definition. This is where the dependency for component1 is declared so its copied to our components directory.

bower.json:

{
"name": "webapp1-test",
"version": "0.0.1",
"dependencies": {
"requirejs": "latest",
"jquery": "latest",
"component1": "../component1"
}
}


Running bower install will copy component1, and install all its dependencies (if not already defined in the application's). Once all the dependencies are in place, we're going to want to be able to test our application differently than how it will run production. I've put an HTML file in the test directory and created this RequireJS config to enable loading all the files individually, including those from component1.

test/config.js:

(function () {

require.config({

baseUrl: "../app/scripts",
paths: {
"jquery": "../../components/jquery/jquery",
"underscore": "../../components/underscore-amd/underscore",
"backbone": "../../components/backbone-amd/backbone",
"tpl": "../../vendor/tpl"
},

packages: [
{
name: 'component1',
location: '../../components/component1/lib',
}
]

});

require(["main"], function(main) {});

})();


Since I told Bower to keep the lib folder when installing the component1 package as a dependency, I can setup a packages hash in the RequireJS config and point to the main.js file. Now, all definitions requiring component1 will load component1's main.js which will cause all the other files to load individually. This is where the relative paths become important. If I did not use those in the component1 define() functions, RequireJS would try to resolve those dependencies relative to the baseUrl defined here (../app/scripts). Obviously, those files are not there but in the components directory. Although not shown, if I decided not to load all the component1 files, I could switch to loading just the optimized component1.js file by removing the packages hash and adding component1 to the paths hash.

Ready for Production

Once development is complete and the application is ready for production, you're going to want to minimize the amount of round trips to the server to load the application. In the example I've built here, I have the main application which includes jQuery, Underscore, Backbone, and RequireJS. Component1 is an additional package I want to load, but maybe defer to avoid loading it right when the main application loads.

For my first attempt, I added component1 to the paths hash in the build.js file and required it immediately in my main.js file.

build/build.js:


({

baseUrl: "../app/scripts",
name: 'main',
out: "../dist/scripts/bundle.js",

paths: {
"requireLib": "../../components/requirejs/require",
"jquery": "../../components/jquery/jquery",
"underscore": "../../components/underscore-amd/underscore",
"backbone": "../../components/backbone-amd/backbone",
"component1": "../../components/component1/component1"
},

include: ['requireLib'],

optimize: 'uglify2'
})


app/scripts/main.js:

require(["jquery", "underscore", "backbone", "component1"], function($, _, Backbone, c1) {

var view1 = new c1.Views.View1();

$(document.body).append(view1.render().$el);

});


This method caused the optimizer to attempt to package everything into one bundled file. However, running r.js using this setup resulted in the follow error:


Tracing dependencies for: main
Error: Error: nope
at check (/usr/local/lib/node_modules/requirejs/bin/r.js:2789:23)


Not the most descriptive error which made it quite hard to find what might have caused the issue. After several failed searches, I finally found this discussion which pointed to the wrapper around the component1 library using a define/factory pattern. Really, I didn't want the optimizer to include the component1, just skip it and load it from a separate file. This would give me the flexibility to load it whenever I wanted as the application started getting larger. I found this discussion that showed how to define the dependency but get the optimizer to basically skip it.

build.js:

paths: {
...
"component1": "empty:" # Note the colon at the end - its important!
},


When I do that, I'll need to copy the file over to the dist directory manually so its available to RequireJS to load as the page loads

build.sh:

cp ../components/component1/component1.min.js ./scripts/component1.js


Another option to avoid this problem and gain the advantage of not requiring the dependency on initial load is to push it down into the application logic and load it only when necessary. By doing this, the optimizer will ignore it and not try to pull it into the build. This is probably a more likely use case anyways since, if you make this component, you probably don't want it loaded until the user needs to use it. Instead of using the above setup, I created a separate app.js module to create a simple start function that loads the component and renders an instance of a view to the DOM.

app/scripts/app.js:

define(["jquery", "underscore", "backbone"], function($, _, Backbone) {
return {

start: function () {

require(['component1'], function ( c1 ) {
// Not the best example but its just a
// proof of concept after all...
var view1 = new c1.Views.View1();

$(document.body).append(view1.render().$el);

});

}
}
});


Now main.js just needs to require app.js and call the start() method. This approach does not need the component1 defined in the paths hash in the build.js.

app/scripts/main.js

define(["app"], function(app) {

app.start();

});


In either version, the final build will create a bundle.js file in the dist directory. The build.sh script copies the component1.min.js file from the components directory and renames it to component1.js so its found when trying to load it later. Finally, a production index.html is copied which has the correct script tag defined to load the bundle and start the application.

build/index.html:

<script data-main="scripts/main" src="scripts/bundle.js"></script>


I took advantage of the option to build RequireJS right into the bundled file to skip that round trip to the server to first load RequireJS and then have it load the optimized bundle. Now, I have just one JS file loading when the application starts and, when necessary, another request to load component1 (which in this demo is basically immediately).

While it doesn't force you to design properly encapsulated, modular applications. This setup does go a long way to promote those design decisions. Developing parts of the application in isolation helps reinforce the intended goal of keeping each part distinct from the other. RequireJS and Bower provide a lot of flexibility to enable simplifying the development workflow and packaging libraries and applications for deployment. The example provided here is a good starting point for exploring other potential ideas to building modular Backbone applications.