Sunday, January 19, 2014

RequireJS Optimization and Dynamic Loading of Shimmed Libraries

I've written a few posts about building module web application using RequireJS as a centerpiece to manage dependencies and help reduce load times. One challenge you encounter when using this design strategy is that not every library you need to use in your application is RequireJS aware. Fortunately, this is accounted for in the RequireJS configuration options in the form of a "shim". However, as I learned when working out all the details of my release life-cycle, there are some caveats you need to address when working with the RequireJS optimizer and dynamically loading shimmed libraries. Before addressing those issues, let me create some context by explaining how I setup my application for development and ready it for release. Below is the simplified structure of my project:


+ web
 |
 + app
 |  - index.html
 |  - config.js
 |  + script
 |  |  - main.js
 |  |  .
 |  |  .
 |  |  .
 + bower_components
 + vendor_components
 .
 .
 + dist (build directory)


Because I am using Bower to install dependencies, I then setup my config file with appropriate paths to the correct files in those directories:


//- web/app/config.js

require.config({

   baseUrl: 'scripts',

   paths: {
      'jquery': '../../bower_components/jquery/jquery',
      'backbone': '../../bower_components/backbone-amd/backbone',
      'underscore': '../../bower_components/lodash/dist/lodash.underscore',
      'jquery-ui': '../../bower_components/jquery-ui/ui/jquery-ui',
      'moment': '../../bower_components/moment/moment',
      'vendor': '../../vendor_components'
   },

   shim: {

     'vendor/bootstrap': {
         deps: ['jquery', 'jquery-ui'],
         exports: '$'
     },

     'jquery-ui': {
         deps: ['jquery'],
         exports: '$'
     }

     // ... and many more

   },

});

require(['main'], function() {});



The vendor path is a catch all for anything that is not Bower aware. As you can see in the shim above, Bootstrap is referenced as vendor/bootstrap. Once those paths are place, my main entry point file loads what's needed to render the initial screen:


//- web/app/script/main.js

require( [
   'jquery',
   'backbone',
   'underscore',
   'vendor/bootstrap',
],
function( $, Backbone, _ ) {
  ...
});


This works great when developing and testing the application, I could root my web server at the web directory and browse to http://localhost/app/index.html to load the site and use all the installed components without having move anything around. Now updating any the Bower components would be as simple as issuing bower update and I could readily see which non-bower components, in the vendor_components directory, I may need to manually check for updates.

What I didn't anticipate was the problems I'd encounter once I created an optimized build of my application. Before looking at that problem, I need to explain my build process for my application. Using Grunt, I assemble a self contained build directory at web/dist. I copy all the bower and vendor components into this directory so they're available as needed. The goal is to be able to setup the web server to be rooted at the dist directory and serve everything relative to it. However, to do that, I can't use the paths in the current config.js file. As I copy the files, I ensure the relative path implied in the module names is maintained (ie jquery is at the root level, all the vendor components are mapped to vendor/*, etc).

Next, using the RequireJS optimizer Grunt task, I create a single bundle file composed of anything referenced in main.js (including main.js). I know that nothing else in my application will be included in the bundle because anything not specifically pre-loaded in main.js will be dynamically loaded as needed. The purpose of referencing it in main.js is to only include the components that will be required to render the first screen the user sees. Anything else, will be loaded later. As part of the optimization, I also include the RequireJS library in that bundle and reconfigure the index.html page to load the bundle and use main.js as the entry point instead of config.js:

from:
  <script data-main="config" src="../bower_components/requirejs/require.js"></script>

to:
  <script data-main="scripts/main" src="scripts/bundle.js"></script>

As a result, when I open http://localhost/dist/index.html, bundle.js loads and executes the code that was in main.js but is now included in the optimized bundle.js. Now the page can load and render as quickly as possible and defer loading the other bells and whistles as needed.

Now that there's some context, I can discuss the only problem I had using this scheme. I had thought that by referencing the config.js in the optimizer's configuration that the shim information would make it into the build. However, as I quickly found out when my first shimmed library failed to dynamically load properly, this is not the case. As it turns out, the shim will only be of value if the shimmed library is included in the optimized bundle. I discovered this when searching the issue log on GitHub. As the discussion in this issue points out, the answer is actually in the RequireJS documentation:

One caveat, mentioned in the docs (near the end of that section): if these errors are occurring after a build, and because one of the shimmed libraries is excluded in the build, that will fail -- for the shimming to work, all the shimmed libraries need to be included in the built file.


Apparently, even if you read the docs more than once, that one important sentence can get lost fairly easily in all the information contained in those docs. Well, since I already have a fairly large file built by the optimizer, adding more code that is only used in a few spots in the application would have defeated the purpose of using RequireJS to dynamically load the code in the first place. While I didn't find any precedence for this solution, it does seem to work: I moved the whole shim section out of config.js and created a second config in main.js specifically for the shim. This would ensure it was included in the optimized bundle file and available for RequireJS to properly load dependencies for the shimmed libraries:


//- web/app/script/main.js

require.config({

   shim: {

     'vendor/bootstrap': {
         deps: ['jquery', 'jquery-ui'],
         exports: '$'
     },

     'jquery-ui': {
         deps: ['jquery'],
         exports: '$'
     }


   },

});

require( [
   'jquery',
   'backbone',
   'underscore',
],
function( $, Backbone, _ ) {
  ...
});


At first, I thought this would clobber the configuration set in config.js and cause problems when developing. However, it appears the two are happily merged together. By splitting it, I can continue to develop and test using one method and roll out a build using the other and everything works as expected.