Thursday, February 28, 2013

Integrating Backbone with jQuery UI Autocomplete to Provide Search Functionality

backbone-jqueryui-autocomplete
Now that I have a fairly reasonable grasp of what Backbone provides, I need to figure out how to use that to build entry forms that contain various UI elements. I certainly would like to use any widget library and properly utilize the encapsulation my collections and models provide. After all, that's one of the strengths of Backbone. One of my first tests was to figure out how to wire up a jQuery UI Autocomplete widget to source its search data from a backbone collection. To facilitate this concept, I chose to use the Rotten Tomatoes API to search for movies so there was some real content to work with. I had several goals in mind when building this example:


  1. Make the view responsible for creation and management of the widget and its events

  2. Use id/value pairs in the autocomplete. This is not the default behavior but can easily be overridden (an example is provided on the jQuery UI site)

  3. Cache searches to reduce the amount of hits to the server

  4. The models created with the search results may not be complete. Once an item is selected, the model can fetch the missing attributes



To begin, I'm going to create a view that will be bound to the collection that will provide the search functionality for looking up movies. This view will do three things:


  1. Create the autocomplete widget

  2. Attach a handler function to the source of the autocomplete which will implement the actual process of fetching data to display in the results

  3. Bind to the required autocomplete events to handle overriding the displayed label and firing an event when an item is finally selected




var MovieFinder = Backbone.View.extend({

render: function () {

// Add some markup to attach the autocomplete to
this.$el.html('<span>Find a Movie: </span><input id="search" type="text" size="30" />');

// Create the widget. Route searches to a custom handler
this.$('#search')
.autocomplete({ source: $.proxy( this.findMovies, this), minLength: 2 });

return this;
},

// Use the bound collection to perform the search.
// The custom search function adds some caching to reduce the
// amount of traffic. Using the Deferred object, wait for the
// response and then call the autocomplete's provided callback
findMovies: function ( request, response ) {

$.when( this.collection.search( request.term ) )
.then(function ( data ) { response( _.map(data, function ( d ) { return { value: d.id, label: d.title + ' ('+ d.year +')' }; }) ); });

},

// We want to show the movie title but use the movie ID to
// fetch the model from the collection and a trigger an event
// so other processing can occur.
events: {
'autocompletefocus' : 'handleFindFocus',
'autocompleteselect' : 'handleFindSelect'
},

handleFindFocus: function ( e, ui ) {

return false;

},

handleFindSelect: function ( e, ui ) {

var m = this.collection.get( ui.item.value );

this.$('#search').val( ui.item.label );

this.trigger(' finder:selected', m );

return false;

}

});


You can see that the findMovies function is essentially creating a bridge between the collection and the UI widget to transform the incoming criteria into the collection's search function and then the results back into the required format of the autocomplete widget.

That takes care of my first two goals. Now, I want to create a caching mechanism so every keystroke doesn't turn into a request to the server. To accomplish this, I'll need to wrap my fetch on the collection inside a function that will manage the caching. Any requests that are cached will be searched locally. The collection will be responsible for applying the rules for searching:


var MovieCollection = Backbone.Collection.extend({

...

search: function ( term ) {

var that = this,
save = this._cache,
crit = term.toLowerCase(),
ckey = crit.slice(0,2),
clen = crit.length,
rslt, matcher;

return $.Deferred(function (dfd) {

// Cache based on first 2 letters typed
// Stop sending requests over and over again
// Filter results locally
if ( save[ckey] ) {

// Search locally
matcher = new RegExp($.ui.autocomplete.escapeRegex(crit), 'i');
rslt = _.filter(save[ckey], function ( m ) { return matcher.test(m.title); });
// Reset collection to match
that.reset(rslt);
dfd.resolve(rslt);

} else {

// Retrieve through an API call
that.fetch({
url: _.result(that, 'url') + '.json?apikey=' + that.apikey + '&q=' + crit,
success: function ( data ) {
save[ckey] = data.toJSON();
dfd.resolve(save[ckey]);
}
});

}

}).promise();
}

});


The search function will always return a Deferred object and resolve it based on whether there is a cache hit or after the results are returned by the API call.

Finally, I'd like to do something once an item is selected. My fourth goal was to use the search API to fill my collection with models that may only have some of the attributes available. Once the item is selected, the corresponding model can be fetched using another API call to retrieve any missing attributes. Based on my initial read of the API docs, it seemed the search API did not include as much detail as it actually did. In the tests I ran, everything related to the movie was returned except the studio name. That seems a little verbose and I didn't see a way to ask for less. However, if it was actually less complete, the final fetch on selected model would grab the missing data so it could be displayed in another Backbone view. Here's the main entry point of the page - it creates the collection that will be used for searching, sets up the view that contains the autocomplete, and then binds to the "finder:selected" event to display the selected movie's details:


$(function() {

movies = new MovieCollection();

var finder = new MovieFinder({ collection: movies });
var selected;

$('.wrapper').append( finder.render().$el );

finder.on('finder:selected', function ( m ) {

if ( selected ) {
selected.remove();
}

// Fill in any missing details - pull entire model
// not just those retrieved by the search.
m.fetch();
selected = new MovieDetail({ model: m });
$('.wrapper').append( selected.render().$el );

});

});



As usual, the full demo is on my sandbox with all the source. While a fairly basic example, its seems like a reasonable approach to cases where you need to allow the user to find something from a reduced set of columns to save on bandwidth and then lazy load the remaining details on the selected item to continue processing somewhere else in the UI.

Sunday, February 17, 2013

Integrating Flot Graphs in a Backbone View

Any data-driven application is going to need to perform a certain amount of transformations to the data in order to display it appropriately. Its generally more common in analytical areas where grouping and pivoting are required. As I've worked with Flot to build graphing solutions, I've needed to transform data from Backbone collections into something Flot could work with.

A Backbone collection's data is organized like this:

data = [
{ columnA: valueA, columnB: valueB, ... },
{ columnA: valueA, columnB: valueB, ... },
...
]


Where each index in data is a row with a hash of column/values.

While a Flot dataset is required to look like this:

data = [
{ label: seriesA, data: [ [x0,y0], [x1,y1] ... ] },
{ label: seriesB, data: [ [x0,y0], [x1,y1] ... ] },
{ label: seriesC, data: [ [x0,y0], [x1,y1] ... ] }
]


Where each index in data is a series with a hash describing the series. The data in the series object is an array of arrays describing each x/y point in the series.

As a more concrete example, consider the following Backbone collection data:

data = [
{ year: 2009, studio: 'Fox', gross: 760507625 },
{ year: 2010, studio: 'Par', gross: 658672302 },
{ year: 2010, studio: 'WB', gross: 623357910 },
{ year: 2009, studio: 'WB', gross: 534858444 },
{ year: 2011, studio: 'Fox', gross: 474544677 }
...
]


If we want to plot each gross by studio by year, we might setup our Flot data to look like this:

data = [
{ label: '2009', data: [ [0, 760507625], [1, 0], [2, 534858444], ... ] },
{ label: '2010', data: [ [0, 0], [1, 658672302], [2, 623357910], ... ] },
{ label: '2011', data: [ [0, 474544677], [1, 0], [2, 0], ... ] }
]

options.xaxis.ticks = [ [0, 'Fox'], [1, 'Par'], [2, 'WB'], ... ]


Each year becomes a different series in the graph and the points are the studios (x-axis) and gross dollars (y-axis). Note that we can't just pass the studio label as the x-axis value in the data array. Since those must be numbered indexes, the axis labels need to be setup in the options hash to describe each tick point. I lined those up to make it more clear what each element in the data array actually means.

Now that we have a basic visualization in mind, its time to build a Backbone View which will convert its collection into the required inputs for Flot to draw the graph. The view setup is pretty straight forward. I split the rendering up into two pieces. The main render() function just added the stub HTML required by Flot to hold the graph canvas. This needs to be styled properly to ensure it has a defined height/width. The graph is actually drawn in the renderGraph() function. This allows it to be called when something changes that requires the graph to be redrawn:


var MovieGraph = Backbone.View.extend({

attributes: { class: 'graph' },

seriesColumn: 'year',
xaxisColumn: 'studio',
yaxisColumn: 'gross',

plotOptions: {
yaxis: {},
xaxis: {},
legend: { show: true, container: '.legend' },
grid: { hoverable: true, clickable: true, autoHighlight: true },
series: {
stack: true,
bars: { show: true, fill: 0.7, barWidth: 0.8, align: 'center' }
}
},

render: function() {

this.$el.html('<div class="legend"></div><div class="plot"></div>');

return this;
},

renderGraph: function() {
/* See below */
}

});


Additionally, I setup my default options for the graph as well as identify which columns should be utilized in the graph dataset. The real work now happens in the drawGraph() function:


renderGraph: function() {

var data = this.collection.toJSON(),
options = _.clone(this.plotOptions),
sC = this.seriesColumn,
xC = this.xaxisColumn,
yC = this.yaxisColumn,
series = {},
xaxis = {},
base = {},
xticks = 0,
i;

if ( data.length > 0 ) {

// All x-axis data must be passed as an index
// from 0 .. n. Setup hash mapping of x-axis values to
// an indexed number so they can be looked up below
// Also, stacked charts don't work well with missing
// values so make sure each series will have a data point
// for each x-axis index.
for ( i=0;i<data.length;i++ ) {

if ( typeof xaxis[data[i][xC]] == 'undefined' ) {
xaxis[data[i][xC]] = xticks;
base[xticks] = 0;
xticks++;
}
}

// Build series data. Sum y-axis values over each series and x-axis point
for ( i=0;i<data.length;i++ ) {

if ( !series[data[i][sC]] ) {
// Initialize series with hash setup above to ensure
// each x-axis point has a value (zero). Set the label for
// the legend
series[data[i][sC]] = { data: _.clone( base ), label: data[i][sC] };
}

// Depending on the data, adding may be desired to group/summarize
series[data[i][sC]].data[ xaxis[data[i][xC]] ] += data[i][yC];
}

// Everything is a hash right now object[idx] = val
// Flot wants an array of arrays [ [idx, val] ... ]
// Use _.map to convert:
options.xaxis.ticks = _.map( xaxis, function ( val, idx ) { return [val, idx]; } );

// Same with the series data
for ( i in series )
series[i].data = _.map( series[i].data, function ( o, i ) { return [ i, o ]; });

// Again, another hash that must be an array of objects
// Convert again with _.map
series = _.map( series, function ( z ) { return z; } );

// Make the y-axis pretty. Assuming Gross dollars is the y-axis
// here. Format accordingly
options.yaxis.tickFormatter = function (val, axis) {
return Globalize.format(val, 'C0');
};

// Now, the chart can be drawn ...
$.plot( this.$('.plot'), series, options );
}

}


The basic process is to pivot the dataset according to the desired column layout and then convert everything into Flot speak before passing it to the plotting function. Its generally easier to work with object hashes when pivoting the data. However, that needs to be converted to arrays to be valid Flot input.

My sandbox has a working demo with full source. By creating variables in the view to define the series and x/y axis columns to use in the collection, you can see that this function is fairly reusable and can serve as the basis to build more complex graphing scenarios. The data is a bit sparse in this example but it should provide a good reference for how to apply it to larger datasets.

Sunday, February 10, 2013

Adding Interaction to Flot Graphs: Tooltips and Labeled Series

flot-interact-item-lines
Out-of-the-box, Flot does an awful lot for you with very little extra work. Even basic interactions are pretty simple to implement by binding to certain events produced by it. And if you really want to dig in and create more advanced interactivity, Flot exposes everything you need to accomplish that task. Probably one of the more common requirements in a graph is to be able to label points - either with the actual value at a given point or more information about the point that might not be evident on the graph. In my case, I wanted to both show the series label and point value when you hovered over a specific point in the graph and, when clicked, show labels for every point in the series. The former turned out to be relative simple, however, the latter took a little more effort to achieve.

For any graph that you want to respond to interactive events, you need to make sure you've enabled the correct options:


options.grid: { hoverable: true, clickable: true, autoHighlight: true };


The auto highlighting feature is the only interaction Flot handles for you. Anything else requires you to bind to the events Flot fires to add additional enhancements. Flot will fire a plothover event whenever the mouse moves over a datapoint but if only is hoverable = true. Additionally, it will fire a plotclick when clickable = true. Each event handler function is passed three arguments which can be used to create interactive features. So the basic setup to create a graph and bind to both events looks like this:


var plot;

plot = $.plot($('.graph'), data, options);

$('.graph').on('plothover', function ( event, pos, item ) {
...
}

$('.graph').on('plotclick', function ( event, pos, item ) {
...
}



I maintain a reference to the Flot object since I might need to call several API functions in the event handlers. Adding a label to the point you're hovering over is fairly straight forward. The information passed to the event handler contains both values about where the mouse is pointing and the normalized position of the data point. For my purposes, I always want the label to be positioned consistently relative to the data point, so I use the item object passed to the handler to find its page coordinates. The process of adding the label is as simple and creating the HTML containing information from the point, adding it to the DOM, and then positioning it over the graph near the point:



$('.graph').on('plothover', function ( event, pos, item ) {
var ofsh, ofsw;

if ( hoverTip )
hoverTip.remove();

if (item) {

hoverTip = $(toolTipHTML( item.series.data[item.dataIndex][1], item.series.label ));

$('.graph').parent().append(hoverTip);

ofsh = hoverTip.outerHeight();
ofsw = hoverTip.outerWidth();

hoverTip.offset({
left: item.pageX - ofsw / 2,
top: item.pageY - ofsh - 15
});
}
}


The item pageX/pageY values will always be the same for a point regardless of where the mouse is positioned. This allows you to create a label in a consistent spot relative to the point. For a line, its the point on the line, however, for a bar graph, the actual point will depend on the alignment of the bar to the axis label. The default will place it at the top/right of the bar. If you change the alignment to "center", then the point will be the top/center of the bar. The above code will result in the following label for a bar graph when you hover over the bar:

flot-interact-hover-datapoint

Since I'm using the stack plugin, finding the original data point value wasn't what I expected. If you look at the Flot documentation, the example shown uses the value in item.datapoint. However, the stack plugin modifies the datapoint values to create start/end point on the graph. If I used that information, the third series would have an y-axis value that was the sum of the two series below it:


// Hover over the third series point: X=3,Y=25
item.datapoint = [3, 34, 9]
item.series.data[item.dataIndex] = [3, 25]


The difference is 25 which is what I want in my label. However, I won't trust this because I may not always be using the stack plugin and the third index in the array won't always be there. Instead, I dug into the item object more and found the original data array (item.series.data) and can get my value from there.

Working with the actual point that is being hovered over or clicked is fairly straight forward. However, what if you want to add labels over the entire series when one of the series points is clicked?

flot-interact-click-series-labels

Since the item object is only providing the pageX/Y for the clicked point, you have to figure out how to find the pageX/Y of all the other points in the series. Fortunately, Flot exposes several helper functions to make this relatively easy. However, the documentation does not go very deep into how to really use them. It took some trial and error to completely understand how to obtain exactly what I wanted.

Internally, Flot has a representation of the data in what is referred to as point-space and canvas-space. The point-space is the actual data values of the series, however, normalized to match how the data will render on the canvas. The canvas-space is the actual pixel coordinates inside the canvas. The API function getAxes() provides functions for each axis called p2c() and c2p() which will translate between point-to-canvas space and canvas-to-point space, respectively. To find where a data point is on the canvas, you just call xaxis.p2c(x) and yaxis.p2c(y). The tricky part here is what is x and y. These are not necessarily the original data value. As I showed above, the stack plugin modified my data to create new values that represented the actual x/y point on the graph. In this case, 9 was the bottom of the bar on the y-axis and 34 was the top of the bar on the y-axis. To find the top of the bar in canvas-space, I need to pass 34 to yaxis.p2c().

Now, I need to find these pageX/Y values across all the points in a series. Going back to the API, I can access all the series data via getData(). Here, I can find the normalized points I need to pass to p2c() to find the positions of all the points in the series. So, to calculate the pageX/Y of each point in the series, I would follow these basic steps:


  1. Get a reference to the axis and series data via getAxis() and getData(), respectively

  2. Find the pageX/Y of the graph canvas since p2c() will return values relative to that point

  3. Iterate over each X-axis point and find it in the series datapoints

  4. Use the p2c() function from each axis to find the X/Y coordinates of the point in canvas-space

  5. Add the previously calculated canvas position to the values returned from p2c() to find the absolute page coordinates of the point



Here's what the code would look like to implement that logic:


$('.graph').on('plotclick', function ( event, pos, item ) {

var x = 0, ttip, fmtd, dp, pz, tmp, xtickl,
ofs = { pageX: 0, pageY: 0, height: 0, width: 0, plotX: 0, plotY: 0 },
axis = plot.getAxes(),
series = plot.getData(),
xcnt = series[0].data.length;

plot.unhighlight();
clearTooltips();

if ( item ) {

// Remember them so they can be removed
clickTips = [];

// Find the canvas offset
tmp = $('.graph').offset();
ofs.plotX = tmp.left;
ofs.plotY = tmp.top;

tmp = plot.getPlotOffset();
ofs.plotX += tmp.left;
ofs.plotY += tmp.top;

// For each point over the x-axis
for ( ;x<xcnt;x++) {

// Let's highlight all of them in the series
plot.highlight(item.seriesIndex, x);

// datapoints is flat, we need to know how the step size
// of each point so we can find our X/Y values
// this is the normalized value relative to the
// graph
pz = series[item.seriesIndex].datapoints.pointsize;
dp = [ series[item.seriesIndex].datapoints.points[x*pz], series[item.seriesIndex].datapoints.points[x*pz+1] ];

// This is the real value to show in the label
fmtd = series[item.seriesIndex].data[x][1];

// Convert to canvas-space and add canvas offset
ofs.pageX = parseInt(axis.xaxis.p2c(dp[0]) + ofs.plotX);
ofs.pageY = parseInt(axis.yaxis.p2c(dp[1]) + ofs.plotY);

// Create the HTML
ttip = $(toolTipHTML( fmtd, (item.dataIndex == x ? item.series.label : null) ));

// Add it to the DOM. Don't pollute the graph element.
// Add it to the parent instead.
$('.graph').parent().append(ttip);

// Figure out how much to offset the label based on its size
ofs.width = ttip.outerWidth();
ofs.height = ttip.outerHeight();

// Position it accordingly
ttip.offset({ left: ofs.pageX - ofs.width / 2, top: ofs.pageY - ofs.height - 15 });

clickTips[x] = ttip;

}
}

});



There's a working demo and full source available on my sandbox. This example can provide a starting point to a lot of different possibilities. Once you understand what Flot has available to find the position of points, adding other interactive features becomes relatively easy.

Monday, February 4, 2013

Animating a Flot Graph with jQuery

The Flot homepage does have an example of animating data. Its actually moving points on and off the chart. It also uses setTimeout() to pace the animation. I was interested in easing in my bar graph so I thought I'd try something with jQuery.animate() and see what I could come up with.

I accomplished it by doing several thing:

  1. Creating the graph with the actual data I want to plot

  2. Using the step callback available in jQuery.animate()

  3. Using the Flot API functions setData and draw to update the graph




    • Here's a small snippet of code showing the basics:



      plot = $.plot($('.graph'), [init], options);

      $('.graph').animate( {tabIndex: 0}, {
      duration: 1000,
      step: function ( now, fx ) {

      var r = $.map( init.data, function ( o ) {
      return [[ o[0], o[1] * fx.pos ]];
      });

      plot.setData( [{ data: r }] );
      plot.draw();

      }
      });



      It was easiest to just create the graph from the final dataset so Flot would create the correct axises and scaling. Grab a reference to the object so it can used and then on each step in the animation, manipulate the y-axis data point to vary from zero to the final value based on fx.pos. I have a working example on my sandbox. I haven't tried it with too many other types of graphs/options yet but it seems like a pretty flexible solution to add some extra pizzazz to your graphs.

Sunday, February 3, 2013

Graphing with Flot: Controlling Series Color

I had the recent pleasure of working the Javascript-based Flot graphing library. It generates graphs by drawing on an HTML Canvas (it also has capability support for older IE). Flot is very easy to use yet provides a lot of customizable options. So far, I haven't found something that I wanted to change that I could not. However, without setting any (or very few) options, you get great looking graphs. Even with the automatic formatting, layout, and color selection, you might find times where some finer control is required.

I discovered that when working with multiple series, I wanted to have more control of the color selections. Occasionally, Flot picked a color too light or too close to another series. I also was adding/removing series from my graph and wanted to try to maintain the color assigned to a given series. Fortunately, there are both a series.color and global colors array options available that enables you to override the auto color picking algorithm in Flot. The color you set depends on what your trying to accomplish. Use the global option to create one master palette regardless of series. Assign a color to the series directly to maintain the color or derive it from the series data. In my test, I just used the global colors array to create a set of colors that Flot would use when drawing the series on the graph.

To get started, I copied a data set used in one of the Flot examples which created several series of data:



var data = [
{
id: "usa",
label: "USA",
data: [[1988, 483994], [1989, 479060], [1990, 457648], ...
},
{
id: "russia",
label: "Russia",
data: [[1988, 218000], [1989, 203000], [1990, 171000], ...
},
{
id: "uk",
label: "UK",
data: [[1988, 62982], [1989, 62027], [1990, 60696], ...
},
{
id: "germany",
label: "Germany",
data: [[1988, 55627], [1989, 55475], [1990, 58464], ...
},
{
id: "denmark",
label: "Denmark",
data: [[1988, 62982], [1989, 62027], [1990, 60696], ...
}
];


I also created a variable and set the fixed settings I was planning to use:


var options = {
series: {
stack: true,
},
yaxis: { min: 0, max: 1000000 },
xaxis: { tickDecimals: 0 }
};



In this case, I also used the Flot plugin to enable creating a stacked chart.

Now that the basics are out of the way, I can focus in on customizing my colors. The default setting is an empty array (don't make it null - it will generate an error):

options.colors = [];
$.plot($(".graph"), data, options);


flot-options-colors-default


I can just add colors that will be assigned in the same order as the series:

options.colors = ['red','orange','green','blue','purple'];
$.plot($(".graph"), data, options);


flot-options-colors-html-names


Or, I can get a little more fancy and use jQuery.Color to generate the colors. The benefit of using that library is you can work in the HSL color space and then convert to something Flot understands. I built two different generators - one creates a monochromatic set of colors by altering the lightness of a base color and the other rotates through a range of hue values while maintaining a constant saturation/lightness for all the colors:


// Shades of blue
options.colors = $.map( data, function ( o, i ) {
return jQuery.Color('blue').lightness(0.7-i/(len*1.2)).toHexString();
});

$.plot($(".graph"), data, options);


flot-options-colors-lightness


// Rotate from red to purple
options.colors = $.map( data, function ( o, i ) {
return jQuery.Color({ hue: (i*360/len), saturation: 0.95, lightness: 0.35, alpha: 1 }).toHexString();
});

$.plot($(".graph"), data, options);


flot-options-colors-hue


If you wanted to use the same approach to set the color directly on the series, you could use $.each over the data series and set o.color equal to the value returned by jQuery.Color(). The approach is pretty flexible and allows for a high degree of control over the colors generated. I put together a small interactive demo on my sandbox that enables switching between color schemes, changing the chart type, and showing/hiding different series. This is just a couple of ideas I was playing with when attempting to assign meaningful colors to the data series I needed to graph. It was a good starting point for exploring the many customizations available in the library.