Archive for 2009

Investigating JavaScript Array Iteration Performance

Note: This post is from 2009, and browsers/libraries have changed a lot since then. Please don’t use any of this information to make decisions about how to write code.

The other day I was working on some JavaScript code that needed to iterate over huge arrays. I was using jQuery’s $.each function just because it was simple, but I had heard from a bunch of articles on the web that $.each was much slower than a normal for loop. That certainly made sense, and switching to a normal for loop sped up my code quite a bit in the sections that dealt with large arrays.

I’d also recently seen an article on Ajaxian about a new library, Underscore.js that claimed to include, among other nice Ruby-style functional building blocks, an each function that was powered by the JavaScript 1.5 Array.forEach when it was available (and degrading for IE). I wondered how much faster that was than jQuery’s $.each, and that got me to thinking about all the different ways to iterate over an array in JavaScript, so I decided to test them out and compare them in different browsers.

This gets pretty long so the rest is after the jump.

I went with six different approaches for my test. Each function would take a list of 100,000 integers and add them together. You can try the test yourself. The first was a normal for loop, with the loop invariant hoisted:

var total = 0;
var length = myArray.length;
for (var i = 0; i < length; i++) {
  total += myArray[i];
}
return total;

And then again without the invariant hoisted, just to see if it makes a difference:

var total = 0;
for (var i = 0; i < myArray.length; i++) {
  total += myArray[i];
}
return total;

Next I tried a for in loop, which is really not a good idea for iterating over an array at all - it’s really for iterating over the properties of an object - but is included because it’s interesting and some people try to use it for iteration:

var total = 0;
for (i in myArray) {
  total += myArray[i];
}
return total;

Next I tried out the Array.forEach included in JavaScript 1.5, on its own. Note that IE doesn’t support this (surprised?):

var total = 0;
myArray.forEach(function(n, i){
  total += n;
});
return total;

After that I tested Underscore.js 0.5.1’s _.each:

var total = 0;
_.each(myArray, function(i, n) {
  total += n;
});
return total;

And then jQuery 1.3.2’s $.each, which differs from Underscore’s in that it doesn’t use the native forEach where available, and this is set to each element of the array as it is iterated:

var total = 0;
$.each(myArray, function(i, n){
  total += n;
});
return total;

I tested a bunch of browsers I had lying around - Firefox 3.5, Chrome 3, Safari 4, IE 8, Opera 10.10, and Firefox 3.7a1pre (the bleeding edge build, because I wanted to know if Firefox is getting faster). I tried testing on IE6 and IE7 in VMs but they freaked out and crashed. The tests iterated over a list of 100,000 integers, and I ran each three times and averaged the results. Note that this is not a particularly rigorous test - other stuff was running on my computer, some of the browsers were run in VMs, etc. I mostly wanted to be able to compare different approaches within a single browser, not compare browsers, though some differences were readily apparent after running the tests.

Time to iterate over an array of 100,000 integers
for loop for loop (unhoisted) for in Array.forEach Underscore.js each jQuery.each
Firefox 3.5 2ms 2ms 78ms 72ms 69ms 225ms
Firefox 3.7a1pre 2ms 3ms 73ms 29ms 34ms 108ms
Chrome 3 2ms 2ms 35ms 6ms 5ms 14ms
Safari 4 1ms 2ms 162ms 16ms 15ms 10ms
IE 8 17ms 41ms 265ms n/a 127ms 133ms
Opera 10.10 15ms 19ms 152ms 53ms 57ms 74ms

I’ve highlighted some particularly interesting good and bad results from the table (in green and red, respectively). Let’s see what we can figure out from these tests. I’ll get the obvious comparisons between browsers out of the way - IE is really slow, and Chrome is really fast. Other than that, they each seem to have some mix of strengths and weaknesses.

First, the for loop is really fast. It’s hard to beat, and it’s clear that if you’ve got to loop over a ton of elements and speed is important to you you should be using a for loop. It’s sad that it’s so much slower in IE and Opera, but it’s still faster than the alternative. Opera’s result is somewhat surprising - while it’s not particularly fast anywhere, it’s not nearly as slow as IE on the other looping methods, but it’s still pretty slow on normal for loops. Notice that IE8 is the only browser where manually hoisting the loop invariant in the for loop matters - by almost 3x. I’m guessing every other browser automatically caches the myArray.length result, leading to roughly the same performance either way, but IE doesn’t.

Next, it turns out that for in loops are not just incorrect, they’re slow - even in otherwise blazingly-fast Chrome. In every browser there’s a better choice, and it doesn’t even get you much in terms of convenience (since it iterates over indices of the array, not values). Safari is particularly bad with them - they’re 10x slower than its next slowest benchmark. Just don’t use for in!

The results for the native Array.forEach surprised me. I expected some overhead because of the closure and function invocation on the passed-in iterator function, but I didn’t expect it to be so much slower. Chrome and Safari seem to be pretty well-optimized, though it is still several times slower than the normal for loop, but Firefox and Opera really chug along - especially Firefox. Firefox 3.7a1pre seems to have optimized forEach a bit (probably more than is being shown, since 3.7a1pre was running in a VM while 3.5 was running on my normal OS).

Underscore.js each‘s performance is pretty understandable, since it boils down to native forEach in most browsers, it performs pretty much the same. However, in IE it has to fall back to a normal for loop and invoke the iterator function itself for each element, and it slows way down. What’s most surprising about that is that having Underscore invoke a function for each element within a for loop is still 10x slower in IE than just using a for loop! There must be a lot of overhead in invoking a function with call in IE - or maybe just in invoking functions in general.

Lastly we have jQuery’s each, which (excluding for in loops) is the slowest method of iterating over an array in most of the browsers tested. The exception is Safari, which consistently does better with jQuery’s each than it does with Underscore’s or the native forEach. I’m really not sure why, though the difference is not huge. IE’s performance here is pretty much expected, since Underscore degrades to almost the same code as jQuery’s when a native forEach isn’t available. Firefox 3.5 is the real shocker - it’s drastically slower with jQuery’s each. It’s even slower than IE8, and by a wide margin! Firefox 3.7a1pre makes things better, but it’s still pretty embarassing. I have some theories on why it’s so slow in Firefox and what could be done about it, but those will have to wait for another post.

It’d be interesting to try out some of the other major JavaScript libraries’ iteration functions and compare them with jQuery and Underscore’s speeds - I’m admittedly a jQuery snob, but it’d be interesting to see if other libraries are faster or not.

It’s worth noting that, as usual, this sort of performance information only matters if you’re actually seeing performance problems - don’t rewrite your app to use for loops or pull in Underscore.js if you’re only looping over tens, or hundreds, or even thousands of items. The convenience of jQuery’s functional-style each means it’s going to stay my go-to for most applications. But if you’re seeing performance problem iterating over large arrays, hopefully this will help you speed things up.

Swoopo Profits Greasemonkey Script - Entertainment Shopping

In the last few weeks I’ve become increasingly obsessed with the evil genius that is Swoopo.com. Swoopo is a penny-auction site - users buy bids for $0.60, and each bid placed on an item increases the price by $0.12. The cost of bids and the amount they increase the price of the item vary depending on the type of auction and the country you’re in. Swoopo does a lot to make it harder to win, though. For example, if the bid is placed within 10 seconds of the end of an auction, the closing time of the auction is extended by 10 seconds or so, so people can have last-second-sniping wars for as long as they want. They also offer a “BidButler” service that will automatically bid for you, and of course if two users in an auction are using it, the BidButlers just fight until they’ve used up all the money they were given. Swoopo’s operation is like cold fusion for money - they make insane amounts of cash off their users, and they only have to drop-ship the item to one user so there’s theoretically very little operating cost (they already have the money from selling bids, and they don’t need to maintain inventory). They’re shameless enough to even have auctions for cash, gold, and even more bids! Because everyone in the auction is paying to participate, even if the winner gets some savings on the item, Swoopo makes far, far more on the sunk bids - sometimes 10x the price of the item in pure profit.

Jeff Atwood (of codinghorror.com) has written about Swoopo multiple times, and some techies have even tried to game the system, but it hasn’t worked. I was introduced to Swoopo through Jeff’s blog but I hadn’t thought about it forever, and for some reason it came up again recently. After looking at it a bit, I was just floored by how they’ve managed to set up such a perfect money-generating system. The company that runs Swoopo is called “Entertainment Shopping”, which I guess is supposed to be a suggestion that it’s like gambling (where it’s “fun” to lose money) though they really, really don’t want to be regulated as gambling. I don’t personally find gambling (or bidding on Swoopo) to be that fun, but I do find it entertaining to watch the astronomical profits tick up as more and more suckers toss money into an auction. So I built a little Greasemonkey script that’ll add the estimated profit to Swoopo above the price of an auction, updating in real time as people place bids.

Example screenshot of Swoopo Profits

It took quite a bit of work to sniff out the prices from the page (I suspect they make it hard to scrape on purpose), but I’ve checked it out and the script works pretty well on current and recent auctions on all of Swoopo’s different sites (US, Canada, UK, Germany, Austria, and Spain). It won’t work on some of their older auctions, where the rules were slightly different (and bid costs were different, too). The basic formula looks like this:

((currentPrice - bidAmount) / bidAmount) * bidCost + currentPrice - worthUpTo

I’m calculating it with all the fairness to Swoopo I can muster. I calculate the number of bids based on the current price and the amount each bid moves the price (bidAmount) times the cost of bids (bidCost). The winner still has to pay the current price, so I add that in, but I subtract what Swoopo says the item is “worth up to” since they probably have to pay around that to drop-ship it to a customer. As the example screenshot shows, this leads to examples like an iMac selling for $364.75 (plus another $392.40 in bids for the winner), but a total pure profit of $9,827.98 for Swoopo. Exciting! I’ll readily admit that my calculation is not always 100% accurate. There are a number of things I don’t take into account - I assume shipping is a wash, so it’s not included. I assume Swoopo’s paying the full retail “worth up to” price when they’re probably not. I count bids as all costing the same even though they might have been won at a “discount” via a bid auction. In cases where I can’t figure out some numbers I default them to hardcoded values, which might be wrong. I also don’t take into account “Swoop it now”, which lets bidders buy the item for its full price minus the money they’ve sunk into bids, effectively getting out of the auction entirely. This would reduce Swoopo’s profits but it isn’t recorded anywhere so I can’t factor it in. So take the number with a grain of salt - it’s entertainment.

Grab the script and start poking around swoopo.com. Hopefully you’ll have as much fun as I have with it. Update: Swoopo.com closed in 2011.

JSONView - View JSON documents in Firefox

I’m a big fan of JSON as a data exchange format. It’s simple, lightweight, easy to produce and easy to consume. However, JSON hasn’t quite caught up to XML in terms of tool support. For example, if you try to visit a URL that produces JSON (using the official “application/json” MIME type), Firefox will prompt you to download the file. If you try the same thing with an XML document, it’ll display a nice formatted result with collapsible sections. I’ve always wanted a Firefox extension that would give JSON the same treatment that comes built-in for XML, and after searching for it for a while I just gave up and wrote my own. The JSONView extension (install) will parse a JSON document and display something prettier, with syntax highlighting, collapsible arrays and objects, and nice readable formatting. In the case that your JSON isn’t really JSON (JSONView is pretty strict) it’ll display an error but still show you the text of your document. If you want to see the original text at any time, it’s still available in “View Source” too.

JSONView logo

I’ve been eager to release this for some time, but I finally pushed it to addons.mozilla.org last night. I actually started development on it about 7 months ago, but work got paused on it for about 6 months due to stuff out of my control, and then I had some other projects I was working on. The actual development only took a few days (including digging through some confusing Unicode bugs). I thought it was funny that right as I was resuming work on JSONView I noticed that a JSON explorer had actually landed for Firebug 1.4, which I’ll also be looking forward to. Initially I had intended to build that functionality as part of my extension. There’s a lot I’d like to add on, like JSONP support and a preference to send the “application/json” MIME type in Firefox’s accept headers.

This is actually my first real open source project - I’ve released some code under open source licenses before, but this is actually set up at Google Code with an issue tracker and public source control and everything. I’ve licensed it under the MIT license. I’m really hoping people get interested in improving the extension with me. I’ve pre-seeded the issue tracker with some known bugs and feature requests.

The extension itself is pretty simple. I wasn’t sure how to approach the problem of supporting a new content type for Firefox, so I followed the example of the wmlbrowser extension and implemented a custom nsIStreamConverter. What this means is that I created a new component that tells Firefox “I know how to translate documents of type application/json into HTML”. And that it does - parsing the JSON using the new native JSON support in Firefox 3 (for speed and security) and then constructing an HTML document that it passes along the chain. This seems to work pretty well, though there are some problems - some parts of Firefox forget the original type of the document and treat it as HTML, so “View Page Info” reports “text/html” instead of “application/json”, “Save as…” saves the generated HTML, Firebug sees the generated HTML, etc. Just recently I came across the nsIURLContentListener interface, which might offer a better way of implementing JSONView, but I’m honestly not sure - the Mozilla documentation is pretty sparse and it was hard enough to get as far as I did. I’m hoping some Mozilla gurus can give me some pointers now that it’s out in the open.

Right now the extension is versioned at “0.1b1” which is a wimpy way of saying “this is a first release and it could use some work”. It’s also trapped in the “sandbox” at addons.mozilla.org, where it will stay until it gets some downloads and reviews. Please check it out, write a little review, and soon people won’t have to log in to install it!

Note: While composing this post I ran across the JSONovich extension which was apparently released in mid-December and seems to do similar stuff to JSONView. No reason we can’t have two competing extensions, though.

Fallout 3 licensed soundtrack with Amazon MP3 links

I just finished Fallout 3 last night. Yeah, that’s one of the reasons I haven’t released anything new in a while. One of my favorite parts of the game was the old music they used. I loved the BioShock soundtrack too. Now that I’m done with the game and won’t be listening to Galaxy News Radio anymore, I figured I’d hunt down the individual songs on Amazon MP3 so I can listen to them while I’m playing other games (my favorite is when Halo or Chrono Trigger music plays over another game). As long as I’m doing that, I thought I’d post the links for everyone else, since I didn’t find a list with links to download the songs anywhere online. I got the list itself from Wikipedia’s Fallout 3 page. Unfortunately not all of the songs are available - hopefully they’ll show up in time.

  1. I Don’t Want To Set The World On Fire” - The Ink Spots
  2. Way Back Home” - Bob Crosby & the Bobcats
  3. Butcher Pete (Part 1)” - Roy Brown
  4. Happy Times” (From the Danny Kaye film The Inspector General) - Bob Crosby & the Bobcats
  5. Civilization (Bongo, Bongo, Bongo)” - Danny Kaye with The Andrews Sisters
  6. Into Each Life Some Rain Must Fall” - Ella Fitzgerald with The Ink Spots
  7. Anything Goes” - Cole Porter
  8. “Fox Boogie” - Gerhard Trede
  9. “I’m Tickled Pink” - Jack Shaindlin
  10. “Jazzy Interlude” - Billy Munn
  11. “Jolly Days” - Gerhard Trede
  12. “Let’s Go Sunning” - Jack Shaindlin
  13. A Wonderful Guy” - Tex Beneke
  14. “Rhythm for You” - Eddy Christiani & Frans Poptie
  15. “Swing Doors” - Allan Gray
  16. Maybe” (Intro song from the original Fallout) - The Ink Spots
  17. Mighty Mighty Man” - Roy Brown
  18. Crazy He Calls Me” - Billie Holiday
  19. Easy Living” - Billie Holiday
  20. “Boogie Man” - Sid Phillips

Update: Amazon added the right version of “Butcher Pete” and I’ve linked it above.