node.js knockout — August 28th & 29th
The node.js Knockout is a competition inspired by the popular Rails Rumble. The rules are simple: Teams of up to 4 people have 48 hours to build a web app that is awesome enough to woo the judges and results in as many votes from the audience as possible. To level the playing field, each team was required to deploy their application to a dedicated hosting environment provided by either heroku.com or joyent.com. The competition started on Saturday at 2:00AM and ended on Monday 2:00 AM. Yes, that’s a weekend right there.
We had done a minimal amount of planning beforehand and agreed that we’d try to visualize traffic on soundcloud.com by hooking into the varnish HTTP accelerator that’s part of our infrastructure. We basically set out to build a pimp HTML5 + WebSockets based version of varnishtop (with a sprinkle of varnishlog, because that’s just so exciting!).
Being confined to the Joyent environment, the first challenge to overcome was not having access to a varnish instance that serves any real traffic. To solve that, we hooked into the Twitter Streaming API and filtered all links to the popular URL shortening service bit.ly. For each bit.ly link a request is made against the local varnish instance to generate traffic. Of course, all of these requests result in (eventually cached) 404s, but oh well. The rest of Saturday saw us create a layout, start working on the C++ extension required to access a running varnish instance’s shared memory to read logs & statistics and generate events in node, culminating in having the basic functionality and deployment configuration done at around 3AM on Sunday.
Lessons learned (in no particular order):
- node.js is fun, building realtime stuff with it is incredibly easy
- deploying early and continuously == win
- Solaris is a pain
- Twitter is full of spam and porn