Archive for the ‘Experiment’ Category You are currently browsing the archives for the Experiment category.
December 9th, 2011

Stop! Hacker Time.

At SoundCloud we like to invent new ideas. But we’re not adverse to implementing really great tried and tested ideas like the 20% time concept made famous by Google.

We’re calling it Hacker Time. We’re still very much in start-up mode so we’re keen to nurture the spirit of hacking. We’ve been testing out Hacker Time for a few weeks now and we’re excited about its potential, from industry-changing initiatives like “Are we playing yet” to unusual passion projects like the “Owl Octave”.

Why we’re doing it

When I arrived at SoundCloud I gathered all the engineers and developers in a room and asked them what they wanted more of. This was one of the top requests:

Like every other fast-paced tech company out there, the everyday demands of development leave little time for pet projects or invention outside of the roadmap. But we shouldn’t let great ideas slip away: it’s wasteful of talent and it’s frustrating for developers themselves. Hacker Time is an attempt to keep both the ideas and the employees focused on making the product even better.

I don’t think there’s a single developer at SoundCloud who isn’t a sound creator. Or at least they quickly become one once they’ve joined. We’re a company of electronic music producers, acoustic musicians, field recordists and social sound fanatics. Of course we’re developing the product for 9 million people, but we’re also developing something we want to use ourselves.

Ironically we’re not trying to invent anything new by initiating Hacker Time. Aside from Google, we’ve been watching the Atlassian experiment and are taking a similar approach; we’ll start with a simple set of rules and adapt to what works best for us. Like Atlassian, we’ll be blogging about our experiences too.

Which projects are engineers allowed to work on?

We’ve compiled a list which is not meant to be restrictive but rather should be a guideline:

  • Pet features/improvements that never made it onto the roadmap
  • Apps based on the Soundcloud API
  • Hack Days
  • Anything for SoundCloudLabs.com
  • “this always annoyed me” bug-fixes or architectural improvements
  • Integration of some technology-du-jour with Soundcloud
  • Contributing to OSS used at Soundcloud
  • Other cool projects using SoundCloud
  • Conferences
  • Writing Blog Posts, Technical Articles

How will time be allocated?

The big decision is how to allocate time to these projects. We don’t want to cannibalize valuable product development time but we do want to give people as much freedom as possible,

There are lots of approaches out there – from reserving one day a week, accumulating time to set aside whole weeks or handling that time like vacation. We’ve decided to put the decision-making in the hands of each team, allowing them to allocate Hacker Time according to each team’s workstyle. It’s an experiment, and something we’ll be reviewing in the early stages.

Demo it!

We’re pretty disciplined about demoing work to the whole company. Hacker Time projects will be included in the demos (which happen every 2 weeks at SoundCloud), giving developers a chance to showcase their projects and sparking interest in their hacks from everyone in the organization.

Watch this blog for further reports!

Many thanks to

Atlassian for blogging about their experiences

Simon Stewart from Google
Jim Webber from Neo4J
Jan Lehnhardt from Couchbase
Stefan Roock from it-agile

for sharing their experiences with us.

Alexander Grosse
November 21st, 2011

Front-end JavaScript bug tracking

Proper and effective error tracking is a common issue for front-end JavaScript code compared to back-end environments.

We felt this pain as well and experimented with different solutions over the past months on the SoundCloud Mobile site.

Analytics

The first approach we had was to track errors with Google Analytics. Their library permits to fire custom events and whenever an ajax error would occur, we would log it.

The biggest benefit of this tool is to monitor the stability of the site and its evolution in longer periods as you can easily go back a few weeks or months to see which events were triggered. Also, it is easy to implement – almost a one-liner!

The drawback, at least for Google Analytics, is that this tool is not meant to track bugs. There is no way to add custom data to these events to get more insight about why and how an error happened, it also doesn’t work in real-time, and you obviously want that when you debug.

So we kept Analytics in place for a long-term view, but took a look at other options for real-time and in-depth tracking.

Airbrake

In our pursuit of getting more insight, we decided to take a look at Airbrake because we were already using it to track back-end errors on our main site.

Our mobile site runs on Node.js, the first thing we did was to integrate an existing plugin for it to handle error tracking on the back-end as well.

Looking a little further we found a front-end notifier, which would catch errors that would fire on window.onerror, but there was no way to report any custom errors.

We decided to take a day to hack this on our own since their API is public and easy to implement.

The benefits of Airbrake were instant. We could see what triggered which error, how, why, in which context, which browser, etc… in real-time!


It also counts errors, which can help you prioritize and include fixes in your roadmap.

However, the lack of filtering, grouping and custom sorting made it difficult to work with. There was also no sense of time or progress, as everything just gets dumped into a single list ordered by time. We needed something a little better than that.

BugSense

That’s when our Android team showed us their BugSense implementation.
BugSense seemed to address all of these issues we had with Airbrake: grouping is more effective, searching and filtering is possible, charts of errors are drawn as well.

There is one more benefit over Airbrake… JSON. No need to convert objects to XML strings anymore!

If you are interested in our BugSense notifier you can find the source on github.

Conclusion

There is still a lot of work needed to make front-end JS debugging as easy as it is for regular back-end environments.
For example, stack traces today aren’t that useful, because of anonymous objects and minified code, but hopefully browser vendors will tackle these issues soon. Maybe Source Maps could be the first milestone in this quest.

At SoundCloud, we will continue to use a combination of these tools because of the different strengths outlined above, but there are also other tools we didn’t try out yet like getexceptional or errorception. If you have tried these, or if you have any suggestion on this subject we’d like to get your feedback in the comments below.

Happy debugging!

Yves
April 28th, 2011

Web Scale Statistics – Failing with MongoDB

As SoundCloud rapidly grows our initial systems need an overhaul. Our scaling strategy has been very realistic, design for 10x our current usage. Our initial statistics system found under http://soundcloud.com/you/stats was made when we were 100k users, living long past its expiration date.

Background

About a year ago we started off with the goal of redesigning the statistics pages to support 500 playbacks a second. We knew that this would be a write-heavy workload and that to sustain bursts of writes, we’d need decent partitioning. Coming from a successful experience moving the Dashboard feature to Cassandra 0.6 we started out prototyping a design that would be easily partitioned.

The write side of this story went very well, Cassandra could keep up with everything we threw at it, however to naively pull out all the aggregates we were collecting took hundreds of queries to the cluster. Cassandra didn’t have atomic counters at the time, so we had a lot of individual counts that needed to be summed on the client. (This is changing with the much anticipated upcoming 0.8 release!)

In a one-night experiment, we re-implemented the Cassandra based prototype to be backed by MongoDB. Not only could this quick prototype consume events as fast as Cassandra, there were some server side features in MongoDB that we could use to simplify a few of the queries that we had for the stats like atomic inplace insert/updates (upserts) to use fewer documents and secondary indexes to build the time series. Plus it was web scale.

Read the rest of this entry »

Sean Treadway