In December a few of us working on processing.js had an idea–what if we could visualize sound data coming out of an <audio> or <video> element?  My colleagues were good at thinking in terms of “how can we make what we have now work?” but I had another idea.  “Let’s try and teach Firefox how to do this.”  In December I set myself a challenge:

This post marks the beginning of what I expect to be a somewhat regular series of posts in which I will document my thinking, learning, and progress related to some Firefox development.  I say “Experiments” above, because I am going to try a number of things.  First, I’m going to push into a part of the Firefox source code where I haven’t worked before, namely, the DOM implementation (note: it scares me, to be honest).  Second, I’m going to do so in an open and pedagogic way, attempting to cast aside my own ego and hesitation at looking foolish–I don’t actually know how to do what I’m going to try, and will learn and fail as I go–on the way to producing an authentic model of open development for my students.  Third, I’m going to work with a few others who are also interested in extending themselves and extending the web.  The only thing I can assure you of at this early date is that these posts will be an honest account of the attempt.

Yesterday, the end result of that work landed in mozilla-central, on its way to inclusion in Firefox 4.  I’m immensely proud of the work we’ve done, and thrilled that my peers in the Mozilla community have also accepted it.  I’m also very tired

There’s lots of things that I could talk about in terms of the code and API, and probably we’ll do some of that soon (you can already use the API in a Firefox nightly, read about the API, and try live demos).  But what I wanted to end this series of posts by saying something about how Firefox, the Mozilla community, and the open web, make what we did possible.

The other day my family had some friends over, and we got talking about what I do.  Of course they had heard of Firefox, and used it themselves.  “But can you explain the difference between this and the other one I use, Internet Explorer.”  One of the big differences, I explained, is participation, the community of involvement, and the accountability that comes with this.

When we started these experiments, we did so without needing permission.  I didn’t have to sign an NDA, go talk to and convince the right people, or get approvals.  I just grabbed the source code and started messing around.  And I did make a mess, at first.  I learned as I went, and we iterated on the API a lot (I have over 80 versions of it here, not to mention the various implementations of those).  We weren’t judged for doing it wrong, or for the pace or directions we took.  Instead, we heard of lot of “this is very cool!” and “have you thought about this?”.  We were able to take one of the world’s premier applications (Firefox) and rework it.  It’s hard to overemphasize how significant this is.  We couldn’t do what we did in very many other contexts.

I said above that participation is paired with accountability, and this is also very important.  In the early months we built something that worked, but not what we have today.  As much as Mozilla made it possible for me to experiment, they also made sure that what got accepted was of the highest quality.  I haven’t blogged about audio much over the past three months, mostly because we’ve been too busy getting the patch fixed up based on reviews.  Before it could land we had to think about testing, security, JS performance, DOM manipulations, memory allocation, etc.  To get this landed we needed lots of advice from various people, who have been generous with their time and knowledge

What’s different about Mozilla?  I think of something Joe Hewitt wrote on twitter back in the spring that struck me:

Bottom line: we can currently only move as fast as employees of browser makers can go, and our imagination is limited by theirs. @joehewitt

Not one of the people who did this work is an employee of the Mozilla corporation.  When we decided to get serious about trying to include this in Firefox 4, one of the people working with us filed a bug, and the response was, “I don’t think we have the cycles to get this done in time.”  “That’s OK, we’ll do it then,” was the reply.  And we did.

The web is too big and too important to only go as fast and as far as a small group of employees can take it.  Mozilla gets this, and values community involvement like ours.  What we did is not unique–there are other great features and bug fixes coming in Firefox 4 that were done by community members vs. employees.  In fact the distinction between the two is often hard to see when you’re working on this stuff–we all work together.

Having said that, let me publicly congratulate my amazing audio peers, without whom this work wouldn’t have happened: Al MacDonald, Corban Brook, Yury Delendik, Charles Cliffe, Ricard Marxer, and our cheerleader and supporter, Chris Blizzard.  Also thanks to the dozen others who wrote demos with our stuff in the early days.  Demos are how you win (Chris Blizzard taught me that).

I’ll end with a video someone shot of my keynote talk at the recent Mozilla Summit.  I was doing a quick demonstration of what is possible with this API.  I look forward to seeing what the rest of the web will do with it.

The grant brings $2.3 Million in funding over 5 years, and is meant to be combined with funding from industry partners.  It will allow us to fund research and co-op positions, like the 9 students who are working this summer on various open source projects.  It means that we can take more of the work we do in the open source courses (OSD600, DPS900, OSD700, DPS911, SBR600, etc.) and have it grow into larger projects, working directly with local industry partners and the open source community.  We’ve known how to do this for a while now but didn’t have the resources to make it happen.  Now we do.

One of the things I thought was key in the announcement of this grant is that the government, Seneca, and our partners all understand that working on open source is the best way to gain industry experience as a student.  I’ve written about this many times before, but I’ll say it again: if you want to work in software, you need experience.  And getting experience means getting a job, but a job needs experience…unless you get involved in global, collaborative, large scale development as a student before you have a job.  The way to do this is to contribute to open source projects.

I’ve had students come to me with all kinds of crazy ideas about what open source means.  Many have this idea that you can’t have a job or make money with open source.  We wouldn’t have this grant if that were true!

“Our government supports innovation because it creates jobs, improves the quality of life of Canadians and strengthens the economy,” said Minister Goodyear. “We are supporting this project at Seneca College to strengthen the competitiveness of small and medium-sized businesses, and enable young Canadians to prepare for the jobs of tomorrow.”

…

“This funding is an important acknowledgment of the valuable, innovative research taking place at Seneca,” says Seneca College President David Agnew. “Building on a reputation of excellence in the open source community, this will help create graduates who have learned not only through hands-on experience, but through applied research that is changing the way we do business.”

The jobs of tomorrow are here today, and they require that you know how to:

• work with large, legacy code bases measured in millions of lines
• know how to collaborate with co-workers, both in person and electronically
• understand how to maintain existing code vs. starting from scratch
• know how to leverage existing technologies and tools
• know how to manage software release cycles and project planning
• know how to solve real world problems vs. made-up assignments
• know how to leverage the open web, and its ways of working

I think current examples are good, so let me give you one.  The other day I was catching up on industry news, and noticed this article on cnet news:

Mainstream microprocessors have been 64-bit for years. Operating systems have followed suit. Now it’s time for a program used by hundreds of millions of people to make the leap: Firefox.

Programmer Armen Zambrano Gasparnian announced the first 64-bit Firefox builds for Windows on Friday, offering an FTP site for those who want to download it. But the software isn’t for mainstream users yet.

That guy making the announcement for Mozilla about the first 64-bit enabled builds of Firefox, know who he is?  He’s a Seneca grad who got involved with open source and now has an amazing job as a Release Engineer with Mozilla Corporation.  How did he get hired into such a position out of school?  He worked on this stuff while he was still a student.  He took our open source courses and got involved with the right projects and the right people.  He got the experience he needed while he was at Seneca in order to jump right into industry, where he’s now doing cutting edge work.

I could tell you many similar stories, but the story I’m most interested in at the moment is the one I’m going to be telling about you in 2-3 years.  My question right now is this: how do we get you, current Seneca student, working on the cutting edge of industry via open source?  We have ways to get your career started early if you’re willing to get involved with what we do.  Your first chance to do this comes in September.  Will you be signed up for the open source classes?  If so, we’ve got some amazing opportunities to discuss with you.

Since my last post, quite a few new people have joined our group, a lot has changed in our implementation, and we’ve achieved a few things worth writing about.  I also can’t keep these demos under wraps any longer, so it’s time for another post.

One of the first pieces of advice I got in the bug, when I started writing this patch to expose audio data in Firefox, was to use Vlad’s new typed arrays (aka WebGL Arrays).  My first implementation used an array-like object to expose the audio data, and JS arrays for writing samples.  Both worked well, but neither was as fast as we’d like, and it meant various hacks to work around performance issues.  Vlad was kind enough to give me a crash course on how to implement them via quickstubs, and over the past few weeks, Yury Delendik and I have worked long hours to rewrite our entire implementation to use them.

Along with the suggestion to use typed arrays also came a less welcome suggestion: remove the FFT calculation from C++ and do it all in JavaScript.  When I suggested this in our #audio irc channel, a lot of people were upset, saying that this was a bad idea, that we’d never be fast enough, etc.  However, I pulled it out anyway in order to force them to try.  Corban responded by rewriting his dsp.js library to use Float32Array, which can now do 1000 FFTs on a 2 channel * 2048 sample buffer in 490ms, or 0.49ms per fft (js arrays take 2.577ms per fft, so a lot faster!).  And one of the biggest critics of my decision to pull the native FFT, Charles Cliffe, went off to prove me wrong, but ended up with two stunning WebGL based audio visualizations (demos here and here, videos here and here).

What I like most about these (other than the fact that he’s written the music, js libs, and demo) is that these combine a whole bunch of JavaScript libraries: dsp.js, cubicvr.js and beatdetection.js, and processing.js.  Some people will tell you that doing anything complex in a browser is going to be slow; but Charles is masterfully proving that you can do many, many things at once and the browser can keep pace.

Corban and Ricard Marxer have been busy exploring how far we can push audio write, and managed to also produce some amazing demos.  The first is by Ricard, and is a graphic equalizer (video is here):

The second is by Corban, and shows a JavaScript based audio sampler.  His code can loop forward or backward, change playback speed, etc. (video is here):

Chris McCormick has been working on porting Pure Data to JavaScript, and already has some basic components built.  Here’s one that combines processing.js and webpd (video is here):

I think that my favourite demo by far this time around is one that I’ve been waiting to see since we first began these experiments.  I’ve written in the past that our work could be used to solve many web accessibility problems.  A few weeks ago I mentioned on irc that someone should take a shot at building a text to speech engine in JavaScript, now that we have typed arrays.  Yury quietly went off and built one based on the flite engine.  When you run this, remember that you’re watching a browser speak with no plugins of any kind. This is all done in JavaScript (demo is here, video is here):

In order to do this he had to overcome some interesting problems, for example, how to load large binary voice databases into the page.  The straightforward approach of using a JS array was brittle, with JS sometimes running out of stack space trying to initialize the array.  After trying various obvious ways, Yury decided to use the web to his advantage, and pushed the binary data into a PNG, then loaded it into a canvas, where getImageData allows him to access the bytes very quickly, using another typed array.  The browser takes care of downloading and re-inflating the data automatically.  Here’s what the database looks like:

What began as a series of experiments by a small group of strangers, has now turned into something much larger.  Our community continues to grow, and the scope and scale of the projects being done on our API is increasing.  At the same time, through the work of Doug Schepers and Chris Blizzard, we’ve managed to get the attention of the W3C, which have now started an Audio Incubator Working Group to look at how to standardize this stuff.  One of my colleagues in these experiments, Al MacDonald, has been asked to chair the group, which already has members from Mozilla, Google, and the BBC.  You can get involved and follow @AudioXG for updates.

If you’d like to stay connected to this work, you can join this bug, where I’ll be posting a patch for review in the next week or so (current patch is here).  You can see our Audio Data API documentation, with tutorials and examples (this was recently completely rewritten, if you’ve looked at it before).  You can also grab builds there, which I’m making right now and will be done in the next few hours.

• Al MacDonald lays out a history and potential future for our work
• Article in Create Digital Music, “Real Sound Synthesis, Now in the Browser; Possible New Standard?”
• Post on Wired’s WebMonkey blog, “New HTML5 Tools Make Your Browser Sing and Dance”
• Feature of Multi-Touch Audio Data Bloop demo on weave.de

A number of comments on these and other blogs, and people on twitter, have talked about how Flash already allows some of this.  Reading it as many times as I did, I wanted to respond and suggest that what we’re doing isn’t simply parity with Flash.  I don’t think it’s exaggerating to say that exposing audio data to the open web has the potential to change sound, audio, and music.  The reason is that HTML5 and JavaScript based audio participates in “View Source,” and that means creating a whole new kind of active and passive audio collaboration.

The reason the web has grown like it has, the reason there is so much innovation, the reason so many people of varying levels of expertise can use it, or as Mike Shaver put it in 2007, the reason the web won, is View Source:

If you choose a platform that needs tools, if you give up the viral soft collaboration of View Source and copy-and-paste mashups and being able to jam jQuery in the hole that used to have Prototype in it, you lose what gave the web its distributed evolution and incrementalism. You lose what made the web great, and what made the web win.

The way View Source functions, with respect to HTML documents, is well understood.  It’s so well understood that its absence becomes something you can’t not see.  The Flash-based music visualization or audio app that runs in the browser today looks and sounds great, but that’s all it does–it can’t lead others to iterate and innovate.  Sure, we can deploy audio on the web, but we can’t tinker with it if we can’t get at how it’s made.  Pressing ‘play’ isn’t the same as playing.

Right now the community of people actively working on audio data in the browser is small (about 12 people that I know of), but it already points to what I’m talking about.  When we made Bloop (a processing.js version of Eno’s Bloom), we did it iteratively.  We learned how to generate simple sounds using JavaScript, then built scales, followed by more complex wave patterns, followed by oscillators, etc.  The code bounced back and forth between people on irc and twitter, before it got extracted into a reusable JavaScript library, which has already allowed Corban and Maciej to start building a multiuser synthesizer/sequencer based on our audio api, node.js, and processing.js (code is here).

This way of working is so common on the web, it’s almost not worth mentioning.  But I do mention it because this was, as far as I can tell, the first time people have collaborated on the web to build music using the technologies of web.  And just as it has for all manner of other things, the web made building music easier and faster.

One of the properties of sound is that it bounces off things, echoes, and changes.  The history of music is filled with people innovating by playing with existing sounds.  Allowing sound to exist in a more manifest and malleable way on the web, to become scriptable, viewable, copy-pastable, means that innovations in sound and music will be more frequent and much, much faster.

It’s a good time to be having this discussion, since the trajectory of music on the web is pointed away from sharing and collaboration (and has for a long time been).  What we’re plotting with an audio data API for the web isn’t a new method of delivering music.  This is about a new way of creating and collaborating that is stolen directly from the web’s play book.  The web beat every other way of working using it.  What will happen to music and sound if we let it do the same?

I’m in Raleigh, North Carolina, with Al MacDonald for the www2010 conference.  We’re here to present our work on exposing audio data in the browser.  Over the past month Corban, Charles, and a bunch of other friends have been working with us to refine the API and get new types of demos ready.  We ended-up with 11 demos, some of which I’ve shown here before.  Here are the others.

The first was done by Jacob Seidelin, and shows many cool 2D visualizations of audio using our API. You can see the live version on his site, or check out this video:

The second and third demos where done by Charles Cliffe, and show 3D visualizations using WebGL and his CubicVR engine. These also show off his JavaScript beat detection code. Is JavaScript fast enough to do real-time analysis of audio and synchronized 3D graphics? Yes, yes it is. The live versions are here and here, and here are some videos:

The fourth demo was done by Corban Brook and shows how audio data can be mixed live using script. Here he mutes the main audio, plays it, passes the data through a low pass filter written in JavaScript, then dumps the modified frames into a second audio element to be played. It’s a technique we need to apply more widely, as it holds a lot of potential. Here’s the live version, and here’s a video (check out his updated JavaScript synthesizer, which we also presented):

The fifth and sixth demos were done by Al (with the help of many). When I was last in Boston, for the Processing.js meetup at Bocoup, we met with Doug Schepers from the W3C. He loved our stuff, and was talking to us about ideas that would be cool to build. He pulled out his iPhone and showed us Brian Eno’s Bloom audio app. “It would be cool to do this in the browser.” Yeah, it is cool, and here it is, written in a few hundred lines of JavaScript and Processing.js (video 1, video 2):

This demo also showcases the awesome work of Felipe Gomes, who has a patch to add multi-touch DOM events to Firefox. The method we’ve used here can be taken a lot further. Imagine being able to connect multiple browsers together for collaborative music creation, layering other audio underneath, mixing fragments vs. just oscillators, etc.  We built this one in a week, and the web is capable of a lot more.

One of the main points of our talk was to emphasize that what we’re talking about here isn’t just a concept, and it isn’t some far away future. This is real code, running in a real browser, and it’s all being done in HTML5 and JavaScript. The web is fast enough to do real-time audio processing now, powerful enough and expressive enough to create music. And the community of digital music and audio hackers, visualizers, etc. are hungry for it. So hungry that they are seeking us out, downloading our hacked builds and creating gorgeous web audio applications.

We want to keep going, and we need help. We need help from those within Mozilla, the W3C, and other browsers to get this stuff into shipping browsers. We need the audio, digital music, accessibility, and web communities to come together in order to help us build js audio libraries and more sample applications. Yesterday Joe Hewitt was talking on twitter about how web browser vendors need to experiment more with non-standard APIs. I couldn’t agree more, and here’s a chance for people to put their money where their mouth is. Let’s make audio a scriptable part of the open web.

I’m currently creating new builds of our updated patch for Firefox, and will post links to them here when I’m done. You can read more about the technical details of our work here, and get involved in the bug here. You can talk more with me on irc in the processing.js channel (I’m humph on moznet), or talk to me on twitter (@humphd) or by email. One way or another, get in touch so you can help us push this forward.

One of my friends said that people don’t make things because it’s hard, but I don’t think it has anything to do with it being hard.  Most people don’t make things because they’ve only ever been taught how to consume.  They live in a plastic world impossible to reproduce, governed by the expert, the manufacturer, the teacher, the broadcaster, the chef.

But it doesn’t have to be so.  You can make stuff.  Tonight I was cleaning up after dinner, and noticed two pieces of cardboard on the table (scraps from a box), each with a perfect hole cut in the centre.  I called to our girls and handed them each one: “Make something cool with these.”  I didn’t have to say anything else.  They knew what to do next.  One made a bird house using tape, more cardboard and a pencil, with the round hole as the opening.  The other made an elaborate one-eyed mask and devised a way of moving while wearing it that made her “like a character in a story.”

People start out knowing how to make things.  They are taught to stop.  It takes time to refine one’s taste such that the homemade won’t do anymore.  You have to work at shutting down the desire to create and build and make.  But you can do it.  Most adults I know wouldn’t know what to do with a piece of cardboard that had a perfect circle cut out of the middle, or what to do with their laptop either.

This release also introduces another layer of automated tests for the project: ref tests.  We’ve had automated parser and unit tests for quite a few releases (there are over 1,200 tests in 0.7), but we were still having to manually inspect the results of our drawing code; and when you’re porting a graphics language, there are a lot of drawing tests.  We ended-up implementing an automated canvas 2D and 3D pixel-based ref test system.  The process works like this:

1. Write a visual test (processing code) and make sure the native Processing (Java) and Processing.js are the same (visual inspection).
2. Generate a ref test using a web-based tool.  The ref test is made by running the code, then extracting pixel data from the canvas and putting it in a comment in the code.  The file is then saved (code + bitmap pixel data) to a .pde file.
3. A web-based ref test runner reads in the test files and does 3 things.  First, draw the pixels back to a canvas using ImageData and putImageData.  Second, run the processing code to produce a live sketch.  Third do a visual pixel diff, comparing the two canvases.

Doing a comparison between the pixels of two canvases is easy for 2D, and a bit more complicated for 3D (Andor has a great post about his 3D method in WebGL).  Using a reference image generated by another browser on another OS and having it match *this* browser/OS combo?  That’s harder.  To compensate for differences in rendering and canvas implementations, we apply a Gaussian blur in JavaScript to the two sets of pixels.  We then compare the colour values using a colour tolerance.  Our initial testing used 2% but further testing points to needing 5% (e.g., 0.02 * 255).

And it works!  We only recently got the ref test infrastructure in place, so don’t have many tests yet (15 at present).  However, today, we caught our first bug using them: a recent check-in that changed curveVertex broke how we interpret control points.  Here’s what it looked like when it failed (red pixels are differences):

This technique is already saving us a lot of time as we test our code on Firefox, Minefield, Safari, Chrome, Opera and on Windows, Linux, Mac, and mobile OSes.  I think other people doing visual work with canvas should explore this method.  You can try it out for yourself here, and see code here.  Thanks to Andor, Yury, and Corban for their work with me to get this working.

In one of his test cases, he had a series of large images that were being flipped through as a slideshow.  Processing provides a draw() function, which is basically a loop allowing for animation to take place: whatever you do in draw() happens at the specified frame rate.  Or, it tries to run at that rate.  When we ran the test with his initial code, we were getting ~6 fps which seemed…slow.  After a bunch of profiling and debugging, we got that to 80-100 fps.

I took a look at his test case under Shark (Benjamin has a great post on how to use it) and noticed that we were spending 9.8% of our time in js_CoerceArrayToCanvasImageData.  Reading through the code it started to make sense.  In Daniel’s code, he had constructed his own ImageData object using a js array:

var imgData = {data: [], width: w, height: h};

Then, whenever he needed to alter pixels, he’d set imgData.data[n] as usual before updating the canvas:

putImageData(imgData, 0, 0);

On paper it looks good.  But it’s not fast.  Every time you do this (and we have to do it a lot, and as fast as possible), the canvas context looks at the data array, sees that it’s just a js array, and then coerces it into the proper type.  And that takes time.

I wrote a little test to demonstrate the issue.  This test picks 100 random gray scale colours and draws them one-by-one, pixel-by-pixel, to a 400 x400 canvas.  It does it first by using a custom ImageData object with a normal js array, then using a native ImageData object, returned via getImageData (NOTE: Opera doesn’t support createImageData, so I’ve used getImageData so I could test this on various browsers).  Here’s what I see on my MacBook Pro running OS X 10.6:

Right off the top, if you want this to work across all browsers supporting canvas, you shouldn’t use custom objects with js arays–it won’t work on WebKit.  But ignoring that, you shouldn’t do this on those browsers that do support it because you pay a big price.  It’s not surprising to see Firefox 3.6 showing no big difference between the two methods, since Vlad’s WebGL arrays aren’t in there.  But you can really see the effect in a trunk build that has them, which is faster than any other browser I tested (NOTE: I didn’t test the equivalent trunk builds for the other browsers, just releases).

As with all performance tests, please don’t take this as the final word, or generalize this as though I’m saying something about canvas performance in general.  I’m talking very specifically about the pixel array container you choose, and the numbers all lean toward using getImageData over [].  It’s a 6 fps vs. 80-100 fps kind of choice in our case.  Not a hard decision.