Things then and now, part II

By david.humphrey | Published: February 8, 2010

Yesterday I wrote about a short newspaper article from March 31, 1949.  While I was looking at it, I also took a moment to read the surrounding articles.  The article itself was actually reprinted as part of the “From Our Early Files” section, and one of a dozen short pieces from 10, 20, 35, 50, and 70 years ago.  The context here is useful for what I’m about to say: small town newspaper from March 31, 1959, with articles quoted back as far as 1884.

A few months ago, David Eaves wrote that the best way to understand Twitter was to think of it like a newspaper vs. email.  I know exactly what he means, and have been interested to see how often I’ve been referred to his post via blogs, twitter, irc, etc. when people are looking for a shorthand way to say, “see, this is what it is.”  However, some people I know who are (or have been) deeply invested in the contemporary newspaper, have commented to me that this metaphor doesn’t work.  I think part of the issue here is that the contemporary newspaper is so unlike the newspaper of the past.  In particular, it has disconnected itself from the local.  It has become what is worth printing instead of what is news, where the arbiter of worth is the national or international news desk.  I would also argue that it is this move from covering here to there, that has killed the newspaper: when there becomes what we all discuss, there’s no reason to get that from someone here.

I think Twitter is like the newspaper as it was before they were all bought-up by the same handful of media corporations.  What follows is anecdotal evidence to prove my point–where the anecdote is necessarily at the heart of the what is important to the community.  Allow me to read you some of what was newsworthy from 1959 to 1884:

1949:

  • Rev. W. E. Aldworth of St. Marys has accepted a call to become the minster of St Pauls United Church.
  • R. A. (Dick) Norman, by a recent appointment of the Tillsonburg Shoe Company, has been named office manager of the plant.

1939:

  • Arrangements are being made to arrange transportation for the boys and girls of Tillsonburg and district to London on Wednesday June 7 to see the royal procession on the occasion of the visit of the King and Queen
  • Alexander Halbert has been appointed chief of police for Tillsonburg at a special meeting of the council on Friday

1924:

  • E. J. Pinch has purchased a vacant lot on Pine Street.
  • S.E. Barret has sold his house and lot on Bidwell Street to Wilbur Prouse

1909:

  • H.G. Coomber has sold the Delmar Sunday school a beautiful Gerhard Heintzman piano which he has placed in their schoolroom this week.
  • Theo Young is have a veranda erected at the front and south side of his house on King Street, which will improve its appearance very much.

1884:

  • Tillsonburg Agricultural Implement Manufacturing Co. has commenced work for the season and their shops present a busy scene.  They have begun the season’s work with about 30 hands.  John McIntyre is manger.

I love this.  Can you imagine anyone caring about a new piano being purchased, a veranda being built, someone buying a vacant lot, the name of a new manager or chief of police?  It’s not worth the paper its printed on, irrelevant, and only of interest to a small community of people.  And that’s exactly the point.  Who cares that people got a new job, are going to an event, are redoing their basement, etc?  Well, when its happening within your community, its interesting to read.  I could turn any of the above into something you’d write on Twitter (first person vs. third) and the subject matter wouldn’t cause anyone reading Twitter to think it out of place with all the other things being discussed on Twitter right now.  It’s not that Twitter has created a new narcissistic space for people to write about trivial aspects of their lives, which no one else will find interesting or useful; it’s that Twitter has tapped into the spirit of the local news, long ago killed by the need for here to be focused on there.  We’re only able to be a community when we can share what’s important to us.

Posted in Digital Swag, Idea Factory | 3 Comments

Things then and now

By david.humphrey | Published: February 7, 2010

I wrote a letter to my grandmother a little while ago.  I was partly reminded of the need for such writing by Luke, but also I had wanted to connect some of my current work with my past.  Yesterday my grandmother was here visiting, and she took me aside just before she left.  “I have something for you.”

I’ve written before (go read that, I’ll wait) about my grandmother, about my grandfather, and about my connection to a man I never met.  He’s been an unseen presence in my life since I was little, a living spirit that infuses my sense of where I came from, what drives me, why I do what I do.  My recent work to bring audio data onto the web is partly a nod to a man who also tinkered with sound.

“I had your mother photocopy this, and I want you to have it,” she told me, and pulled a yellowed newspaper article from her purse.  In it was told the story of my grandfather making a TV out of radio parts:

What is believed to be the first television receiving set in this district has been constructed by Alex (Scotty) Cassells, 5 Townline, and has been in operation about three weeks.  The set will pick up two Cleveland, Ohio stations and one station from Buffalo, N.Y., and the builder states that the receoption is exceptoinoally clear on the Cleveland stations  The set was built at a cost of $160, with many parts taken from old radios and some of the parts homemade.  The reception is entirely dependent on the weather and it will not operate in damp or rainy weather.  It took three months of Mr. Cassell’s spare time to build the set.  At present time the aerial is a simple dipole, about 20 feet in length, but the builder hopes to get great reception when he installes a stacked array of aerials, and extends the height to approximately 40 feet.  There are 27 tubes in the set. — Tillsonburg News March 31, 1949

So that’s about $1,500 in today’s money, or roughly the amount it costs to get a decent laptop, three months of spare time in his basement, and a bunch of recycled and homemade parts?  Sounds really familiar.  Lots of things change in 60 years.  But not everything.

Posted in Experiments with audio, Idea Factory, family | 3 Comments

Writing Automated Tests for Processing.js

By david.humphrey | Published: February 5, 2010

I finished up some work today on the Processing.js automated test tools.  Previously I wrote a test harness to allow JavaScript based unit tests, as well as parser tests.  However, after watching some of my students struggle through bugs during the lead up to their 0.5 release, I decided we needed a way to also be able to write them in Processing.  Right now I see a lot of tests being drawn into sketches, and then manually comparing results.  It’s sub-optimal to say the least.

Having written the unit test functions already, I decided to simply load them into Processing via a library.  Processing.js, like Processing, has the capability to load “native” libraries.  In Processing.js, such libraries are JavaScript functions written like so:

(function() {
  Processing.lib.Foo = function() {
    this.something = function() {...};
    ...
  }
})();

Then, in your Processing.js sketch code, you initialize the library by calling its constructor (e.g., Foo();), and then any function (e.g., something();) will be available to your code.

Using this technique I added the same set of _check* functions already available to JavaScript unit tests, such that I’m now able to write small unit tests in Processing like this:

// Tests for Processing str() function - http://processingjs.org/reference/str%28%29
boolean b = false;
byte y = -28;
char c = 'R';
float f = -32.6;
int i = 1024;

_checkEqual('false', str(b));
_checkEqual('-28', str(y));
_checkEqual('R', str(c));
_checkEqual('-32.6', str(f));
_checkEqual('1024', str(i));

I’ve fully documented the setup and use of the new automated testing features. All Processing.js developers (especially my students), should read through this. Beginning with the 0.6 code, we’ll start using this by default for all regression and API testing.

Posted in CDOT, Mozilla Education, Seneca, Teaching Open Source | 1 Comment

Another road

By david.humphrey | Published: February 2, 2010

There is another road.  It is, for the most part, hidden from view.  I have seen it only a few times, and always it has been those walking upon it that have made it appear to me.

I saw it for the first time in a wicked snow storm, just as the light was succumbing to the night.  Out of the corner of my eye I saw them, walking in single file, bent low beneath the wind, wolves.  Their path cut across the lake.  They walked with purpose out of the woods on one side and onto the frozen water.  So straight was their path that they seemed to be crossing a bridge I could not see.  They were on the road.

I saw it yesterday as I passed a creek, stopped solid with ice.  One, two, three deer went across, filling the space between the trees, their bodies revealing its direction.  They didn’t pause to look at me, didn’t stop to consider the sun coming out from behind a cloud.  They were on the road.

The road, this road, was not built.  It cannot be seen, only walked, only witnessed.  It is drawn with the precision of a draftsman’s ruler, and always at a right-angle to the one we would draw.  It is across our path, over the creek, and through.  It moves across the horizon, cuts us in two.  When you see it you are made aware that the road you are on, the one that felt so solid before, the one that seemed to be going somewhere, is really just a way point on the other road.  It is this other road that I am always looking for, this other road that I sometimes see.

Posted in Idea Factory, Nature | Leave a comment

The case of the open file handle

By david.humphrey | Published: February 2, 2010

I did a rewrite of the Processing.js test suite code to get around a bug Al hit on Linux.  In doing so, I hit another nasty bug that Ted eventually solved.  I needed to get an escaped version of JavaScript or Processing code loaded into the js shell via the -f flag.  Previously I had been passing this as a string via -e, but we were erroring out as the size of the files increased beyond the max arg length allowed.  My solution was to create a temporary file in Python, stick the encoded script in, pass that into the shell for processing, then delete it.

It worked great on OS X, but on Linux we were getting an exception about half-way through the test run “Too many open files.”  We have 855 test files at present, so it seemed unlikely that we had hit a real OS maximum.  I tried adjusting the number of file descriptors per process (ulimit -n 8192) and it worked.  Looking at my code I first thought that I must be accumulating pipes (I use stdout and stderr to communicate with the js shell).  Nothing I could think of was fixing it, and it seemed crazy to me that subprocess.Popen() wouldn’t close them properly.

Eventually Ted figured out that my issue was that I wasn’t closing my temp files properly.  Here’s the relevant bit of what I was doing:

tmp = tempfile.mkstemp()

t = open(tmp[1], 'w')
t.write(es)
t.close()

“But I am closing it,” I protested. “No, sir. You’re not!” said Ted, and he was right. Looking again at the docs for mkstemp(), and reading more carefully, I noticed this:

mkstemp() returns a tuple containing an OS-level handle to an open file (as would be returned by os.open()) and the absolute pathname of that file, in that order.

Did you catch that?  “…an open file…”  I missed it totally.  I have said it before, but it is worth repeating: Python has really amazing docs.  I should read them more often.  Here’s the fix:

tmp = tempfile.mkstemp()
os.close(tmp[0])
Posted in CDOT, Seneca | Leave a comment

Flatbreads

By david.humphrey | Published: February 1, 2010

On the weekends I try to devote myself to as much time in the kitchen as I can, and to focus on attempting something I haven’t tried before.  This past weekend, for reasons I’m not clear on myself, I was interested in flatbread.  I wanted to share a number of observations I made:

  1. As with all bread, and despite the fact that they don’t rise to the same heights, flatbreads need time.
  2. You can help the yeast, and the flavour, by adding some sugar to the dough when you mix it.
  3. Adding some fat (butter, ghee, oil) is often a good idea.  Fat in bread makes it lighter and softer.  That white bread you ate as a kid that isn’t good for you?  It has fat in it.  Bread isn’t soft on its own.
  4. When stretching dough, the most important tip is to let it rest.  If you’re pulling it, and it simply snaps back, it’s not ready.  Invert a bowl over it and let it rest for 7-15 minutes before trying again.
  5. Cut, never tear dough.  Use scissors, a sharp knife, or, if you have it, a bench scraper.
  6. You want it thinner than you think is wise (it will rise in the heat), but not too thin or it will burn.
  7. Flatbreads are somewhat hard for children to work with, since they require a margin of error much narrower than normal loaves.
  8. If you don’t have a Tandoori (who doesn’t?), you can use your broiler on high heat in a pinch.
Posted in Food, family | Leave a comment

Bad Gr@mm3r?

By david.humphrey | Published: February 1, 2010

The Globe picked up a Canadian Press article today, whose title echos the favourite refrain of big media and other institutional onlookers of so called social media: “Students failing because of Twitter, texting.”  In it the author cites some “solid evidence” for the demise of writing skills at the post secondary level:

Ontario’s Waterloo University is one of the few post-secondary institutions in Canada to require the students they accept to pass an exam testing their English language skills.

Almost a third of those students are failing.

…

“What has happened in high school that they cannot pass our simple test of written English, at a minimum?” she asks.

Even those with good marks out of Grade 12, so-called elite students, “still can’t pass our simple test,” she says.

Poor grammar is the major reason students fail, says Barrett.

“If a student has problems with articles, prepositions, verb tenses, that’s a problem.”

We’ve heard all this before.  Poor writing?  Increased use of social media?  Let’s connect the dots.  A little further into the article, we find some other interesting bits; let’s start with grammar:

Some students in public schools are no longer being taught grammar, she believes.

…

“When I went to high school in the ’70s I was never taught grammar in English. I learned grammar from Latin classes.”

…

“We haven’t taught grammar for 30-40 years…(and it) hasn’t worked.”

“It’s not that hard to teach basic grammar,” he says.

I learned grammar studying classical Greek through four years of university.  I was never properly taught grammar in high school, because that final statement above is false: it is hard to teach grammar, or perhaps it is hard to teach it well.  Grammar, as an abstract system, is very difficult to grasp when you only know one language.  Without being able to step outside English and examine it from the vantage point of Greek, I don’t think I would have ever understood it.  But in the context of learning a second language, especially its literature, one is forced to map grammatical structures back and forth.  You can be a very good speaker or writer of English and not know the names for the constructions you use.  You can’t learn to read another language without them.  In other words, the need for grammar has to be real if one is to learn and then retain it.  I think many of the high school students who failed these tests did in fact learn grammar; they just didn’t retain it beyond the final exam.

Another issue discussed at length is spelling, contractions, and other short forms:

Emoticons, happy faces, sad faces, cuz, are just some of the writing horrors being handed in, say professors and administrators at Simon Fraser.

…

“Instead of ‘because’, it’s ‘cuz’.

…

“The words ‘a lot’ have become one word, for everyone, as far as I can tell. ‘Definitely’ is always spelled with an ‘a’ -’definitely’. I don’t know why,” says Paul Budra, an English professor and associate dean of arts and science at Simon Fraser.

I find the last example particularly funny.  The use of alot vs. a lot frustrates me, I’ll be honest.  I have a friend who uses ‘alot’ exclusively, and one day I decided to send him a private note telling him of his mistake.  He replied politely that he knew, and had made a conscious decision to change the spelling as a way of normalizing ‘alot.’  I was blown away at first, but later, upon reflection, had some respect for his decision.  Spelling reforms have happened throughout history, whether through decree or by means of popular usage shifts.  If ‘definitely’ is always spelled ‘definately,’ it soon will be.  Is it E-Mail, E-mail, e-mail, or email?  Spelling is very much influenced by usage, just as meaning.

Students haven’t learned grammar.  Students haven’t learned to spell.  Ergo:

Cellphone texting and social networking on Internet sites are degrading writing skills, say even experts in the field.

“even experts” notwithstanding, how do we make the leap from an obvious lack of focused study on particular areas of language learning, and the increased use of cellphones, social networking, and the web?  Furthermore, how is it that these are degrading writing skills?

I have spent the past decade teaching software engineering to third and fourth year undergraduate students.  I teach in what is often thought to be an illiterate discipline (in addition to English and Greek philology, I also studied Computer Science, so I can speak from experience).  What does programming have to do with writing?  A lot (or should I say, Alot).

In my courses I expect students to do a great deal of writing and reflecting.  Students are asked to keep a blog, use Twitter, collaborate on wikis, and communicate in real time using irc.  My experience in doing this over the years is that students have a lot to say, are thinking in ways that help move the class forward when shared, and most of all, that they have much to teach the other students and myself.

Do I see them make mistakes in terms of punctuation, grammar, and style?  I do.  Do they see me do the same thing?  Most certainly they do.  But we’re all writing.  Can students write?  Yes, if you let them.  Yes, if you model writing, and reading for them.  One of my students asked me a few weeks ago if I’d ever done a word count on my blog.  I hadn’t but decided to try.  It turns out I’ve written the equivalent of 3 good sized novels since 2006.  I haven’t written a novel, though, and don’t expect I ever will.  Instead, blogging, Twitter, social networking, and the web have insured that I exist in a literate age, where writing every day is the new normal.  I would argue that if students were allowed to use the web more, they’d be in even better shape.  As David Eaves notes of his blog, “if writing is a muscle, this is my gym.”

I’m less skeptical of students’ ability to write than I am of the intentions of educators decrying the demise of writing.  When all we have of writing is a “simple test,” education is reduced to a set of questions you can use in order to substitute real reading and ideas for a bell curve.  We’ve seen “simple tests” before, and they often have very little to do with real writing.  Writing, like many worthy pursuits, is easily stifled by pedants, who confuse the rules of writing for writing itself.

Posted in CDOT, Mozilla Education, Seneca, Teaching Open Source | 3 Comments

Giving the parser a B-

By david.humphrey | Published: January 29, 2010

Today I finished-up my work writing automated tests for the Processing.js parser and a full unit test infrastructure.  It has been an interesting bit of work, and resulted in a lot of good info.  Today I imported Ben and Casey’s sample code from their Processing books, and it was interesting to watch things pass and fail.

Of the 855 Processing files I tested today, 623 were parsed correctly (72%).  This leaves 232 that fail for one or more reasons.  It’s actually not as bad as it sounds, since there are six main errors I see over and over, for which I’ve filed bugs, and a bunch of other failures that are likely related to my python/js scripts.

I’m pumped to have some new students about to start writing tests and doing more investigations into these bugs.  At the same time, we’ve got a new project coming online to do an antlr-based rewrite of the parser.

If you’ve got Processing code that you want to make sure works on the web, let us know so we can add it to our automated test suite.

Posted in CDOT, Mozilla Education, Seneca, Teaching Open Source | Leave a comment

On the joys of the test harness

By david.humphrey | Published: January 25, 2010

This is the story of my favourite kind of programming.  The kind of programming where you have a handful of existing tools, a bunch of data, and you need to make it all fit together somehow.  There’s no neat-shrink-wrapped-sofware-download-it-from-the-web way to do it (none of this stuff was meant to go together), so you start hacking your tools, abusing data formats, work in multiple languages, and otherwise get round pegs to slide smoothly into square holes.

Today’s fun comes as a result of the need for a test harness to drive automated tests for the Processing.js parser.  I tell it in large part to help my students see how one leverages open technologies and formats on the way to solving otherwise unsolvable problems.  There’s no button in Visual Studio for what we’re about to do here.

The story is set within the Processing for the Web project, which is making a JavaScript version of Processing.  The past few days we’ve been trying to track down some bugs related to the parser.  Right now, there is a clever hack in place to take Processing code, and using a series of regular expressions, turn it into pure JavaScript.  It’s a nice bit of code, and works pretty well turning the Java-style syntax into something the browser can understand.  It’s also very brittle.

What we really need here, and what Processing itself uses, is an antlr based grammar to build our parser on top of JS.  Processing basically subclasses the Java antlr grammar, and we could do the same thing, only use the JavaScript target.  Probably doing this in parallel to a parser-lite version in the browser, so that you could choose to create your pure JS Processing code once, and then include that in your web pages.  I’ve filed a bug, and some unsuspecting student is going to get the chance to work on it soon…

But!  before you can go reimplementing a parser, and before you can properly fix regressions in the current one, you need a proper test suite.  We really need a simple way to take valid Processing code, and make sure our parser produces valid JavaScript.  And we need it now, so we don’t break more code while we fix these bugs.  I debated making it a student project for this term, but Al convinced me that we needed it for the next release.  So today I wrote a test harness.

The problem is actually pretty simple, but automating it required some thinking.  I have a parser written in JS that is meant to be run in a web page, and I need to take lots of Processing code and run it through, some of which I want to pass, some of which I want to fail, etc.  In order to automate this, I really want to decouple this from the browser, and do it from the command-line.

A lot of these problems are already solved in the Mozilla context.  With Firefox we have a whole slew of testing frameworks and tools to drive our various automated tests.  To run a lot of these, we don’t need a full browser, just a JS shell that can execute our JS code and report to stdout.  I spoke with Ted today about our issues, and he recommended modeling it on how xpcshell tests work (i.e., js + python + makefile).

The first step was to take a directory of Processing files and turn them into something I could feed into a test harness running in the jsshell.  The jsshell can run the Processing.js parser, since it’s just JS, and as long as I can feed the Processing code in, I’m good.  There’s even a handy load() function I can use to get files into the jsshell.  However, what it loads has to be valid JS and Processing code is anything but.  I needed a way to trick the jsshell into loading the Processing code in so I could get it into the parser and tested.

I chose to use python, and decided to write it such that it would scan the test dir for all the files, and turn them into JSON strings, which I could pass on the command-line to the jsshell executable (i.e, ./js -e ‘{code: “x =7;\ny=5;\n…}’), along with my test harness scripts.  I then wrote my test harness in JS, which also loads the Processing.js code (i.e., the parser), and then I basically do this:

try {
  eval(Processing(canvas, processing-code));
  _pass()
} catch (e) {
  _fail()
}

The Processing function takes a canvas element (where the graphics will get drawn), and the Processing code, and tries to parse it.  What it gives back is pure JS, so we can use eval() to try and execute it.  If that works, the generated JS is good, so our test passes; if not, we fail.  But there’s another problem: the Processing code assumes there’s a canvas element and a DOM–basically, it assumes it’s running in a web browser as part of a page.  But we’re nowhere near a web browser when this runs, so we have to fake it.

The nice thing about JS is that you can create and modify objects, data, functions, whatever on the fly.  In my case, I needed to fool the Processing.js parser into thinking that it had access to various DOM functions and objects (e.g., document, setInterval, etc.).  My solution was to create dummy functions and object literals that could stand-in for the real thing.  The parser doesn’t actually need to call this code; it just needs to be there and non-null.  Here’s my setInterval implementation:

var setInterval = function() {};

Using this trick I was able to slowly write all the bits of the fake DOM I needed to get the parser running.  As I’ve tried running a few other larger tests, I’ve noticed other bits of the DOM I need to implement, but it’s pretty trivial to add them as I go.

I’ve filed a bug and am waiting for some feedback before a I fix a few more things and get it checked-in.  Once I do, I plan to find some other students to write a ton of tests to try and break the parser even more, and then file bugs so we can fix them.  Writing tests may not be the most fun you can have, but honestly, writing a test harness is pretty close.

Posted in CDOT, Mozilla Education, Seneca, Teaching Open Source | 2 Comments

San Francisco goes Open Source

By david.humphrey | Published: January 22, 2010

There’s a great post over at Mashable by San Francisco mayor, Gavin Newsom, on the city’s new open source policy.  In it he discusses the democratic, fiscal, and technical advantages of open source.  He writes:

One of the greatest technology changes of our times is the rise of open source software…Open source software is created by the people for the people and as such is ideal for government. To that goal, I am extremely proud to announce today the nation’s first open source software policy for city government.

San Francisco’s new policy requires city departments to consider open source software equally with commercial products when purchasing new software. The opportunities with open source are tremendous: lower costs, greater agility, better reliability, improved security, and increased innovation.

Under the leadership of our City CIO, Chris Vein, and the Department of Technology, we have witnessed the benefits of open source with shorter implementation times and lower costs. We have seen this with my web site, DataSF.org, RecoverySF.org and our 311 integration with Twitter.

This week I’ve been introducing our students to open source, and a lot of our discussions have focused on how open source isn’t a fringe movement anymore–individuals use it, big companies use it, and governments use it.  As government budgets continue to shrink, and fiscal responsibility becomes front page news again, open source is being discussed by an ever increasing number of people.  Developers in San Francisco have long understood the value of open source on a technical level, but it’s another thing when the mayor’s office gets in the game, too.

If you’re interested to watch this same phenomenon happening in Canada, keep one eye on David Eaves.  Hey Mayor Miller, where’s our open source policy?

Posted in CDOT, Mozilla Education, Seneca, Teaching Open Source | 1 Comment