A Brief History of Test Frameworks

So the other night I was reading Beautiful Code by Oram & Wilson et al, and got to the article Beautiful Tests by Alberto Savoia. In this chapter, Alberto discusses the origins of JUnit. What he said about the ambitions, goals, and accomplishments of JUnit was perfectly true, but it got me to thinking about the “good old days”, and in particular by the work that was done at Taligent on test frameworks. I believe this work actually predates or at least parallels the whole SUnit/JUnit history, and I thought it might be interesting to people to hear the story. I also want to make sure that two talented engineers get the credit they deserve for their little slice of Test Driven Development history. I should also note that I’m not writing this to criticize Kent Beck or Erich Gamma, whose contributions to the same history are far greater than what I’m discussing here today.

From 1988-1990 (in case you haven’t figured it out yet, I’m old – in programmer years I’m like 113!), I worked for Apple in the inimitable Mac Developer Technical Support (MacDTS) group, the most fun I’ve ever had in any job, by far. I was part of Rick Blair’s “Futures” team, whose job was to keep track of upcoming technologies from Apple and make sure DTS was ready to support them when they were released to the public. In particular, I was responsible for covering the Pink project, an insanely ambitious effort to create an entirely object-oriented operating system and application framework using the brand-spankin-new C++ language. Like I said, insane. But I totally bought in and became very passionate about the project and its possibilities.

So I decided that I wanted to join the team in 1990, which at the time was something like 25-30 people. Art Cabral championed me into the organization (not sure if I should thank him or blame him!), and I ended up taking a job as manager of the newly minted “Instrumentation” group, building performance, build, logging and quality infrastructure for a team that had almost none of the above. I had two initial employees, one build engineer and one QA engineer. The rest I had to hire. To make matters worse, I had to write the first QA Plan for the entire project, and I knew absolutely nothing about QA and had very few QA resources to work with. So I talked to people, read a lot of books, and tried to figure out how we could ship an entire OS platform and a suite of applications without a lot of QA people. I wasn’t entirely successful, but I did figure out that we’d need to put a lot of the unit testing load on the individual engineers.

And of course, the Pink project was all about building object-oriented frameworks. So we decided we needed a framework for unit testing. At the time I did some research to see if anyone else had done something similar, but this was back in the days before search engines and I didn’t really find anything. So I did what managers did best: I delegated the task to some engineers on my team and went off to some meetings.

The engineers I speak of are Alan Liu and David McCusker. Alan went on to fame and fortune working on the internationalization/unicode features of Java (including IBM’s ICU4J), and David went on to work for Netscape and OSAF. Both great engineers. The system they built, beginning in 1991, had four main components, which were described as follows in the book I wrote for Taligent, The Power of Frameworks:

  • Assertions – “Provide a mechanism for asserting invariants in a program, generating exceptions when these invariants are not met”
  • Test Framework – “Provides a framework for executing, logging, and evaluating tests”
  • User Interface Testing – “Provides tools and services for driving the user interface of a CommonPoint application under test”
  • Utility Tests – “Provide standard tests for common object behaviors, including hashing and streaming”

Of these four components, the Test Framework is of the most interest historically – the assertions system and the UI Testing capability weren’t anything all that new. The Test Framework class hierarchy that looked like this (sorry for the poor quality image):

Taligent Test Framework Class Hierarchy

By contrast, the original JUnit class hierarchy, as described in “JUnit: A Cook’s Tour”:

JUnit Class Hierarchy

The similarities seem fairly striking to me. Factor in the fact that Erich Gamma worked at Taligent at the time the Test Framework was developed, and in fact gave the engineering team some good feedback, and its pretty hard to argue that Alan and David don’t deserve at least some credit for developing one of the original ancestors of JUnit.

But what of SUnit? Its website bills it as “The mother of all testing frameworks”, and its clearly been around for a while. Several sources on the web say that JUnit was derived from it. Unfortunately, I couldn’t find anything about it on google dated earlier than 2000, but I know that date is wrong, since JUnit was being written about in 1998. Could it be that SUnit and the Taligent Test Framework were developed in parallel, serendipitously? Absolutely. I’d love to hear from someone who knows, and will be sending a link to this article to both Erich and Kent in the hopes that they can fill in the blanks.

I’d also like to see the wikipedia entry on JUnit updated appropriately, but don’t want to do so myself for fear I won’t be able to remain unemotional about things.

Finally, for reference, here is the Testing section of the book Inside Taligent Technology by Sean Cotter with Mike Potel, June 1995, ISBN 0-201-40970-4. I include the scanned PDF here because the book is long out of print. It includes a lot more detail about how to write tests using the Test Framework, which should further convince anyone who might be skeptical of the similarities.

[Update 08-27-2007] Received a reply from Kent Beck, and decided to publish it and the prior response from Erich in the body of the post instead of adding it to the comments. As I suspected might be the case, it looks like it really was a serendipity that Erich was in the same place for the development of the two frameworks.

From: Erich Gamma
Sent: Wednesday, August 22, 2007 10:25 PM
To: Andrew Shebanow
Cc: Kent Beck
Subject: Re: test framework history

Hi Andy,

the starting point for the work on JUnit was Kent’s SUnit. SUnit was written in Smalltalk. It is unrelated to the Taligent Test framework work. What Kent and I did together was to implement SUnit in Java while preserving the original spirit of SUnit. This went somewhat like this Kent “in Smalltalk I would”, me: “you don’t do that (or may be more often… you cannot do that) in Java”…

At that point I have to confess that I didn’t write any test when I was a member of the Hoops team at Taligent. It was Kent that has infected me with this testing stuff.

Funny timing… 3 weeks ago I was standing with my family in front of the Taligent building after more than 10 years and now you point me to a blog with a Taligent class diagram … it was a great time…

–erich


From: Kent Beck
Date: Monday, August 27, 2007 10:54 AM
To: Andrew Shebanow
CC: Erich Gamma
Subject: RE: test framework history

Andrew,

Thank you for the invitation for a trip down memory lane. I wrote SUnit in 1994 I think. I had experimented with various other ways of running automated tests but none of them really stuck. I was on my way to a client in Chicago the next day and I knew I would be advising them to write tests. I knew I should also tell them how. I took the workspaces I used to drive development in Smalltalk and objectified them. The workspace would start out like this:

| a b |
a := ...
b := ...
...some code involving a and b

I’d select the code and print the value until I got the answer I expected. Then I’d copy the first part and change the code:

| a b |
a := ...
b := ...
...some code involving a and b

| a b |
a := ...
b := ...
...different code involving a and b

I’d print the second one until I got the answer I expected.

The objectification went like this:

  • the workspace became a class
  • the temporary variables became instance variables
  • the assignments of the variables became the setup method (avoiding duplication)
  • the rest of the code involving the variables became a method
  • the judgement about whether the right thing was calculated became assertions

I wrote this up as a little three class (TestCase, TestSuite, TestResult) framework with, I think, twelve methods. The clients were able to understand and use it (they still are to this day).

The framework was so simple that I couldn’t imagine that it had value, even though I thought it was aesthetically cool, so I gave it to the smartest programmer I knew and asked him to try it. When I didn’t hear back for a month I figured, “Okay, cool little idea but not big deal.” Then he wrote me back ecstatic. He had some really complicated code to try to make work and he couldn’t debug it because it was so low level. Instead of continuing to beat his head against the wall, he decided to write unit tests for his most basic data structures. He found a bunch of defects doing so, and when he was done the whole system ran.

I’m sorry that Mr. McCusker feels slighted. To have been working on moving testing into the inner loop of development at that time was far-sighted. I was aware of some of the stuff going on a Taligent (I had moved into an office vacated by a couple of the original Pink guys at Apple), but I knew nothing of their testing activity and I’m quite sure I never saw the testing framework.

Regards,

Kent Beck
Three Rivers Institute

About these ads

~ by Andrew Shebanow on 21Aug07.

8 Responses to “A Brief History of Test Frameworks”

  1. David McCusker has posted a followup:

    http://www.briarpig.com/log/aug07.html#test-frameworks

    I should note that lke David, I noticed the similarities long ago, when JUnit 1.0 was first released. Reading Beautiful Code just motivated me to blog about it.

  2. I received a reply from Erich Gamma, and he kindly gave me permission to post it here. I’ve edited the email to remove the actual email addresses, but otherwise it is untouched. My only comment is that it really makes we want to hear more about the history of SUnit.

    From: Erich Gamma
    Sent: Wednesday, August 22, 2007 10:25 PM
    To: Andrew Shebanow
    Cc: Kent Beck
    Subject: Re: test framework history

    Hi Andy,

    the starting point for the work on JUnit was Kent’s SUnit. SUnit was written in Smalltalk. It is unrelated to the Taligent Test framework work. What Kent and I did together was to implement SUnit in Java while preserving the original spirit of SUnit. This went somewhat like this Kent “in Smalltalk I would”, me: “you don’t do that (or may be more often… you cannot do that) in Java”…

    At that point I have to confess that I didn’t write any test when I was a member of the Hoops team at Taligent. It was Kent that has infected me with this testing stuff.

    Funny timing… 3 weeks ago I was standing with my family in front of the Taligent building after more than 10 years and now you point me to a blog with a Taligent class diagram … it was a great time…

    –erich

  3. [...] Andrew Shebanow: [...]

  4. “Simple Smalltalk testing” Kent Beck Smalltalk Report 4(2) Oct 1994

    A pdf was available on Donald Macqueens website, but it doesn’t seem to be responding today:

    http://www.macqueen.us/apache2-default/www.macqueen.us/smalltalkReport/ST/91_95/SMAL0402.PDF

  5. Hi Andrew,

    I am writing my Computer Science Master Thesis on tests creation and I would like to put an appendix with the mail Kent Beck wrote you about how he created SUnit. Is it possible? I will put a reference to you, your blog and your post.

    Thank you in advance,

    Sback

  6. Sback,

    Yes, you can, absolutely. Thanks!

  7. [...] A Brief History of Test Frameworks에 나온 Kent Beck의 [...]

  8. OK, I know it has been years since I wrote this, but I was just re-reading it and decided to follow the link in Isaac Gouy’s comment to the Smalltalk Report article from 1994. The link is still broken, but some editing made it work. Highly recommended: http://www.macqueen.us/smalltalkReport/ST/91_95/SMAL0402.PDF

Comments are closed.

 
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: