Lessons learned in TDD

Leave a comment

This was an excellent talk Ian Cooper gave at NDC a couple years ago.
It hits on a lot of the challenges and common problems Ian and others have seen with test suites and TDD in general.

It’s a great talk. Which you can watch in it’s entirety here.

Here are some notes.

The Problem

Screen Shot 2016-02-16 at 8.54.34 AM

Where did it all go wrong

Screen Shot 2016-02-16 at 8.54.40 AM

Two keep points
Not writing tests against behaviors.
Coupling our implementation details to our tests.

If we fix these we will have smaller test suites, much more self explaining, and much less painful to own.

There was advice in early TDD that said when you are given a new method on a class, that was the trigger point for writing a new test.

And that’s really wrong.

The trigger for writing a new test is to have some piece of behavior that I want to implement. The test needs to capture that behavior.

The reason why when you go back to your tests that you find them so hard to read is there is a lost connection between the low level test you are writing and the high-level feature you are trying to solve. It’s lost in the noise.

What you need to do is express in your test that we are testing a given behavior of our system.

Before you get to the point where you are going to put the implementation details down, you first put down the behaviours you are trying to capture, and then the implementation. Don’t jump ahead too early.

Testing outside in

Start with the use case, the story, the scenario or feature we want to solve, on the outside, then work your way in from there.

Recommendation: don’t start this at the UI level. Start one level beneath at the plain old object level. Then if your UI changes, it won’t matter. Focus on the domain models.

Test the public API of classes. Not the internals. Test the surface only, and it should be quite narrow.

As soon as you start testing the internals you are coupling your tests to your implementation details. To change your implementation details you now have to change your tests. And that’s the key problem.

So your surface area should be much narrowly than many people are testing today. Just the API.

That means you will write less tests, tests against the use cases, and refactoring the contract remains the same, the internals change, and you don’t break any tests.


This is what BDD is all about. Testing behavior, and not low level details, and methods on classes.

What is a unit test

Screen Shot 2016-02-16 at 8.54.48 AM

The simplest thing

Screen Shot 2016-02-16 at 8.54.55 AM

The real problem with TDD is Kent asked us to go green as fast as possible, committing as many sins as we want in that step. So go to project, cut and paste some code, and stick in application. Hack it. Make it work. Don’t worry about the design. Get it green ASAP.

Kent’s point here is that it’s difficult to do two things at once. Make the test pass and design at the same time. Better to separate out.

Green is about solving the problem. Refactoring is about doing the design.

Screen Shot 2016-02-16 at 8.55.00 AM

You do not write new tests here when refactoring. You already have the high-level behavior tests written. So long as those pass, you are OK. If you need another test, write it at that level.

But when you are refactoring, you are free to focus on design and engineer it right.

If you write tests here you bind your implementation details to your tests.

Coupling is the biggest problem in software. It is the enemy. Forget DRY. Coupling will kill you. Do you couple tests to implementation details.

Screen Shot 2016-02-16 at 8.55.06 AM

Dependency is the key problem in software development at all scales.

We need to eliminate the dependency between our tests and our code. Tests should not depend on details. Tests should depend on contracts or public interfaces. This allows us to change implementations without changes tests.

Test behaviors not implementations.

Means less tests.
Moving faster.
Going quickly to green.
Now you are refactoring and making it cleaner.
Better forward progress, the tests aren’t slowing you down so much.

Screen Shot 2016-02-16 at 8.55.13 AM

XP Days at Spotify


Screen Shot 2015-10-30 at 1.08.46 PMToday I had the pleasure of hosting the second day of a two day XP style workshop with some incredible developers at Spotify. Here’s what we did.

Build something fast!

We kick things off by asking the class to build a Reverse Polish Notation (RPN) style calculator. In 10 minutes.

We do this partially by design – to create a little stress. But it’s also an ice breaker. Just to verify everyone’s machine is up and running. As well as gauge the experience level of the class.

We then have a discussion about what some of the challenges are when building software. As well as what quality means, and how we can shoot for that in our software.

The birth of XP

This then segues into the birth of XP. Here I share Kent and Ward’s original vision of what XP was. Talk a little about the C3 project. And basically explain the XP attitudes behind if certain things are hard, we are going to continuously do them, all the time. Testing, integration, pairing, design, etc.

I also draw the cost of change curve and share how Kent wanted a way to develop software, such that the cost didn’t increase exponentially over time.

Screen Shot 2015-10-30 at 1.08.55 PM

Elephant Carpaccio

Partly because smaller stories are a good idea, and partly because it helps them with the project they do on day two, we then do an Elephant Carpaccio story slicing exercise.


This is great because it reminds people about what makes stories valuable in the first place, while also giving advice on how to slice. Basics, but really good stuff for people who may not have read or had any story training.

TDD together

This is where we formally introduce TDD, and revisit the RPN calculator we built at the beginning of the course.

As a class, I start with one other person, showing them the mechanics behind writing the test first, making it pass, and then … sometimes refactoring. Sometimes I don’t refactor, just to let the duplication build up and save for conversation later.

But we do this as a group, and it gives students a nice intro on a codebase and a problem they already know.


Something that’s surprised me as a middle aged programmer (cough) is how many people still haven’t read Martin Fowlers seminal book on refactoring. I have a lot of fun with this.

Refactoring, even in really tight circles of programming, is still a very misunderstood term. Usually it just means rewriting code. But people don’t appreciate always that when you are refactoring you aren’t adding any new value. You are only improving the existing design.

So I really hit home that the reason why that word appears in a menu at the top of your favorite ID is because of this book. And then I ask how people refactor without unit tests.

It’s a trap of course. Because you can’t. But a lot of people still do. So we talk about that.

So that’s day one. And during the day we also sprinkle in discussions about YAGNI, production readiness, and other XPisms and attitudes.

Code me a game

Day two is all about putting TDD, unit testing, refactoring, continuous integration, collective code ownership, and pairing all into practice. In teams of four, the class then goes about building a text based adventure game.

Screen Shot 2015-10-30 at 1.12.24 PM

Text based games are great for learning TDD. They are fun, most people understand how they work, and they don’t contain a lot of hairiness. But they contain enough hairiness to give a taste of what applying TDD in the real world is like (like how do you handle all those input output streams?).

We have a few rules for the text based game we are going to build.

  1. No production code without a failing unit test.
  2. No code not written in pairs.
  3. Refactoring happens continuously while the code is being developed.
  4. Check in early and often.

Screen Shot 2015-10-30 at 1.14.34 PM

It’s really fun seeing how teams of x4 tackle handling a new code base. We also have the challenge that many people come from different parts of the world. So everyone has their own keyboard which can make pairing tricky.

But people overcome. We have lots of discussion about how to test, how to design, and how to tackle all the hairiness that seems to come with coding, even on a simple text based game.

We usually do x3 one hour iterations. With a demo, code review, and discussion at the end of each.

We also track number of commit, number of stories, and code coverage.

After three to four hours people are pretty pumped and exhausted. We then sit back and reflect on what we’ve learned.

It’s really interesting watching people try TDD for the first time. On the surface, when you see other people do it, it looks really easy. But it’s not. At least at first. It’s hard. It goes against everything not only everything you’ve been taught in school. It also not the natural state for people who are just used to hacking.

But once people get it, you can see a light go one. They design their code differently. They find they need less code. And it takes a lot of the guesswork away. They only end up creating what they need. And for many that’s a revelation.

Of course TDD is only one leg of the XP stool. And all the other practices also come into play – particularly pairing. People often ask how do you keep the discipline up. You play a big part of that I say. But your pair can help you too.

Insights, Actions, and Histories

We wrap up the day with Insights, Actions, and Mysteries. Here we ask people what they learned, things they want to do, and things we still wonder about.

Screen Shot 2015-10-30 at 5.09.06 PM

Screen Shot 2015-10-30 at 5.09.15 PM

Screen Shot 2015-10-30 at 5.09.22 PM
Big insights today were the awesomeness of pairing (you can learn a lot working with someone else). Actions were YAGNI – try hard not to over engineer. And mysteries were things like how fast to move when doing TDD vs going really slow and gearing down to really really small tests.

Overall it was a great day. I always learn as much if not more than the attendees. And this was a great group to work with. See you out there!

Screen Shot 2015-10-30 at 1.13.53 PM

Classical vs Mockist testing


I’ve been looking for language to help articulate the difference between two styles of unit testing, and really like how Martin describes it in Mocks Aren’t Stubs.

Mocks use behaviour verification.
Classical uses state verification.


Classicist Mockist
Like working with real objects Prefer working with fake objects
State verification Behavior verification
Use mocks to test collaborations Use mocks all the time
Will hard code collaboration Will mock collaborations
TDD from middle out TDD from the outside in
Use ObjectMothers/Factories for test setup Will mock only what they need for test collaboration
Test tend to be more coarse grained – approaching more integration style tests Tests tend to be very fine grained – may miss integrations
Classists don’t couple tests to implementation Mockists do
Classists don’t like thinking about implementation when writing tests Mockists do
Don’t mind creating query methods to support testing Mockists typically don’t have to
Classists style can encourage Asking Not Telling design Mockists style encourages Tell Don’t Ask

Classical vs Mocking

I generally prefer the classical style to the mocking mostly because I don’t like thinking about implementation when writing a test.

I do however like having the mocking option in my back pocket, for those cases when a system is difficult to test, an object has a nasty setup, or I am refactoring legacy code.

The other piece of advice I recommend (my ex colleague Jonas Claesson turned me onto this) is to keep your calculations separate from your orchestrations.

If you can do this you get the best of both worlds. Calculations can be classically testing. Thin orchestrations mocked. Should keep both set easy to read, and easy to setup.

Here are some more summary notes from Martins article.

Article summary

Regular tests

  • state verification

Mock tests

  • behavior verification

Classical and Mockist Testing

  • Classical style uses real objects
  • Mocking style is to do everything with mocks
  • Classicists will use mocks for creating doubles

Choosing between the differences

  • If collaboration of objects is easy – classic is obvious choice
  • If collaboration between objects is awkward – mock
  • No big choice to be made here
  • Where things get more interesting is when we talk TDD

Driving TDD

  • Mockist will TDD there systems from the outside in
  • They will start with the GUI, describe their collaborations, and progressively marching deeper and deeper into the stack one layer at a time
  • Classical TDD doesn’t offer that kind of guidance
  • Whenever a classist requires something from a collaborator they simply hard code it
  • But classists take a slightly different approach – middle out
  • Here a classist will create an object for whatever domain they need and once they are working then you layer on the UI
  • Doing this you may never need to fake anything
  • Focuses the attention on the domain model, which helps keep the logic from leaking into the UI

Fixture Setup

  • Classic TDD needs to create SUT with all it’s collaborators
  • Mockists need only create SUT and mocks for immediate neighbours
  • Classics will do this using ObjectMothers – reuse whole objects
  • Mockists will say this is more work
  • Classists will say we do this once and reuse – you setup every time

Test Isolation

  • If you introduce a bug mocking, only those tests whose SUT contains the bug will fail
  • In classic, any tests of client objects also fail – ripple effect throughout system
  • Mocksts will say make finding bug harder to find
  • Classists say its usually pretty obvious – little bit more noise


  • Classic tests can be more coarse grained than mockists as they test overall interaction of objects
  • In essence classic xunit tests are not just unit tests, but also mini-integration tests
  • Classists like this
  • Mocksts lose that quality – also run the risk that the mocks are incorrect masking inherent errors
  • Which ever way you go you need both – coarse and fine grained

Coupling Tests to implementation

  • Mockists couple their tests to implementation
  • Classiss don’t care about implementation – only the final state – not how it was arrived
  • With mock testing, writing the test makes you think about the implementation behavior
  • Mocksist like this – classists dont
  • Coupling implementation also interfere with refactoring, since implementaiton changes are more likely to break tests than classic testing

Design style

  • Classist are more comfortable creating query methods to support testing
  • Mockists don’t have to do that as much
  • Mocking also encourages Tell Don’t Ask
  • Classis can encourage Asking Not Telling – though in practice this is easily hadnleld
  • Mocking can also result in more classes (interfaces in various languages)
  • May not be as true now, but i remember many more interfaces and classes being ccreated than was necseary when mocking
  • Also DHH dislikes the designs mocking sometimes creates – mockists like it

Classic or mockist?

  • I have never seen a reason to go all mock
  • Mockists focus solely on implementation details of the SUT – never felt nature to me


  • Don’t combine logic with orchestrations
  • Logic classis – classic mock
  • Orchestrations – simple as possible, no calcs, mockist

Links that help



How to create a xcode command line project with tests xcode 6


Create new command line project


Add new test target by clicking ‘+’ sign in the white area under Targets (Foo)


And then selecting ‘Cocoa Testing Bundle’



Edit your project scheme


Select the ‘Test’ scheme


Hit the ‘+’ sign again, and select your test bundle.



Open logging window (Shift + Command + Y)
Command + U show now run all the tests.



Note: Whenever you add any new project files, you will need to include them in the test target (command line tool apps only).



Jasmine example – selectors and textboxes


Here’s a walk through of a Jasmine example for testing the functionality of the selector arrows show below.

Basically I want to test the jQuery selectors and button functionality. When I click the > button, I want to test that the contents of the ‘Default Trading Group’ box slide over to the ‘Available Trading Groups’ text box.

Step 1: Just get it working.

Before I Jasmine up any of my code, I find it useful just to get the UI working. I am not strong enough in Jasmine or Javascipt yet to TDD stuff like this out. So first pass through, I just hack enough to get the basic controls working, the jQuery selectors setup, and the basic functionality working.

Step 2: Define the model.

With the basic UI going, I was now ready to think about the design.

How did I want this thing to work?

I decided to start with a model (ala MVC). I wanted a model that I could talk to to get the state of any control I wanted on the page. I then wanted this model to be used by a controller (which would actually do this to the page).

Because I already got the page going, my model basically consisted of jQuery selectors to return things off the page. It looked something like this:

function UpdateTradingGroupModel() {

UpdateTradingGroupModel.prototype.defaultTradingGroupId = function() {
    return $('#TextDefaultTradingGroup').attr("defaultTradingGroupId");

UpdateTradingGroupModel.prototype.defaultTradingGroupName = function() {
    return $('#TextDefaultTradingGroup').val();


There’s more, but I think you get the gist of it. Anything I want to know the state of, I just talk to my model and it gives it to me.

And then here’s sampling of the controller:

function UpdateTradingGroupController(theModel) {
    this.model = theModel;

UpdateTradingGroupController.prototype.configureArrowButtons = function() {

    that = this; // important!

        $('#SelectedTradingGroups option:selected').appendTo('#AvailableTradingGroups');

        $('#AvailableTradingGroups option:selected').appendTo('#SelectedTradingGroups');



        if (that.model.defaultTradingGroupIsEmpty()) {
        } else {


UpdateTradingGroupController.prototype.updateTradingGroups = function(model) {

    params = model.saveParameters();

        type: "POST",
        traditional: true,
        url: getControllerAddress('User') + "/@Model.User.Id/UpdateTradingGroups2",
        dataType: 'json',
        success: function(result)

The controller basically sets up the buttons, and actually does the work (like saving any changes done to the page).

And I would add this functionality to the javascript header section of my webpage with something like this:

    var controller = new UpdateTradingGroupController(new UpdateTradingGroupModel());

Alright. With that out of the way, we can now look at some Jasmine tests.

Step 3: Create your Jasmine tests.

The first test I wanted to write verified that when a user clicked the > button, the text in the default trading group got removed and added to the available trading group.

That test looked something like this:

describe("UpdateTradingGroupSpec - When clicking the #removeDefault button", function(){

    var model;
    var controller;

    beforeEach(function() {

        '<input type="text" id="TextDefaultTradingGroup" defaulttradinggroupid="10" value="ctg1">' +
        '<input id="removeDefault" type="button" value=" &gt; ">' +
        '<select id="AvailableTradingGroups" multiple="multiple" name="AvailableTradingGroups"></select>');

        model = new UpdateTradingGroupModel();
        controller = new UpdateTradingGroupController(model);

    it("should slide selected field from TextDefaultTradingGroup to available trading groups", function(){

    it("should be able to determine if TextDefaultTradingGroup is empty", function(){


The first thing to note is this setup fixture line:

        '<input type="text" id="TextDefaultTradingGroup" defaulttradinggroupid="10" value="ctg1">' +
        '<input id="removeDefault" type="button" value=" &gt; ">' +
        '<select id="AvailableTradingGroups" multiple="multiple" name="AvailableTradingGroups"></select>');

It’s beautiful because it enables us to take just the html we need from our page, and set it up however we want it. Just view source on your page, grab what you need and put it in here. You can also ‘loadFixture‘ which will load an entire html file and set that up as your test harnass if you like.

In this case I want the TextDefaultTradingGroup textbox to have an id and value of 10 and ctg1 respectfully (note this doesn’t match what’s on the screen), with a remove button, and select box called AvailableTradingGroups which is empty.

I can then configure the buttons on my page:

        model = new UpdateTradingGroupModel();
        controller = new UpdateTradingGroupController(model);

And always remember to include whatever javascript you use in your app in your jasmine test runner file (else it won’t find your javascript).

Now I am ready to test what happens when someone clicks the remove button.

By simply asking my model I can verify that things start like this:


Then they hit the removeDefault button


And end up like this:


It was a lot of working setting this up, but now that the tracks are laid, we can setup any configuration of buttons and values we want.

You could test every other button combination and configuration of data (we had bugs in there).

Or if you wanted to display a warning dialog in the event someone tries to save a blank default trading group you could write something like this:

describe("UpdateTradingGroupSpec - When clicking the #save button", function(){

    var model;
    var controller;

    beforeEach(function() {

        '<input type="text" id="TextDefaultTradingGroup" defaulttradinggroupid="10" value="ctg1">' +
        '<select id="AvailableTradingGroups" multiple="multiple" name="AvailableTradingGroups"></select>' +
        '<select id="SelectedTradingGroups" multiple="multiple" name="SelectedTradingGroups">' +
            '<option value="11">ctg2</option>' +
            '<option value="12">ctg3</option>' +
        '</select>' +
        '<input type="submit" id="save" value="Save">');

        model = new UpdateTradingGroupModel();
        controller = new UpdateTradingGroupController(model);

        spyOn(Application, 'rootDir').andReturn("/");

     it ("should display a warning dialog if defaultTradingGroupIsEmpty", function() {
        spyOn(model, 'defaultTradingGroupIsEmpty').andReturn(true);
        spyOn(controller, 'showWarningDialog');

    it ("should not display a warning dialog if TextDefaultTradingGroup has data", function() {
        spyOn(model, 'defaultTradingGroupIsEmpty').andReturn(false);
        spyOn(controller, 'showWarningDialog');

    it ("should collect parameters for save", function() {
        expect(model.saveParameters()).toEqual({ defaultTradingGroupId : '10', selectedCompanyTradingGroups : [ '11', '12' ] });


Or if I wanted to verify an ajax call was made to the backend I could write something like this:

    it ("should make an ajax call to do the save", function() {
        spyOn($, 'ajax').andReturn('Successful save.');

I’ve only just scratched the surf of what we can do with Jasmine. But I wanted to get a few examples up to show what it can look like.

One thing I still haven’t got a great grasp out yet is Javascript closures and use of ‘this‘ and ‘that‘.

If you are coming from an enterprise language background like C# or Java, just beware that object state and scope is different in Javascript which is why passing in a model to a controller in Javascript has lines like

UpdateTradingGroupController.prototype.configureArrowButtons = function() {

    that = this; // important!

I am still figuring it out and will write more once I do.

But give Jasmine a chance. It’s helped us reduce the number of bugs in our app and I am looking forward to writing many more.

If you are just starting out checkout Evan Hahn’s Jasmine tutorial. It’s easy peasy.

Best software engineering course ever!


This week I received an email from Tobias Pfeiffer, a student studying software engineering at the Hasso Plattner Institut in Germany, saying … well I will let him say it:

Hello Jonathan,

I am a german student. I am currently making my bachelor degree in IT-Systems Engineering at the Hasso-Plattner-Institute Potsdam.
I just wanted to let you know what a great read your book “the agile samurai” was to me. I started reading it the evening before my exam in Software Engineering II (which is mainly about Scrum) basically because I didn’t know what else to learn and I was looking for some enlightment.
It was so entertaining that it was hard for me to stop reading it and go to bed. Now I read the whole book and really kudos to you, I learned very much despite having finished two full blown courses about agile software practices. The humor and all the pictures are great as they lighten up the book.

The inception deck is great and I wish we would have done one at the beginning of our bachelor project.

You tipped my preference for a future job a bit more into the direction of an agile team member/ project manager/ agile coach.

Also thanks for all the work experiences you shared and all the books you recommended, I’ll make sure to read some of them.

So keep up the good work and have a great week!
Tobias Pfeiffer

After thanking Tobias for his kind words I dug a little deeper into what this second year software engineering course he was taking was about. In a nutshell it looks awesome.

Check out this course outline:

(may need to accept a certificate before viewing)

* 22.10.2010: Course Introduction
* 22.10.2010: Rails Introduction
* 22.10.2010: Exercise
* 29.10.2010: ERP Introduction
* 29.10.2010: CRM Introduction
* 29.10.2010: Collaboration Environment
* 05.11.2010: Lego Exercise
* 05.11.2010: Git Introduction
* 12.11.2010: BDD & Testing (in Rails)

And of course I dig their recommended reading of Pragmatic Programmer Books:

This is the kind of course I would have given my … would have loved to have when I was studying engineering.

You not only learn agile, you get to practice it in a team, building a product using Rails (a framework built for agility) while getting to practice all the best software engineering techniques like TDD, refactoring, continuous integration, and unit testing.

Check out the links above. It’s excellent material and it’s all free.

If you are doing this at your school or university let me know. I would love to give you a shout out as this stuff is important.

Thank you Tobias for the kind words. Would love to meet some day. As someone currently learning Rails myself this is the kind of course I would love to take.

Video of Tobias doing lego exercise (he is the curly haired fellow on the left).

%d bloggers like this: