Launching Spotify on the Sony Playstation

1 Comment

Delivering a new product to 60 million users

Last week was an exciting one for me and my team here at Spotify. We launched the Spotify player on the PS3 and PS4 and brought the Spotify music experience to over 60 million PlayStation users. Here are some notes on how we did it.

How it was run

This was a big project – one that spanned many tribes and cut right across the entire organization. Marketing, sales, licensing, distribution, all of tech. It was also very distributed with Sony and Spotify teams spread across San Francisco, London, Stockholm, Tokyo, and New York.

Spotify doesn’t have project managers. But for this project we did appoint someone to play the role of an overall programme manager. This was key because there were a lot of moving parts, and someone needed to help coordinate.

We did several onsite visits. Many teams came to Stockholm. We went to Tokyo. And there were a lot of early morning/late night calls (not to mention a tonne of email and Slack chats). Every Friday we also had a standing lunch where anyone interested could get an update on where things were at along with the current problems we were facing.

My team was responsible for building the app that would run the Spotify player on the PS3 & PS4. Four developers, three QA, a designer, a Product Owner, and me the coach.

Challenges we faced

Spotify doesn’t normally do projects. Instead we prefer giving teams a goal, and then letting them decide when best to release. We didn’t have that luxury. Sony’s old music application was due to shutdown at end of March, and we needed to be ready with our player to take it’s place. So in that sense, this was a very traditional project with a really hard deadline.

How we overcame

We ran this project like a typical Scrum project, only we quickly resorted to a kind of code and fix mentality that often comes when you realize you have too much to do and not enough time.

In addition to playing coach I also played the role of project tracker (tracking things like our team velocity and keeping an eye on overall deadline). For pretty much the entire project things didn’t look good. We had too many features. Not enough time. Typical project stuff.

Spotify’s approach to situations like these is to leave the team alone and let them sort things out for themselves. If they want extra help it’s there. But really it’s up to the team to solve their own problems.

What was interesting on this project was that the team (mostly the dev’s) didn’t initially want any outside help. Even when it became apparent that there was too much to do.

There was an inflection point in the project were senior management insisted we take someone else on (a fourth developer). This turned out to be a very good choice. Our new addition ended up making an invaluable contribution and we wouldn’t have made it without him.

But this leads to an interesting question for management – how far should you trust the team, and how do you know when to step in and ‘overrule’ their judgement. It’s not an easy call.

Finally, and perhaps most importantly, the one thing we did have going for us is we were the #1 priority project for the entire company. This meant that whatever we wanted – we got. A project this cross cutting and of this size can’t be handled alone. And the entire company was needed to pitch in. So when we needed people, meeting rooms, resources, or just decisions, people on the team just pulled out these ‘magic’ project cards and things got done. It was a fun way to work. But one you must be careful not to abuse.

Finally, this project required a lot of overtime, weekend, and late nights to bring across the finish line. We made it. We delivered. But there was definitely a human cost in terms of time, capital, and technical debt.

The launch

Now for the fun stuff! Here are some pictures of the war room on launch day.

There was a lot of coordination and monitoring that needed to happen on launch day. We created a war room, with each team represented at their own workstation.


We also had sample Sony Playstation on standby, so we could log in to different regions around the world and make sure things were working.


We also had a real cool analytics team that specializes in tracking usage in real time around the world.


The Apollo launch feeling was created by design. Several pictures a phones like these were in full force that day.


And this is what things looked like just before launch.



We kept the war room going for about 48 hours. Once Sony pushed their package out to all the PlayStations, we slowly saw people start to come online around the world, and it was great to see no major hiccups or disruption our backend services couldn’t handle.

To be honest I was amazed. I wasn’t expecting anything to go smoothly. What isn’t easy to see is the amount of work that goes on behind the scenes to make something like this happen. So many things can go wrong.

But somehow it worked. It all came together. It wasn’t Agile. It wasn’t any process, or secret sauce. It’s that same old cliche – it was the people. The talented, passionate group that went the extra mile, stayed the extra night, and worked the extra weekend to make this thing happen.

In Summary

I don’t know if I will ever get to participate in an event of this magnitude ever again. It’s pretty incredible to see this much focused work come together in this timeframe, in a single effort.

But I am thankful to have been a part of it. And I want to thank my team for all their hard work and making it happen.



If you like this newsletter, you can find more here:

Pragmatic Programming in Java 8 with JUnit

Leave a comment

Jeff Langr recently put out a new book Pragmatic Unit Testing in Java 8 with JUnit which I highly recommend. I’ve recently come back to Java (mostly through teaching XP courses) and wanted a refresher on some unit testing best practices. Jeff’s book didn’t disappoint.

Here are some take aways and language for for me from this great gook.

AAA – Arrange – Act – Assert

Exposing Private Data Versus Exposing Private Behavior

Try not to test private methods/data too much.
Occasionally is OK. But if you are constantly exposing deep, gobs of interesting behavior it’s usually a violations of the SRP (Single Responsibility Rule) which states that classes should be small and single-purpose

Your best bet is to extract the interesting private behavior and move to to another class where it can be public behavior.

What to test?
Flex your Right-BICEP.

Right – are the results right?
B – Boundary conditions
I – Inverse Relationships
C – Cross check results with other means
E – Error conditions
P – Performance

Remember Boundary Conditions with CORRECT
Conformance – Does the value conform to the expected format (i.e. email)
Ordering – Is the set of values ordered or unordered as appropriate
Range – Are values within acceptable min/max range.
Reference – Does the code reference anything external not under direct control of code itself
Existence – Does the value exist (is it non-null, nonzero, present in a set, so on).
Cardinality – Are there exactly enough values (0 values, 1 value, 10 values etc, off by one)
Time – Is everything happening in order? At the right time? In time?

Anyways. This is a great book and one you should definitely checkout if you are into unit testing in Java.

Classical vs Mockist testing


I’ve been looking for language to help articulate the difference between two styles of unit testing, and really like how Martin describes it in Mocks Aren’t Stubs.


Classicist Mockist
Like working with real objects Prefer working with fake objects
State verification Behavior verification
Use mocks to test collaborations Use mocks all the time
Will hard code collaboration Will mock collaborations
TDD from middle out TDD from the outside in
Use ObjectMothers/Factories for test setup Will mock only what they need for test collaboration
Test tend to be more coarse grained – approaching more integration style tests Tests tend to be very fine grained – may miss integrations
Classists don’t couple tests to implementation Mockists do
Classists don’t like thinking about implementation when writing tests Mockists do
Don’t mind creating query methods to support testing Mockists typically don’t have to
Classists style can encourage Asking Not Telling design Mockists style encourages Tell Don’t Ask

Classical vs Mocking

I generally prefer the classical style to the mocking mostly because I don’t like thinking about implementation when writing a test.

I do however like having the mocking option in my back pocket, for those cases when a system is difficult to test, an object has a nasty setup, or I am refactoring legacy code.

The other piece of advice I recommend (my ex colleague Jonas Claesson turned me onto this) is to keep your calculations separate from your orchestrations.

If you can do this you get the best of both worlds. Calculations can be classically testing. Thin orchestrations mocked. Should keep both set easy to read, and easy to setup.

Here are some more summary notes from Martins article.

Article summary

Regular tests

  • state verification

Mock tests

  • behavior verification

Classical and Mockist Testing

  • Classical style uses real objects
  • Mocking style is to do everything with mocks
  • Classicists will use mocks for creating doubles

Choosing between the differences

  • If collaboration of objects is easy – classic is obvious choice
  • If collaboration between objects is awkward – mock
  • No big choice to be made here
  • Where things get more interesting is when we talk TDD

Driving TDD

  • Mockist will TDD there systems from the outside in
  • They will start with the GUI, describe their collaborations, and progressively marching deeper and deeper into the stack one layer at a time
  • Classical TDD doesn’t offer that kind of guidance
  • Whenever a classist requires something from a collaborator they simply hard code it
  • But classists take a slightly different approach – middle out
  • Here a classist will create an object for whatever domain they need and once they are working then you layer on the UI
  • Doing this you may never need to fake anything
  • Focuses the attention on the domain model, which helps keep the logic from leaking into the UI

Fixture Setup

  • Classic TDD needs to create SUT with all it’s collaborators
  • Mockists need only create SUT and mocks for immediate neighbours
  • Classics will do this using ObjectMothers – reuse whole objects
  • Mockists will say this is more work
  • Classists will say we do this once and reuse – you setup every time

Test Isolation

  • If you introduce a bug mocking, only those tests whose SUT contains the bug will fail
  • In classic, any tests of client objects also fail – ripple effect throughout system
  • Mocksts will say make finding bug harder to find
  • Classists say its usually pretty obvious – little bit more noise


  • Classic tests can be more coarse grained than mockists as they test overall interaction of objects
  • In essence classic xunit tests are not just unit tests, but also mini-integration tests
  • Classists like this
  • Mocksts lose that quality – also run the risk that the mocks are incorrect masking inherent errors
  • Which ever way you go you need both – coarse and fine grained

Coupling Tests to implementation

  • Mockists couple their tests to implementation
  • Classiss don’t care about implementation – only the final state – not how it was arrived
  • With mock testing, writing the test makes you think about the implementation behavior
  • Mocksist like this – classists dont
  • Coupling implementation also interfere with refactoring, since implementaiton changes are more likely to break tests than classic testing

Design style

  • Classist are more comfortable creating query methods to support testing
  • Mockists don’t have to do that as much
  • Mocking also encourages Tell Don’t Ask
  • Classis can encourage Asking Not Telling – though in practice this is easily hadnleld
  • Mocking can also result in more classes (interfaces in various languages)
  • May not be as true now, but i remember many more interfaces and classes being ccreated than was necseary when mocking
  • Also DHH dislikes the designs mocking sometimes creates – mockists like it

Classic or mockist?

  • I have never seen a reason to go all mock
  • Mockists focus solely on implementation details of the SUT – never felt nature to me


  • Don’t combine logic with orchestrations
  • Logic classis – classic mock
  • Orchestrations – simple as possible, no calcs, mockist

Links that help

Turn that ship around

Leave a comment

My friend Jason Yip shared with me the work book for Turn this Ship Around which comes as a highly recommended book on leadership.

He also had a cool deck of cards I wanted to save to later. Here are some images for when I am ready to do so.






The Apple Watch in New York

Leave a comment

So this weekend I have the pleasure of seeing the Apple watch for the first time in two distinct iconic locations – Grand Central Station, and the flagship 5th Avenue store.

Grand Central Station


First off, when I first walked into Grand Central Station (or Juncture here as they call it) I didn’t even know that hand an Apple store tucked away in the back.

Not thinking much, I walked over, and loe and behold – watches!


This was the first day the watches had been on display to the public and needless to say people were excited.


The watches come in all different shapes and sizes (completely different marketing from computers where there are very few models). And there was a watch for just about everyone.



Speaking to a salesman I asked if the watches were for sale. He said no. You could only pre-order. And they were sold out till November! Woah.


I don’t know if that’s true of not, but still. There seems to be a pretty healthy demand for these times.

I got to play with some of the watches on display. I have to say I am impressed. They feel good. The user interface works well. You can speak with Siri and it works. So I could see these things doing well.

I walked in having no desire for owning an Apple watch. But after playing with one for a while. I was beginning to think these things were pretty cool. Maybe I was just getting swept up in the moment.


Anyways. After that it was off to me real destinatation. The iconic 5th avenue store.

5th Avenue Store

This is the store that Walter Isaccson talked about in the Steve Jobs biography. Clear cube on top. Spiral stair case going to the bottom. It is simply beautiful.


And over course they had the watches on display too. So things were getting busy in there.


I just stood back and marvelled at the place. It was truly a beautiful store. It’s located right on 5th ave. Right beside Central Park. And is definitely iconic not only it’s exterior design, but on that day what it was selling inside too.


I don’t know whats going to happen to the watch long term. I suspect Apple will iterate on this and it will improve over time.

But I bet you this store won’t be quiet until after Christmas. And I suspect more than one person will be finding a new Apple Watch.




And I just love that staircase!

How to add command line interface to xcode project

Leave a comment

This is how you can add command line interface to youx XCode command line app.

Add a new Objective-C file



Rename the Objective-C file extension to .mm



Add a new header file


Copy in the following code to the header file


#import <Foundation/Foundation.h>

@interface IOHelper : NSObject

- (NSString *)readLine;

- (void)outputLine:(NSString *)line;



Copy in the following code to the .mm file

#import "IOHelper.h"
#include <string>
#include <iostream>

@implementation IOHelper

- (NSString *)readLine
    std::string input;

    getline(std::cin, input);

    return [NSString stringWithCString:input.c_str()
                              encoding:[NSString defaultCStringEncoding]];

- (void)outputLine:(NSString *)line
    std::string outputLine([line cStringUsingEncoding:[NSString defaultCStringEncoding]]);

    std::cout << outputLine << std::endl;



Copy in the following code to the main.m file


#import <Foundation/Foundation.h>
#import "IOHelper.h"

int main(int argc, const char * argv[]) {
    @autoreleasepool {
        IOHelper *io = [IOHelper new];
        [io outputLine:@"Hi! Welcome to XpStuff 2014-06-12."];
        [io outputLine:@"Who are you?"];
        NSString *name = [io readLine];
        [io outputLine:[NSString stringWithFormat:@"OK %@, go build a game!", name]];
    return 0;

Run (Command + R). Should see the following in the console (Command + Shift + Y).


Reinventing Organizations – Frédéric Laloux



The following are notes taken from Frédéric Laloux’s excellent talk on his new book Reinventing Organizations – The emergence of a new management paradigm.

It’s basically a summary of the next way in which we may all one day be working at companies. Watch this if you are into organizational theory, liked Dan Pink’s book Drive, and are looking for examples of how some incredible companies today are re-organizing themselves around:

1. Self management.
2. Wholeness.
3. Evolutionary purpose.

Here’s the video (1h:42min). Notes below.

There is something broken with the way we manage or companies today.
It feels exhausted. Disengaged employees.
Show up with bodies, but not hearts.
Not just bottom. But leaders at the top too.
They are tired of the rat race.
Tired of the endless meetings, politics, infighting, bureaucracy.
Another tedious budget cycle, targets that need to be hit.
There should be something more.
There is no meaning.
For many corporations this isn’t an easy question to answer.
Nurse and doctors, teachers are leaving their professions in droves.
Something new is about to be emerge.

Big leaps in human thinking (14:30)

Each stage of human organizational evolution resulted in a new management paradigm.
A new way to run organizations.


Magic/Tribal level

One tribe agains the other.
Some modern groups today still do this.
Street gangs, mafias, mecenary armies
Boss needs to constantly inspire fear.


World of rules.
Institutionalized religion – god given rules.
Highly stratified caste systems.
Invented Catholic Church, armies, goverment agencies, public school systems

Key breakthroughs
1. Formal hierarchy
2. Replicable prcocesses

By creating the org chart, for the first time the priests can be priest are aren’t interested in stabbing the boss in the back.

Means can have large levels of organizations with lots of levels of hierarchy.

And by being able to to long term planning, these organizations were able to do things not previously possible with the other groups.
Long term planning.


World of enlightenment.
Scientific enquiry.
Constant innovation, optimization, to gain profit market share.
Most multi-national organizations operate from here.

Key breakthroughts
1. Innovation
2. Accountability
3. Meritocracy

All the things we enjoy today are because companies have innovated.
R&D departments. Marketing deparments.

In previous organizations bosses gave orders – workers followed.
Here for the first time bosses set targets, people figure out how to achieve.

Management by objective. 360 feedbacks. Budgets. Targets. Annual appraisals.

In all previous organizations Pope was from nobel family. Priests from peasantry.
The best people can rise. Huge liberation.
Organizations as machines. People are cogs.


Now starting to look at the soft aspects.
Cultural driven organizations.
Culture eats strategy for breakfast. – Peter Drucker
Green organizations – Starbucks, Zappos, Ben & Jerry’s
If people are happy – everything else will be alright.
Company values are meaningful here.
Fanatical about empowerment.
Pusing decisions to the lowest level.
Moving away from shareholder model, to stakeholder model.
These organizations, like predecessors, dramatically outperform previous levels.
We are a family.

Wolf packs, armies, machines, and now families.
So what’s next?

Three breakthroughs for next level

1. Self-management.
2. Wholeness.
3. Evolutionary purpose.

Self-management (37:00)

These companies are able to operate at large scale (10,000 employees) without the pyramid.
Here is a different way to think about hierarchy in organizations that is more truthful.
Hiearchy works OK in environments with low complexity.
But as some as you have complex thinking, hierarchy is out of it’s depth.
Hiearchy pushes all the complexity up to the top.
And there is only so much complexity a few people at the top can handle.
The global economy has no boss.
North Korea and Cuba are the only two countries in the world that operate from a pyramid.
We laugh, but this is exactly how we choose to operate of companies.
In this new way of thinking you have lots of structure and coordinating systems – but you don’t have a boss.
Morning traffic. Self regulating system.
Single cell. Our brain. Forests. Many examples in nature.
Having bosses isn’t natural. Nature has not boss. It regulates itself.
Not how complex systems work.
As world becomes more complex, will have to shift how we run organizations to these principles.
Our pyramids can not cope.
However, if you replace a pyramid, you need to reinvent everything.
Decision making, organizational structure, information flows, meetings, conflict management.
All this needs to be recreated and figured out.

Decision making

Traditionally two ways to make decisions.
Hierarchical decision making or consensus.
Here the boss decides or the group decides.
Boss isn’t always right.
Group is long and exhausting – make a decision already!
There is a third way – Advice process.

Advice Process – any person can make any decision (including spending company money) under two conditions.

1. They must have sought advice from people with expertise.
2. They mush have sought advice from people how will be impacted by the decision.

Don’t need to integrate my decision into watered down consensus.
This works because it’s a process of collective intelligence.
But it stays an individual decision. So if I feel strongly about something I can just make it happen. No one can stop me. Every is empowered to do whatever they feel is necessary.
Imagine the kind of energy that liberates.

And it leads to interesting dynamics. Because one of the few ways people can get kicked out of these organizations is if they don’t respect their peers advice. And instead just go and randomly make decisions. You are putting whole system at risk.

Who makes how much money

Morning Star (a tomato paste distribution company in the US).
Once a year you write a letter in which you state I grant myself a raise of X% and then you state all the reasons you feel this is justified. 360 degree f/b.
They each plant, elects a committee, which puts all these letters beside each other, and the only thing this committee does is give advice.
Committee gives advice.
All information is public.
And then you decide.
Either you earn the raise or you don’t.
People are incredibly good at choosing their salaries.
Forces everyone to grown up.
People don’t talk salary here because it isn’t an issue.
If you don’t like your salary raise it – and see what happens.
All the strategizing, haggling and complaining, falls away.
Everyone is just an adult.


Most organizations there is an expectation that we show up in a specific way.
A professional sefl.
Most orgs push us to wear a mask.
It’s become acceptable to show up and work, with ego, and to fight for what we want.
But showing up to work, and asking questions with deeper meaning – isn’t.

55:00 Imagine for example the advertizing executive who shows up Monday morning and says:

Hey guys. I want have a really important disucssion with you guys. I sometimes wonder what we are busy with. We are creating all these needs for people, for goods that they don’t really need, for goods that get produced in China, pollute the world, get shipped over, used once, get thrown away, trash the planet, and all those people who can’t afford them become unhappy. What we are really dealing with.

The person that calls in the meeting proably won’t have a very long career.

Same for a doctor in a hospital. We’ve changed hospitals into these kind of factories and we’ve forget about what it really truly means to care. The relationship with the client.

What these companies have discoved is that when this is the case, people only show up to work with 1/16 of their abilities. Their hearts, minds, and passions that same for other things. They don’t bring these to work.

But those companies that have found ways to align these, are recieving peoples best. And outperforming all others. They have create very deliberate practices to open up this window and tap that energy. Full glory of how humanity of who we are. People here brim with energy because they can be who they truly are.

It’s about creating a safe space. When we show up we are vulnerable. So they work hard to create a safe place.

Meetings without egos (1h:00)

At every meeting they have these hand symbols.
When ever he or she feels that some is speaking from their ego, or trying to win an argument for the sake of winning an argument, or serving themselves and their career, or group, someone chimes two bells.

The rule is while the bell rings everyone is supposed to be silent for a minute and ask themselves who they are trying to serve. Am I serving me? Or am I hear in service of something greater.

All these organizations have these meeting practices because meetings tend to be these places where egos tend to come out. Meetings without egos.

This is only possible when people have been trainined in active listening, non-violent communication. A whole common knowledge around this.

There is a public school in Berlin that is entire self managing.
But what they’ve really nailed in how to make kids truly be themselves.

One pracice they do is they gather every Friday afternoon, for 45min, they start by singing a song (to get everyone in tune). And then they have this practice of open microphone, and the only rule is you walk up to the microphone to either thank someone or make a compliment.

What happens is people tell mini-stories. But actually what they are reviling is things about themselves. Adolesents how go out there and thank their classmates for helping them in all sorts of things.

You have kids daring to be authentic and vulnerable infront of 500 people. This school has no violence problems. Children are just so passionate to learn because they are accepted for how they are. No masks.

Evolutionary Purpose (1:06)

Most companies prioritize making money above most other things.
Most people are cynical about their mission statements.
Most companies hide their competitive advantages from competitors.
No Berjekalen. They did the opposite.
He wrote a book explaining in exact detail how he and his company (who have managed to secure 80% of the market) to each of his compeitors.
Because for him, the purpose is not his organization.
It’s greater than that.
Market share doesn’t matter.


In most traditional organizations we believe it is the role of the leader to determine the vision and the stragey. And then some execution plans to get there.

That way of thinking makes sense if you believe organizations are static, inanimte object. Like a machine. You need to program the machine. It’s a ship. You need a captain.

But organizations are like living beings.
The organization has a sense of direction.
It’s own creative spark about something it wants to manifest.
And our role as leaders is to listen to where does this organization naturally want to do.
And align with that.

Noe of these companies have a strategy.
They have a very clear intent and purpose.
But everything flows from that.
Most don’t have budgets or targets.
Anything they put in the ground, in terms of a plan, is a distraction from reality.

When we try to predict the future, we stop listening to reality.
When we try so hard to stick to the plan we stop listening.
Instead we should have a very clear intent on where we should do.
And then we should listen intently on how to get there and constantly adjust.

1:15 Great story of how Bjorkshallen got into prevention, shared it with the CEO, and asked him if he thought they should change direction. He said I don’t know, let’s share with everyone else and see if there is energy to support it. Some ideas/innovations flourish. Some don’t. The company as a whole decides. If someone wants to drive and champion they will.

Not playing God. Listening to what is happening.


It might not be crazy to think there is a new way to run organizations.
They are built on these three breakthroughs.

1. Self management – can operate without hierarchy
2. Wholeness – bringing our wholeselves to work
3. Purpose – will become central. More about clear intention

I was personally blown away by this talk. I think many of us in the Agile community feel this already. And I am excited to see how quickly these new ideas spread and catch on. I will take years. But I believe the change has already begun.

Cheers – Jonathan

Older Entries


Get every new post delivered to your Inbox.

Join 332 other followers

%d bloggers like this: