Launching Spotify on the Sony Playstation

2 Comments

Delivering a new product to 60 million users

Last week was an exciting one for me and my team here at Spotify. We launched the Spotify player on the PS3 and PS4 and brought the Spotify music experience to over 60 million PlayStation users. Here are some notes on how we did it.

How it was run

This was a big project – one that spanned many tribes and cut right across the entire organization. Marketing, sales, licensing, distribution, all of tech. It was also very distributed with Sony and Spotify teams spread across San Francisco, London, Stockholm, Tokyo, and New York.

Spotify doesn’t have project managers. But for this project we did appoint someone to play the role of an overall programme manager. This was key because there were a lot of moving parts, and someone needed to help coordinate.

We did several onsite visits. Many teams came to Stockholm. We went to Tokyo. And there were a lot of early morning/late night calls (not to mention a tonne of email and Slack chats). Every Friday we also had a standing lunch where anyone interested could get an update on where things were at along with the current problems we were facing.

My team was responsible for building the app that would run the Spotify player on the PS3 & PS4. Four developers, three QA, a designer, a Product Owner, and me the coach.

Challenges we faced

Spotify doesn’t normally do projects. Instead we prefer giving teams a goal, and then letting them decide when best to release. We didn’t have that luxury. Sony’s old music application was due to shutdown at end of March, and we needed to be ready with our player to take it’s place. So in that sense, this was a very traditional project with a really hard deadline.

How we overcame

We ran this project like a typical Scrum project, only we quickly resorted to a kind of code and fix mentality that often comes when you realize you have too much to do and not enough time.

In addition to playing coach I also played the role of project tracker (tracking things like our team velocity and keeping an eye on overall deadline). For pretty much the entire project things didn’t look good. We had too many features. Not enough time. Typical project stuff.

Spotify’s approach to situations like these is to leave the team alone and let them sort things out for themselves. If they want extra help it’s there. But really it’s up to the team to solve their own problems.

What was interesting on this project was that the team (mostly the dev’s) didn’t initially want any outside help. Even when it became apparent that there was too much to do.

There was an inflection point in the project were senior management insisted we take someone else on (a fourth developer). This turned out to be a very good choice. Our new addition ended up making an invaluable contribution and we wouldn’t have made it without him.

But this leads to an interesting question for management – how far should you trust the team, and how do you know when to step in and ‘overrule’ their judgement. It’s not an easy call.

Finally, and perhaps most importantly, the one thing we did have going for us is we were the #1 priority project for the entire company. This meant that whatever we wanted – we got. A project this cross cutting and of this size can’t be handled alone. And the entire company was needed to pitch in. So when we needed people, meeting rooms, resources, or just decisions, people on the team just pulled out these ‘magic’ project cards and things got done. It was a fun way to work. But one you must be careful not to abuse.

Finally, this project required a lot of overtime, weekend, and late nights to bring across the finish line. We made it. We delivered. But there was definitely a human cost in terms of time, capital, and technical debt.

The launch

Now for the fun stuff! Here are some pictures of the war room on launch day.

There was a lot of coordination and monitoring that needed to happen on launch day. We created a war room, with each team represented at their own workstation.

spotify-sony-playstation-1

We also had sample Sony Playstation on standby, so we could log in to different regions around the world and make sure things were working.

spotify-sony-playstation-2

We also had a real cool analytics team that specializes in tracking usage in real time around the world.

spotify-sony-playstation-3

The Apollo launch feeling was created by design. Several pictures a phones like these were in full force that day.

spotify-sony-playstation-4

And this is what things looked like just before launch.

spotify-sony-playstation-7

spotify-sony-playstation-8

We kept the war room going for about 48 hours. Once Sony pushed their package out to all the PlayStations, we slowly saw people start to come online around the world, and it was great to see no major hiccups or disruption our backend services couldn’t handle.

To be honest I was amazed. I wasn’t expecting anything to go smoothly. What isn’t easy to see is the amount of work that goes on behind the scenes to make something like this happen. So many things can go wrong.

But somehow it worked. It all came together. It wasn’t Agile. It wasn’t any process, or secret sauce. It’s that same old cliche – it was the people. The talented, passionate group that went the extra mile, stayed the extra night, and worked the extra weekend to make this thing happen.

In Summary

I don’t know if I will ever get to participate in an event of this magnitude ever again. It’s pretty incredible to see this much focused work come together in this timeframe, in a single effort.

But I am thankful to have been a part of it. And I want to thank my team for all their hard work and making it happen.

umbrella-team

Newsletter

If you like this newsletter, you can find more here:

http://www.agilenutshell.com/newsletter

Turn that ship around

Leave a comment

My friend Jason Yip shared with me the work book for Turn this Ship Around which comes as a highly recommended book on leadership.

He also had a cool deck of cards I wanted to save to later. Here are some images for when I am ready to do so.

turn-that-ship-around-1
turn-that-ship-around-2

turn-that-ship-around-3

turn-that-ship-around-4

turn-that-ship-around-5

turn-that-ship-around-6

XP Tech Practices Workshop at Spotify

1 Comment

Last week, a group of us did an XP Tech Practices workshop here in Stockholm. We covered Test-Driven Development, Refactoring, Unit Testing, Pair Programming, and a host of other Old school XP practices. Here are some notes and a write up about the course.

xp1

What is XP

XP (Extreme Programming) was the first really popular Agile method. Not only did it introduce Agile planning concepts such as user stories, adaptive planning, and iterative development. It helped popularize engineering practices such as unit testing, refactoring, test-driven development, and continuous integration.

It isn’t as widely practiced today as it was in the early 2000’s (Scrum has largely taken over). But it has many wonderful practices and ideas that deserve repeating and are still very much relevant for software teams today.

This workshop aimed at revitalizing some of software these practices, showing participants who they were originally meant to be performed, and then discussing how they can be introduced into their workplace.

How do you feel about your work?

We start off the course by asking participants how they feel about their work. Specifically around quality and design. We do this to start a conversation around what quality is, how we know when we have good design, and what obstacles prevent us from reaching them in our software.

Build something!

We then break the ice by asking participants to code something up (like a calculator). We do this to get the juices flowing, engage the participants, and just get them hacking. This helps us see where they are coming from, their familiarity with the programming language, and how comfortable they are writing code.

We don’t judge or do anything with the code that we produce here. We simply park it, and revisit it later. We then introduce XP.

XP Intro

XP can best be described as the software methodology that turns the knobs up to 11.

If code reviews are good, we will review code all the time (pair programming).
If testing is good, we will test continuously (acceptance tests, unit tests, TDD, demos).
If design is good, we will design continuously, every minute and every day (refactoring).
If simplicity is good, we vow to always leave the system in its simplest state.
If integration is good, we will integration continuously (continuous integration).
And if iterative development is good, we will iterate from day one, continuously getting feedback on our product, plan, and ability to please our customer.

XP Demo

Once we’ve covered the spirit and intent of XP, we demo the practices. We pair program up a sample problem, applying TDD, refactoring, unit testing, and pair programming in an extreme way, just to give the participants a sense of how all the practices work.

We then revisit the exercise they did in the ice breaker, and try it again, this time using the XP practices, specifically writing the tests first.

Build a game!

We then unleash them on their own codebase – specifically a game. They take what we’ve covered, and apply the practices in as an extreme a way as they can, and we build a fun interactive game over the course of several iterations.

XPisms

Over the span of the course we look for opportunities to introduce XPisms. Things like:

YAGNI – You aint gonna need it. A reminder to keep things simple, and not add excess functionality / code unless absolutely required (a good test is whether you can write a failing test first before adding the new code).

xp2

Production Readiness – A system spends a lot more time in production than in development. So why don’t we start treating it that way, and write code as if we were in production from day one.

Doing the simplest thing that could possibly work

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. – C. A. R. Hoare

xp3

Test everything that could possibly break – We can’t test everything. But we know we can’t test nothing. So we start by testing everything we think could break. And evolving our suites from there.

Dials to 11 – XP’s take on hard problems is to take them head on. Taking this approach takes many of the traditionally hard problems on software projects, and breaks them down into much smaller, easier to digest pieces.

xp5

Courage – Have the courage to write that failing unit test. Have the courage to say no to the hack on your finger tips. Have the courage to say no to the unrealistic schedule, and help another team out when they really need it.

Why we feel this is important

It may seem that XP is all about engineering. It’s not. It’s just as much about culture. But the culture XP wants the team to operate under doesn’t happen if certain practices aren’t in place. This course is about explaining the practices, showing how they work, and then helping teams adopt them their work place.

A special thanks to Henrik Kniberg for the initial course outline and for his support in making this course happen.

Quotes

“I was supposed to have a Hackday today, but this is WAY better”

“I found the course to be a lot of fun, and learned a lot about TDD/XP in the process. The exercises, particularly the four-part exercise to build a game for a customer, were a very good balance of difficulty and fun, and allowed me to learn at a good pace without being bored by trival tasks or overwhelmed by difficult ones.

In fact, the game task actually, perhaps accidentally, was almost tricking me into learning stuff. You know you give a kid a toy that’s actually teaching them maths without them knowing because they’re having fun with the flashy lights? It was kinda like that.

★★★★★ Five stars, genuinely one of the best in-house workshops I’ve attended.”

“Got a good amount of new perspective on XP and TDD. Got into the habit of writing unit tests during instead of after development. And it was a lot of fun!”

A constructive question

1 Comment

Yesterday I attended a session put on the People and Organizational Change group here at Spotify, and I liked the question we used to start off the workshop.

This task/workshop/activity/tool/doc would have a positive impact on the experience of joining Spotify.

I like the question because instead of just griping and complaining, it forces the attendees to switch from problem mode to solution mode.

Great question. Going to use this sometime in the future.

Greetings from the Iteration – Retrospective

Leave a comment

Tried something new style of retrospective this week. It’s called Greetings from the Iteration.

greetings-from-the-iteration

Basic idea is you write yourself a postcard, talking about and visualizing how last iteration went.You can create pictures, write words – whatever you want.

For this retro, we started with a blank wall, containing three areas: sunny for great iteration, rainy for bad, and an island in between for OK.

greetings-from-the-iteration-start

Then team members spent about 10 minutes creating their post cards

spotify-retro

After 10 minutes, team members standup, share their experiences with the team, and then stick it on the board/wall indicating how they feel (sunny parts for great sprint, rainy parts for not so great).

greetings-from-the-iteration-finished-product

It’s a neat exercise. What I like about it is it gives the team members a blank canvas to express how they feel. It’s also something different and new to get the juices flowing and lead to some interesting discussion.

I got the idea from Filipe Albero Pomar via Corinna Baldauf’s website

http://plans-for-retrospectives.com/?id=85

A wonderful website containing over 91 different ideas and activities for retros with you teams (thank you Corinna). So if you are looking to try something different with your teams, head on over, check it out, and see if you can’t mix things up a bit at your next retro.

Top 100 Agile Blogs

1 Comment

Looking for a distilled list of the Top 100 Agile blogs? Checkout Luis Goncalves Top 100.

http://lmsgoncalves.com/2014/10/26/100-top-agile-blogs/

Chances are good you’ll find something new in there you haven’t seen before.

Thanks Louis!

XP Customer and Developer Bill of Rights

4 Comments

To help customers and the development team work together, XP came up with a customer and developer bill of rights.

Customer Bill of Rights

  • You have the right to an overall plan, to know what can be accomplished when and at what cost.
  • You have the right to get the most possible value out of every programming week.
  • You have the right to see progress in a running system, proven to work by passing repeatable tests that you specify.
  • You have the right to change your mind, to substitute functionality, and to change priorities without paying exorbitant costs.
  • You have the right to be informed of schedule changes, in time to choose how to reduce the scope to restore the original date.
  • You can cancel at any time and be left with a useful working system reflecting investment to date.

Programmer Bill of Rights

  • You have the right to know what is needed, with clear declarations of priority.
  • You have the right to produce quality work at all times.
  • You have the right to ask for and receive help from peers, managers, and customers.
  • You have the right to make and update your own estimates.
  • You have the right to accept your responsibilities instead of having them assigned to you.

The bill of rights were a powerful means of setting expectations with folks, and letting them know who was responsible for what.

For example, XP made it pretty clear that developers shouldn’t be making calls about business priorities. That was the domain of customer.

In exchange, customers would stop telling engineers how long it would take to do there jobs. Or anything technical in nature (like how to build the system). That was the domain of the developers.

You may not have this problem today, but if you do, pull these out and put them infront of your team to ensure everyone knows what role they are planning, and what they should be entitled to on your project.

And if these rights don’t work for you. Create your own. Just don’t let the issue fester. Get everyone together, get into a room, and make it clear who gets to own what.

Making this clear could be the most important thing you do for your project.

Older Entries

%d bloggers like this: