Copy with care

2 Comments

I understand the desire to copy others. I do it all the time. I read an article about Facebook having no QA department, or see first hand how Spotify delivers projects with no Project Managers, and I can’t help but wonder how if they can do it, why can’t we do it here?

There’s just one problem. Copying others is never that simple.

When we see or admire what one company is doing, we are only seeing it through the lens of the end result. And not the underlying foundations that were setup to get them there.

Facebook can get away with not having a QA department because of the huge level of accountability they place on their engineers. Not to mention the insane level of talent.

Spotify doesn’t have formal Project Managers. But it empowers, and gives a level of autonomy and independence to it’s teams, so that teams are inspired to manage themselves.

For fun, this is the conversation I imagine trying to have with the Oil and Gas executive back home.

“OK. For next year we are going to try something new. No annual budgets. When Toronto asks us how much we are plan on spending, we will simply say we are going to spend whatever the cost is for us to work here this year. Our burn. That’s our budget. And in exchange for that they will empower us to do whatever it takes to meet the goals we have jointly agreed upon and laid out before us.”

Needless to say this wouldn’t fly.

But that’s my point. It wouldn’t fly because most companies won’t give the trust and accountability necessary to get the level of commitment and passion they see and read about in others. For their employees it will always be a job.

So am I saying we should give up, stop looking to others to inspiration, and just accept that we work for companies that will never give us the levels of autonomy, accountability, and independence we crave.

Hell no. Just remember. Not everyone wants this level of accountability. For many people work IS just a job. And trust is something that needs to be earned.

So keep reading. Keep studying. And keep looking to others. Just understand. What works at Apple, or Google, won’t necessarily work out of the gate at yours. Don’t get discouraged, keep trying to make where you work a better place, because at the end of the day, what other choice is there.

Deep Inside Facebook

Leave a comment

In this excellent Stanford ecorner episode, Director of Engineering Jocelyn Goldfein takes us on a trip inside the innovative culture of Facebook. In this illuminating conversation with STVP Executive Director Tina Seelig, Goldfein explains why code wins arguments, employees must have the right to take risks, and how Facebook strives to remain a hungry, yet humble, company.

The following is a semi-transcript of the parts of the interview I found most interesting.

facebook

Culture of Facebook

Interviewer: Can you paint a picture of the culture of Facebook?

You would experience unfinished ceilings.
Concrete floors.
Desks everywhere – no offices. Not even Mark has an office.
And writing all over the walls.
And posters.

And you would see company slogans like:

– This journey is 1% finished.
– Move fast and break things.
– Fail harder

And facebook was born out of disruption and out of trying things and seeing what happened. And trying again and trying harder. And never being daunted by failure. Doesn’t mean we set out to fail. It means we are not afraid of it. And we are willing to keep taking risks.

And the entire environment is meant to keep you from feeling complacent, or comfortable or we’ve won. We never want to feel like we have won. We are pretty hungry. And someone could come and eat our lunch tomorrow. Because somebody could. And we never want to take it for granted. It’s the most humble successful company I have ever known. It may sound strange to say that facebook is a humble company. But it really is.

We don’t take our success for granted. We think our users are choosing to be there and could just as easily choose not to be there if we don’t deliver a great service.

The Facebook workspace

Interviewer: What sort of things did people think about in creating the facebook workspace?

It’s very deliberate. It’s absolutely deliberate. You can not just think of the culture you want and then create it. Culture arises from so many small things. A professor of mine once said:

Culture is the behaviors you reward and punish.

At the end of the day, people look around, and mimic the behaviors that reward them, and avoid the ones that punish.

But you’ve also got to try and show people the behaviors you want. And the space is one of those things that sneaks under your radar as not being important. But it truly is.

When you walk on those concrete floors you know we are not finished.
We are not luxurious.
We are not taking it for granted.
We’re scrappy.

The openspace is another huge one. You know that if you program you need focused attention. You need flow time. And you know that even small interruptions takes a long time to get back into the stream of things.

And so the idea of having programmers set out in open space at open desks with desks all around with walking and conversation and foot traffic … that’s controversial. For many years it was the ‘gold standard’ in silicon valley to have offices. Engineers were housed in offices.

When we took over the campuses from Sun we told the contracts to start knocking down every wall that wasn’t structural. Got as big an empty space in the office as we could and we would fill it in from there.

And why? Why was it so important to us that we would even consider sacrificing our productivity. Which is what we are doing having everyone out in these open spaces.

It’s because one of the key values of facebook is to be open.

It’s what the product is for. It’s fundamental to our DNA as a culture too. We expect every individual to be informed and in the know about what is going on empowers you to make good decisions. And so sort of set the expectation that everything is out in the open. Everybody is plugged in and aware of what is going on.

And then we put headphones in the vending machines and try to create private spaces that way. Give everybody a laptop and have pretty loose rules about working from home policy. So we do everything we can to mitigate the productivity impact of that openness. But every time we have to choose we choose openness.

Onboarding at Facebook

We call it boot camp.
It is a six week on boarding program.
And everybody goes into it.

As the VP of Engineering at VMWare, I had not written code for 7 years and I went to facebook and they said:

Here’s your desk.
Here’s your laptop.
Here’s your unix account.
And here’s five bugs that are assigned to you to fix.

And you spend your first 6 weeks at Facebook fixing bugs and implementing small features all over the site. And attending lectures and you have a boot camp mentor who is a full time software engineer who’s job is to help you get code reviews, to help you figure out what tasks to work on. And at the end of the six weeks they will help you find a team.

I give facebooks introduction to culture for bootcampers. And one of the many great things boot camp does is build empathy. Before you identify with a particular team you fix code. Even if you want in determined to do back end services, we will make you fix some UI bugs. We will make you fix an issue on mobile. No matter what you are going to do we are going to at least give you the tools to inspect, investigate, to know, about what is going on in other parts of the world.

It really sends the message we want to send which is – engineers are connected to one another. The facebook employees is just another subset of the social graph.

And so in boot camp you hook up your social graph. With your first setup of co-workers who then spread out to the 4 corners of engineering and give you connections in every part of the company.

But you know a little bit about something of everything.

And that’s an incredible asset for a new hire.

Interviewing at Facebook

Interviewer: So what are you looking for when you interview at Facebook? I know the standards are incredibly high. How important are technical skills as well as creativity and being about to work on teams.

The number one thing we are looking for in hiring a software engineer is ability to write code. And the ability to reason, be analytical, find mistakes and fix them quickly, and to analyze the running time so later on you can demonstrate you have the potential to solve hard systems issues.

If you are a new grad, that is the primary thing.

We also want to make sure you are not a jerk. Want people at facebook who are nice. ut we want people how have enthusiasm. Have fire in their belly. Aren’t going to take no for an answer. People who are not afraid. Going to attack making software with gusto.

The reason you see so much writing on the walls at facebook is we want people to write and share things the see – good or bad.

Pop open the hood.
The cement is never dry.
You can mess with it.
You can change it.

One thing that made engineers really successful at facebook is they had really good intuition for what would be a really good feature.

The difference between good intuition and just working iteratively is that the person with good intuition will get there in three hops while the person without will take x10 tries. So we started interviewing specifically for intuition. Its something that distinguishes good from great software engineers.

Mark Zuckerberg

Interviewer: How involved is Mark in decision very granular decision making?

Mark will definitely give you advice about pixels. Mark has organized the company so he can spend the build of his time on product and product strategy. He has basically set up his calendar so that for each day of the week theres a theme, and on day the theme might be mobile, platform, and then there will be a block of 4 hours and teams just rotate through and present stuff to him and talk through it with him and it’s amazing because he is probably the most gifted product thinker in the company, and maybe in the valley, maybe in the world (that’s probably stretching it) but definitely in our space.

He so like having a half an hour of his time is just amazing. You do have to parse the advice he is giving you and know whether he has is CEO hat on, designer hat, or Product Manager hat because he can operate at some many levels of abstraction.

So, yes. He will be very involved and he has structured his time to be involved. But he can only pay attention to so many things at a time to. So if he is not paying attention, he expects you to run forward while he is not looking.

Advice for grads

(31:31)

Interviewer: Imagine you are flashing back in time and you are a student back here at Stanford again. What advice do you wish someone had given you?

Ah man. That’s a hard question… Um…

I think that umm… you know you hear this all the time but be brave. Like you really don’t have any bad choices you. You can’t go wrong. Now is the time in your life to take a lot of risks.

I think that ahh.. I would have said… don’t worry about what you are going to be when you grow up. Just be. It will come and find you.

I feel like, you know, when I was in highschool and college so many people were like follow your passion, follow your dream and I was like: “How do I find my passion. How do I find my dream.”

No one had particularly good advice for what to do. To figure out what you are passionate about. So I would say:

It’s OK not to know.

Just keep trying things and you will find stuff that you are passionate about and excited about.

I would have told myself not be afraid to code for a living. I was afraid to code. I was afraid that other people would be better at it than me. I was afraid that I would be buried in detail and not get to be strategic. And I learned over time that writing code is one of the most strategic things you can do. And every time I meet a college grad who wants to skip being a software engineer and move straight into Product Management I am like uh… you learn so much from writing code that the devil is truly in the details. Embrace it. Relish it. You may not end up doing it forever but every minute you spend coding will make you 1000x better at being a manager or product manager or anything else. Even if you decide you want to do something else, your time writing code will make you better for it. If you want any career in software. Any career.

This was a great interview. I highly recommend you download and listen to the audio.

Cheers – Jonathan

How Facebook pushes new code live

16 Comments

This weekend at Hackernews I stumbled across a great video showing how Facebook pushes new code live.

It’s the most advanced push deployment system I have seen.

The video is excellent. If you are into software, looking for advanced build and deploy techniques, or just looking for ammunition to improve your own build and deploy process, watch this video.

Here are some of the highlights.

Culture

We operate at ludicrous speed and massive scale.

Tools are not going to solve your problem.
It’s about culture.

No big fat layer of QA, managers and adult supervision.

500 core engineers.

As a developer you will shepherd your changes out from the time you check it into trunk, to the time you release it out to your mom.

There is no army of people who are going to vet it and check it.
You are accountable.

Subversion and git are used for version control.
UI tests are Watir and Selenium.

Oncall duties are serious. When you are on call, you are the guy.

Branching

Generally don’t branch. All work done off trunk. You work. You checking. Bam. You’re done.

Cut one week release branches.

Your change can go out with the weekly deploy, or you can bump it up into the daily deploy.

Testing

Everyone tests at Facebook all the time.
Anyone can open a bug – there is a Facebook group internally anyone can go to to see the latest bugs, and open any new ones they find.

Everything is automated. Here are few of the tools Facebook uses internally to do pushes.

IRC bots

These bots are there to tell you the state of your push.
Don’t bug a deployment engineer asking where you code is.
Ask the bot.

When you push is going live, the bot will ping you and ask you if you are here.
You are to respond, and let support know that you are here to help if needed.
You are on standby.
For a daily push, if you don’t do this, your rev doesn’t go out.

Test Console

Built there own test console to show the state of there tests.
Use Watir, Selenium, + bug suite of unit tests.

Console will not only show which tests are broken, they will show when the test broke, and who’s change broke it.

Shadow branch

Production + changes + tests

Shadowing prod.
This is the working prod changes changes are merged to.

Error tracking (18min)

php errors. Exceptions. Fatals. All the things going wrong.

Can see calling stack for all errors on the site.
Will show subversion blame for that line of code.

Gatekeeper (24min)

This is the tool/process that impressed me most out the the entire press.

Gatekeeper always Facebook to incrementally push changes to the live website, and then turn them on or off in very complicated selected ways (basically a big conditionally).

For example, you could push some new changes live (that you aren’t about) and then only expose them to:
– Employees only
– By country
– Age
– IP
– East coast/West coast
– Anyone but TechCrunch 🙂

You can bump public up ot 1%. In minutes you will get a million hits.
You can grab the data, turn it back down. Make changes. And then turn it on again.

Super cool feature that let’s you ease it out.

Push Karma (25min)

Basically a Karma system where by you are assigned a Karma score (4 stars) and every time you screw up a push, you lose Karma.

Great way for putting accountability on the engineers to make sure their changes make it live OK, and don’t cause the build engineers any pain.

HipHop for PHP (29min)

PHP compiler.
PHP is crappy and slow. So they make a compiled version of PHP.
Generates highly optimized C++ and converts into giant 1 GB binary – which is Facebook in it’s entirety.
Takes a couple minutes.
Savings are 50% performance boost.
Less hardware required.
Open source.

BitTorrent (31min)


Facebook pushes it’s 1GB binary of compiled PHP to it’s 10,000s of servers using BitTorrent.

Very cool. Rack affinity – looks locally first before going out to neighbours.
Ridiculous data speeds.
Can roll Facebook.com in about 15min.
Whole site.
Incredible.
Minimal user impact.

Summary

Tools alone won’t save you.

But you need the right people.
But you need the right culture.
But you need the right company.

Watch the video.

Why you may find your software lacking

1 Comment

On my drive to work this morning I was listening to the excellent Pragmatic Podcast interview with Miles Forrest interviewing Uncle Bob when Miles asked Bob:

Why is it that sites like Facebook can support so many features and can be so easy to use and yet the vast majority of software systems are so ridiculously complicated and can only support a few thousand concurrent users.

“This is a very good question.” chuckled Uncle Bob.

Uncle Bob went on to explain that when you bring the people building the software closer to the people needing it good things tend to happen. Basically, agile.

And while I wholeheartedly agree with Uncle Bob’s answer, I have a few other ideas why companies like Facebook, Google, and Amazon get software right while the vast majority of us (companies) seem to get it wrong.

1. Most companies don’t live and die by their software.

For the vast majority of companies out there software is a necessary evil.

Given the choice most would rather buy their software then have to build it themselves. This attitude and lack of software expertise means most companies don’t put their heart and souls into the quality of their software and system the way say Google would.

Because these companies don’t see their software as a competitive advantage, they treat it like a deprecating asset, and let it decay over time.

2. Too many rely on QA departments.

Did you know every engineer at Facebook supports 1.2M (that’s million) users on average and they do it all without a formal QA department?

They don’t wait for someone from QA to submit a bug. Every engineer takes direct responsibility for the code they produce and works closely with operations to ensure that what they ship works, and they to support it quickly when things go wrong.

Amazon does something similar. They call it the two pizza rule.

I am not saying most companies shouldn’t have QA departments (it’s probably a good idea for the vast majority of them to be there). What I am saying is that until they start thinking about how to write software such that they aren’t needed in the first place, they are going to continue to build lousy software.

3. Too many ship their org charts.

I once worked at a start-up that had no less then six segmented, vertically sliced silos of teams each responsible for a different portion of their application. Six teams for one product.

Shipping your org chart is natural for most companies. It means you don’t have to coordinate and talk to others. You are the master of your own little silo. And your bonus certainly isn’t contingent on someone else meeting their goals.

Microsoft is currently falling prey to this. As Ray Ozzie recently pointed out in his farewell address to the company, Microsoft runs the risk of being left behind unless they can get their act together and start shipping integrated products beyond the PC. The have to stop shipping their org chart.

4. Most don’t see the value of software talent.

I know this sounds obvious but the quality of your product is directly proportional to the talent of the people you have building it. This cliche is compounded x10 when it comes to software.

Contrary to popular belief software is more then just typing.

You don’t have to explain this to companies like Microsoft, Google, and Facebook. They are currently locked in a waging war for talent. Anyone doubting the impact a single developer can have should ask Google where GMail, Googletalk, and Googlenews came from.

Google Making Extraordinary Counteroffers To Stop Flow Of Employees To Facebook

So that next time someone asks you how Google can organize the worlds data, and your company can’t seem to push a single release of their software live, check the size of your companies QA department, see if your company is shipping their org chart, and ask yourself if you truly live and die by the quality of your software.

Create great looking mock-ups with Balsamiq

1 Comment

Several people have asked me what tool I used to create the screen shots used in the YAGNI vs Usability post.

The tool is called Balsamiq. And if you are looking for a simple, quick, easy way to create great looking screen mockups, it is a tool I highly recommend.

Dead easy to use

One of the things I like most about the tool is how easy it is to use.

With its easy drag-and-drop interface, designers could learn a lot from just studying how Balsamiq was built and designed.

mockup1

Good plugin community

And because Balsamiq is so open and configurable, there is a decent plugin community for Outlook, Facebook, and iPhone apps.

mockup_2

Making mock-ups is just fun

For no other reason, I like this tool because it’s funky, fun, and actually makes the creation of paper based looking mock-ups really cool and fun.

So if you are in the market for a tool like this, check out Balsamiq. It is well work the low price of $79.