MySpace’s death spiral due to no test environment, no source control, no code review

I just saw an extremely insightful comment by Nick Kwiatkowski quoted whole by the High Scalability blog. Unfortunately, they buried the lede very deep, while I don’t trust the original medium of Disqus comments to have any longevity.

Here’s the comment for posterity:

Having been in a position where I was able to work with some of the programmers who worked at MySpace, the issue wasn’t the engine (whether it was on ColdFusion or .NET), it was the environment they choose to breed for their developers.Management would say “We need X feature NOW to remain competitive”. They would then select a group of developers to implement that feature. The biggest problem was they didn’t allow the developers to have staging or testing serversthey deployed on the production servers on the first go-around. Sometimes these developers were given 5 or 10 projects that they had to deploy in very little time. And all this with no change management or control. No versioning either.

MySpace management never wanted to go back and review code or make it more efficient. Their solution was “more servers”. They ended up hiring a crew whose sole job was to install more servers. Meanwhile they had developers checking in buggy code and they were racking up technical debt at an alarming rate. At the time MySpace was running two major versions of their application server behind what was recommended for use. When Microsoft & New Atlanta came around, they jumped at the idea to essentially sell off their technical debt (like a mortgage to a financial firm), and have somebody else take care of their problem.

The problem then was Microsoft was not updating their old code, they simply were adding new features on .NET. This didn’t solve their problems and left them in a situation where they still needed to fix the old stuff, all the while updating new code.

The issue with MySpace was this : they are a classic example of when you don’t listen and you accumulate too much technical debt. Fixing old stuff should be a priority, and doing things like change management, version control, testing and development servers, etc. are all a must. This is why the bookface is able to deploy new changes with little impact — they have everything tested and proofed out before they let their actual users play with it.

(I will take this down on request. I appreciate that quoting something whole is generally a bad idea.)

Maybe this is just my bias as a developer, but to me this fully explains the failure of MySpace and the success of Facebook. The latter has a reputation of following good software engineering practices — separate development, testing, and production environments; source control; and so on. The former turns out to have gone out of its way to sabotage their own software.