The History of Scrum: How, when and WHY

Understand the reasons, not just the process.

Scrum has been around for over two decades and is helping many people successfully develop new products faster and more efficiently. This framework was officially first introduced to the public in 1995 by Jeff Sutherland and Ken Schwaber, but as the authors say, it’s not anything that hasn’t been done before.

For both Schwaber and Sutherland, the formal Scrum model was the result of many years of experiments and learning. Understanding the process behind Scrum development and the reasons leading to it, will help you understand Agile principles, give you a better grasp of Scrum ceremonies, and will overall help you view the whole process of product development as the authors intended.

From fighter jets to wasteful projects: why we needed Scrum

Both creators of Scrum, Jeff Sutherland and Ken Schwaber, fought in the Vietnam war between the years 1967 and 1975. Necessary everyday risk assessment already influenced their way of thinking and naturally affected their later work. Next, they followed similar career paths and stayed in touch.

Ken Schwaber started his own company in 1985 after working in a corporate environment for nearly ten years. He says the corporate work created a lot of frustration for him, as he could never figure out what was wrong or why it was so hard to finish projects successfully and pull people together to work as a team. At the time, roughly 45% of the projects he was working on were successful, and the rest were considered somewhat a waste. People didn’t like their job, and they kept running away, trying to find anything else.

Only 36% of product features are used.

Schwaber was a waterfall enthusiast and tried to make it work, but he just couldn’t. He knew there was a need for change in the industry. In the paper SCRUM Development Process, he wrote later explaining waterfall is a method that assumes software development is a well-understood process without any unexpected outcomes or problems. However, that is not true; software development is very unpredictable for multiple reasons.

„We were supposed to do a huge project for one million, but we weren’t sure if we could. So based on our calculations, we predicted we would need two years and four million, but it ended being even more. My point is the huge uncertainty in development.“ Ken Schwaber

Jeff Sutherland started his career with a dissertation in medicine and complex systems theory. When later in 1986, he became responsible for the ATMs business unit in Saddlebrook Corporation. Beforehand the unit used traditional project management, which led to late delivery, poor quality, and user dissatisfaction. Sutherland was supposed to fix this. The ATMs unit was problematic and the least profitable branch of the company; in short, six months, he brought it up to be the most successful unit.

„Software development in interesting and fun, it should stay that way.“ Jeff Sutherland

Sutherland compares landing a sprint to landing a plane, slowly descending with each task.

He helped the team by teaching them how to do the process better comparing it to learning how to land a plane. Slowly descending with completing each task and ultimately landing the sprint, creating a burndown chart. In his Ted talk, he explains the success with three important methods he brought to the team: making their work visible, giving a responsibility to the team to fix the problem, and self-organization of the team to make it all happen. This experience showed him there’s a need for a new, better approach to development that would produce happy customers, developers, and early projects.

„Self-organizing teams just do a better job no matter what they do.“ Jeff Sutherland

Looking for a better way, Schwaber later worked with DuPont and was strongly influenced by their advanced process control methods. They advised to shorten the planning period to decrease the risk and instead, try for a shorter period, see what works and what doesn’t, adjust and do it again.

Based on this, Sutherland and Schwaber spent 4-5 years building and researching a new approach to developing software that would be more flexible, give people more control, allow the creative process to bloom, and would allow controlling the risk and expense value. Because software is fun and you do it for fun, you need to support your family and make the industry.

January 1994: first Scrum team is scrumming

In 1993 Jeff Sutherland was hired into Easel Corporation as Chief Engineer for developing a new product. At the time, they didn’t know what exactly they wanted, but they knew it had to be something faster and better than what is already on the market. Knowing this was a huge challenge, he and the team first reviewed the literature to see what was already done and what could be taken beyond.

They found, according to Sutherland, the best paper in Harvard Business Review, The new new product development game (not a typo, authors chose to emphasize the word new) by Hirotaka Takeuchi and Ikujiro Nonaka. A paper showing a new, holistic approach to product development. The authors of this paper compared the new approach to a rugby game – a team moving down the field in a scrum formation as a unit. This paper provided Sutherland with a formal model and the name for a new framework created for the IT industry: Scrum.

The first Scrum team was built on the principles of Nonaka’s and Takeuchi’s paper. Another important factor for building the team was research from Bell Labs that showed how specialized silos of activity slowed down communication and made teamwork impossible. This resulted in having a few roles in the team as possible. Starting with the team and the team leader – scrum master, and later adding the product owner role responsible for clearly specifying the backlog. From here, the first Scrum team started scrumming in January 1994.

In 1995, after Easel Corporation was acquired by VMARK, Schwaber spent two weeks with the Scrum team, and he agreed to expand Scrum with Sutherland outside of Easel Corporation into the software industry worldwide.

In 1995 Ken Sutherland and Jeff Schwaber wrote a paper officially introducing Scrum and published it at OOPSLA’95 (Object-Oriented Programming, Systems, Languages and Applications) conference. They both went on to introduce Scrum to multiple companies like Motorola, Fidelity, or GE Medical, and after seeing great results and creating a community, they co-created the Agile manifesto in 2001.

„We’re not going to tell people what to do; the idea is you’re doing creative work, you need guidelines to help you, not constraint you. So why don’t we come up with a bunch of principles and values, alternatives, that might lead you to what you need to do.“ Ken Schwaber about creating Agile manifesto

New new product development game and principles that inspired

The paper New new product development game published by Hirotaka Takeuchi and Ikujiro Nonaka in 1984 serves as the formal model for Scrum. In the paper, the authors looked at successful companies in Japan and the United States that used a new, holistic method for product development as opposed to an older, sequential approach (like the waterfall). Companies like Fuji – Xerox, Canon, or Honda used different principles when developing a product, which pushed them ahead of competitors.

Takeuchi and Nonaka highlight that the new approach they presented can act as a change agent, a vehicle for introducing creative, market-driven ideas and processes into an old, rigid organization. These teams reminded the authors of sports teams, particularly rugby and scrum formation, in which the whole team moves as one, working together towards one goal around any impediments.

Sutherland believed this paper demonstrated exactly what was needed for development in the software industry and used the holistic approach presented as a base to create a set of rules to generate high-performing teams. The main characteristics of these teams were:

• Autonomy: teams chose the work and decided how to do it. They had challenging goals from management, but they didn’t tell the team how to achieve them.
• Oriented towards transcendence: pulling power from the people and building on their passion.
• Learning from each other: cross-fertilization and creating built-in stability.

„Scrum doesn’t solve a problem; it’s a tool you can use to gain into your excellence. It will tell you how you’re doing by creating transparency in your organization.“ Ken Schwaber

Example of Toyota developing a new car as described in Takeuchi’s and Nonaka’s paper: In Toyota, they needed to develop a car that gets twice the gas mileage, in half the time they ever shipped a car. It was impossible by any approach they’ve used before, so the management said: „Let’s try something new!“. The team working on the new concept started cooperating and asking each other questions, „What should we do? How do we do it?“. This pushed the team into a self-organizing state, increased cooperation, and naturally introduced overlapping development phases to speed up the process.

Challenging the old and accepting the unknown

The Scrum framework challenges the old way of thinking. Here, there’s a possibility that the outcomes are different than predicted, which requires a fair amount of integrity to keep the transparency, admit failure, and make decisions around it.

There was a need for a different system that would be more reliable and responsive to changes. Considering how nowadays our lives are dependent on technology and how software products are being used in our bodies or in planes flying at 10 km above the ground. These products are too complex and unpredictable to be developed like they were 70 years ago.

„When a plane falls, the explosion is bad, so it gets attention. But projects in the bank don’t create commotion, so they keep crashing, and nothing changes.“ Jeff Sutherland