The College Way

~“Scrum is only useful in a college programming environment”

When I was in college, doing full-blown project management seemed like a far-off dream that was only available to “professional” software engineers.  I had studied hard about the science of software engineering. I read all those books that talked about the 5 phases of software engineering: requirements, design, implementation, verification, and maintenance. It was clear that this was the process that “real” software engineering projects used to achieve success.

In the “real world” of software engineering, a development team would take months putting together the perfect requirements document that would drive the entire development project.  There were many different ways to denote requirements and an entire language to describe them:  BRDs and MRDs, Use Cases and UML diagrams. These requirements would be paired with a development timeline that would perfectly, and elegantly, lay out the entire development schedule. Requirements documents would flow into Gantt charts, Gantt charts would describe features and dates. PMPs would feel the rush of victory as the development team put together meticulously thought-out designs and fed a team of programmers with requirements as the software architects churned-out technical specifications. I knew in my heart that this is how aircraft control systems and rockets were built!

I read deeply into Frederick Brooks’, “The Mythical Man Month” and the first edition of Steve McConnell’s, “Code Complete.” I learned about requirements and design, estimation and planning. By the end of my studies in college, I felt that I had read enough about the foundations of software engineering that I would one day grow up to be a fine programmer. Unfortunately, none of this mirrored my actual college experience building software products for my various engineering classes.

My real experience building software in college went something like this: time-box everything and do all the “required” stuff first. As a student, you know when your project starts (you get the assignment) and you know when it ends (the due date). There was no “letting the date slip” in this environment.  Failure to deliver working code resulted in a grade of “F.” Because you couldn’t let the date slip, it was important to focus on the most critical functions first. Do the 20 percent of the work first that yielded the 80 percent of what the professor was looking for in the assignment. If there was time left over, you would focus on the “polish.” This sometimes left time for other activities like LAN parties, sports, or dating.

This programming work was usually done alone or in small teams; it was the team assignments that were always the most challenging. The team would take the assignment (requirements), break it up to a high-level design, and divide up the work that needed to be complete by feature or technical area. Then everyone would code like crazy to get their part done and merge all the code back together. Then test everything and fix any obvious bugs. This usually left the minimum amount of time for documentation, which was always required for a project submission. So the documents would be very focused on specific aspects of the software and usually done at a high-enough level that if the software had to change at the last minute, the documentation would still be accurate. This documentation would basically be used to remind myself what I did and explain to the professor the purpose of the software and design. I did a lot of diagrams and high-level feature explanation back then. This “just get it done” mindset, where we tried to get the most important requirements from the assignment complete and submitted to the professor before the due date, did not fit into my vision for how software engineering should really happen. This was just “college work.”

Fast-forward to “the real world” of software development in the mid-to-late 90s. Before the Internet bubble burst. Back when IE3 and Netscape were battling it out on the desktop for the soul of the Internet. Back then I was working in Seattle at a company called IDX Systems. IDX had recently purchased a company called PHAMIS in the mid-1990s, and I was a widget in that acquisition. I had worked on a number of different projects, and they were all run in a similar way. Start with the business requirements document (BRDs), put together technical specifications, estimate timelines and put these timelines and milestones into Microsoft Project. This was followed by development and testing, then into the maintenance phase.

This was just like I expected software engineering to be done when I was in college, except for one thing – the results were not what I had expected. Instead of an elegant flow from requirements to software, all flowing through a tightly controlled project plan, I had encountered the realities of software projects. Pinning-down both the requirements and the release dates lead to a number of strange circumstances. Often times the release date slipped. It simply took longer than expected to get all the requirements completed. Sometimes there were “missing” requirements and a special team had to be put together to do the gap analysis, leading to further project delays. Sometimes the projects were released on time, but since the development time ran longer than expected, some “optional” phases were compressed, such as testing or documentation. Was this better than slipping the date? Who knows? At least we were able to say we delivered all the features on time. Sometimes all the pieces would come together perfectly, but this was a rare event worth celebrating.

IDX was a company who had its headquarters in Burlington, Vermont. One day in 1997, the technical leadership in Burlington sent a guy by the name of Jeff Sutherland to Seattle to talk to the engineers about a new way of doing software development. He had this “crazy idea” that software development should be done more like I did it in college and less like we did it in “the real world.” He didn’t call it “the college way,” like I thought it mirrored. Instead, he called it “Scrum.”

Developing business-focused software applications is more like programming projects in college than I would ever have imagined. It is important to deliver software early and often to get that critical feedback needed from the business owners. Did the new features meet their needs? Have their needs/wants changed with the delivery of new functions? The only way that you’ll know if your customer’s will give you an “A” is by ensuring that you are on-track with their needs and vision. I know that in college, I regularly discussed the state of my projects with my professors to try to understand what was needed to achieve an “A” in the course.

Time-boxing the development (called the “sprint” in the language of Scrum) is as critical in professional software development as it was in a college classroom setting. When you say that you are going to deliver software, you’d better be delivering software. Failure to deliver is one sure way to get an “F” from your customers, especially if they have changes to their business processes dependent on a software release. What exactly will be in the hands of your customers? It’s sometimes hard to say, because it is possible that not everything that was put into your backlog was completed. But one thing is likely: if you focus on doing the most critical features first, your stand a good chance of making your customers happy. Something is better than nothing, both in college courses and in software development. And if your customers derive 80 percent of the value from 20 percent of the features (like most customers do), then delivering features early and often will provide maximum benefit, in the same way that this allowed your professor to provide maximum guidance to your programming class software project. Additionally, time-boxing the development cycles will also leave time for other activities, such as keg parties, family functions, or more time at the Aikido dojo.

One important factor that allows professional software development to be run more like college is the rise of web-based applications run over the Internet (or intranet) in the browser.  Since, in this environment, every user can be upgraded or updated simultaneously with very little effort, it has changed the economics of software deployments. In the web space, there is no ability for users to “opt out” of an upgrade or to have multiple users with different versions of the software. The software is just a single version, and it can be refreshed very frequently so it makes sense to release software as often as possible. In this environment, bug fixes have a lower cost, the benefits of incremental product improvements can be realized almost immediately, and users can be involved in constant, ongoing self-training on these incremental improvements rather than waiting for a huge class on new versions.

In this way, software – like people – is constantly evolving and improving. It is less focused on the “end product,” and more focused on continual improvement; both for the software features and for the customers who consume them. Constant incremental improvement of the software allows the software to closely mirror the needs of the customer and follow them as their needs and desires change. To me, this is the Zen of the Scrum development process; process over product.

Welcome

Welcome to Zen and the Art of Scrum! I don't mean "welcome" like I would say to my best friend who is visiting after a long absence, because I really don't know you that well.  I also don't mean "welcome" like I would say to a perfect stranger who I am meeting in a clandestine way on the first day in some 12-step program. This "welcome" is somewhere in between; it is a welcome of shared knowledge and experiences that can only be achieved after working for years in this crazy, turbulent, and wild world of professional software development.

I, like you, have been fighting the battles and driving towards success in the non-stop, fast-paced, and usually insane world of software development. I have been working with software for most of my life. In the time that I have been doing this, I have seen technologies come and go – only to return again like a rewrapped and regifted Snuggie at a white elephant party. I have also met many brilliant minds in the field, some of whom have burned-out and moved thousands of miles away to escape. I have seen – and been a part of – some great software successes and a few total catastrophic failures. I tend to obsess mostly on the failures. These are the battle scars that all software development vets wear with pride. It is these experiences that let me understand the complex and often misunderstood art of managing software development teams.

As an American, I know that we have trouble driving in certain situations. This is especially true when encountering a “roundabout.”  A roundabout is elegant in its design and brilliant in its simplicity. It’s a circular junction where multiple roads converge. It is simple in that you only need to yield to the traffic to your left (or right in the UK) and you can connect to any of the adjoining roadways. It keeps traffic flowing, reduces stops, and even saves on fuel. Unfortunately, it is almost totally alien to Americans and for that brief moment when one is encountered, completely baffling. I once sat on a patio at a Gloria’s restaurant that overlooked a roundabout in the middle of an outdoor mall in Garland, Texas for almost 2 hours watching the traffic. It was possibly the most hilarious study of traffic patterns ever devised (of course the sangria didn’t hurt). I watched cars stop before entering, which backed-up traffic for blocks. Time and time again, I saw cars that refused to yield, almost causing a pileup. I even caught a few cars that went the wrong way in the roundabout! If you can think of a wrong way to use a roundabout, I’m sure there was a car using it on that day.

Why does all of this roundabout stuff matter? Well, because many times business leaders and IT managers have grand visions for how they will run their software product teams. They have great maps about how much ground they will cover and how fast they will get their product / enhancement / customization to market. Unfortunately, these lengthy, tightly-designed road maps often overlook the simplest details, and the development teams fail to reach their destination because they get stuck in a roundabout somewhere along the way. They spin round-and-round on the path instead of focusing on the destination. It is important that everyone who is involved in the creation of software in these teams understand the processes and expectations (i.e. rules of the road) before a development team can truly be successful. Unfortunately, the rules of the road change in every country, and the best software development practices tend to be slightly different at each company, and within each development team.

In recent years, "agile" has become the software craze. "Go agile and everything will be better" is what development teams and software managers sell to their executives as a way to get better results in a faster timeline. To many under-experienced business folks, “agile” means “go faster”, which in turn means “more profit.” It is easy to take aim at the neophyte, “non-programmer” types in business as some type of necessary overhead, obviously unaware of the needs of the development community. But it turns out that these folks who do not contribute a line to the codebase actually care about something meaningful; winning against the competition which wants nothing more than to put you (and your company) out of business. For the business leaders, it is important to understand what agile is, and is not, so that its benefits can be realized in an organization focused on software and profit.

To some in the software development world, “agile” means “no processes.” I have interviewed hundreds of prospective employees who’s resumes assure me of their “expertise in agile.” But many of these candidates seem to think that agile means that they write no documentation and meet every day for 15 minutes. Although it is certainly possible to be a successful software company only following these two rules, I have always felt that it was important to understand why I was doing what I was doing – not to just do it.

For many software professionals who have read deeply on agile and followed its steady rise over the past decade, the agile philosophy just seems to Make Sense. Trust people over processes. Working software is the best form of documentation. Stay close to your customers and adapt. Unfortunately, many of the truisms of these agile approaches are completely lost on the decision-makers within an organization. Many times, some of the details are even lost on the teams! A battle-cry of "go agile" that yields an initial failure can have catastrophic consequences to a development team, or even an entire IT organization. In this blog, I hope to explore some of those truisms in an irreverent and whimsical way, and try to get to the underlying reality that is the processes and management of software development teams.