Lean and Agile Software Development

Once in a while, a new idea comes along that changes the way you look at something. It changes the way something gets done. It makes you realize that there may be a better way to do it.

The lean and agile software development process is such a new idea; it changes the way you think how software can be created. It takes the best ways to create software and puts them together. The end result is accelerated time-to-market, maximized quality and dramatically reduced costs.

The traditional processes for software development have demonstrated their flaws over the years. In fact, their failure rates are astounding. Numerous studies have shown that more software projects fail than succeed. In theory, the traditional software development processes promise delivery of projects on-time and on-budget. In practice, that rarely happens.  The lean/agile software development process was designed to address these shortcomings. It is an exciting and proven process that combines sound principles and best practices in a new way to develop software.

Some of the Principles
Drawn from the lessons of the quality movement, the principles of lean/agile development have become widely known and embraced in recent years, because they simply work better than the old ways. Some of these principles are:

• Eliminate waste.
• Build quality in.
• Deliver fast.
• Get rapid feedback.
• Assume simplicity.
• Respond empirically.
• Make incremental changes.
• Ensure daily visibility.

Eliminate Waste
Waste means anything that does not add value. Consider, for instance, the cost of unneeded requirements and the development, testing and documentation of unneeded features (which are often a staggering two-thirds of all features in custom software). Partially done software (unsynchronized, untested, undocumented, or undeployed code) is a form of inventory, with all its attendant costs. And wasting time in developing software puts a company at a huge disadvantage against more nimble competitors.

Build Quality In 
Developers should focus on building quality into the code from the outset, not on tracking defects or testing at the end of the process. Today we have the tools for practices such as unit tests, acceptance tests and continuous integration, which enable developers to avoid most defects before they write the associated code. Building quality in means that the code does what it’s supposed to do at that moment (not that the code should never have to be changed).

Deliver Fast
When you eliminate waste, you increase speed. Organizations can go overboard with very detailed specs, process plans, workflow documentation, job descriptions, and the like; all that can slow development down. Better to have teams of bright, engaged, trustworthy people moving quickly, making decisions, improving processes and solving problems as they write clean code much faster than competitors. And then deliver iterations of that code to customers in small, useful feature sets that help them do their job.

Get Rapid Feedback
The old “waterfall” method of development assumed that the knowledge of requirements exists apart from the coding itself.  But developing software in a lean/agile way itself creates knowledge, and a good part of that knowledge comes from ongoing, rapid feedback from testers, customers and other stakeholders. The developers put that information back into the system, increasing knowledge and improving the next iteration. Of course, delivering small batches of code is much better for getting rapid feedback than the old large-batch, all-or-nothing approach.

Assume Simplicity
Don’t try to plan too much for the future, to design for repurposing. That brings complexity, and complexity makes code brittle and adds significant cost. Instead, focus on doing things right today, e.g., in testing and communicating. Assume problems are simple and easily solved. This approach works for all but a tiny percentage of problems, and the time saved is more than enough to deal with those exceptions. A corollary: don’t automate complexity.

Make Incremental Changes
Make big changes all at one time, and you invite big problems. Instead, make small, incremental changes to plans, code or the team—at the smallest level that makes a difference. A corollary to this principle is the idea that in the early stages of development, we should avoid committing to crucial design decisions that are very difficult to change.

Respond Empirically
We can categorize the ways of transforming ideas into products as deterministic or empirical. The deterministic approach sets out to create an exhaustive definition of the product, and then proceeds to realize that detailed definition. That way leads to complexity and waste. The lean/agile way is to develop a high-level concept of the product, and set up tests and other clear feedback mechanisms on the model of Deming’s “plan, do, check, act” cycle. These offer empirical responses and enable the development team to adapt to changes as it works on the best ways to bring the concept to reality.

Ensure Daily Visibility
Problems in software development can usually be traced to someone who did not communicate something important to the customer, to a programmer, to a manager. The lean/agile approach uses several techniques that can’t be done without communicating among the team and with the customer. These include daily team sessions, electronic dashboards showing relevant metrics, and big visual charts.

Some of the Best Practices
The lean/agile software development process puts those principles to work with a set of best practices. A best practice is a technique that, through experience and research, has proven to lead reliably to an optimal result. In software development, it is a well-defined technique that contributes to a successful step in writing effective code. Its combination of the best practices described below makes the lean/agile software development process so effective. Some of these best practices are:

• Iterative development.
• Automated testing.
• Continuous integration.
• Refactoring

Iterative Development
The lean/agile software development process offers numerous advantages over the traditional methodologies. One of these advantages is iterative development. The typical traditional software development methodology utilizes a waterfall approach whereby the each phase of the project lifecycle is performed and completed before the next phase is started.  So, assume a project lifecycle has the following phases: requirements, design, code, test, and deploy.  A waterfall development approach would look like this:

On the other hand, an iterative development approach would look like this:

So, in a traditional software development methodology, each of the phases is performed only once, within the entire life cycle, and each phase is completed before moving onto the next phase.

In the lean/agile software development methodology, the process is iterative; that is, the project lifecycle consists of short iterations. Within each iteration, a small part of the software system is created. Each iteration includes gathering requirements, designing, coding, testing and deploying for that small part of the software system.  In some situations, the software product cannot actually be deployed into production until the final iteration. In these cases, all but the final deployment will be into a staging area.  Even so, there are advantages in going through each of the other phases iteratively.

For one thing, since a small part of the software system is deployed in each iteration, the customer is able to see parts of the system earlier, as they are created. This ensures that the software system is in line with the requirements and expectations of the customer.

Another advantage of the iterative approach is the ability to accommodate changes in requirements. Often, the complete requirements for the software system are not known at the beginning of a software project. Typically, the requirements change as the project progresses, often well into the project. The traditional waterfall approach does not easily accommodate these changes in requirements, since it assumes that all of the requirements are completed at the beginning of the project. Consequently, as the project progresses and the requirements change, re-work often needs to be performed to accommodate the change. With the lean/agile process, however, it is not necessary to determine the complete set of requirements at the beginning of the project. The software system is designed and coded as each requirement is determined. Therefore, the amount of re-work is minimized, and costs are reduced.

Also, an iterative approach better supports more frequent releases of the software product.  Why are frequent releases desirable?  Well, you don’t start realizing your ROI unless the software product actually hits production.  Since an iterative approach forces you to manage the requirements, design, coding, and testing of features in smaller chunks, it makes it easier to deploy each set of features into production.

Automated Testing 
Lean/agile development uses lightweight frameworks to create tests quickly and run them automatically. These include unit tests and user acceptance tests. That way, the development team catches defects faster. And the savings from catching those bugs early are enormous: it costs 10 times more to fix a defect at each successive stage in the development process, and 100 times more to fix it once the software is in production use.

Continuous Integration 
All the code the development team writes gets compiled, tested (it must pass 100 percent) and integrated continuously. This automated process dramatically reduces the risk that the code won’t work with previously developed software.

Refactoring
Over time, as code is added to and changed, the design tends to break down—software’s version of entropy. Maintaining the code becomes harder, and adding features becomes more costly. Refactoring doesn’t change the external behavior of the software; rather, it improves the code’s internal consistency and clarity, making it more change-tolerant and easier to maintain. Refactoring has proven its value as a way to “build quality in.”

How ATI Can Help
ATI has extensive experience with the practical implementation and extension of the lean/agile software development process. We have seen the effectiveness of this process in many different situations. We have provided workshops, given formalized training, helped companies implement the process, and conducted complete outsourced projects using it. Whether it's helping you leverage this paradigm-shifting approach on your next project, or integrating lean/agile best practices into your existing development methodology, ATI can help you maximize quality and ROI while accelerating time-to-market and significantly reducing costs.