To help you assess the current state of your software development processes, Borland has compiled a list of the top ten integration and testing blunders. Compiled through consultation with industry experts and our own work with countless software organizations worldwide, the top ten list can help you identify areas of weakness that are holding your business back from accelerating the development of reliable software products.
Blunder #1: Lack of Visibility
While it’s very common to see businesses adopt open source software today, it’s rare to see those businesses adopt any of the practices that help make open source software successful. One key characteristic of open source development projects is visibility – everyone can see what everyone else is doing, fixing, or breaking. Visibility encourages both peer accountability and recognition of individual contributions, which is especially important when working with distributed development teams (i.e., the right hand often doesn’t know what the left hand is doing because it’s asleep when the other one’s working). At the same time, visibility allows all project members to learn from one another and improve their skills over time.
But few development teams have the visibility they need today. Typical managers, for example, still use three variables to manage software development projects: 1) requirements (what do we need to create?), 2) a calendar (how much time will it take to complete?), and 3) a defects list (how many defects are in the code?). To estimate how much more time is needed to complete a project or phase, they must rely on ballpark estimates provided by project leads – essentially hunches that are based on experience, not fact.
The Solution: Visibility
But imagine if you had access to data that provided accurate, detailed insight into the development process – for example, you could see that code complexity is increasing and code coverage is decreasing. Armed with this insight, you could better assess where things really stand and work with project leads to proactively address problems. You would also know when teams are working well and projects are increasing in terms of stability, quality, and timing. Armed with this knowledge, you could create a positive team environment that celebrates individual and team accomplishments. Your software development organization would be well served by adopting practices and technologies that help increase visibility. This could involve, for example, publishing build and test results in a central location for everyone to see, or sending out automated notifications when certain thresholds are reached (either positive or negative).
Blunder #2: Ignoring History
If you regularly build and test your software but don’t store the results in a database, you’re missing the opportunity to gain valuable insights into your development process and software projects. There’s a wealth of information in the form of build reports, test output, and other artifacts just waiting to be mined if only they were stored in a convenient way. Even relatively simple queries could uncover previously unknown relationships or dependencies. For example, finding out that half of your build failures occurred after someone touched an obscure configuration file would give you a fast, simple way to optimize your build process.
The Solution: Capture and Mine Data
As a best practice, implement software that helps you turn data into actionable information. You won’t gain much from analytics software that simply captures and reports on granular detail or provides isolated data points. Choose a product that helps you see trends relating to complexity, code coverage, duplications, check-in frequency, and more so you can see how your project is getting better or worse over time across key metrics. This kind of insight helps you change the behaviour of your engineering team – and improve your software development process over time.
Blunder 3: Relying on ”Gut Feel” to Determine Release Readiness
As a manager, it’s not enough to just cross your fingers and hope that a release is really ready for prime time. Why? Because releasing something before it is ready can have significant consequences, such as higher maintenance costs, an increased likelihood of costly re-work, and unhappy customers. True, your business users do not care (or even understand) “unit test code coverage” – but they do understand when applications are buggy and make it harder for them to do their jobs. At the same time, if your customers can’t count on your company to reliably repeat releases of some basic level of quality and business value, you risk losing the confidence of your users and ultimately lowering customer satisfaction.
Your comfort level and confidence in software readiness needs to be based on what you know about your code and other software artifacts – in other words, what you can actually measure and quantify, not what your gut instinct tells you. Anyone involved in software development would agree with this. But all too often, development organizations lack an explicit process and set of measurements to ensure a release is ready to be shipped. And because they lack proper visibility into their software artifacts, it’s impossible to make fact-based decisions about release readiness.
The Solution: Implement an Metrics-Driven Release Readiness Process for an Official “Stamp of Approval”
Establish an explicit release readiness process driven by unbiased metrics – similar to the check list a pilot needs to go through before certifying that a plane is ready for take-off. Keep it simple by coming up with a set of core metrics that actually affect the quality and business value of a release. The metrics you choose must be immediately actionable. For example, by simply setting a team-wide target of 80% compliance (with defined coding standards) and publishing the current rate on a regular basis to the entire team, you can have a remarkable impact on team performance.
Equally important, select metrics that you can receive as part of an automated process, as this prevents massaging of data. Keep records of these metrics over time so that you can better understand what’s required before you give a product your “stamp of approval.” (For example, you may find that you have lower maintenance costs on projects where unit test coverage is at least 75%.) And don’t forget to publicize what your actions will be when certain metrics are met or missed. For example, what needs to happen if you set an initial unit testing threshold of 80% and you dip below that? When everyone understands what’s required to “right the ship,” you’ll have a more reliable, effective software delivery organization.
Blunder # 4: Building Infrequently
According to Steve McConnell, author of Code Complete, only 25 to 30 percent of software development organizations are scheduling daily build events.3 And this means that the majority of software development teams are building far too infrequently. Why? Because most developers have degrees in computer science – not software engineering – and are trained to focus primarily on coding features. They graduate with little or no exposure to build automation software or source code management systems, which are essential to working efficiently and effectively with others on a software project. And finally, putting pieces of code together is simply not a problem that interests engineers, who enjoy focusing on solving problems.
As a result, building frequently is often a low priority. It’s not uncommon for organizations to schedule a massive integration effort after weeks of developer work. And the result is usually the same: a painful software build process that takes days and even weeks to complete and results in structural misalignment across the software. In addition to breaking the build, these types of speculative, big-bang integrations can generate a host of problems, including the following:
• Progress comes to a grinding halt as developers scramble to fix the build.
• Costs can skyrocket as developers sift through massive amounts of code to find and fix errors.
• QA teams may waste time identifying bugs that are really the result of developer conflicts that should have been resolved before testing began.
• It’s difficult to plan a project around such high-risk events. For example, they can’t predict how long it will take to integrate the various pieces of code, address errors, or finalize the product for release to customers.
• It’s impossible to gauge the quality of the software. Big-bang integrations only provide development teams with quantitative information about the build (for example, did the code compile or not?) – not quantitative information regarding code coverage, compliance with coding standards, and completion of unit tests.
The Solution: Continuous Integration Enabled by Build Automation
Most of these issues can be avoided by building frequently – preferably continuously, but at the very least daily. Building software is a critical validation step because it’s hard to know if something is broken until you try putting the software together and running it. Building frequently gives you greater confidence that you know the true state of your software development project at any given point in time.
With continuous integration, the work completed by your developers is checked in every few hours and added to the main line repository. Ideally, developers first perform automated builds on their own development machines by pulling down a working copy of the main line code, integrating their changes, and fixing problems until they achieve a good build. When the build is successful, developers can start to commit their changes to main line repository. As a rule, if the mainline build fails – which is often caused by conflicts with recent changes made by other developers – it must be fixed immediately so that all developers on the team can continue to work from a stable base of code.
You can optimize the efficiency of your development team by using software to automate the build process, as it can be a time consuming and potentially error-prone process. As a best practice, you should include everything in the automated build – from compiling to loading schemas onto databases – so that anyone can check the sources out of the version control library and use a single command to have a running system on their machine.
Continuous integration allows everyone on your team to work from a shared, up-to-date, and stable base that’s bug-free and works the way it’s supposed to. Ideally, the build automation software should also help you enforce best practices around testing and integration by, for example, automatically performing unit tests – tasks that are often overlooked during manual builds.
Blunder #5: Not Validating Changes Against Target Environments
When customers have a problem with your software, the last thing they want to hear are the words, “Well, it worked on my machine.” Developers often assume that their local environment is an acceptable proxy for the “real world.” And this oversight is the hidden cause of many problems that ultimately lead to high support costs, low customer satisfaction, and lost customers. It’s unrealistic to expect that developers will have the right setup on their desktops to fully validate their changes, especially when they need to build and test on multiple platforms or have test environments that require external resources.
The Solution: Server-Based Builds
As a best practice, you should validate code changes regularly against all of the environments that your product is intended to support (operating systems, application servers, databases, etc.). Each target environment should be maintained on separate servers and duplicate customer realities in as many ways as possible. It’s equally important to set up an automated process that publishes mainline code to the different servers modeling each environment. Automated build and test tools make the validation process more efficient by speeding the build process, ensuring codewise integration across a build, and making the build process structured and predictable, rather than ad hoc. This predictability allows you to, for example, regularly schedule events for publishing the most current mainline code to various servers. Your team can then run multi-day tests on each stack, validating assertions that a particular build runs properly on different target environments. And if it doesn’t run properly, your team has time to fix problems as they come up – rather than just before you go to market – which reduces development costs and ensures that you can release the product on time.
Equally important, you should be able to reproduce the target build environments in their entirety when deploying each new build – from reloading the database to making detailed configurations. This is vital to developing a more reliable build process. It’s not uncommon for developers testing code on a different environment to make small tweaks to it in order to have the build run properly. If these tweaks get lost in the rush to code the next feature – and there can be many tweaks over time – future builds won’t run. And when tweaks are not reproduced in the build each time, they are quickly forgotten by those who first made them.
Blunder #6: Not Testing the Build
Many organizations fail to take advantage of the build process as an opportunity to test earlier, more consistently, and for a wider range of potential problems. They’re satisfied when code passes unit tests and compiles successfully. Unit-level tests are certainly vital to the development process, but they don’t catch certain types of bugs or serious issues relating to functional integration, for example; it is entirely possible to have developers writing perfect code that does nothing to solve problems or further the project functionally. At the same time, you need other types of build-level tests to help your developers pay more attention to issues such as coding standards, code duplication, code coverage, dependency analysis, and complexity monitoring. All of these areas can impact the quality of your products, as well as the maintainability of the product’s code over time.
The Solution: Automated Build Validation Tests
One of the best ways to address these issues is to include comprehensive, automated build validation tests in the build process. You can create self-testing code that automatically tests the majority of the code base for you. Automated software inspection and build validation tests can detect regressions and other problems early, when they are easier to find and cheaper to fix (not least of all because the changes in question will be recent enough for developers to remember what they were thinking when they wrote that piece of code). When defects live for a long time in the code, lots of other code becomes dependent on that defect. So, when the defect is finally found and fixed, other dependent code breaks – and development costs go up.
You should be able to initiate the self-testing code process using a single command, and failure of any single test should cause the build to fail. Plan ahead for which tests should be performed as part of every build and which should be performed on a nightly or weekly basis. Tests that can be performed quickly should be run as part of every build event. Extended tests that may take an hour or more – and would significantly delay build processes for developers – can be scheduled for the nightly build, when they are least disruptive to your team.
It’s important to note that 1) you can’t count on tests to find every bug and 2) automated build validation tests are only as effective as the unit tests that your developers create and the code inspection tools that they choose to deploy. The value of automation is that it helps you enforce build-level testing as a best practice, rather than leaving it as an option for developers rushing to complete the next new unit of code.
Blunder #7: Interpreting Code Coverage in Isolation
The most frequently cited code coverage-related blunder is probably “wasting time trying to get all the way to 100 percent code coverage” or “believing that very high coverage means your code works.” Code coverage is actually a metric that tells you how much of the source code has been tested and gives you an indication about how much confidence you should have in the quality of the product being produced; as developers write code, they should be creating new tests to prove assertions that the code will build successfully and meet functional and other requirements. It’s not uncommon for development teams to hide their code coverage figures – usually because they know they are not as high as they need to be.
Generally, if code coverage is staying the same relative to the growth of a project, that’s okay. If it’s going up, it’s even better. If it’s going down, then you have a problem. And if you have spikes, you should be able to correlate to them to events in the project – such as a large introduction of new, untested source code developed by outside firm that requires extensive testing internally.
But high code coverage does not guarantee quality, as the quality of the tests can vary, and developers simply can’t test for everything. From a management perspective, you also need to look at relative code coverage – a commonly overlooked issue. If you can’t compare coverage between different projects, then you won’t know where scarce testing resources will be best spent. At the same time, if you don’t know (or can’t remember) what your code coverage was last week, last month, or on the previous version of the software, it’s hard to evaluate today’s number. Code coverage is far more useful when you and your team can view it in context.
The Solution: Relative Coverage and Trend Analysis
As a best practice, define a target for code coverage that’s properly aligned with 1) the complexity of the product being developed and 2) how often that code is accessed by the application. It doesn’t do you much good to increase code coverage for simple code (e.g., with a cyclomatic complexity of less than three) because the chances of something being wrong is quite low. But as complexity increases – or as code is accessed more frequently – set higher expectations for code coverage. For example, plan for one test for when code complexity falls between three and ten, two tests when it falls between eleven and twenty, and so on.
In addition, you can create a positive feedback loop by giving developers real-time insight into code coverage (for example, via a live web dashboard – not just weekly reports) and providing incentives for increasing coverage (such as a catered lunch when the target it hit). Ideally, your build automation software should monitor code coverage for you in real time and display code coverage data for all developers to see.
Blunder #8: Infrequent Check-Ins
Making large, infrequent check-ins is not just a recipe for losing lots of work when you accidentally corrupt your hard drive or your laptop battery explodes. It also limits the value of other best practices that you may implement, such as frequently executing and testing the build. Continuous integration is only as continuous as the version control activity driving it. If your developers only commit your changes once a day, there’s little point in doing anything more frequent than daily builds. If you check in your changes less than once a day, you significantly increase the likelihood of code conflicts with other developers. And the more time that goes by, the greater the risk and the costs associated with fixing problems. You also lose visibility into how code is being developed over time. For example, when a large chunk of code is checked in by a developer, it’s hard for you as the manager to know whether that employee just stayed up all night and accomplished a lot at once, stole the code from an unapproved library, or included open-source code that will significantly change your licensing requirements.
The Solution: Frequent Check-Ins
As a best practice, commit single code changes as they are completed and manage each one in isolation. Checking in smaller revisions more often makes it easier to keep track of your work and track down the cause of any developer conflicts that are discovered. It’s not unusual for developers to avoid frequent check-ins because they are worried about breaking the build. You can remove this fear by implementing best practices such as sandboxing, whereby defective code is quarantined until defects are fixed. Sandboxing can even be used to encourage more senior developers to perform optimistic check-ins, during which they perform only minimal validation on their local environment before committing changes to the server. As long as these developers are disciplined about fixing defects immediately once they are detected by automated unit and build validation tests, this approach can save time and increase efficiency.
Blunder #9: Not Publishing Builds
Collaboration is the key to successful software projects – and working code is the best artifact around which to collaborate. But less mature development teams often fail to publish builds in a central location so that, at a minimum, everyone can access the latest executable version and run it. Instead, they may rely upon emailed versions sent by the person who made the most recent changes. This is a recipe for disaster, resulting in version control problems that can be costly and time consuming to fix. You may have one version being used as a demo for by the sales force, different versions being tested by QA staff, and several others being updated by different developers simultaneously.
The Solution: Build and Distribution Management
Giving other developers and QA engineers easy access to the latest builds helps keep everyone in sync. And best of all, it’s easy to do. Simply determine a well-known location, define a self-describing numbering schema for files, and publish all builds there so everyone can access them. Ideally, you want your build system to automatically publish successful builds so you don’t have to deal with managing distributions manually. And be sure to also archive your build configurations so you can reliably reproduce customer issues on older versions of your software.
Blunder #10: Relying Only on Unit Tests
For development teams that have historically decoupled development from testing – or waste time manually performing tests – automating unit tests is a good start to improving the development process. But if you limit yourself to only doing unit testing, you’re missing an opportunity to catch a much broader range of potential problems.
Since typically up to 80 percent of the cost of code is tied up in maintenance, it pays off to take a closer look at your compiled code. Don’t make the mistake of thinking that just because code compiles successfully that it is quality code. Compiling code is like using spell check on a document – you can use it to detect incorrectly spelled words, but you won’t catch glaring grammar problems or instances of repeat paragraphs. Similarly, unless you run a broad range of build-level tests that break the build when problems are detected, your executable binary could be full of hidden problems.
The Solution: Comprehensive Build Validation
Valuable tools ranging from static analysis tools (such as FindBugs or NDepend) to enterprise products for security analysis, license compliance auditing, and functional testing can help you detect a variety of issues early. They are all forms of build-level testing that can give you the qualitative, “big picture” status of a project early on. Because some of these tests can take considerable time to perform, you may not want to run all of them continuously. But as a best practice, they should be performed regularly and consistently. The best way to ensure that they happen as required is to automate the testing process. Start by making your unit tests part of the source code repository and automating them so that are part of every build event. Then start adding (and automating) other tests as well, such as component tests that allow you to see how units interact together, functional tests that allow you to see how the components interact together, and so on. The goal is to test as much as you can and move toward a more efficient assembly line process for software development.
Source: Borland White Paper, February 2009
